We are having an intermittent issue of delayed menu conversation response time when attempting to retrieve voicemail.
Problem description: subscriber accesses Unity via phone, immediate response for ID/password as well as subscriber name but then there will be a delay of up to 20+ seconds before hearing the menu options. As soon as a minute or so later try again and all is functioning normal. This happens multiple times a day.
Unity system info w/ failover:
W2k3, Unity 5.0(1)GUSI: Patches installed:=CU5.0(1)_ES73, CU5.0(1)_ES78, TSP 8.3(1.0)
Exchange cluster info:
Exch2k7/W2k8 and Exch2k3/W2k3, partner Exch is the Exch2k7
The following is generated in the Event Log on the Unity server:
Event Type: Warning
Event Source: CiscoUnity_AvWm
Event Category: Warning
Event ID: 29021
A MAPI function call to the Microsoft Exchange server took 20328 milliseconds, which exceeds the system threshold of 5000 milliseconds. The most likely causes are an overloaded Exchange server or poor network connectivity to the Exchange server. This is likely to affect the operation of the Cisco Unity server in a number of ways, including slow response or delays during phone calls, ports getting stuck or not answering calls, and Unity failover if a secondary server is active.
To troubleshoot this problem, enable all AvWM micro traces using the Cisco Unity Diagnostic Tool (UDT), and investigate the Exchange server for high system load and other indications that maintainance is required. For more information, please consult the Troubleshooting Guide For Cisco Unity with Microsoft Exchange.
Method IMAPITable::QueryRows (returned 0x00000000) Exchange Server: exch2k7cluster Outstanding Server MAPI Requests: 1 Parent Process: AvCsMgr.exe
Our MS engineer is seeing no errors on Exchange side when this occurs. I have triple-checked that McAfee isn't attempting to scan anything. Have had a TAC case open at P3 level for a while, just requested an escalation.
I haven't found anywhere in the forums where this is occurring elsewhere. I appreciate any feedback on this issue.
Here are a couple of things to check:
1. Check your speed and duplex settings for Unity NIC and switchport and Exchange NIC and switchport. If they are set to auto negotiate, hard code to 100/Full.
2. In the settings for Unity/Exchagne NICs (Right-click network connection > Properties > Configure > Advanced), set Checksum Offload to None.
I ran into an issue at one time where Unity periodically thought the Exchange server was offline even though it was not. The above settings took care of the problem.
Hope this helps.
Thanks for the quick reply!
From the Unity side, I have verified all of the above previously (also due to previous experience but with backups) and all was set as you suggested except for #2. I made that change and see how it goes. I have also forwarded your response onto the MS Engineer.
Well, we found that the duplex/speed settings were set to auto on Exchange ports. Made the change, but have since received the error again.
How is Unity connected to Exchange? Same switch? Multiple hops away? Any interface problems between the two? When the problem occurs, how are ping times from Unity to Exchange?
Same location, one hop between Unity to Exchange. Network engineer has not found any problems between the two.
Haven't been able to isolate a ping time directly to when the error occurred but in a constant ping haven't seen one time higher then 15ms.
Attached is a MAPI latency test our MS engineer did from a laptop in the same vlan as the Unity server to the Exchange server.
Not yet. So far actions by TAC:
1. Re-applied/verified security. Had me re-run the message store configuration wizard again.
2. In Unity Advanced Settings: 'Messaging-72 or more voice portss-number of MAPI sessions per Exchange server', changed current value to '1' on both servers, restarted the pair after hours.
3. Last suggestion, installed ES80.
***Due to an Exch 2k7 mailstore crash last week, the latest rollup has been installed on Exchange (not SP2).
The issue continues. I tried having the case re-queued once and the same engineer maintained ownership? At any rate further action was taken so i didn't push the issue. Our MS engineer currently has a case open with MS as well. If there is no progress from that side my next step is to contact a duty manager and escalate as we are receiving more and more complaints on the issue.
I would greatly appreciate feedback from anyone that has experienced this issue and has resolved or made progress on resolving the issue.
Still no resolution and unfortunately appear to be at a stalemate. TAC's stance is that the error is pointing directly to either Exchange or network issues. MS is saying that there is nothing happening on the Exchange side when the errors occur. The number of times a day the errors occur continues to grow. Still have cases open with both. I will be sure to post any updates. I've started to see others inquiring/posting on this isse so I'm hoping with multiple eyes looking into this the smoking gun will be found.
Had exactly the same error messages 3-4 times an hour. Delays were in the 10-20 sec range.
A few times the Unity-Exchange just went crazy and MWI would not work properly for most users. We then had that MAPI error message 4-5 times a minute and had to restart Unity services to get back to normal.
Some users were complaining about VM delays but not much.
Exchange guys could not find a specific problem but some users had delays while reading their normal emails in Outlook.
TAC confirmed the Unity was behaving properly.
Eventually the Exchange guys upgraded their SAN and all error messages went away.
Not saying this is your problem, but in our case Unity was not the culprit.
Unity UM 7.0(2) + Exchange 2007 cluster.
Latest ES and SP everywhere.
Thank you for taking the time to post your information on this issue, much appreciated!!
Is there any chance you could elaborate on version info in regard to the SAN?
Thank you for your time.
Hello, I apologize for not posting an update sooner. Ultimately this has been the "resolution", have the affected user(s) archive, delete, move, etc. e-mail messages out of Outlook Inbox, Deleted Items, and Sent Items. MS and TAC agreed that this ultimately is causing the delay. Not being an expert, the best way I understood the issue is that Exchange periodically updates an index of message count. Depending on when Unity is accessing a user mailbox this indexing could cause the delay.
I have tested with multiple affected users and had them simply move x-amount of messages to "Inbox-Archive" (for example) subfolder under their Inbox to reduce message count to under a thousand. Note, I found that some of the users had 35k plus messages!!! At any rate this has appeared to resolve the issue. The only .02 I'll add on this is, yes, that is crazy insane to have so many messages, but when partnered with Exch2k3 we very, very rarely ever experienced this issue.
Any experts out there that could clarify on MS and TAC findings may better assist anyone researching this issue.
High Item counts on the Exchange inbox always cause delays on Cisco Unity; back on Exchange 2003, having more than 5000 items will be a problem and more than 20000 items on Exchange 2007 will cause the same issue too; the problem is that when the user access his mailbox through the TUI, Cisco Unity makes a MAPI call to Exchange querying for the amount of voice messages (IPM .Note .Unity) then, Exchange needs to count the amount of messages with that specific message class in order to reply back to this query; the following TechNet article has additional information regarding the impact of high items counts on Exchange inboxes:
If you rarely had this issue when partnered to Exchange 2003, I would start checking the performance and resource utilization of the Exchange 2007 servers.
Hope this helps!