I have a CM 3.2.2c and Unity 3.1.5. All of a sudden, according to the customer, the MWIs stopped working. There are no errors in Event Viewer. TSP is configured and working. There are 16 ports in Unity configured, 4 are set for MWI dial out. CM shows 19 ports configured. I testedwith a subscriber. I called her and left a message, the light did not go on. When I refreshed the status, it went on. Same thing when deleting the message, the light stayed on until I refreshed the status. The port utilization was very low, 1 or 2 ports being used. So, apparently, the MWI on/off codes are correct. The service, however, is not sending out the codes until it gets "pushed" by the refresh status. I did a Resync all MWIs under the IP Switch setting, didn't help. All Unity services are running. Where do I go from here?
There's a specific issue with Exchange that sounds like the problem here.
First off, could you verify that MWI's have worked in the past? This is not a new issue?
If MWI's have been running properly, it's possible a UDP packet recieved by Exchange stopped notifications per these KB links:
If the patch for either the E5.5 or E2K system helps, please post your reply here. If not, TAC should be contacted for escalation.
The MWIs have worked in the past. This is an "all of a sudden" problem. Are there any errors or events associated with the UDP issue you mention? I would like to be sure of the problem before installling the patch discussed in the above link. Does it have the same symptoms that I describe above, only working with the Refresh status button?
The symptom is the only event associated with this UDP issue. Scenario like:
MWI's working fine, then stop with no events and all subscribers are affected.
A stop/start of the MsgStoreMonitor service and/or restart of the server brings MWI features back in service. There is no known frequency for the issue, meaning Unity could run a few more weeks before it happens again.
I stopped/restarted the AvMsgStoreMonitor service and that did the trick. All the MWIs are working now. Thanks. Issue is resolved! Is there anything to look for in the future in case it happens again?
Great - the service restart re-initializes all Unity connections to Exchange so any new msg notifications will be recieved.
Any system that's experienced this issue should have the patches from MS listed in the prior post installed on the Exchange server(s).
If Exchange notifications aren't received even after the patch is installed, we'd need to get instrumented Exchange diags enabled and network traces to take to MS. This scenario has been very rare, but should be escalated if it's seen (even on a frequency of every few weeks).
The problem has returned today, I'm not sure exactly what time, it may have happened over night. Unfortunately, restarting the service didn't help, they still have no MWIs. I saw some errors in the app log everytime I restarted the AvMsgStoreMonitor, they are below. How can I be sure if this is the MAPI thread issue? Are there some traces I can run? Also, this customer is upgrading to CM 3.3 and Unity 4.0 in the next several weeks. Would the upgrade fix it, or is this an Exchange specific problem?
If it's truly the UDP thread issue, there will not be any errors in the event log. The only way to know for sure if it is the UDP thread issue is to take a memory dump of the AvMsgStoreMonitorSvr service and look for code that's executing in the MAPI client (stuff that's beyond the scope of the forum and better handled with a TAC case).
The errors you mention could provide some light, but they didn't make it into the post.
Exchange Monitor traces of "Component" and "MAPI notifications" will help in the investigation. These traces will be found in the AvMsgStoreMonitorSvr diagnostic files.
Without knowing what the problem is, we can't answer if a Unity upgrade is going to fix the problem.
I don't know what I did but here are are the errors. The only other events were TSP and AvDirSync warnings, I don't think they are related:
From the System log
Event Type: Error
Event Source: Service Control Manager
Event Category: None
Event ID: 7031
Time: 4:51:20 PM
The AvCsMgr service terminated unexpectedly. It has done this 1 time(s). The following corrective action will be taken in 0 milliseconds: No action.
From the app log after restarting the AvMsgStoreMonitor service:
Event Type: Error
Event Source: AvNotifier_MC
Event Category: Run
Event ID: 1021
Time: 12:29:39 PM
[Thread 0x00000DEC] Fatal error: Exception caught in CAvNotifier::resyncThreadProc().
Event Type: Information
Event Source: AvMsgStoreMonitor_MC
Event Category: Devices
Event ID: 1009
Time: 12:43:36 PM
The description for Event ID ( 1009 ) in Source ( AvMsgStoreMonitor_MC ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. The following information is part of the event: .
The first one looks scary, but it's not. You can see that sometimes when Unity is brought down. The third one there is actually a goof on our part, it should say somehting about that service has started. The second one is the one that makes me want to say, "OUCH!"
That's the problematic one with regards to MWI when restarting the AvMsgStoreMonitorSvr service. Does that one come up everytime that service is restarted?
Yes, everytime I restart the service, that error pops in to the Event Viewer and the MWIs still don't work. I just spoke with the customer and he rebooted Unity and Exchange last night and then again this morning. When he did it this morning, the MWIs started working. So, for right now, they are working. Is there something else I can do to determine if it is the MAPI thread issue?
I can tell you that the issue with the error message showing up is NOT the MAPI UDP thread issue. If there is an MWI outage where there are no errors in the event log, there's no definitive way to tell. However, there is one very strong symptom: Everyone loses MWI. This would be the case regardless of the home Exchange server the Unity subscribers reside. If the AvMsgStoreMonitorSvr is restarted and MWIs start working again for everyone, then the evidence of the MAPI UDP issue is stronger. If some people's MWI still work while others don't, it's ceratainly not the MAPI UDP thread issue.
To definitively know, analysis of the memory dump of the AvMsgStoreMonitorSvr service is necessary.
It's not a bad idea to install the Microsoft fix for the MAPI client DLL on the Unity server (that's the only place it needs to go) anyway. Sites that had run into the MAPI UDP issue, have successfully installed and avoided further MAPI UDP issues.
This definitely affects everyone when it does "break." How do I get a memory dump of the AvMsgStoreMonitorSvr service? Do I need Cisco TAC to look at it? And what is the name of the Microsoft fix for the MAPI client DLL? Can I get that off of Microsoft's website? Does that mean that I won't need the service pack for the Exchange server mentioned above?
The MAPI client is \Winnt\System32\EMSMDB32.dll. There's an existing DDTS entry on this issue CSCdx43466. The fixes from Microsoft are available at:
Exchange 2000 KB article:
XCCC: MAPI Thread That Is Monitoring the New Mail Notification Quits (Q329024)
The Exchange on Unity will have to be at least SP3 before applying this.
Exchange 5.5 KB article:
XCCC: MAPI Thread That Is Monitoring the New Mail Notification Quits When It
Receives a UDP Packet (Q329415)
The Exchange on Unity will have to be at least SP4 before applying this.
I'd suggest grabbing the appropirate fix and applying it. It does sound like you're hitting this issue. If you want to gather a memory dump, opening a TAC case would be the best bet. Devepoers will have to analyze the dump, it's not something like a text file.
Thanks for all the info. Just one more question. Do you know if upgrading Unity to 4.0 will have any effect on the problem, without installing the fix? Or is it specific only to Exchange, regardless of the version of Unity.
Where the MWI problems show up as you've described (particularly where we don't see Unity event messages showing the failure) - it is specifically an Exchange issue.
Can you clarify :-
If the exchange mailbox stores are located off the unity server, does the MS fix have to be loaded on the Unity server or the Exchange server or both?
Is SP3 for exchange is a pre-requisite?
The fix is only needed on the Unity server. For E2K systems, SP3 is a req, for E55 systems SP4 is a req.