Unity 3.0.3, with RDB patch, TSP-3.0.3 with CCM 3.1.2c.b. MWI's were working until yesterday at approximately 14:20, the only things in the event log up until that point are LicenseService errors for Exchange 2000 and Win2k, telling us that we are out of licenses and to use License Mgr to see how many more we need. The first failed MWI is noted in the event logs as: EVENT ID: 1045; "Failed to set Message Waiting Lamp for CIT, Test, EXT 2364322, Switch 0, Cluster 0, Reason: Set MWI failed with 0x80004004"<br><br>The problem was noticed this morning and here is what we have done for testing...<br>First we verified that our TSP and CCM MWI settings are correct, that was ok. Then attempted to lamp by dialing the MWI code at the phone, which is successful. Then looked at the test subscriber to make sure that MWI was turned on, it was set to lamp "X". We added, as a secondary, the same number that appears in the "extension" field on the profile page, so that both "X" and "2364322" appear. That clears the problem and MWI's turn on and off as expected. When that secondary MWI point is removed, the lamp ceases to function.<br><br> As a side note, the time of the first event in the Event Viewer corresponds closely to another problem we are experiencing, where users connecting via terminal services and trying to connect to their phone to administer recordings receive, "Unknown problems are preventing completion of the call." We can go into the options, and specify another Unity server, registered to a different CCM Cluster, and the call completes. This occurred when two users were trying to administer the same set of greetings. My assessment has been that a stop and start of Unity should clear that problem.<br><br> I am trying to get MWI's working without a restart of Unity, as the window for an outage is still a few hours off, so any ideas would be helpful.<br><br> Adding the Extension number to the MWI field is also going to be a time consuming process as there are several hundred subscribers.<br><br>--Cael
Since your MWIs worked at one point, I think you should try to figure out why they stopped working before you make all those changes to each subscriber's MWI extension. It really should work with X in there, I've never heard of a case where it didn't work with X but did work with the subscriber's extension.
Can you confirm that no subscriber has a working MWI right now, or is it just some of your users?
Is there anything that you can think of that changed around the time that the MWIs stopped working?
There were no other changes made at the time of the MWI stop that I can locate. It is every subscriber on the server, I have tested at least 3 phones in every location. This server has 4 locations, each with its own set of Partitions and Calling Search Spaces on the same cluster of CCM's.
I can fix and reproduce the problem by adding/removing the explicit extension number in the MWI field.
Any other changes that I can think of that we made on the CCM occurred AFTER the MWI's started throwing the events in the logs.
No TSP, patch or other changes have happened on the Unity box in question for over a week.
Hmmm...unless someone else out on the Forum pipes in, I think you should bring this to TAC. They should have some good information on how to troubleshoot MWI problems.
I'm officially freaked out that adding the static number in the MWI field works but X does not. All my test boxes with various versions of 3.0/3.1 work with the X just fine. This basic logic hasn't changed in a while... I have no snappy theories for why this would be happening.
I could certainly write you a quick script that would just run through and slam in the DTMF_ACCESS_ID (extension number) of every subscriber in their MWI collection for you in one shot but I'd really like to figure out what's causing this to happen. If you open a TAC case, email me the ticket number and the person working on it... I'll want to take a look myself if I can.
Unity Product Architect/Answer Monkey
http://www.AnswerMonkey.net (new page for Unity support tools and scripts)
Sorry, the customer is requesting a reboot ASAP, we have a maintenance window to work in, and a large CCM cut tonight, so unfortunately we've got to take the shortest route between no lights and lights. I am pretty sure that the restart will take care of it, I can send in event logs, or whatever else you want to look at, but I've got to go ahead with the restart.
I was as stumped by you as to why X does not work and adding extension explicitly does.
One other thing since last posting that I have tried is to run a "TEST" from the TSP, and different ports are receiving an "Error Registering (port name)" each time I run the test (8 times now and counting).
The only thing we can collectively come up with that occurred at about the same time the MWI issue cropped up is the inabililty to us the "Connect to Phone" feature of the SA for administering Call Handler greetings. The details of that are: Two Admins connected to the Unity server via Terminal Services. They each went into the same subscriber box, to the greetings page, and attempted to connect to their own phone. One connected, the second person received a message stating that "Unknown problems are preventing completion of the call" (or something very similar to that) Then the first Admin type disconnected, then later tried to reconnect and got the same error as the second Admin. Now anyone trying to connect to their phone through the SA gets the same error when using that Unity server. Through the SA, we can specify another Unity servers IP, and connect to the phone and do what needs to be done.
I am not convinced that the two are related, but it is the only event that occurred that was out of the normal operation of the server.
Ok, the reboot fixed the problem but, further testing before we were able to perform that reboot allowed us to figure out that the first message left in a box with only X in the MWI field would usually lamp the phone. When a subscriber went in and listened to that message, the lamp would not turn off. A second message could be left and listened to, and then the lamp would usually go out.
Also, the MWI fields HAD to contain both X and the extension number for MWI to work consistently. When we had only the extension number in the MWI field, MWI was working better, but not flawlessly. When we had both X and Extension, the lamps worked AOK, I am wondering if anyone has seen this "double on/off" thing before?
So just so I am straight, single lamp requests to a given extension were not functioning properly. When 2 requests were sent to the same extension, lamps worked "properly."
Is there a correlation between 'Set MWI failed' errors in the event log and the occurrence of an MWI not turning on or off properly?
If it always takes multiple MWI requests to get the lamp on/off properly, then there's something going on that's causing a lot of failures.
It's possible that you are having collisions between incoming calls and outgoing MWIs. If you haven't already, you might try dedicating a few ports for 'MWI only' via the SA (Ports page). This helps minimize the possibility of a collision, in which an incoming call arrives on a port at the same time that Cisco Unity takes the port off-hook to dial out. This is a more likely scenario if your system is handling a lot of incoming calls.
If this does not fix the problem then you should probably open a case with TAC.
The problem has gone away since a reboot of the server.
I figured while I had it "broken" I would test to see under what exact conditions it would fail and NOT fail. The system is a 40 port, with 3 ports dedicated for MWI. At the time of the failure and testing we were preparing for a Call Manager cutover, so all ports were idle anyway. There were no active phones on the CCM, or more directly, there were not alot of registration requests/software phone loads occurring at the time, Network monitors showed no congestion on the network at the time.
We truly went through and narrowed this down to a Unity specific issue.
Like I said, having a single MWI field would almost always lamp the phone, but never turn the lamp off. We could leave a message, have the user go in and listen to it, delete it, and have the lamp not turn off. Then on the SAME box with no config changes, leave a second message, and repeat, and the lamp would go out.
I have never seen behavior like that before from a Unity box... but a reboot cleared the whole mess so I will continue with my theory that if flaky things are going on, reboot.
Well, if you are OK with rebooting every day or so I'm not going to stop you. But it seems you should still work to resolve this issue. I have two more ideas, but beyond this I would see TAC for assistance (didn't I say that before)?
1) It is possible that MWIs are just really slow, so they don't appear to be turning off at all until the second test, when in fact the request that is extinguishing the lamp originated from the first test. Try this (assuming Exchange is installed on the Unity box -if not, I don't think this applies):
If MWIs are very slow in being lit, the problem can often be addressed by running Exchange Optimizer. Exchange Optimizer should be run on the Unity server after adding around 200 users. When a new user is created on Unity, no matter where the user is homed, Unity creates a Primary Call Handler (which is a mail user object to Exchange) on the Unity box. If 200 users homed on other Exchange servers are imported, there are 200 new mail objects created on the Unity server. Running Exchange Optimizer will allocate additional threads in Exchange to do notification.
When running the utility and selecting the number of users, select a range that is double the number of subscribers if homed on the Unity server, since there are two mail objects for each subscriber.
2) If you set up a user for message notification, do you see the same problems? Does a new message trigger a message notification dialout? Then if you delete the message, will the message dialout continue even though there are no longer new messages (make sure the device is configured for to retry after so many minutes). If message notification appears to work flawlessly, then it's likely an integration issue.
You never mentioned whether there is an event log error corresponding to 'failed' MWI attempts.
The original post has the Event Log detail.
When we had the system in the "broken" state, we tested for immediate notifications. We allowed between 2 and 5 minutes before leaving/retrieving the second message, so slow lamps and servers are not the issue. The symptom was reproducable on demand.
Which processes are utilized for completing the MWI on/off? Is it possible that with the other issues going on (read original post) one of those underlying processes was hung and also affecting MWI?
I have an issue with the MWI. I have 8 ports for Unity & one is used for MWI only. Initially the MWI was working well. Some days later, the MWI will turn on although there's no voice message. THere was no changes made until the problem was reported. Here's what I get from the Event Viewer.
Component Miu: Thread 0x00000E9C had a Failure on Port 8 in Method CAvTSPAbstraction::Selsius_SetMWI()
DESCRIPTION: HardFailure from lineDevSpecific.
I am having similar problems to these posts and just created a case with TAC (after following all of the recommendations in the forum here and related MWI trouble-shooting docs on Cisco.com. I am running CCM 3.1 (2c) with Unity 3.1.2 (TSP 3.1.2), also running Exchange 5.5 same server. MWI will not light for all subscribers and I am getting events 521 in the application event log - AvMiu_MC as well as Notifier error stating it could not light the user's extension. My TAC case is C356927 - so will keep you updated as to our progress.