Notice: UC560 Voice Mail / AA issue being investigated

Unanswered Question
Sep 23rd, 2010

We are currently investigating several cases where UC560s running on  the latest SWP 8.0.4 are experiencing Audo Chop and distortion with  Voice Mail Greeting & Recorded Voice Mail Messages.

We  believe we understand the issue now is with CUE 8.0.2 and are close to implementing a  solution

Please watch CSCti85760  in your Bug Toolkit for status.

In the mean time if  you do experience this issue, use CCA to reload CUE (we have seen this  help when the problem occurs and found very agressive proactive action  of every 2-3 days will prevent this).

If you have a factory UC560 ready to install,  please only upgrade the IOS to 15.0(1)XA3A.  Do not upgrade CUE (Stay with CUE 7.1.3)

We removed the UC560 SWP from CCO until we have a solution.  /* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-priority:99; mso-style-qformat:yes; mso-style-parent:""; mso-padding-alt:0in 5.4pt 0in 5.4pt; mso-para-margin:0in; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"Calibri","sans-serif"; mso-ascii-font-family:Calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:Calibri; mso-fareast-theme-font:minor-latin; mso-hansi-font-family:Calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"Times New Roman"; mso-bidi-theme-font:minor-bidi;}

*If you need 15.0(1)XA3a IOS image, you can get it from 8.0.4 software pack for uc540 platform which is still posted and has no problem.

NEW INFO:

Downgrading is NOT an option for CUE.  DO NOT attempt to downgrade CUE to 7.1.3 if you are running CUE 8.0.2 as it is not supported.

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (1 ratings)
Loading.
Steven Holl Fri, 09/24/2010 - 07:33

I've got someone seeing this on an AIM running 7.2.2, as well, and am currently working with CUE developers on it.  May be realted, or may not.

I see a high jitter value (~800ms) on a single pack from CUE to CME.

I'm working with development in regards to this, and currently pending some coredump collections and rtp pacer debugs.  If it ends up being a new bug, I'll try to remember to report back to this thread since it could be related.  My issue may be related to CSCtf70144 and just never ported to the AIM platform, but I know CSCtf70144 was fixed for the UC5xx platforms.

MICHAEL JOHNSON Fri, 09/24/2010 - 08:18

Steve,

As a Cisco partner, I appreciate the honesty that such a post required. Most vendors including Cisco of the past, would simply cover this up as just another "bug", but this is not just another bug. This "bug" impacts everyone using a system and the core of what a phone system is supposed to provide not just a feature.

As a partner having two customers with this problem, I appreciate the effort Cisco has put into helping me, the partner, and my customer. In the end, we need the problem resolved as quickly as possible so that it does not delay future orders we have in the pipeline.

Steven DiStefano Fri, 09/24/2010 - 09:40

Here is how to RELOAD ONLY CUE if you find you need to using CCA 2.2.5 (click to improve picture)

Steven DiStefano Sat, 09/25/2010 - 16:44

Latest Status and request for Partners who want to try workaround.

As we now have reproduced the problem and have a team of developers looing at this as I write, we also are taking the action to produce a work around.

We will be using the Event Manager to create a sort of CRON job that can kick off and reboot CUE every morning at 4AM, since we know that it takes about 2 days to get into the problem where CUE is sending packets with 40ms packetization delay (should be 20ms).

We are advising Partners that rebooting CUE (not the whole UC560), will prevent the problem from occuring, but we realize this is a manual task.

When we do have this ready, are there any Partners who would like to use it?

Send me an EMAIL [email protected]

We are also considering the following if we cannot figure out the fix in the next couple days, but its a bit complex and time consuming.  Let me explain.

Putting CUE back on CUE 7.1.3 will eliminate this problem.  However, the data structure of CUE 8.0.2 is NOT backward compatible with CUE 7.1.3 (so it cant be saved and restored).   Also, as you all know, these systems are configured in such a way where configuration is sent to BOTH IOS/CME and CUE, so there is no way to simply restore CUE to factory defaults and load the old CUE SW and rebuilt only it.   The only way to go back to CUE 7.1.3, is to factory default the entire system, downgrade CUE (I would probably factory reset again after that) and then rebuild the system.  This obviously will take 3-4 hours if you have all the data you need.

The only value an RMA option would provide is getting you on older code, and you would upgrade IOS to 15.0(1)XA3A (but leave CUE alone on 7.1.3) and then rebuild it in a staging (offline) way.  This is very possible using text files for phones and users as I have documented in TEL#25 on this community.  Then it would be more like a physical cutover of one box to the other.  You would have to transfer licenses and make absolutely sure the HW VWICs are identical in both systems to have this work, and this also is manually intensive, so we are really not encouraging this option, but want to let you know it is being considered on the table at this point.

Steven DiStefano Sun, 09/26/2010 - 14:44

/* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-priority:99; mso-style-qformat:yes; mso-style-parent:""; mso-padding-alt:0in 5.4pt 0in 5.4pt; mso-para-margin:0in; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"Calibri","sans-serif"; mso-ascii-font-family:Calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"Times New Roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:Calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"Times New Roman"; mso-bidi-theme-font:minor-bidi;}

This is IOS CLI, entered on the UC560 with privilege level 15 in Configuration Terminal mode.   When done, save it using 'write memory' which copies running to startup-config.  While IOS will not reboot with this, it's still a good idea to save it in case the router looses power or reloads due to something else.


This is intended to be a short term work around only, until we address the root cause of the CUE packetization problem that seems to occur after a period of time ranging from a couple days to 4-5 days.   The reboot of CUE initializes it preventing the longevity issue.  UC560 CUE reboot should take about 5 minutes on a UC560 with about 50 voicemail boxes.

The 4am is a time we picked, but you can adjust if that's not a good time.  When CUE reloads, all call processing continues as normal, just voice mail and AA are reported as unavailable, please try later.

NOTE:  PLEASE REMEMBER TO TAKE THIS OUT WHEN WE FIX THE ROOT CAUSE


Since this is fairly new, the SBCS is being briefed on this as well so this is pretty much real time.  We tried this script on several systems and it worked well.

event manager applet cue_reset

event timer cron cron-entry "0 4 * * *"

action 1 cli command "enable"

action 2 cli command "service-module integrated-Service-Engine 0/0 reset" pattern "confirm"

action 3 cli command "y"

wr mem

***If you made any changes in CUE and didnt save via CCA, please do that before executing this script (normal best practice anyway here).

Some have asked how do you know if CUE was reset.  here is how...

UC_560#service-module integrated-Service-Engine 0/0 session
Trying 10.1.10.2, 2002 ... Open
banner login ^Cisco Configuration Assistant. Version: 2.2 (5). Mon Sep 27 17:08:07 EDT 2010^

User Access Verification

Username: ciscoadmin
Password: ************
se-10-1-10-1# show hardware
se-10-1-10-1 uptime is 0 weeks, 0 days, 1 hours, 44 minutes
# of Processors:              2
CPU:                          e500v2
Revision:                     3.0 (pvr 8021 0030)
BogoMIPS:                     133.12
SKU:                          Unavailable
UDI:                          Unavailable
Chassis Type:                 UC500
Chassis Serial:               FHH1313P04X
Module Type:                  Integrated-Service-Engine
Module Serial:               
Compact Flash:                2097MB
SDRAM (MByte):                1024
se-10-1-10-1#

antyeung Thu, 09/30/2010 - 08:20

There have been a few inquiries on how to change the time of day that the EEM script reloads CUE. The script that Steve mentions reloads the CUE everyday at 4AM.

event  manager applet cue_reset

event timer cron cron-entry "0 4 * *  *"

action 1 cli command "enable"

action 2 cli command "service-module  integrated-Service-Engine 0/0 reset" pattern "confirm"

action 3 cli  command "y"

To change the time of day when this script executes modify the "event timer" configuration line. Here is a description of the different parameters-

http://www.cisco.com/en/US/docs/ios/netmgmt/command/reference/nm_06.html#wp1157622

cron-entry

Text string that consists of five fields separated by spaces. The fields  represent the times and dates when CRON timer events will be triggered.  Fields and corresponding values are as follows:

minute—A  number in the range from 0 to 59 that specifies when a CRON timer event  is triggered.

hour—A  number in the range from 0 to 23 that specifies when a CRON timer event  is triggered.

day-of-month—A  number in the range from 1 to 31 that specifies the day of the month  when a CRON timer event is triggered.

month—A  number in the range from 1 to 12 or the first three letters (not  case-sensitive) of the name of the month in which a CRON timer event is  triggered.

day-of-week—A  number in the range from 0 to 6 (Sunday is 0) or the first three  letters (not case-sensitive) of the name of the day when a CRON timer  event is triggered.

Dan Smith Thu, 09/30/2010 - 15:46

Is it supported to upgrade new UC560 systems with IOS and phone loads to 8.0.4 and leave CUE 7.1.6?

MICHAEL JOHNSON Sun, 10/03/2010 - 20:40

Is anyone having issue with this work-around? I have one client that having voice mail issues after the auto reboot. I have now had 3 consecutive days where the cue has failed to reload properly after the routinge, forcing us to reboot in and manully reload the CUE again.

I am trying to figure out if there is anyone else having that same problem or similar CUE issues when leveraging the work around.

CISCO: Is there any debug I can run to help Cisco support figure out the cause of the hanging?

NOTE: This is not garbling. The last time it happened, the auto attendent greeting was playing, but no extensions or options worked from within the auto attendent. When I "serv inte 0/0 sess" from clI, it would go into EXEC mode (unpriviledged) with the factory standard name "integrated engine>". Again, the correct AA greeting would play, but no extensions would connect nor would the AA options connect.

antyeung Wed, 10/06/2010 - 09:41

UPDATE1: CSCti85760 -  Choppy VM / AA - SWP 8.0.4

In lieu of the current manual proactive reboot procedure as a workaround to  avoid this issue we have an EEM script to automate this effort.

We have been able to reproduce the voice quality issue experienced with CUE in our lab over the past week on UC560. Since then we have been running Release Candidate images on our UC560’s and have seen positive results. We will be conducting more tests on the Release Candidate images before it is made available for general use. If you would like to help be a part of the feedback process before a general release is made available please send your request to [email protected] /* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-priority:99; mso-style-qformat:yes; mso-style-parent:""; mso-padding-alt:0in 5.4pt 0in 5.4pt; mso-para-margin:0in; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"Calibri","sans-serif"; mso-ascii-font-family:Calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"Times New Roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:Calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"Times New Roman"; mso-bidi-theme-font:minor-bidi;} with your Cisco.com User ID in the subject line of the email. You will then receive an email notification with instructions on how to download the firmware.

Removed EEM script from this post since we now have RC1 and RC2 images available>

If there are any questions/concerns feel free to reach out to me at [email protected]

Thanks,
Anthony

Message was edited by: Anthony Yeung

rturnertech Tue, 11/16/2010 - 09:59

Has anyone had "real" world success with RC3 solving the Audio Chop issue.  My customer has been dealing with this for quite a while and I don't want to implement something that doesn't have in production successes.  They don't deserve anymore anxiety with this system.

Regards,

Rick Turner

CCIE (Voice) #25838

antyeung Wed, 11/17/2010 - 00:20

Hi Rick,

From the customers that have provided feedback we have not received any reports of CSCti85760 reoccuring. Our engineering team is also confident that the Release Candidate image properly addresses CSCti85760. Based on the engineering effort, results from internal tests, and the positive responses we received with respect to the Release Candidate image we will be bundling it into the next Software Pack Release.

If you have any questions or concerns please feel free to reach out to me directly - [email protected]

Regards,

Anthony

antyeung Wed, 11/17/2010 - 15:30

There will be two Software Packs (8.1.0 and 8.0.5) that will contain CUE 8.0.3. SWP 8.1.0 is planned to be released mid December and SWP 8.0.5 is planned to be released next week. The UC500 SWP page has been updated to reflect this schedule-

https://supportforums.cisco.com/docs/DOC-9827

econsystems Sat, 11/27/2010 - 18:12

I see the 8.05 is available in the downloads.  But did anyone check the zip file.  I tried the 540 and 560 zip file and both are corrupt.  Can anyone please check and verify if these files even work.  I tried multiple times using different browsers and all downloads report files as corrupt.

antyeung Fri, 12/03/2010 - 10:31

Can you try downloading them again and see if you still have the same problem?

econsystems Fri, 12/03/2010 - 10:39

Anthony, thanks for the reply.  I think, as of Tuesday, they were fixed.

Under testing, I am however having an issue regarding MWI.  When I build a new config using CCA 2.2.5 or 2.2.6, on 8.0.5, the WMI does not work at all.  However, it works just fine in 8.0.2 or 8.0.4 under new config and after it is upgraded to 8.0.5.

After Steve's recommendation, I tried wmi refresh all and it did not do the trick.

Can you please test this out in your lab.  I tried uc560 and 540 and both show the same behavior.

antyeung Fri, 12/03/2010 - 10:45

I have a lab UC540 that's running SWP 8.0.5 and I'm able to get MWI. I just saw your other thread post. I'll repond to that thread.

antyeung Wed, 10/20/2010 - 19:00

UPDATE2: CSCti85760 -   Choppy VM / AA - SWP 8.0.4

Over the past two weeks our engineering team has come up with two Release Candidate images. The RC1 image was built to contain the issue and was provided to several customers upon request. We will soon have a RC2 image, which will have the fix for CSCti85760. We are undergoing verification and testing before we make the RC2 image available. If you are interested in participating in the feedback process for the RC2 image, please send an email to [email protected].

Since there are Release Candidate images available we no longer recommend the EEM script as a workaround. Please remove the EEM script from deployment.

Thanks,

Anthony

Message was edited by: Anthony Yeung

tcekada Fri, 10/22/2010 - 16:55

Anthony,

     Would you happen to know or could you elaborate on whether the ISM-SRE-300-K9 engine running CUE 8.0.2 in a 2911 router would be affected by this issue?  Or is it only CUE 8.0.2 in the UC560?

Thanks in advance,

TonyC

antyeung Mon, 10/25/2010 - 08:29

Hi Tony,

It’s only the UC560 platform that is being affected by CSCti85760. If you are also experiencing similar symptom it may be for a different reason and may be best to get a TAC case open to help diagnose the issue.

Thanks
Anthony

bagattij81 Wed, 11/03/2010 - 17:33

Do we have an update on when the RC's are going to be closer to release?

Even with the reboot fix, my voicemail system is running insanely slow. It takes 20-25 seconds after entering my pin to get to the menu and another 25 seconds to start the first message with a gap of 20-30 seconds before the next message starts playing.

glanders Wed, 11/03/2010 - 22:10

Justin - I'm not aware of 'slow' beeing an issue. Did you apply RC1 or RC2 already or just the reboot work-around ?

RC2 is undergoing additional validation, but at this point anticipated to be the released code. If you do not already have a case investigating the performance, I would consider opening one for that.

bagattij81 Wed, 11/03/2010 - 22:32

I have spent the last 6 hours on the phone with SBSC and TAC, currently im talking to an engineer based out of Sydney trying to restore connectivity to the CUE module to keep my system stable until a replacement box can get to my office.

I really want to give my thoughts on the level of support i have had from the Small Business team, but i need time to calm down before i post anything.

glanders Wed, 11/03/2010 - 22:53

Sorry to hear, that doesn't sound good. Hardware issue ? If you are not satisfied with the engineer, you can escalate to the duty manager/manager on Duty and express your concerns. This is the fastest path to resolution.

Certainly understand you are in the thick of it right now.  Once you are stable I can be reached at [email protected], we can figure out what went wrong here.

DataPathInc Mon, 12/06/2010 - 20:53

Has anyone else updated to the 8.0(5) pack on a UC560 that hasn't had the MWI problems?  I'm curious if there are any other issues people are having.  We have a couple clients we have been waiting to update because of the chop issue.

antyeung Wed, 02/23/2011 - 14:44

To close the loop on this Software Pack 8.0.5 and Software Pack 8.1.0 is available for download, which contain the fix for the issue reported in this forum thread.

Software Packs can be downloaded from the following link-

https://supportforums.cisco.com/docs/DOC-9829/

For more information about SWP refer to the following page-

https://supportforums.cisco.com/docs/DOC-9827

Regards,

Anthony

paulmulrainey Tue, 04/26/2011 - 02:23

I have a customer who is experiencing the symptoms of choppy voicemail as described in CSCti85760 on a UC560 platform but they are on Software Pack 8.1.0. Have you came across this and should i implement the cue reboot process to see if this eliminates the issue. ?

Actions

This Discussion