I have a Call Manager 4.01 Sr1a implementation with approximately 55 7940/7960 IP Phones. They are connected with 2 3560 48 port inline power phones, a 3550 24 port and then a 3750 24 Port Gigabit switch. I have a voice and data vlan with all the voice servers and phones in the voice vlan. I recently upgraded to the 6.05 firmware load on the phone and noticed a strange thing. For off-net and on-net calls sometimes, maybe 3 out of 10 calls users will hear a delay in the audio. Once a call is established you try to speak and hear nothing for 3 seconds. After the 3 seconds you can hear the other caller. Some delay in the RTP stream. I've opened a TAC case and they helped me take the firmware back to 6.04 but I am still hearing the same delays. They've wanted me to gather packet traces but since this is so random it is difficult to track down. I essentially have to do a port span and be right at the phone to capture the traces correctly. Wondering if anybody has seen this before?
I've swapped switch ports, ethernet cables, phones. I'm using auto qos voip cisco-phone on the switch ports for the QOS features. There's no inter-vlan routing going on for the voice traffic.
The switches are in a hub-and-spoke design with all the switches hanging off the gigabit switch. They're uplinked with each other via copper gigabit.
I have the exact same problem, however, we are running CM 3.3.3 SR4, and the topology is different. But the problem is exactly the same. The only common thing is that the phones are on 6.05. I know this is not an answer to your problem, but maybe if two people have the same problem, somebody in the Forum will answer more promptly.
And you are right, this is impossible to catch with a Sniffer. I once had this problem with fw 3.x, about a year ago, and the TAC engineer said it was a bug in the firmware where the ARP request from the phone has to wait 10 seconds or less, until the stream gets established. However, I would think this bug was solved then, and it should not reappear.
We have observed this on some customers with older echo-cancellation code on their gateways, the G.165 ECAN as opposed to the new G.168 ECAN. Assuming you are using IOS gateways, go into the voice-port configuration and set 'no echo-cancel enable'. We didn't troubleshoot extensively after pinning it down to the ECAN, but we thought it had something to do with the G.165 suppressor feature where the conversation ends up being half-duplex for the first few seconds (by design). Obviously you wouldn't leave this in permanently, but run through a few dozen test calls and see if you can replicate the problem with the ECANs out of the loop. Upgrading IOS to the most recent IOS revisions, which support the new G.168 ECAN and use it by default, would resolve the issue if that's the case.
There are some other issues that could cause it. You could look at CSCdz52758, which gets flushed out by a hardware issue specific to the 3640 router if that's what you have for a voice gateway. You can see if that's the case by double-tapping the ? button on the phone and seeing if the MaxJtr value has gone raving bonkers. Even if it hasn't, use of the ? button should be able to help you figure out if the phone is receiving RTP for those first few seconds by comparing approximate Rx and Tx packet counts. So long as you don't put the call on hold (and even this doesn't matter on very recent phone loads) the counts should be reasonably even.
Thanks for the response. However, my problem happens between IP Phones connected to the same switch. So, there is no gateway in the middle. The switch is a 6500 and the phones are connected in the same VLAN (different module, however) and AutoQos is configured on the 6500.
This is the bug that I was referring earlier, but it is supposed to be fixed by now:
CSCeb19725: Intermittent one way, no way audio for the first 10 seconds between IP phones with no gateway involved. CallManager traces show the calls being connected but no voice stream is being passed. Review of a sniffer trace between phones show that the streams do not get established until the phone issues an ARP reqeust roughly 10 seconds after the skinny message comes in to cut through audio.
Maybe it is a different thing, but what I've heard it could be also present in 5.0.6 IP Phone fw.
Just to let everyone know, I have 333sr4a, phone load 6.0(5) and TAC told me to roll back to 6.0(3) for the 7940/60 phones (if you have a 7914 sidecar use the compatible 4.0(0) fw).
Hopefully this will work.
I do not see rolling back to older firmware as a good solution. I'm holding out for Cisco to release new firmware that will address the issue in their current release.
Are there any plans for this in the near future?
I have the same issue with delay between ip phones. This delay only appears to happen between 6509 switches (over the gig connection). So far, tac has me going in circles. I'm running CCM 4.0.2 with the latest phone load. The biggest problem is i'm unable to reproduce the problem when tac wants me too. All QOS parameters are in place and I still get about a 3 second delay. Very strange and difficult to troubleshoot.
From reading these messages and taking note on my own network, I believe that the issue is independent of device types. It appears to happen randomly on various types of switches and gateways. From what I see it must be a telephone firmware issue.
I've got a customer experiencing the same problem. Is even happening between IP phones in the same 6513 switch.
Call Mgr 4.0 , phones have 6.05 load.
Has anyone been given a fix ? What is Cisco saying about it?
I haven't heard anything from Cisco, but I am testing the very latest firmware with some phones. This is 7.1.1, just released yesterday. Hopefully it comes with the fix, even though is not mentioned in the release notes. Still too early to tell if this one makes a difference.
I'm going to try and apply the firmware updates at our offices. If there's no problems I'll give it a try at our clients office and see if it resolves the issue. Hopefully it will. Thanks for all the replies, I didn't know it was such a big issue with everybody. Thought it was something I was doing.
Same problem, CCM version is now 3.3(4)sr2, gateway is a 6608 T1 PRI on a 6509. We also upgraded to the 6.0.5 software...but we had this problem back CCM 3.3.3sr4a with the 5.0.6 software.
I have a customer having the same problem. Happens on GW calls and IP-IP.
New firmware load out on 11-24 (or so) is supposed to fix the problem. We don't have it loaded yet, so I can't report on the results.
Same problem, seems to be more frequent on calls that are placed on hold, but CM 4.0.2aES9.1 with phone load 6.0.5. We are on 6.0.5 because of a bug in 6.0.4 concerning line status with shared lines. Rolling back is not an issue. I've put load 7.1.1 on the server but have not applied it to any but 1 phone. Will let the forum know the results of that change when I make it.
Post upgrade to 7.1.1, no complaints of pausing. 7.1.1 seems to be THE load for CM4.0.2a. I wouldn't even try 6.0.4 or 6.0.5 from now on.
I have been experiencing this problem at my site. We are running 4.0(2)a. I have loaded the 7.1.1 version on a few phones. I would like to use the T017 Load that people are getting from TAC. I am willing to open a case to get that if that is what I need to do or can I just request it?
Does anyone have the BugID for this and does any one know when they will release the T017 Load for full release?
its my understanding that the 7.1.1. load w/ 4.0 fixes the problem. I've upgraded my 4.0 customer to that load and it seems to have fixed the delay.
I've gotton the T017 from tac which I put on a 3.3.4 cm site.
You of course need a CCO login. There's a link to this on the Call Manager download page. This is also non-sip.