3 Second Delay in Audio

Unanswered Question
Nov 15th, 2004

I have a Call Manager 4.01 Sr1a implementation with approximately 55 7940/7960 IP Phones. They are connected with 2 3560 48 port inline power phones, a 3550 24 port and then a 3750 24 Port Gigabit switch. I have a voice and data vlan with all the voice servers and phones in the voice vlan. I recently upgraded to the 6.05 firmware load on the phone and noticed a strange thing. For off-net and on-net calls sometimes, maybe 3 out of 10 calls users will hear a delay in the audio. Once a call is established you try to speak and hear nothing for 3 seconds. After the 3 seconds you can hear the other caller. Some delay in the RTP stream. I've opened a TAC case and they helped me take the firmware back to 6.04 but I am still hearing the same delays. They've wanted me to gather packet traces but since this is so random it is difficult to track down. I essentially have to do a port span and be right at the phone to capture the traces correctly. Wondering if anybody has seen this before?

I've swapped switch ports, ethernet cables, phones. I'm using auto qos voip cisco-phone on the switch ports for the QOS features. There's no inter-vlan routing going on for the voice traffic.

The switches are in a hub-and-spoke design with all the switches hanging off the gigabit switch. They're uplinked with each other via copper gigabit.

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Average Rating: 0 (0 ratings)
ikobisher3 Tue, 11/16/2004 - 06:56

I have the exact same problem, however, we are running CM 3.3.3 SR4, and the topology is different. But the problem is exactly the same. The only common thing is that the phones are on 6.05. I know this is not an answer to your problem, but maybe if two people have the same problem, somebody in the Forum will answer more promptly.

And you are right, this is impossible to catch with a Sniffer. I once had this problem with fw 3.x, about a year ago, and the TAC engineer said it was a bug in the firmware where the ARP request from the phone has to wait 10 seconds or less, until the stream gets established. However, I would think this bug was solved then, and it should not reappear.

Thanks

IK

jasyoung Tue, 11/16/2004 - 07:16

We have observed this on some customers with older echo-cancellation code on their gateways, the G.165 ECAN as opposed to the new G.168 ECAN. Assuming you are using IOS gateways, go into the voice-port configuration and set 'no echo-cancel enable'. We didn't troubleshoot extensively after pinning it down to the ECAN, but we thought it had something to do with the G.165 suppressor feature where the conversation ends up being half-duplex for the first few seconds (by design). Obviously you wouldn't leave this in permanently, but run through a few dozen test calls and see if you can replicate the problem with the ECANs out of the loop. Upgrading IOS to the most recent IOS revisions, which support the new G.168 ECAN and use it by default, would resolve the issue if that's the case.

There are some other issues that could cause it. You could look at CSCdz52758, which gets flushed out by a hardware issue specific to the 3640 router if that's what you have for a voice gateway. You can see if that's the case by double-tapping the ? button on the phone and seeing if the MaxJtr value has gone raving bonkers. Even if it hasn't, use of the ? button should be able to help you figure out if the phone is receiving RTP for those first few seconds by comparing approximate Rx and Tx packet counts. So long as you don't put the call on hold (and even this doesn't matter on very recent phone loads) the counts should be reasonably even.

ikobisher3 Tue, 11/16/2004 - 07:20

Thanks for the response. However, my problem happens between IP Phones connected to the same switch. So, there is no gateway in the middle. The switch is a 6500 and the phones are connected in the same VLAN (different module, however) and AutoQos is configured on the 6500.

Thanks

IK

jeffkelso Tue, 11/16/2004 - 09:13

I am also having the same issue running CM 4.02a with load 6.04.

togi@perlegen.com Tue, 11/16/2004 - 11:31

I'm having the same, random problem. I'm on 33(3) sr4 also, with the phone load being the 6.0.5 one.

mpaul@innovativ... Tue, 11/16/2004 - 12:17

Thanks for your replies. A little more detail for you. A 1760 voice gateway using MGCP. It happens on the Lan and over the PSTN.

ikobisher3 Tue, 11/16/2004 - 12:22

This is the bug that I was referring earlier, but it is supposed to be fixed by now:

CSCeb19725: Intermittent one way, no way audio for the first 10 seconds between IP phones with no gateway involved. CallManager traces show the calls being connected but no voice stream is being passed. Review of a sniffer trace between phones show that the streams do not get established until the phone issues an ARP reqeust roughly 10 seconds after the skinny message comes in to cut through audio.

Maybe it is a different thing, but what I've heard it could be also present in 5.0.6 IP Phone fw.

Thanks

IK

jeffkelso Tue, 11/16/2004 - 13:23

I also have a 7200 voice gateway running IOS version 12.2(12.14)T.

togi@perlegen.com Fri, 11/19/2004 - 10:05

Just to let everyone know, I have 333sr4a, phone load 6.0(5) and TAC told me to roll back to 6.0(3) for the 7940/60 phones (if you have a 7914 sidecar use the compatible 4.0(0) fw).

Hopefully this will work.

T

jeffkelso Fri, 11/19/2004 - 13:08

I do not see rolling back to older firmware as a good solution. I'm holding out for Cisco to release new firmware that will address the issue in their current release.

Cisco Techs:

Are there any plans for this in the near future?

tavery2000 Fri, 11/19/2004 - 14:02

I have the same issue with delay between ip phones. This delay only appears to happen between 6509 switches (over the gig connection). So far, tac has me going in circles. I'm running CCM 4.0.2 with the latest phone load. The biggest problem is i'm unable to reproduce the problem when tac wants me too. All QOS parameters are in place and I still get about a 3 second delay. Very strange and difficult to troubleshoot.

jeffkelso Mon, 11/22/2004 - 06:15

From reading these messages and taking note on my own network, I believe that the issue is independent of device types. It appears to happen randomly on various types of switches and gateways. From what I see it must be a telephone firmware issue.

jocoates Wed, 11/24/2004 - 20:32

Has anyone got a resolution to this? We also have a customer that is experiencing similar pain.

rvincent Thu, 11/25/2004 - 05:55

I've got a customer experiencing the same problem. Is even happening between IP phones in the same 6513 switch.

Call Mgr 4.0 , phones have 6.05 load.

Has anyone been given a fix ? What is Cisco saying about it?

ikobisher3 Thu, 11/25/2004 - 06:04

I haven't heard anything from Cisco, but I am testing the very latest firmware with some phones. This is 7.1.1, just released yesterday. Hopefully it comes with the fix, even though is not mentioned in the release notes. Still too early to tell if this one makes a difference.

rvincent Thu, 11/25/2004 - 06:09

thanks

I was thinking the same thing, but was hoping to see it mentioned in the fixes.

Please let me know if it fixed it.

Rob

jocoates Sat, 11/27/2004 - 03:36

Thanks for the reply.... please do let me know how you go. Cheers John

mpaul@innovativ... Sat, 11/27/2004 - 18:58

I'm going to try and apply the firmware updates at our offices. If there's no problems I'll give it a try at our clients office and see if it resolves the issue. Hopefully it will. Thanks for all the replies, I didn't know it was such a big issue with everybody. Thought it was something I was doing.

jerake Sun, 11/28/2004 - 03:46

Same problem, CCM version is now 3.3(4)sr2, gateway is a 6608 T1 PRI on a 6509. We also upgraded to the 6.0.5 software...but we had this problem back CCM 3.3.3sr4a with the 5.0.6 software.

TMH

mciarfello Sun, 11/28/2004 - 23:25

I have a customer having the same problem. Happens on GW calls and IP-IP.

New firmware load out on 11-24 (or so) is supposed to fix the problem. We don't have it loaded yet, so I can't report on the results.

jerake Sun, 11/28/2004 - 23:28

I saw that new load as well - I'm going to apply it to our 600+ IP phones Monday 11-29, I will see if that fixes the problem.

TMH

alshort Tue, 11/30/2004 - 05:48

Same problem, seems to be more frequent on calls that are placed on hold, but CM 4.0.2aES9.1 with phone load 6.0.5. We are on 6.0.5 because of a bug in 6.0.4 concerning line status with shared lines. Rolling back is not an issue. I've put load 7.1.1 on the server but have not applied it to any but 1 phone. Will let the forum know the results of that change when I make it.

alshort Wed, 12/08/2004 - 12:11

Post upgrade to 7.1.1, no complaints of pausing. 7.1.1 seems to be THE load for CM4.0.2a. I wouldn't even try 6.0.4 or 6.0.5 from now on.

JASON NIELSEN Fri, 12/10/2004 - 10:06

I have been experiencing this problem at my site. We are running 4.0(2)a. I have loaded the 7.1.1 version on a few phones. I would like to use the T017 Load that people are getting from TAC. I am willing to open a case to get that if that is what I need to do or can I just request it?

Does anyone have the BugID for this and does any one know when they will release the T017 Load for full release?

Thanks

rvincent Fri, 12/10/2004 - 11:38

its my understanding that the 7.1.1. load w/ 4.0 fixes the problem. I've upgraded my 4.0 customer to that load and it seems to have fixed the delay.

I've gotton the T017 from tac which I put on a 3.3.4 cm site.

Rob

jeffkelso Mon, 11/29/2004 - 06:25

Where do I obtain IP Telephone Firmware? I can't find it under the CallManager Software Page.

jerake Mon, 11/29/2004 - 06:42

It's not under the CallManager software page - it's on it's own page: Cisco IP Phone FW 7900 Series (NON SIP)

TMH

jeffkelso Mon, 11/29/2004 - 07:15

Where do I obtain IP Telephone Firmware? I can't find it under the CallManager Software Page.

jeffkelso Mon, 11/29/2004 - 08:48

I found it. It was right under my nose and I was looking over it. Thanks, I'm going to give it a try.

aattravi Mon, 11/29/2004 - 00:23

I have the same problem in a ip to ip calls.

CCM3.3(3)sr4 , 50 ip phone, two cat3550.

rvincent Mon, 11/29/2004 - 07:33

Would anyone from Cisco care to comment on this?

I've got two customers, one with cm 4.0, the other with 3.3.4, and both are experiencing this delay in the audio.

Rob

tavery2000 Mon, 11/29/2004 - 07:42

The problem we are "ALL" facing is a combination of issues. First issue is in all verison of 3.3(?) call manager. According to our DE, the problem was not discovered until 4.0. Another issue is within the signaling setup bit. As of CCM 3.3(4) and up Cisco changed the call signaling for voice and video to PHB CS3 or DSCP 24, previously classified as AF31 or DSCP 34. Voice Bearer has not changed and is still EF or DSCP 46. The following steps were taken to resolve our delay issues.

1). Upgraded to CCM 4.0.2a. We made the upgrade to support Cisco VT Advantage and resolve a few known issues in 3.3.3. If you are running IPCC 3.5.1 or higher, with AD, you will need patch ES-13.

2). changed all LAN qos to conform with the new CS3 and DSCP 24.

I don't believe the lastest phone load will resolve the problem. Working for a Brokerage Firm, with over 2k ip phones, 3 sec delay is not acceptable. Since making the changes our delay issue has been resolved. Unfortunitly I'm not sure if CCM 4.0.2 fixed the problem or changes to QoS was the fix.

Jumping to 4.0 for 3.3.4 is a big change and should be the last resort. However, skip 4.0.1 if you plan to deploy any video (Tandberg or Cisco VT Advantage).

Thanks

Tom

mpaul@innovativ... Mon, 11/29/2004 - 08:27

I'm curious if you were using the auto-qos feature on your switches. For the site that is experiencing this, I noticed this command and used it for the first time, otherwise I would've manually put out the commands I've used at some other sites, 3560/3550 switches.

interface FastEthernet0/2

switchport access vlan 20

switchport voice vlan 10

srr-queue bandwidth share 10 10 60 20

srr-queue bandwidth shape 10 0 0 0

mls qos trust device cisco-phone

mls qos trust cos

no mdix auto

auto qos voip cisco-phone

spanning-tree portfast

The auto qos voip cisco-phone command automatically added the MLS QOS statements and the srr-queue statements.

For those ports that aren't attached to the cisco phones we're not using this command on the switch port, printers, servers.

mciarfello Mon, 11/29/2004 - 08:53

I'm also curious what your switch configs looked like too because changing the DSCP types should not matter to the switch, the switch will still give it priority. (Unless you were specifically classifying specific DSCP types.) You bring up an interesting topic and I think we all would like to explore this some more.

Our switches are set to trust DSCP markings on CCM, GW ports and trunk ports and use AutoQos for phone ports. I have verified end-end QoS using a sniffer at various points in the network.

I am going to start deploying 7-1-1 on a limited basis tomorrow, so we'll see if a phone load fixes it. It will take me some time to report back findings since the delay didn't happen every day. Running CCM 3.3.3SR4.

If anyone else had any success with firmware upgrades, I would LOVE to hear from ya.

tavery2000 Mon, 11/29/2004 - 09:52

Cat IOS version should be around 12.2.20 or 18. The QoS is vlan-base to avoid TCAM Errors.

A basic config is as follows (without srr-queueing);

qos map cos 3 to dscp 24

qos map cos 4 to dscp 34

qos map cos 5 to dscp 46

qos

class-map match-all Voice-control

description VoIP Control Traffic

match access-group name Voice-Control

class-map match-all Gold-Data

description Unity Server

match access-group name Gold-Data

class-map match-all Tanberg

match access-group name Tandberg-Ports

class-map match-all Video

description Video Traffic only

match ip precedence 4

class-map match-all Voice

description VoIP Bearer Traffic

match access-group name Voice

policy-map Tandberg

class Tanberg

set ip dscp 34

policy-map ACCESS-LAN-EDGE-IN

class Voice

set ip dscp 46

class Gold-Data

set ip dscp 18

class Voice-Control

set ip dscp 24

interface FastEthernet4/9

switchport trunk encapsulation dot1q

switchport trunk native vlan XXX

switchport mode trunk

switchport voice vlan XXX

qos trust device cisco-phone

qos trust cos (not needed, depends on IOS/OS)

tx-queue 3

priority high

ip access-list extended Gold-Data

remark Match IP Address of Unity Server

permit ip any host 10.250.17.9

permit ip host 10.250.17.9 any

ip access-list extended Tandberg-Ports

permit tcp any any eq 4224

permit udp any any eq 5445

remark Tanberg and Cisco VT Advant ports

ip access-list extended VoIP-Control

permit tcp any any eq 1720

permit tcp any any range 11000 11999

permit tcp any any range 2000 2002

permit udp any any eq 1719

ip access-list extended Voice

remark Match UDP ports for VoIP Bearer

permit udp any range 16384 32767 any (not recommended)

For CATOS use the auto qos feature along with aux voice vlans. you will want to trust CoS on the trunk links on both sides...

- set port qos 1/1 autoqos trust cos (again you can do the range of gig 1/1-2)

Thanks

tavery2000 Mon, 11/29/2004 - 09:03

The auto qos command is a great tool for gathering voice port info, srr- queues, etc. We generally use it as a baseline or start for network wide qos parameter.

jeffkelso Mon, 11/29/2004 - 08:52

I am running CCM 4.02a already. I am noticing the problem with this version, so the issue must be with the QOS. What changes exactly are needed to conform to the new standards? Are we talking about IOS updates?

togi@perlegen.com Mon, 11/29/2004 - 09:04

The cisco guys have given me a phone load of P0030600T017 which isn't even out yet to solve this problem. Hopefully, I'll implement it tonight if not this week.

T

mciarfello Mon, 11/29/2004 - 09:38

What version does that translate to when displayed on the phone? Actually, strike that, I already have that image.

Here is my log entry from the 4th of Nov:

• Put firmware back to 5.0(6.0) due to problems with intermittent delay in receiving audio. First manifested itself on the 29th as delayed audio when transferring. David converted the 2nd floor back to 5.0(6.0). On the 30th, we converted users at Health that did a lot of xfers. 11-4-2004 saw audio delays on direct calls also.

Looks like you have two choices: Try the Engineering special image and see if that resolves the issue, or try the 7-1-1 image (which is supposed to contain the fixes in the ES image) and see if that resolves your problem.

PLEASE report back your findings. And thanks

jerake Tue, 11/30/2004 - 12:22

I applied the 7.1.1 load to all 600+ of our IP phones, both 7960's and 7940's. It's only been on for one day, but my "chronic" ppl who, without fail, always tell me if they experience the delay issue and echo have said that it has not happened. I will have to give it at least a week before I can know for sure that the problem has been resolved.

However, another issue has appeared: phones are randomly freezing up. This usually seems to happen when a user is switching over to a call that has come in on call waiting. The only way to fix it is to unplug the phone/plug it back in. Today alone it's happened five times over our three locations. We are using CCM version 3.3(4)sr2, OS version 2000.2.6sr5.

If the freezing keeps happening, I will have to switch back to load 6.0.5. Also, this load apparently can determine whether or not the call is in "secure" mode by showing a small lock icon next to the call timer. I find it odd that this icon shows up on some calls, but not all, and that it's even appearing at all since we're using a version of CCM that doesn't support those enhanced security features.

TMH

mpaul@innovativ... Fri, 12/03/2004 - 18:18

I wanted to wait about a week before I posted back on this one. On Sunday night this past week I applied the latest firmware updates to the 7960/40 phones at our site. Monday went OK so I felt confident in applying it to my client's site. On Monday evning I applied the firmware, 7.1.1 to my client site having this problem. I was on-site on Thursday and asked them if they were hearing the 3 second delay anymore. A quiet poll around the office and nobody has heard the delay, at least since Monday. Still hearing some echo to the PSTN but that's on the gateway of course and I'm massaging that one along.

I did not experience the freezing to the phones Jerake had experienced in the previous post. Both of my sites have 4.01 call managers and that's a little different from Jerake's site.

As a side note, they were experiencing a clipping when they went to unity, 'ello, cisco unity.' As of Monday this problem went away as well. The clipping has stopped and they hear 'Hello, Cisco Unity.' Thanks for the replies on this one. I'll keep an eye on both sites and post back if we have a problem.

jerake Fri, 12/03/2004 - 18:32

An addendum:

Whatever the freezing was, I think it had to do with the security features in 7.1.1 made for 4.0, while we're using 3.3(4) (just my speculation). The freezing problem disappeared when I went to load T017, given to me by Cisco TAC (which seems to be based off of the 6.0.X loads).

TMH

ikobisher3 Fri, 12/03/2004 - 19:09

Is there any way to get this firmware (T017) in some other way without opening a TAC case? Is it restricted to be shared ?

Thanks

IK

rvincent Sat, 12/04/2004 - 05:25

I'd like to make sure I'm understanding this:

The latest 7.1 phone load on a 4.0 cm has eliminated the audio delay. But it is casuing other problems with 3.3 cm even though it says it works with 3.3 cm.

The special T017 load with 3.3 cm has eliminated the audio delay without causing any other problems.

Have I got that right?

Thanks

Rob

tavery2000 Sat, 12/04/2004 - 09:58

Does anyone have expirence with 7.0 phone load and 7914 side cards with CCM 4.0.2 ?

Thanks

Tom

jeffkelso Mon, 12/06/2004 - 05:52

You must update the 7914 firmware when you upgrade the 7940/7960 firmware.

jerake Sat, 12/04/2004 - 12:22

rvincent:

That is what I am experiencing, yes, you've got it right.

The fact that it says 7.1.1 allegedly works with 3.3 was the first thought that came to mind when I started experiencing problems.

TMH

rvincent Wed, 12/08/2004 - 09:17

I'm having the same problem with a 3.3.4 customer. Has the P0030600T017 load you got from tac resolved the delay & your freezing?

thanks

Rob

togi@perlegen.com Wed, 12/08/2004 - 10:50

Rob,

Sorry about the delay but now I'm having some problems with the TFTP server serving out that darned version of the FW. Attempting to have a specialist come out soon and fix that, so I can try to fix the FW issue. Stay tuned.

T

Actions

Login or Register to take actions

This Discussion

Posted November 15, 2004 at 11:05 PM
Stats:
Replies:71 Avg. Rating:
Views:689 Votes:0
Shares:0
Tags: No tags.

Discussions Leaderboard