FXO Trunk - Choppy Audio

Unanswered Question
Jan 31st, 2008
User Badges:

Hi all,


When calls come into an FXO port on a MGCP controlled 2821 they are routed over the WAN to an auto attendant on a Unity box. The auto attendant speech is very choppy. QoS best practices are properly set and I have played with input gain/output attenuation and still no luck. Jitter and lost packets aren't the culprits either. Any other suggestions to check before I involve the telco?


Thanks

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
davex4285 Fri, 02/01/2008 - 06:49
User Badges:

1. The codec I am using is G729 and this problem is only occuring at one of the remote sites on one FXO port.


2. Thanks for the link for the sweep method, I will have to try that.

Paolo Bevilacqua Fri, 02/01/2008 - 07:03
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    Founding Member

Hi, the problem with g.729 is that is very sensitive to packet jitter and loss. So if the circuit is less than good, that is the effect. Check your packet loss statistics in unity of from an IP phone pressing ? twice.


Then if you use g.711 instead, it will take up more bandwidth, but when a packet is lost or delayed, the effect is not as bad as with G.729.


Hope this helps, please rate post if it does!



davex4285 Fri, 02/01/2008 - 07:07
User Badges:

Yeah I am suspecting the 768kpbs link is causing a problem, although when I gather statistics using the ? packet loss and jitter don't seem to be a problem. I do not have LFI on this link so I am wondering if delay could be causing a problem.

Paolo Bevilacqua Fri, 02/01/2008 - 07:09
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    Founding Member

Hi, a true 768 link very reasonably should not cause any problem.


Delay in itself also do not cause problems. Only high jitter and packet loss. Try calling a phone co-located with unity, or check stats at the GW.

davex4285 Fri, 02/01/2008 - 07:13
User Badges:

A phone that is co-located on the same LAN as Unity goes through clear, as does internal calls to Unity across the WAN.


What stats do you suggest I check on the GW?

Paolo Bevilacqua Fri, 02/01/2008 - 07:21
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    Founding Member

Output of "show call active voice brief".

Now if the call is OK to a phone, but choppy to unity, wouldn't that point to unity in first instance ?

davex4285 Fri, 02/01/2008 - 07:25
User Badges:

Well when the call comes into the gateway it is immediately routed over the WAN to Unity's call handler. Sometimes it is clear as day and other times it is choppy. If I transfer to an IP phone it seems to be choppy as well. All other calls to unity work fine except from this Line.


This is a remote site so I am going to have a cabling resource out there next week to put a butt set on the line and see if the quality is the same.


Thanks for the help.

Paolo Bevilacqua Fri, 02/01/2008 - 14:22
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    Founding Member

No problem. All symptoms are pointing to a packet loss or jitter issue. Let us know what you find.

davex4285 Fri, 02/01/2008 - 14:30
User Badges:

Thats what I though initially but the statistics are showing jitter and packet loss as a non-issue. Since there are 5 analog lines coming in and only 1 line is having an issue, I am starting to suspect the telco line or the connections in the closet.

pcameron Fri, 02/01/2008 - 16:24
User Badges:
  • Cisco Employee,

Try swapping the suspect line to a different port and seeing if the problem follows it. Take a working line from another port and see if it has the same issue on the suspect voice port. Make sure you DO NOT have command 'compand-type a-law' configured on the analogue ports as this will cause distorted audio between other ports (all analogue ports use ulaw coding by default)


Otherwise, if you still suspect a network issue, can you do a "SH CALL ACTIVE VOICE" command when it occurs. This will give details on the overall quality of the call.


If there is excessive latency on the network (most often caused by larger MTU packets, the voice traffic may get delayed slightly) The adaptive de jitter buffers on the voice ports try to accomodate for this , but by the time they have noticed the increasing delay, the problem has already occured.


The output of the command is pretty verbose, but what we are interested in are these stats -


Syd_Rtr1#sh call active voice


…


OnTimeRvPlayout=930280

GapFillWithSilence=1240 ms <----

GapFillWithPrediction=5090 ms <----

GapFillWithInterpolation=0 ms

GapFillWithRedundancy=0 ms

HiWaterPlayoutDelay=150 ms

LoWaterPlayoutDelay=50 ms

ReceiveDelay=143 ms <----

LostPackets=0

EarlyPackets=299

LatePackets=401 <----

VAD = disabled


Over the duration of the call, refresh the command and see if the stats that have '<----' beside them increment.If they do, then there is variable delay across your network which causes the traffic to the router voice to exceed the default de jitter (playout) buffer time. The router attempts to fill the missing audio samples with a predicted value (gapfill with prediction) or silence (gapfill with silence). These are evident as choppy audio.


As the IP phones have a larger buffer (maximum of 1000msec), jitter across the network is not so apparent as they will wait a bit longer for the rtp media to come in and hence be more tolerant of gaps between packets.


So what can you do ?


1) Use Link Fragmentation on the serial interface - even though your link speed is 768K, for all you know there may a slower connection in the data path somewhere in the middle which is the source of the problem.


2) Under the voice port, add this command -


playout-delay nominal 250


This will Increase the default de-jitter buffer to the max of 250msec, so it will accomodate much greater periods of jitter.

davex4285 Sat, 02/02/2008 - 14:17
User Badges:

Wow pcameron thanks for the detailed assistance. I will be troubleshooting this issue Tuesday morning so I will definitely be testing out your ideas.


Thanks again.

davex4285 Fri, 02/08/2008 - 06:20
User Badges:

Hey all here are some of the statistics I gathered from a show call active voice.


9:30am - 30 Second Period

Gaps Filled with Silence = 1240ms

Gaps Filled with Prediction = 5090ms

Late Packets = 265


9:45am - 30 Second Period

Gaps Filled with Silence = 780ms

Gaps Filled with Prediction = 3080ms

Late Packets = 243


10:15am - 30 Second Period

Gaps Filled with Silence = 1330ms

Gaps Filled with Prediction = 5050ms

Late Packets = 454


10:30am - 30 Second Period

Gaps Filled with Silence = 2460ms

Gaps Filled with Prediction = 7740msms

Late Packets = 754


11:15am - 30 Seconds

Gaps Filled with Silence = 1120ms

Gaps Filled with Prediction = 4650ms

Late Packets = 379


I believe this is showing Jitter, although when I changed the jitter buffer using playout-delay nominal 250 I was still having the same issues.


pcameron Sat, 02/09/2008 - 20:39
User Badges:
  • Cisco Employee,

this is showing that you have a big problem with delay and jitter across your newtwork. Gap fill with silence means the DSP was not able to fill a pause between audio traffic, so it just left it as a silent period, gap fill with prediction means the DSP predicted what should have been inserted into the Audio stream and it made a best guess. Late packets means packets that arrived , but exceeded the playout buffer window.


You need to set up all the QOS correctly to esnure the voice traffic is prioritised, and use the LFI/interleaving to ensure larger packets are not causing serialisation delay.


Refer here for more details -


http://www.cisco.com/en/US/tech/tk652/tk698/technologies_tech_note09186a0080094660.shtml


If this does not help, it suggests there is excessive delay on the connection to the sites and you would need to get this looked at by the service provider.



davex4285 Sun, 02/10/2008 - 11:22
User Badges:

QoS is set up correctly, the only issue is LFI is not enabled. We would need to do this on site or with OOB modems which isn't an option at this point. The service provider is looking into it. This is the only remote site with issues.


Thanks again for your help pcameron.

Actions

This Discussion