Problem with one-way audio over VPN and it's not a routing issue!

Unanswered Question
Feb 11th, 2010

I am having a problem with one-way audio over VPN.  CCM 6.1 with 7940s.  Several PIX 501s have a VPN tunnel setup to a central 515E (all on different subnets) and EVERYTHING works perfectly, even the 7940s and voice calls.  The problem is that if the phone is not used for a while then a call from another extension results in one way audio.  If you hang up and call right back, everything is fine.  In fact everything is fine as long as you use the phone on a regular basis but after a couple of hours a call from an extension results in one-way audio again (but for this first call only) and then everything is fine again.

If it was solidly one way audio it would be pretty easy to troubleshoot and fix but you only get one chance a day or so to "catch a falling star".  Some clues are that trunk calls (PSTN) via multiple gateways and PRIs (either in or out) always work fine and of course calls to Unity always work fine.

Everything (normal IP traffic) is flawless and you can ping any phone anywhere on the network; we just have this intermittent naging one-way audio problem in one out of 20 calls.  I turned off "fixup protocol skinny" at both ends as there is no NAT happening with these calls but it didn't make a difference. Any suggestions appreciated.

Thanks in advance!

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
steigja Thu, 02/11/2010 - 15:32

I think your problem is probably with the p2p tunnel, and that it is not up at the time of the first call, and then it times out without activity.  Do a "show crypto ipsec sa" during downtime on one of the remote sites, when there are no calls going on.  Do you see one tunnel up to the subnet for the publisher and one to the subscriber?  Perhaps when the first call is made the tunnel hasn't come up yet, then it is up after the first call.  Is the one way audio always in the same direction?

you should see the following for each tunnell.

local ident (addr/mask/prot/port): (X.X.X.X/255.255.255.0/0/0)
remote ident (addr/mask/prot/port): (X.X.X.X/255.255.255.0/0/0)

smoores Fri, 02/12/2010 - 05:31

Thanks for your answer; yeah I already looked at that.  Seems that SCCP is pretty chatty (a few packets every couple of seconds) so the tunnel is always up as the phone is constantly chatting with call manager when it's registered (even when idle).  And of course the phone wouldn't ring in the first place if the tunnel was down.  If I call a remote user they can hear me (but I can't hear them), similarily if I am at a remote site and call an extension, I can't hear them but they can't hear me so yeah I guess it's always in the same direction.

We do have another "clue".  I told the user that the first time they go off hook to immediately place the caller on hold (for 1/2 sec) and then hit resume and presto, two-way perfect audio.  I seem to remember that there was a bug with the softphones for a while and this was the work around but these are hard phones (7940) but it seems related somewhat.

Any insight from anyone on where to look next would be much appreciated.

CHRIS CHARLEBOIS Fri, 02/12/2010 - 09:54

The putting-on-hold 'work-around' makes sense, because taking someone off hold is alot like making a new call; In both cases, a new RTP stream is created.  So something must be wonky with the SCCP commands to setup that first RTP stream.  And since they can hear you, but you can't hear them, the problem must be at the remote site.  Or at least traffic coming from the remote site.  Can you capture the traffic between the CUCM and the remote phone?  Specifically, the SCCP traffic?  And decode it to see what address the phone is told to establish the RTP stream with.

Chris

CHRIS CHARLEBOIS Fri, 02/12/2010 - 09:57

Wait, I'm confused about the direction of the problem.  Is it true that, regardless who dials whom, the remote user (i.e. the one *not* local to the CUCM) can hear the local user, and the local user cannot hear the remote user?

smoores Fri, 02/12/2010 - 11:07

Just the opposite, it's the other way around; the remote user cannot hear (no matter who calls who).  The distrubing thing is that incoming and outgoing PSTN trunk calls always work fine; just one extension to another is a problem.  Yet if you take note of some phone IPs and ping them from the remote site there is never a lost packet. Everything I've done to test connectivity is perfect.  I guess I am into 'catching a failing star' with a packet capture.  It almost seems like an ARP table issue doesn't it?  Like once we cache something in the ARP table we are good until it expires.

dsitsgroup Sat, 01/29/2011 - 08:20

Did you ever find the fix for this? I have the same problem now.

paulburkecs Tue, 02/01/2011 - 07:58

Sorry I was not involved in this. I will forward to our consultant and see if he can help.

Thanks

smoores Fri, 12/20/2013 - 08:23

The problem was resolved!  Turns out it was something weird the PIX was doing (even though the traffic wasn't being NATed or rather specifically exempted from NAT and there was therefore "no fixup skinny").  When I replaced the 515 with a real "router" (1841) that has an access list to only allow IPsec in, everything started working perfectly, as an added plus the router does spoke to spoke traffic via the hub site so remote sites can talk to remote sites via the hub. All the spokes are dynamic IPs (DHCP).  There are a mixture of PIX 501s (to be replaced with 5505s), ASA5505s and other routers at spoke sites and everything talks to everything perfectly now. While the PIX makes a great firewall, nothing routes like a router. Surprisingly I set this up without DMVPN (PIX and ASAs can't do DMVPN anyways) but have the benefits of full connectivity DMVPN provides between spoke sites.

ip access-list extended IPSEC-ONLY

  remark Allow only ISAKMP and ESP traffic inbound

  permit udp any host X.X.X.X eq isakmp

  permit esp any host X.X.X.X

  deny   ip any any


interface FastEthernet0/0

  description Internet facing external interface

  ip address X.X.X.X

  ip access-group IPSEC-ONLY in

Actions

This Discussion