Can't get MoH working with H.323 Gateway

Answered Question
Apr 18th, 2007
User Badges:
  • Bronze, 100 points or more

I've converted an MGCP gateway to H.323, and am having major problems with Music on Hold. It worked just fine with MGCP. When a user makes an outbound call (G.711) and hits the hold button, the call drops either immediately, or upon resuming. When the call drops immediately, I get DSP error messages like the following:


Apr 18 19:17:10 PDT: %C5510-1-C5510_CHPI_ERROR: cHPI error for pa_bay 0 pump 0 dsp 1.

Apr 18 19:17:13 PDT: %C5510-1-NO_RING_DESCRIPTORS: No more ring descriptors available on slot 0 dsp 1.


I found this TAC case:


http://www.ciscotaccc.com/kaidara-advisor/voice/showcase?case=K24619534


So I have changed one of my Music on Hold servers to Multicast, then added that Gateway and a test phone to a MRGL that uses the Multicast server. The calls are no longer dropping, but still no MoH. Any ideas? Gateway config:


!

network-clock-participate wic 1

network-clock-select 1 T1 0/1/0

!

voice service voip

allow-connections h323 to h323

fax protocol t38 ls-redundancy 0 hs-redundancy 0 fallback pass-through g711ulaw

h323

no h225 timeout keepalive

call preserve

session transport udp

!

!

voice class codec 1

codec preference 1 g711ulaw

codec preference 2 g729r8

!

!

!

voice class h323 1

h225 timeout tcp establish 3

!

controller T1 0/1/0

framing esf

linecode b8zs

pri-group timeslots 1-24

!

interface Loopback0

ip address 10.1.1.2 255.255.255.255

h323-gateway voip interface

h323-gateway voip h323-id GatewayB

h323-gateway voip bind srcaddr 10.1.1.2

!

interface Serial0/1/0:23

no ip address

encapsulation hdlc

isdn switch-type primary-ni

isdn incoming-voice voice

no cdp enable

!

voice-port 0/1/0:23

!

ccm-manager music-on-hold

!

dial-peer voice 1 voip

incoming called-number .

codec transparent

no vad

!

dial-peer voice 100 pots

description Outbound calls

destination-pattern .T

port 0/1/0:23

!

dial-peer voice 400 voip

destination-pattern ^[1-8]...$

voice-class codec 1

voice-class h323 1

session target ipv4:10.1.2.2

dtmf-relay h245-alphanumeric

no vad

!

dial-peer voice 401 voip

preference 1

destination-pattern ^[1-8]...$

voice-class codec 1

voice-class h323 1

session target ipv4:10.1.2.3

dtmf-relay h245-alphanumeric

no vad

!


IOS version of the Gateway is 12.4(9)T3, DSPware 8.4.4. CCM version is 4.1(3)sr4b

Correct Answer by thisisshanky about 10 years 1 week ago

Have you also made sure multicasting is enabled in your network ? If the router is in a different subnet than MOH server, all l3 devices in the path should be enabled for multicasting.


on a router add following commands.


ip multicast-routing

int fa0/0

ip pim sparse-dense-mode.


You already have the ccm-manager music-on-hold command. So thats good. Also add this command for safety.


ccm-manager music-on-hold bind fa0/0.


Use sh ccm-manager music-on-hold to verify moh streaming.


HTH


Sankar


PS: please remember to rate posts!



  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (2 ratings)
Loading.
Correct Answer
thisisshanky Wed, 04/18/2007 - 18:53
User Badges:
  • Purple, 4500 points or more

Have you also made sure multicasting is enabled in your network ? If the router is in a different subnet than MOH server, all l3 devices in the path should be enabled for multicasting.


on a router add following commands.


ip multicast-routing

int fa0/0

ip pim sparse-dense-mode.


You already have the ccm-manager music-on-hold command. So thats good. Also add this command for safety.


ccm-manager music-on-hold bind fa0/0.


Use sh ccm-manager music-on-hold to verify moh streaming.


HTH


Sankar


PS: please remember to rate posts!



johnnylingo Wed, 04/18/2007 - 19:15
User Badges:
  • Bronze, 100 points or more

You're on to it...I just checked the core 6500s here and found multicasting to be disabled. I'll have to check with the other Network guys here before turning that on.

johnnylingo Wed, 04/18/2007 - 19:19
User Badges:
  • Bronze, 100 points or more

Something interesting - I also didn't have "Allow multicasting" check in the Music on Hold Audio Source. When I did that and tried to put a user on hold, I again got the DSP crash mentioned above.

kelvin.blair Wed, 04/18/2007 - 19:35
User Badges:
  • Silver, 250 points or more

First let me say this, Multicast MOH will not work if multicast isn't configured in the network. Secondly, I've see this same problem in multiple H323 gateways running 12.9.4T IOS. The DSP firmware that your running is the cause of this issue. It is a bug but only effects h323 gateways. So that is why you didn't see this problem with the MGCP configuration. You need to upgrade the DSPfirmware. The version that I found that fixed the problem amoung other issues that I was experiencing like dropped POTS calls was firmware 9.2.0. I would find an IOS with this DSP firmware builtin or open a TAC case and have them give it to you. That is how I got it.

johnnylingo Wed, 04/18/2007 - 19:37
User Badges:
  • Bronze, 100 points or more

Thanks for the input. I did find it really strange the a lack of Multicast support on the network would case a DSP crash, so that explains the other part of it.


We really want H.323 call preservation so calls stay up when a CallManager is rebooted (there's a 24/7 call center in the mix), so I think 12.4(9)T is my only option, if I'm understanding correctly.

thisisshanky Wed, 04/18/2007 - 20:30
User Badges:
  • Purple, 4500 points or more

I have two gateways running h323, 12.3.14T7, MMOH works just fine. Looks like with 12.4, newer dsp firmware creates this problem.

kelvin.blair Thu, 04/19/2007 - 04:52
User Badges:
  • Silver, 250 points or more

That is true, its the DSP Firmware that does create the problem. I would upgrade the IOS or DSP FIRMWARE. Since you're wanting call preserve, Then go to 12.4.11T1 It should have the fix.


Be sure to rate the post if it helps

johnnylingo Fri, 04/20/2007 - 08:58
User Badges:
  • Bronze, 100 points or more

Something I'm confused on... With 12.3(14)T7, the DSPware version is 4.4.708 and with 12.4(7e) it's 4.4.24. Which is the older version? I would assume 4.4.24, but 12.3(14)T7 was released over a year ago so that doesn't seem to make sense.

thisisshanky Fri, 04/20/2007 - 09:23
User Badges:
  • Purple, 4500 points or more

I got confused the first time i saw that too. I believe 12.4 should have newer dspware built into ios.

johnnylingo Fri, 04/20/2007 - 13:20
User Badges:
  • Bronze, 100 points or more

Just reviewed my logs, and can certainly confirm IOS 12.4(9)T3 w/ DSPware 8.4.4 is buggy. One of the PRIs is still MGCP, and it has bounced about 5 times in the last 24 hours. The MGCP-controlled PRI did not have any calls on it at the time of the bounce, but the H.323 PRI did. An identically configured gateway still running 12.3(7e) has had no problems.


johnnylingo Mon, 04/23/2007 - 10:05
User Badges:
  • Bronze, 100 points or more

Ooop, so much for that. My MGCP PRI did bounce a couple times over the weekend, although that's not nearly as much as the other Gateway running 12.4(9)T3. Error messages were:


Apr 21 20:47:50 PDT: %C5510-1-C5510_CHPI_ERROR: cHPI error for pa_bay 0 pump 0 dsp 1.

Apr 21 20:47:51 PDT: %C5510-1-NO_RING_DESCRIPTORS: No more ring descriptors available on slot 0 dsp 1.

Apr 21 20:47:56 PDT: %C5510-1-NO_RING_DESCRIPTORS: No more ring descriptors available on slot 0 dsp 1.

Apr 21 20:48:01 PDT: %C5510-1-NO_RING_DESCRIPTORS: No more ring descriptors available on slot 0 dsp 1.

Apr 21 20:48:06 PDT: %C5510-1-NO_RING_DESCRIPTORS: No more ring descriptors available on slot 0 dsp 1.

Apr 21 20:48:07 PDT: %LINK-3-UPDOWN: Interface 0/0/1:23(1), changed state to Administrative Shutdown

...

Apr 21 20:48:12 PDT: %LINK-3-UPDOWN: Interface 0/0/1:23(16), changed state to Administrative Shutdown

Apr 21 20:48:11 PDT: %C5510-1-NO_RING_DESCRIPTORS: No more ring descriptors available on slot 0 dsp 1.

Apr 21 20:48:14 PDT: %DSPRM-5-UPDOWN: DSP 1 in slot 0, changed state to up

Apr 21 20:48:15 PDT: %LINK-3-UPDOWN: Interface 0/0/1:23(1), changed state to up

...

Apr 21 20:48:20 PDT: %LINK-3-UPDOWN: Interface 0/0/1:23(16), changed state to up


Since this is running 12.4(7e), I will be curious to hear what TAC recommends.

kelvin.blair Mon, 04/23/2007 - 10:16
User Badges:
  • Silver, 250 points or more

I running 12.9.4T1 with dspware 9.2.2 I upgraded my DSPware to this version per TAC recommendation. It worked for me..

johnnylingo Tue, 04/24/2007 - 12:08
User Badges:
  • Bronze, 100 points or more

Have an issue with our contract (it was inherited from the previous vendor), so haven't been able to open a TAC case yet. However, did downgrade to 12.3(14)T7 last night and have had no problems today.

thisisshanky Mon, 04/23/2007 - 10:32
User Badges:
  • Purple, 4500 points or more

Johnny,


Have you considered asking TAC about hardware issues with the DSP ? Can you do a dsp test to see if they are fully operational ? With pvdm-12s you could easily run test dsp command (hidden command). But with the C5510 dsp's you have to use a different command ( I unfortunately dont remember off top of my head, what the command was. I will try to find this).

johnnylingo Mon, 04/23/2007 - 11:40
User Badges:
  • Bronze, 100 points or more

"test voice driver", then select Keepalive status.


Yep, ran that and it shows all DSPs as alive.


While I doubt this is a hardware problem, I did have a report a few weeks ago where a group said calls were dropping after placing users on hold. But I couldn't find anything in the logs. At the time, the gateway was running 12.3(11)T and just MGCP.



thisisshanky Tue, 04/24/2007 - 12:10
User Badges:
  • Purple, 4500 points or more

Yeah, That IOS is very stable (atleast so far i have not had any issues).

johnnylingo Wed, 05/02/2007 - 13:18
User Badges:
  • Bronze, 100 points or more

So I've flipped on Multicast on all my Layer 3 devices and have Multicast MoH working between the phones, but still same result when I try to use it on the Gateway. This happens both in IOS 12.3(14)T7 and 12.4(11)T1. Ideas? Interface Gig0/1 is attached to the same switch as the CCM (different VLAN though)


Config:


interface GigabitEthernet0/1

ip address 10.1.2.3 255.255.255.0

ip pim sparse-dense-mode

duplex auto

speed auto

!

ccm-manager music-on-hold bind GigabitEthernet0/1



#debug ccm-manager music-on-hold all


May 2 21:33:48.458: moh_update_rtp: callID 199 dstCallID 200

May 2 21:33:48.458: moh_update_rtp: callID 199 dstCallID 200

May 2 21:33:48.458: moh_update_rtp: callID 199 dstCallID 200

May 2 21:33:48.458: moh_update_rtp: callID 199 dstCallID 200

May 2 21:33:48.482: moh_process_ccb: dstadr 239.1.1.1, callid 200, port 16384,

codec 5, moh_en 0, moh_addr 0.0.0.0

May 2 21:33:48.482: moh_process_ccb:multicast addr add_ccb

May 2 21:33:48.482: moh_add_ccb: ip addr 239.1.1.1 port 16384 callid 200

May 2 21:33:48.482: moh_add_ccb: vmccb does not exists - creating a

new one for 239.1.1.1 through IGMP

May 2 21:33:48.482: moh_join_group_command called for 239.1.1.1

May 2 21:33:48.482: moh_join_group_command: igmp issued on grp 239.1.1.1 in interface Gi0/1

May 2 21:33:48.486: moh_create_session: called

May 2 21:33:48.486: moh_create_session : dstadr 239.1.1.1 does not exist - creating a control block

May 2 21:33:48.486: moh_insert_multicast_hashtable:moh_insert_multicast_hashtable buc 0

May 2 21:33:48.486: moh_create_session : Created a new vmccb for 239.1.1.1

May 2 21:33:48.486: moh_add_ccb: Done inserting CCB for 239.1.1.1

May 2 21:33:48.486: moh_update_rtp: callID 199 dstCallID 200


#show ccm-manager music-on-hold

Current active multicast sessions : 1

Multicast RTP port Packets Call Codec Incoming

Address number in/out id Interface

===================================================================

239.1.1.1 16384 3965/3965 193 g711ulaw Gi0/1


EH-VG-Main-2#sh ip igmp groups 239.1.1.1

IGMP Connected Group Membership

Group Address Interface Uptime Expires Last Reporter Group Accounted

239.1.1.1 GigabitEthernet0/1 00:01:45 00:02:50 10.1.2.3


#show ip mroute count

Group: 239.1.1.1, Source count: 2, Packets forwarded: 1, Packets received: 20385

RP-tree: Forwarding: 0/0/0/0, Other: 0/0/0

Source: 1.2.3.4/32, Forwarding: 0/-50/0/0, Other: 20383/0/20383

Source: 10.1.2.1/32, Forwarding: 1/0/96/0, Other: 2/1/0

# DSP Crash:


May 2 14:27:08 PDT: %C5510-1-C5510_CHPI_ERROR: cHPI error for pa_bay 0 pump 0 dsp 1.

May 2 14:27:11 PDT: %C5510-1-NO_RING_DESCRIPTORS: No more ring descriptors available on slot 0 dsp 1.

May 2 14:27:16 PDT: %C5510-1-NO_RING_DESCRIPTORS: No more ring descriptors available on slot 0 dsp 1.

May 2 14:27:21 PDT: %C5510-1-NO_RING_DESCRIPTORS: No more ring descriptors available on slot 0 dsp 1.

May 2 14:27:26 PDT: %C5510-1-NO_RING_DESCRIPTORS: No more ring descriptors available on slot 0 dsp 1.

May 2 14:27:30 PDT: %DSPRM-5-UPDOWN: DSP 1 in slot 0, changed state to up



thisisshanky Wed, 05/02/2007 - 13:27
User Badges:
  • Purple, 4500 points or more

Johnny,


Do you see any multicast traffic on the router when you put a pstn call on hold. You can use command sh ip mroute. Also use sh ccm-manager music-on-hold to see if a stream is being played back to the pstn.

johnnylingo Wed, 05/02/2007 - 13:36
User Badges:
  • Bronze, 100 points or more

Yes, when I do "show ccm-manager music-on-hold" I see the group added, and can even watch the patckets increase.


I will add my debug output now.

thisisshanky Wed, 05/02/2007 - 13:39
User Badges:
  • Purple, 4500 points or more

So is your dspware now back to 4.4.708 ?

thisisshanky Wed, 05/02/2007 - 14:44
User Badges:
  • Purple, 4500 points or more

I have two 3845s, h323, 12.3.14T7 and dsp ware 4.4.708. Dont see these dsp crashes on the router. MMOH plays just fine. Only other thing i can recommend is try swapping the dsp's.

johnnylingo Wed, 05/02/2007 - 15:21
User Badges:
  • Bronze, 100 points or more

Yes, I'm starting to suspect this is hardware related. I'll do some hardware swaps and post results.

johnnylingo Thu, 05/03/2007 - 17:14
User Badges:
  • Bronze, 100 points or more

Tried swapping the DSPs; no luck.


Can you post your config?

johnnylingo Wed, 05/02/2007 - 15:20
User Badges:
  • Bronze, 100 points or more

Actually on IOS 12.4(11)T1 now, so 9.X. But same exactly problem when I go back to 4.4.708



thisisshanky Thu, 05/03/2007 - 19:26
User Badges:
  • Purple, 4500 points or more

Nothing big in the config:


ip multicast-routing


ccm-manager music-on-hold


interface GigabitEthernet0/0

ip address 255.255.255.0

ip pim sparse-dense-mode

duplex auto

johnnylingo Fri, 05/04/2007 - 14:26
User Badges:
  • Bronze, 100 points or more

If you do "show ip igmp groups 239.1.1.1", do you always see entries, or just when a call is on hold?

thisisshanky Fri, 05/04/2007 - 15:00
User Badges:
  • Purple, 4500 points or more

Since PIM is enabled, IGMP should automatically be enabled on the interface. Check the output below, when i just put the pstn caller off hold, so now there are no moh sessions playing back to pstn, but the igmp group registration is still active all the time.



ROUTER# sh ccm-manager music-on-hold

Current active multicast sessions : 0

ROUTER#

ROUTER#

ROUTER#sh ip igmp groups 239.22.32.17

IGMP Connected Group Membership

Group Address Interface Uptime Expires Last Reporter

239.22.32.17 GigabitEthernet0/0 2w1d 00:02:19 1.1.1.1

ROUTER#

ROUTER#

ROUTER#sh ip igmp groups 239.22.32.17

IGMP Connected Group Membership

Group Address Interface Uptime Expires Last Reporter

239.22.32.17 GigabitEthernet0/0 2w1d 00:02:15 1.1.1.1



johnnylingo Tue, 05/08/2007 - 13:43
User Badges:
  • Bronze, 100 points or more

Got a TAC case open, but it's moving slowly.


I've been able to eliminate hardware, because I'm bring up a 3845 and having the exactly same problem, again regardless of IOS/DSPWare versions. I think this is a multicast problem, because even though my membership looks good:


VG-3845-TEST#sh ip igmp groups 239.1.1.1

IGMP Connected Group Membership

Group Address Interface Uptime Expires Last Reporter Group Accounted

239.1.1.1 GigabitEthernet0/0 20:06:18 00:02:50 1.2.3.4


Running "debug ccm-manager music-on-hold" shows the following:


May 8 21:37:44.776: moh_update_rtp: callID 24 dstCallID 25

May 8 21:37:44.776: moh_update_rtp: callID 24 dstCallID 25

May 8 21:37:44.776: moh_update_rtp: callID 24 dstCallID 25

May 8 21:37:44.784: moh_update_rtp: callID 24 dstCallID 25

May 8 21:37:44.800: moh_process_ccb: dstadr 239.1.1.1, callid 25, port 16384,

codec 5, moh_en 0, moh_addr 0.0.0.0

May 8 21:37:44.800: moh_process_ccb:multicast addr add_ccb

May 8 21:37:44.800: moh_add_ccb: ip addr 239.1.1.1 port 16384 callid 25

May 8 21:37:44.800: moh_add_ccb: vmccb does not exists - creating a

new one for 239.1.1.1 through IGMP

May 8 21:37:44.800: moh_join_group_command called for 239.1.1.1

May 8 21:37:44.800: moh_join_group_command: igmp issued on grp 239.1.1.1 in interface Gi0/0

May 8 21:37:44.800: moh_create_session: called

May 8 21:37:44.800: moh_create_session : dstadr 239.1.1.1 does not exist - creating a control block

May 8 21:37:44.800: moh_insert_multicast_hashtable:moh_insert_multicast_hashtable buc 0

May 8 21:37:44.800: moh_create_session : Created a new vmccb for 239.1.1.1

May 8 21:37:44.800: moh_add_ccb: Done inserting CCB for 239.1.1.1

May 8 21:37:44.800: moh_update_rtp: callID 24 dstCallID 25

May 8 14:37:44 PDT:


Also if I check the multicast routing table, there is no outbound interface for the route:


VG-3845-TEST#sh ip mroute 239.1.1.1

IP Multicast Routing Table

Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,

L - Local, P - Pruned, R - RP-bit set, F - Register flag,

T - SPT-bit set, J - Join SPT, M - MSDP created entry,

X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,

U - URD, I - Received Source Specific Host Report,

Z - Multicast Tunnel, z - MDT-data group sender,

Y - Joined MDT-data group, y - Sending to MDT-data group

Outgoing interface flags: H - Hardware switched, A - Assert winner

Timers: Uptime/Expires

Interface state: Interface, Next-Hop or VCD, State/Mode

(*, 239.1.1.1), 20:08:27/stopped, RP 1.2.4.3, flags: SJPL

Incoming interface: GigabitEthernet0/0, RPF nbr 1.2.3.1

Outgoing interface list: Null

(4.4.4.4, 239.1.1.1), 20:07:41/00:02:59, flags: PLT

Incoming interface: GigabitEthernet0/0, RPF nbr 0.0.0.0

Outgoing interface list: Null


1.2.3.1 is the Server Switch, and 4.4.4.4 is the Unicast IP address of the CCM. It runs NIC teaming, which I'm starting to suspect as the problem.


thisisshanky Tue, 05/08/2007 - 14:59
User Badges:
  • Purple, 4500 points or more

Nic teaming should not matter. I am running CM 5.x in this setup and have failover type nic teaming configured. It works just fine.

johnnylingo Tue, 05/08/2007 - 15:02
User Badges:
  • Bronze, 100 points or more

Just curious - if you do a "show ip mroute 239.22.32.17" what does it show for the outbound interface?

johnnylingo Wed, 05/09/2007 - 14:17
User Badges:
  • Bronze, 100 points or more

I found out today I get the same crash when I try to join a conference. Also, if I click "Media Termination Point Required" in CCM, the DSP crash occurs as soon as the remote phone is rings.


So essentially, as soon as the H.323 gateway tries to talk to one of the CCM's, the DSP crash occurs.

johnnylingo Wed, 05/09/2007 - 14:52
User Badges:
  • Bronze, 100 points or more

Found a fix. Changed this:


dial-peer voice 1 voip

codec transparent

incoming called-number .

!


dial-peer voice 1 voip

voice-class codec 1

incoming called-number .

!


So looks like it's a Codec negotiation bug in the H.323 stack. Will post bug ID when I have it.


Thank you for everyone's input.

thisisshanky Wed, 05/09/2007 - 14:58
User Badges:
  • Purple, 4500 points or more

I am using "codec g711ulaw". IOS 12.3.14T7. What did you have in your voip dial-peer before ?

johnnylingo Mon, 06/04/2007 - 10:05
User Badges:
  • Bronze, 100 points or more

I had "codec transparent". Either specifying the codec or using a voice-class fixes it.


Unfortunately the TAC engineer could not duplicate this in the lab, so I never got a bug to come out of it. Would be curious if anyone else can duplicate it.

Actions

This Discussion