SRST

teddywijaya84 · ‎06-07-2010

Hi All,

I have an UCM version 7.0 and several sites with voice gateways 2801 (SRST gateway) that registered to UCM using H.323 protocol.

The problem is the phones can register/unregister to the SRST gateway more than 3 times although the WAN link is UP. And I'm sure the WAN link is quite reliable, because I have test ping with size 8000 repeat 1000 and the result is 100% success. This problem only happened at 1 site, all the others sites are working fine. Also this problem occurs in the last 2 months, before there is no problem.

The configuration on all the SRST gateways are same. Here is the SRST configuration :

call-manager-fallback

max-conferences 4 gain -6

transfer-system full-consult

ip source-address 10.21.110.1 port 2000

max-ephones 30

max-dn 60

Here is the logging from the SRST gateway

Log Buffer (10000 bytes):

RMAL: ephone-7:SEP002497355DF0 IP:10.21.110.52 Socket:1 DeviceType:Phone has unregistered normally.

001565: Jun 7 00:00:13 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP002497355DF0 Load = SCCP41.8-4-1S Last=TCP-timeout

001566: Jun 7 00:00:13 WIB: %IPPHONE-6-REGISTER_NEW: ephone-7:SEP002497355DF0 IP:10.21.110.52 Socket:1 DeviceType:Phone has registered.

001567: Jun 7 00:00:13 WIB: %VOICE_CODEC-6-FALLBACK: ephone-7: Configured codec is not supported by phone , fallback to codec g711ulaw

001568: Jun 7 00:00:17 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP00249735A319 Load= SCCP41.8-4-1S Last=TCP-timeout

001569: Jun 7 00:00:21 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP002497355F08 Load= SCCP41.8-4-1S Last=TCP-timeout

001570: Jun 7 00:00:21 WIB: %IPPHONE-6-REGISTER_NEW: ephone-6:SEP002497355F08 IP:10.21.110.47 Socket:5 DeviceType:Phone has registered.

001571: Jun 7 00:00:21 WIB: %VOICE_CODEC-6-FALLBACK: ephone-6: Configured codec is not supported by phone , fallback to codec g711ulaw

001572: Jun 7 00:02:51 WIB: %IPPHONE-6-UNREGISTER_NORMAL: ephone-6:SEP002497355F08 IP:10.21.110.47 Socket:5 DeviceType:Phone has unregistered normally.

001573: Jun 7 00:03:04 WIB: %IPPHONE-6-UNREGISTER_NORMAL: ephone-7:SEP002497355DF0 IP:10.21.110.52 Socket:1 DeviceType:Phone has unregistered normally.

001574: Jun 7 01:18:27 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP00249735658C Load= SCCP41.8-4-1S Last=TCP-timeout

001575: Jun 7 01:18:27 WIB: %IPPHONE-6-REGISTER_NEW: ephone-5:SEP00249735658C IP:10.21.110.51 Socket:1 DeviceType:Phone has registered.

001576: Jun 7 01:18:27 WIB: %VOICE_CODEC-6-FALLBACK: ephone-5: Configured codec is not supported by phone , fallback to codec g711ulaw

001577: Jun 7 01:18:30 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP002497355DF0 Load= SCCP41.8-4-1S Last=TCP-timeout

001578: Jun 7 01:21:02 WIB: %IPPHONE-6-UNREGISTER_NORMAL: ephone-5:SEP00249735658C IP:10.21.110.51 Socket:1 DeviceType:Phone has unregistered normally.

001579: Jun 7 02:03:15 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP00249735A319 Load= SCCP41.8-4-1S Last=TCP-timeout

001580: Jun 7 02:03:15 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP002497355F08 Load= SCCP41.8-4-1S Last=TCP-timeout

001581: Jun 7 02:03:15 WIB: %IPPHONE-6-REGISTER_NEW: ephone-6:SEP002497355F08 IP:10.21.110.47 Socket:3 DeviceType:Phone has registered.

001582: Jun 7 02:03:15 WIB: %VOICE_CODEC-6-FALLBACK: ephone-6: Configured codec is not supported by phone , fallback to codec g711ulaw

001583: Jun 7 02:05:46 WIB: %IPPHONE-6-UNREGISTER_NORMAL: ephone-6:SEP002497355F08 IP:10.21.110.47 Socket:3 DeviceType:Phone has unregistered normally.

001584: Jun 7 02:14:19 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP002497355F08 Load= SCCP41.8-4-1S Last=TCP-timeout

001585: Jun 7 02:14:19 WIB: %IPPHONE-6-REGISTER_NEW: ephone-6:SEP002497355F08 IP:10.21.110.47 Socket:2 DeviceType:Phone has registered.

001586: Jun 7 02:14:19 WIB: %VOICE_CODEC-6-FALLBACK: ephone-6: Configured codec is not supported by phone , fallback to codec g711ulaw

001587: Jun 7 02:14:20 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP00249735658C Load= SCCP41.8-4-1S Last=TCP-timeout

001588: Jun 7 02:14:20 WIB: %IPPHONE-6-REGISTER_NEW: ephone-5:SEP00249735658C IP:10.21.110.51 Socket:3 DeviceType:Phone has registered.

001589: Jun 7 02:14:20 WIB: %VOICE_CODEC-6-FALLBACK: ephone-5: Configured codec is not supported by phone , fallback to codec g711ulaw

001590: Jun 7 02:14:21 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP002497355DF0 Load= SCCP41.8-4-1S Last=TCP-timeout

001591: Jun 7 02:14:21 WIB: %IPPHONE-6-REGISTER_NEW: ephone-7:SEP002497355DF0 IP:10.21.110.52 Socket:1 DeviceType:Phone has registered.

001592: Jun 7 02:14:21 WIB: %VOICE_CODEC-6-FALLBACK: ephone-7: Configured codec is not supported by phone , fallback to codec g711ulaw

001593: Jun 7 02:14:23 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP002497361D10 Load= SCCP41.8-4-1S Last=TCP-timeout

001594: Jun 7 02:14:23 WIB: %IPPHONE-6-REGISTER_NEW: ephone-1:SEP002497361D10 IP:10.21.110.49 Socket:4 DeviceType:Phone has registered.

001595: Jun 7 02:14:23 WIB: %VOICE_CODEC-6-FALLBACK: ephone-1: Configured codec is not supported by phone , fallback to codec g711ulaw

001596: Jun 7 02:16:40 WIB: %IPPHONE-6-UNREGISTER_NORMAL: ephone-6:SEP002497355F08 IP:10.21.110.47 Socket:2 DeviceType:Phone has unregistered normally.

001597: Jun 7 02:16:52 WIB: %IPPHONE-6-UNREGISTER_NORMAL: ephone-7:SEP002497355DF0 IP:10.21.110.52 Socket:1 DeviceType:Phone has unregistered normally.

001598: Jun 7 02:17:04 WIB: %IPPHONE-6-UNREGISTER_NORMAL: ephone-1:SEP002497361D10 IP:10.21.110.49 Socket:4 DeviceType:Phone has unregistered normally.

001599: Jun 7 02:17:05 WIB: %IPPHONE-6-UNREGISTER_NORMAL: ephone-5:SEP00249735658C IP:10.21.110.51 Socket:3 DeviceType:Phone has unregistered normally.

001600: Jun 7 02:48:18 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP002497361D10 Load= SCCP41.8-4-1S Last=TCP-timeout

001601: Jun 7 02:48:18 WIB: %IPPHONE-6-REGISTER_NEW: ephone-1:SEP002497361D10 IP:10.21.110.49 Socket:1 DeviceType:Phone has registered.

001602: Jun 7 02:48:18 WIB: %VOICE_CODEC-6-FALLBACK: ephone-1: Configured codec is not supported by phone , fallback to codec g711ulaw

001603: Jun 7 02:48:20 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP00249735658C Load= SCCP41.8-4-1S Last=TCP-timeout

001604: Jun 7 02:50:59 WIB: %IPPHONE-6-UNREGISTER_NORMAL: ephone-1:SEP002497361D10 IP:10.21.110.49 Socket:1 DeviceType:Phone has unregistered normally.

001605: Jun 7 06:45:13 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP002497361D10 Load= SCCP41.8-4-1S Last=TCP-timeout

001606: Jun 7 06:45:13 WIB: %IPPHONE-6-REGISTER_NEW: ephone-1:SEP002497361D10 IP:10.21.110.49 Socket:1 DeviceType:Phone has registered.

001607: Jun 7 06:45:13 WIB: %VOICE_CODEC-6-FALLBACK: ephone-1: Configured codec is not supported by phone , fallback to codec g711ulaw

001608: Jun 7 06:45:16 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP002497355F08 Load= SCCP41.8-4-1S Last=TCP-timeout

001609: Jun 7 06:45:16 WIB: %IPPHONE-6-REGISTER_NEW: ephone-6:SEP002497355F08 IP:10.21.110.47 Socket:3 DeviceType:Phone has registered.

001610: Jun 7 06:45:16 WIB: %VOICE_CODEC-6-FALLBACK: ephone-6: Configured codec is not supported by phone , fallback to codec g711ulaw

001611: Jun 7 06:47:54 WIB: %IPPHONE-6-UNREGISTER_NORMAL: ephone-1:SEP002497361D10 IP:10.21.110.49 Socket:1 DeviceType:Phone has unregistered normally.

001612: Jun 7 06:47:57 WIB: %IPPHONE-6-UNREGISTER_NORMAL: ephone-6:SEP002497355F08 IP:10.21.110.47 Socket:3 DeviceType:Phone has unregistered normally.

001613: Jun 7 09:00:46 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP002497361D10 Load= SCCP41.8-4-1S Last=TCP-timeout

001614: Jun 7 09:00:46 WIB: %IPPHONE-6-REGISTER_NEW: ephone-1:SEP002497361D10 IP:10.21.110.49 Socket:1 DeviceType:Phone has registered.

001615: Jun 7 09:00:46 WIB: %VOICE_CODEC-6-FALLBACK: ephone-1: Configured codec is not supported by phone , fallback to codec g711ulaw

001616: Jun 7 09:00:48 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP002497355DF0 Load= SCCP41.8-4-1S Last=TCP-timeout

001617: Jun 7 09:00:52 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP00249735A319 Load= SCCP41.8-4-1S Last=TCP-timeout

001618: Jun 7 09:03:17 WIB: %IPPHONE-6-UNREGISTER_NORMAL: ephone-1:SEP002497361D10 IP:10.21.110.49 Socket:1 DeviceType:Phone has unregistered normally.

001619: Jun 7 12:03:17 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP00249735A319 Load= SCCP41.8-4-1S Last=TCP-timeout

001620: Jun 7 12:03:17 WIB: %IPPHONE-6-REGISTER_NEW: ephone-8:SEP00249735A319 IP:10.21.110.50 Socket:1 DeviceType:Phone has registered.

001621: Jun 7 12:03:17 WIB: %VOICE_CODEC-6-FALLBACK: ephone-8: Configured codec is not supported by phone , fallback to codec g711ulaw

001622: Jun 7 12:03:18 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP002497355F08 Load= SCCP41.8-4-1S Last=TCP-timeout

001623: Jun 7 12:03:19 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP002497355DF0 Load= SCCP41.8-4-1S Last=TCP-timeout

001624: Jun 7 12:03:19 WIB: %IPPHONE-6-UNREGISTER_ABNORMAL: ephone-8:SEP00249735A319 IP:10.21.110.50 Socket:1 DeviceType:Phone has unregistered abnormally.

001625: Jun 7 12:03:19 WIB: %IPPHONE-6-REGISTER_NEW: ephone-7:SEP002497355DF0 IP:10.21.110.52 Socket:3 DeviceType:Phone has registered.

001626: Jun 7 12:03:19 WIB: %VOICE_CODEC-6-FALLBACK: ephone-7: Configured codec is not supported by phone , fallback to codec g711ulaw

001627: Jun 7 12:03:20 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP002497361D10 Load= SCCP41.8-4-1S Last=TCP-timeout

001628: Jun 7 12:06:00 WIB: %IPPHONE-6-UNREGISTER_NORMAL: ephone-7:SEP002497355DF0 IP:10.21.110.52 Socket:3 DeviceType:Phone has unregistered normally.

001629: Jun 7 12:47:07 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP00249735A319 Load= SCCP41.8-4-1S Last=TCP-timeout

001630: Jun 7 12:47:07 WIB: %IPPHONE-6-REGISTER_NEW: ephone-8:SEP00249735A319 IP:10.21.110.50 Socket:3 DeviceType:Phone has registered.

001631: Jun 7 12:47:07 WIB: %VOICE_CODEC-6-FALLBACK: ephone-8: Configured codec is not supported by phone , fallback to codec g711ulaw

001632: Jun 7 12:49:48 WIB: %IPPHONE-6-UNREGISTER_NORMAL: ephone-8:SEP00249735A319 IP:10.21.110.50 Socket:3 DeviceType:Phone has unregistered normally.

001633: Jun 7 15:03:48 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP00249735A319 Load= SCCP41.8-4-1S Last=TCP-timeout

001634: Jun 7 15:03:48 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP002497355F08 Load= SCCP41.8-4-1S Last=TCP-timeout

001635: Jun 7 15:17:04 WIB: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0, changed state to down

001636: Jun 7 15:17:05 WIB: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0, changed state to up

001637: Jun 7 15:17:42 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP00249735658C Load= SCCP41.8-4-1S Last=TCP-timeout

001638: Jun 7 16:19:07 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP002497355DF0 Load= SCCP41.8-4-1S Last=TCP-timeout

001639: Jun 7 16:19:09 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP002497361D10 Load= SCCP41.8-4-1S Last=TCP-timeout

001640: Jun 7 16:19:11 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP002497355F08 Load= SCCP41.8-4-1S Last=TCP-timeout

Does anyone experienced the same problem? Please advice..

Regards,

Teddy

William Bell · ‎06-07-2010

Teddy,

SRST is engaged when an IP phone fails over to the SRST reference (as configured in CUCM). Once one phone fails over than the IOS configs related to SRST are engaged and used by the gateway. Each phone's failover is an independant action. Since you have multiple phones failing over, I would say that you need to look into flaps on your network. Check for interface errors on any switch uplinks and WAN links. Look for state transition counts on WAN links. You have one site exhibiting this behavior. If the phone models, firmware versions, etc. are similar across multiple sites then it is prudent to start looking at PHY, L2, and L3 network issues first.

Don't use ping as your sole testing tool in this case. Remember that the phones maintain a TCP connection and if that connection is reset for some reason, the phones may engage SRST. Depends on the specifics of your environment.

Note that checking your gateway config isn't a bad idea in general, but gateways do not initiate SRST failover for phones. Phones initiate this failover on their own. Like H.323, SRST is an "every man for himself" protocol.

HTH.

Regards,
Bill

Please remember to rate helpful posts.

HTH -Bill (b) http://ucguerrilla.com (t) @ucguerrilla

Please remember to rate helpful responses and identify

teddywijaya84 · ‎06-07-2010

Hi Bill,

Thank you for your informations.

Actually the first thing that i have done for troubleshoot this problem is checked the PHY, L2, and L3 network issues. And i found nothing problems.

As example I checked the switch port that directly connect to the phone, there is no error on the interface.

CM-SW-Lt.1#sh int fa0/20
FastEthernet0/20 is up, line protocol is up (connected)
Hardware is Fast Ethernet, address is 0024.9883.2e94 (bia 0024.9883.2e94)
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
     reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, media type is 10/100BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:25, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 1000 bits/sec, 2 packets/sec
     569958 packets input, 92823199 bytes, 0 no buffer
     Received 52171 broadcasts (47430 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 47430 multicast, 0 pause input
     0 input packets with dribble condition detected
     2952238 packets output, 366813356 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

Are there any other suggestions? Please advice..

Regards,

Teddy

William Bell · ‎06-07-2010

I am assuming you also checked switch uplinks to the WAN router and the WAN router links to the WAN?

You could check RTT to see if there is a noticeable drop in latency from previous tests you may have performed. You did say that the problem just started, so there could have been a change in the WAN. Check the latency against other remote sites. You should also check your CUCM application even logs to see what type of errors you are getting when the devices unregister. There are reason codes in the events that you can lookup and that may give you some additional clues.

HTH.

Regards, Bill

HTH -Bill (b) http://ucguerrilla.com (t) @ucguerrilla

Please remember to rate helpful responses and identify

don.glidewell · ‎10-22-2010

Teddy and William,

Hope all is well. I'm having this same exact issue.

CM 6.1 at the hub

2800 Voice H323 gateways at the spoke

Random phones unregistering and registering within 5 mins of eachother. I feel it's a network issue but on the router, the interfaces never go down we only know this is happening with the unregistering and registiring of the phones. Our monitoring software alerts us as if the site is down. when we look at the site there is no interfaces down but just the unregestiring and re-registering of the phones.

It's happening at multiple sites all at the same time but different phones and appears to be random times.

Any help would be much appreciated.

dksingh · ‎10-22-2010

Multiple phones at a remote losing CCM registration and falling back on SRST router,

would indicate some sort of WAN issue.

What type of WAN connection u have between remote and CUCM site?

Is the remote SRST router also acting as the WAN router ? If not, checkout WAN router interfaces.

Even if the SRST router interfaces look good and its being used as the WAN router as well, do we

have proper QoS in place to give enough BW guarantee to the signaling traffic?

Is there any policy drops ?

Anything in between that could be affecting TCP traffic and/or SCCP keepalives?

Also CCM logs will provide info on why the phone unregistered in the first place. So that will

be the place to start looking.

don.glidewell · ‎10-22-2010

Thank you for the fast response. I've set traps on one of the spokes that has been giving me trouble. I've also been working with someone and so far this is what we've came up with....Please let me know if I'm on the right track.... I opened another thread as I didnt know if I would get anyone on this old post.

Here is the link to the new thread...and I'd appreciate your advice as well!

don.glidewell · ‎10-22-2010

Dilip,

I realize I didnt answer all your questions... It's late and I'm about frazzled.

1) I too believe it to be some sort of WAN issue. The unregister and reregister is all within less than 5 mins with random spokes but showing no int down.

2) 10mb metro e at the hub and Intergrated circuit (T1 broken out for both voice and data) at spokes.

3) Yes single routers are in place at the spokes so it's both WAN and SRST.

4) QOS is in place for the spokes for voice.

5) I dont see any policy drops or anything in between that woudl cause the issue.

6) I've set traps to capture logs for a spoke that normally gives us problems.

Also, I've worked with the carrier to check the circuits at not only the hub but most of the spokes. According to them, they see no issues. One thing that they told me that has caused me some worry is that the 10mb metro e is delivered in bundled T1's. They did say that they saw errors on 2 of the T1's. This would make sense if it was the problem but We've had issues since they supposedly resolved the T1 issues... I'm requesting that they bring the 10mb connection straight in with fiber. I think this can only help the situation.

As far as bandwidth, the spokes are getting now where near the limit for the T1. 200k is the max i've seen on any of the circuits. I'll post the sys logs when I catch something in the traps.

Thanks again for all the help.

dksingh · ‎10-22-2010

no worries...I understand...

So is it a multilink bundle from provider to the hub?

For the WAN link between the hub and the remote,

is it a single t1 ?

Can u post one of the problem remote's config for a sanity

check? Feel free to remove any sensitive info (IP/pwd etc.)

Pl. include sh policy-map inteface as well from both end

of the link.

Also, how may CCM server's are in the phone's CMGroup

excluding the SRST router?