06-07-2010 03:11 AM - edited 03-15-2019 11:07 PM
Hi All,
I have an UCM version 7.0 and several sites with voice gateways 2801 (SRST gateway) that registered to UCM using H.323 protocol.
The problem is the phones can register/unregister to the SRST gateway more than 3 times although the WAN link is UP. And I'm sure the WAN link is quite reliable, because I have test ping with size 8000 repeat 1000 and the result is 100% success. This problem only happened at 1 site, all the others sites are working fine. Also this problem occurs in the last 2 months, before there is no problem.
The configuration on all the SRST gateways are same. Here is the SRST configuration :
call-manager-fallback
max-conferences 4 gain -6
transfer-system full-consult
ip source-address 10.21.110.1 port 2000
max-ephones 30
max-dn 60
Here is the logging from the SRST gateway
Log Buffer (10000 bytes):
RMAL: ephone-7:SEP002497355DF0 IP:10.21.110.52 Socket:1 DeviceType:Phone has unregistered normally.
001565: Jun 7 00:00:13 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP002497355DF0 Load = SCCP41.8-4-1S Last=TCP-timeout
001566: Jun 7 00:00:13 WIB: %IPPHONE-6-REGISTER_NEW: ephone-7:SEP002497355DF0 IP:10.21.110.52 Socket:1 DeviceType:Phone has registered.
001567: Jun 7 00:00:13 WIB: %VOICE_CODEC-6-FALLBACK: ephone-7: Configured codec is not supported by phone , fallback to codec g711ulaw
001568: Jun 7 00:00:17 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP00249735A319 Load= SCCP41.8-4-1S Last=TCP-timeout
001569: Jun 7 00:00:21 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP002497355F08 Load= SCCP41.8-4-1S Last=TCP-timeout
001570: Jun 7 00:00:21 WIB: %IPPHONE-6-REGISTER_NEW: ephone-6:SEP002497355F08 IP:10.21.110.47 Socket:5 DeviceType:Phone has registered.
001571: Jun 7 00:00:21 WIB: %VOICE_CODEC-6-FALLBACK: ephone-6: Configured codec is not supported by phone , fallback to codec g711ulaw
001572: Jun 7 00:02:51 WIB: %IPPHONE-6-UNREGISTER_NORMAL: ephone-6:SEP002497355F08 IP:10.21.110.47 Socket:5 DeviceType:Phone has unregistered normally.
001573: Jun 7 00:03:04 WIB: %IPPHONE-6-UNREGISTER_NORMAL: ephone-7:SEP002497355DF0 IP:10.21.110.52 Socket:1 DeviceType:Phone has unregistered normally.
001574: Jun 7 01:18:27 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP00249735658C Load= SCCP41.8-4-1S Last=TCP-timeout
001575: Jun 7 01:18:27 WIB: %IPPHONE-6-REGISTER_NEW: ephone-5:SEP00249735658C IP:10.21.110.51 Socket:1 DeviceType:Phone has registered.
001576: Jun 7 01:18:27 WIB: %VOICE_CODEC-6-FALLBACK: ephone-5: Configured codec is not supported by phone , fallback to codec g711ulaw
001577: Jun 7 01:18:30 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP002497355DF0 Load= SCCP41.8-4-1S Last=TCP-timeout
001578: Jun 7 01:21:02 WIB: %IPPHONE-6-UNREGISTER_NORMAL: ephone-5:SEP00249735658C IP:10.21.110.51 Socket:1 DeviceType:Phone has unregistered normally.
001579: Jun 7 02:03:15 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP00249735A319 Load= SCCP41.8-4-1S Last=TCP-timeout
001580: Jun 7 02:03:15 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP002497355F08 Load= SCCP41.8-4-1S Last=TCP-timeout
001581: Jun 7 02:03:15 WIB: %IPPHONE-6-REGISTER_NEW: ephone-6:SEP002497355F08 IP:10.21.110.47 Socket:3 DeviceType:Phone has registered.
001582: Jun 7 02:03:15 WIB: %VOICE_CODEC-6-FALLBACK: ephone-6: Configured codec is not supported by phone , fallback to codec g711ulaw
001583: Jun 7 02:05:46 WIB: %IPPHONE-6-UNREGISTER_NORMAL: ephone-6:SEP002497355F08 IP:10.21.110.47 Socket:3 DeviceType:Phone has unregistered normally.
001584: Jun 7 02:14:19 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP002497355F08 Load= SCCP41.8-4-1S Last=TCP-timeout
001585: Jun 7 02:14:19 WIB: %IPPHONE-6-REGISTER_NEW: ephone-6:SEP002497355F08 IP:10.21.110.47 Socket:2 DeviceType:Phone has registered.
001586: Jun 7 02:14:19 WIB: %VOICE_CODEC-6-FALLBACK: ephone-6: Configured codec is not supported by phone , fallback to codec g711ulaw
001587: Jun 7 02:14:20 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP00249735658C Load= SCCP41.8-4-1S Last=TCP-timeout
001588: Jun 7 02:14:20 WIB: %IPPHONE-6-REGISTER_NEW: ephone-5:SEP00249735658C IP:10.21.110.51 Socket:3 DeviceType:Phone has registered.
001589: Jun 7 02:14:20 WIB: %VOICE_CODEC-6-FALLBACK: ephone-5: Configured codec is not supported by phone , fallback to codec g711ulaw
001590: Jun 7 02:14:21 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP002497355DF0 Load= SCCP41.8-4-1S Last=TCP-timeout
001591: Jun 7 02:14:21 WIB: %IPPHONE-6-REGISTER_NEW: ephone-7:SEP002497355DF0 IP:10.21.110.52 Socket:1 DeviceType:Phone has registered.
001592: Jun 7 02:14:21 WIB: %VOICE_CODEC-6-FALLBACK: ephone-7: Configured codec is not supported by phone , fallback to codec g711ulaw
001593: Jun 7 02:14:23 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP002497361D10 Load= SCCP41.8-4-1S Last=TCP-timeout
001594: Jun 7 02:14:23 WIB: %IPPHONE-6-REGISTER_NEW: ephone-1:SEP002497361D10 IP:10.21.110.49 Socket:4 DeviceType:Phone has registered.
001595: Jun 7 02:14:23 WIB: %VOICE_CODEC-6-FALLBACK: ephone-1: Configured codec is not supported by phone , fallback to codec g711ulaw
001596: Jun 7 02:16:40 WIB: %IPPHONE-6-UNREGISTER_NORMAL: ephone-6:SEP002497355F08 IP:10.21.110.47 Socket:2 DeviceType:Phone has unregistered normally.
001597: Jun 7 02:16:52 WIB: %IPPHONE-6-UNREGISTER_NORMAL: ephone-7:SEP002497355DF0 IP:10.21.110.52 Socket:1 DeviceType:Phone has unregistered normally.
001598: Jun 7 02:17:04 WIB: %IPPHONE-6-UNREGISTER_NORMAL: ephone-1:SEP002497361D10 IP:10.21.110.49 Socket:4 DeviceType:Phone has unregistered normally.
001599: Jun 7 02:17:05 WIB: %IPPHONE-6-UNREGISTER_NORMAL: ephone-5:SEP00249735658C IP:10.21.110.51 Socket:3 DeviceType:Phone has unregistered normally.
001600: Jun 7 02:48:18 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP002497361D10 Load= SCCP41.8-4-1S Last=TCP-timeout
001601: Jun 7 02:48:18 WIB: %IPPHONE-6-REGISTER_NEW: ephone-1:SEP002497361D10 IP:10.21.110.49 Socket:1 DeviceType:Phone has registered.
001602: Jun 7 02:48:18 WIB: %VOICE_CODEC-6-FALLBACK: ephone-1: Configured codec is not supported by phone , fallback to codec g711ulaw
001603: Jun 7 02:48:20 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP00249735658C Load= SCCP41.8-4-1S Last=TCP-timeout
001604: Jun 7 02:50:59 WIB: %IPPHONE-6-UNREGISTER_NORMAL: ephone-1:SEP002497361D10 IP:10.21.110.49 Socket:1 DeviceType:Phone has unregistered normally.
001605: Jun 7 06:45:13 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP002497361D10 Load= SCCP41.8-4-1S Last=TCP-timeout
001606: Jun 7 06:45:13 WIB: %IPPHONE-6-REGISTER_NEW: ephone-1:SEP002497361D10 IP:10.21.110.49 Socket:1 DeviceType:Phone has registered.
001607: Jun 7 06:45:13 WIB: %VOICE_CODEC-6-FALLBACK: ephone-1: Configured codec is not supported by phone , fallback to codec g711ulaw
001608: Jun 7 06:45:16 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP002497355F08 Load= SCCP41.8-4-1S Last=TCP-timeout
001609: Jun 7 06:45:16 WIB: %IPPHONE-6-REGISTER_NEW: ephone-6:SEP002497355F08 IP:10.21.110.47 Socket:3 DeviceType:Phone has registered.
001610: Jun 7 06:45:16 WIB: %VOICE_CODEC-6-FALLBACK: ephone-6: Configured codec is not supported by phone , fallback to codec g711ulaw
001611: Jun 7 06:47:54 WIB: %IPPHONE-6-UNREGISTER_NORMAL: ephone-1:SEP002497361D10 IP:10.21.110.49 Socket:1 DeviceType:Phone has unregistered normally.
001612: Jun 7 06:47:57 WIB: %IPPHONE-6-UNREGISTER_NORMAL: ephone-6:SEP002497355F08 IP:10.21.110.47 Socket:3 DeviceType:Phone has unregistered normally.
001613: Jun 7 09:00:46 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP002497361D10 Load= SCCP41.8-4-1S Last=TCP-timeout
001614: Jun 7 09:00:46 WIB: %IPPHONE-6-REGISTER_NEW: ephone-1:SEP002497361D10 IP:10.21.110.49 Socket:1 DeviceType:Phone has registered.
001615: Jun 7 09:00:46 WIB: %VOICE_CODEC-6-FALLBACK: ephone-1: Configured codec is not supported by phone , fallback to codec g711ulaw
001616: Jun 7 09:00:48 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP002497355DF0 Load= SCCP41.8-4-1S Last=TCP-timeout
001617: Jun 7 09:00:52 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP00249735A319 Load= SCCP41.8-4-1S Last=TCP-timeout
001618: Jun 7 09:03:17 WIB: %IPPHONE-6-UNREGISTER_NORMAL: ephone-1:SEP002497361D10 IP:10.21.110.49 Socket:1 DeviceType:Phone has unregistered normally.
001619: Jun 7 12:03:17 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP00249735A319 Load= SCCP41.8-4-1S Last=TCP-timeout
001620: Jun 7 12:03:17 WIB: %IPPHONE-6-REGISTER_NEW: ephone-8:SEP00249735A319 IP:10.21.110.50 Socket:1 DeviceType:Phone has registered.
001621: Jun 7 12:03:17 WIB: %VOICE_CODEC-6-FALLBACK: ephone-8: Configured codec is not supported by phone , fallback to codec g711ulaw
001622: Jun 7 12:03:18 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP002497355F08 Load= SCCP41.8-4-1S Last=TCP-timeout
001623: Jun 7 12:03:19 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP002497355DF0 Load= SCCP41.8-4-1S Last=TCP-timeout
001624: Jun 7 12:03:19 WIB: %IPPHONE-6-UNREGISTER_ABNORMAL: ephone-8:SEP00249735A319 IP:10.21.110.50 Socket:1 DeviceType:Phone has unregistered abnormally.
001625: Jun 7 12:03:19 WIB: %IPPHONE-6-REGISTER_NEW: ephone-7:SEP002497355DF0 IP:10.21.110.52 Socket:3 DeviceType:Phone has registered.
001626: Jun 7 12:03:19 WIB: %VOICE_CODEC-6-FALLBACK: ephone-7: Configured codec is not supported by phone , fallback to codec g711ulaw
001627: Jun 7 12:03:20 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP002497361D10 Load= SCCP41.8-4-1S Last=TCP-timeout
001628: Jun 7 12:06:00 WIB: %IPPHONE-6-UNREGISTER_NORMAL: ephone-7:SEP002497355DF0 IP:10.21.110.52 Socket:3 DeviceType:Phone has unregistered normally.
001629: Jun 7 12:47:07 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP00249735A319 Load= SCCP41.8-4-1S Last=TCP-timeout
001630: Jun 7 12:47:07 WIB: %IPPHONE-6-REGISTER_NEW: ephone-8:SEP00249735A319 IP:10.21.110.50 Socket:3 DeviceType:Phone has registered.
001631: Jun 7 12:47:07 WIB: %VOICE_CODEC-6-FALLBACK: ephone-8: Configured codec is not supported by phone , fallback to codec g711ulaw
001632: Jun 7 12:49:48 WIB: %IPPHONE-6-UNREGISTER_NORMAL: ephone-8:SEP00249735A319 IP:10.21.110.50 Socket:3 DeviceType:Phone has unregistered normally.
001633: Jun 7 15:03:48 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP00249735A319 Load= SCCP41.8-4-1S Last=TCP-timeout
001634: Jun 7 15:03:48 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP002497355F08 Load= SCCP41.8-4-1S Last=TCP-timeout
001635: Jun 7 15:17:04 WIB: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0, changed state to down
001636: Jun 7 15:17:05 WIB: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0, changed state to up
001637: Jun 7 15:17:42 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP00249735658C Load= SCCP41.8-4-1S Last=TCP-timeout
001638: Jun 7 16:19:07 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP002497355DF0 Load= SCCP41.8-4-1S Last=TCP-timeout
001639: Jun 7 16:19:09 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP002497361D10 Load= SCCP41.8-4-1S Last=TCP-timeout
001640: Jun 7 16:19:11 WIB: %IPPHONE-6-REG_ALARM: 10: Name=SEP002497355F08 Load= SCCP41.8-4-1S Last=TCP-timeout
Does anyone experienced the same problem? Please advice..
Regards,
Teddy
06-07-2010 05:04 AM
Teddy,
SRST is engaged when an IP phone fails over to the SRST reference (as configured in CUCM). Once one phone fails over than the IOS configs related to SRST are engaged and used by the gateway. Each phone's failover is an independant action. Since you have multiple phones failing over, I would say that you need to look into flaps on your network. Check for interface errors on any switch uplinks and WAN links. Look for state transition counts on WAN links. You have one site exhibiting this behavior. If the phone models, firmware versions, etc. are similar across multiple sites then it is prudent to start looking at PHY, L2, and L3 network issues first.
Don't use ping as your sole testing tool in this case. Remember that the phones maintain a TCP connection and if that connection is reset for some reason, the phones may engage SRST. Depends on the specifics of your environment.
Note that checking your gateway config isn't a bad idea in general, but gateways do not initiate SRST failover for phones. Phones initiate this failover on their own. Like H.323, SRST is an "every man for himself" protocol.
HTH.
Regards,
Bill
Please remember to rate helpful posts.
Please remember to rate helpful responses and identify
06-07-2010 09:22 PM
Hi Bill,
Thank you for your informations.
Actually the first thing that i have done for troubleshoot this problem is checked the PHY, L2, and L3 network issues. And i found nothing problems.
As example I checked the switch port that directly connect to the phone, there is no error on the interface.
CM-SW-Lt.1#sh int fa0/20
FastEthernet0/20 is up, line protocol is up (connected)
Hardware is Fast Ethernet, address is 0024.9883.2e94 (bia 0024.9883.2e94)
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, media type is 10/100BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:25, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 1000 bits/sec, 2 packets/sec
569958 packets input, 92823199 bytes, 0 no buffer
Received 52171 broadcasts (47430 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 47430 multicast, 0 pause input
0 input packets with dribble condition detected
2952238 packets output, 366813356 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out
Are there any other suggestions? Please advice..
Regards,
Teddy
06-07-2010 09:28 PM
I am assuming you also checked switch uplinks to the WAN router and the WAN router links to the WAN?
You could check RTT to see if there is a noticeable drop in latency from previous tests you may have performed. You did say that the problem just started, so there could have been a change in the WAN. Check the latency against other remote sites. You should also check your CUCM application even logs to see what type of errors you are getting when the devices unregister. There are reason codes in the events that you can lookup and that may give you some additional clues.
HTH.
Regards, Bill
Please remember to rate helpful responses and identify
10-22-2010 04:03 PM
Teddy and William,
Hope all is well. I'm having this same exact issue.
CM 6.1 at the hub
2800 Voice H323 gateways at the spoke
Random phones unregistering and registering within 5 mins of eachother. I feel it's a network issue but on the router, the interfaces never go down we only know this is happening with the unregistering and registiring of the phones. Our monitoring software alerts us as if the site is down. when we look at the site there is no interfaces down but just the unregestiring and re-registering of the phones.
It's happening at multiple sites all at the same time but different phones and appears to be random times.
Any help would be much appreciated.
10-22-2010 05:56 PM
Multiple phones at a remote losing CCM registration and falling back on SRST router,
would indicate some sort of WAN issue.
What type of WAN connection u have between remote and CUCM site?
Is the remote SRST router also acting as the WAN router ? If not, checkout WAN router interfaces.
Even if the SRST router interfaces look good and its being used as the WAN router as well, do we
have proper QoS in place to give enough BW guarantee to the signaling traffic?
Is there any policy drops ?
Anything in between that could be affecting TCP traffic and/or SCCP keepalives?
Also CCM logs will provide info on why the phone unregistered in the first place. So that will
be the place to start looking.
10-22-2010 06:13 PM
Thank you for the fast response. I've set traps on one of the spokes that has been giving me trouble. I've also been working with someone and so far this is what we've came up with....Please let me know if I'm on the right track.... I opened another thread as I didnt know if I would get anyone on this old post.
Here is the link to the new thread...and I'd appreciate your advice as well!
10-22-2010 06:55 PM
Dilip,
I realize I didnt answer all your questions... It's late and I'm about frazzled.
1) I too believe it to be some sort of WAN issue. The unregister and reregister is all within less than 5 mins with random spokes but showing no int down.
2) 10mb metro e at the hub and Intergrated circuit (T1 broken out for both voice and data) at spokes.
3) Yes single routers are in place at the spokes so it's both WAN and SRST.
4) QOS is in place for the spokes for voice.
5) I dont see any policy drops or anything in between that woudl cause the issue.
6) I've set traps to capture logs for a spoke that normally gives us problems.
Also, I've worked with the carrier to check the circuits at not only the hub but most of the spokes. According to them, they see no issues. One thing that they told me that has caused me some worry is that the 10mb metro e is delivered in bundled T1's. They did say that they saw errors on 2 of the T1's. This would make sense if it was the problem but We've had issues since they supposedly resolved the T1 issues... I'm requesting that they bring the 10mb connection straight in with fiber. I think this can only help the situation.
As far as bandwidth, the spokes are getting now where near the limit for the T1. 200k is the max i've seen on any of the circuits. I'll post the sys logs when I catch something in the traps.
Thanks again for all the help.
10-22-2010 07:32 PM
no worries...I understand...
So is it a multilink bundle from provider to the hub?
For the WAN link between the hub and the remote,
is it a single t1 ?
Can u post one of the problem remote's config for a sanity
check? Feel free to remove any sensitive info (IP/pwd etc.)
Pl. include sh policy-map inteface as well from both end
of the link.
Also, how may CCM server's are in the phone's CMGroup
excluding the SRST router?
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: