Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements

Welcome to Cisco Support Community. We would love to have your feedback.

For an introduction to the new site, click here. And see here for current known issues.

New Member

Phones registered to different subscriber cannot communicate

We have 4 CUCM in PUB-SUB scenario

1 -PUB and 3 SUBs with cucm 6.0

everything was working fine but from last few days we are facing an issue,

Phones registered with Subscriber A can communiate with phones on Subscriber B and C both

Phones registered to Subscriber B can commnicate with subscriber A but not Subscriber C

Phones registered to sub Subscriber C cannot communicate with subscriber B but they can communicate with subscriber A

There are times when subscriber C phones can communicate with Subscriber B phones but its very rare.

I have verified DB replication status and its 2

I have tried to insert phones from subscriber C Admin page and it works.

There no latency or packet drops issues between Subscriber to Subscriber and Publisher to subscriber.

Any suggestion to solve the issue.

2 ACCEPTED SOLUTIONS

Accepted Solutions
Cisco Employee

Re: Phones registered to different subscriber cannot communicate

If replication is fine you might need to look at SDL layers to find out the issue.

Look for version mismatchs or errors in the app log

HTH

java

If this helps, please rate

www.cisco.com/go/pdihelpdesk

HTH

java

if this helps, please rate

www.cisco.com/go/pdi

Re: Phones registered to different subscriber cannot communicate

This confirms what Java was referring to earlier.  You have a DB replication issue within the cluster.  Basically, while the SDL link to that node is out of service - the replication for the entire cluster is considered bad.  With the Linux appliance, I've found that a reboot typically works but depending on how long the issue has been going on and the trigger for problem, it may not.  You'll need to take a look at the utils dbreplication commands.  There are some that you run on each server first, see what happens, and then if that doesn't work then you can initiate a repair operation from the publisher server.  This is all done via CLI.  Note that the repair operation may (and likely will) take quite a while to complete.  Once you start it, let it run and wait for it to complete.  Then use RTMT to verify DB replication.  You can also use command line operation on each server or the Unified Reporting.  Personally, I prefer RTMT.

If you need further info, just shout.

16 REPLIES
Cisco Employee

Re: Phones registered to different subscriber cannot communicate

Sounds like SDL issues, i would recommend a cluster reboot to resync the SDL

HTH

java

If this helps, please rate

www.cisco.com/go/pdihelpdesk

HTH

java

if this helps, please rate

www.cisco.com/go/pdi
New Member

Re: Phones registered to different subscriber cannot communicate

hi,

java thanks for your reply, i will let you know after cluster reboot.

Regards,

Iftikhar Ahmed

New Member

Re: Phones registered to different subscriber cannot communicate

Hi,

I have rebooted the cluster, publisher first then all subscribers.

Problem still persists.

Any further suggestions?

Cisco Employee

Re: Phones registered to different subscriber cannot communicate

If replication is fine you might need to look at SDL layers to find out the issue.

Look for version mismatchs or errors in the app log

HTH

java

If this helps, please rate

www.cisco.com/go/pdihelpdesk

HTH

java

if this helps, please rate

www.cisco.com/go/pdi
New Member

Re: Phones registered to different subscriber cannot communicate

hi,

how can i enable the logs? or you asking me to enable trace for that?

Regards,

Iftikhar

New Member

Re: Phones registered to different subscriber cannot communicate

Hi,

Since the problem started, I am getting this error message in application logs

Dec 10 23:04:19 CMSUB-ISL-IPT Error Cisco CallManager  : 26: Dec 10 18:04:19.6 UTC :  %CCM_CALLMANAGER-CALLMANAGER-3-SDLLinkOOS: SDL link to remote application out of service. Local node ID:12 Local Application ID.:100 Remote IP address of remote application:10.100.200.12 RemoteNodeID:1 Remote application ID.:100 Unique Link ID.:12:100:1:100 Cluster ID:StandAloneCluster Node ID:CMSUB-ISL-IPT

any suggestion?

Re: Phones registered to different subscriber cannot communicate

This confirms what Java was referring to earlier.  You have a DB replication issue within the cluster.  Basically, while the SDL link to that node is out of service - the replication for the entire cluster is considered bad.  With the Linux appliance, I've found that a reboot typically works but depending on how long the issue has been going on and the trigger for problem, it may not.  You'll need to take a look at the utils dbreplication commands.  There are some that you run on each server first, see what happens, and then if that doesn't work then you can initiate a repair operation from the publisher server.  This is all done via CLI.  Note that the repair operation may (and likely will) take quite a while to complete.  Once you start it, let it run and wait for it to complete.  Then use RTMT to verify DB replication.  You can also use command line operation on each server or the Unified Reporting.  Personally, I prefer RTMT.

If you need further info, just shout.

New Member

Re: Phones registered to different subscriber cannot communicate

Hi,

I have started repair process for all node, I am still wondering why its dbreplication issue if RTMT is sying that dbreplication status is 2 for all servers?

I will let you know once db replication is done.

Thanks for replying to my queries.

Regards,

Iftikhar AHmed

Cisco Employee

Re: Phones registered to different subscriber cannot communicate

EXACTLY what was the procedure you followed for the cluster reboot???

There is a method to perform a cluster reboot

HTH

java

if this helps, please rate

www.cisco.com/go/pdihelpdesk

HTH

java

if this helps, please rate

www.cisco.com/go/pdi
New Member

Re: Phones registered to different subscriber cannot communicate

Hi,

It seems to be working so far after dbreplication repair on all nodes,

As far as cluster reboot is concerned, I rebooted the publisher first and when it was online again i rebooted the subscriber one by one.

What is the correct way of reboting cluster by the way?

Thanks to both of you for support.

Regards,

Iftikhar

New Member

Re: Phones registered to different subscriber cannot communicate

Hi,

I am observing same problem again,

Again i am observing same error in RTMT application log

Dec 11 12:44:27CMSUB-ISL-IPTErrorCisco CallManager: 62: Dec 11 07:44:27.901 UTC : %CCM_CALLMANAGER-CALLMANAGER-3-SDLLinkOOS: SDL link to remote application out of service. Local node ID:12 Local Application ID.:100 Remote IP address of remote application:10.100.200.11 RemoteNodeID:3 Remote application ID.:100 Unique Link ID.:12:100:3:100 Cluster ID:StandAloneCluster Node ID:CMSUB-ISL-IPT

after db replication repair it was solved but now after few hours I am facing same issue again.

Any comments.

Regards,

Iftikhar Ahmed

New Member

Re: Phones registered to different subscriber cannot communicate

One more thing everytime when problem starts i see following in application log just before SDL error

Dec 11 12:01:01

CMSUB-ISL-IPTNoticelogrotate

ALERT exited abnormally with [1]

Is it something that is creating issue?

Regards,

Iftikhar Ahmed

Re: Phones registered to different subscriber cannot communicate

On the surface, I would say that the logrotate alert is likely not related to an SDL issue.  However, if I were you - I would run a battery of tests to verify that you dont have an issue at the network - either logical or physical.  For some reason, the Pub either can't or thinks it can't communicate at the SDL layer to this particular subscriber.  So, a lot of what you would need to test would depend on topology and etc.  You may also have a bad NIC or failed teaming configuration - there are lots of possibilities.  Java and I have touched on the primary fixes - 1 being reboot and 2 being repair replication if all else fails.  Some things to look for is a bad NIC, packet loss over the network, QoS, etc.  If you can rule those things out, then like Java said earlier you may need to start digging into the SDL layers.  In your case, I don't know your experience level but I would recommend that you open a TAC case with Cisco.  Other factors could be if this is a subscriber that was recently added to the cluster or if there was a recent upgrade on the nodes that may be causing an issue.

Step 1) TAC case

Step 2) Examine potential network-related issues

Step 3) Goes with #2, try to rule out any hardware issues such as bad NIC

That's my best at the moment.

New Member

Re: Phones registered to different subscriber cannot communicate

I would recommend you to run thea test on a server to check all SDL link or connectivity issue

Command :-  Utils dialgnose test

Check if the test fails at certain point send me the output of the test .

New Member

Re: Phones registered to different subscriber cannot communicate

Hi,

Thanks to all of you for replying,

As i posted earlier, there are no issues with WAN link, link between these two sites is STM1 link with no packet loss and latency around 12ms.

MTU for the network path is 1500.

If it is a Bad NIC issue then problem should occur between all three sites.

Vivek,

Utils diagnose test is not a valid command in CUCM 6.

Unfortunately we don't have essential software support at this moment to open TAC case.

Kindly let me know which DSL trace can be used to check issue with SDL linkOSS. as this is the only error displayed on RTMT application Log

Regards,

Iftikhar Ahmed

New Member

Re: Phones registered to different subscriber cannot communicate

Hi,

I have checked realtime SDL traces and i see following

Remote NodeId: 1, AppId: 300, Address: [10.100.200.12:8004]
000549312| 2009/12/12 15:51:05.838| 012| SdlError  |                                       |                               |                                 |                                 |                                         | - Unable to connect to remote node.> Remote NodeId: 1, Remote AppId: 300, Address: [10.100.200.12:8004]
000549313| 2009/12/12 15:51:08.567| 012| LnkState  |                                       |                               |                                 |                                 |                                         | Remote NodeId: 3, AppId: 300, Address: [10.100.200.11:8004]
000549314| 2009/12/12 15:51:08.577| 012| SdlError  |                                       |                               |                                 |                                 |                                         | - Unable to connect to remote node.> Remote NodeId: 3, Remote AppId: 300, Address: [10.100.200.11:8004]
000549315| 2009/12/12 15:51:08.677| 012| LnkState  |                                       |                               |                                 |                                 |                                         | Remote NodeId: 2, AppId: 300, Address: [10.120.200.11:8004]
000549316| 2009/12/12 15:51:08.704| 012| SdlError  |                                       |                               |                                 |                                 |                                         | - Unable to connect to remote node.> Remote NodeId: 2, Remote AppId: 300, Address: [10.120.200.11:8004]
000549317| 2009/12/12 15:51:10.477| 012| LnkState  |                                       |                               |                                 |                                 |                                         | Reason: Send Error, NodeId: 3, AppId: 100, TCPAddr: 10.100.200.11:8002
000549318| 2009/12/12 15:51:10.477| 012| LnkState  |                                       |                               |                                 |                                 |                                         | NodeId: 3 AppId: 100 [10.100.200.11:8002]
000549319| 2009/12/12 15:51:10.477| 012| LnkState  |                                       |                               |                                 |                                 |                                         | NodeId: 3, AppId: 100, TCPAddr: 10.100.200.11:8002
000549320| 2009/12/12 15:51:10.478| 012| LnkState  |                                       |                               |                                 |                                 |                                         | <:SENDTHREAD -="" waiting="" for="" closecomplete=""> NodeId: 3, AppId: 100, TCPAddr: 10.100.200.11:8002
000549321| 2009/12/12 15:51:10.478| 012| LnkState  |                                       |                               |                                 |                                 |                                         | <:RECEIVETHREAD -="" going="" to="" closecomplete=""> NodeId: 3, AppId: 100, TCPAddr: 10.100.200.11:8002
000549322| 2009/12/12 15:51:10.478| 012| LnkState  |                                       |                               |                                 |                                 |                                         | <:SENDTHREAD -="" going="" to="" connecting=""> NodeId: 3, AppId: 100, TCPAddr: 10.100.200.11:8002
000549323| 2009/12/12 15:51:10.490| 012| AlarmErr  |                                       |                               |                                 |                                 |                                         | AlarmClass: CallManager, AlarmName: SDLLinkOOS, AlarmSeverity: Error AlarmMessage: , AlarmDescription: SDL link to remote application out of service., AlarmParameters:  LocalNodeId:12, LocalApplicationID:100, RemoteIPAddress:10.100.200.11, RemoteNodeID:3, RemoteApplicationID:100, LinkID:12:100:3:100, AppID:Cisco CallManager, ClusterID:StandAloneCluster, NodeID:CMSUB-ISL-IPT,
000549324| 2009/12/12 15:51:15.287| 012| LnkState  |                                       |                               |                                 |                                 |                                         | Remote NodeId: 1, AppId: 100, Address: [10.100.200.12:8002]
000549325| 2009/12/12 15:51:15.298| 012| LnkState  |                                       |                               |                                 |                                 |                                         | - Connection Accepted Waiting to go InService - Remote NodeId: 1, Remote AppId: 100 Hostname: [10.100.200.12]
000549326| 2009/12/12 15:51:15.568| 012| SdlSig    | ReapOldTokenRegistrationsTimer        | wait                          | SIPStationInit(12,100,77,1)     | SdlTimerService(12,100,3,1)     | (12,100,89,2).1-(*:*)                   | [R:HP - HP: 0, NP: 0, LP: 0, VLP: 0, LZP: 0 DBP: 0]
000549327| 2009/12/12 15:51:15.837| 012| LnkState  |                                       |                               |                                 |                                 |                                         | Remote NodeId: 1, AppId: 300, Address: [10.100.200.12:8004]
000549328| 2009/12/12 15:51:15.847| 012| SdlError  |                                       |                               |                                 |                                 |                                         | - Unable to connect to remote node.> Remote NodeId: 1, Remote AppId: 300, Address: [10.100.200.12:8004]
000549329| 2009/12/12 15:51:16.307| 012| LnkState  |                                       |                               |                                 |                                 |                                         | <:RECEIVEDTHREAD -="" sending="" linkconnectionattempt="" message=""> NodeId: 1, AppId 100, TCPAddr: 10.100.200.12:8002
000549330| 2009/12/12 15:51:18.568| 012| LnkState  |                                       |                               |                                 |                                 |                                         | Remote NodeId: 3, AppId: 300, Address: [10.100.200.11:8004]
000549331| 2009/12/12 15:51:18.580| 012| SdlError  |                                       |                               |                                 |                                 |                                         | - Unable to connect to remote node.> Remote NodeId: 3, Remote AppId: 300, Address: [10.100.200.11:8004]
000549332| 2009/12/12 15:51:18.698| 012| LnkState  |                                       |                               |                                 |                                 |                                         | Remote NodeId: 2, AppId: 300, Address: [10.120.200.11:8004]
000549333| 2009/12/12 15:51:18.723| 012| SdlError  |                                       |                               |                                 |                                 |                                         | - Unable to connect to remote node.> Remote NodeId: 2, Remote AppId: 300, Address: [10.120.200.11:8004]
000549334| 2009/12/12 15:51:20.488| 012| LnkState  |                                       |                               |                                 |                                 |                                         | Remote NodeId: 3, AppId: 100, Address: [10.100.200.11:8002]
000549335| 2009/12/12 15:51:20.497| 012| LnkState  |                                       |                               |                                 |                                 |                                         | - Connection Accepted Waiting to go InService - Remote NodeId: 3, Remote AppId: 100 Hostname: [10.100.200.11]
000549336| 2009/12/12 15:51:21.498| 012| LnkState  |                                       |                               |                                 |                                 |                                         | <:RECEIVEDTHREAD -="" sending="" linkconnectionattempt="" message=""> NodeId: 3, AppId 100, TCPAddr: 10.100.200.11:8002

connections is established on port 8002 and 8003 but on port 8004 servers are unable to communucate. There is no port blockage at network level over the WAN.

Any suggestions now?

Regards,

Iftikhar

1433
Views
4
Helpful
16
Replies