TC6.2.1 issues after losing network connectivity

[Nick Halbert-Lillyman]

We've had a bit of a reccuring issue recently where, intermittently, if an endpoint running TC6.2.1 loses then restores network connectivity, it can't be called.  It re-registers to VCS and can dial out OK, but incoming calls fail.  In all cases, a reboot fixes the issue.

We've observed this on multiple EX90s, C-series and MX300s.  It doesn't seem to be a timeout issue - in many cases, the endpoint has been back on the network for an hour or more but still won't accept incoming calls until it's had a reboot, but in all cases can dial out fine.  We've also never seen this issue in TC5.X.

VCS search history shows "No route to destination" for the calls that don't work.

Endpoints are running TC6.2.1, registered with H323 to VCS-C 7.2 cluster.

[Nick Halbert-Lillyman]

I should also add, that when I check the registration on VCS after restoring network connectivity, it still shows the old registration rather than a new registration.

Diagnostics show no errors (other than "auto negotiation" as we need to have duplex hard set).  TMS Systems > Navigator shows the endpoint as registered also.

[Martin Koch]

Hi Nick.

I think I red about this behavior here before.

I did not find anything in the cisco bug tool. If you really can rule out

some firewall or router timeout and you have a service contract why dont

you escalate this to TAC and let us know what the result was!

Please remember to rate helpful responses and identify helpful or correct answers.

[Nick Halbert-Lillyman]

Hi Martin,

Yes I can definitely rule firewall etc out - in the testing I've been conducting, the endpoints and VCS are just one router hop away with nothing of note in the network configuration that might be interfering.

I guess it's TAC time.

[Nick Halbert-Lillyman]

So I've logged this with the TAC, the issue is still not solved but we've done quite alot of testing. Here's what we've found:

I've tested every version between TC6.1.1 and TC6.3.0 - it seems the issue affects TC6.2.0 and above. TC6.1.2 and below are unaffected - I couldn't find anything in the TC6.2.0 release notes that would indicate what might be causing the issue between the versions?

• The issue only occurs while using H323 - if I register as SIP, it's not a problem.
  
• I tried registering to different VCS which was running X7.2.2 but the problem still occured.
  
• It doesn't matter what the "calling" endpoint is - SIP phones, H323 video endpoints or TP server, whether from the local VCS or a neighbouring VCS/CUCM, all fail to call the affected endpoint.
  
• I tried manually clearing the registration on VCS after replacing the network cable and waiting for the endpoint to re-register. It re-registers OK but still can't recieve a call.
  
• Tried factory resseting one of the affected EX90s and using minimal configuration for H323 calls (gatekeeper, E164 and H323 ID). The problem still occured when these were the only non-default settings configured.
  
• Lastly, I was incorrect when I said the issue was intermittent - it's 100% reproducable and is fixed 100% of the time by a reboot, at least in my environment. The TAC has thus far been unable to reproduce it.
  

[Martin Koch]

Hi Nick,

thx for the feedback (+5)

Did you do some trace? I still can picture that there is some timer in your network / router which might

be the reason, ...

Do you have any chance to put a system on the same logical network as your vcs? Just to see if it would

happen there as well?

I recall that some behavoir on h323 regards timeouts had changed, there was a forum post regards this

but I neither recall the tc version when it changed nor did I find the posting when I was just trying to find it.

Please remember to rate helpful responses and identify helpful or correct answers.

[Nick Halbert-Lillyman]

Hi Martin,

I've had both endpoints and the VCS all in the same subnet, literally nothing but a switch in between, issue still occurs.  No port security, 8021.x or anything else strange.

Remembering that while the endpoint can't recieve incoming calls, it re-registers to VCS, can make outgoing calls and I can ping/browse to the GUI once the network connection is plugged back in, which indicates to me that the network is working fine.  Given that I manually cleared the old registration on VCS and allowed the endpoint to re-register and it's still occuring, I don't think it's a registration timeout issue (unless the issue is localized to the endpoint).

[Wayne DeNardi]

Hi Nick,

I had a similar issue with one of my C60s earlier this week.  The C60 could call out to anywhere (SIP and H.323 registered), but no H.323 calls inbound would connect (I didn't try a SIP call to it).

As per your troubleshooting, a reboot of the codec fixed it and I haven't see it re-occur.

Have you had a chance to try TC6.3 to see if there's a fix in there somewhere?

Wayne

--

Please remember to rate responses and to mark your question as answered if appropriate.

[Martin Koch]

Wayne: Nick posted that he tried TC6.3 (at least I assume in between also includes TC6.3  :-)

Still, doing a network trace could be interesting as well as to enable h323 debug logging.

Besides that I guess the best is to wait what TAC has to say.

Please remember to rate helpful responses and identify helpful or correct answers.

[Nick Halbert-Lillyman]

Just to clarify, yes I've tried TC6.3.

You're probably right that I will need to do a trace...

Wayne: I wonder if you pull the network cable of the C60 for a minute (until it says "No Lan") then replace it and see if the issue re-occurs.

What version are you running on the C60s? As per my previous post, it seems TC6.2.0 and above are affected.

[Wayne DeNardi]

Sorry, missed that post about you trying TC6.3.  Will be interesting to see what you can find with your traces and if you can see something missing.  I'll try to have a better look at this end if I see it occur on the C60 that did it before again.

Wayne

--

Please remember to rate responses and to mark your question as answered if appropriate.

[Patrick Pettit]

Is it possible you do this and run a packet capture AFTER you reconnect the network back in? In the trace, do you see 1720 being RST when VCS tries to open? Just curious. 

VR

Patrick

[Nick Halbert-Lillyman]

Hi Patrick,

To update, I've logged this with the TAC (SR 627952653).  I ran a packet capture on both endpoints and the VCS.  The packed capture for the affected endpoint contained data from both before and after the network was plugged back in.

It's also probably worth mentioning that I've upgraded to TC7.0.1 but no change - it still seems all versions after 6.1.1 are affected.

[Nick Halbert-Lillyman]

Further update:

The TAC said I am probably hitting this bug:

CSCum04312:  C40 cant receive H323 call.Port 1720 doesnt open after upgrade to TC 6.2

I don't have permissions to view the bug so I'm not sure if/when the target fix version is.  It may be worth changing to include other TC endpoints (we are seeing it on C60, C20, EX90 and MX300 G1.  Haven't tried SX20 or EX60).

[Patrick Pettit]

Looking at you EX90 pcap file that you've uploaded, frames 30 and 31.  Seem to fit the bill. 

The bug listed seems to fit, but will have to confirm.  Will look a bit more later tomorrow.  I think its easily reproducible by doing what your describing, so will take that and see where it goes. 

VR

Patrick