Telco nightly "REX" tests hang BRI ports on 3700

d.anthony.hadley_2 · ‎05-08-2007

Several nights out of the week telco does a "REX" [?] test. When they do this, it hangs the BRI channels and they go in to a Deactivated state. I have to reboot the router to get them out of that state, bouncing the BRI interface doesn't fix it. Telco said that it was a bug in the router. Of course, they couldn't elaborate on that. Neverthelesss, I still need to know if there is something in the code or something with telco that I can check. I'm at the typical telco stalemate. There are 27 sites with the same hardware and software configuration and only this one site has the problem. Any ideas? Below is the hardware/software and some debugs that were captured. For now, I had telco turn the auto tests off so I don't have to reboot the router twice a week but I need to go back and re-visit this issue.

Voice NM-HD-2V

VIC2-2BRI-NT/TE

c3745-ipvoice-mz.123-11.T8.bin

isdn switch-type basic-5ess

Jan 29 02:20:49 172.200.95.1 283077: Jan 29 02:09:04.269: ISDN BR2/2 Q921: User TX -> RRp sapi=0 tei=89 nr=53

Jan 29 02:21:15 172.200.95.1 283091: Jan 29 02:09:34.330: ISDN BR2/2 Q921: User TX -> RRp sapi=0 tei=89 nr=53

Jan 29 02:21:23 172.200.95.1 283092: Jan 29 02:09:34.342: ISDN BR2/2 Q921: User RX <- RRf sapi=0 tei=89 nr=74

Jan 29 02:21:40 172.200.95.1 283099: Jan 29 02:09:46.039: ISDN BR2/2 Q921: User RX <- UI sapi=0 tei=127

Jan 29 02:27:17 172.200.95.1 283205: Jan 29 02:14:47.105: ISDN BR2/2 Q921: User RX <- UI sapi=0 tei=127

Jan 29 02:36:05 172.200.95.1 283230: Jan 29 02:34:40.064: ISDN BR2/2 Q921: User RX <- UI sapi=0 tei=127

Jan 29 02:36:13 172.200.95.1 283231: Jan 29 02:34:40.064: ISDN BR2/2 Q931: MANAGEMENT_INFO pd = 8 callref = 0x00

Jan 29 02:36:47 172.200.95.1 283235: Jan 29 02:34:41.170: ISDN BR2/2 Q921: User RX <- UI sapi=0 tei=127

Jan 29 02:36:55 172.200.95.1 283236: Jan 29 02:34:41.170: ISDN BR2/2 Q931: MANAGEMENT_INFO pd = 8 callref = 0x00

Jan 29 02:55:50 172.200.95.1 283260: .Jan 29 02:54:30.392: ISDN BR2/2 Q921: User RX <- UI sapi=0 tei=127

Jan 29 02:55:58 172.200.95.1 283261: .Jan 29 02:54:30.392: ISDN BR2/2 Q931: MANAGEMENT_INFO pd = 8 callref = 0x00

Jan 29 02:56:32 172.200.95.1 283265: .Jan 29 02:54:31.490: ISDN BR2/2 Q921: User RX <- UI sapi=0 tei=127

Jan 29 02:56:41 172.200.95.1 283266: .Jan 29 02:54:31.490: ISDN BR2/2 Q931: MANAGEMENT_INFO pd = 8 callref = 0x00

Feb 8 09:29:34.579: ISDN BR2/2 Q921: User TX -> RR sapi=0 tei=81 nr=3

Feb 8 09:29:34.579: ISDN BR2/2 **ERROR**: host_information: Unexpected INFO - call id 0x0

Feb 8 09:29:34.603: ISDN BR2/2 Q921: User RX <- INFO sapi=0 tei=81, ns=3 nr=3

Feb 8 09:29:34.603: ISDN BR2/2 Q931: INFORMATION pd = 8 callref = N/A

Locking Shift to Codeset 6

Codeset 6 IE 0x22 i = 0x01

Codeset 6 IE 0x32 i = 0x01

Codeset 6 IE 0x3B i = 0x10

Feb 8 09:29:34.603: ISDN BR2/2 Q921: User TX -> RR sapi=0 tei=81 nr=4

Feb 8 09:29:34.603: ISDN BR2/2 **ERROR**: host_information: Unexpected INFO with no channel id - call id 0x0

Feb 8 09:29:35.040: %LINEPROTO-5-UPDOWN: Line protocol on Interface BRI2/2:1, changed state to down

Feb 8 09:29:35.040: %LINEPROTO-5-UPDOWN: Line protocol on Interface BRI2/2:2, changed state to down

Danilo Dy · ‎05-09-2007

Hi,

Nothing much in the output that I can see. It seems that layer1 detects link down then link up, but layer2 and layer3 may have remain in active state during the transition from up-down-up. If it is, check the link below for CSCsg50202

http://www.cisco.com/univercd/cc/td/doc/product/software/ios123/123relnt/xprn123/123mcavs.htm

Dandy

d.anthony.hadley_2 · ‎05-10-2007

Thanks for the reply. The ports get hung in such a way that only rebooting the router will clear them. I have "bounced" the interfaces, taken out the configs and put them back in but it doesn't clear them. I suspect that it's telco I need to find something else that is happening to the circuit. Any other known bugs in the IOS regarding this issue? Thanks.

Danilo Dy · ‎05-11-2007

Hi,

Nope.

If you have SmartNet or back-to-back maintenance with vendor, I advice you to update it and come back if you still experiencing problem.

Always take advantage of the update (if there's a bug/vulnerability in your current IOS) as you pay for it yearly (if you did), it's unlike MS patch that available for free as long as you bought a license product.

Dandy

paolo bevilacqua · ‎05-13-2007

Hi,

actually blaming a bug does not seem out of place in this case, can you try latest 12.4 on this router, and persisting the problem, seek advice by the tac. If you have captured the "rex" packet that make the port hung, that is a good start.

paolo bevilacqua · ‎05-13-2007

Additionally, I've found some more info about the REX (Routine EXercise) test. Possibly, you are connected to a 5ess switch with loop extended line, if so telco would have used programming codes ASDFI266 (Prohibit REX on BRI lines per office) or ASDFI267 (Prohibit REX on BRI lines per line).

Still, it not admissible that a received messages sends the port out of service, so I would suggest that after upgrading IOS, you consult the TAC on this one.

d.anthony.hadley_2 · ‎05-14-2007

Thanks. That's really good information. It is a 5ess switch. How would I be able to capture the ASDFI266 and ASDFI267 packets from telco? Is there a debug for this? Upgrading the IOS is not so simple as this is a 3745 CME/CUE. Upgrading requires a new CME version and will not be standardized with the 26 other sites and I'll need a specific justification for it.

paolo bevilacqua · ‎05-14-2007

Hi, the codes I gave are supposed to be 5ess commands to disable rex, thing that they did already for you.

If you can confirm that interface became unusable immediately after the packed above, you have captured the packet already.

I would not worry about bumping the CCME version just for a test. Newer IOS as still many bugfixes for ISDN. On the ISR, you can keep two images on flash and go back as required, however there are no compatibility issue.

As telco disabled the test now and probably will not press to reinstate it, you have resolved but if the bug exist and is not know, that would benefit all the cisco users.