Outgoing and incoming calls failing periodically (T1/PRI)

[mitchell helton]

Hey folks!

We're continuing to have a problem which we think is on the provider's side (isn't it always?).  However, they keep pushing back on us and telling us they don't see anything wrong with their equipment.  They'll admit they see the B and D channels dropping and resyncing (I think that's the terminology they use but I'm not a Voice guy) but all of their interfaces are clean.  The problem lasts for 5-10 minutes and happens sporadically.  It may happen 3 times 1 day, and not happen again for a week.  It's starting to happen more frequently though.  Anyway, here are the facts:

• Outages last for 5-10 minutes
• Internal (on-net) calls work perfectly fine
• Outgoing/Incoming calls fail
• We use Solarwinds, syslog and RTMT to monitor our systems
  • None of these have indicated we have a problem internally (ie no alerts generated during outages)
• TAC deduced the problem was with our network or the provider since calls aren't making it to CUCM
• Our setup is:  Smartjack ->2901 -> 3750 -> 4510 -> Redundant BE6K servers
• We've replaced the Cat5 cable connecting our voice gateway to the smartjack

I've attached the output of various show and debug commands and hope they'll be helpful.  Please let me know if there is additional information you'd like.

Thanks very much!

mitch

[michael o'nan]

can you post show run | i clock

[mitchell helton]

Sure!

network-clock-participate wic 0
network-clock-select 1 T1 0/0/0

[Jayanth Velkuri]

looks like your later 2 was down

do you have deb isdn q921 ?

HQ-VGW1#  show isdn status
Global ISDN Switchtype = primary-ni
ISDN Serial0/0/0:23 interface
	dsl 0, interface ISDN Switchtype = primary-ni
    Layer 1 Status:
	ACTIVE
    Layer 2 Status:
	TEI = 0, Ces = 1, SAPI = 0, State = TEI_ASSIGNED
    Layer 3 Status:
	1 Active Layer 3 Call(s)
    Active dsl 0 CCBs = 0
    The Free Channel Mask:  0x807FFFFF
    Number of L2 Discards = 0, L2 Session ID = 123
    Total Allocated ISDN CCBs = 0

[mitchell helton]

I was not debugging q921 at that time.  I'll make sure to do that if we have the issue again.

Thanks!

[aseanand89]

Create a T1 loopback plug and then plug it into the T1 port. Run debug isdn q921 output & check if the SABME was sent

this message received should be:

    RX <- BAD FRAME(0x00017F)Line may be looped!

The BAD FRAME messages indicate that the router performs correctly. The router sends out the SABME packet. The message is looped back to the router, because of which, the router receives the same SABME message that was sent. The router marks it as a BAD FRAME, and presents the error message. The error message states that the line is probably looped. This is the expected behavior for the looped circuit. Therefore, the problem lies within the Telco ISDN switch or the cabling from the demarc to the Telco switch.

Refer to the link below:

http://www.cisco.com/c/en/us/support/docs/wan/t1-e1-t3-e3/8131-T1-pri.html

[mitchell helton]

That sounds really interesting - I'll look into that.

Thanks!

[mitchell helton]

This still isn't resolved, but at least the telco recognizes it's a problem on their end now that another customer is having the exam same issue as we are.

Thanks for the help everyone.