10-06-2010 08:22 PM - edited 03-04-2019 10:01 AM
Equipment Background Information:
sh ver:
Cisco Internetwork Operating System Software
IOS (tm) C2600 Software (C2600-JS-M), Version 12.3(26), RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2008 by cisco Systems, Inc.
Compiled Mon 17-Mar-08 15:24 by dchih
ROM: System Bootstrap, Version 12.2(6r), RELEASE SOFTWARE (fc1)
cisco 2650 (MPC860P) processor (revision 0x00) with 126976K/4096K bytes of memory.
Processor board ID JAD06200P0W (2846690197)
M860 processor: part number 5, mask 2
Bridging software.
X.25 software, Version 3.0.0.
SuperLAT software (copyright 1990 by Meridian Technology Corp).
TN3270 Emulation software.
1 FastEthernet/IEEE 802.3 interface(s)
2 Serial network interface(s)
32K bytes of non-volatile configuration memory.
32768K bytes of processor board System flash (Read/Write)
Abreviated Sh run:
class-map match-all QoS
match access-group 100
!
!
policy-map QoSWAN
class QoS
set dscp ef
class class-default
set dscp default
!
!
!
!
interface Multilink1
ip address #.#.#.# 255.255.255.252
load-interval 30
no cdp enable
ppp multilink
ppp multilink group 1
service-policy output QoSWAN
!
interface FastEthernet0/0
ip address 192.168.7.3 255.255.255.0
load-interval 30
duplex auto
speed auto
!
interface Serial0/0
no ip address
encapsulation ppp
load-interval 30
no fair-queue
service-module t1 timeslots 1-24
ppp multilink
ppp multilink group 1
!
interface Serial0/1
no ip address
encapsulation ppp
load-interval 30
no fair-queue
service-module t1 timeslots 1-24
ppp multilink
ppp multilink group 1
!
no ip http server
ip classless
ip route 0.0.0.0 0.0.0.0
!
!
access-list 100 permit ip host 192.168.7.21 any
access-list 100 permit ip any host 192.168.3.20
access-list 100 permit ip any host 192.168.4.20
access-list 100 permit ip any host 192.168.5.20
access-list 100 permit ip any host 192.168.6.20
access-list 100 permit ip any host 192.168.8.20
access-list 100 permit ip any host 192.168.9.20
no cdp run
Background information on problem:
Intermittently taking large number of service affecting errors. Circuit bounces frequently. (see counters below). ATT sees errors in PM logs but can't capture event during testing. Service is Frame Relay across ILEC Frame Cloud to ISP Talk to Circuit. Router is configured for QoS.
Questions:
All questions are related but separated here to make reading easier.
1. Sh controller output shows a Recieve Ring and Transmit Ring. The Receive Ring consists of 24 individual lines. The Transmit Ring consists of 16 individual lines. Does each line in the Receive and Transmit Ring represent 1 of 24 channels forming the T1 circuit?
2. Assuming each line represents a unique channel in the T1 circuit, Is the Transmit Ring only displaying 16 channels because the router is using hardware to reserve 8 channels for QoS configuration?
3. The Transmit Ring shows most lines having a status of status 5C00. At different times, different lines will have a status of status 0. Does status 0 represent a "down" channel?
4. What would cause different lines to randomly show status 0?
5. Does this indicate an issue with a multiplexer or other physical issue with the circuit and can this information be used to narrow down the problematic equipment?
Actual log entries from Sh controller output of router:
Receive Ring
rmd(680126B0): status 9000 length 5E8 address 7D6BDA4
rmd(680126B8): status 9000 length 5E8 address 7D6D7A4
rmd(680126C0): status 9000 length 5E8 address 7D6EB24
rmd(680126C8): status 9000 length 5E8 address 7D6A3A4
rmd(680126D0): status 9000 length 4BB address 7D66FA4
rmd(680126D8): status 9000 length 36 address 7D76024
rmd(680126E0): status 9000 length 45 address 7D78DA4
rmd(680126E8): status 9000 length 45 address 7D662A4
rmd(680126F0): status 9000 length 36 address 7D6F824
rmd(680126F8): status 9000 length 40 address 7D72C24
rmd(68012700): status 9000 length 36 address 7D696A4
rmd(68012708): status 9000 length 4A address 7D78724
rmd(68012710): status 9000 length 5E8 address 7D759A4
rmd(68012718): status 9000 length 5E8 address 7D766A4
rmd(68012720): status 9000 length 5E8 address 7D6AA24
rmd(68012728): status 9000 length 5E8 address 7D773A4
rmd(68012730): status 9000 length 5E8 address 7D6E4A4
rmd(68012738): status 9000 length 5E8 address 7D732A4
rmd(68012740): status 9000 length 5E8 address 7D725A4
rmd(68012748): status 9000 length 5E8 address 7D79424
rmd(68012750): status 9000 length 5E8 address 7D67624
rmd(68012758): status 9000 length 5E8 address 7D6FEA4
rmd(68012760): status 9000 length 5E8 address 7D73FA4
rmd(68012768): status B000 length 5E8 address 7D74CA4
Transmit Ring
tmd(68012770): status 5C00 length 43 address 7D1F0E4
tmd(68012778): status 0 length 8 address 7D01B70
tmd(68012780): status 5C00 length 3D address 7D47B28
tmd(68012788): status 5C00 length 43 address 7D28664
tmd(68012790): status 5C00 length 43 address 7D195E4
tmd(68012798): status 5C00 length 12A address 7D9064E
tmd(680127A0): status 5C00 length 12A address 7C02DAE
tmd(680127A8): status 5C00 length 64 address 7D8FF4A
tmd(680127B0): status 0 length 8 address 7D01B70
tmd(680127B8): status 5C00 length 31 address 7D1FE1A
tmd(680127C0): status 0 length 8 address 7D01B70
tmd(680127C8): status 5C00 length 3D address 7D245A8
tmd(680127D0): status 5C00 length 32 address 7D2A064
tmd(680127D8): status 5C00 length 43 address 7D3F8E4
tmd(680127E0): status 5C00 length 32 address 7D2A6E4
tmd(680127E8): status 2000 length 8 address 7D01B70
tx_limited=0(16)
SCC GENERAL PARAMETER RAM (at 0x68013C00)
Rx BD Base [RBASE]=0x26B0, Fn Code [RFCR]=0x18
Tx BD Base [TBASE]=0x2770, Fn Code [TFCR]=0x18
Max Rx Buff Len [MRBLR]=1548
Rx State [RSTATE]=0x18008440, BD Ptr [RBPTR]=0x2750
Tx State [TSTATE]=0x18800511, BD Ptr [TBPTR]=0x27E0
SCC HDLC PARAMETER RAM (at 0x68013C38)
CRC Preset [C_PRES]=0xFFFF, Mask [C_MASK]=0xF0B8
Errors: CRC [CRCEC]=736, Aborts [ABTSC]=6485, Discards [DISFC]=0
Nonmatch Addr Cntr [NMARC]=0
Retry Count [RETRC]=0
Max Frame Length [MFLR]=1608
Rx Int Threshold [RFTHR]=0, Frame Cnt [RFCNT]=38869
User-defined Address 0000/0000/0000/0000
User-defined Address Mask 0x0000
buffer size 1524
PowerQUICC SCC specific errors:
11374743 input aborts on receiving flag sequence
0 throttles, 0 enables
0 overruns
0 transmitter underruns
0 transmitter CTS losts
957977 aborted short frames
Example of Circuit Bounces
sh log: (Date/Time command issued 10/7/2010 3:50 AM EST)
Oct 4 20:18:03: %LINK-3-UPDOWN: Interface Serial0/0, changed state to down
Oct 4 20:18:04: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0, chan
ged state to down
Oct 4 20:21:25: %LINK-3-UPDOWN: Interface Serial0/0, changed state to up
Oct 4 20:21:40: %LINK-3-UPDOWN: Interface Serial0/0, changed state to down
Oct 4 21:26:48: %LINK-3-UPDOWN: Interface Serial0/0, changed state to up
Oct 4 21:27:40: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0, chan
ged state to up
Oct 4 21:29:21: %LINK-3-UPDOWN: Interface Serial0/0, changed state to down
Oct 4 21:29:22: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0, chan
ged state to down
Oct 4 21:34:39: %LINK-3-UPDOWN: Interface Serial0/0, changed state to up
Oct 4 21:34:53: %LINK-3-UPDOWN: Interface Multilink1, changed state to down
Oct 4 21:34:53: %LINK-3-UPDOWN: Interface Multilink1, changed state to up
Oct 4 21:40:42: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0, chan
ged state to up
Oct 5 00:16:54: %LINK-3-UPDOWN: Interface Serial0/1, changed state to down
Oct 5 00:16:55: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/1, chan
ged state to down
Oct 5 00:22:25: %LINK-3-UPDOWN: Interface Serial0/1, changed state to up
Oct 5 00:22:41: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/1, chan
ged state to up
Oct 5 00:23:33: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/1, chan
ged state to down
Oct 5 00:28:50: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/1, chan
ged state to up
Oct 5 00:36:41: %LINK-3-UPDOWN: Interface Serial0/1, changed state to down
Oct 5 00:36:42: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/1, chan
ged state to down
Oct 5 00:58:30: %LINK-3-UPDOWN: Interface Serial0/1, changed state to up
Oct 5 00:58:45: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/1, chan
ged state to up
Oct 5 00:59:47: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/1, chan
ged state to down
Oct 5 01:28:12: %LINK-3-UPDOWN: Interface Serial0/1, changed state to down
Oct 5 01:38:16: %LINK-3-UPDOWN: Interface Serial0/1, changed state to up
Oct 5 07:47:39: %LINK-3-UPDOWN: Interface Serial0/1, changed state to down
Oct 5 07:52:55: %LINK-3-UPDOWN: Interface Serial0/1, changed state to up
Oct 5 07:52:56: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/1, chan
ged state to up
sh service-module (Date/Time command issued 10/7/2010 3:50 AM EST)
Module type is T1/fractional
Hardware revision is 0.152, Software revision is 20050811-0.3,
Image checksum is 0x80ACD1B3, Protocol revision is 0.1
Receiver has no alarms.
Framing is ESF, Line Code is B8ZS, Current clock source is line,
Fraction has 24 timeslots (64 Kbits/sec each), Net bandwidth is 1536 Kbits/sec.
Last module self-test (done at startup): Passed
Last clearing of alarm counters 1d20h
loss of signal : 0,
loss of frame : 0,
AIS alarm : 0,
Remote alarm : 0,
Module access errors : 0,
Total Data (last 96 15 minute intervals):
0 Line Code Violations, 11 Path Code Violations
0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 7 Degraded Mins
7 Errored Secs, 4 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
Data in current interval (625 seconds elapsed):
0 Line Code Violations, 0 Path Code Violations
0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
Module type is T1/fractional
Hardware revision is 0.96, Software revision is 20050811-0.3,
Image checksum is 0x80ACD1B3, Protocol revision is 0.1
Receiver has no alarms.
Framing is ESF, Line Code is B8ZS, Current clock source is line,
Fraction has 24 timeslots (64 Kbits/sec each), Net bandwidth is 1536 Kbits/sec.
Last module self-test (done at startup): Passed
Last clearing of alarm counters 1d20h
loss of signal : 0,
loss of frame : 1, last occurred 1d19h
AIS alarm : 1, last occurred 1d19h
Remote alarm : 0,
Module access errors : 0,
Total Data (last 96 15 minute intervals):
0 Line Code Violations, 2 Path Code Violations
0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 2 Degraded Mins
2 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
Data in current interval (636 seconds elapsed):
0 Line Code Violations, 0 Path Code Violations
0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
I have attempted to do my own research for information regarding the output of this command but have been unsuccessful. Hopefuly your collective knowledge can help shed some light on my issue or at the very least my curiosity.
Regards,
Chris
Solved! Go to Solution.
10-07-2010 05:25 AM
Hello,
1. Sh controller output shows a Recieve Ring and Transmit Ring. The Receive Ring consists of 24 individual lines. The Transmit Ring consists of 16 individual lines. Does each line in the Receive and Transmit Ring represent 1 of 24 channels forming the T1 circuit?
The receive/transmit rings are a pre-allocated set of buffers that are used to DMA frames between controller and the host (router CPU). The tx/rx ring does not represent "channels". Without going into the internals of how they work, surfice it to say they are not related to the serial problems that you are describing.
2. Assuming each line represents a unique channel in the T1 circuit, Is the Transmit Ring only displaying 16 channels because the router is using hardware to reserve 8 channels for QoS configuration?
No, the tx-ring does not represent a "channel. The tx-ring for this chip defaults to 16 but can be changed using the tx-ring-limit command on the interface. Please see the following url for more information (BTW, I would not suggest tuning this value in your case):
http://www.cisco.com/en/US/tech/tk39/tk824/technologies_tech_note09186a00800fbafc.shtml
3. The Transmit Ring shows most lines having a status of status 5C00. At different times, different lines will have a status of status 0. Does status 0 represent a "down" channel?
The Status is a indicator as to who owns the buffer (either the controller, or the CPU) as well as some other information which is used by the inner workings of the driver. It does not pertain to the physical line or channels.
4. What would cause different lines to randomly show status 0?
Please see answer 3.
5. Does this indicate an issue with a multiplexer or other physical issue with the circuit and can this information be used to narrow down the problematic equipment?
No
Based on your problem description, you are seeing the T1 line bounce periodically. To troubleshoot this, please refer to the following documents:
http://www.cisco.com/en/US/tech/tk713/tk628/technologies_tech_note09186a00800a5f40.shtml
http://www.cisco.com/en/US/tech/tk713/tk628/technologies_tech_note09186a00800a754b.shtml
http://www.cisco.com/en/US/tech/tk713/tk628/technologies_tech_note09186a00800a7599.shtml
http://www.cisco.com/en/US/tech/tk713/tk628/technologies_tech_note09186a00800a758d.shtml
Can you attach a show interface as well?
Thanks
Tim
10-07-2010 05:25 AM
Hello,
1. Sh controller output shows a Recieve Ring and Transmit Ring. The Receive Ring consists of 24 individual lines. The Transmit Ring consists of 16 individual lines. Does each line in the Receive and Transmit Ring represent 1 of 24 channels forming the T1 circuit?
The receive/transmit rings are a pre-allocated set of buffers that are used to DMA frames between controller and the host (router CPU). The tx/rx ring does not represent "channels". Without going into the internals of how they work, surfice it to say they are not related to the serial problems that you are describing.
2. Assuming each line represents a unique channel in the T1 circuit, Is the Transmit Ring only displaying 16 channels because the router is using hardware to reserve 8 channels for QoS configuration?
No, the tx-ring does not represent a "channel. The tx-ring for this chip defaults to 16 but can be changed using the tx-ring-limit command on the interface. Please see the following url for more information (BTW, I would not suggest tuning this value in your case):
http://www.cisco.com/en/US/tech/tk39/tk824/technologies_tech_note09186a00800fbafc.shtml
3. The Transmit Ring shows most lines having a status of status 5C00. At different times, different lines will have a status of status 0. Does status 0 represent a "down" channel?
The Status is a indicator as to who owns the buffer (either the controller, or the CPU) as well as some other information which is used by the inner workings of the driver. It does not pertain to the physical line or channels.
4. What would cause different lines to randomly show status 0?
Please see answer 3.
5. Does this indicate an issue with a multiplexer or other physical issue with the circuit and can this information be used to narrow down the problematic equipment?
No
Based on your problem description, you are seeing the T1 line bounce periodically. To troubleshoot this, please refer to the following documents:
http://www.cisco.com/en/US/tech/tk713/tk628/technologies_tech_note09186a00800a5f40.shtml
http://www.cisco.com/en/US/tech/tk713/tk628/technologies_tech_note09186a00800a754b.shtml
http://www.cisco.com/en/US/tech/tk713/tk628/technologies_tech_note09186a00800a7599.shtml
http://www.cisco.com/en/US/tech/tk713/tk628/technologies_tech_note09186a00800a758d.shtml
Can you attach a show interface as well?
Thanks
Tim
10-08-2010 03:17 AM
ATT is going to test their Span segment by segment this weekend. I'll let you know if they find an issue there. After ATT tests are complete I will loop each of the serial interfaces, generate some traffic and look for errors. Also, I wanted to thank you for directly answering each of my questions. It let me move on from my nagging questions.
As requested here are the interface statistics:
Serial0/0 is up, line protocol is up
Hardware is PQUICC with Fractional T1 CSU/DSU
MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation PPP, LCP Open, multilink Open, loopback not set
Keepalive set (10 sec)
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 2d22h
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
30 second input rate 2000 bits/sec, 4 packets/sec
30 second output rate 2000 bits/sec, 5 packets/sec
10860860 packets input, 1679275340 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
106 input errors, 17 CRC, 59 frame, 0 overrun, 0 ignored, 30 abort
11351141 packets output, 1545097301 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
DCD=up DSR=up DTR=up RTS=up CTS=up
Serial0/1 is up, line protocol is up
Hardware is PQUICC with Fractional T1 CSU/DSU
MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation PPP, LCP Open, multilink Open, loopback not set
Keepalive set (10 sec)
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 2d22h
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
30 second input rate 1000 bits/sec, 3 packets/sec
30 second output rate 1000 bits/sec, 3 packets/sec
10690222 packets input, 1565550554 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 1 giants, 0 throttles
271 input errors, 3 CRC, 261 frame, 0 overrun, 0 ignored, 7 abort
11216102 packets output, 1513811962 bytes, 0 underruns
0 output errors, 0 collisions, 114 interface resets
0 output buffer failures, 0 output buffers swapped out
2 carrier transitions
DCD=up DSR=up DTR=up RTS=up CTS=up
Multilink1 is up, line protocol is up
Hardware is multilink group interface
MTU 1500 bytes, BW 3072 Kbit, DLY 100000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation PPP, LCP Open, multilink Open
Open: IPCP, loopback not set
Keepalive set (10 sec)
DTR is pulsed for 2 seconds on reset
Last input 00:00:00, output never, output hang never
Last clearing of "show interface" counters 2d22h
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 4160
Queueing strategy: fifo
Output queue: 0/40 (size/max)
30 second input rate 7000 bits/sec, 12 packets/sec
30 second output rate 9000 bits/sec, 11 packets/sec
21450716 packets input, 3114533655 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
1 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
18078211 packets output, 2878577656 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
10-07-2010 10:08 AM
Chris,
This frequent bounces looks like a keepalive mismatch to me.
Could you check these values match at both the ends ?
10-08-2010 03:23 AM
Keepalive is set to (10 sec) on each end for both circuits. What drew your attention to the possiblity of a Keepalive mismatch?
10-08-2010 03:31 AM
Such frequent bounces are normally caused by mismatched keep alives.
Let the carrier come out with the testing results.
Do share the outcome in here, it will help all of us.
Cheers !
10-08-2010 04:40 AM
Looking at the show interface we can see a number of physical errors such as crc's, aborts and framing. The show controller also shows a number of aborts. I suspect the problem is between the router and the telco (likely in the telco setup).
Tim
10-08-2010 06:37 AM
Well FYI,
The circuit can still work with all the errors you told, be it abort, crc or input, but it is less likely to bounce the interface in that way..-:)
My bad, i didnt deep dive at the counter level...-:)
And if you break it down the issue can exist anywhere starting from the
1.WIC card
2.corroded cable
3.Smart Jack
4.DACS
5.Issue at the ADM
or
even go up to a faulty card in the Transport box SONET ring and even furthur....
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: