cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
824
Views
0
Helpful
7
Replies

Interpret sh controller output on 2650 w/ Integrated FT1 CSU/DSU module

chollin611
Level 1
Level 1

Equipment Background Information:

sh ver:

Cisco Internetwork Operating System Software
IOS (tm) C2600 Software (C2600-JS-M), Version 12.3(26), RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2008 by cisco Systems, Inc.
Compiled Mon 17-Mar-08 15:24 by dchih

ROM: System Bootstrap, Version 12.2(6r),  RELEASE SOFTWARE (fc1)

cisco 2650 (MPC860P) processor (revision 0x00) with 126976K/4096K bytes of memory.
Processor board ID JAD06200P0W (2846690197)
M860 processor: part number 5, mask 2
Bridging software.
X.25 software, Version 3.0.0.
SuperLAT software (copyright 1990 by Meridian Technology Corp).
TN3270 Emulation software.
1 FastEthernet/IEEE 802.3 interface(s)
2 Serial network interface(s)
32K bytes of non-volatile configuration memory.
32768K bytes of processor board System flash (Read/Write)

Abreviated Sh run:

class-map match-all QoS

  match access-group 100

!

!

policy-map QoSWAN

  class QoS

   set dscp ef

  class class-default

   set dscp default

!

!

!

!

interface Multilink1

ip address #.#.#.# 255.255.255.252

load-interval 30

no cdp enable

ppp multilink

ppp multilink group 1

service-policy output QoSWAN

!

interface FastEthernet0/0

ip address 192.168.7.3 255.255.255.0

load-interval 30

duplex auto

speed auto

!

interface Serial0/0

  no ip address

encapsulation ppp

load-interval 30

no fair-queue

service-module t1 timeslots 1-24

ppp multilink

ppp multilink group 1

!

interface Serial0/1

no ip address

encapsulation ppp

load-interval 30

no fair-queue

service-module t1 timeslots 1-24

ppp multilink

ppp multilink group 1

!

no ip http server

ip classless

ip route 0.0.0.0 0.0.0.0

!

!

access-list 100 permit ip host 192.168.7.21 any

access-list 100 permit ip any host 192.168.3.20

access-list 100 permit ip any host 192.168.4.20

access-list 100 permit ip any host 192.168.5.20

access-list 100 permit ip any host 192.168.6.20

access-list 100 permit ip any host 192.168.8.20

access-list 100 permit ip any host 192.168.9.20

no cdp run

Background information on problem:

Intermittently taking large number of service affecting errors.  Circuit bounces frequently.  (see counters below).  ATT sees errors in PM logs but can't capture event during testing.  Service is Frame Relay across ILEC Frame Cloud to ISP Talk to Circuit.  Router is configured for QoS.

Questions:

All questions are related but separated here to make reading easier.

1.  Sh controller output shows a Recieve Ring and Transmit Ring. The Receive Ring consists of 24 individual lines.  The Transmit Ring consists of 16 individual lines.  Does each line in the Receive and Transmit Ring represent 1 of 24 channels forming the T1 circuit?

2.  Assuming each line represents a unique channel in the T1 circuit, Is the Transmit Ring only displaying 16 channels because the router is using hardware to reserve 8 channels for QoS configuration?

3.  The Transmit Ring shows most lines having a status of  status 5C00.  At different times, different lines will have a status of  status 0.  Does status 0 represent a "down" channel?

4.  What would cause different lines to randomly show status 0?

5.  Does this indicate an issue with a multiplexer or other physical issue with the circuit and can this information be used to narrow down the problematic equipment?

Actual log entries from Sh controller output of router:

Receive Ring
        rmd(680126B0): status 9000 length 5E8 address 7D6BDA4
        rmd(680126B8): status 9000 length 5E8 address 7D6D7A4
        rmd(680126C0): status 9000 length 5E8 address 7D6EB24
        rmd(680126C8): status 9000 length 5E8 address 7D6A3A4
        rmd(680126D0): status 9000 length 4BB address 7D66FA4
        rmd(680126D8): status 9000 length 36 address 7D76024
        rmd(680126E0): status 9000 length 45 address 7D78DA4
        rmd(680126E8): status 9000 length 45 address 7D662A4
        rmd(680126F0): status 9000 length 36 address 7D6F824
        rmd(680126F8): status 9000 length 40 address 7D72C24
        rmd(68012700): status 9000 length 36 address 7D696A4
        rmd(68012708): status 9000 length 4A address 7D78724
        rmd(68012710): status 9000 length 5E8 address 7D759A4
        rmd(68012718): status 9000 length 5E8 address 7D766A4
        rmd(68012720): status 9000 length 5E8 address 7D6AA24
        rmd(68012728): status 9000 length 5E8 address 7D773A4
        rmd(68012730): status 9000 length 5E8 address 7D6E4A4
        rmd(68012738): status 9000 length 5E8 address 7D732A4
        rmd(68012740): status 9000 length 5E8 address 7D725A4
        rmd(68012748): status 9000 length 5E8 address 7D79424
        rmd(68012750): status 9000 length 5E8 address 7D67624
        rmd(68012758): status 9000 length 5E8 address 7D6FEA4
        rmd(68012760): status 9000 length 5E8 address 7D73FA4
        rmd(68012768): status B000 length 5E8 address 7D74CA4

Transmit Ring
        tmd(68012770): status 5C00 length 43 address 7D1F0E4
        tmd(68012778): status 0 length 8 address 7D01B70
        tmd(68012780): status 5C00 length 3D address 7D47B28
        tmd(68012788): status 5C00 length 43 address 7D28664
        tmd(68012790): status 5C00 length 43 address 7D195E4
        tmd(68012798): status 5C00 length 12A address 7D9064E
        tmd(680127A0): status 5C00 length 12A address 7C02DAE
        tmd(680127A8): status 5C00 length 64 address 7D8FF4A
        tmd(680127B0): status 0 length 8 address 7D01B70
        tmd(680127B8): status 5C00 length 31 address 7D1FE1A
        tmd(680127C0): status 0 length 8 address 7D01B70
        tmd(680127C8): status 5C00 length 3D address 7D245A8
        tmd(680127D0): status 5C00 length 32 address 7D2A064
        tmd(680127D8): status 5C00 length 43 address 7D3F8E4
        tmd(680127E0): status 5C00 length 32 address 7D2A6E4
        tmd(680127E8): status 2000 length 8 address 7D01B70

tx_limited=0(16)

SCC GENERAL PARAMETER RAM (at 0x68013C00)
Rx BD Base [RBASE]=0x26B0, Fn Code [RFCR]=0x18
Tx BD Base [TBASE]=0x2770, Fn Code [TFCR]=0x18
Max Rx Buff Len [MRBLR]=1548
Rx State [RSTATE]=0x18008440, BD Ptr [RBPTR]=0x2750
Tx State [TSTATE]=0x18800511, BD Ptr [TBPTR]=0x27E0

SCC HDLC PARAMETER RAM (at 0x68013C38)
CRC Preset [C_PRES]=0xFFFF, Mask [C_MASK]=0xF0B8
Errors: CRC [CRCEC]=736, Aborts [ABTSC]=6485, Discards [DISFC]=0
Nonmatch Addr Cntr [NMARC]=0
Retry Count [RETRC]=0
Max Frame Length [MFLR]=1608
Rx Int Threshold [RFTHR]=0, Frame Cnt [RFCNT]=38869
User-defined Address 0000/0000/0000/0000
User-defined Address Mask 0x0000


buffer size 1524

PowerQUICC SCC specific errors:
11374743 input aborts on receiving flag sequence
0 throttles, 0 enables
0 overruns
0 transmitter underruns
0 transmitter CTS losts
957977 aborted short frames

Example of Circuit Bounces

sh log: (Date/Time command issued 10/7/2010 3:50 AM EST)

Oct  4 20:18:03: %LINK-3-UPDOWN: Interface Serial0/0, changed state to down
Oct  4 20:18:04: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0, chan
ged state to down
Oct  4 20:21:25: %LINK-3-UPDOWN: Interface Serial0/0, changed state to up
Oct  4 20:21:40: %LINK-3-UPDOWN: Interface Serial0/0, changed state to down
Oct  4 21:26:48: %LINK-3-UPDOWN: Interface Serial0/0, changed state to up
Oct  4 21:27:40: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0, chan
ged state to up
Oct  4 21:29:21: %LINK-3-UPDOWN: Interface Serial0/0, changed state to down
Oct  4 21:29:22: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0, chan
ged state to down
Oct  4 21:34:39: %LINK-3-UPDOWN: Interface Serial0/0, changed state to up
Oct  4 21:34:53: %LINK-3-UPDOWN: Interface Multilink1, changed state to down
Oct  4 21:34:53: %LINK-3-UPDOWN: Interface Multilink1, changed state to up
Oct  4 21:40:42: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0, chan
ged state to up
Oct  5 00:16:54: %LINK-3-UPDOWN: Interface Serial0/1, changed state to down
Oct  5 00:16:55: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/1, chan
ged state to down
Oct  5 00:22:25: %LINK-3-UPDOWN: Interface Serial0/1, changed state to up
Oct  5 00:22:41: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/1, chan
ged state to up
Oct  5 00:23:33: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/1, chan
ged state to down
Oct  5 00:28:50: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/1, chan
ged state to up
Oct  5 00:36:41: %LINK-3-UPDOWN: Interface Serial0/1, changed state to down
Oct  5 00:36:42: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/1, chan
ged state to down
Oct  5 00:58:30: %LINK-3-UPDOWN: Interface Serial0/1, changed state to up
Oct  5 00:58:45: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/1, chan
ged state to up
Oct  5 00:59:47: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/1, chan
ged state to down
Oct  5 01:28:12: %LINK-3-UPDOWN: Interface Serial0/1, changed state to down
Oct  5 01:38:16: %LINK-3-UPDOWN: Interface Serial0/1, changed state to up
Oct  5 07:47:39: %LINK-3-UPDOWN: Interface Serial0/1, changed state to down
Oct  5 07:52:55: %LINK-3-UPDOWN: Interface Serial0/1, changed state to up
Oct  5 07:52:56: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/1, chan
ged state to up

sh service-module (Date/Time command issued 10/7/2010 3:50 AM EST)

Module type is T1/fractional
    Hardware revision is 0.152, Software revision is 20050811-0.3,
    Image checksum is 0x80ACD1B3, Protocol revision is 0.1
Receiver has no alarms.
Framing is ESF, Line Code is B8ZS, Current clock source is line,
Fraction has 24 timeslots (64 Kbits/sec each), Net bandwidth is 1536 Kbits/sec.
Last module self-test (done at startup): Passed
Last clearing of alarm counters 1d20h
    loss of signal        :    0,
    loss of frame         :    0,
    AIS alarm             :    0,
    Remote alarm          :    0,
    Module access errors  :    0,
Total Data (last 96 15 minute intervals):
    0 Line Code Violations, 11 Path Code Violations
    0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 7 Degraded Mins
    7 Errored Secs, 4 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
Data in current interval (625 seconds elapsed):
    0 Line Code Violations, 0 Path Code Violations
    0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
    0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
Module type is T1/fractional
    Hardware revision is 0.96, Software revision is 20050811-0.3,
    Image checksum is 0x80ACD1B3, Protocol revision is 0.1
Receiver has no alarms.
Framing is ESF, Line Code is B8ZS, Current clock source is line,
Fraction has 24 timeslots (64 Kbits/sec each), Net bandwidth is 1536 Kbits/sec.
Last module self-test (done at startup): Passed
Last clearing of alarm counters 1d20h
    loss of signal        :    0,
    loss of frame         :    1, last occurred 1d19h
    AIS alarm             :    1, last occurred 1d19h
    Remote alarm          :    0,
    Module access errors  :    0,
Total Data (last 96 15 minute intervals):
    0 Line Code Violations, 2 Path Code Violations
    0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 2 Degraded Mins
    2 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
Data in current interval (636 seconds elapsed):
    0 Line Code Violations, 0 Path Code Violations
    0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
    0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs

I have attempted to do my own research for information regarding the output of this command but have been unsuccessful.  Hopefuly your collective knowledge can help shed some light on my issue or at the very least my curiosity.

Regards,

Chris

1 Accepted Solution

Accepted Solutions

Timothy Stewart
Cisco Employee
Cisco Employee

Hello,

1.  Sh controller output shows a Recieve Ring and Transmit Ring. The Receive Ring consists of 24 individual lines.  The Transmit Ring consists of 16 individual lines.  Does each line in the Receive and Transmit Ring represent 1 of 24 channels forming the T1 circuit?

The receive/transmit rings are a pre-allocated set of buffers that are used to DMA frames between controller and the host (router CPU).  The tx/rx ring does not represent "channels".  Without going into the internals of how they work, surfice it to say they are not related to the serial problems that you are describing.

2.  Assuming each line represents a unique channel in the T1 circuit, Is the Transmit Ring only displaying 16 channels because the router is using hardware to reserve 8 channels for QoS configuration?

No, the tx-ring does not represent a "channel.  The tx-ring for this chip defaults to 16 but can be changed using the tx-ring-limit command on the interface.  Please see the following url for more information (BTW, I would not suggest tuning this value in your case):

http://www.cisco.com/en/US/tech/tk39/tk824/technologies_tech_note09186a00800fbafc.shtml

3.  The Transmit Ring shows most lines having a status of  status 5C00.  At different times, different lines will have a status of  status 0.  Does status 0 represent a "down" channel?

The Status is a indicator as to who owns the buffer (either the controller, or the CPU) as well as some other information which is used by the inner workings of the driver.  It does not pertain to the physical line or channels.

4.  What would cause different lines to randomly show status 0?

Please see answer 3.


5.  Does this indicate an issue with a multiplexer or other physical issue with the circuit and can this information be used to narrow down the problematic equipment?

No

Based on your problem description, you are seeing the T1 line bounce periodically.  To troubleshoot this, please refer to the following documents:

http://www.cisco.com/en/US/tech/tk713/tk628/technologies_tech_note09186a00800a5f40.shtml

http://www.cisco.com/en/US/tech/tk713/tk628/technologies_tech_note09186a00800a754b.shtml

http://www.cisco.com/en/US/tech/tk713/tk628/technologies_tech_note09186a00800a7599.shtml

http://www.cisco.com/en/US/tech/tk713/tk628/technologies_tech_note09186a00800a758d.shtml

Can you attach a show interface as well?

Thanks

Tim

View solution in original post

7 Replies 7

Timothy Stewart
Cisco Employee
Cisco Employee

Hello,

1.  Sh controller output shows a Recieve Ring and Transmit Ring. The Receive Ring consists of 24 individual lines.  The Transmit Ring consists of 16 individual lines.  Does each line in the Receive and Transmit Ring represent 1 of 24 channels forming the T1 circuit?

The receive/transmit rings are a pre-allocated set of buffers that are used to DMA frames between controller and the host (router CPU).  The tx/rx ring does not represent "channels".  Without going into the internals of how they work, surfice it to say they are not related to the serial problems that you are describing.

2.  Assuming each line represents a unique channel in the T1 circuit, Is the Transmit Ring only displaying 16 channels because the router is using hardware to reserve 8 channels for QoS configuration?

No, the tx-ring does not represent a "channel.  The tx-ring for this chip defaults to 16 but can be changed using the tx-ring-limit command on the interface.  Please see the following url for more information (BTW, I would not suggest tuning this value in your case):

http://www.cisco.com/en/US/tech/tk39/tk824/technologies_tech_note09186a00800fbafc.shtml

3.  The Transmit Ring shows most lines having a status of  status 5C00.  At different times, different lines will have a status of  status 0.  Does status 0 represent a "down" channel?

The Status is a indicator as to who owns the buffer (either the controller, or the CPU) as well as some other information which is used by the inner workings of the driver.  It does not pertain to the physical line or channels.

4.  What would cause different lines to randomly show status 0?

Please see answer 3.


5.  Does this indicate an issue with a multiplexer or other physical issue with the circuit and can this information be used to narrow down the problematic equipment?

No

Based on your problem description, you are seeing the T1 line bounce periodically.  To troubleshoot this, please refer to the following documents:

http://www.cisco.com/en/US/tech/tk713/tk628/technologies_tech_note09186a00800a5f40.shtml

http://www.cisco.com/en/US/tech/tk713/tk628/technologies_tech_note09186a00800a754b.shtml

http://www.cisco.com/en/US/tech/tk713/tk628/technologies_tech_note09186a00800a7599.shtml

http://www.cisco.com/en/US/tech/tk713/tk628/technologies_tech_note09186a00800a758d.shtml

Can you attach a show interface as well?

Thanks

Tim

ATT is going to test their Span segment by segment this weekend.  I'll let you know if they find an issue there.  After ATT tests are complete I will loop each of the serial interfaces, generate some traffic and look for errors.  Also, I wanted to thank you for directly answering each of my questions. It let me move on from my nagging questions.

As requested here are the interface statistics:

Serial0/0 is up, line protocol is up
  Hardware is PQUICC with Fractional T1 CSU/DSU
   MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation PPP, LCP Open, multilink Open, loopback not set
  Keepalive set (10 sec)
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 2d22h
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  30 second input rate 2000 bits/sec, 4 packets/sec
  30 second output rate 2000 bits/sec, 5 packets/sec
     10860860 packets input, 1679275340 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     106 input errors, 17 CRC, 59 frame, 0 overrun, 0 ignored, 30 abort
     11351141 packets output, 1545097301 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions
     DCD=up  DSR=up  DTR=up  RTS=up  CTS=up

Serial0/1 is up, line protocol is up
  Hardware is PQUICC with Fractional T1 CSU/DSU
   MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation PPP, LCP Open, multilink Open, loopback not set
  Keepalive set (10 sec)
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 2d22h
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  30 second input rate 1000 bits/sec, 3 packets/sec
  30 second output rate 1000 bits/sec, 3 packets/sec
     10690222 packets input, 1565550554 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 1 giants, 0 throttles
     271 input errors, 3 CRC, 261 frame, 0 overrun, 0 ignored, 7 abort
     11216102 packets output, 1513811962 bytes, 0 underruns
     0 output errors, 0 collisions, 114 interface resets
     0 output buffer failures, 0 output buffers swapped out
     2 carrier transitions
     DCD=up  DSR=up  DTR=up  RTS=up  CTS=up

Multilink1 is up, line protocol is up
  Hardware is multilink group interface
  MTU 1500 bytes, BW 3072 Kbit, DLY 100000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation PPP, LCP Open, multilink Open
  Open: IPCP, loopback not set
  Keepalive set (10 sec)
  DTR is pulsed for 2 seconds on reset
  Last input 00:00:00, output never, output hang never
  Last clearing of "show interface" counters 2d22h
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 4160
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  30 second input rate 7000 bits/sec, 12 packets/sec
  30 second output rate 9000 bits/sec, 11 packets/sec
     21450716 packets input, 3114533655 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     1 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     18078211 packets output, 2878577656 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions

Tharak Abraham
Level 3
Level 3

Chris,

This frequent bounces looks like a keepalive mismatch to me.

Could you check these values match at both the ends ?

Keepalive is set to (10 sec) on each end for both circuits.  What drew your attention to the possiblity of a Keepalive mismatch?

Such frequent bounces are normally caused by mismatched keep alives.

Let the carrier come out with the testing results.

Do share the outcome in here, it will help all of us.

Cheers !

Looking at the show interface we can see a number of physical errors such as crc's, aborts and framing.  The show controller also shows a number of aborts.  I suspect the problem is between the router and the telco (likely in the telco setup).

Tim

Well FYI,

The circuit can still work with all the errors you told, be it abort, crc or input, but it is less likely to bounce the interface in that way..-:)

My bad, i didnt deep dive at the counter level...-:)

And if you break it down the issue can exist anywhere starting from the

1.WIC card

2.corroded cable

3.Smart Jack

4.DACS

5.Issue at the ADM

or

even go up to a faulty card in the Transport box SONET ring and even furthur....

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: