Serial Interface Resets

InTech1976 · ‎11-19-2008

Many thanks to all who have helped sincce my router guy up and left...

I have one other pressing issue I need assistance on. One of our remote offices is connected to the main office via direct T1. I have been told applications consistently lose connection with our servers from users in this remote office. Here are the stats fom the remote office router's serial interface:

Remote Office Router: 2600

Serial0/0 is up, line protocol is up

Hardware is PQUICC with Fractional T1 CSU/DSU

Description: Wan to HQ

Internet address is 192.168.2.18/30

MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,

reliability 255/255, txload 2/255, rxload 1/255

Encapsulation HDLC, loopback not set

Keepalive set (10 sec)

Last input 00:00:00, output 00:00:01, output hang never

Last clearing of "show interface" counters never

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 127

Queueing strategy: weighted fair

Output queue: 0/1000/64/123 (size/max total/threshold/drops)

Conversations 0/41/256 (active/max active/max total)

Reserved Conversations 0/0 (allocated/max allocated)

Available Bandwidth 1158 kilobits/sec

5 minute input rate 3000 bits/sec, 2 packets/sec

5 minute output rate 16000 bits/sec, 2 packets/sec

74378626 packets input, 1848314845 bytes, 0 no buffer

Received 2961633 broadcasts, 7 runts, 60 giants, 0 throttles

10630 input errors, 3265 CRC, 6793 frame, 0 overrun, 0 ignored, 572 abort

61204959 packets output, 3853218004 bytes, 0 underruns

0 output errors, 0 collisions, 3 interface resets

0 output buffer failures, 0 output buffers swapped out

5 carrier transitions

DCD=up DSR=up DTR=up RTS=up CTS=up

Thinking it may just be their router, I see their routers serial interface has reset a number of times and has many, many drops. Out of curiosity, I checked our core router and saw the following:

Corporate Core Router: 2821

Serial0/3/0 is up, line protocol is up

Hardware is GT96K with integrated T1 CSU/DSU

Description: Wan to Office1

Internet address is 192.168.2.17/30

MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,

reliability 255/255, txload 3/255, rxload 3/255

Encapsulation HDLC, loopback not set

Keepalive set (10 sec)

Last input 00:00:01, output 00:00:05, output hang never

Last clearing of "show interface" counters 3d00h

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 3278

Queueing strategy: weighted fair

Output queue: 0/1000/64/3278 (size/max total/threshold/drops)

Conversations 0/31/256 (active/max active/max total)

Reserved Conversations 0/0 (allocated/max allocated)

Available Bandwidth 1158 kilobits/sec

5 minute input rate 23000 bits/sec, 6 packets/sec

5 minute output rate 20000 bits/sec, 6 packets/sec

387261 packets input, 115143728 bytes, 0 no buffer

Received 30332 broadcasts, 0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

487387 packets output, 371441638 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 output buffer failures, 0 output buffers swapped out

0 carrier transitions

DCD=up DSR=up DTR=up RTS=up CTS=up

This also shows drops from the branch office...

I am at a loss where to begin troubleshooting this, so I am here to humbly ask for your help....Thanks again for helping me learn!

ropethic · ‎11-19-2008

First let me direct you to the what the meaning of the output stats

http://www.cisco.com/en/US/docs/ios/12_1/interface/command/reference/irdshoin.html#wp1020497

The branch router looks okay. 126 drops for over 61 million packets is very good, as well inut traffic.

Less than 1% drops on core, however this is over 3 days. keep an eye on this one. The branch router counters never clearing since last router reboot.

Tasks to be completed:

1. Show log - this will show if serial up / down status

2. Clear counters on remote office router

clear counters interface s0/0

Post show logs output.

I would also check to see if you are running backups, virus updates / scans during the day to users in office. I have seen whole departments lose connectivity during scheduled backups and virus scans.

InTech1976 · ‎11-19-2008

Many thanks, once again. I will review the link you provided. Here is the info you requested. (I have to split this between 2 posts since it is loinger than 4000 characters - the board maximum)

Branch Office Router's sh log:

boffice1#show log

Syslog logging: enabled (0 messages dropped, 1 messages rate-limited, 0 flushes,

0 overruns, xml disabled)

Console logging: level debugging, 227 messages logged, xml disabled

Monitor logging: level debugging, 0 messages logged, xml disabled

Buffer logging: disabled, xml disabled

Logging Exception size (4096 bytes)

Count and timestamp logging messages: disabled

Trap logging: level informational, 230 message lines logged

Core Router sh log:

Syslog logging: enabled (1 messages dropped, 1 messages rate-limited,

0 flushes, 0 overruns, xml disabled, filtering disabled)

No Active Message Discriminator.

No Inactive Message Discriminator.

Console logging: level critical, 0 messages logged, xml disabled,

filtering disabled

Monitor logging: level debugging, 0 messages logged, xml disabled,

filtering disabled

Buffer logging: level debugging, 69 messages logged, xml disabled,

filtering disabled

Logging Exception size (4096 bytes)

Count and timestamp logging messages: disabled

Persistent logging: disabled

Trap logging: level debugging, 71 message lines logged

InTech1976 · ‎11-19-2008

Log Buffer (4096 bytes):

.0.131)

000033: Jul 7 09:29:29.647 Chicago: %SYS-5-CONFIG_I: Configured from console by bhaase on vty1 (192.168.0.131)

000034: Jul 7 10:12:23.881 Chicago: %SYS-6-CLOCKUPDATE: System clock has been updated from 10:12:23 Chicago Mon Jul 7 2008 to 10:12:23 Chicago Mon Jul 7 2008, configured from console by last on vty0 (192.168.0.52).

000035: Jul 7 10:12:23.905 Chicago: %SYS-6-CLOCKUPDATE: System clock has been updated from 10:12:23 Chicago Mon Jul 7 2008 to 10:12:23 Chicago Mon Jul 7 2008, configured from console by last on vty0 (192.168.0.52).

000036: Jul 7 10:12:24.073 Chicago: % Multiple self signed certificates in config

certificate for trust point TP-self-signed-2057303117 ignored

000037: Jul 7 10:12:26.430 Chicago: %PKI-4-NOAUTOSAVE: Configuration was modified. Issue "write memory" to save new certificate

000038: Jul 7 10:13:24.054 Chicago: %SYS-5-CONFIG_I: Configured from console by someuser on vty0 (192.168.0.52)

000039: Jul 7 10:15:22.676 Chicago: %SYS-5-CONFIG_I: Configured from console by someuser on vty0 (192.168.0.52)

000040: Jul 7 10:21:06.421 Chicago: %SYS-5-CONFIG_I: Configured from console by someuser on vty0 (192.168.0.52)

000041: Jul 7 10:22:41.297 Chicago: %SYS-5-CONFIG_I: Configured from console by someuser on vty0 (192.168.0.52)

000042: Jul 17 12:39:29.298 Chicago: %SYS-5-CONFIG_I: Configured from console by someuser on vty0 (192.168.0.55)

000043: Jul 17 12:45:12.877 Chicago: %SYS-5-CONFIG_I: Configured from console by someuser on vty0 (192.168.0.55)

000044: Aug 11 10:28:44.241 Chicago: %SYS-5-CONFIG_I: Configured from console by bh on vty0 (192.168.0.131)

000045: Oct 1 08:41:39.235 Chicago: %SYS-5-CONFIG_I: Configured from console by someuser on vty0 (192.168.0.50)

000046: Oct 1 15:12:55.821 Chicago: %SYS-5-CONFIG_I: Configured from console by someuser on vty0 (192.168.0.50)

000047: Nov 4 14:38:27.301 Chicago: %SYS-5-CONFIG_I: Configured from console by someuser on vty0 (192.168.0.53)

000048: Nov 4 14:38:27.621 Chicago: %LINK-3-UPDOWN: Interface Serial0/2/0, changed state to up

000049: Nov 4 14:38:28.621 Chicago: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/2/0, changed state to up

000050: Nov 4 15:03:44.214 Chicago: %SYS-5-CONFIG_I: Configured from console by someuser on vty0 (192.168.0.53)

000051: Nov 5 08:51:53.908 Chicago: %SYS-5-CONFIG_I: Configured from console by bh on vty1 (192.168.0.131)

000052: Nov 5 09:10:17.411 Chicago: %CLEAR-5-COUNTERS: Clear counter on interface Serial0/2/0 by bh on vty1 (192.168.0.131)

000053: Nov 5 09:14:18.207 Chicago: %CLEAR-5-COUNTERS: Clear counter on interface Serial0/2/0 by bh on vty1 (192.168.0.131)

000054: Nov 5 12:03:23.732 Chicago: %SYS-5-CONFIG_I: Configured from console by bh on vty1 (192.168.0.131)

000055: Nov 6 17:30:28.685 Chicago: %SYS-5-CONFIG_I: Configured from console by someuser on vty0 (192.168.0.50)

000056: Nov 6 17:31:08.450 Chicago: %SYS-6-CLOCKUPDATE: System clock has been updated from 17:31:08 Chicago Thu Nov 6 2008 to 17:31:08 Chicago Thu Nov 6 2008, configured from console by last on vty0 (192.168.0.50).

000057: Nov 6 17:31:08.478 Chicago: %SYS-6-CLOCKUPDATE: System clock has been updated from 17:31:08 Chicago Thu Nov 6 2008 to 17:31:08 Chicago Thu Nov 6 2008, configured from console by last on vty0 (192.168.0.50).

000058: Nov 6 17:31:08.662 Chicago: % Multiple self signed certificates in config

certificate for trust point TP-self-signed-2057303117 ignored

000059: Nov 6 17:31:11.190 Chicago: %SYS-5-CONFIG_I: Configured from console by someuser on vty0 (192.168.0.50)

000060: Nov 6 17:32:56.591 Chicago: %SYS-5-CONFIG_I: Configured from console by someuser on vty0 (192.168.0.50)

000061: Nov 11 13:30:37.873 Chicago: %SYS-5-CONFIG_I: Configured from console by bh on vty0 (192.168.0.131)

000062: Nov 11 13:31:12.290 Chicago: %SYS-5-CONFIG_I: Configured from console by bh on vty0 (192.168.0.131)

000063: Nov 11 13:31:46.378 Chicago: %SYS-5-

glen.grant · ‎11-19-2008

Looks like buffer logging is turned off on the branch router so nothing will ever be put in the log on the router . I would enable buffer logging on that router so that you can see if anything funny is going on . Also when people happen to be complaining I would take a look at the utilization on that link , as you get near the limit it very well could start dropping packets. A T1 isn't much bandwidth nowadays . Most people have a lot more than that just to surf the net at home . It would basically take 1 or 2 people downloading something to suck up a T1.

conf t

logging buffer 16384 informational

end

write mem

stephenshaw · ‎11-20-2008

Hi, input and CRC errors are typically the result of a defective CSU/DSU or actual circuit problem. If the router shows no log entries for the actual CSU/DSU you may need to involve the Telco to start running some loopback tests on the circuit to pinpoint where the errors start occuring in the circuit path. Hopefully, you won't need to replace the CSU/DSU card.

Regards,

Steve