Weird serial interface problem

Unanswered Question
Dec 16th, 2008
User Badges:
  • Purple, 4500 points or more

We have a 3745 that has a DS3 connection into a frame cloud. Yesterday we noticed that the serial interface had almost 100% of bandwidth being used outbound. I set up netflow to see what was using it, and it wasn't consistent. I tried to block ports and servers in an acl, and I applied it inbound on the serial interface, but that didn't make a difference. The configuration for the port is here and I've bolded the lines that we're concerned with:


Serial1/0 is up, line protocol is up

Hardware is DSXPNM Serial

Description: DS3 connection for the GO

MTU 4470 bytes, BW 44210 Kbit, DLY 200 usec,

reliability 255/255, txload 92/255, rxload 8/255

Encapsulation FRAME-RELAY IETF, crc 16, loopback not set

Keepalive set (10 sec)

LMI enq sent 5539, LMI stat recvd 5539, LMI upd recvd 0, DTE LMI up

LMI enq recvd 0, LMI stat sent 0, LMI upd sent 0

LMI DLCI 1023 LMI type is CISCO frame relay DTE

FR SVC disabled, LAPF state down

Broadcast queue 0/64, broadcasts sent/dropped 923/0, interface broadcasts 0

Last input 00:00:00, output 00:00:00, output hang never

Last clearing of "show interface" counters 15:23:06

Input queue: 1/75/0/0 (size/max/drops/flushes); Total output drops: 10003

Queueing strategy: fifo

Output queue: 0/40 (size/max)

30 second input rate 1521000 bits/sec, 494 packets/sec

30 second output rate 16042000 bits/sec, 3334 packets/sec

35238211 packets input, 2648515031 bytes, 0 no buffer

Received 0 broadcasts, 0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

192251637 packets output, 1216186812 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 output buffer failures, 0 output buffers swapped out

0 carrier transitions

DSU mode 0, bandwidth 44210, real bandwidth 44210, scramble 0


a.) The DSU line that we have is a 20mb connection, but the default is 44210. The line is in the running config as "dsu bandwidth 44210", but according to Cisco is the default. I think this is primarily for calculations, and that would mean that my "tx load" would be even higher than it shows now.


b.) The tx load is almost at 100% if we take into calculation that it's around 30% now, but it "thinks" it's a 45mb pipe. If we convert that to a 20mb calculation, then the transmit rate would naturally increase to something around 225/255. (Does that make sense?)


c.) The 30 second output is at 16mb, which means that there's 16mb of data going out CONSTANT out of a 20mb connection. This isn't normal for this connection.


d.) Do I need to be concerned with the dropped packets?


Also, when we're talking about output for this interface, is that output respective of the direction? I mean if I'm sending traffic through the router from one side and it's going into my network, does that register as output as well as sending traffic out of my network?


I'm at a loss as to why there's so much traffic, and where it's coming from and I hope you guys can give me some pointers as to what's going on. This is not a new connection either by the way; this just started happening yesterday morning.


Thanks!

John

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
patrickvanham Tue, 12/16/2008 - 04:21
User Badges:

You could put the command bandwidth 20000 to verify your bandwidth use. This is mainly administrative and will be used for bandwidth calculations as well as well as routing protocols.


The output interface is what your network is sending, so it is sourced in your network. If there is a sudden increase you may want to investigate whether there is a sudden increase in demand on a server in your network, otherwise a worm or virus may have managed to find its way onto one or more workstations.

John Blakley Tue, 12/16/2008 - 04:23
User Badges:
  • Purple, 4500 points or more

We tried to block several different ports that were coming through netflow, but there was no difference. At first we thought that the server group may have been doing upgrades, but that wasn't the case either.



patrickvanham Tue, 12/16/2008 - 04:26
User Badges:

If you see no load increase on any of your servers it seems likely a worm or similar has become active. Since these do not always use "known ports" it's difficult to find. If you have stats on workstations in the LAN you could use that to identify any source or sources for this increase in load.

John Blakley Tue, 12/16/2008 - 04:27
User Badges:
  • Purple, 4500 points or more

Other than netflow, what's another really good way to determine what address is sending major traffic? I've tried ACLs that permit anything through and then changed the logging update threshold, but I just seem to get a ton of crap because of the way the connection is into the cloud.

Richard Burts Tue, 12/16/2008 - 04:46
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

John


While you might try something like IP accounting to categorize and identify the traffic, I think that NetFlow is the optimum tool for this purpose. Are you reading the NetFlow results manually on the router or are you exporting NetFlow to some device that gathers data and reports results over some time period (the better solution)?


Does the code that you are running support the NetFlow top talker functionality? If so that would be the easy way to find what is generating the most traffic.


Your original post talked about bandwidth (which is just administrative) and about dsu bandwidth which could be a controller command and if so is more than just administrative. Could you post the controller/service module (if present) and interface configuration?


HTH


Rick

John Blakley Tue, 12/16/2008 - 04:51
User Badges:
  • Purple, 4500 points or more

Rick,


I'm using the free version of Solarwinds Netflow to collect. I set up top talkers on the router, but the data isn't consistent with what I'm seeing on the output of the interface. For instance, my top 10 doesn't equal 80% of the bandwidth (all of them are maxing out at a certain kb, but only equates to about 2mb of speed). I don't have a collector that can write to a database for trending.


Here's the controller and interface information:


controller T1 0/0

framing esf

linecode b8zs

channel-group 0 timeslots 1-24 speed 64

!

controller T1 0/1

framing esf

linecode b8zs

channel-group 0 timeslots 1-24 speed 64

!

controller T3 1/0

clock source line

!

interface Serial1/0

description DS3 connection for the GO

no ip address

encapsulation frame-relay IETF

no ip route-cache cef

no ip mroute-cache

load-interval 30

dsu bandwidth 44210

frame-relay lmi-type cisco



interface Serial1/0.115 point-to-point

description DS3 connection for the GO

ip address 172.x.x.1 255.x.x.x

frame-relay interface-dlci 1002


Thanks!

John

Actions

This Discussion