12-16-2008 04:01 AM - edited 03-04-2019 12:43 AM
We have a 3745 that has a DS3 connection into a frame cloud. Yesterday we noticed that the serial interface had almost 100% of bandwidth being used outbound. I set up netflow to see what was using it, and it wasn't consistent. I tried to block ports and servers in an acl, and I applied it inbound on the serial interface, but that didn't make a difference. The configuration for the port is here and I've bolded the lines that we're concerned with:
Serial1/0 is up, line protocol is up
Hardware is DSXPNM Serial
Description: DS3 connection for the GO
MTU 4470 bytes, BW 44210 Kbit, DLY 200 usec,
reliability 255/255, txload 92/255, rxload 8/255
Encapsulation FRAME-RELAY IETF, crc 16, loopback not set
Keepalive set (10 sec)
LMI enq sent 5539, LMI stat recvd 5539, LMI upd recvd 0, DTE LMI up
LMI enq recvd 0, LMI stat sent 0, LMI upd sent 0
LMI DLCI 1023 LMI type is CISCO frame relay DTE
FR SVC disabled, LAPF state down
Broadcast queue 0/64, broadcasts sent/dropped 923/0, interface broadcasts 0
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 15:23:06
Input queue: 1/75/0/0 (size/max/drops/flushes); Total output drops: 10003
Queueing strategy: fifo
Output queue: 0/40 (size/max)
30 second input rate 1521000 bits/sec, 494 packets/sec
30 second output rate 16042000 bits/sec, 3334 packets/sec
35238211 packets input, 2648515031 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
192251637 packets output, 1216186812 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
DSU mode 0, bandwidth 44210, real bandwidth 44210, scramble 0
a.) The DSU line that we have is a 20mb connection, but the default is 44210. The line is in the running config as "dsu bandwidth 44210", but according to Cisco is the default. I think this is primarily for calculations, and that would mean that my "tx load" would be even higher than it shows now.
b.) The tx load is almost at 100% if we take into calculation that it's around 30% now, but it "thinks" it's a 45mb pipe. If we convert that to a 20mb calculation, then the transmit rate would naturally increase to something around 225/255. (Does that make sense?)
c.) The 30 second output is at 16mb, which means that there's 16mb of data going out CONSTANT out of a 20mb connection. This isn't normal for this connection.
d.) Do I need to be concerned with the dropped packets?
Also, when we're talking about output for this interface, is that output respective of the direction? I mean if I'm sending traffic through the router from one side and it's going into my network, does that register as output as well as sending traffic out of my network?
I'm at a loss as to why there's so much traffic, and where it's coming from and I hope you guys can give me some pointers as to what's going on. This is not a new connection either by the way; this just started happening yesterday morning.
Thanks!
John
12-16-2008 04:21 AM
You could put the command bandwidth 20000 to verify your bandwidth use. This is mainly administrative and will be used for bandwidth calculations as well as well as routing protocols.
The output interface is what your network is sending, so it is sourced in your network. If there is a sudden increase you may want to investigate whether there is a sudden increase in demand on a server in your network, otherwise a worm or virus may have managed to find its way onto one or more workstations.
12-16-2008 04:23 AM
We tried to block several different ports that were coming through netflow, but there was no difference. At first we thought that the server group may have been doing upgrades, but that wasn't the case either.
12-16-2008 04:26 AM
If you see no load increase on any of your servers it seems likely a worm or similar has become active. Since these do not always use "known ports" it's difficult to find. If you have stats on workstations in the LAN you could use that to identify any source or sources for this increase in load.
12-16-2008 04:27 AM
Other than netflow, what's another really good way to determine what address is sending major traffic? I've tried ACLs that permit anything through and then changed the logging update threshold, but I just seem to get a ton of crap because of the way the connection is into the cloud.
12-16-2008 04:46 AM
John
While you might try something like IP accounting to categorize and identify the traffic, I think that NetFlow is the optimum tool for this purpose. Are you reading the NetFlow results manually on the router or are you exporting NetFlow to some device that gathers data and reports results over some time period (the better solution)?
Does the code that you are running support the NetFlow top talker functionality? If so that would be the easy way to find what is generating the most traffic.
Your original post talked about bandwidth (which is just administrative) and about dsu bandwidth which could be a controller command and if so is more than just administrative. Could you post the controller/service module (if present) and interface configuration?
HTH
Rick
12-16-2008 04:51 AM
Rick,
I'm using the free version of Solarwinds Netflow to collect. I set up top talkers on the router, but the data isn't consistent with what I'm seeing on the output of the interface. For instance, my top 10 doesn't equal 80% of the bandwidth (all of them are maxing out at a certain kb, but only equates to about 2mb of speed). I don't have a collector that can write to a database for trending.
Here's the controller and interface information:
controller T1 0/0
framing esf
linecode b8zs
channel-group 0 timeslots 1-24 speed 64
!
controller T1 0/1
framing esf
linecode b8zs
channel-group 0 timeslots 1-24 speed 64
!
controller T3 1/0
clock source line
!
interface Serial1/0
description DS3 connection for the GO
no ip address
encapsulation frame-relay IETF
no ip route-cache cef
no ip mroute-cache
load-interval 30
dsu bandwidth 44210
frame-relay lmi-type cisco
interface Serial1/0.115 point-to-point
description DS3 connection for the GO
ip address 172.x.x.1 255.x.x.x
frame-relay interface-dlci 1002
Thanks!
John
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: