Solved: overrun errors on gigabit switch port

mattcisconet · ‎12-16-2005

Can somebody please explain to me what overrun errors on an interface are and what may cause them? Thanks.

5 minute input rate 3542000 bits/sec, 714 packets/sec

5 minute output rate 1865000 bits/sec, 694 packets/sec

9077339068 packets input, 9161744841831 bytes, 0 no buffer

Received 48704918 broadcasts (46547259 multicast)

0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 139716 overrun, 0 ignored

0 watchdog, 0 multicast, 0 pause input

0 input packets with dribble condition detected

7612190579 packets output, 1859866741694 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier, 0 PAUSE output

0 output buffer failures, 0 output buffers swapped out

Bobby Thekkekandam · ‎12-16-2005

The WS-X6548-GE-TX is what is known as a "blocking" card, meaning that the ports are oversubscribed, for this particular card, at a rate of 8:1. In other words, every block of 8 ports shares 1Gb of bandwidth.

Also, this card shares a 1Mb buffer between groups ports (1-8, 9-16, 17-24, 25-32, 33-40, 41-48) since each block of eight ports is 8:1 oversubscribed. If any port in this range is receiving or transmitting traffic at a rate that exceeds its bandwidth or utilizing a large amount of buffers to handle bursts of traffic, the other ports in the same range of 8 may experience packet loss.

There are a couple of ways to address this.

1)Isolate any ports that may be consistently oversubscribed (SPAN destinations, servers with NFS, slower speed than the other ports in the range, etc.) to their own range of 8 ports to minimize the impact of drops to other interfaces.

2)Disable head of line blocking which will utilize the interface buffers instead of the shared buffers. This will result in only the single over utilized port having drops, but since the interface buffers (32k) are significantly smaller than the 1Mb shared buffer, there may be more lost traffic to individual ports. This is only recommended for extreme cases where slower clients or span ports cannot be moved to other line cards that offer dedicated interface buffers

6500(config)#service internal

6500(config)#interface gigabit 1/1

6500(config-if)#hol-blocking disable

%HOL Blocking is Disabled on: Gi1/1 Gi1/2 Gi1/3 Gi1/4 Gi1/5 Gi1/6 Gi1/7 Gi1/8

6500#show hol-blocking module 1

Interface Hol-Blocking

-------------- ---------------

Gi1/1 Disable

Gi1/2 Disable

Gi1/3 Disable

Gi1/4 Disable

Gi1/5 Disable

Gi1/6 Disable

Gi1/7 Disable

Gi1/8 Disable

Once this is disabled, the drops will move to the interface counters and can be seen with 'show interface gigabit '. The other ports will no longer be affected provided that they are not individually bursting also. Since it is recommended to keep hol-blocking enabled, this information can be used to find the device that is overrunning the buffers on the range of port and move it to another card or an isolated range on the card so hol-blocking can be re-enabled.

6500#show interface gigabit 1/1

Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 25542147 <-----------

Additionally, this document may provide some more insight.

Buffers, Queues & Thresholds on Catalyst 6500 Ethernet Modules

http://www.cisco.com/en/US/products/hw/switches/ps708/products_white_paper09186a0080131086.shtml

HTH,

Bobby

View solution in original post

Bobby Thekkekandam · ‎12-16-2005

Overruns are the number of times the receiver hardware was unable to hand received data to a hardware buffer.

This is generally caused by the input rate of traffic exceeding the receiver's ability to handle the data.

What type of module is this? More than likely this is a blocking card who is receiving more data than can be put into the shared buffer. Depending on the type of card, we may be able to mitigate this by spreading the top talkers among different port groups.

This is documented here:

http://www.cisco.com/warp/public/473/53.shtml#overrun

HTH,

Bobby

mattcisconet · ‎12-16-2005

Thanks Bobby,

I'm surprised a Gigabit interface would ever receive more data than it's buffer can process, however I'm no expert. Wouldn't this mean the input rate would have to exceed 1000Mb/s?

Here are my module details:

auto-neg to Full-duplex, 1000Mb/s

!

interface GigabitEthernet4/29

logging event link-status

switchport

switchport access vlan 101

switchport mode access

no cdp enable

spanning-tree portfast

end

Mod Ports Card Type Model Serial No.

--- ----- -------------------------------------- ------------------ -----------

4 48 SFM-capable 48 port 10/100/1000mb RJ45 WS-X6548-GE-TX SAD0835026Z

Mod MAC addresses Hw Fw Sw Status

--- ---------------------------------- ------ ------------ ------------ -------

4 0001.c9df.b802 to 0001.c9df.b831 10.1 7.2(1) 8.3(2.10)TFW Ok

Mod Online Diag Status

--- -------------------

4 Pass

Bobby Thekkekandam · ‎12-16-2005

The WS-X6548-GE-TX is what is known as a "blocking" card, meaning that the ports are oversubscribed, for this particular card, at a rate of 8:1. In other words, every block of 8 ports shares 1Gb of bandwidth.

Also, this card shares a 1Mb buffer between groups ports (1-8, 9-16, 17-24, 25-32, 33-40, 41-48) since each block of eight ports is 8:1 oversubscribed. If any port in this range is receiving or transmitting traffic at a rate that exceeds its bandwidth or utilizing a large amount of buffers to handle bursts of traffic, the other ports in the same range of 8 may experience packet loss.

There are a couple of ways to address this.

1)Isolate any ports that may be consistently oversubscribed (SPAN destinations, servers with NFS, slower speed than the other ports in the range, etc.) to their own range of 8 ports to minimize the impact of drops to other interfaces.

2)Disable head of line blocking which will utilize the interface buffers instead of the shared buffers. This will result in only the single over utilized port having drops, but since the interface buffers (32k) are significantly smaller than the 1Mb shared buffer, there may be more lost traffic to individual ports. This is only recommended for extreme cases where slower clients or span ports cannot be moved to other line cards that offer dedicated interface buffers

6500(config)#service internal

6500(config)#interface gigabit 1/1

6500(config-if)#hol-blocking disable

%HOL Blocking is Disabled on: Gi1/1 Gi1/2 Gi1/3 Gi1/4 Gi1/5 Gi1/6 Gi1/7 Gi1/8

6500#show hol-blocking module 1

Interface Hol-Blocking

-------------- ---------------

Gi1/1 Disable

Gi1/2 Disable

Gi1/3 Disable

Gi1/4 Disable

Gi1/5 Disable

Gi1/6 Disable

Gi1/7 Disable

Gi1/8 Disable

Once this is disabled, the drops will move to the interface counters and can be seen with 'show interface gigabit '. The other ports will no longer be affected provided that they are not individually bursting also. Since it is recommended to keep hol-blocking enabled, this information can be used to find the device that is overrunning the buffers on the range of port and move it to another card or an isolated range on the card so hol-blocking can be re-enabled.

6500#show interface gigabit 1/1

Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 25542147 <-----------

Additionally, this document may provide some more insight.

Buffers, Queues & Thresholds on Catalyst 6500 Ethernet Modules

http://www.cisco.com/en/US/products/hw/switches/ps708/products_white_paper09186a0080131086.shtml

HTH,

Bobby

mattcisconet · ‎12-16-2005

bobby,

Thanks very much for your extremely informative response.

I do have 1 more question. How would overruns manifest themselves on the application layer? Would users experience slowness? TCP Re-transmissions?

Bobby Thekkekandam · ‎12-16-2005

Slowness and TCP retransmissions are very likely scenarios that would correlate on the upper layers as a result of this, depending of course, on what type of traffic is being dropped.

mattcisconet · ‎12-16-2005

Thanks again Bobby. Wow you are a wealth of knowledge :)

Wouldn't I see output drops on my interface is packets were in fact being dropped as a result of the overruns? Or do I assume that because I'm seeing overruns there is packet loss?

Bobby Thekkekandam · ‎12-16-2005

If head-of-line blocking is disabled, you'll use the port level buffers, and then you would see the drops on the interface (example in my longer post above).

However, you can assume that there is packet loss if you are seeing overruns to begin with.

-Bobby