PortChannel1 overruns ?

Jason Aarons · ‎06-01-2010

I have a telnet based application where the Agents log into a Server. However the applications is noticing a network communication error and logs the Agents out. Via show mac-address-table I see the agent traffic is crossing a pair of 6500s with a Port Channel bundle of 5 GigE interfaces.

One of the interfaces is having overruns and I can watch the counters increment on overruns. Are packets being discarded when the overruns occur?

Sup720-33

12.2(33)SXH1

6509-02#show interface po1
Port-channel1 is up, line protocol is up (connected)
Hardware is EtherChannel, address is 0017.5945.0e8e (bia 0017.5945.0e8e)
Description: ETHERCHANNEL TO CORE01
MTU 1500 bytes, BW 5000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 16/255, rxload 15/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is unknown
input flow-control is off, output flow-control is off
Members in this channel: Gi1/47 Gi2/44 Gi3/48 Gi4/48 Gi7/48
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output never, output hang never
Last clearing of "show interface" counters 00:41:35
Input queue: 0/2000/18/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 310186000 bits/sec, 54843 packets/sec
5 minute output rate 323428000 bits/sec, 51695 packets/sec
     104391769 packets input, 56682979173 bytes, 0 no buffer
     Received 363725 broadcasts (102097 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 18 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     92400441 packets output, 56229589814 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

6509-02# show interfaces port-channel 1 capabilities
Port-channel1
Model:                 NO IDPROM
Type:                  unknown
Speed:                 10,100,1000,auto
Duplex:                half,full
Trunk encap. type:     802.1Q,ISL
Trunk mode:            on,off,desirable,nonegotiate
Channel:               yes
Broadcast suppression: none
Flowcontrol:           rx-(off,on,desired),tx-(off,on,desired)
Membership:            static
Fast Start:            yes
QOS scheduling:        rx-(NotDef-t), tx-(NotDef-t)
QOS queueing mode:     rx-(none), tx-(none)
CoS rewrite:           yes
ToS rewrite:           yes
Inline power:          no
Inline power policing: no
SPAN:                  source/destination
UDLD                   no
Link Debounce:         no
Link Debounce Time:    no
Ports on ASIC:         UNAVAILABLE
Remote switch uplink: no
Dot1x:                 no
Port-Security:         no

6509-02#show interfaces port-channel 1 etherchannel
Age of the Port-channel   = 766d:08h:47m:06s
Logical slot/port   = 14/1          Number of ports = 5
GC                  = 0x00000000      HotStandBy port = null
Port state          = Port-channel Ag-Inuse
Protocol            =    -
Fast-switchover     = disabled

Ports in the Port-channel:

Index   Load   Port     EC state        No of bits
------+------+------+------------------+-----------
1     21     Gi1/47   On    2
0     42     Gi2/44   On    2
2     84     Gi3/48   On    2
4     08     Gi4/48   On    1
3     10     Gi7/48   On    1

Time since last port bundled: 754d:20h:06m:32s Gi2/44
Time since last port Un-bundled: 754d:20h:12m:29s Gi2/44

Ganesh Hariharan · ‎06-01-2010

I have a telnet based application where the Agents log into a Server. However the applications is noticing a network communication error and logs the Agents out. Via show mac-address-table I see the agent traffic is crossing a pair of 6500s with a Port Channel bundle of 5 GigE interfaces.

One of the interfaces is having overruns and I can watch the counters increment on overruns. Are packets being discarded when the overruns occur?

Sup720-33

12.2(33)SXH1

6509-02#show interface po1
Port-channel1 is up, line protocol is up (connected)
Hardware is EtherChannel, address is 0017.5945.0e8e (bia 0017.5945.0e8e)
Description: ETHERCHANNEL TO CORE01
MTU 1500 bytes, BW 5000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 16/255, rxload 15/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is unknown
input flow-control is off, output flow-control is off
Members in this channel: Gi1/47 Gi2/44 Gi3/48 Gi4/48 Gi7/48
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output never, output hang never
Last clearing of "show interface" counters 00:41:35
Input queue: 0/2000/18/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 310186000 bits/sec, 54843 packets/sec
5 minute output rate 323428000 bits/sec, 51695 packets/sec
     104391769 packets input, 56682979173 bytes, 0 no buffer
     Received 363725 broadcasts (102097 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 18 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     92400441 packets output, 56229589814 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

6509-02# show interfaces port-channel 1 capabilities

Port-channel1

Model: NO IDPROM

Type: unknown

Speed: 10,100,1000,auto

Duplex: half,full

Trunk encap. type: 802.1Q,ISL

Trunk mode: on,off,desirable,nonegotiate

Channel: yes

Broadcast suppression: none

Flowcontrol: rx-(off,on,desired),tx-(off,on,desired)

Membership: static

Fast Start: yes

QOS scheduling: rx-(NotDef-t), tx-(NotDef-t)

QOS queueing mode: rx-(none), tx-(none)

CoS rewrite: yes

Hi,

Overrun defination means The number of times the receiver hardware was unable to hand received data to a hardware buffer and common cause would be the input rate of traffic exceeded the ability of the receiver to handle the data.

Check out the below link for troubleshooting ethernet inetrfaces issue

http://www.cisco.com/en/US/products/hw/switches/ps700/products_tech_note09186a008015bfd6.shtml

Hope to Help !!

Ganesh.H

Jason Aarons · ‎06-02-2010

Sorry, was looking for someone who might have seen this before and wondering what they did..

Customer is using WS-X6148-GE cards that have a 8:1 oversubscription with only a 1MB buffer for 8 port blocks, My hypothesis is that these cards are the problem as qos3Outlost are incrementing.

6509-02# show counters int gig2/44 | include qos3Outlos

53. qos3Outlost = 114195974

Connectivity Problem or Packet Loss with WS-X6548-GE-TX and WS-X6148-GE-TX Modules used in a Server Farm

When you use either the WS-X6548-GE-TX or WS-X6148-GE-TX modules, there is a possibility that individual port utilization can lead to connectivity problems or packet loss on the surrounding interfaces. Especially when you use EtherChannel and Remote Switched Port Analyzer (RSPAN) in these line cards, you can potentially see the slow response due to packet loss. These line cards are oversubscription cards that are designed to extend gigabit to the desktop and might not be ideal for server farm connectivity. On these modules there is a single 1-Gigabit Ethernet uplink from the port ASIC that supports eight ports. These cards share a 1 Mb buffer between a group of ports (1-8, 9-16, 17-24, 25-32, 33-40, and 41-48) since each block of eight ports is 8:1 oversubscribed

Giuseppe Larosa · ‎06-02-2010

Hello Jason,

I agree it is likely a problem of excessive use of the 8 ports group

to confirm this you can look for overruns or other errors in the other seven ports of the same group.

I did so for WS-4548 on C4500 that has the same problem, in my case I could see increasing output drops on the ports served by the same ASIC chip as a confirmation of performance issues.

Also if you have a 5 Gbps channel the load sharing algorithm may use one link much more then the other members so I would look at use of ports in the bundle with sh etherchannel

Hope to help

Giuseppe