PortChannel1 overruns ?

Unanswered Question
Jun 1st, 2010

I have a telnet based application where the Agents log into a Server. However the applications is noticing a network communication error and logs the Agents out. Via show mac-address-table I see the agent traffic is crossing a pair of 6500s with a Port Channel bundle of 5 GigE interfaces.

One of the interfaces is having overruns and I can watch the counters increment on overruns.  Are packets being discarded when the overruns occur?

Sup720-33

12.2(33)SXH1

6509-02#show interface po1
Port-channel1 is up, line protocol is up (connected)
  Hardware is EtherChannel, address is 0017.5945.0e8e (bia 0017.5945.0e8e)
  Description: ETHERCHANNEL TO CORE01
  MTU 1500 bytes, BW 5000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 16/255, rxload 15/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is unknown
  input flow-control is off, output flow-control is off
  Members in this channel: Gi1/47 Gi2/44 Gi3/48 Gi4/48 Gi7/48
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output never, output hang never
  Last clearing of "show interface" counters 00:41:35
  Input queue: 0/2000/18/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 310186000 bits/sec, 54843 packets/sec
  5 minute output rate 323428000 bits/sec, 51695 packets/sec
     104391769 packets input, 56682979173 bytes, 0 no buffer
     Received 363725 broadcasts (102097 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 18 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     92400441 packets output, 56229589814 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

6509-02#  show interfaces port-channel 1 capabilities
Port-channel1
  Model:                 NO IDPROM
  Type:                  unknown
  Speed:                 10,100,1000,auto
  Duplex:                half,full
  Trunk encap. type:     802.1Q,ISL
  Trunk mode:            on,off,desirable,nonegotiate
  Channel:               yes
  Broadcast suppression: none
  Flowcontrol:           rx-(off,on,desired),tx-(off,on,desired)
  Membership:            static
  Fast Start:            yes
  QOS scheduling:        rx-(NotDef-t), tx-(NotDef-t)
  QOS queueing mode:     rx-(none), tx-(none)
  CoS rewrite:           yes
  ToS rewrite:           yes
  Inline power:          no
  Inline power policing: no
  SPAN:                  source/destination
  UDLD                   no
  Link Debounce:         no
  Link Debounce Time:    no
  Ports on ASIC:         UNAVAILABLE
  Remote switch uplink:  no
  Dot1x:                 no
  Port-Security:         no

6509-02#show interfaces port-channel 1 etherchannel
Age of the Port-channel   = 766d:08h:47m:06s
Logical slot/port   = 14/1          Number of ports = 5
GC                  = 0x00000000      HotStandBy port = null
Port state          = Port-channel Ag-Inuse
Protocol            =    -
Fast-switchover     = disabled

Ports in the Port-channel:

Index   Load   Port     EC state        No of bits
------+------+------+------------------+-----------
  1     21     Gi1/47   On    2
  0     42     Gi2/44   On    2
  2     84     Gi3/48   On    2
  4     08     Gi4/48   On    1
  3     10     Gi7/48   On    1

Time since last port bundled:    754d:20h:06m:32s    Gi2/44
Time since last port Un-bundled: 754d:20h:12m:29s    Gi2/44

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
Ganesh Hariharan Tue, 06/01/2010 - 22:46

I have a telnet based application where the Agents log into a Server. However the applications is noticing a network communication error and logs the Agents out. Via show mac-address-table I see the agent traffic is crossing a pair of 6500s with a Port Channel bundle of 5 GigE interfaces.

One of the interfaces is having overruns and I can watch the counters increment on overruns.  Are packets being discarded when the overruns occur?

Sup720-33

12.2(33)SXH1

6509-02#show interface po1
Port-channel1 is up, line protocol is up (connected)
  Hardware is EtherChannel, address is 0017.5945.0e8e (bia 0017.5945.0e8e)
  Description: ETHERCHANNEL TO CORE01
  MTU 1500 bytes, BW 5000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 16/255, rxload 15/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is unknown
  input flow-control is off, output flow-control is off
  Members in this channel: Gi1/47 Gi2/44 Gi3/48 Gi4/48 Gi7/48
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output never, output hang never
  Last clearing of "show interface" counters 00:41:35
  Input queue: 0/2000/18/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 310186000 bits/sec, 54843 packets/sec
  5 minute output rate 323428000 bits/sec, 51695 packets/sec
     104391769 packets input, 56682979173 bytes, 0 no buffer
     Received 363725 broadcasts (102097 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 18 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     92400441 packets output, 56229589814 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

6509-02#  show interfaces port-channel 1 capabilities

Port-channel1

  Model:                 NO IDPROM

  Type:                  unknown

  Speed:                 10,100,1000,auto

  Duplex:                half,full

  Trunk encap. type:     802.1Q,ISL

  Trunk mode:            on,off,desirable,nonegotiate

  Channel:               yes

  Broadcast suppression: none

  Flowcontrol:           rx-(off,on,desired),tx-(off,on,desired)

  Membership:            static

  Fast Start:            yes

  QOS scheduling:        rx-(NotDef-t), tx-(NotDef-t)

  QOS queueing mode:     rx-(none), tx-(none)

  CoS rewrite:           yes

Hi,

Overrun defination means The number of times the receiver                     hardware was unable to hand received data to a hardware buffer and common cause would be the input rate of traffic                     exceeded the ability of the receiver to handle the data.

Check out the below link for troubleshooting ethernet inetrfaces issue

http://www.cisco.com/en/US/products/hw/switches/ps700/products_tech_note09186a008015bfd6.shtml

Hope to Help !!

Ganesh.H

Jason Aarons Wed, 06/02/2010 - 04:16

Sorry, was looking for someone who might have seen this before and wondering what they did..

Customer is using WS-X6148-GE cards that have a 8:1 oversubscription with only a 1MB buffer for 8 port blocks, My hypothesis is that these cards are the problem as qos3Outlost are incrementing.

6509-02# show counters int gig2/44 | include qos3Outlos

53.                        qos3Outlost = 114195974

Connectivity Problem or Packet Loss with WS-X6548-GE-TX and WS-X6148-GE-TX Modules used in a Server Farm

When you use either the WS-X6548-GE-TX or WS-X6148-GE-TX modules, there is a possibility that individual port utilization can lead to connectivity problems or packet loss on the surrounding interfaces. Especially when you use EtherChannel and Remote Switched Port Analyzer (RSPAN) in these line cards, you can potentially see the slow response due to packet loss. These line cards are oversubscription cards that are designed to extend gigabit to the desktop and might not be ideal for server farm connectivity. On these modules there is a single 1-Gigabit Ethernet uplink from the port ASIC that supports eight ports. These cards share a 1 Mb buffer between a group of ports (1-8, 9-16, 17-24, 25-32, 33-40, and 41-48) since each block of eight ports is 8:1 oversubscribed

Giuseppe Larosa Wed, 06/02/2010 - 07:32

Hello Jason,

I agree it is likely a problem of excessive use of the 8 ports group

to confirm this you can look for overruns or other errors in the other  seven ports of the same group.

I did so for WS-4548 on C4500 that has the same problem, in my case I could see increasing output drops on the ports served by the same ASIC chip as a confirmation of performance issues.

Also if you have a 5 Gbps channel the load sharing algorithm may use one link much more then the other members so I would look at use of ports in the bundle with sh etherchannel

Hope to help

Giuseppe

Actions

This Discussion