WS-C3550-24 output discard problem

phil_carter · ‎04-06-2010

Hello,

We are seeing problems when sending file transfers across a WAN link that terminates on a 3550. Sending a large number of pings across the link simulates the same problem, with upto 3% packet loss at times. The problem is intermittent.

HAVANT_W_CX355_01#ping 10.4.230.117 re 10000

Type escape sequence to abort.
Sending 10000, 100-byte ICMP Echos to 10.4.230.117, timeout is 2 seconds:

Success rate is 99 percent (9983/10000), round-trip min/avg/max = 8/12/44 ms

HAVANT_W_CX355_01#sh interfaces f0/4 counters errors

Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize
Fa0/4 0 0 37247 0 0

However, this problem appears across numerous WAN links that all terminate on the 3550. some of the WAN links have been fully tested to eliminate them from the problem.

The 3550 acts as a Layer-3 Switch running OSPF and IOS: c3550-ipservices-mz[1].122-50.SE.bin

Removing user traffic off the link and running the same tests seems to clear the problem, so it looks as though the switch is not handling the load. However, the utilisation across the links is small:

HAVANT_W_CX355_01#sh controllers utilization
Port       Receive Utilization Transmit Utilization
Fa0/1              1                    5
Fa0/2              2                    0
Fa0/3              1                    1
Fa0/4              37                   48
Fa0/5              0                    0
Fa0/6              0                    0
Fa0/7              0                    0
Fa0/8              0                    0
Fa0/9              0                    0
Fa0/10             0                    0
Fa0/11             0                    5
Fa0/12             0                    0
Fa0/13             0                    0
Fa0/14             0                    0
Fa0/15             0                    0
Fa0/16             0                    0
Fa0/17             16                   50
Fa0/18             0                    0
Fa0/19             0                    0
Fa0/20             0                    0
Fa0/21             0                    0
Fa0/22             0                    0
Fa0/23             0                    0
Fa0/24             0                    0
Gi0/1              5                    3
Gi0/2              0                    0

Total Ports : 26
Switch Receive Bandwidth Percentage Utilization : 0
Switch Transmit Bandwidth Percentage Utilization : 0

Switch Fabric Percentage Utilization : 0

HAVANT_W_CX355_01#sh int f0/4
FastEthernet0/4 is up, line protocol is up (connected)
Hardware is Fast Ethernet, address is 000a.8a1d.d400 (bia 000a.8a1d.d400)
Description: CARDGA_CC375_06 F1/0/24 (IDN106600ETH100)
Internet address is 10.4.230.118/30
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
     reliability 255/255, txload 88/255, rxload 73/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, media type is 10/100BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:03, output 00:00:01, output hang never
Last clearing of "show interface" counters 5d03h
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/0 (size/max)
5 minute input rate 28675000 bits/sec, 22169 packets/sec
5 minute output rate 34818000 bits/sec, 15000 packets/sec
     635595324 packets input, 4042652913 bytes, 0 no buffer
     Received 118415 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 118415 multicast, 0 pause input
     0 input packets with dribble condition detected
     452938874 packets output, 2342123048 bytes, 37270 underruns <<<<<<<<<< underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     37270 output buffer failures, 0 output buffers swapped out    <<<<<<<<<< buffer failures

Any advice welcomed.

Phil

Leo Laohoo · ‎04-06-2010

Can you post the output to the command "sh controller ethernet f0/4"?

phil_carter · ‎04-07-2010

HAVANT_W_CX355_01#sh contr ethernet-controller f0/4

Transmit FastEthernet0/4           Receive
2150026521 Bytes               2021793111 Bytes
160226828 Unicast frames       240612493 Unicast frames
     18094 Multicast frames         15976 Multicast frames
         0 Broadcast frames             0 Broadcast frames
        81 Discarded frames             0 No dest, unicast
         0 Too old frames               0 No dest, multicast
         0 Deferred frames              0 No dest, broadcast
         0 1 collision frames
         0 2 collision frames          0 FCS errors
         0 3 collision frames          0 Oversize frames
         0 4 collision frames          0 Undersize frames
         0 5 collision frames          0 Collision fragments
         0 6 collision frames
         0 7 collision frames   12217534 Minimum size frames
         0 8 collision frames   82924790 65 to 127 byte frames
         0 9 collision frames   88930402 128 to 255 byte frames
         0 10 collision frames    1312173 256 to 511 byte frames
         0 11 collision frames     841586 512 to 1023 byte frames
         0 12 collision frames   54401984 1024 to 1518 byte frames
         0 13 collision frames
         0 14 collision frames          0 Flooded frames
         0 15 collision frames          0 Overrun frames
         0 Excessive collisions         0 VLAN filtered frames
         0 Late collisions              0 Source routed frames
         0 Good (1 coll) frames         0 Valid oversize frames
         0 Good(>1 coll) frames         0 Pause frames
         0 Pause frames                 0 Symbol error frames
         0 VLAN discard frames          0 Invalid frames, too large
         0 Excess defer frames          0 Valid frames, too large
         0 Too large frames             0 Invalid frames, too small
46016428 64 byte frames               0 Valid frames, too small
   9134183 127 byte frames
38024967 255 byte frames
   1640322 511 byte frames
   2853334 1023 byte frames
62582974 1518 byte frames

I cleared this counter yesterday.... so the discarded frames are not excessive currently.... However, it's only the discarded frames that seems to increase.

Regards,

Phil

Leo Laohoo · ‎04-07-2010

Looks like everything to be fine. Yes, there are Discards on the Transmit. What is the BW of your subscription? Do you have anything to shape your traffic?

phil_carter · ‎04-07-2010

Hello,

The links are 100m ports (set 100m Full Duplex). Utilisation never gets above 50%, yet we see discards and performance issues. Outside normal working hours, when user traffic is minimal, the problem goes away.

HAVANT_W_CX355_01#sh contr ether f0/4

Transmit FastEthernet0/4           Receive
4177189702 Bytes                773223015 Bytes
270604689 Unicast frames       387477642 Unicast frames
     20875 Multicast frames         18425 Multicast frames
         0 Broadcast frames             0 Broadcast frames
      1994 Discarded frames             0 No dest, unicast
         0 Too old frames               0 No dest, multicast
         0 Deferred frames              0 No dest, broadcast

HAVANT_W_CX355_01#sh inter f0/4 counter err

Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize
Fa0/4 0 0 1994 0 0

Port Single-Col Multi-Col Late-Col Excess-Col Carri-Sen Runts Giant
s
Fa0/4 0 0 0 0 0 0
0

To make things even more confusing, when sending a file transfer to the remote site, it takes half the time of receiving the same sized file (even though the discards appear to be on the transmit direction only on the 3550!).

No QoS or shaping happening, just plain old FIFO. I thought about using traffic shaping, but if utilisation never gets anywhere near the 100m, it would never use it.

Your thoughts welcomed.

Thanks,

Phil