Leo I have pasted the physical interface stats, FastEthernet1/36, look at the original mesg.

thanks, p

Output discards on a FastEthernet port???  What client is connected to this port?  

yes on fast ether port. set for full/100. Right now there is a cisco catalyst set for full/100 with no errors. its an L2 switch with about 5 linux smtp servers.

I just can't figure out what is causing the output drops on the 6500 side.

paul

The 6548 line cards are no match to the Linux servers.  The 6548 line cards are not meant to "live" or "do" data centre work.  

so what do you think is happening, because we have other servers/switches connected to the 6548 line card that sees no output drops.

also the interface is connected to another L2 switch which sees no output drops. The servers are not connected directly to the 6548 linecard.

What I think is happening is each of the Linux server is overwhelming the notoriously shallow memory buffer of the 6548 line cards. 

The solution to this are: 

1.  Apply QoS; or

2.  6748 line cards.

Unfortunately, QoS is not my strong suits.  One of the Cisco VIPs, Joseph, is good at it and if he's not busy he normally lurks around and he'll chime in.

is there a way to see memory buffers on those cards? also shouldn't i see drops across all ports on that card? I'm only seeing it for some ports, the busiest ports.

The other thing is the L2 switch connected to it is an old cisco catalyst 2900 and there are no output drops. Could that line card be really that bad ?

You can use show buffers input-interface -interface x/x- packet/header command. however, you may be hitting this bug if you are only seeing the drops increase but there is no actual network degradation:

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCdz02952/?reffering_site=dumpcr

Hope it helps, best regards!

JC

The buffer command will not show any packets buffer info, i have tried that. Also the line card i have runs software 12.2 so its not affected by that bug. It's just really strange to have the output counter increment for now reason, well none that i can see.

Leo, i have done research am im not convinced that the issue is what your described above and its a problem with the hardware queue, thus also why setting the software queue didnt make a difference. I have mls qos enabled and its using the hardware queues on the linecards.

my question is the 6548 shows 1088KB for a TX queue buffer while the 6748 shows 1.2MB do you think that ~ and extra 300KB will make a difference ?

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages wha2tsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

and extra 300KB will make a difference ?

Possibly, although a bigger difference using a 6748 might be being running at gig rather that 100.  (Does your 2900 support gig on a 2900 uplink port?)

Generally, unless you're bumping into some kind of bug, Cisco device queues drop packets when they overflow.  When overall utilization appears low, microbursts are a common cause, although you say you believe this isn't happening.  (How did you obtain the packet dump you used?  If you used Cisco's SPAN, I'm unsure its replication would be accurate enough to represent what the egress port hardware is actually "seeing" frame/packet timing wise.)

As you mention you're using MLS QoS, it's possible, some tuning of the interface's QoS settings (e.g. wrr-queue queue-limit and/or wrr-queue bandwidth) might remediate your drops.  (I don't recall how a 6548 shares buffers between its queues.)

also shouldn't i see drops across all ports on that card? I'm only seeing it for some ports, the busiest ports.

Since on this particular card, buffers are allocated per port, what you state makes sense, i.e. you only see drops on the busiest ports.

Joseph, it was set to 1000 using an HP switch before i switched it to the 2900 at 100. The thing is i never seeing input errors on the 2900 or HP switch.  Do you think going with the WS-X6748-GE-TX would make a difference, im not 100% sure.

I used span and wireshark IO graph to look at the bandwidth used in real time. I was seeing output drops at 3 megs/5 megs and never saw any indication of microbursts. although I'm assuming I should be seeing spikes near the 100Meg limit which im not seeing anything near that. 

I tweaked the WRR queue bandwidth, dont have queue limit option,  several times with no affect, although i have mls qos enabled and dscp trust on that port most packets destined for this port are smtp traffic with low level qos marking and the packets never take advantage of the priority queue and are dropped on the 1st queue.

  Packets dropped on Transmit:
    BPDU packets:  0
    que thr            dropped          30-s bytes          peak bytes 5-mins avg bps   peak bps  [co
s-map]
    -------------------------------------------------------------------------------------------------
-----

    1    1                1256                   0                   0            0            0   [0
 1 ]

thanks, paul

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages wha2tsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Joseph, it was set to 1000 using an HP switch before i switched it to the 2900 at 100. The thing is i never seeing input errors on the 2900 or HP switch.  Do you think going with the WS-X6748-GE-TX would make a difference, im not 100% sure.

I wouldn't expect egress drops to cause input error unless the receiving device couldn't deal with 100 Mbps.

Again, a 6748 might help, because of the extra 300K, because if you can run at gig rather than 100, because it should support the queue-limit command too.

What you might try, if you don't have a 6748 port at hand, is a sup port.  The sup720's ports aren't the best for uplinks, but it might be interesting to see if your results vary.

I used span and wireshark IO graph to look at the bandwidth used in real time. I was seeing output drops at 3 megs/5 megs and never saw any indication of microbursts. although I'm assuming I should be seeing spikes near the 100Meg limit which im not seeing anything near that. 

Again, unsure SPAN doesn't distort timings.  What port were you SPANing, the egress port?

Could more than one port be sending traffic to the egress port?  If so, consider just two concurrent 100 Mbps streams would be sending 200 Mbps of traffic to a 100 Mbps port.  If that happens, the question is whether the concurrent traffic will exceed the allocated buffer space.  If the whole MB of buffer space was available, one would expect it to often be ample, but if the interface reserves it for egress queues, and if you cannot adjust them, then you're more likely to overflow an individual queue's buffers.