07-24-2009 11:35 AM - edited 03-06-2019 06:56 AM
I need some help to determine where packets are being dropped in the network for a new QoS configuration I've rolled out.
My test setup is running iperf from two source systems to a single target system. The two data flows are labelled at different priority levels and sure enough, my lower priority traffic gets dropped as expected .. so all is looking good. What I can't tell is where the data is getting dropped.
The data route traverses a 6509 (IOS), 4507 (IOS), 3750 all with QoS enabled, queueing setup and appropriate service policies or trust state applied.
I've looked at all the counters in the route and I can see that my traffic being dropped on the final leg by the queueing, so all good once again.
Cat4507 command - show interface counter mod x
Cat 6500 command - show counters interface x/x
Cat3750 commands - show platform pm if-numbers
show platform port-asic stats drop port x asic y
However, one thing that came to light is the mismatch of tx rate at one end of a port channel to the rx rate of the other end. Anybody got any idea why the two don't look the same? I cleared the counters at both end and left it running for 10+ minutes so expected the 5 minute rates to be the same:
CAT 6509
5 minute input rate 3722000 bits/sec, 4271 packets/sec
5 minute output rate 127230000 bits/sec, 11332 packets/sec
CAT 4507
5 minute input rate 96120000 bits/sec, 8553 packets/sec
5 minute output rate 2544000 bits/sec, 3263 packets/sec
How come I'm missing 30 odd Mbps of rx traffic on the 4507r? Are the algortithms different for calculating the rx rate on the platforms or something?
Thanks
Jed
07-26-2009 02:35 PM
If nobody can help with the above, how about this related question.
I can see packets are getting dropped from Q2 on my C3750 (12.2(20)SE4) whenever I try to load the interface even lightly.
FastEthernet1/0/26 is up, line protocol is up (connected)
Hardware is Fast Ethernet, address is 0013.6009.009e (bia 0013.6009.009e)
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 17/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, media type is 10/100BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:49, output 00:00:01, output hang never
Last clearing of "show interface" counters 00:38:45
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 172000 bits/sec, 318 packets/sec
5 minute output rate 6996000 bits/sec, 619 packets/sec
792794 packets input, 56452222 bytes, 0 no buffer
Received 276 broadcasts (0 multicast)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 61 multicast, 0 pause input
0 input packets with dribble condition detected
1566924 packets output, 2212696242 bytes, 0 underruns
0 output errors, 0 collisions, 2 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out
sw01#sho mls qos int fa1/0/26 queu
FastEthernet1/0/26
Egress Priority Queue : enabled
Shaped queue weights (absolute) : 3 0 0 0
Shared queue weights : 1 70 25 5
The port bandwidth limit : 100
The port is mapped to qset : 1
sw01#sho mls qos int fa1/0/26 buffer
FastEthernet1/0/26
The port is mapped to qset : 1
The allocations between the queues are : 25 25 25 25
sw01#show platform port-asic stats drop port 3
...
Port 3 TxQueue Drop Statistics
Queue 0
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 1
Weight 0 Frames 347
Weight 1 Frames 0
Weight 2 Frames 0
Queue 2
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 29111
Queue 3
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 6527492
29111 DSCP 0 packets have been dropped, yet the Tx rate of the port is well below capacity so why is it I'm seeing drops at this low utilisation level. I'm getting an actual HTTP download of about 700kbps for this workstation, yet without the QoS it can push higher.
It seems that the buffering kicks in way before the link is heavily utilised which wasn't what I was expecting. Before I applied any of the QoS settings (derived from the SRND) we weren't dropping packets and throughput was much better.
Why am I seeing the buffering kick in so early?
Thanks
Jed
07-27-2009 12:23 AM
I've found a very similar post stream under http://forums.cisco.com/eforum/servlet/NetProf?page=netprof&forum=Network%20Infrastructure&topic=LAN%2C%20Switching%20and%20Routing&topicID=.ee71a04&fromOutline=&CommCmd=MB%3Fcmd%3Ddisplay_location%26location%3D.2cd40f09 so I'll transfer my focus to that stream.
Jed
07-27-2009 04:01 AM
"Why am I seeing the buffering kick in so early? "
I suspect it might just be a poor or inadequate allocation of port resources when QoS is enabled. If just "poor", a later IOS might correct the issue as might manual QoS tuning.
As to manual tuning, the documentation I've seen, to me, is weak explaining both the amount of buffers actually available (I have in mind % references) and the optimal interplay between reserved and common pool buffers.
A high drop rate with low utilization often indicates insufficient egress queue capacity to handle expected traffic rate bursting. What's interesting about your posted stats, the W2 drops in queues 2 and 3, since this, I believe, indicates tail drop for the total size of the queue.
What you might try, since you have:
Shared queue weights : 1 70 25 5
but
The allocations between the queues are : 25 25 25 25
Try setting buffer allocations (somewhat) the inverse of your queue bandwidth allocations.
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: