WS-C6716-10G-3C overrun

[odamis]

Hi,

we built up a test environment with 2 Cat6500 (VS-S720 and WS-X6716 Linecard). The 2 Cat6500 works as VSS. The ports Te1/1/5, Te1/1/9, Te2/1/5 and 2/1/9 working in performance mode (oversubscription disabled) and forms a portchannel. The Portchannel is connected to 2 N5k in VPC mode. On both N5k is a Fluke OptiviewXG connected.

If I run a 10G throughput test, the ports on the Cat6500 increases the overrun counter. With small packets i get only about 70% of 10G. 

The traffic flow is from N5k to Te2/1/5, then via Te2/1/9 back to the other N5k. The second Fluke generates also a 10G Dataflow. The traffic is bi-directional and has a total bandwith of 20G.

The Linecard can forward 10G per ASIC. The DFC can forward 48Mpps. With only small packets I calculate about 24Gbps (64byte = 512bit, 512bit*48Mpps = 24,5 Gbps).

Q: Why I get only 7Gbps with small packets? Where is my fault?

The Test report of the Fluke Throughput test.

Size	Local Tx	Remote Rx	% Remote Rx	Remote Tx	Local Rx	% Local Rx
1518	10,00 Gbps	9,860 Gbps	98,6%	10,00 Gbps	9,859 Gbps	98,6%
1280	10,00 Gbps	9,849 Gbps	98,5%	10,00 Gbps	9,849 Gbps	98,5%
1024	10,00 Gbps	9,812 Gbps	98,1%	10,00 Gbps	9,812 Gbps	98,1%
512	10,00 Gbps	9,638 Gbps	96,4%	10,00 Gbps	9,638 Gbps	96,4%
256	10,00 Gbps	9,325 Gbps	93,3%	10,00 Gbps	9,325 Gbps	93,2%
128	10,00 Gbps	8,810 Gbps	88,1%	10,00 Gbps	8,810 Gbps	88,1%
64	10,00 Gbps	7,197 Gbps	72%	10,00 Gbps	7,197 Gbps	72%

regards

Olaf

[pille1234]

Hi odamis,

I believe the given output by Fluke is rather misleading. Check this http://www.cisco.com/web/about/security/intelligence/network_performance_metrics.html

The maximum packet rate for 10G Ethernet is 14.880.960 packets/s with a packet size of 64 byte this results in 952381440 byte/s = 7,62Gbit/s. This is the max throughput that could be serializied with that packet size, so I doubt the Fluke is actually transmitting more than that number. It would be more interesting to see the actual counters of forwarded packets. You could check the interface stats on the Cat6500 and N5k during the test.

However you shouldn't see any overruns anyway, as the switch is supposed to forward at line rate, so there might still be a problem somewhere.

I was able to witness the 14.8 mio packets with a WS-C6708 linecard without any overruns and I assume the 6716 is supposed to behave in the same way.

Regards

Pille

[odamis]

Hi Pille,

you're right, 14,88 Mpps is the maximum packet rate per second on the wire. But this results in the calculation of 10.000 Mbps with a minimum frame size of 64 bytes + 20 bytes overhead (64 bytes + SFD + Preemble + IFG = 84 Bytes on the wire). [10.000 Mbps / 672 bits = 14,88 Mpps]

The first Interface where the Fluke is connected (eth1/9) receives 14,88 Mpps. The Interface to Cat6500 (eth1/8) sends also 14,88 Mpps. So the N5k is forwarding 10Gbps with 64 byte frames (14,88 Mpps) with no drops. The incomming interface of the Cat6500 receives 14,88 Mpps and drops a few (6M packets) of them. So I receive only 11,1 Mpps, what leads to a bandwidth of only 7,5Gbps.

I tested the same scenario with only the N5k and got no drops. With the C6k5 I got about 25% packet loss.

Is there a known problem with C6k5 and small packets?

regards
Olaf

The sh int output for the test with N5k and C6k5 in a 2 second period.

Test-N5k-1# sh int eth 1/9
Ethernet1/9 is up
Dedicated Interface
Hardware: 1000/10000 Ethernet, address: 547f.ee62.5010 (bia 547f.ee62.5010)
Description: Fluke_10G
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 2/255
Encapsulation ARPA
Port mode is access
full-duplex, 10 Gb/s, media type is 10G
Beacon is turned off
Input flow-control is off, output flow-control is off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
Last link flapped 3d17h
Last clearing of "show interface" counters 00:00:55
30 seconds input rate 510370952 bits/sec, 996812 packets/sec
30 seconds output rate 381473560 bits/sec, 745060 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 85.00 Mbps, 165.99 Kpps; output rate 63.53 Mbps, 124.07 Kpps
RX
29761918 unicast packets 0 multicast packets 1 broadcast packets
29761919 input packets 1904763877 bytes
0 jumbo packets 0 storm suppression bytes
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
22245327 unicast packets 42 multicast packets 1 broadcast packets
22245370 output packets 1423705679 bytes
0 jumbo packets
0 output errors 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause
0 interface resets


Test-N5k-1# sh int eth 1/8
Ethernet1/8 is up
Dedicated Interface
Belongs to Po5
Hardware: 1000/10000 Ethernet, address: 547f.ee62.500f (bia 547f.ee62.500f)
Description: TEST_65-02
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
reliability 255/255, txload 2/255, rxload 1/255
Encapsulation ARPA
Port mode is trunk
full-duplex, 10 Gb/s, media type is 10G
Beacon is turned off
Input flow-control is off, output flow-control is off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
Last link flapped 2d16h
Last clearing of "show interface" counters 00:00:57
30 seconds input rate 9968 bits/sec, 10 packets/sec
30 seconds output rate 9128 bits/sec, 9 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 63.14 Mbps, 116.05 Kpps; output rate 80.63 Mbps, 148.19 Kpps
RX
22251771 unicast packets 18 multicast packets 0 broadcast packets
22251789 input packets 1513148903 bytes
0 jumbo packets 0 storm suppression bytes
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
28413888 unicast packets 9 multicast packets 1 broadcast packets
28413898 output packets 1932169715 bytes
0 jumbo packets
0 output errors 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause
0 interface resets


TEST_C6500_R.161#sh int te 2/1/5
TenGigabitEthernet2/1/5 is up, line protocol is up (connected)
Hardware is C6k 10000Mb 802.3, address is 0025.84f0.c4bc (bia 0025.84f0.c4bc)
Description: Test-N5k-01
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 10Gb/s, media type is 10Gbase-SR
input flow-control is on, output flow-control is off
Clock mode is auto
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:16, output 00:00:39, output hang never
Last clearing of "show interface" counters 00:02:06
Input queue: 0/2000/6162176/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 49897000 bits/sec, 91725 packets/sec
5 minute output rate 39075000 bits/sec, 71832 packets/sec
28414494 packets input, 1932234173 bytes, 0 no buffer
Received 20 broadcasts (18 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 6162176 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
22252451 packets output, 1513220487 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out