We are using Transit vpc cloudformation template with ax lincenseincluded image for CSRs. We started to noticed output packet drops when traffic is slightly over 2Gbps (in+out).
The CSR showed extremely high-throughput setting of 200000000kbps, which means it shouldn't be rate limited by license. Since we use c4.8xlarge instance type, we expect performance of 4.5Gbps. Has anyone had experienced this? The drop is tail drop. And csr data plane showed credit error for RX. Any suggestions on how to fix this?
PS. while this image is licenseincluded, cisco tech can't find the serial number (show license detailed showed no license installed) and couldn't provide TAC support. Does this subscription license include TAC support or should we buy it separately from somewhere else?
ip-xxxxxxx# show platform hardware throughput level
The current throughput level is 200000000 kb/s
ip-xxxxxxx#show int gi1
GigabitEthernet1 is up, line protocol is up
Hardware is CSR vNIC, address is 06ff.0f57.d3cc (bia 06ff.0f57.d3cc)
Internet address is 10.128.6.118/25
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 255/255, rxload 255/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full Duplex, 1000Mbps, link type is auto, media type is RJ45
output flow-control is unsupported, input flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:30, output 00:00:30, output hang never
Last clearing of "show interface" counters never
Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 570020659
Queueing strategy: fifo
Output queue: 0/40 (size/max)
30 second input rate 1104644000 bits/sec, 110865 packets/sec
30 second output rate 1053357000 bits/sec, 105471 packets/sec
458391357560 packets input, 529164908101076 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
457842026058 packets output, 526861603555761 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
ip-10-128-6-118#show int gi1
GigabitEthernet1 is up, line protocol is up
Hardware is CSR vNIC, address is 06ff.0f57.d3cc (bia 06ff.0f57.d3cc)
Internet address is 10.128.6.118/25
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 255/255, rxload 255/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full Duplex, 1000Mbps, link type is auto, media type is RJ45
output flow-control is unsupported, input flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:32, output 00:00:32, output hang never
Last clearing of "show interface" counters never
Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 570072490
Queueing strategy: fifo
Output queue: 0/40 (size/max)
30 second input rate 1104644000 bits/sec, 110865 packets/sec
30 second output rate 1053357000 bits/sec, 105471 packets/sec
458391357560 packets input, 529164908101076 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
457842026058 packets output, 526861603555761 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
ip-xxxxxxx#show platform hardware qfp active statistics drop
-------------------------------------------------------------------------
Global Drop Stats Packets Octets
-------------------------------------------------------------------------
Disabled 233 269468
Icmp 1 70
IpTtlExceeded 53295 7231829
IpsecIkeIndicate 3470 3522605
IpsecInput 97484 115321755
IpsecInvalidSa 1286 1522173
IpsecOutput 343285 487097831
Ipv4NoAdj 947 1020520
Ipv4NoRoute 348326 434525048
Ipv4Null0 364 29217
Ipv4RoutingErr 2206 123536
ReassDrop 4428 229279
ReassNoFragInfo 2944 4104548
ReassTimeout 5884 2113466
TailDrop 573162728 678027384676
UnconfiguredIpv4Fia 28 1752
Unresolved 946 946530
ip-xxxxxxx#show platform hardware qfp active datapath utilization summary
CPP 0: 5 secs 1 min 5 min 60 min
Input: Total (pps) 104925 105233 105199 106035
(bps) 1074255416 1072888872 1072727696 1074089008
Output: Total (pps) 102623 103071 103058 103750
(bps) 1050293744 1050295320 1050285288 1050283432
Processing: Load (pct) 27 28 28 28
ip-xxxxxxx#show platform hardware qfp active datapath infrastructure sw-nic
PMAP info:
poll err 0; epochs: pmap 0 wait_all 0
poll calls 0
PMD info:
DPDK RTE 2.2.0
epochs: pmd 1 wait_all 1 waiting 37863375893
pmd c99cae40 device Gi1 out of credit 0
RX:pkts 458541301462 bytes 527521744082020 pkts/burst 7 cycl/pkt 206
sw mac filter: 0 filters 0 pkts filtered 0
pkts dropped 0 bad len 0
Ring read 131013968011, empty 70917363568 credit cb 40480282
alloc err 0 credit err 40480282
TX:pkts 457988710102 bytes 525216371366694 send 53584478118 sendnow 1784578963
forced 51799899166 poll 0 thd_poll 0 blocked 0
retries 0 mbuf-alloc err 0 cycl/pkt 137
full 0 lowater 0 hiwater 31 pkts/burst 8
IF:admin_state 0, speed 0 Mbps, duplex 0, mtu 0
RTE:link status: up, speed: 11152 Mbps, duplex: half-duplex, mtu 1500
Promiscuous(ovrd): 0(0), All mc: 0
MAC addr(06ff.0f57.d3cc)
Ring pending:rx( 1024) rx_mini( 0) rx_jumbo( 0) tx( 512)
