Nexus 7009 dropping packets

abbas.ali · ‎12-04-2013

I have a customer running Nexus 7009 running NX-OS 6.0.(4) and having some serious iSCSI issues which is affecting SQL and some critical applications. The basic problem is dropped iSCSI packets which seems to indicate a network problem. We tried eliminating MPIO at the host (Windows 2003 and Win 2008), as well as upgrading Proliant/Broadcom NIC drivers and MS iSCSI initiators, yet the problem persists.

Port 3/17-20 are connected to HP 3PAR Array (4 Nodes), and Port 4/19 as a test server. I don't know if it is a bug with the current Nexus software 6.0.4 or issue with the port settings.

See below the brief output (version, show interface stats etc)

show version

Cisco Nexus Operating System (NX-OS) Software

TAC support: http://www.cisco.com/tac

Documents: http://www.cisco.com/en/US/products/ps9372/tsd_products_support_series_home.html

The copyrights to certain works contained in this software are

owned by other third parties and used and distributed under

license. Certain components of this software are licensed under

the GNU General Public License (GPL) version 2.0 or the GNU

Lesser General Public License (LGPL) Version 2.1. A copy of each

such license is available at

http://www.opensource.org/licenses/gpl-2.0.php and

http://www.opensource.org/licenses/lgpl-2.1.php

Software

BIOS: version 3.22.0

kickstart: version 6.0(4)

system: version 6.0(4)

BIOS compile time: 02/20/10

kickstart image file is: bootflash:///n7000-s1-kickstart.6.0.4.bin

kickstart compile time: 12/25/2020 12:00:00 [06/22/2012 12:26:16]

system image file is: bootflash:///n7000-s1-dk9.6.0.4.bin

system compile time: 6/6/2012 18:00:00 [06/22/2012 14:03:20]

Hardware

cisco Nexus7000 C7009 (9 Slot) Chassis ("Supervisor module-1X")

Intel(R) Xeon(R) CPU with 8251592 kB of memory.

Processor Board ID JAF1649ABCL

Device name: IAD-MDF-CSW1

bootflash: 2048256 kB

slot0: 0 kB (expansion flash)

Kernel uptime is 291 day(s), 19 hour(s), 41 minute(s), 31 second(s)

Last reset

Reason: Unknown

System version: 6.0(4)

Service:

plugin

Core Plugin, Ethernet Plugin

CMP (Module 1) ok

CMP Software

CMP BIOS version: 02.01.05

CMP Image version: 6.0(1) [build 6.0(0.66)]

CMP BIOS compile time: 8/ 4/2008 19:39:40

CMP Image compile time: 6/6/2012 18:00:00

CMP (Module 2) ok

CMP Software

CMP BIOS version: 02.01.05

CMP Image version: 6.0(1) [build 6.0(0.66)]

CMP BIOS compile time: 8/ 4/2008 19:39:40

CMP Image compile time: 6/6/2012 18:00:00

`show module`

Mod Ports Module-Type Model Status

--- ----- ----------------------------------- ------------------ ----------

1 0 Supervisor module-1X N7K-SUP1 active *

2 0 Supervisor module-1X N7K-SUP1 ha-standby

3 32 1/10 Gbps Ethernet Module N7K-F132XP-15 ok

4 48 10/100/1000 Mbps Ethernet XL Module N7K-M148GT-11L ok

Mod Sw Hw

--- -------------- ------

1 6.0(4) 3.0

2 6.0(4) 3.0

3 6.0(4) 1.2

4 6.0(4) 1.2

Mod MAC-Address(es) Serial-Num

--- -------------------------------------- ----------

1 84-78-ac-55-f8-b0 to 84-78-ac-55-f8-b8 JAF1649ABCL

2 84-78-ac-55-f3-e0 to 84-78-ac-55-f3-e8 JAF1649AASC

3 ac-f2-c5-e6-5f-a0 to ac-f2-c5-e6-5f-e4 JAF1647BHDK

4 e0-2f-6d-02-af-ac to e0-2f-6d-02-af-e0 JAF1648AAFJ

Mod Online Diag Status

--- ------------------

1 Pass

2 Pass

3 Pass

4 Pass

----------------------------------------------------------------------------------------------

Ethernet3/17 is up

Dedicated Interface

Hardware: 1000/10000 Ethernet, address: acf2.c5e6.5fb0 (bia acf2.c5e6.5fb0)

MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA

Port mode is access

full-duplex, 10 Gb/s, media type is 10G

Beacon is turned off

Auto-Negotiation is turned on

Input flow-control is off, output flow-control is off

Rate mode is dedicated

Switchport monitor is off

EtherType is 0x8100

Last link flapped 21week(s) 2day(s)

Last clearing of "show interface" counters never

30 seconds input rate 1240528 bits/sec, 1893 packets/sec

30 seconds output rate 44657920 bits/sec, 3720 packets/sec

Load-Interval #2: 5 minute (300 seconds)

input rate 1.07 Mbps, 1.79 Kpps; output rate 44.11 Mbps, 3.62 Kpps

RX

6212152185 unicast packets 429281 multicast packets 1217 broadcast packets

6215450735 input packets 8325559809873 bytes

0 jumbo packets 0 storm suppression packets

0 runts 0 giants 0 CRC 0 no buffer

0 input error 0 short frame 0 overrun 0 underrun 0 ignored

0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop

0 input with dribble 2868052 input discard

0 Rx pause

TX

4365896409 unicast packets 6994873 multicast packets 5685775 broadcast packets

4378577057 output packets 3636859601639 bytes

0 jumbo packets

0 output error 0 collision 0 deferred 0 late collision

0 lost carrier 0 no carrier 0 babble 0 output discard

0 Tx pause

1 interface resets

Ethernet3/18 is up

Dedicated Interface

Hardware: 1000/10000 Ethernet, address: acf2.c5e6.5fb1 (bia acf2.c5e6.5fb1)

MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA

Port mode is access

full-duplex, 10 Gb/s, media type is 10G

Beacon is turned off

Auto-Negotiation is turned on

Input flow-control is off, output flow-control is off

Rate mode is dedicated

Switchport monitor is off

EtherType is 0x8100

Last link flapped 21week(s) 2day(s)

Last clearing of "show interface" counters never

30 seconds input rate 88 bits/sec, 0 packets/sec

30 seconds output rate 832 bits/sec, 0 packets/sec

Load-Interval #2: 5 minute (300 seconds)

input rate 168 bps, 0 pps; output rate 616 bps, 0 pps

RX

816332607 unicast packets 429272 multicast packets 4688 broadcast packets

816770935 input packets 1029653115941 bytes

0 jumbo packets 0 storm suppression packets

0 runts 0 giants 0 CRC 0 no buffer

0 input error 0 short frame 0 overrun 0 underrun 0 ignored

0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop

0 input with dribble 4368 input discard

0 Rx pause

TX

559486852 unicast packets 6994692 multicast packets 5682191 broadcast packets

572163735 output packets 488780855539 bytes

0 jumbo packets

0 output error 0 collision 0 deferred 0 late collision

0 lost carrier 0 no carrier 0 babble 0 output discard

0 Tx pause

1 interface resets

Ethernet3/19 is up

Dedicated Interface

Hardware: 1000/10000 Ethernet, address: acf2.c5e6.5fb2 (bia acf2.c5e6.5fb2)

MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA

Port mode is access

full-duplex, 10 Gb/s, media type is 10G

Beacon is turned off

Auto-Negotiation is turned on

Input flow-control is off, output flow-control is off

Rate mode is dedicated

Switchport monitor is off

EtherType is 0x8100

Last link flapped 21week(s) 2day(s)

Last clearing of "show interface" counters never

30 seconds input rate 957736 bits/sec, 1838 packets/sec

30 seconds output rate 44128776 bits/sec, 3643 packets/sec

Load-Interval #2: 5 minute (300 seconds)

input rate 941.03 Kbps, 1.75 Kpps; output rate 43.49 Mbps, 3.53 Kpps

RX

656897825 unicast packets 429277 multicast packets 4017 broadcast packets

657333669 input packets 807689030859 bytes

0 jumbo packets 0 storm suppression packets

0 runts 0 giants 0 CRC 0 no buffer

0 input error 0 short frame 0 overrun 0 underrun 0 ignored

0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop

0 input with dribble 2550 input discard

0 Rx pause

TX

492336795 unicast packets 6994825 multicast packets 5682939 broadcast packets

505014559 output packets 437551853812 bytes

0 jumbo packets

0 output error 0 collision 0 deferred 0 late collision

0 lost carrier 0 no carrier 0 babble 0 output discard

0 Tx pause

1 interface resets

Ethernet3/20 is up

Dedicated Interface

Hardware: 1000/10000 Ethernet, address: acf2.c5e6.5fb3 (bia acf2.c5e6.5fb3)

MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA

Port mode is access

full-duplex, 10 Gb/s, media type is 10G

Beacon is turned off

Auto-Negotiation is turned on

Input flow-control is off, output flow-control is off

Rate mode is dedicated

Switchport monitor is off

EtherType is 0x8100

Last link flapped 21week(s) 2day(s)

Last clearing of "show interface" counters never

30 seconds input rate 32 bits/sec, 0 packets/sec

30 seconds output rate 832 bits/sec, 0 packets/sec

Load-Interval #2: 5 minute (300 seconds)

input rate 88 bps, 0 pps; output rate 552 bps, 0 pps

RX

691372990 unicast packets 429273 multicast packets 4555 broadcast packets

691824812 input packets 850421645740 bytes

0 jumbo packets 0 storm suppression packets

0 runts 0 giants 0 CRC 0 no buffer

0 input error 0 short frame 0 overrun 0 underrun 0 ignored

0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop

0 input with dribble 17994 input discard

0 Rx pause

TX

516293350 unicast packets 6994665 multicast packets 5682306 broadcast packets

528970321 output packets 457198205525 bytes

0 jumbo packets

0 output error 0 collision 0 deferred 0 late collision

0 lost carrier 0 no carrier 0 babble 0 output discard

0 Tx pause

1 interface resets

Ethernet4/19 is up

Dedicated Interface

Hardware: 10/100/1000 Ethernet, address: e02f.6d02.afbe (bia e02f.6d02.afbe)

MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA

Port mode is access

full-duplex, 1000 Mb/s

Beacon is turned off

Auto-Negotiation is turned off

Input flow-control is on, output flow-control is on

Auto-mdix is turned on

Switchport monitor is off

EtherType is 0x8100

Last link flapped 15:39:53

Last clearing of "show interface" counters never

30 seconds input rate 72 bits/sec, 0 packets/sec

30 seconds output rate 1016 bits/sec, 0 packets/sec

Load-Interval #2: 5 minute (300 seconds)

input rate 264 bps, 0 pps; output rate 600 bps, 0 pps

RX

65076 unicast packets 0 multicast packets 377 broadcast packets

65453 input packets 50864500 bytes

0 jumbo packets 0 storm suppression packets

0 runts 0 giants 0 CRC 0 no buffer

0 input error 0 short frame 0 overrun 0 underrun 0 ignored

0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop

0 input with dribble 0 input discard

0 Rx pause

TX

49177 unicast packets 33361 multicast packets 22469 broadcast packets

105007 output packets 41816639 bytes

0 jumbo packets

0 output error 0 collision 0 deferred 0 late collision

0 lost carrier 0 no carrier 0 babble 0 output discard

0 Tx pause

8 interface resets

Luc Paulin · ‎10-18-2019