jumbo frames dropped when pining from nexus 7k to iscsi port appliance

Unanswered Question
Mar 18th, 2014
User Badges:

I have an iscsi appliance attached to the nexus 7k and port on the appliance is set at 9216.

 

but when i ping from the switch to the appliance with

 

 

# ping 10.60.1.106 packet-size 9216 c 20
PING 10.60.1.106 (10.60.1.106): 9216 data bytes
9224 bytes from 10.60.1.106: icmp_seq=0 ttl=63 time=10.943 ms
Request 1 timed out
9224 bytes from 10.60.1.106: icmp_seq=2 ttl=63 time=8.93 ms
Request 3 timed out
9224 bytes from 10.60.1.106: icmp_seq=4 ttl=63 time=17.072 ms
Request 5 timed out
9224 bytes from 10.60.1.106: icmp_seq=6 ttl=63 time=8.977 ms
Request 7 timed out
9224 bytes from 10.60.1.106: icmp_seq=8 ttl=63 time=8.984 ms
Request 9 timed out
9224 bytes from 10.60.1.106: icmp_seq=10 ttl=63 time=9.028 ms
Request 11 timed out
9224 bytes from 10.60.1.106: icmp_seq=12 ttl=63 time=8.827 ms
Request 13 timed out
9224 bytes from 10.60.1.106: icmp_seq=14 ttl=63 time=9.171 ms
Request 15 timed out
9224 bytes from 10.60.1.106: icmp_seq=16 ttl=63 time=9.054 ms
Request 17 timed out
9224 bytes from 10.60.1.106: icmp_seq=18 ttl=63 time=8.76 ms
Request 19 timed out

 

 

they time out. 

 

 

what would be the issue?

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (1 ratings)
Loading.
Steve Fuller Wed, 03/19/2014 - 01:56
User Badges:
  • Blue, 1500 points or more
  • Community Spotlight Award,

    Member's Choice, September 2015

Hi,

There are a couple of things to consider here.

The value you specify with the packet-size option on the ping command is actually the data portion of the packet i.e., the value not including the ICMP and IP header. The packet sent by the router in the test you did would have been 9244-bytes (9216 data + 8-bytes ICMP + 20-bytes IP). The value you specify should be 28-bytes less than the MTU size to accommodate this.

Also when testing jumbo frames you should use the df-bit option, otherwise the router will simply fragment the packet that is sent and so you won't properly be able to determine the MTU size.

So if you have an end system with an MTU of 9216-bytes the command you should use to test end to end jumbo frame connectivity would be ping <ip-address> df-bit packet-size 9188. That way when the 8-byte ICMP and 20-byte IP header are added, the packet size will be 9216-bytes.

Just on this point, I would also confirm the MTU value set on the storage is 9216-bytes. Ethernet jumbo frames are not properly specified within any standard and so different vendors use different values. On end systems I've seen and played around with the MTU is generally 9000-bytes.

The other area to watch out for is Control Plane Policing (CoPP) on the Nexus. The Nexus 7000 runs CoPP by default to limit traffic hitting the CPU on switch supervisor. I've seen issues in the past with ICMP as the default value used within the policer quite low, and if you're specifying large packet sizes with the ping it's possible CoPP is dropping the traffic. You can use the show policy-map interface control-plane | in "class-map|violate" command to quickly check if you have drops in any of the classes. If the Nexus is running the default CoPP configuration then the class-map is likely to be copp-system-class-monitoring. Remember also note that you can only check copp from the default VDC.

If CoPP is dropping the ICMP traffic then you can change the police cir <value> within the policy-map, but obviously consider the reason for CoPP and don't increase the value to the point where CoPP is no longer effective.

Regards

tonywong123 Wed, 03/19/2014 - 14:07
User Badges:

Will not work

 

 

Any packet above 1472 gets dropped

 

 

--- 10.60.1.106 ping statistics ---

1 packets transmitted, 0 packets received, 100.00% packet loss

# ping 10.60.1.106 df-bit packet-size 1472

PING 10.60.1.106 (10.60.1.106): 1472 data bytes

1480 bytes from 10.60.1.106: icmp_seq=0 ttl=63 time=1.481 ms

1480 bytes from 10.60.1.106: icmp_seq=1 ttl=63 time=0.864 ms

1480 bytes from 10.60.1.106: icmp_seq=2 ttl=63 time=1.187 ms

1480 bytes from 10.60.1.106: icmp_seq=3 ttl=63 time=1.468 ms

1480 bytes from 10.60.1.106: icmp_seq=4 ttl=63 time=1.229 ms

 

--- 10.60.1.106 ping statistics ---

5 packets transmitted, 5 packets received, 0.00% packet loss

round-trip min/avg/max = 0.864/1.245/1.481 ms

# ping 10.60.1.106 df-bit packet-size 1473

PING 10.60.1.106 (10.60.1.106): 1473 data bytes

^C

--- 10.60.1.106 ping statistics ---

1 packets transmitted, 0 packets received, 100.00% packet loss

WSGRCSA# ping 10.60.1.106 df-bit packet-size 1473

PING 10.60.1.106 (10.60.1.106): 1473 data bytes

Request 0 timed out

Request 1 timed out

Request 2 timed out

Request 3 timed out

Request 4 timed out

 

 

----------------------------------------------------------------------------------------------

# show policy-map interface control-plane | in "class-map|violate"
    class-map copp-system-p-class-critical (match-any)
        violated 0 bytes; action: drop 
        violated 0 bytes; action: drop 
    class-map copp-system-p-class-important (match-any)
        violated 564980520 bytes; action: drop 
        violated 16564096880 bytes; action: drop 
    class-map copp-system-p-class-management (match-any)
        violated 0 bytes; action: drop 
        violated 0 bytes; action: drop 
    class-map copp-system-p-class-normal (match-any)
        violated 38979438 bytes; action: drop 
        violated 102233820206 bytes; action: drop 
    class-map copp-system-p-class-normal-dhcp (match-any)
        violated 15555618 bytes; action: drop 
        violated 257350720 bytes; action: drop 
    class-map copp-system-p-class-normal-dhcp-relay-response (match-any)
        violated 0 bytes; action: drop 
        violated 0 bytes; action: drop 
    class-map copp-system-p-class-redirect (match-any)
        violated 0 bytes; action: drop 
        violated 0 bytes; action: drop 
    class-map copp-system-p-class-exception (match-any)
        violated 0 bytes; action: drop 
        violated 0 bytes; action: drop 
    class-map copp-system-p-class-monitoring (match-any)
        violated 278218 bytes; action: drop 
        violated 274881 bytes; action: drop 
    class-map copp-system-p-class-l2-unpoliced (match-any)
        violated 0 bytes; action: transmit 
        violated 0 bytes; action: transmit 
    class-map copp-system-p-class-undesirable (match-any)
        violated 8578 bytes; action: drop 
        violated 20752 bytes; action: drop 
    class-map copp-system-p-class-l2-default (match-any)
        violated 68855 bytes; action: drop 
        violated 5191298 bytes; action: drop 
    class-map class-default (match-any)
        violated 39535752498 bytes; action: drop 
        violated 78583238971 bytes; action: drop 

 

 

 

i have a lot of dropped or violated packets.

Steve Fuller Thu, 03/20/2014 - 00:34
User Badges:
  • Blue, 1500 points or more
  • Community Spotlight Award,

    Member's Choice, September 2015

Hi,

Let's forget the Control Plane Policing at the moment then as there's a more fundamental issue in that you don't have jumbo frames configured end-to-end. This can be seen as you're dropping any ICMP packets larger than 1472-bytes with the DF bit set.

Can you provide details of how the storage is connected specifically the following:

  • Is the iSCSI storage device connected directly to the Nexus 7K and if so is the interface it is connected to configured with mtu 9216?
  • Does the storage connect via multiple interfaces e.g., port-channel, and if so are all interfaces configured for jumbo frames?
  • Are you also routing on the Nexus 7K and if so do the VLAN interfaces have mtu 9216 configured ?

Can you post the relevant show interface commands from the Nexus, and if possible an ifconfig (or equivalent command) from the storage?

Regards

tonywong123 Mon, 03/24/2014 - 17:44
User Badges:

hello

Storage is connected directly to the 7k. Verified that the port is set to mtu 9216.

Also verified on the appliance itself that the port is at mtu 9000.

 

storage connects via multiple interfaces but not port channel. EAch iscsi interface functiona as individual interfaces. 

  • Are you also routing on the Nexus 7K and if so do the VLAN interfaces have mtu 9216 configured ?

how do I check this?

 

 sh interface ethernet 3/34
Ethernet3/34 is up
admin state is up, Dedicated Interface
  Hardware: 1000/10000 Ethernet, address: 6073.5c04.98cd (bia 6073.5c04.98cd)
  Description: storesimple data3 primary
  MTU 9216 bytes, BW 1000000 Kbit, DLY 10 usec
  reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, medium is broadcast
  Port mode is access
  full-duplex, 1000 Mb/s, media type is 1G
  Beacon is turned off
  Auto-Negotiation is turned off
  Input flow-control is off, output flow-control is off
  Auto-mdix is turned off
  Rate mode is dedicated
  Switchport monitor is off 
  EtherType is 0x8100 
  EEE (efficient-ethernet) : n/a
  Last link flapped 5d06h
  Last clearing of "show interface" counters never
  81 interface resets
  30 seconds input rate 0 bits/sec, 0 packets/sec
  30 seconds output rate 3320 bits/sec, 1 packets/sec
  Load-Interval #2: 5 minute (300 seconds)
    input rate 0 bps, 0 pps; output rate 2.70 Kbps, 1 pps
  RX
    57989568648 unicast packets  188 multicast packets  6254 broadcast packets
    57989437082 input packets  94861146540597 bytes
    1481562539 jumbo packets  0 storm suppression packets
    0 runts  0 giants  0 CRC  0 no buffer
    0 input error  0 short frame  0 overrun   0 underrun  0 ignored
    0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop
    0 input with dribble  0 input discard
    0 Rx pause
  TX
    17057991061 unicast packets  42719697 multicast packets  42453871 broadcast packets
    17143026620 output packets  19149860787894 bytes
    708264551 jumbo packets
    0 output error  0 collision  0 deferred  0 late collision
    0 lost carrier  0 no carrier  0 babble  0 output discard
    0 Tx pause

# sh interface ethernet 3/36
Ethernet3/36 is up
admin state is up, Dedicated Interface
  Hardware: 1000/10000 Ethernet, address: 6073.5c04.98cf (bia 6073.5c04.98cf)
  Description: storesimple secondary data1
  MTU 9216 bytes, BW 1000000 Kbit, DLY 10 usec
  reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, medium is broadcast
  Port mode is access
  full-duplex, 1000 Mb/s, media type is 1G
  Beacon is turned off
  Auto-Negotiation is turned off
  Input flow-control is off, output flow-control is off
  Auto-mdix is turned off
  Rate mode is dedicated
  Switchport monitor is off 
  EtherType is 0x8100 
  EEE (efficient-ethernet) : n/a
  Last link flapped 21week(s) 1day(s)
  Last clearing of "show interface" counters never
  6 interface resets
  30 seconds input rate 0 bits/sec, 0 packets/sec
  30 seconds output rate 2608 bits/sec, 3 packets/sec
  Load-Interval #2: 5 minute (300 seconds)
    input rate 0 bps, 0 pps; output rate 1.79 Kbps, 3 pps
  RX
    12217843366 unicast packets  0 multicast packets  145 broadcast packets
    12217705503 input packets  8077006642708 bytes
    0 jumbo packets  0 storm suppression packets
    0 runts  0 giants  0 CRC  0 no buffer
    0 input error  0 short frame  0 overrun   0 underrun  0 ignored
    0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop
    0 input with dribble  0 input discard
    0 Rx pause
  TX
    35911585661 unicast packets  28648470 multicast packets  90374246 broadcast packets
    36030470368 output packets  52448125979820 bytes
    0 jumbo packets
    0 output error  0 collision  0 deferred  0 late collision
    0 lost carrier  0 no carrier  0 babble  0 output discard
    0 Tx pause

 

 

tonywong123 Tue, 03/25/2014 - 11:49
User Badges:

hi 

when i look at my vlan interface

I see this

 


interface Vlan60
  no ip redirects
  ip address 10.60.0.1/16
  no ipv6 redirects
  ip proxy-arp
  no hsrp bfd
  hsrp version 2
  hsrp delay minimum 0 reload 0 
  no hsrp use-bia
  hsrp 60 
    authentication cisco
    name hsrp-Vlan60-60
    mac-address 0000.0C9F.F03C
    no preempt
    priority 100 forwarding-threshold lower 1 upper 100
    timers 3 10
    ip 10.60.0.5 
  description SERVERS60
  no shutdown
  mtu 1500
  bandwidth 1000000
  delay 1
  medium broadcast
  snmp trap link-status
  carrier-delay msec 100
  load-interval counter 1 60
  load-interval counter 2 300
  no load-interval counter 3
  mac-address d867.d907.ffc1 
  no management

 

 

looks like mtu is at 1500. 

 

do I need to change mtu on vlan interfaces as well? According to cisco document they did not mention it.

 

http://www.cisco.com/c/en/us/support/docs/switches/nexus-5000-series-swi...

Steve Fuller Tue, 03/25/2014 - 23:05
User Badges:
  • Blue, 1500 points or more
  • Community Spotlight Award,

    Member's Choice, September 2015

Hi,

If all your hosts and storage are in the same Layer-2 broadcast domain i.e., no Layer-3 hops between them, then there should be no need to configure mtu 9216 i.e., jumbo frames, on your Nexus 7000 VLAN interfacces.

If there are any layer-3 hops between the hosts and storage, or if you want to be able to ping the storage or hosts with jumbo frames from the Nexus 7000, then you will need to add mtu 9216 to the VLAN interfaces.

That's an interesting document that you link to above, and in my humble opinion, not very well produced. In the first part of the document it is shown that ping 172.16.0.1 packet-size 9216 c 20 fails, but not really explained why. There's no use of the df-bit option, and as seen, this is key when testing jumbo frames. Additionally after jumbo frames are enabled, you are told to verify with the ping -l 9000 command. Why do they not verify using the same ping as originally?

Regards

Mukund M Deshpande Wed, 09/28/2016 - 03:56
User Badges:



Hi,


I have configured HSRP with nexus 7000 (primary) and 4500 (Secondary)

I am able to ping hsrp virtual ip from cmd with 15000 byte but not able to ping same ip with 17000 byte for nexus device.



Please help me on this

Actions

This Discussion