what about the storage array, - Page 2

Dragomir · ‎03-17-2014

I have 2 iscsi nics configured on my service profile and they are set to 9000

but in vcenter it is still showing the nics at 1500mtu

any idea?

Manuel Velasco · ‎03-18-2014

Tony,

How is your UCS connected to the storage (topology)?

Dragomir · ‎03-18-2014

everything is connected directly to a pair of redundant 7k

2 redundant FIs, 2 uplinks each to each 7k.

Dragomir · ‎03-18-2014

from the core switch itself when i try to ping the appliance ip I get this

# ping 10.60.1.106 packet-size 9216 c 20
PING 10.60.1.106 (10.60.1.106): 9216 data bytes
9224 bytes from 10.60.1.106: icmp_seq=0 ttl=63 time=10.943 ms
Request 1 timed out
9224 bytes from 10.60.1.106: icmp_seq=2 ttl=63 time=8.93 ms
Request 3 timed out
9224 bytes from 10.60.1.106: icmp_seq=4 ttl=63 time=17.072 ms
Request 5 timed out
9224 bytes from 10.60.1.106: icmp_seq=6 ttl=63 time=8.977 ms
Request 7 timed out
9224 bytes from 10.60.1.106: icmp_seq=8 ttl=63 time=8.984 ms
Request 9 timed out
9224 bytes from 10.60.1.106: icmp_seq=10 ttl=63 time=9.028 ms
Request 11 timed out
9224 bytes from 10.60.1.106: icmp_seq=12 ttl=63 time=8.827 ms
Request 13 timed out
9224 bytes from 10.60.1.106: icmp_seq=14 ttl=63 time=9.171 ms
Request 15 timed out
9224 bytes from 10.60.1.106: icmp_seq=16 ttl=63 time=9.054 ms
Request 17 timed out
9224 bytes from 10.60.1.106: icmp_seq=18 ttl=63 time=8.76 ms
Request 19 timed out

--- 10.60.1.106 ping statistics ---
20 packets transmitted, 10 packets received, 50.00% packet loss
round-trip min/avg/max = 8.76/9.974/17.072 ms
# ping 10.60.1.106 packet-size 8000 c 20
PING 10.60.1.106 (10.60.1.106): 8000 data bytes
8008 bytes from 10.60.1.106: icmp_seq=0 ttl=63 time=2.49 ms
8008 bytes from 10.60.1.106: icmp_seq=1 ttl=63 time=6.139 ms
8008 bytes from 10.60.1.106: icmp_seq=2 ttl=63 time=7.22 ms
Request 3 timed out
8008 bytes from 10.60.1.106: icmp_seq=4 ttl=63 time=2.126 ms
8008 bytes from 10.60.1.106: icmp_seq=5 ttl=63 time=5.761 ms
8008 bytes from 10.60.1.106: icmp_seq=6 ttl=63 time=6.429 ms
Request 7 timed out
8008 bytes from 10.60.1.106: icmp_seq=8 ttl=63 time=1.78 ms
8008 bytes from 10.60.1.106: icmp_seq=9 ttl=63 time=6.466 ms
8008 bytes from 10.60.1.106: icmp_seq=10 ttl=63 time=6.21 ms
Request 11 timed out
8008 bytes from 10.60.1.106: icmp_seq=12 ttl=63 time=1.716 ms
8008 bytes from 10.60.1.106: icmp_seq=13 ttl=63 time=6.277 ms
8008 bytes from 10.60.1.106: icmp_seq=14 ttl=63 time=14.51 ms
Request 15 timed out
8008 bytes from 10.60.1.106: icmp_seq=16 ttl=63 time=2.685 ms
8008 bytes from 10.60.1.106: icmp_seq=17 ttl=63 time=5.212 ms
8008 bytes from 10.60.1.106: icmp_seq=18 ttl=63 time=6.959 ms
Request 19 timed out

Dragomir · ‎03-18-2014

tried different sizes and still some got dropped

ping 10.60.1.106 packet-size 800 c 20
PING 10.60.1.106 (10.60.1.106): 800 data bytes
808 bytes from 10.60.1.106: icmp_seq=0 ttl=63 time=1.995 ms
808 bytes from 10.60.1.106: icmp_seq=1 ttl=63 time=1.151 ms
808 bytes from 10.60.1.106: icmp_seq=2 ttl=63 time=1.235 ms
808 bytes from 10.60.1.106: icmp_seq=3 ttl=63 time=1.005 ms
808 bytes from 10.60.1.106: icmp_seq=4 ttl=63 time=1.844 ms
808 bytes from 10.60.1.106: icmp_seq=5 ttl=63 time=2.016 ms
808 bytes from 10.60.1.106: icmp_seq=6 ttl=63 time=1.425 ms
808 bytes from 10.60.1.106: icmp_seq=7 ttl=63 time=0.993 ms
808 bytes from 10.60.1.106: icmp_seq=8 ttl=63 time=1.612 ms
808 bytes from 10.60.1.106: icmp_seq=9 ttl=63 time=1.056 ms
808 bytes from 10.60.1.106: icmp_seq=10 ttl=63 time=2.187 ms
808 bytes from 10.60.1.106: icmp_seq=11 ttl=63 time=2.292 ms
808 bytes from 10.60.1.106: icmp_seq=12 ttl=63 time=2.011 ms
808 bytes from 10.60.1.106: icmp_seq=13 ttl=63 time=2.579 ms
808 bytes from 10.60.1.106: icmp_seq=14 ttl=63 time=2.947 ms
808 bytes from 10.60.1.106: icmp_seq=15 ttl=63 time=2.287 ms
808 bytes from 10.60.1.106: icmp_seq=16 ttl=63 time=0.956 ms
808 bytes from 10.60.1.106: icmp_seq=17 ttl=63 time=1.65 ms
808 bytes from 10.60.1.106: icmp_seq=18 ttl=63 time=2.35 ms
808 bytes from 10.60.1.106: icmp_seq=19 ttl=63 time=1.629 ms

--- 10.60.1.106 ping statistics ---
20 packets transmitted, 20 packets received, 0.00% packet loss
round-trip min/avg/max = 0.956/1.76/2.947 ms
# ping 10.60.1.106 packet-size 1800 c 20
PING 10.60.1.106 (10.60.1.106): 1800 data bytes
1808 bytes from 10.60.1.106: icmp_seq=0 ttl=63 time=1.21 ms
1808 bytes from 10.60.1.106: icmp_seq=1 ttl=63 time=1.428 ms
1808 bytes from 10.60.1.106: icmp_seq=2 ttl=63 time=1.385 ms
1808 bytes from 10.60.1.106: icmp_seq=3 ttl=63 time=1.63 ms
1808 bytes from 10.60.1.106: icmp_seq=4 ttl=63 time=1.506 ms
1808 bytes from 10.60.1.106: icmp_seq=5 ttl=63 time=1.942 ms
1808 bytes from 10.60.1.106: icmp_seq=6 ttl=63 time=1.601 ms
1808 bytes from 10.60.1.106: icmp_seq=7 ttl=63 time=1.893 ms
1808 bytes from 10.60.1.106: icmp_seq=8 ttl=63 time=1.354 ms
Request 9 timed out
1808 bytes from 10.60.1.106: icmp_seq=10 ttl=63 time=1.485 ms
1808 bytes from 10.60.1.106: icmp_seq=11 ttl=63 time=1.894 ms
1808 bytes from 10.60.1.106: icmp_seq=12 ttl=63 time=1.67 ms
1808 bytes from 10.60.1.106: icmp_seq=13 ttl=63 time=1.406 ms
1808 bytes from 10.60.1.106: icmp_seq=14 ttl=63 time=1.611 ms
1808 bytes from 10.60.1.106: icmp_seq=15 ttl=63 time=1.929 ms
1808 bytes from 10.60.1.106: icmp_seq=16 ttl=63 time=1.454 ms
1808 bytes from 10.60.1.106: icmp_seq=17 ttl=63 time=1.488 ms
1808 bytes from 10.60.1.106: icmp_seq=18 ttl=63 time=1.482 ms
Request 19 timed out

Manuel Velasco · ‎03-18-2014

Hi Tony,

Looks like there are some problems on your 7K. How did you configure Jumbo frames on the Nexus 7K? The configuration should look something like this

switch(config)#policy-map type network-qos jumbo
switch(config-pmap-nq)#class type network-qos class-default
switch(config-pmap-c-nq)#mtu 9216
switch(config-pmap-c-nq)#exit
switch(config-pmap-nq)#exit
switch(config)#system qos
switch(config-sys-qos)#service-policy type network-qos jumbo

Did you set the MTU size on the storate to 9000?

Dragomir · ‎03-18-2014

this is what i see.... Looks like it is not enabled?

sh policy-map

Type qos policy-maps
====================

policy-map type qos ISCSI
class class-default
set cos 5

Type queuing policy-maps
========================

policy-map type queuing default-in-policy
class type queuing in-q1
queue-limit percent 50
bandwidth percent 80
class type queuing in-q-default
queue-limit percent 50
bandwidth percent 20
policy-map type queuing default-out-policy
class type queuing out-pq1
priority level 1
queue-limit percent 16
class type queuing out-q2
queue-limit percent 1
class type queuing out-q3
queue-limit percent 1
class type queuing out-q-default
queue-limit percent 82
bandwidth remaining percent 25
policy-map type queuing default-4q-8e-in-policy
class type queuing 2q4t-8e-in-q1
queue-limit percent 10
bandwidth percent 50
class type queuing 2q4t-8e-in-q-default
queue-limit percent 90
bandwidth percent 50
policy-map type queuing default-4q-8e-out-policy
class type queuing 1p3q1t-8e-out-pq1
priority level 1
class type queuing 1p3q1t-8e-out-q2
bandwidth remaining percent 33
class type queuing 1p3q1t-8e-out-q3
bandwidth remaining percent 33
class type queuing 1p3q1t-8e-out-q-default
bandwidth remaining percent 33

Type control-plane policy-maps
==============================

policy-map type control-plane copp-system-p-policy-strict
class copp-system-p-class-critical
set cos 7
police cir 39600 kbps bc 250 ms conform transmit violate drop
class copp-system-p-class-important
set cos 6
police cir 1060 kbps bc 1000 ms conform transmit violate drop
class copp-system-p-class-management
set cos 2
police cir 10000 kbps bc 250 ms conform transmit violate drop
class copp-system-p-class-normal
set cos 1
police cir 680 kbps bc 250 ms conform transmit violate drop
class copp-system-p-class-normal-dhcp
set cos 1
police cir 680 kbps bc 250 ms conform transmit violate drop
class copp-system-p-class-normal-dhcp-relay-response
set cos 1
police cir 900 kbps bc 500 ms conform transmit violate drop
class copp-system-p-class-redirect
set cos 1
police cir 280 kbps bc 250 ms conform transmit violate drop
class copp-system-p-class-exception
set cos 1
police cir 360 kbps bc 250 ms conform transmit violate drop
class copp-system-p-class-monitoring
set cos 1
police cir 130 kbps bc 1000 ms conform transmit violate drop
class copp-system-p-class-l2-unpoliced
police cir 8 gbps bc 5 mbytes conform transmit violate transmit
class copp-system-p-class-undesirable
set cos 0
police cir 32 kbps bc 250 ms conform drop violate drop
class copp-system-p-class-l2-default
police cir 100 kbps bc 250 ms conform transmit violate drop
class class-default
set cos 0
police cir 100 kbps bc 250 ms conform transmit violate drop

Type network-qos policy-maps
============================
policy-map type network-qos default-nq-4e-policy
class type network-qos c-nq-4e-drop
congestion-control tail-drop
mtu 1500
class type network-qos c-nq-4e-ndrop-fcoe
pause
mtu 2112
class type network-qos c-nq-4e-ndrop
pause
mtu 2112
policy-map type network-qos default-nq-6e-policy
class type network-qos c-nq-6e-drop
congestion-control tail-drop
mtu 1500
class type network-qos c-nq-6e-ndrop-fcoe
pause
mtu 2112
class type network-qos c-nq-6e-ndrop
pause
mtu 2112
policy-map type network-qos default-nq-7e-policy
class type network-qos c-nq-7e-drop
congestion-control tail-drop
mtu 1500
class type network-qos c-nq-7e-ndrop-fcoe
pause
mtu 2112
policy-map type network-qos default-nq-8e-policy
class type network-qos c-nq-8e
congestion-control tail-drop
mtu 1500

Manuel Velasco · ‎03-19-2014

Hi Tony,

Yes, I don't see jumbo fram enabled on your nexus 7k. When you confiugre jumbo frames you need to make sure that any device where these frames are going to traverse jumbo frames is enable. It won't work if you only enable jumbo frames in one place.

Dragomir · ‎03-19-2014

is the policy map a global enable of jumbo frames on the entire 7k? I remember it used to be system mtu 9216 or something like that on the ios switches.

Manuel Velasco · ‎03-19-2014

Hey Tony,

Sorry, I gave you the jumbo frame configuration fo the n5k. For the 7K you will need to do the following

http://www.cisco.com/c/en/us/support/docs/switches/nexus-5000-series-switches/112080-config-mtu-nexus.html

Complete these steps in order to set the jumbo frame in a Nexus 7010 Switch:

Nexus-7010


!--- Set the MTU to its maximum  
!--- size (9216 bytes) in order  
!--- to enable the Jumbo MTU
!--- for the whole switch. 


switch(config)#system jumbomtu 9216


!--- Set the MTU specification for an interface.

switch(config)#interface ethernet x/x


!--- By default, Cisco NX-OS configures Layer 3 parameters.
In order to configure Layer 2 parameters, use this command.

switch(config-if)#switchport

switch(config-if)#mtu 9216
switch(config-if)#exit

Dragomir · ‎03-19-2014

i already have that enabled already

# sh run all | i jumbomtu
system jumbomtu 9216

also on all the data iscsi ports

interface Ethernet3/33
switchport access vlan 60
spanning-tree port type edge
speed 1000
mtu 9216
no shutdown

# sh run interface ethernet 3/34

!Command: show running-config interface Ethernet3/34
!Time: Wed Mar 19 10:57:22 2014

version 6.1(2)

interface Ethernet3/34
switchport access vlan 60
spanning-tree port type edge
mtu 9216
no shutdown

# sh run interface ethernet 3/36

!Command: show running-config interface Ethernet3/36
!Time: Wed Mar 19 10:57:23 2014

version 6.1(2)

interface Ethernet3/36
switchport access vlan 15
spanning-tree port type edge
speed 1000
mtu 9216
no shutdown

Manuel Velasco · ‎03-19-2014

what about the storage array, is the MTU set to jumbo frames? What about the CoS on storage array is it the same as your 7K?

At this point I'm running out of ideas, I would recomend to open a TAC case with the Nexus 7K to help you with the configuration.

Dragomir · ‎03-24-2014

yes verified on the storage array that the iscsi ports are set to mtu 9000.

I do not have any cos on the storage array.

Dragomir · ‎03-25-2014

hi there

I checked my vlan interface and it shows this

interface Vlan60
no ip redirects
ip address 10.60.0.1/16
no ipv6 redirects
ip proxy-arp
no hsrp bfd
hsrp version 2
hsrp delay minimum 0 reload 0
no hsrp use-bia
hsrp 60
authentication cisco
name hsrp-Vlan60-60
mac-address 0000.0C9F.F03C
no preempt
priority 100 forwarding-threshold lower 1 upper 100
timers 3 10
ip 10.60.0.5
description SERVERS60
no shutdown
mtu 1500
bandwidth 1000000
delay 1
medium broadcast
snmp trap link-status
carrier-delay msec 100
load-interval counter 1 60
load-interval counter 2 300
no load-interval counter 3
mac-address d867.d907.ffc1
no management

looks like it is set at 1500. Do i have to change it there as well? the document did not mention it

thanks

Manuel Velasco · ‎03-19-2014

Before you make the changes Tony, are you using FCoE on the Nexus 7 K?

Dragomir · ‎03-19-2014

hi

No I am not using fcoe

mtu 1500 shows in vcenter but service profile template is at 9000