11-12-2013 08:37 AM - edited 03-07-2019 04:34 PM
Hallo all,
I see a very strange behavior on my two nexus switches.
Both are Nexus 5548 with L3-daughter-cards. Both do l2 and l3-switching, ACL-filtering and other things. Furthermore I have a set of servers connected to both switches in a vPC-setup. All in all I do nothing special.
After reloading the primary switch (vpc-primary, root-bridge for all vlans and hsrp-active with preemption for all SVIs) the switche comes back online and after getting up all links and reconverging everthing the network breaks. After a lot of debugging and curses and connection tries and a few additional gray hairs later I have got it to work by pinging all ip-addresses from the switch that I have previously rebooted.
Later I do some tests to find out what was going wrong. I found out that if I clear the arp-cache I will get the same issue. Pinging from server A in one subnet to server B in another subnet doesn't lead to success, because the switch issues no arp-requests. To make it work just ping server B from the switch and all works fine. The switch does arp, the arp-table is updated and the pings from the server A will reach the server B.
Any ideas?
Regards
Thomas
^^°-°^^
11-12-2013 11:01 PM
Can you share the config?
11-13-2013 12:08 AM
I can post a extraction of the relevant config items.
- --[ vpc-primary
cfs01# sh run
!Command: show running-config
!Time: Wed Nov 13 08:46:18 2013
version 5.2(1)N1(1b)
cfs eth distribute
vrf context CEPH
vrf context management
ip route 172.31.0.0/20 172.31.8.190
vlan 14
name 172.31.50.0/26_CN/NN/OSDs@DMZ
vlan 4080
name 172.31.48.64/26_NAS.Infrastr@DMZ
spanning-tree vlan 1-129,131-3967,4048-4093 priority 0
udld aggressive
vpc domain 1
role priority 1
peer-keepalive destination 172.31.8.179 source 172.31.8.178
peer-config-check-bypass
delay restore 150
peer-gateway
auto-recovery
ip arp synchronize
interface Vlan14
no shutdown
mtu 9216
description CN/NN/OSDs@DMZ
ip access-group acl-vl14-in in
vrf member CEPH
no ip redirects
ip address 172.31.50.61/26
no ip port-unreachable
hsrp version 2
hsrp 3
authentication md5 key-string 3-14
preempt delay minimum 30 reload 60
priority 255
ip 172.31.50.62
interface Vlan4080
no shutdown
mtu 9216
description NAS.Infrastr@DMZ
ip access-group acl-vl4080-in in
vrf member CEPH
no ip redirects
ip address 172.31.48.125/26
no ip port-unreachable
hsrp version 2
hsrp 3
authentication md5 key-string 3-4080
preempt delay minimum 30 reload 60
priority 255
ip 172.31.48.126
interface port-channel7
switchport mode trunk
switchport trunk native vlan 991
spanning-tree port type network
speed 10000
vpc peer-link
interface port-channel100
switchport mode trunk
switchport trunk native vlan 991
switchport trunk allowed vlan 2,14-19,991,4078-4080
speed 10000
vpc 100
interface port-channel102
switchport mode trunk
switchport trunk native vlan 991
switchport trunk allowed vlan 2,14,18,991,1299-1400
speed 10000
vpc 102
interface Ethernet1/1
no cdp enable
switchport mode trunk
switchport trunk native vlan 991
switchport trunk allowed vlan 2,14-19,991,4078-4080
channel-group 100 mode active
interface Ethernet1/3
no cdp enable
switchport mode trunk
switchport trunk native vlan 991
switchport trunk allowed vlan 2,14,18,991,1299-1400
channel-group 102 mode active
interface Ethernet1/29
description cfs02_Eth29
switchport mode trunk
switchport trunk native vlan 991
channel-group 7 mode active
interface Ethernet1/30
description cfs02_Eth30
switchport mode trunk
switchport trunk native vlan 991
channel-group 7 mode active
interface Ethernet1/31
description cfs02_Eth31
switchport mode trunk
switchport trunk native vlan 991
channel-group 7 mode active
interface Ethernet1/32
description cfs02_Eth32
switchport mode trunk
switchport trunk native vlan 991
channel-group 7 mode active
interface mgmt0
description oam01_Gi0/19
ip address 172.31.8.178/26
cfs01#
- --[ vpc-secondary
cfs02# sh run
!Command: show running-config
!Time: Wed Nov 13 08:46:05 2013
version 5.2(1)N1(1b)
cfs eth distribute
vrf context CEPH
vrf context management
ip route 172.31.0.0/20 172.31.8.190
vlan 14
name 172.31.50.0/26_CN/NN/OSDs@DMZ
vlan 4080
name 172.31.48.64/26_NAS.Infrastr@DMZ
spanning-tree vlan 1-129,131-3967,4048-4093 priority 4096
udld aggressive
vpc domain 1
role priority 2
peer-keepalive destination 172.31.8.178 source 172.31.8.179
peer-config-check-bypass
delay restore 150
peer-gateway
auto-recovery
ip arp synchronize
interface Vlan14
no shutdown
mtu 9216
description CN/NN/OSDs@DMZ
ip access-group acl-vl14-in in
vrf member CEPH
no ip redirects
ip address 172.31.50.60/26
no ip port-unreachable
hsrp version 2
hsrp 3
authentication md5 key-string 3-14
priority 254
ip 172.31.50.62
interface Vlan4080
no shutdown
mtu 9216
description NAS.Infrastr@DMZ
ip access-group acl-vl4080-in in
vrf member CEPH
no ip redirects
ip address 172.31.48.124/26
no ip port-unreachable
hsrp version 2
hsrp 3
authentication md5 key-string 3-4080
priority 254
ip 172.31.48.126
interface port-channel7
switchport mode trunk
switchport trunk native vlan 991
spanning-tree port type network
speed 10000
vpc peer-link
interface port-channel100
switchport mode trunk
switchport trunk native vlan 991
switchport trunk allowed vlan 2,14-19,991,4078-4080
speed 10000
vpc 100
interface port-channel102
switchport mode trunk
switchport trunk native vlan 991
switchport trunk allowed vlan 2,14,18,991,1299-1400
speed 10000
vpc 102
interface Ethernet1/1
no cdp enable
switchport mode trunk
switchport trunk native vlan 991
switchport trunk allowed vlan 2,14-19,991,4078-4080
channel-group 100 mode active
interface Ethernet1/3
no cdp enable
switchport mode trunk
switchport trunk native vlan 991
switchport trunk allowed vlan 2,14,18,991,1299-1400
channel-group 102 mode active
interface Ethernet1/29
description cfs01_Eth29
switchport mode trunk
switchport trunk native vlan 991
channel-group 7 mode active
interface Ethernet1/30
description cfs01_Eth30
switchport mode trunk
switchport trunk native vlan 991
channel-group 7 mode active
interface Ethernet1/31
description cfs01_Eth31
switchport mode trunk
switchport trunk native vlan 991
channel-group 7 mode active
interface Ethernet1/32
description cfs01_Eth32
switchport mode trunk
switchport trunk native vlan 991
channel-group 7 mode active
interface mgmt0
description oam02_Gi0/19
ip address 172.31.8.179/26
cfs02#
11-14-2013 07:00 AM
fuck. The faulty behavior disappears. Just rebooting the nexus-switch. Two days to view a lots of logg-messages, error discovery, tests... For what? For nothing. And now I'm not absolutely sure that the fault will not raise up again. That does not inspire me with confidence.
^^°-°^^
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: