I have been attempting upgrade on a pair of Nexus 5010s in a vPC configuration. I want to go from n5000-uk9.5.0.3.N1.1c to n5000-uk9.5.2.1.N1.4 as that is the minimum recommended version. I have 4 FEXs that are dual homed to the N5ks and 4 old CatOS switches that are port channeled / vPC as well. Since I have switches running spanning tree downstream I can’t do ISSU, so I did a disruptive upgrade.
I followed the procedure here under the “Upgrading A Dual-Homed FEX Access Layer“ procedures: http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/upgrade/503_N1_1/n5k_upgrade_downgrade_503.html#wp641001
On the primary N5k I entered:
install all kickstart bootflash:n5000-uk9-kickstart.5.2.1.N1.4.bin system bootflash:n5000-uk9.5.2.1.N1.4.bin
The primary NK5 reloaded and came back up with the new code, everything good so far.
I set the boot parameters on the secondary switch:
boot kickstart bootflash:n5000-uk9-kickstart.5.2.1.N1.4.bin
boot system bootflash:n5000-uk9.5.2.1.N1.4.bin
copy run start
I reloaded the FEXs and waited until they came online running the new version of code on the Primary N5K. My continuous pings were all responding.
I reloaded the secondary switch to complete the upgrade. When I did this I immediately began losing continuous pings I had going to nodes connected to the FEXs and my management interfaces on the CatOS switches behind the N5Ks. Some of them dropped completely, some of them were intermittent, no response for 5 pings, respond for 3 ping etc. After the secondary N5k came back up, the issue remained.
I verified both N5Ks were running the same new code, I did a config comparison and it looked fine, I did observere that the new code added a “vlan configuration [vlans]” that is not in the old software and it reorded some of the port channel config. FEXs were all online on both switches, running new code, the vPC was up, peer link up, peer keep alive up, vPC roles were correct, all the port channels were up, all the interfaces were up, logs didn’t give any indication of problems, spanning tree (MST) looked ok, I rebooted both switches again one at a time, I cleared the mac address tables, ... nothing seemed wrong, nothing fixed the problem.
It doesn't appear to be a problem with the FEXs, as my CatOS switches are directly conencted to the N5ks and they are experiencing the same problem.
I continued to have the same intermittent/complete connectivity loss. I have attempted this upgrade twice with the same result, and both times I had to revert back to the original code to get everything working normally. When I reverted back to the old code everything started working!
Any ideas for locating and resolving this problem?
