TIP OF THE DAY: N9K Switch Conversion Process from NXOS Standalone Mode to ACI Mode

Tomas de Leon · ‎10-22-2014

Today's TIP OF THE DAY is a supplemental example of the Nexus 9000 Series Switch Conversion Process from NXOS Standalone Mode to ACI Mode. This process should provide you steps that will successfully convert your N9K to ACI Mode. Please note the "Verify Installed Certificates" section towards the bottom of this article. This is an important check that will ensure that your switches will successfully complete the ACI Fabric Discovery process.

Nexus 9000 Series Switch Conversion Process

from NXOS Standalone Mode to ACI Mode

Prerequisites:

* The Nexus 9000 Series Switch must be running the latest EPLD version for the installed hardware.

* To list the EPLDs running on your switch, use the "show version module module_number epld" command. If any of the versions that you list are older than what is listed in the Cisco Nexus 9000 Series FPGA/EPLD Upgrade Release Notes , we recommend that you update the EPLDs.

For Example (for Cisco Nexus 93xx):

switch# show version

Cisco Nexus Operating System (NX-OS) Software

TAC support: http://www.cisco.com/tac

Software

BIOS: version 07.05 [last: ]

NXOS: version 6.1(2)I2(1)

BIOS compile time: 01/29/2014 [last: ]

NXOS image file is: bootflash:///n9000-dk9.6.1.2.I2.1.bin

NXOS compile time: 3/15/2014 19:00:00 [03/16/2014 04:26:07]

Hardware

cisco Nexus9000 C9396PX Chassis

Intel(R) Pentium(R) CPU @ 1 with 8157508 kB of memory.

Processor Board ID SAL17267Z7R

## output above is abbreviated ##

switch# show version module 1 epld

EPLD Device Version

---------------------------------------

MI FPGA 0x11

IO FPGA 0x8

MI FPGA2 0x15

Note: For Nexus 95xx switches you will need to perform command for “each” installed hardware module. Cross reference each module command output to the versions in the release notes.

Each EPLD image that you can download from http://www.cisco.com is a bundle of EPLD upgrades. To see the updated EPLD versions for the current release, check the Cisco Nexus 9000 Series FPGA/EPLD Upgrade Release Notes for the latest EPLD release information.

http://www.cisco.com/c/en/us/support/switches/nexus-9000-series-switches/products-release-notes-list.html

If any of the versions that you list are older than what is listed in the Cisco Nexus 9000 Series FPGA/EPLD Upgrade Release Notes, we recommend that you update the EPLDs.

After comparing the above "show version module 1 epld" output of the Cisco Nexus 9396PX switch (N9K-C9396PX) with the Table 2 in release notes. The Cisco Nexus 9396PX switch needs to upgrade the EPLD firmware.

Software Download for Cisco Nexus 9000 Series Switches

http://software.cisco.com/download/navigator.html?mdfid=285954710

You can use the “show install all impact epld bootflash:” command to determine whether the EPLDs can be upgraded for all the modules or for specific modules on a switch. This command indicates the current EPLD images, new EPLD images, and whether the upgrades would be disruptive to switch operations.

The following example shows the results of using the show install all impact epld bootflash: command on a Cisco Nexus 93 xxx switch.

For example:

switch# show install all impact epld bootflash:n9000-epld.6.1.2.I2.2b.img

Compatibility check:

Module Type Upgradable Impact Reason

------ ----------------- ---------- ---------- ------

1 SUP Yes disruptive Module Upgradable

2 Expansion Yes disruptive Module Upgradable

Retrieving EPLD versions... Please wait.

Images will be upgraded according to following table:

Module Type EPLD Running-Version New-Version Upg-Required

------ ---- ------------- --------------- ----------- ------------

1 SUP MI FPGA 0x11 0x14 Yes

1 SUP IO FPGA 0x08 0x13 Yes

2 SUP MI FPGA2 0x15 0x15 No

As you can see an upgrade is required for the EPLDs on this switch. Use the "install epld bootflash:<image.img> module all" command to upgrade the EPLDs.

For example:

switch# install epld bootflash:n9000-epld.6.1.2.I2.2b.img module all

Compatibility check:

Module Type Upgradable Impact Reason

------ ----------------- ---------- ---------- ------

1 SUP Yes disruptive Module Upgradable

Retrieving EPLD versions... Please wait.

Images will be upgraded according to following table:

Module Type EPLD Running-Version New-Version Upg-Required

------ ---- ------------- --------------- ----------- ------------

1 SUP MI FPGA 0x11 0x14 Yes

1 SUP IO FPGA 0x08 0x13 Yes

2 SUP MI FPGA2 0x15 0x15 No

The above modules require upgrade.

The switch will be reloaded at the end of the upgrade

Do you want to continue (y/n) ? [n]

Select "y" to upgrade EPLDs. After Upgrading EPLDs, you can start the Migration from NXOS to ACI mode using one of the following migration methods.

Nexus 9000 Conversion Process from NXOS Standalone to ACI Mode

Method 1: Migration to ACI using NXOS configuration

Download the latest Cisco Nexus 9000 Switch ACI firmware from http://www.cisco.com

http://software.cisco.com/download/type.html?mdfid=285968390&catid=null
Select APIC Software
In the Related Software section, download the latest Cisco Nexus 9000 Series ACI-Mode Switch Firmware Release
Hover over the "Cisco Nexus 9000 Series ACI-Mode Switch Firmware Release" and screen capture the firmware details. You will use this information later to verify MD5 and file size to verify the downloaded file is a valid image.

* After downloading image, verify MD5 checksum & file size for the downloaded file. Compare the MD5 checksum & file size of the downloaded file with the MD5 Checksum & file size captured in the screenshot of file details.

For example:

osx:downloads$ md5 aci-n9000-dk9.11.0.1b.bin

MD5 (aci-n9000-dk9.11.0.1b.bin) = 06113fdb3e98c9c4446d221587c48bb0

osx:downloads$ ls -la aci*

-rw-r--r--@ 1 cisco staff 480490140 Aug 5 02:37 aci-n9000-dk9.11.0.1b.bin

* If necessary, upload the latest Cisco Nexus 9000 Series ACI-Mode Switch Firmware Release to your host server (scp, ftp, tftp, etc...)

* From the Cisco Nexus 9000 Series switch, verify IP connectivity to default gateway and server hosting the Cisco Nexus 9000 Series ACI-Mode Switch Firmware Release from the mgmt0 port. Use ping as a simple test. Once connectivity is verified, copy the latest Cisco Nexus 9000 Series ACI-Mode Switch Firmware Release to bootflash.

For example (using SCP):

switch# copy scp://10.122.254.224/FCS/aci-n9000-dk9.11.0.1b.bin bootflash:

Enter vrf (If no input, current vrf 'default' is considered): management

Enter username: scpuser

scpuser@10.122.254.224's password:

aci-n9000-dk9.11.0.1b.bin 100% 458MB 14.3MB/s 00:32

Copy complete, now saving to disk (please wait)...

* After uploading image to bootflash, verify file size for the downloaded file. Compare the file size of the uploaded file with the file size captured in the screenshot of file details.

switch# dir

480490140 Sep 05 18:34:55 2014 aci-n9000-dk9.11.0.1b.bin

* Configure Cisco Nexus 9000 Series switch to boot into loader.

- Enter configuration mode. Use "configure terminal" (config t).

- Force boot to loader. Use the "no boot nxos command"

- Save configuration. Use the "copy running-config startup-config" command

- Reload switch. Use "reload" command.

switch# configure terminal

Enter configuration commands, one per line. End with CNTL/Z.

switch(config)# no boot nxos command

switch(config)# exit

switch# copy running-config startup-config

[########################################] 100%

Copy complete.

switch# reload

This command will reboot the system. (y/n)? [n] y

2014 Sep 5 18:39:49 switch %$ VDC-1 %$ %PLATFORM-2-PFM_SYSTEM_RESET: Manual system restart from Command Line Interface

[15289.768956] [1409942399] writing reset reason 9

Note: If the Cisco Nexus 9000 Series switch does not boot to "loader", Reload the switch again and use Ctrl + C for break command to halt the boot sequence and break into "loader".

* Verify that the Cisco Nexus 9000 Series ACI-Mode Switch Firmware Release image is listed in bootflash. Use the "dir" command to verify image is present.

loader > dir

bootflash::

aci-n9000-dk9.11.0.1b.bin

* Boot Cisco Nexus 9000 Series ACI-Mode Switch Firmware Release image in bootflash. Use the "boot <aci-image.bin>" command.

loader > boot aci-n9000-dk9.11.0.1b.bin

Booting aci-n9000-dk9.11.0.1b.bin

Trying diskboot

Filesystem type is ext2fs, partition type 0x83

The PCI BIOS has not enabled this device!

Updating PCI command 0000->0007. pci_bus 05 pci_device_fn 00

PCI latency timer (CFLT) is unreasonably low at 0. Setting to 32 clocks.

Formed cmdline console=ttyS0,9600n8nn card_index=21022 mpa_index=1 loader_ver="6.10" quiet debug

Formed cmdline console=ttyS0,9600n8nn card_index=21022 mpa_index=1 loader_ver="6.10" quiet debug ksimg=aci-n9000-dk9.11.0.1b.bin

Booting kickstart image: bootflash::3:/aci-n9000-dk9.11.0.1b.bin....file_size 5220352 real_size 4460 prot_kernel_size 5204480 setupsects 30

Trying to allocate 1271 pages for VMLINUZ

Allocated prot_mode_mem 0xbd257000 real_mode_mem 0x0009e000

[Linux-EFI, setup=0x116c, size=0x4f6a00]

Read length 5204480Read length 164929536

....

## output above is abbreviated ##

*** Running INXOS PE IFC image ***

Found card_index=21022

[ 130.908083] t2usd_tor (4892) Ran 4158 msecs in last 5000 msecs

Note: You may see the "*** Running INXOS PE IFC image ***" and the console just sits there. Hit "ENTER" key to get to the "(none) login:" prompt.

* The Cisco Nexus 9000 Series switch should boot the Cisco Nexus 9000 Series ACI-Mode image. After the boot process, the switch will prompt the user for login. Use the username "admin" with "NO" password. The is no initial password for admin user. A banner is displayed informing the user that the fabric discovery is in progress.

User Access Verification

(none) login: admin

********************************************************************************

Fabric discovery in progress, show commands are not fully functional

Logout and Login after discovery to continue to use show commands.

********************************************************************************

(none)#

This completes Method 1: Migration to ACI using NXOS configuration.

Method 2: Migration to ACI using "Loader" of a non-configured Cisco Nexus 9000 Series switch

This method is used when there is "no" configuration on the Cisco Nexus 9000 Series switch and it boots into "Loader"

* Configure IP configuration for the Cisco Nexus 9000 Series switch. Use the "set ip <ip address> <netmask>" and "set gw <ip address>" commands to configure the switch IP configuration.

Loader Version 6.10

loader > set ip 10.122.254.248 255.255.255.0

Correct - ip addr is 10.122.254.248, mask is 255.255.255.0

Finding driver for NIC vendor 8086 Device 438

## output above is abbreviated ##

loader > set gw 10.122.254.1

Correct gateway addr 10.122.254.1

Address: 10.122.254.248

Netmask: 255.255.255.0

Server: 0.0.0.0

Gateway: 10.122.254.1

* Boot the Cisco Nexus 9000 Series ACI-Mode image from hosting server. For this method, Use TFTP. Use the command, "boot tftp://<tftpserver_ip>/<path>/<ACI_IMAGE.bin>".

loader > boot tftp://10.122.254.224/FCS/aci-n9000-dk9.11.0.1b.bin

Booting tftp://10.122.254.224/FCS/aci-n9000-dk9.11.0.1b.bin

Trying netboot

Address: 10.122.254.248

Netmask: 255.255.255.0

Server: 10.122.254.224

Gateway: 10.122.254.1

Filesystem type is tftp, using whole disk

PCI latency timer (CFLT) is unreasonably low at 0. Setting to 32 clocks.

.file_size 5220352 real_size 4486 prot_kernel_size 5204480 setupsects 30

Trying to allocate 1271 pages for VMLINUZ

Allocated prot_mode_mem 0xbd257000 real_mode_mem 0x0009e000

[Linux-EFI, setup=0x1186, size=0x4f6a00]

...........................................................................

Image valid

MD5Sum match

## output above is abbreviated ##

--> Rebooting

The progress of the boot process will be displayed with periods (ie ...........). Once the file is transferred, the Cisco Nexus 9000 Series switch will reboot into the Cisco Nexus 9000 Series ACI-Mode image. After the boot process, the switch will prompt the user for login. Use the username "admin" with "NO" password. The is no initial password for admin user. A banner is displayed informing the user that the fabric discovery is in progress

*** Running INXOS PE IFC image ***

Found card_index=21022

User Access Verification

(none) login: admin

********************************************************************************

Fabric discovery in progress, show commands are not fully functional

Logout and Login after discovery to continue to use show commands.

********************************************************************************

(none)#

Note: The boot TFTP may stall and you do not see any progress (....) indicators. CTRL-C to abort the boot process and repeat command until you see progress (....) indicators. Also, if you see "Retry 1: Congestion or cable issue", this is ok as long as you see progress (....) indicators after the error(s). If you return to "loader >" prompt again, repeat the entire method 2 again.

This completes Method 2: Migration to ACI using "Loader" of a non-configured Cisco Nexus 9000 Series switch.

Migration Caveats:

The Cisco Nexus 95xx Series switches may be configured and have two supervisors installed in HW slots 27 & 28. If the Cisco Nexus 95xx Series switch has two supervisors, the standby supervisor (slot 28) needs to be removed before you perform the migration Method 1 or Method 2 listed above. Perform migration Method 1 or Method 2 on the primary supervisor in slot 27 first. After completing the migration on the primary supervisor in slot 27, remove the supervisor in slot 27 and replace it with the standby supervisor. Repeat migration Method 1 or Method 2 on the standby supervisor now in slot 27. After completing the migration of the standby supervisor now in slot 27, you can reinsert the removed supervisor module into slot 28. Both supervisors are now migrated to the Cisco Nexus 9000 Series ACI-Mode image.

Verify Installed Certificates (CERTs) for Cisco Nexus 9000 Series ACI-Mode

* At the "login" prompt, login as username "admin" with "NO" password.

* At the (none)# prompt, verify that the admin user can access the necessary certificate directory and files..

- Go to directory "/securedata/ssl/"

- Check to see if the "server.crt, server.key, and server.csr" files exist

(none)# cd /securedata/ssl/

(none)# ls -al server*

lrwxrwxrwx 1 root root 40 Sep 5 20:59 server.crt -> /isan/plugin/0/securedata/ssl/server.crt

lrwxrwxrwx 1 root root 40 Sep 5 20:59 server.csr -> /isan/plugin/0/securedata/ssl/server.csr

lrwxrwxrwx 1 root root 40 Sep 5 20:59 server.key -> /isan/plugin/0/securedata/ssl/server.key

* Validate installed Certificates. Fields to verify: "ISSUER", "SERIAL NUM & PID", and "DATE RANGE". User the following commands to gather the Certificate information for EACH migrated Cisco Nexus 9000 Series ACI-Mode switch.

Commands:

openssl x509 -in /securedata/ssl/server.crt -text -noout -issuer -subject -dates | grep 'issuer\|subject\|notBefore\|notAfter'

openssl asn1parse < /securedata/ssl/server.crt | grep PRINTABLESTRING

Valid Certificates should match the following output. The "SERIAL NUM & PID" information is unique to each switch. The "ISSUER" & "DATE RANGE" fields are the key indicators for the valid Certificate.

(none)# openssl x509 -in /securedata/ssl/server.crt -text -noout -issuer -subject -dates | grep ‘issuer\|subject\|notBefore\|notAfter'

WARNING: can't open config file: /usr/lib/ssl/openssl.cnf

issuer= /O=Cisco Systems/CN=Cisco Manufacturing CA

subject= /serialNumber=PID:N9K-C9396PX SN:SAL17267Z7R/CN=SAL17267Z7R

notBefore=Aug 1 17:34:08 2014 GMT

notAfter=Aug 1 17:44:08 2024 GMT

(none)# openssl asn1parse < /securedata/ssl/server.crt | grep PRINTABLESTRING

WARNING: can't open config file: /usr/lib/ssl/openssl.cnf

51:d=5 hl=2 l= 13 prim: PRINTABLESTRING :Cisco Systems

75:d=5 hl=2 l= 22 prim: PRINTABLESTRING :Cisco Manufacturing CA

142:d=5 hl=2 l= 30 prim: PRINTABLESTRING :PID:N9K-C9396PX SN:SAL17267Z7R

183:d=5 hl=2 l= 11 prim: PRINTABLESTRING :SAL17267Z7R

If your Certificate "ISSUER" & "DATE RANGE" parameters do not match the above display output, problems may be seen when trying to join an ACI Fabric Solution. Please contact Cisco Technical Support, if you have discrepancies with installed Certificates on Cisco Nexus 9000 Series ACI-Mode switches.