Upgrading a Cisco Nexus 7000 to 6.2(16) with one Supervisor.

The Cisco Nexus 7000 series switches have several features to make upgrading firmware safe and reliable, especially if you don't have a redundant supervisor. The release notes for each software version has a table that clearly shows what previous software versions can be updated to the chosen version.

The Nexus 7000 are Linux based switches and they use kickstart to automate the installation. This means you will be copying a new n7000-s2-kickstart.x.x.xx.bin file along with the n7000-s2-dk9.x.x.xx.bin firmware file. Note that the kickstart and firmware file must be the same version.

Before the upgrade

Make a Backup of the running configuration of all VDCs and key files

The Nexus switch has a couple USB slots that mount as usb1: and USB2:. I used a 1GB flash drive formatted with the FAT file system to back up the current configuration. While not a requirement for an upgrade I like to have a backup in case anything goes wrong. The Nexus can read larger USB drives but I keep several 1GB drives handy for this type of work.

The command that backs up the running configuration of all Virtual Device Contexts (VDC) at once to the USB stick is:

• copy running-config usb1:MY-N7K.txt vdc-all

Just to be safe I copied the vlan.dat and license files to the USB drive. The license files use a .lic file extension so they are easy to identify. Once the configurations were backed up I put the USB stick into my laptop and verified that the backup was good.

I also looked at the boot settings before and after the upgrade
show boot
Current Boot Variables:

sup-1
kickstart variable = bootflash:/n7000-s2-kickstart.6.2.8a.bin
system variable = bootflash:/n7000-s2-dk9-npe.6.2.8a.bin
No module boot variable set

Download the new kickstart and Nexus software files 
You will need a CCO account and a current contract to get the software. Once the software is downloaded verify that the files are valid using the MD5 hash on the download page. If you are not familiar with verifying hashes it is very easy.

On Linux
mhubbard@1S1K-SYS76:/media/mhubbard/783E-8CFE$ md5sum n7000-s2-dk9.6.2.16.bin
f6ad2c2ea750fb15fc455d670277340c  n7000-s2-dk9.6.2.16.bin

On Windows 7
certutil -hashfile C:\tftp-root\n7000-s2-dk9.6.2.16.bin md5

or

with powershell 4 or above and the community extensions installed.
$PSVersionTable.PSVersion
PS C:\Users\mhubbard> get-hash C:\tftp-root\n7000-s2-dk9.6.2.16.bin -algorithm MD5

If you need more information on how to verify hashes on Windows you can see my blog on Using iPerf3 to verify Link Quality. Scroll down to "Installing iPerf3 on Windows".

Copy the new files to from the USB to bootflash

• copy usb1: bootflash:

n7000-s2-dk9.6.2.16.bin

• copy usb1: bootflash:

n7000-s2-kickstart.6.2.16.bin

Run the installer

install all kickstart bootflash:n7000-s2-kickstart.6.2.16.bin system bootflash:n7000-s2-dk9.6.2.16.bin parallel

The parallel keyword allows all modules to be upgraded in parallel to save time.

A lot of feed back is given during the upgrade so you can see if everything is proceeding correctly.

***********************************************************

Installer will perform compatibility check first. Please wait.

Verifying image bootflash:/n7000-s2-kickstart.6.2.16.bin for boot variable "kickstart".
[####################] 100% -- SUCCESS

Verifying image bootflash:/n7000-s2-dk9-npe.6.2.16.bin for boot variable "system".
[####################] 100% -- SUCCESS

Verifying image type.
[####################] 100% -- SUCCESS

Extracting "system" version from image bootflash:/n7000-s2-dk9-npe.6.2.16.bin.
[####################] 100% -- SUCCESS

Extracting "kickstart" version from image bootflash:/n7000-s2-kickstart.6.2.16.bin.
[####################] 100% -- SUCCESS

Extracting "bios" version from image bootflash:/n7000-s2-dk9-npe.6.2.16.bin.
[####################] 100% -- SUCCESS

Extracting "lc1n7k" version from image bootflash:/n7000-s2-dk9-npe.6.2.16.bin.
[####################] 100% -- SUCCESS

Extracting "fexth" version from image bootflash:/n7000-s2-dk9-npe.6.2.16.bin.
[####################] 100% -- SUCCESS

Performing module support checks.
[####################] 100% -- SUCCESS

Notifying services about system upgrade.
[####################] 100% -- SUCCESS



Compatibility check is done:
Module  bootable          Impact  Install-type  Reason
------  --------  --------------  ------------  ------
     1       yes      disruptive         reset  Reset due to single supervisor
     3       yes      disruptive         reset  Reset due to single supervisor
     4       yes      disruptive         reset  Reset due to single supervisor
   101       yes      disruptive         reset  Reset due to single supervisor
   102       yes      disruptive         reset  Reset due to single supervisor
   103       yes      disruptive         reset  Reset due to single supervisor



Images will be upgraded according to following table:
Module       Image                  Running-Version(pri:alt)           New-Version  Upg-Required
------  ----------  ----------------------------------------  --------------------  ------------
     1      system                                   6.2(8a)               6.2(16)           yes
     1   kickstart                                   6.2(8a)               6.2(16)           yes
     1        bios   v2.12.0(05/29/2013):v2.12.0(05/29/2013)   v2.12.0(05/29/2013)            no
     3      lc1n7k                                   6.2(8a)               6.2(16)           yes
     3        bios       v2.0.32(12/16/13):v2.0.32(12/16/13)     v2.0.32(12/16/13)            no
     4      lc1n7k                                   6.2(8a)               6.2(16)           yes
     4        bios       v2.0.32(12/16/13):v2.0.32(12/16/13)     v2.0.32(12/16/13)            no
   101       fexth                                   6.2(8a)               6.2(16)           yes
   102       fexth                                   6.2(8a)               6.2(16)           yes
   103       fexth                                   6.2(8a)               6.2(16)           yes


Additional info for this installation:
--------------------------------------

Service "lacp" in vdc 1: LACP: Upgrade will be disruptive as 0 switch ports and 10 fex ports are not upgrade ready!!
      Issue the "show lacp issu-impact" cli for more details.



Switch will be reloaded for disruptive upgrade.
Do you want to continue with the installation (y/n)?  [n] y

Install is in progress, please wait.

Performing runtime checks.
[####################] 100% -- SUCCESS

Setting boot variables.
[####################] 100% -- SUCCESS

Performing configuration copy.
[####################] 100% -- SUCCESS

Module 1:  Upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[####################] 100% -- SUCCESS

Module 3:  Upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[####################] 100% -- SUCCESS

Module 4:  Upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[####################] 100% -- SUCCESS

Finishing the upgrade, switch will reboot in 10 seconds.

*****************************************************************

Upgrade any Fabric Extenders that connected

Once the switch reloaded it will notice that the Fabric extenders need to be upgraded and will automatically start the process. You can view the progress using:

show fex detail
FEX: 102 Description: < SAN Hosts - Backup Rack >   state: Image Download
  FEX version: 6.2(8a) [Switch version: 6.2(16)]

Once the Fabric Extenders have upgraded they will automatically reload.

You can use the following command to see a log of all upgrade steps:

show install all status

References

Cisco Nexus 7000 Series NX-OS Release Notes, Release 6.2

Download Software Nexus 7000 4-Slot Switch

Boot System and Boot kickstart commands

Cisco Nexus 7000 Series NX-OS Software Upgrade and Downgrade Guide, Release 6.x

N7K-SUP2/E: eUSB Flash Failure or Unable to Save Configuration

I recently had a customer that installed a new firewall and made several changes to their Cisco Nexus 7004 core switch. Once all the configuration changes were made they issued a wr to save the changes (wr is an alias. See the references for details). But they didn't get the usual complete message. Instead, they received "Configuration update aborted: request was aborted".

You never want to see that but especially not on a core switch that is responsible for routing 72 sites! I opened a TAC case with Cisco and was told to run these four  commands and send the output back to the TAC engineer:

show module
show version
show system internal raid (Hidden Command)
slot x show system internal raid ( x = standby sup )

Note: you can use slot x show system internal raid and replace x with the slot the supervisor is in regardless if it's the standby. For example, slot 1 show system internal raid gives the same output as show system internal raid with only one supervisor.

The key output was from the command:
MY-MDF-DC1# show system internal raid
Current RAID status info:
RAID data from CMOS = 0xa5 0xc3 < ----------- Both primary and alternate failed.

and from the show module command:

Mod  Online Diag Status
---  ------------------
1    Pass
3    Pass

4    Fail

TAC said this meant that both eUSB flash memory cards were failed. Since we didn't have a redundant supervisor the only way to recover was to reboot the switch. The "Failed" eUSB memory cards aren't failed as in they don't work but that they are full. The References section below has a link to the actual bug report (CSCus22805). It explains in detail how to recover if only one eUSB is failed or in you have a redundant supervisor.

The Problem

The customer had made several configuration changes and wasn't able to save the running configuration. Obviously, all changes would be lost during the reload.

The Solution

The Nexus switch has a couple USB slots and a command that backs up the running configuration of all Virtual Device Contexts (VDC) up to the USB stick:

copy running-config usb1:MY-N7K.txt vdc-all

Once the configurations were backed up I put the USB stick into my laptop and verified that the backup was good.

Since this switch has so many routes and some of the changes that were made were routing related I wanted to make sure all routes came up after the reboot. I saved the output from:

show ip route summary
Number of routes per mask-length:
  /0 : 1       /8 : 2       /16: 82      /23: 2       /24: 113
  /25: 2       /26: 1       /27: 5       /28: 1       /29: 2
  /30: 1       /32: 788

to a text file so that I could compare after the reboot.

I also saved the output from
show interface status | i connected
show cdp ne det | i Dev

These two commands gave me a quick summary of the interfaces that were up and the neighboring switches.

Finally, I copied the all the license files and vlan.dat file to a tftp server.

The Reload

The maintenance window arrived and I had a plan in place. All that was left now was to reload. I consoled in and entered reload. The switch came back up and I reran the four commands. Show module was all "pass" and the RAID report was 0xa5 0xf0. The 0xf0 meaning the eUSB memory was working correctly.

The Clean Up

I reran the "show ip route summary" command and was missing some routes. In addition, some interface configurations were missing. This was to be expected since the changes were lost.

I ran "copy running-config usb1:MY-N7K1.txt vdc-all" and inserted the USB stick into my laptop. I use a great file diff program called MELD. I put a link to it in the references. I opened both files in MELD and it instantly highlighted the differences between the current running configuration and the backup I made before the reboot. It was a simple task to add the changes back and all routes came up.

Comparing two files in MELD

References

N7K-SUP2/E: eUSB Flash Failure or Unable to Save Configuration CSCus22805
Meld - Open source file diff tool
Write Command On Nexus Switches - How to create an alias for copy run start

Cisco 6800 Instant Access (IA) switch trunk ports

The 6800 Instant access switch allows you to extend the core switch into access closets throughout the campus. I wrote a blog on configuring the 6880-x to work with the 6800IA here.

From the 6800IA Cisco Catalyst Instant Access FAQ:

Q. What is Cisco Catalyst ® Instant Access?
A. Instant Access is a solution that uses Cisco IOS ® Software to connect Cisco ® Catalyst 6800ia access switches to Cisco Catalyst 6500 or 6800 Series core switches. Once connected, the entire configuration works as a single extended switch with a single management domain. The solution is intended to simplify your campus network operations and management.

What does that mean? It means you can connect the 6800IA to a 6880 or 6500 series core switch and manage it from the core switch. But the 6800IA isn't a standalone switch, it's a Fabric Extender (FEX) and has some limitations that a standalone switch doesn't.

Again, from the FAQ:
Q. Why is the default configuration of Instant Access client host port configuration “switch trunk allowed vlan 1” and not “all”?

A. Each Instant Access host port can be configured in access or trunk mode (default is dynamic). If in trunk mode, there is a constraint as to how many VLANs can be trunked on each port.
Note: No more than 1,000 VLANs can be associated with a single FEX ID, divided by the number of Instant Access trunk ports.

To make sure that this constraint is followed, implementation requires specifying explicitly which VLANs will be trunked. We recommend no more than 20 VLANs per Instant Access trunk port 
(up to the total of 1000 per FEX), to limit the amount of BPDU processing.

On a standalone Cisco switch, by default, a trunk port passes all VLANs. The 6800IA by default only passes VLAN1!  Here is an example of a trunk port configured to work with an access point that needs vlans 1, 4, 201, 202, 203 and 204. If you don't explicitly allow a vlan it isn't passed (other than vlan1 of course).

It is very easy to forget this if you are replacing older switches with IAs and basically copying the configs! Obviously, if the port is only passing vlan1 and you need 4, 201, 202, 203 and 204 the SSIDs won't work correctly.

interface GigabitEthernet101/1/0/1
 description < Access Point >
 switchport
 switchport trunk allowed vlan 1,4,201,202,203,204
 switchport mode trunk
 logging event trunk-status