ESXi vSwitch Configuration for Cisco Cross Stack Etherchannel

There are a lot of examples of how to set up an ESXi vSwitch to work with Cisco Etherchannel available on the Internet. But I still get asked quite often how to do it so I decided to document how I do it.

References

Best Practices for Virtual Networking
Host requirements for link aggregation for ESXi and ESX (1001938)
Sample configuration of EtherChannel / Link Aggregation Control Protocol (LACP) with ESXi/ESX and Cisco/HP switches (1004048)
VMware KB: Understanding IP Hash load balancing

The Hardware

In this example there are two Dell servers running ESXi.

There are two Cisco 2960x switches with stacking modules but this example will work with most Cisco Catalyst switches that are configured as a stack. 

Server Connections

Each server has a four built in Gb interfaces and a four port add in Gb adapter. We will use one port from each adapter for VMware management and one port from each adapter for Guest traffic. You can easily scale the example to use all eight NICs.

Two ports for VM management

• Dell Built in port 1 (vmnic0)
• Dell Add on Port far Left (vmnic7)

vSwitch0 used for Management

vmk0, vlan 54

Click on Properties… and select vSwitch, Edit.

vSwitch0

Click the NIC Teaming tab

Set Load Balancing to "Route based on IP hash"

Set Network Failover Detection to "Link status only"

Set Notify Switches to "Yes"

Set Fallback to "Yes"

Make sure vnmic0 and vmnic7 are the Active Adapters. You may have to select and then use the "Move Up" button if one of them is in standby.

Click Ok and select the Management Network, Edit.

Select the NIC Teaming tab

Set Load Balancing to "Route based on IP hash"

Set Network Failover Detection to "Link status only"

Set Notify Switches to "Yes"

Set Fallback to "Yes"

Make sure vnmic0 and vmnic7 are the Active Adapters. You may have to select and then use the "Move Up" button if one of them is in standby. Click Ok.

Two Ports for Guest Traffic

• Dell Built in port 4 (vmnic3)
• Dell Add on port Far right (vmnic4)

vSwitch1 used for Guest traffic

vmk1, vlan 50

Click on Properties… and select vSwitch, Edit.

Select the NIC Teaming tab

Set Load Balancing to "Route based on IP hash"

Set Network Failover Detection to "Link status only"

Set Notify Switches to "Yes"

Set Fallback to "Yes"

Make sure vnmic3 and vmnic4 are the Active Adapters. You may have to select and then use the "Move Up" button if one of them is in standby.

Click Ok and select the Management Network, Edit.

Select the NIC Teaming tab

Set Load Balancing to "Route based on IP hash"

Set Network Failover Detection to "Link status only"

Set Notify Switches to "Yes"

Set Fallback to "Yes"

Make sure vnmic3 and vmnic4 are the Active Adapters. You may have to select and then use the "Move Up" button if one of them is in standby.

Here's a look at the physical network adapters

That's it for the server. 

Cisco Switch Configuration

From VMware KB 1001938
The switch must be set to perform 802.3ad link aggregation in static mode ON and the virtual switch must have its load balancing method set to Route based on IP hash.
Ensure that the participating NICs are connected to the ports configured on the same physical switch or stacked switch.

If this is a new deployment you can configure the switch to use src-dst-ip for Etherchannel and connect the ESXi servers.

But if you are adding ESXi servers to an existing switch that already has Etherchannels defined you should check the current port-channel load balancing mode before making any changes (and make a backup of the current switch configuration).

Use Show Etherchannel load-balance to see the current setting. For a 2960x the default is src-mac.

TEST#sh etherchannel load-balance
EtherChannel Load-Balancing Configuration:
        src-mac

EtherChannel Load-Balancing Addresses Used Per-Protocol:
Non-IP: Source MAC address
  IPv4: Source MAC address
  IPv6: Source MAC address

You can see that the default uses the Source MAC address instead the IP address. I haven't had any problems connecting a 2960x to a 4507R+E using Etherchannel after making the change but you should be aware of what is in use before making a change.

In global configuration mode:

port-channel load-balance src-dst-ip

This sets the Etherchannel load balancing to match the ESXi vSwitch mode.

In the interface configuration we will add:

channel-group X mode on

On is used instead of Active because VMware doesn’t support negotiation. 

Switch configurations

port-channel load-balance src-dst-ip

interface Port-channel1

 description < Uplink to Core >

 switchport mode trunk

no shut

!

interface Port-channel2

 description < ESX02 management >

 switchport trunk allowed vlan 54

 switchport mode trunk

no shut

!

interface Port-channel3

 description < ESX02 VM >

 switchport mode trunk

no shut

!

interface Port-channel4

 description < ESX01 management>

 switchport trunk allowed vlan 54

 switchport mode trunk

 no shut

!

interface Port-channel5

 description < ESX01 VM >

switchport mode trunk

no shut

!

interface GigabitEthernet1/0/45

 description < ESX01-MG1 >

 switchport trunk allowed vlan 54

 switchport mode trunk

 channel-group 4 mode on

!

interface GigabitEthernet1/0/46

 description < ESX01-VM1 >

 switchport trunk allowed vlan 50

 switchport mode trunk

 channel-group 5 mode on

!

interface GigabitEthernet1/0/47

 description < ESX02-MG1 >

 switchport trunk allowed vlan 54

 switchport mode trunk

 channel-group 2 mode on

!

interface GigabitEthernet1/0/48

 description < ESX02-VM1 >

 switchport trunk allowed vlan 50

 switchport mode trunk

 channel-group 3 mode on

!

interface GigabitEthernet2/0/45

 description <  ESX01-MG2 >

 switchport trunk allowed vlan 54

 switchport mode trunk

 channel-group 4 mode on

!

interface GigabitEthernet2/0/46

 description < ESX01-VM2 >

 switchport trunk allowed vlan 50

 switchport mode trunk

 channel-group 5 mode on

!

interface GigabitEthernet2/0/47

 description <  ESX02-MG2 >

 switchport trunk allowed vlan 54

 switchport mode trunk

 channel-group 2 mode on

!

interface GigabitEthernet2/0/48

 description < ESX02-VM2 >

 switchport trunk allowed vlan 50

 switchport mode trunk

 channel-group 3 mode on

!

Show commands

• Show Etherchannel Sum – Shows status of the port channel and each port
• Show Etherchannel load-balance – Shows load balance mode. Should be src-dst-ip.
• Show Etherchannel detail – Shows detailed information about the port channel.

Show Etherchannel Sum

TEST#Show Etherchannel Sum

Flags:  D – down P - bundled in port-channel

I - stand-alone s - suspended

H - Hot-standby (LACP only)

R - Layer3      S - Layer2

U - in use      f - failed to allocate aggregator

M - not in use, minimum links not met

u - unsuitable for bundling

w - waiting to be aggregated

d - default port

Number of channel-groups in use: 5

Number of aggregators:           5

| Group | Port-channel | Protocol | Ports                   |

|:-----: |:------------: |:--------: |------------------------- |

|   1   |    Po1(SU)   |   LACP   | Te1/0/1(P)  Te2/0/1(P)   |

|   2   |    Po2(SU)   |     -     | Gi1/0/47(P) Gi2/0/47(P) |

|   3   |    Po3(SU)   |     -     | Gi1/0/48(P) Gi2/0/48(P) |

|   4   |    Po4(SU)   |     -     | Gi1/0/45(P) Gi2/0/45(P) |

|   5   |    Po5(SU)   |     -     | Gi1/0/46(P) Gi2/0/46(P) |

Show Etherchannel load-balance

TEST#Show Etherchannel load-balance

EtherChannel Load-Balancing Configuration:

        src-dst-ip

EtherChannel Load-Balancing Addresses Used Per-Protocol:

Non-IP: Source XOR Destination MAC address

  IPv4: Source XOR Destination IP address

  IPv6: Source XOR Destination IP address

Test the load balancing 

test etherchannel load-balance interface port-channel 3 <source IP> <dest IP>

TEST#test etherchannel load-balance interface port-channel 3 ip 10.26.50.100 10.26.50.101

Would select Gi1/0/48 of Po3

TEST#

TEST#test etherchannel load-balance interface port-channel 3 ip 10.26.50.100 10.26.54.102

Would select Gi2/0/47 of Po3

TEST#

Trouble Shooting

The reference link "IP Hash Load Balancing" lists the following under "Disadvantages" of using Port-channels:

Beacon probing is not supported with IP Hash. Only link status can be used as a failure detection method. If a link fails without the link state going down, there is no way to avoid network communication issues on the vSwitch.

This is a serious problem if one of the links is UP but the protocol is down. Symptoms are some servers are reachable and some aren't. The first time you run into this it can be a challenge to figure out, especially if you are remote and depending on someone on site to give you information. Keep this in the back of your mind.

Calculate the Fingerprint from an RSA Public Key

Updated July 5th, 2017.
SSH is a great protocol that encrypts traffic between the client and the server (among many other things that it does). But it doesn't provide authentication. That can leave you open to a Man in the Middle (MiTM) attack. You have probably started an SSH session to a switch, router or server and seen this dialog (or a similar one depending you the OS you are using):

The message reinforces the fact that SSH doesn't provide authentication. So what to do? 

In a previous blog Discovering SSH host keys with NMAP I showed you how to use NMAP to pull the fingerprint or full SSH key from a Cisco device. The problem here is that you still can't be sure that the device you scanned is actually the device you want to connect to.  

With newer Cisco IOS versions you can easily display the full RSA key of the device. If you are connected with a console cable there is no doubt that you are connected to the correct device. So you can record the key right after you generate it during the initial setup with a console cable.

Once you have the full key it's easy to get the fingerprint using OpenSSH on Linux/MAC. For windows users Didier Stevens over at the SANS InfoSec forums wrote a Python script you can use. His post can be found at SSH Fingerprints Are Important.

Here is the output on the switch

3750x#sh ip ssh

SSH Enabled - version 2.0

Authentication methods:publickey,keyboard-interactive,password

Encryption Algorithms:aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc,aes192-cbc,aes256-cbc

MAC Algorithms:hmac-sha1,hmac-sha1-96

Authentication timeout: 120 secs; Authentication retries: 3

Minimum expected Diffie Hellman key size : 1024 bits

IOS Keys in SECSH format(ssh-rsa, base64 encoded):

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQDQbH8FnvSpDpEsL3OzzWal1DCFBQmiImg4WWhSwlCu9mdHb53YigPEXJOlsVdX9KTKIACvwyHu5+yRQXa6+7Ahp7f98c1ac8wRW4Q8EP35kI00l+40LoC5G8eMiZ5Pz8o8UyAD/zmY1vWJzxs8k4mtdvgI4Pf5C3mmJ8r2nu0Zjw==

Some things to note here 

SSH Enabled - Version 2.0. It is critical that you don't allow your devices to use ANY version of SSH except 2.0. You do this by issuing "IP SSH Version 2" from global config mode. If you see version 1.99 then your device will accept SSH 1.0 connections and any version before 2.0 is broken.

The device will accept 3des-cbc encryption. 3des is an outdated encryption algorithm.  

Finally, the Hash Message Authentication Code (HMAC) is SHA1 which has been deprecated for SSL since the end of 2015.

Continuing on:

Copy everything from ssh-rsa to ==.

From a terminal:

Create an empty file to paste the key into. I used nano to create a file in the .ssh folder in my home directory. I called the file ios.pub. Obviously a more descriptive name would be used in production.

nano ~/.ssh/ios.pub 

Paste the key into the file. MAKE sure that the key is a continuos string. You will probably have to remove a couple line feeds after you paste the key in. Save the file.

ssh-keygen uses arguments to select the action to perform. To calculate the fingerprint we will use the following argurments:

-f filename - Filename of the key file.

-l - Show fingerprint of key file. (lowercase L)

Run this command:

ssh-keygen -lf ~/.ssh/ios.pub

1024 73:e9:e6:e3:f6:52:22:05:fc:d7:5d:d8:d7:ef:12:4d /home/mhubbard/.ssh/ios.pub (RSA)

You can see that the fingerprint matches the one in the dialog so I know for sure that I am connecting to the correct device and no MiTM is present.

This will work for any rsa .pub file. For example, you can run ssh-keygen -lf /etc/ssh/ssh_host_rsa_key >> $HOME/keys.txt on your Linux/Mac rig and keys.txt will contain the fingerprints for your rig.

cat keys.txt

1024 c1:c8:63:45:ce:56:66:c3:1e:7c:58:d9:c2:8f:28:1e  root@1S1K-DO (DSA)

256 1d:24:82:44:4f:56:40:4e:53:44:92:7a:1b:e5:43:e0  root@1S1K-DO (ECDSA)

2048 2c:84:a5:cd:81:eb:71:0d:42:10:68:c0:fe:28:3c:d4  root@1S1K-DO (RSA)

Didier Stevens from SANS wrote a cool Python 2 script for calculating the fingerprint from the key - Calculating a SSH Fingerprint From a (Cisco) Public Key | Didier Stevens. His blog explains the various parts of the key, worth reading for sure.

References
SSH Mastery: OpenSSH, PuTTY, Tunnels and Keys (ebook) - An excellent reference on SSH.
What is a SSH key fingerprint and how is it generated?

Upgrading Cisco IOS XE switches

IOS XE based switches like the 3850 series are Linux based. Compared to a 3750 switch where the tar file was under 15MB the 3850s have HUGE files. On the order of 300MB or 20 times the size of the 3750. Since the image is so large Cisco doesn't recommend using tftp to copy the image over. Here is the method TAC gave me and it has worked successfully on every switch I have used it on.

The switches can run in Bundle or install mode. Install mode is the recommended and is the method discussed in this blog.

Downloading the file

Log into Cisco.com's support site with your CCO credentials and search for the IOS XE file. Once you find the file click on the file name and copy the MD5 checksum value. You can drag your mouse over the hash to select it and then ctrl+c to copy it. We will need that to verify the file after we copy it to flash.

In our case - MD5 Checksum: 5fa29c3d9df48f882f4d6439b81bc2ce

Using SCP

I wrote a blog on using SCP a while back. You can refer to it for more detail if needed Using SCP to transfer files. On MAC/Linux SCP is built in.

On windows you will need to download an SCP tool. Since I use SuperPutty when I'm on Windows the Putty version of SCP (PSCP) is what I use. You can download it here. No install is required for PSCP and I saved it to my downloads folder.

If you prefer a GUI, WinSCP is a free open source graphical program for Windows.

On the switch, from global configuration mode, add "ip scp server enable" to enable SCP. You will need to be running a "K9" firmware version to use SCP. You can do a "sh ver | i K9" to determine if the switch is running a crypto image. Here is an example:

My-3850#sh ver | i K9
Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch  Software (cat4500es8-UNIVERSALK9-M), Version 03.06.04.E RELEASE SOFTWARE (fc2)
Hubbard_Test#

You will need the following information to copy the file:
IP Address of the switch - 10.140.68.110
Username and password of a user on the switch - mhubbard / mW!yV1s^L1Bx
The filename to copy - cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin

Once the download is complete open a cmd window, CD to the folder where you saved pscp and enter:
pscp -scp -pw mW!yV1s^L1Bx C:\tftp-root\cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin mhubbard@10.140.68.110:flash:cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin

pw - this is the password for the user. In this case mW!yV1s^L1Bx for user mhubbard
mhubbard@10.140.68.110 - mhubbard is the user on the switch and 10.140.68.110 is the IP

You will get some feedback on how fast the copy is going and a percent complete counter.
cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin | 8784 kB | 798.5 kB/s | ETA: 00:06:35 |   2%

If SCP isn't enabled on the switch you will receive this message:
Administratively disabled.
Fatal: Received unexpected end-of-file from server

Copy switch to switch

If one switch already has the firmware on it you can use SCP to copy the image to anohter switch:

My-3850#copy scp: flash:
Address or name of remote host [10.140.68.247]?
Source username [mhubbard]?
Source filename [cat3k_caa-universalk9.SPA.03.06.05.E.152-2.E5.bin]?
Destination filename [cat3k_caa-universalk9.SPA.03.06.05.E.152-2.E5.bin]?

Verifying the copy

This is the key step! Once the copy is complete log onto the switch and run a dir of flash:

My-3850#dir
Directory of flash:/

40402  -rwx     2097152  Mar 16 2016 16:58:43 -07:00  nvram_config
40403  -rw-        2574  Oct 24 2014 04:58:09 -07:00  aup.html
40404  -rw-         344  Oct 24 2014 04:58:09 -07:00  failed.html
40405  -rw-        4082  Oct 24 2014 04:58:09 -07:00  login.html
40406  -rw-         318  Oct 24 2014 04:58:09 -07:00  loginscript.js
40407  -rw-        1116  Oct 24 2014 04:58:09 -07:00  logout.html
40408  -rw-       70123  Oct 24 2014 04:58:10 -07:00  yourlogo.jpg
40409  -rw-        2846  Oct 24 2014 04:58:09 -07:00  consent.html
40410  -rw-         984  Mar 15 2016 19:18:26 -07:00  vlan.dat
40418  -rw-        1234  Oct 24 2014 04:27:48 -07:00  packages.conf.00-
48483  -rw-    82653508  Dec 30 2014 19:27:39 -08:00  cat3k_caa-base.SPA.03.06.01E.pkg
40419  -rw-        1236  Dec 30 2014 19:27:48 -08:00  packages.conf
72722  drwx        4096  Oct 24 2014 04:28:16 -07:00  mnt
40412  -rw-    82672260  Oct 24 2014 04:27:36 -07:00  cat3k_caa-base.SPA.03.06.00E.pkg
40413  -rw-     6601404  Oct 24 2014 04:27:36 -07:00  cat3k_caa-drivers.SPA.03.06.00E.pkg
40414  -rw-    33747948  Oct 24 2014 04:27:36 -07:00  cat3k_caa-infra.SPA.03.06.00E.pkg
40415  -rw-    42769724  Oct 24 2014 04:27:37 -07:00  cat3k_caa-iosd-universalk9.SPA.152-2.E.pkg
40416  -rw-    25711500  Oct 24 2014 04:27:37 -07:00  cat3k_caa-platform.SPA.03.06.00E.pkg
40417  -rw-    98462528  Oct 24 2014 04:27:37 -07:00  cat3k_caa-wcm.SPA.10.2.102.0.pkg
72723  drwx        4096  Feb 20 2016 17:09:01 -08:00  dc_profile_dir
48484  -rw-     6625980  Dec 30 2014 19:27:39 -08:00  cat3k_caa-drivers.SPA.03.06.01E.pkg
48485  -rw-    33749996  Dec 30 2014 19:27:39 -08:00  cat3k_caa-infra.SPA.03.06.01E.pkg
48486  -rw-    42827072  Dec 30 2014 19:27:39 -08:00  cat3k_caa-iosd-universalk9.SPA.152-2.E1.pkg
64644  -rw-    25727884  Dec 30 2014 19:27:39 -08:00  cat3k_caa-platform.SPA.03.06.01E.pkg
64645  -rw-    99240768  Dec 30 2014 19:27:40 -08:00  cat3k_caa-wcm.SPA.10.2.111.0.pkg
40411  -rw-   302988468  Mar 16 2016 18:03:48 -07:00  cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin

The file is in flash and the size, time/date stamp looks correct.

Now run the MD5 verify command. A lot of dots will scroll by before it completes, I truncated all but one row!

verify /md5 cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin
.............................................................................................................................................................
.........................Done!
verify /md5 (flash:cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin) = 5fa29c3d9df48f882f4d6439b81bc2ce

Compare it to the value from the Cisco site - 5fa29c3d9df48f882f4d6439b81bc2ce.

As you can see the file passed the verify operation.

Now we can start the installation:
My-3850#software install file flash:cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin new verbose
Preparing install operation ...
[1]: Copying software from active switch 1 to switch 2
[1]: Finished copying software to switch 2
[1 2]: Starting install operation
[1 2]: Expanding bundle flash:cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin
[1 2]: Copying package files
[1 2]: Package files copied
[1 2]: Finished expanding bundle flash:cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin
[1 2]: Verifying and copying expanded package files to flash:
[1 2]: Verified and copied expanded package files to flash:
[1 2]: Starting compatibility checks
[1 2]: Finished compatibility checks
[1 2]: Starting application pre-installation processing
[1 2]: Finished application pre-installation processing
[1]: Old files list:
    Removed cat3k_caa-base.SPA.03.06.01E.pkg
    Removed cat3k_caa-drivers.SPA.03.06.01E.pkg
    Removed cat3k_caa-infra.SPA.03.06.01E.pkg
    Removed cat3k_caa-iosd-universalk9.SPA.152-2.E1.pkg
    Removed cat3k_caa-platform.SPA.03.06.01E.pkg
    Removed cat3k_caa-wcm.SPA.10.2.111.0.pkg
[2]: Old files list:
    Removed cat3k_caa-base.SPA.03.06.01E.pkg
    Removed cat3k_caa-drivers.SPA.03.06.01E.pkg
    Removed cat3k_caa-infra.SPA.03.06.01E.pkg
    Removed cat3k_caa-iosd-universalk9.SPA.152-2.E1.pkg
    Removed cat3k_caa-platform.SPA.03.06.01E.pkg
    Removed cat3k_caa-wcm.SPA.10.2.111.0.pkg
[1]: New files list:
    Added cat3k_caa-base.SPA.03.06.04.E.pkg
    Added cat3k_caa-drivers.SPA.03.06.04.E.pkg
    Added cat3k_caa-infra.SPA.03.06.04.E.pkg
    Added cat3k_caa-iosd-universalk9.SPA.152-2.E4.pkg
    Added cat3k_caa-platform.SPA.03.06.04.E.pkg
    Added cat3k_caa-wcm.SPA.10.2.140.0.pkg
[2]: New files list:
    Added cat3k_caa-base.SPA.03.06.04.E.pkg
    Added cat3k_caa-drivers.SPA.03.06.04.E.pkg
    Added cat3k_caa-infra.SPA.03.06.04.E.pkg
    Added cat3k_caa-iosd-universalk9.SPA.152-2.E4.pkg
    Added cat3k_caa-platform.SPA.03.06.04.E.pkg
    Added cat3k_caa-wcm.SPA.10.2.140.0.pkg
[1 2]: Creating pending provisioning file
[1 2]: Finished installing software.  New software will load on reboot.
[1 2]: Committing provisioning file

[1 2]: Do you want to proceed with reload? [yes/no]: n

Notice that I selected no instead of yes to reload the switch. There is one more step to take before reloading. Initially I used tftp to copy and install the file in one step. I didn't verify or do this step. I ended up with one switch at rommon mode and one switch stack that was missing its configuration!

Run the following to see the packages:
My-3850#dir | i 3.06.04
40408  -rw-    82665136  Mar 15 2016 18:30:27 -07:00  cat3k_caa-base.SPA.03.06.04.E.pkg
40409  -rw-     4913852  Mar 15 2016 18:30:27 -07:00  cat3k_caa-drivers.SPA.03.06.04.E.pkg
40410  -rw-    33784816  Mar 15 2016 18:30:27 -07:00  cat3k_caa-infra.SPA.03.06.04.E.pkg
40412  -rw-    27417488  Mar 15 2016 18:30:28 -07:00  cat3k_caa-platform.SPA.03.06.04.E.pkg

My-3850#more flash:packages.conf
#! /usr/binos/bin/packages_conf.sh

Notice that this command runs a Linux shell command.

When it finishes make sure that all the packages are set to the 03.06.04 pkg.

sha1sum: b67332dea64aae6b5d80f92b19713413707f9c27
iso   rp 0 0   rp_base       cat3k_caa-base.SPA.03.06.04.E.pkg
iso   rp 0 0   rp_infra       cat3k_caa-infra.SPA.03.06.04.E.pkg
iso   rp 0 0   rp_platform       cat3k_caa-platform.SPA.03.06.04.E.pkg
iso   rp 0 0   rp_iosd       cat3k_caa-iosd-universalk9.SPA.152-2.E4.pkg
iso   rp 0 0   rp_wcm       cat3k_caa-wcm.SPA.10.2.140.0.pkg
iso   rp 0 0   drivers       cat3k_caa-drivers.SPA.03.06.04.E.pkg

#
# -start- superpackage .pkginfo
#
# pkginfo: Name: rp_super
# pkginfo: PackageFileType: iso
# pkginfo: BuildTime: Sat Feb 13 04:00:36 PST 2016
# pkginfo: ReleaseDate: Sat Feb 13 04:00:36 PST 2016
# pkginfo: .BuildArch: mips
# pkginfo: RouteProcessor: mips
# pkginfo: Platform: ng3k
# pkginfo: User: abhakat
# pkginfo: PackageName: cat3k_caa-universalk9
# pkginfo: Build: 03.06.04.E
# pkginfo: Dependencies: PROVIDES:cat3k_caa-base,03.06.04.E,mips;cat3k_caa-infra,03.06.04.E,mips;cat3k_caa-platform,03.06.04.E,mips;cat3k_caa-iosd-universalk9,152-
2.E4,mips;cat3k_caa-wcm,10.2.140.0,mips;cat3k_caa-drivers,03.06.04.E,mips;
# pkginfo: .SupportedBoards: unknown
# pkginfo: .BuildPath: unknown
# pkginfo: BuildType: Production
#
# -end- superpackage .pkginfo
#

At this point reload and go get a cup of coffee (or two) while it restarts!

But what if they don't match up? 

I asked TAC and here is the response:
**************************************************
After the package files are expanded and copied to flash, the running provisioning file (flash:packages.conf) is updated to reflect the newly installed packages, and the controller displays a reload prompt.

If this is not pointing to the correct version, run the software install command again and make sure you are not getting a new provisioning file, sometimes the .conf file name is changed and we need to modify the boot variable to the correct provisioned file.

Cisco refers to “packages.conf” as the provision file. Sometimes when we expand the images this file name might change, at the moment of the expand the same system let you know about the file name change, when this happen the only thing you need to do is to modify the boot variable to point to the new file.

Watch the install messages closely and if you see a message stating that the packages.conf file name has changed you MUST update the boot variable before reloading. If you don't the switch will reload to the switch: prompt.
**************************************************

Note: you will see packages.conf.00- if this is the first upgrade and packages.conf.01- if it's the second and so on. You can run more (a standard Linux command!) on these files just as packages.conf.

A Failed Upgrade

I mentioned earlier that before I started following this procedure I had a switch that booted to switch: after the upgrade. Here is a dir from that switch. You can see that the packages have 0 bytes. TAC said that was probably because I used tftp for the transfer and the .bin was corrupt.
switch: dir flash:
Directory of flash:/

16161  drwx  4096       .
    2  drwx  4096       ..
16162  -rwx  2097152    nvram_config
16163  -r--  0          vlan.dat
16164  -rw-  1236       packages.conf.00-
    0  -rw-  1236       cat3k_caa-base.SPA.03.06.01E.pkg
    0  -rw-  1236       cat3k_caa-drivers.SPA.03.06.01E.pkg
    0  -rw-  1236       cat3k_caa-infra.SPA.03.06.01E.pkg
    0  -rw-  1236       cat3k_caa-iosd-universalk9.SPA.152-2.E1.pkg
16184  -rw-  1234       packages.conf.01-
16185  -rw-  1236       packages.conf
16170  drwx  4096       mnt
    0  drwx  4096       cat3k_caa-base.SPA.03.06.00E.pkg
    0  drwx  4096       cat3k_caa-drivers.SPA.03.06.00E.pkg
    0  drwx  4096       cat3k_caa-infra.SPA.03.06.00E.pkg
    0  drwx  4096       cat3k_caa-iosd-universalk9.SPA.152-2.E.pkg
    0  drwx  4096       cat3k_caa-platform.SPA.03.06.00E.pkg
    0  drwx  4096       cat3k_caa-wcm.SPA.10.2.102.0.pkg
16169  -rw-  2574       aup.html
16173  -rw-  344        failed.html
16174  -rw-  4082       login.html
16175  -rw-  318        loginscript.js
16176  -rw-  1116       logout.html
16177  -rw-  577536     yourlogo.jpg
16178  -rw-  2846       consent.html
16179  drwx  4096       dc_profile_dir
    0  drwx  4096       cat3k_caa-platform.SPA.03.06.01E.pkg
    0  drwx  4096       cat3k_caa-wcm.SPA.10.2.111.0.pkg
    0  drwx  4096       cat3k_caa-base.SPA.03.06.03E.pkg
    0  drwx  4096       cat3k_caa-drivers.SPA.03.06.03E.pkg
    0  drwx  4096       cat3k_caa-infra.SPA.03.06.03E.pkg
    0  drwx  4096       cat3k_caa-iosd-universalk9.SPA.152-2.E3.pkg
    0  drwx  4096       cat3k_caa-platform.SPA.03.06.03E.pkg
    0  drwx  4096       cat3k_caa-wcm.SPA.10.2.131.0.pkg

1557549056 bytes available (29929472 bytes used)

Operation timed out waiting for switch 2

I had this happen recently. I verified the copy but the install timed out on switch 2.

software install file flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin
Preparing install operation ...
[1]: Copying software from active switch 1 to switch 2
[1]: Finished copying software to switch 2
[1 2]: Starting install operation
[1 2]: Expanding bundle flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin
[1 2]: Copying package files
[1 2]: Package files copied
[1 2]: Finished expanding bundle flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin
[1]: % Operation timed out waiting for switch 2 to respond. Operation aborted.

It didn't say what operation timed out so I ran a dir on switch 2:

dir flash-2:
Directory of flash-2:/

 8083  -rwx     2097152   Jul 9 2019 14:14:23 -07:00  nvram_config
 8084  -rw-        1344  Feb 27 2019 00:21:09 -08:00  vlan.dat
 8082  -rw-   302112348  Jun 27 2019 15:33:33 -07:00  cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin
 8101  -rw-        1236   Mar 7 2017 15:45:15 -08:00  packages.conf
40402  drwx        4096  Sep 11 2014 03:06:13 -07:00  mnt
40403  drwx        4096  Feb 27 2019 00:20:01 -08:00  dc_profile_dir
 8089  -rw-        2574  Sep 11 2014 03:39:08 -07:00  aup.html
 8092  -rw-         344  Sep 11 2014 03:39:08 -07:00  failed.html
 8093  -rw-        4082  Sep 11 2014 03:39:03 -07:00  login.html
 8094  -rw-         318  Sep 11 2014 03:39:08 -07:00  loginscript.js
 8095  -rw-        1116  Sep 11 2014 03:39:08 -07:00  logout.html
 8096  -rw-       70123  Sep 11 2014 03:39:08 -07:00  yourlogo.jpg
 8097  -rw-        2846  Sep 11 2014 03:39:08 -07:00  consent.html
 8103  -rw-    83293932   Mar 7 2017 15:45:12 -08:00  cat3k_caa-base.SPA.03.06.06E.pkg
 8104  -rw-     3982012   Mar 7 2017 15:45:12 -08:00  cat3k_caa-drivers.SPA.03.06.06E.pkg
 8105  -rw-    33788908   Mar 7 2017 15:45:12 -08:00  cat3k_caa-infra.SPA.03.06.06E.pkg
 8106  -rw-    43074880   Mar 7 2017 15:45:13 -08:00  cat3k_caa-iosd-universalk9.SPA.152-2.E6.pkg
16161  -rw-    28394380   Mar 7 2017 15:45:13 -08:00  cat3k_caa-platform.SPA.03.06.06E.pkg
16162  -rw-   111233856   Mar 7 2017 15:45:13 -08:00  cat3k_caa-wcm.SPA.10.2.160.0.pkg 

The .bin file is on switch 2 and the file size looks correct. I ran "more packages.conf" and "dir pack*.*" and everything looked  go to boot into the old image. I reloaded and then the install completed successfully.

The switch was on 3.6.6 and hadn't been rebooted in over a year. I think that 3.6.6 may have a memory leak because I have had issues with this version where the console isn't reponsive or other issues and a reload always corrects it. 

References

Catalyst 3850 Series Switch Upgrade, Management, and Recovery Techniques - If you have a switch that boots to the "switch:" prompt you can follow the instructions in this document to recover.

Configure Network Devices with Excel and Word Mail Merge

If you have several of the same type devices to configure it will be faster and more accurate to create a template and then use a Word feature called "Mail Merge" to create the actual configurations. Mail Merge can be linked to an Excel spread sheet to pull in the data.

For example, assume we have a company with 5 data closets and each closet uses a different vlan for the access ports. A customer requirement is to have the switch serial number configured as an SNMP location. The customer also has a standard for the hostname and uses RADIUS authentication against a Microsoft NPS server. Additionally, you use SuperPutty to manage the switches.

In Excel you can create a formula to build the NPS server script and the SuperPutty string used in the XML file. If you haven't used Excel to create formulas you should download my sample file and look it over. Excel is a great tool for network configuration once you get formulas down. A simple google for Excel calculation will turn up many websites with examples.

The items in the header row become mail merge items in Word. Here are several of the rows from the sample spread sheet.

With the spread sheet created it's time to build the Word document for the switch configuration. Once that is complete click the "Mailings" menu. That brings up several new selections in the Word Ribbon. Now click on "Select Recipients and Use Existing List..."

A File Open dialog box with open. It ALWAYS opens in "C:\Program Files (x86)\Microsoft Office\Office14\QUERIES" which is very annoying if you use a cloud storage provider for your files. Anyway, navigate to the Excel spread sheet and select it. In my case I have more than one work sheet defined so I saw the following dialog:

This dialog let's you select the workbook to use with the merge. In this case I have two workbooks defined - SITE1 and SITE2. If I am configuring switches for site1 I would obviously select SITE1. Same logic if I was configuring site 2.

Now that the spread sheet is linked to the Word document we can insert the mail merge fields. Simply navigate to where you need a field then click "Mailings, Insert Merge Field" and select the field. Here is an example of inserting the HostName field: 

hostname «HostName». 

When the mail merge is executed the «HostName» field will be replaced with the value from the spread sheet. You can create as many merge fields as you need. 

Execute the Merge

Once you have added all the merge fields and saved the document you are ready to run it. Click the "Mailings Menu, then Finish and Merge, Edit Individual Documents...

A dialog will open allowing you to select All, Current or From.

The first column in my work book is named item. I don't use it in my switch configuration, it is in the work book so that if I want to configure only one switch I can pick it out easily. To configure the first two switches you would select From: and enter 1 and 2. 

As soon as you click Ok the merge will run and a new Word document will open. It will be named Letters1 and will contain all of the configurations. At this point I normally click ctrl+a and then ctrl+x to select all the text and cut it. I then paste it into a notepad to remove any hidden characters that Word may have inserted.

Below is a link to a sample Word document and spread sheet. You will need to change the column headers to meet your needs. If  you start with my spread sheet and change the headers you are prompted when running the merge that fields were not found. Simply click "Remove from document" and Word will remove my fields.

I encourage you to download my samples and try it. Once you get the hang of it you will find it's much better than using "Search and replace" to create configurations. It also allows you to "Outsource" creating configs because you can create the template, fill out the spread sheet and then send it to a co-worker who can do the actual configuring. Plus, you can get peer review on the spread sheet to reduce the chance of error.

Sample template and spread sheet

The sample spread sheet has columns for SuperPutty and Network Policy Server.

SuperPutty

To use the SuperPutty item simply edit the formula (use caution as there are a lot of """ character sets in it) to match your SuperPutty format. Then close SuperPutty, open the sessions.xml file in the SuperPutty folder and paste it in. I use Notepad++ and it color codes the XML file making it easier to view. Save the changes and then close the file. When you re-open SuperPutty you will find a folder with the new switches in it.

Microsoft Network Policy Server

Setting up a Microsoft Network Policy Server is very easy and will give you a free RADIUS server for switch (or 802.1x) Authentication. Server Standard is limited to 50 devices, Data Center is unlimited.

It is a Cisco Best Practice to use RADIUS (or TACACS+) for authentication. If you are using NPS the last column in my spread sheet creates the code needed to configure the device in NPS. Simply open a command line as an administrator and paste the code in. Super simple and it will save a lot of time over manually creating devices in NPS.

My Template
I always try to follow Cisco best practices whenever possible. As you look at the template hopefully you will find some best practices that you can implement in your environment.