Upgrade Cisco 2960 firmware using SCP instead of TFTP

I recently needed to upgrade a couple dozen Cisco 2960-x switches located at several different sites. Obviously, I didn't want to drive to each site and insert flash drives to do the upgrade but I also didn't want to use TFTP and risk a failed transfer causing a reboot issue.

The solution was to use Secure Copy or SCP. Unlike TFTP, SCP is TCP based so you get the advantages of a TCP - Reliability, Flow Control, error checking and error recovery. SCP adds encryption. While encryption isn't so important for a firmware transfer it is important when transferring configurations so that the passwords aren't sent in plain text.

The drawback to SCP is that you need to set up an SCP server and create a username/password combination. For Windows, the Solarwinds SFTP/SCP server is free and works well. It supports secure FTP and secure copy protocols. You can download the server here. While free, it does require that you enter a valid email address. Once you have it installed, click on configure and create a new user/password.

On Ubuntu, you can use the OpenSSH server from the Ubuntu repository. For a How To on configuring the server click here.

Once you have the SCP server installed, download the tar file from Cisco and save it to the SCP servers root directory. For the Solarwinds server, the default is c:\sftp-root. For Ubuntu, you will specify the path after the IP address.

I recommend you check the hash of the firmware file after you download it from Cisco. To do that, simply click on the file name on the Cisco download page. You will see the MD5 hash on the popup, right click, copy and paste it into an editor.

On windows
From a cmd window run "certutil -hashfile <filename> -MD5.

Linux
From a terminal windows run md5sum <filename>.

Compare the result to the hash saved in the text file. If the hashes match, proceed to the next step.

On the switch

You will need to set up SCP first. See my blog Using Secure Copy (SCP) to Transfer Files for a tutorial on setting up SCP. 

Once that is complete, log in and go to enable mode.

Enter the following (substitute the appropriate IP address, filename, and credentials). The switch puts the password: prompt on the screen but the password will be passed automatically. Since I used the /reload switch no further intervention is required - the firmware will be upgraded and the switch will reload. You can kick off several switches in parallel as the load on the SCP server isn't too heavy. Using an Ubuntu desktop VM with the defaults set in ESXi I have upgraded 10 switches at once with no problems.

archive download-sw /overwrite /reload scp://mhubbard:hU9*b2Sis@192.168.10.221/c2960x-universalk9-tar.152-2.E7.tar
Password:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Password:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
examining image...
extracting info (109 bytes)
extracting c2960x-universalk9-mz.152-2.E7/info (786 bytes)
extracting info (109 bytes)

Stacking Version Number: 1.56

System Type:             0x00000000
  Ios Image File Size:   0x0144DA00
  Total Image File Size: 0x019ACA00
  Minimum Dram required: 0x08000000
  Image Suffix:          universalk9-152-2.E7
  Image Directory:       c2960x-universalk9-mz.152-2.E7
  Image Name:            c2960x-universalk9-mz.152-2.E7.bin
  Image Feature:         IP|LAYER_2|SSH|3DES|MIN_DRAM_MEG=128
  FRU Module Version:    No FRU Version Specified

References

Using Secure Copy (SCP) to Transfer Files
2960-X Switch System Management Command Reference, Cisco IOS Release 15.0(2)EX
Configuring Secure Shell (SSH)
Security Configuration Guide
10 SCP Commands to Transfer Files/Folders in Linux
Example syntax for Secure Copy (scp)

Upgrading Cisco IOS XE switches

IOS XE based switches like the 3850 series are Linux based. Compared to a 3750 switch where the tar file was under 15MB the 3850s have HUGE files. On the order of 300MB or 20 times the size of the 3750. Since the image is so large Cisco doesn't recommend using tftp to copy the image over. Here is the method TAC gave me and it has worked successfully on every switch I have used it on.

The switches can run in Bundle or install mode. Install mode is the recommended and is the method discussed in this blog.

Downloading the file

Log into Cisco.com's support site with your CCO credentials and search for the IOS XE file. Once you find the file click on the file name and copy the MD5 checksum value. You can drag your mouse over the hash to select it and then ctrl+c to copy it. We will need that to verify the file after we copy it to flash.

In our case - MD5 Checksum: 5fa29c3d9df48f882f4d6439b81bc2ce

Using SCP

I wrote a blog on using SCP a while back. You can refer to it for more detail if needed Using SCP to transfer files. On MAC/Linux SCP is built in.

On windows you will need to download an SCP tool. Since I use SuperPutty when I'm on Windows the Putty version of SCP (PSCP) is what I use. You can download it here. No install is required for PSCP and I saved it to my downloads folder.

If you prefer a GUI, WinSCP is a free open source graphical program for Windows.

On the switch, from global configuration mode, add "ip scp server enable" to enable SCP. You will need to be running a "K9" firmware version to use SCP. You can do a "sh ver | i K9" to determine if the switch is running a crypto image. Here is an example:

My-3850#sh ver | i K9
Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch  Software (cat4500es8-UNIVERSALK9-M), Version 03.06.04.E RELEASE SOFTWARE (fc2)
Hubbard_Test#

You will need the following information to copy the file:
IP Address of the switch - 10.140.68.110
Username and password of a user on the switch - mhubbard / mW!yV1s^L1Bx
The filename to copy - cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin

Once the download is complete open a cmd window, CD to the folder where you saved pscp and enter:
pscp -scp -pw mW!yV1s^L1Bx C:\tftp-root\cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin mhubbard@10.140.68.110:flash:cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin

pw - this is the password for the user. In this case mW!yV1s^L1Bx for user mhubbard
mhubbard@10.140.68.110 - mhubbard is the user on the switch and 10.140.68.110 is the IP

You will get some feedback on how fast the copy is going and a percent complete counter.
cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin | 8784 kB | 798.5 kB/s | ETA: 00:06:35 |   2%

If SCP isn't enabled on the switch you will receive this message:
Administratively disabled.
Fatal: Received unexpected end-of-file from server

Copy switch to switch

If one switch already has the firmware on it you can use SCP to copy the image to anohter switch:

My-3850#copy scp: flash:
Address or name of remote host [10.140.68.247]?
Source username [mhubbard]?
Source filename [cat3k_caa-universalk9.SPA.03.06.05.E.152-2.E5.bin]?
Destination filename [cat3k_caa-universalk9.SPA.03.06.05.E.152-2.E5.bin]?

Verifying the copy

This is the key step! Once the copy is complete log onto the switch and run a dir of flash:

My-3850#dir
Directory of flash:/

40402  -rwx     2097152  Mar 16 2016 16:58:43 -07:00  nvram_config
40403  -rw-        2574  Oct 24 2014 04:58:09 -07:00  aup.html
40404  -rw-         344  Oct 24 2014 04:58:09 -07:00  failed.html
40405  -rw-        4082  Oct 24 2014 04:58:09 -07:00  login.html
40406  -rw-         318  Oct 24 2014 04:58:09 -07:00  loginscript.js
40407  -rw-        1116  Oct 24 2014 04:58:09 -07:00  logout.html
40408  -rw-       70123  Oct 24 2014 04:58:10 -07:00  yourlogo.jpg
40409  -rw-        2846  Oct 24 2014 04:58:09 -07:00  consent.html
40410  -rw-         984  Mar 15 2016 19:18:26 -07:00  vlan.dat
40418  -rw-        1234  Oct 24 2014 04:27:48 -07:00  packages.conf.00-
48483  -rw-    82653508  Dec 30 2014 19:27:39 -08:00  cat3k_caa-base.SPA.03.06.01E.pkg
40419  -rw-        1236  Dec 30 2014 19:27:48 -08:00  packages.conf
72722  drwx        4096  Oct 24 2014 04:28:16 -07:00  mnt
40412  -rw-    82672260  Oct 24 2014 04:27:36 -07:00  cat3k_caa-base.SPA.03.06.00E.pkg
40413  -rw-     6601404  Oct 24 2014 04:27:36 -07:00  cat3k_caa-drivers.SPA.03.06.00E.pkg
40414  -rw-    33747948  Oct 24 2014 04:27:36 -07:00  cat3k_caa-infra.SPA.03.06.00E.pkg
40415  -rw-    42769724  Oct 24 2014 04:27:37 -07:00  cat3k_caa-iosd-universalk9.SPA.152-2.E.pkg
40416  -rw-    25711500  Oct 24 2014 04:27:37 -07:00  cat3k_caa-platform.SPA.03.06.00E.pkg
40417  -rw-    98462528  Oct 24 2014 04:27:37 -07:00  cat3k_caa-wcm.SPA.10.2.102.0.pkg
72723  drwx        4096  Feb 20 2016 17:09:01 -08:00  dc_profile_dir
48484  -rw-     6625980  Dec 30 2014 19:27:39 -08:00  cat3k_caa-drivers.SPA.03.06.01E.pkg
48485  -rw-    33749996  Dec 30 2014 19:27:39 -08:00  cat3k_caa-infra.SPA.03.06.01E.pkg
48486  -rw-    42827072  Dec 30 2014 19:27:39 -08:00  cat3k_caa-iosd-universalk9.SPA.152-2.E1.pkg
64644  -rw-    25727884  Dec 30 2014 19:27:39 -08:00  cat3k_caa-platform.SPA.03.06.01E.pkg
64645  -rw-    99240768  Dec 30 2014 19:27:40 -08:00  cat3k_caa-wcm.SPA.10.2.111.0.pkg
40411  -rw-   302988468  Mar 16 2016 18:03:48 -07:00  cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin

The file is in flash and the size, time/date stamp looks correct.

Now run the MD5 verify command. A lot of dots will scroll by before it completes, I truncated all but one row!

verify /md5 cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin
.............................................................................................................................................................
.........................Done!
verify /md5 (flash:cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin) = 5fa29c3d9df48f882f4d6439b81bc2ce

Compare it to the value from the Cisco site - 5fa29c3d9df48f882f4d6439b81bc2ce.

As you can see the file passed the verify operation.

Now we can start the installation:
My-3850#software install file flash:cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin new verbose
Preparing install operation ...
[1]: Copying software from active switch 1 to switch 2
[1]: Finished copying software to switch 2
[1 2]: Starting install operation
[1 2]: Expanding bundle flash:cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin
[1 2]: Copying package files
[1 2]: Package files copied
[1 2]: Finished expanding bundle flash:cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin
[1 2]: Verifying and copying expanded package files to flash:
[1 2]: Verified and copied expanded package files to flash:
[1 2]: Starting compatibility checks
[1 2]: Finished compatibility checks
[1 2]: Starting application pre-installation processing
[1 2]: Finished application pre-installation processing
[1]: Old files list:
    Removed cat3k_caa-base.SPA.03.06.01E.pkg
    Removed cat3k_caa-drivers.SPA.03.06.01E.pkg
    Removed cat3k_caa-infra.SPA.03.06.01E.pkg
    Removed cat3k_caa-iosd-universalk9.SPA.152-2.E1.pkg
    Removed cat3k_caa-platform.SPA.03.06.01E.pkg
    Removed cat3k_caa-wcm.SPA.10.2.111.0.pkg
[2]: Old files list:
    Removed cat3k_caa-base.SPA.03.06.01E.pkg
    Removed cat3k_caa-drivers.SPA.03.06.01E.pkg
    Removed cat3k_caa-infra.SPA.03.06.01E.pkg
    Removed cat3k_caa-iosd-universalk9.SPA.152-2.E1.pkg
    Removed cat3k_caa-platform.SPA.03.06.01E.pkg
    Removed cat3k_caa-wcm.SPA.10.2.111.0.pkg
[1]: New files list:
    Added cat3k_caa-base.SPA.03.06.04.E.pkg
    Added cat3k_caa-drivers.SPA.03.06.04.E.pkg
    Added cat3k_caa-infra.SPA.03.06.04.E.pkg
    Added cat3k_caa-iosd-universalk9.SPA.152-2.E4.pkg
    Added cat3k_caa-platform.SPA.03.06.04.E.pkg
    Added cat3k_caa-wcm.SPA.10.2.140.0.pkg
[2]: New files list:
    Added cat3k_caa-base.SPA.03.06.04.E.pkg
    Added cat3k_caa-drivers.SPA.03.06.04.E.pkg
    Added cat3k_caa-infra.SPA.03.06.04.E.pkg
    Added cat3k_caa-iosd-universalk9.SPA.152-2.E4.pkg
    Added cat3k_caa-platform.SPA.03.06.04.E.pkg
    Added cat3k_caa-wcm.SPA.10.2.140.0.pkg
[1 2]: Creating pending provisioning file
[1 2]: Finished installing software.  New software will load on reboot.
[1 2]: Committing provisioning file

[1 2]: Do you want to proceed with reload? [yes/no]: n

Notice that I selected no instead of yes to reload the switch. There is one more step to take before reloading. Initially I used tftp to copy and install the file in one step. I didn't verify or do this step. I ended up with one switch at rommon mode and one switch stack that was missing its configuration!

Run the following to see the packages:
My-3850#dir | i 3.06.04
40408  -rw-    82665136  Mar 15 2016 18:30:27 -07:00  cat3k_caa-base.SPA.03.06.04.E.pkg
40409  -rw-     4913852  Mar 15 2016 18:30:27 -07:00  cat3k_caa-drivers.SPA.03.06.04.E.pkg
40410  -rw-    33784816  Mar 15 2016 18:30:27 -07:00  cat3k_caa-infra.SPA.03.06.04.E.pkg
40412  -rw-    27417488  Mar 15 2016 18:30:28 -07:00  cat3k_caa-platform.SPA.03.06.04.E.pkg

My-3850#more flash:packages.conf
#! /usr/binos/bin/packages_conf.sh

Notice that this command runs a Linux shell command.

When it finishes make sure that all the packages are set to the 03.06.04 pkg.

sha1sum: b67332dea64aae6b5d80f92b19713413707f9c27
iso   rp 0 0   rp_base       cat3k_caa-base.SPA.03.06.04.E.pkg
iso   rp 0 0   rp_infra       cat3k_caa-infra.SPA.03.06.04.E.pkg
iso   rp 0 0   rp_platform       cat3k_caa-platform.SPA.03.06.04.E.pkg
iso   rp 0 0   rp_iosd       cat3k_caa-iosd-universalk9.SPA.152-2.E4.pkg
iso   rp 0 0   rp_wcm       cat3k_caa-wcm.SPA.10.2.140.0.pkg
iso   rp 0 0   drivers       cat3k_caa-drivers.SPA.03.06.04.E.pkg

#
# -start- superpackage .pkginfo
#
# pkginfo: Name: rp_super
# pkginfo: PackageFileType: iso
# pkginfo: BuildTime: Sat Feb 13 04:00:36 PST 2016
# pkginfo: ReleaseDate: Sat Feb 13 04:00:36 PST 2016
# pkginfo: .BuildArch: mips
# pkginfo: RouteProcessor: mips
# pkginfo: Platform: ng3k
# pkginfo: User: abhakat
# pkginfo: PackageName: cat3k_caa-universalk9
# pkginfo: Build: 03.06.04.E
# pkginfo: Dependencies: PROVIDES:cat3k_caa-base,03.06.04.E,mips;cat3k_caa-infra,03.06.04.E,mips;cat3k_caa-platform,03.06.04.E,mips;cat3k_caa-iosd-universalk9,152-
2.E4,mips;cat3k_caa-wcm,10.2.140.0,mips;cat3k_caa-drivers,03.06.04.E,mips;
# pkginfo: .SupportedBoards: unknown
# pkginfo: .BuildPath: unknown
# pkginfo: BuildType: Production
#
# -end- superpackage .pkginfo
#

At this point reload and go get a cup of coffee (or two) while it restarts!

But what if they don't match up? 

I asked TAC and here is the response:
**************************************************
After the package files are expanded and copied to flash, the running provisioning file (flash:packages.conf) is updated to reflect the newly installed packages, and the controller displays a reload prompt.

If this is not pointing to the correct version, run the software install command again and make sure you are not getting a new provisioning file, sometimes the .conf file name is changed and we need to modify the boot variable to the correct provisioned file.

Cisco refers to “packages.conf” as the provision file. Sometimes when we expand the images this file name might change, at the moment of the expand the same system let you know about the file name change, when this happen the only thing you need to do is to modify the boot variable to point to the new file.

Watch the install messages closely and if you see a message stating that the packages.conf file name has changed you MUST update the boot variable before reloading. If you don't the switch will reload to the switch: prompt.
**************************************************

Note: you will see packages.conf.00- if this is the first upgrade and packages.conf.01- if it's the second and so on. You can run more (a standard Linux command!) on these files just as packages.conf.

A Failed Upgrade

I mentioned earlier that before I started following this procedure I had a switch that booted to switch: after the upgrade. Here is a dir from that switch. You can see that the packages have 0 bytes. TAC said that was probably because I used tftp for the transfer and the .bin was corrupt.
switch: dir flash:
Directory of flash:/

16161  drwx  4096       .
    2  drwx  4096       ..
16162  -rwx  2097152    nvram_config
16163  -r--  0          vlan.dat
16164  -rw-  1236       packages.conf.00-
    0  -rw-  1236       cat3k_caa-base.SPA.03.06.01E.pkg
    0  -rw-  1236       cat3k_caa-drivers.SPA.03.06.01E.pkg
    0  -rw-  1236       cat3k_caa-infra.SPA.03.06.01E.pkg
    0  -rw-  1236       cat3k_caa-iosd-universalk9.SPA.152-2.E1.pkg
16184  -rw-  1234       packages.conf.01-
16185  -rw-  1236       packages.conf
16170  drwx  4096       mnt
    0  drwx  4096       cat3k_caa-base.SPA.03.06.00E.pkg
    0  drwx  4096       cat3k_caa-drivers.SPA.03.06.00E.pkg
    0  drwx  4096       cat3k_caa-infra.SPA.03.06.00E.pkg
    0  drwx  4096       cat3k_caa-iosd-universalk9.SPA.152-2.E.pkg
    0  drwx  4096       cat3k_caa-platform.SPA.03.06.00E.pkg
    0  drwx  4096       cat3k_caa-wcm.SPA.10.2.102.0.pkg
16169  -rw-  2574       aup.html
16173  -rw-  344        failed.html
16174  -rw-  4082       login.html
16175  -rw-  318        loginscript.js
16176  -rw-  1116       logout.html
16177  -rw-  577536     yourlogo.jpg
16178  -rw-  2846       consent.html
16179  drwx  4096       dc_profile_dir
    0  drwx  4096       cat3k_caa-platform.SPA.03.06.01E.pkg
    0  drwx  4096       cat3k_caa-wcm.SPA.10.2.111.0.pkg
    0  drwx  4096       cat3k_caa-base.SPA.03.06.03E.pkg
    0  drwx  4096       cat3k_caa-drivers.SPA.03.06.03E.pkg
    0  drwx  4096       cat3k_caa-infra.SPA.03.06.03E.pkg
    0  drwx  4096       cat3k_caa-iosd-universalk9.SPA.152-2.E3.pkg
    0  drwx  4096       cat3k_caa-platform.SPA.03.06.03E.pkg
    0  drwx  4096       cat3k_caa-wcm.SPA.10.2.131.0.pkg

1557549056 bytes available (29929472 bytes used)

Operation timed out waiting for switch 2

I had this happen recently. I verified the copy but the install timed out on switch 2.

software install file flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin
Preparing install operation ...
[1]: Copying software from active switch 1 to switch 2
[1]: Finished copying software to switch 2
[1 2]: Starting install operation
[1 2]: Expanding bundle flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin
[1 2]: Copying package files
[1 2]: Package files copied
[1 2]: Finished expanding bundle flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin
[1]: % Operation timed out waiting for switch 2 to respond. Operation aborted.

It didn't say what operation timed out so I ran a dir on switch 2:

dir flash-2:
Directory of flash-2:/

 8083  -rwx     2097152   Jul 9 2019 14:14:23 -07:00  nvram_config
 8084  -rw-        1344  Feb 27 2019 00:21:09 -08:00  vlan.dat
 8082  -rw-   302112348  Jun 27 2019 15:33:33 -07:00  cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin
 8101  -rw-        1236   Mar 7 2017 15:45:15 -08:00  packages.conf
40402  drwx        4096  Sep 11 2014 03:06:13 -07:00  mnt
40403  drwx        4096  Feb 27 2019 00:20:01 -08:00  dc_profile_dir
 8089  -rw-        2574  Sep 11 2014 03:39:08 -07:00  aup.html
 8092  -rw-         344  Sep 11 2014 03:39:08 -07:00  failed.html
 8093  -rw-        4082  Sep 11 2014 03:39:03 -07:00  login.html
 8094  -rw-         318  Sep 11 2014 03:39:08 -07:00  loginscript.js
 8095  -rw-        1116  Sep 11 2014 03:39:08 -07:00  logout.html
 8096  -rw-       70123  Sep 11 2014 03:39:08 -07:00  yourlogo.jpg
 8097  -rw-        2846  Sep 11 2014 03:39:08 -07:00  consent.html
 8103  -rw-    83293932   Mar 7 2017 15:45:12 -08:00  cat3k_caa-base.SPA.03.06.06E.pkg
 8104  -rw-     3982012   Mar 7 2017 15:45:12 -08:00  cat3k_caa-drivers.SPA.03.06.06E.pkg
 8105  -rw-    33788908   Mar 7 2017 15:45:12 -08:00  cat3k_caa-infra.SPA.03.06.06E.pkg
 8106  -rw-    43074880   Mar 7 2017 15:45:13 -08:00  cat3k_caa-iosd-universalk9.SPA.152-2.E6.pkg
16161  -rw-    28394380   Mar 7 2017 15:45:13 -08:00  cat3k_caa-platform.SPA.03.06.06E.pkg
16162  -rw-   111233856   Mar 7 2017 15:45:13 -08:00  cat3k_caa-wcm.SPA.10.2.160.0.pkg 

The .bin file is on switch 2 and the file size looks correct. I ran "more packages.conf" and "dir pack*.*" and everything looked  go to boot into the old image. I reloaded and then the install completed successfully.

The switch was on 3.6.6 and hadn't been rebooted in over a year. I think that 3.6.6 may have a memory leak because I have had issues with this version where the console isn't reponsive or other issues and a reload always corrects it. 

References

Catalyst 3850 Series Switch Upgrade, Management, and Recovery Techniques - If you have a switch that boots to the "switch:" prompt you can follow the instructions in this document to recover.

Using Secure Copy (SCP) to Transfer Files

Secure Copy or SCP uses SSH for data transfer and uses the SSH mechanisms for authentication, thereby ensuring the authenticity and confidentiality of the data in transit.

On Cisco network equipment you can enable SCP and use it instead of TFTP for most file transfers. This has the advantage of not having to setup a TFTP server on your computer and the file transfers are encrypted. Encryption should (must?) be used if you are copying files over an insecure link like the Internet.

On Windows you can use the Putty companion program pscp. You can download pscp here PSCP.

Configuration

SSH and "aaa new model" with exec authorization have to be enabled or a local username/password with level 15 priv. In other words, when you log in over SSH, you must end up at a # prompt. See the Cisco Support forum article "Privilege Denied message" in the Reference section below if you get a Privilege denied message.

aaa new-model
aaa authentication login default local
aaa authorization exec default local

Enable SCP
ip scp server enable

For this example:
Cisco device IP - 192.168.10.100
Username - cisco
password – cisco1
My workstation - 192.168.10.50

Copy files from flash:
pscp -scp -pw cisco1 cisco@192.168.10.100:flash:test.pcap
test.pcap             | 30 kB |  30.6 kB/s | ETA: 00:00:00 | 100%

Copy files to flash:
pscp -scp -pw cisco1 C:\tftp-root\firmware.bin cisco@192.168.10:flash:firmware.bin

Using the Archive command with SCP

The free Solarwinds SFTP/SCP server can be used on Windows as an SCP server. Once you have downloaded and installed the server use this syntax on the network device:

archive download-sw scp://mhubbard@192.168.10.50/c3750-ipbaselmk9-tar.122-55.SE10.tar

On the server you will see the authentication and then the file transfer:

On Mac or Linux

SCP is built into Mac OSX and most Linux/Unix distributions.
To copy a file from the network device to the host the syntax is:

scp user@host:flash:filename filename

Note: Make sure you have permission to save files in the directory you run the command from. If not you will see “Operation not supported
Closing Connection”

Here’s an example:
1s1k:~ mhubbard$ scp cisco@192.168.10.50:flash:config.text config.text
Password:
config.text                                       100% 7267     7.1KB/s   00:00

To copy a file from the host to the network device the syntax is:

scp file user@host:flash:file

Here’s an example:
1s1k:~ mhubbard$ scp a.txt cisco@192.168.10.50:flash:a.txt
Password:
a.txt                                                         100% 7267     7.1KB/s   00:00

Copy files to flash from the network device
In this example the file is in my Downloads folder so the absolute path is /home/mhubbard/Downloads.

copy scp://mhubbard@192.168.10.50//home/mhubbard/Downloads/cat3k_caa-universalk9.SPA.03.06.06.E.152-2.E6.bin flash:

NOTE: there are two // after the IP address and the full path to the file.

Using a Password with an @ Symbol 

Just like an SNMP Community string can't contain an @ symbol, neither can the password used for SCP. Here is an example: 

C:\TFTP-Root>pscp -scp -pw b@w0rk@7 c:\tftp-root\nxos.7.0.3.I2.2d.bin admin@172.16.40.5:flash:nxos.7.0.3.I2.2d.bin

PuTTY Secure Copy client

Release 0.64

w0rk@7' is not recognized as an internal or external command, operable program or batch file.

Notice that it took the "b" then terminated on the @ symbol and returned the rest of the password as an error. The user was doing the copy with WinSCP and it popped up a dialog saying it received invalid data. 

I switched to Putty SCP and it was immediately clear what the problem was. I created a new user with a valid password and the SCP transfer succeeded: 

C:\TFTP-Root>pscp -scp -ow 9x3kaqq! c:\tftp-root\nxos.7.0.3.I2.2d.bin mhubbard@172.16.40.5:flash:nxos.7.0.3.I2.2d.bin

nxos.7.0.3.I2.2d.bin | 315980 kB | 381.2 kB/s | 

Troubleshooting

OpenSSH dropped support for SSH V1 in mid 2017. I recently went to scp a file to a 3850 switch and received this on the switch (ip 10.42.250.40):

%Error opening scp://*@10.42.52.172/cat3k_caa-universalk9.16.03.07.SPA.bin (Undefined error)

On the laptop I ran:

systemctl status sshd
 
ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2018-12-12 21:31:56 PST; 21h ago
Process: 19309 ExecReload=/bin/kill -HUP $MAINPID (code=exited, status=0/SUCCESS)
Process: 19305 ExecReload=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
Main PID: 2184 (sshd)
Tasks: 1 (limit: 4915)
CGroup: /system.slice/ssh.service
└─2184 /usr/sbin/sshd -D

Dec 13 18:46:11 1S1K-G5-5587 sshd[19813]: Protocol major versions differ for 10.40.250.40 port 23825: SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.1 vs. SSH-

I remembered that support for SSH V1 had been dropped. I ran
ip ssh ver 2
on the switch and it resolved the issue.

Resources

Cisco link for SCP configuration

Solarwinds SFTP/SCP
SCP Copy - Cisco Support Forums
Privilege denied message

OpenSSH Removes SSHv1 Support