Signature Verification failed while upgrading a Cisco IOS-XE Based Switch

While upgrading some Cisco 3850 switches from 3.6.1 to 3.6.10 I ran into this message:

test-sw#software install file flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin new verbose
Preparing install operation ...
[1]: Starting install operation
[1]: Expanding bundle flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin
[1]: % Signature Verification failed on cat3k_caa-base.SPA.03.06.10E.pkg. Operation aborted.
[1]: % Failed to extract consolidated content. Operation aborted.
[1]: % An internal error was encountered. Operation aborted.

I had verified the .bin file using the “verify /md5 <filename>” command before starting so I knew the file was good. I reloaded the switch and tried again but got the same results.

I opened a TAC case and they knew what to do! There is a bug that requires booting into bundle mode, and then manually expanding the .bin file. It’s actually very easy and doesn’t take much longer than the straight upgrade.

The bug has been fixed in 3.6.4 so you should only run into this on 3.6.0 to 3.6.3.

Check the current boot variable

test-sw#sh boot
---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable does not exist

Boot Variables on next reload:
BOOT variable = flash:packages.conf

Obviously the switch is running install mode because it’s booting from “packages.conf.”

List the files in flash

test-sw#dir
Directory of flash:/

24242  -rwx     2097152  Sep 11 2019 23:00:03 -07:00  nvram_config
24243  -rw-         804  Jun 18 2015 14:28:13 -07:00  vlan.dat
24244  -rw-   302112348   Sep 9 2019 21:01:50 -07:00  cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin
24251  -rw-        1236  Jun 16 2015 06:51:25 -07:00  packages.conf
56561  drwx        4096  Jun 16 2015 06:51:53 -07:00  mnt
24252  -rw-    82653508  Jun 16 2015 06:51:13 -07:00  cat3k_caa-base.SPA.03.06.01E.pkg
24253  -rw-     6625980  Jun 16 2015 06:51:13 -07:00  cat3k_caa-drivers.SPA.03.06.01E.pkg
24254  -rw-    33749996  Jun 16 2015 06:51:13 -07:00  cat3k_caa-infra.SPA.03.06.01E.pkg
24255  -rw-    42827072  Jun 16 2015 06:51:13 -07:00  cat3k_caa-iosd-universalk9.SPA.152-2.E1.pkg
24256  -rw-    25727884  Jun 16 2015 06:51:13 -07:00  cat3k_caa-platform.SPA.03.06.01E.pkg
24257  -rw-    99240768  Jun 16 2015 06:51:14 -07:00  cat3k_caa-wcm.SPA.10.2.111.0.pkg
56562  drwx        4096  Aug 14 2019 18:18:34 -07:00  dc_profile_dir

1562509312 bytes total (919986176 bytes free)

Verify the .bin file's md5 hash

test-sw#verify /md5 cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin
..........................................................................................................................…
Done!
verify /md5 (flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin) = ac15e03a732a23e894d3943d667ec168

The hash is correct so we can continue.

Set the boot variable to boot the .bin file

test-sw#conf t
test-sw(config)#no boot system
test-sw(config)#boot system sw all flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin
test-sw(config)#end
test-sw#wr mem
Building configuration...
Compressed configuration from 33642 bytes to 11246 bytes[OK]

Check the boot variable

test-sw#sh boot
---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable = flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin;

Boot Variables on next reload:
BOOT variable = flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin;

Note: If you check the boot variable immediately after the wr mem it may still show

Boot Variables on next reload:
BOOT variable = flash:packages.conf

If it does, just wait 10 seconds and try again.

Boot into bundle mode

test-sw#reload
Reload command is being issued on Active unit, this will reload the whole stack
Proceed with reload? [confirm]

----------------------After the reload-----------------------------

Expand the .bin file

test-sw#software expand file flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin
Preparing expand operation ...
[1]: Expanding bundle flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin
[1]: Copying package files
[1]: A different version of provisioning file packages.conf already exists in flash:.
    The provisioning file from the expanded bundle will be saved as
    flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.conf
[1]: Package files copied
[1]: Finished expanding bundle flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin

Update the packages.conf file

test-sw#delete flash:packages.conf
Delete filename [packages.conf]?
Delete flash:/packages.conf? [confirm]
test-sw#rename flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.conf packages.conf
Destination filename [packages.conf]?

Verify that packages.conf contains the 3.6.10 files

test-sw#more flash:packages.conf
#! /usr/binos/bin/packages_conf.sh

sha1sum: b844cd3af271a3dd0fcfd90d38bb4f58f8b4f531
iso   rp 0 0   rp_base       cat3k_caa-base.SPA.03.06.10E.pkg
iso   rp 0 0   rp_infra       cat3k_caa-infra.SPA.03.06.10E.pkg
iso   rp 0 0   rp_platform       cat3k_caa-platform.SPA.03.06.10E.pkg
iso   rp 0 0   rp_iosd       cat3k_caa-iosd-universalk9.SPA.152-2.E10.pkg
iso   rp 0 0   rp_wcm       cat3k_caa-wcm.SPA.10.2.200.0.pkg
iso   rp 0 0   drivers       cat3k_caa-drivers.SPA.03.06.10E.pkg

Update the Boot Variable for install mode

test-sw#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
test-sw(config)#no boot system
test-sw(config)#boot system flash:packages.conf
test-sw(config)#end
test-sw#wr mem
Building configuration...
Compressed configuration from 33566 bytes to 10974 bytes[OK]
test-sw#sh boot
---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable = flash:packages.conf;

Boot Variables on next reload:
BOOT variable = flash:packages.conf;
Allow Dev Key = yes
Manual Boot = no
Enable Break = no

Reload the switch to boot into 3.6.10

test-sw#reload

Here are the commands ready to paste in:

sh boot
dir
verify /md5  cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin
conf t
no boot system
boot system sw all flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin
end
wr mem
sh boot
reload

software expand file flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin
delete flash:packages.conf
rename flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.conf packages.conf

more flash:packages.conf
conf t
no boot system
boot system flash:packages.conf
end
wr mem
sh boot