Sunday, October 25, 2020

Bad Neighbor cve-2020-16898

On October 13, 2020 Microsoft issued a security vulnerability notice - Windows TCP/IP Remote Code Execution Vulnerability. CVE-2020-16898 which affects Server 2019 and Windows 10.

From the notice:

"A remote code execution vulnerability exists when the Windows TCP/IP stack improperly handles ICMPv6 Router Advertisement packets. An attacker who successfully exploited this vulnerability could gain the ability to execute code on the target server or client.

To exploit this vulnerability, an attacker would have to send specially crafted ICMPv6 Router Advertisement packets to a remote Windows computer.

The update addresses the vulnerability by correcting how the Windows TCP/IP stack handles ICMPv6 Router Advertisement packets."

This is an unauthenticated vulnerability meaning you don't need any credentials on the domain to exploit it. A security researcher named Adam wrote a Proof of Concept exploit in python. The write up is extremely detailed and walks you through his development. The blog is available here - CVE-2020-16898 – Exploiting “Bad Neighbor” vulnerability The python script needed to exploit the vulnerability is available on the blog.

Mitigation

Microsoft does not recommend completely disabling IPv6 to mitigate. As a workaround, they provide this netsh script

Disable ICMPv6 RDNSS


netsh int ipv6 set int *INTERFACENUMBER* rabaseddnsconfig=disable

To find the interface number I found this Powershell script Enumerate IPv6 interfaces

The same GitHub has a script to disable RDNSS.

Example - I ran this from the Powershell ISE program:

PS C:\Windows\system32> Get-NetIPInterface -AddressFamily ipv6 | foreach{
   [PSCustomObject]@{
        "IfIndex"   = (& netsh int ipv6 show int $_.ifIndex) -match 'IfIndex' -replace "ifindex\s*:","" | Out-String
        "RFC"   = (& netsh int ipv6 show int $_.ifIndex) -match '(RFC 6106)' -replace "RA Based DNS Config \(RFC 6106\)\s*:","" | Out-String
    }
}

IfIndex RFC       
------- ---       
 5...    enable...
 6...    enable...
 1...    enable...

PS C:\Windows\system32> netsh int ipv6 set int 5 rabaseddnsconfig=disable
Ok.


Mitigation using a Cisco network switch

If you are using Cisco switches in your environment you can use the following to mitigate. I show this in the video.

From global configuration mode:

  • SW1(config)#ipv6 nd inspection policy policy-name HOST-POLICY
  • SW1(config-nd-inspection)#device-role host
Note: host is the default role so you don't have to enter the device-role.

From interface configuration mode

  • SW1(config)#int gig0/1
  • SW1(config-if)#ipv6 nd raguard attach-policy HOST-POLICY

Verify

SW1#sh ipv6 nd raguard policy HOST-POLICY

Policy RAGUARD configuration:

device-role host

Policy HOST-POLICY is applied on the following targets:

Target               Type Policy               Feature       Target range

Gi0/1               PORT RAGUARD            RA guard       vlan all


Mitigation using an Aruba switch running Provision software

In this example, an Aruba 5412 switch is used

sw2(config)# ipv6 ra-guard ports i1 log

Verify
sw2# show ipv6 ra-guard | exclude  No    0

 IPv6 RA Guard Information

  Port  Block RAs Blocked Redirs Blocked Log
  ----- ----- ----------- -------------- ---
  I1    Yes   0           0              Yes


Watch Bad Neighbor in action

I made a 2-minute video showing a Windows 2019 server blue screening when Adam's script is run against the server. After showing the blue screen I enable ra guard on a cisco switch and rerun the script. This prevents the blue screen. Here is a link to the video - Bad Neighbor cve 2020 16898

Juniper network devices

Juniper has announced that JunOS is vulnerable under certain conditions. Here is their bulletin:

2020-10 Security Bulletin: Junos OS:


References

CVE-2020-16898: Windows ICMPv6 Router Advertisement RRDNS Option Remote Code Execution Vulnerability - A detailed write up by Johannes Ulrich of SANS.

Cisco RA Guard blog

MITRE CVE for 2020-16898

ZEEK package to detect Bad Neighbor

There Goes The Neighborhood - Rapid 7 blog on Bad Neighborhood

Sunday, June 21, 2020

Disable Weak SSH/SSL Ciphers in Cisco IOS

For backward compatibility, most companies still ship deprecated, weak SSH, and SSL ciphers. Cisco is no exception. For the security of your network and to pass a penetration test you need to disable the weak ciphers, disable SSH v1 and disable TLS version 1.0 and 1.1.

Firefox, Chrome and Microsoft all have committed to dropping support for TLS1.1. Firefox had actually done it in May 2020 but so many US Government sites quit working (during the Covid19 Hysteria) that they rolled back. Microsoft has set July 2020 to remove TLS 1.0/1.1 from IE, Edge Legacy, and Edge Chromium.

This blog covers Cisco IOS software. I plan to do another blog on IOS-XE and Nexus in the future.

SSH

Network device manufacturers (all of them I think) enabling SSH v1 by default really bothers me. Most Windows users connect with Putty which supports SSH v2. You should set Putty to default to SSH V2:




MAC/Linux users will be using OpenSSh which also supports SSH V2. You may run into situations on MAC/Linux where the weak ciphers are used and OpenSSH won't connect.

You will see a message similar to
ssh mhubbard@10.20.1.7
Unable to negotiate with 10.20.1.7 port 22: no matching key exchange method found. Their offer: diffie-hellman-group1-sha1
 
This is easy to resolve:

1. Open the SSH config file - gedit ~/.ssh/config
2. Add the necessary host IP and ciphers. KEX is Key Exchange:
        host 10.20.1.7
              KexAlgorithms +diffie-hellman-group1-sha1
              Ciphers 3des-cbc

On a really old switch, I ran into a host key exchange algorithm that I had never even heard of "ssh-dss". I had to add HostKeyAlgorithms=+ssh-dss to connect.

If you will only log into this device once or twice you can use the following without modifying the SSH config file:
ssh -oKexAlgorithms=+diffie-hellman-group1-sha1 10.20.1.7
You can use the "-G" switch and SSH will show you the ciphers that SSH is offering:
ssh -G mhubbard@10.20.1.7


The OpenSSH site has a page dedicated to legacy ciphers
openssh legacy ciphers

Removing weak SSH algorithms


All of the commands shown are from a 2960x running:
Version 15.2(4)E8 - Mainstream deployment (MD) from 18-Mar-2019

First, let's look at the default SSH setup
show ip ssh
SSH Enabled - version 1.99
Authentication methods:publickey,keyboard-interactive,password
Authentication Publickey Algorithms:x509v3-ssh-rsa,ssh-rsa
Hostkey Algorithms:x509v3-ssh-rsa,ssh-rsa
Encryption Algorithms:aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc,aes192-cbcc
MAC Algorithms:hmac-sha1,hmac-sha1-96
Authentication timeout: 120 secs; Authentication retries: 3
Minimum expected Diffie Hellman key size : 1024 bits
IOS Keys in SECSH format(ssh-rsa, base64 encoded): TP-self-signed-1676064512
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQCjsPhP/zpPgra0d3wzzt8fDZnKL4sUtCh0DVmV0fH6
m+/Xke7IRMvxg2OEk333uHlKD+Ww6w8D2eMOzY7/R6edHA4UtKXwohJN1OZKS1ltL4tDSZSIeLO3juOL
GfxKBtvGd30Y2jzYYMmTQGP9u1VrKdQRKAU13/c+iOiQPi3Q4w==          

The "version 1.99" means that it supports SSH v1 and v2. We want to disable v1 and remove the cbc and 3Des ciphers. These are "Cipher Block Chain" algorithms and will cause a failure during a penetration test.

From global configuration mode enter the following:
ip ssh version 2 !disable V1

ip ssh server algorithm encryption aes256-ctr aes128-ctr

ip ssh server algorithm mac hmac-sha1

no ip ssh server algorithm mac hmac-sha1-96

You should also perform the following to harden SSH

crypto key generate rsa modulus 4096 label SSH-KEYS

!Note that generating 4096 bit keys can take up to 3 minutes.

ip ssh rsa keypair-name SSH-KEYS !associate keys to SSH

!set minimum bit size for client connection                                

ip ssh dh min size 2048


Let's see what SSH looks like now:
show ip ssh
SSH Enabled - version 2.0
Authentication methods:publickey,keyboard-interactive,password
Authentication Publickey Algorithms:x509v3-ssh-rsa,ssh-rsa
Hostkey Algorithms:x509v3-ssh-rsa,ssh-rsa
ip ssh server algorithm encryption aes256-ctr aes128-ctr
MAC Algorithms:hmac-sha1
Authentication timeout: 120 secs; Authentication retries: 3
Minimum expected Diffie Hellman key size : 2048 bits
IOS Keys in SECSH format(ssh-rsa, base64 encoded): SSH-KEYS
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC5KQxmPn8tyfK+9fq6NC75whEQD02POopz9SE/SKeP
ibO8KM7kSVdwy7anUhmgiX5jGmpecTFoP+txdA+KuEszAL5x8aeNZsPAykqBU6JClIz3fnMKjgoIqFlZ
mwhL0Qow4OGrd52EkRNRxAc2TYpBr5p0ICdaxeHd7etzgXjkwcZpQ1e2kqvV9XU94LBO1R93AgYYLCsT
nFsKga4tvvikXqKuwe3tfWKzNfO4LY1mZE9FXecoNW0Kb8p4U/pO/w69oEbHmmH7BfgWSHCCVZlgBhcf
DtJa+oVnqHrMwVza+ViTMQLghvt63zewvTN2I235K6W+GhgUmx6p+Q62Rsrfrc+4K5ECVKNf7fzmlg6X
Zs+P3WKgP8rh2z7ObTT917pp1VXw4pUkeqCCtMEmkiICO0TzU1dXyuoEPNGeES8wxYOSdaMA0DGEL34p
Ccb6hb1RQbHjSjQZfDOXaZ0UwXtVJ07v7PR7fOhFHem58w2P+qmCwnEYFZrZhizR1y1SUDxs6Z7vZV98
cyoTo98dWG4WDGiHM1loLq3SA3OMfceq5g2waPVBNmpZlzXitCTern1bZ15zdLvhxY1589A/TaSZuMeP
lhjQ1mlYp3qf0Jt7eoaWNPRV/i0VUaRfxNBefiNBI5pS8ybj3bhfWpZe8QOOMAHRahAPPI9PasOBuMHR

In 2020, this is still pretty lame, but keep reading! Cisco has been adding newer ciphers and removing some deprecated ciphers in newer IOS versions. You can check what's available on your version using:

test(config)#ip ssh server algorithm encryption ?
  3des-cbc    Three-key 3DES in CBC mode
  aes128-cbc  AES with 128-bit key in CBC mode
  aes128-ctr  AES with 128-bit key in CTR mode
  aes192-cbc  AES with 192-bit key in CBC mode
  aes192-ctr  AES with 192-bit key in CTR mode
  aes256-cbc  AES with 256-bit key in CBC mode
  aes256-ctr  AES with 256-bit key in CTR mode

test(config)#ip ssh server algorithm mac ?
  hmac-sha1     HMAC-SHA1 (digest length = key length = 160 bits)
  hmac-sha1-96  HMAC-SHA1-96 (digest length = 96 bits, key length = 160 bits)

If you look at Authentication in the output you notice that Public key is an option. I wrote a blog showing how to use SSH keys instead of passwords -
Authenticating to Cisco devices using SSH keys

 

Weak SSL ciphers


First, we will look at the current secure server settings. To see all possible secure server settings:
sh ip http server
  all             HTTP server all information
  connection      HTTP server connection information
  external        HTTP external registration
  history         HTTP server history information
  secure          HTTP secure server status information
  session-module  HTTP server application session module information
  statistics      HTTP server statistics information
  status          HTTP server status information

sh ip http server all        
HTTP server status: Disabled
HTTP server port: 80
HTTP server authentication method: local
HTTP server access class: 0
HTTP server base path: flash:/c2960x-universalk9-mz.152-4.E8/html
HTTP server help root:
Maximum number of concurrent server connections allowed: 16
Maximum number of secondary server connections allowed: 5
Server idle time-out: 180 seconds
Server life time-out: 180 seconds
Server session idle time-out: 180 seconds
Maximum number of requests allowed on a connection: 25
HTTP server active session modules: ALL
HTTP secure server capability: Present
HTTP secure server status: Enabled
HTTP secure server port: 443
HTTP secure server ciphersuite:  dhe-aes-128-cbc-sha dhe-aes-256-cbc-sha
        edche-rsa-aes-256-cbc-sha edche-rsa-rc4-128-sha

HTTP secure server client authentication: Disabled
HTTP secure server trustpoint:
HTTP secure server active session modules: ALL

To see who is connected to the switch over TLS:
sh ip http server connection

HTTP server current connections:
local-ipaddress:port  remote-ipaddress:port in-bytes   out-bytes
  192.168.10.31:443    192.168.10.211:55014 1394       586227


Viewing available current cipher suites


ip http secure-ciphersuite ?

  aes-128-cbc-sha            Encryption type tls_rsa_with_aes_cbc_128_sha
                             ciphersuite
  aes-256-cbc-sha            Encryption type tls_rsa_with_aes_cbc_256_sha
                             ciphersuite
  dhe-aes-128-cbc-sha        Encryption type tls_dhe_rsa_with_aes_128_cbc_sha
                             ciphersuite
  dhe-aes-256-cbc-sha        Encryption type tls_dhe_rsa_with_aes_256_cbc_sha
                             ciphersuite
  edche-rsa-aes-256-cbc-sha  Encryption type tls_ecdhe_rsa_aes_256_cbc_sha
                             ciphersuite
  edche-rsa-rc4-128-sha      Encryption type tls_ecdhe_rsa_rc4_128_sha
                             ciphersuite
  null-sha                   Encryption type tls_rsa_with_null_sha ciphersuite



Notice that rc4 and Null are supported!

To verify what was being offered by the switch I ran the nmap ssl-cert and ciphers script.
sudo nmap --script ssl-cert,ssl-enum-ciphers -p 443 192.168.10.31
Nmap scan report for 10.241.3.40
Host is up, received echo-reply ttl 254 (0.10s latency).
Scanned at 2020-06-18 15:28:06 PDT for 3s

PORT     STATE  SERVICE       REASON
443/tcp  open   https         syn-ack ttl 254
| ssl-cert: Subject: commonName=IOS-Self-Signed-Certificate-1302447744
| Issuer: commonName=IOS-Self-Signed-Certificate-1302447744
| Public Key type: rsa
| Public Key bits: 1024
| Signature Algorithm: sha1WithRSAEncryption
| Not valid before: 2020-06-16T22:55:16
| Not valid after:  2030-01-01T00:00:00
| MD5:   c522 61ff 31c4 c9aa 971d 7cfd 4eb7 14de
| SHA-1: 50fb 7c7d d6a8 86c0 ba67 1293 11d7 f529 058e e1de
| -----BEGIN CERTIFICATE-----
| MIICKzCCAZSgAwIBAgIBATANBgkqhkiG9w0BAQUFADAxMS8wLQYDVQQDEyZJT1Mt
| U2VsZi1TaWduZWQtQ2VydGlmaWNhdGUtMTMwMjQ0Nzc0NDAeFw0yMDA2MTYyMjU1
| MTZaFw0zMDAxMDEwMDAwMDBaMDExLzAtBgNVBAMTJklPUy1TZWxmLVNpZ25lZC1D
| ZXJ0aWZpY2F0ZS0xMzAyNDQ3NzQ0MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKB
| gQDCgxwOBYowFY7GgS3Q81u6CRTzcaEb2SwZvzSsjTLmHPqrB7OYgGukAgs19+Xa
| 8jRS3jY4Q492RtpyBAb4BU9naHXRKvD2zB5e9QDreeFOf73If6f8V/BtjqSozYZW
| N0RPpgqIWVbgQbkr1eBbnXgE1/TO7czYcjae/OTSZwQL1QIDAQABo1MwUTAPBgNV
| HRMBAf8EBTADAQH/MB8GA1UdIwQYMBaAFDL08Ihv1OFKYBqkbHJ5wpXt3G7IMB0G
| A1UdDgQWBBQy9PCIb9ThSmAapGxyecKV7dxuyDANBgkqhkiG9w0BAQUFAAOBgQCH
| GxSZ29CUBrvCkDU4knDw9WmdLKqgMl88+dpZmOO758+o4B8lMT0f+Ixny7drFIJ7
| rrkhrqpCHnLDJtXYcINiaKASs3tPIpQ21nQ1r5WTdW8GqaTVcOBIFG0KWlJGVmsF
| RepCnGblGV/3mrUWImNU8xwY+uZS2vAFKAVXYVLk5w==
|_-----END CERTIFICATE-----
| ssl-enum-ciphers:
|   TLSv1.1:
|     ciphers:
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (secp256r1) - A
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (secp256r1) - A
|     compressors:
|       NULL
|     cipher preference: client
|     warnings:
|       Weak certificate signature: SHA1
|   TLSv1.2:
|     ciphers:
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (secp256r1) - A
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (secp256r1) - A
|     compressors:
|       NULL
|     cipher preference: client
|     warnings:
|       Weak certificate signature: SHA1
|_  least strength: A
465/tcp  closed smtps         reset ttl 254
993/tcp  closed imaps         reset ttl 254
995/tcp  closed pop3s         reset ttl 254
3389/tcp closed ms-wbt-server reset ttl 254

NSE: Script Post-scanning.
NSE: Starting runlevel 1 (of 1) scan.
Initiating NSE at 15:28
Completed NSE at 15:28, 0.00s elapsed
Read data files from: /usr/local/bin/../share/nmap
Nmap done: 1 IP address (1 host up) scanned in 2.71 seconds
           Raw packets sent: 9 (372B) | Rcvd: 6 (232B)


To secure TLS I upgraded to 15.2.7E2. This release allows TLS 1.0 and 1.1 to be disabled. To pass a penetration test you will need to disable both. Once the upgrade is complete run the following:
test(config)#ip http secure-ciphersuite ?
  aes-128-cbc-sha            Encryption type tls_rsa_with_aes_cbc_128_sha ciphersuite
  aes-256-cbc-sha            Encryption type tls_rsa_with_aes_cbc_256_sha ciphersuite
  dhe-aes-128-cbc-sha        Encryption type tls_dhe_rsa_with_aes_128_cbc_sha ciphersuite
  edche-rsa-aes-256-cbc-sha  Encryption type tls_ecdhe_rsa_aes_256_cbc_sha ciphersuite


test(config)#ip http secure-ciphersuite edche-rsa-aes-256-cbc-sha aes-256-cbc-sha
test(config)#ip http tls-version ?
  TLSv1.0  Set TLSv1.0 version Only
  TLSv1.1  Set TLSv1.1 version Only
  TLSv1.2  Set TLSv1.2 version Only
test(config)#ip http tls-version tlsv1.2
To verify, I re-ran the nmap ssl-cert and ciphers scripts. This time only TLS 1.2 is enabled.
sudo nmap --script ssl-cert,ssl-enum-ciphers -p 443 192.168.10.31
Nmap scan report for 192.168.10.31
Host is up, received echo-reply ttl 254 (0.0072s latency).
Scanned at 2020-06-18 15:50:03 PDT for 3s

PORT    STATE SERVICE REASON
443/tcp open  https   syn-ack ttl 254
| ssl-cert: Subject: commonName=IOS-Self-Signed-Certificate-1302447744
| Issuer: commonName=IOS-Self-Signed-Certificate-1302447744
| Public Key type: rsa
| Public Key bits: 1024
| Signature Algorithm: sha1WithRSAEncryption
| Not valid before: 2020-06-16T22:55:16
| Not valid after:  2030-01-01T00:00:00
| MD5:   c522 61ff 31c4 c9aa 971d 7cfd 4eb7 14de
| SHA-1: 50fb 7c7d d6a8 86c0 ba67 1293 11d7 f529 058e e1de
| -----BEGIN CERTIFICATE-----
| MIICKzCCAZSgAwIBAgIBATANBgkqhkiG9w0BAQUFADAxMS8wLQYDVQQDEyZJT1Mt
| U2VsZi1TaWduZWQtQ2VydGlmaWNhdGUtMTMwMjQ0Nzc0NDAeFw0yMDA2MTYyMjU1
| MTZaFw0zMDAxMDEwMDAwMDBaMDExLzAtBgNVBAMTJklPUy1TZWxmLVNpZ25lZC1D
| ZXJ0aWZpY2F0ZS0xMzAyNDQ3NzQ0MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKB
| gQDCgxwOBYowFY7GgS3Q81u6CRTzcaEb2SwZvzSsjTLmHPqrB7OYgGukAgs19+Xa
| 8jRS3jY4Q492RtpyBAb4BU9naHXRKvD2zB5e9QDreeFOf73If6f8V/BtjqSozYZW
| N0RPpgqIWVbgQbkr1eBbnXgE1/TO7czYcjae/OTSZwQL1QIDAQABo1MwUTAPBgNV
| HRMBAf8EBTADAQH/MB8GA1UdIwQYMBaAFDL08Ihv1OFKYBqkbHJ5wpXt3G7IMB0G
| A1UdDgQWBBQy9PCIb9ThSmAapGxyecKV7dxuyDANBgkqhkiG9w0BAQUFAAOBgQCH
| GxSZ29CUBrvCkDU4knDw9WmdLKqgMl88+dpZmOO758+o4B8lMT0f+Ixny7drFIJ7
| rrkhrqpCHnLDJtXYcINiaKASs3tPIpQ21nQ1r5WTdW8GqaTVcOBIFG0KWlJGVmsF
| RepCnGblGV/3mrUWImNU8xwY+uZS2vAFKAVXYVLk5w==
|_-----END CERTIFICATE-----
| ssl-enum-ciphers:
|   TLSv1.2:
|     ciphers:
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (secp256r1) - A
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (secp256r1) - A
|       TLS_RSA_WITH_AES_256_CBC_SHA (rsa 1024) - A
|     compressors:
|       NULL
|     cipher preference: client
|     warnings:
|       Weak certificate signature: SHA1
|_  least strength: A

NSE: Script Post-scanning.
NSE: Starting runlevel 1 (of 1) scan.
Initiating NSE at 15:50
Completed NSE at 15:50, 0.00s elapsed
Read data files from: /usr/local/bin/../share/nmap
Nmap done: 1 IP address (1 host up) scanned in 2.99 seconds
           Raw packets sent: 5 (196B) | Rcvd: 2 (72B)

Results

You can see that it still uses SHA1 as the certificate signature. You can use  Ciphersuite Info to compare different ciphers.

What about SSH?

Let's see what's new for SSH in 15.7.2E2.
ip ssh server algorithm mac ?
  hmac-sha1      HMAC-SHA1 (digest length = key length = 160 bits)
  hmac-sha1-96   HMAC-SHA1-96 (digest length = 96 bits, key length = 160 bits)
  hmac-sha2-256  HMAC-SHA2-256 (digest length = 256 bits, key length = 256
                 bits)
  hmac-sha2-512  HMAC-SHA2-512 (digest length = 512 bits, key length = 512
                 bits)


ip ssh serv algorithm encryption ?


3des-cbc    Three-key 3DES in CBC mode
  aes128-cbc  AES with 128-bit key in CBC mode
  aes128-ctr  AES with 128-bit key in CTR mode
  aes192-cbc  AES with 192-bit key in CBC mode
  aes192-ctr  AES with 192-bit key in CTR mode
  aes256-cbc  AES with 256-bit key in CBC mode
  aes256-ctr  AES with 256-bit key in CTR mode


Now we can eliminate the ancient HMAC-SHA1 and CBC ciphers from our switch!

First, we will add the sha2 HMACs
ip ssh server algorithm mac hmac-sha2-256 hmac-sha2-512

Then remove the sha1 hmacs
no ip ssh server algorithm mac hmac-sha1
no ip ssh server algorithm mac hmac-sha1-96

And now the encryption
ip ssh server algorithm encryption aes256-ctr aes192-ctr aes128-ctr

The results

show ip ssh

SSH Enabled - version 2.0
Authentication methods:publickey,keyboard-interactive,password
Authentication Publickey Algorithms:x509v3-ssh-rsa,ssh-rsa
Hostkey Algorithms:x509v3-ssh-rsa,ssh-rsa
Encryption Algorithms:aes256-ctr,aes192-ctr,aes128-ctr
MAC Algorithms:hmac-sha2-256,hmac-sha2-512
KEX Algorithms:diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1
Authentication timeout: 120 secs; Authentication retries: 3
Minimum expected Diffie Hellman key size : 2048 bits
IOS Keys in SECSH format(ssh-rsa, base64 encoded): SSH-KEYS
Modulus Size : 4096 bits
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDCsyuZ8/lMCNHSLREb6vGQoBVehYQQI0+eJlanuyq5
f+iTqFcceR7vvXP14JhHmXe2lkygOZ8VIeilMJkpS8q748TaBL9QfmUAdDkbbk1wYPNKM2sLn/ACuerf
ImNa4vQFNaP28zqaCMhre/Z0DCRJvDnOXs2fepQnQZ6ZvbOgwMRw6rvTiLcPYlB46VlaS6T1ogEbsPLz
HG1e2UeGOnxyIU9j99+sUq3h5omoxtOd33c7ygyBgghBm+G4rHoD4EsJmejK2/Ai1PsjHIN16EaTAB0Y
MiIFByAYr4/Hr+6ANejxDrFpeY3DDBTvXIcES3S+C/Ch6JEoFVfHufc5ni8OReE7KQhrBctNfhoXvFRO
wITNNyyu/jk1LLDTaLFbL/auw/eXGXlXXerWRFY6HvmAbQannl9wryvy97Hm4LJVO+DtTspwvw4IKrQT
HDMdyXvTI6RMjIlGb/7hiUeFb33wx7sw/DwkgjyUCWh8R8nCEoLfpz7qOchW2/WSj+608m62Eh6WDy5q
qkDpstQRD7AbE2OBtiuYgYJaNJfZ1qhIQXlvtQCTgRRS2TvInnoGg+STD2+lWR5WufgKEO778tNDXt3H
YRSdD2N1YcjXG+y0hB/xjvWSoMkr+G2Btxtm8QPgvXQRe9aFU/kALMBKBJ6Q+rDXr2QbyA7zpDudkAn3

Security Header Enhancements

For IOS-XE devices, starting with 16.4.1, the Nginx/HTTP headers have the following settings for increased security:

Nginx – Web user interface - 
Nginx applications take care of the headers for their response. As Web UI is one of the NginX application, it adds the security headers. 

The three headers are the following:

  •     X-XSS-Protection: 1; mode=block
  •     X-Frame-Options: SAMEORIGIN
  •     X-Content-Type-Options: nosniff


Do the same thing to your Linux Servers

There is a good chance your organization is running some Linux servers. Out of the box, CentOS/Ubuntu will have several weak ciphers. It's very easy to correct that but you will need root privileges.

First, we will check what ciphers your server is offering. If the server has a public IP address you can go to https://sshcheck.com and enter the FQDN or the IP address. You will get back a comprehensive report back with suggestions on which items should be disabled.

If the server is internal you can use nmap's ssh-enum script:
sudo nmap --script ssh2-enum-algos 192.168.10.239

This will return a list of the crypto offered by your server.

Update the sshd config file

The sshd config file is located at /etc/ssh. We need to open it and add the suites we want. First we will make a backup copy.

sudo cd /etc/ssh
sudo cp sshd_config sshd_config.bak
sudo nano sshd_config

Add the following (Make sure these fit your company's security policies)
ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr

macs umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,umac-128@openssh.com,hmac-sha2-256

KexAlgorithms diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha256,curve25519-sha256,curve25519-sh$

You can add these anywhere. I put them just below the section
"# Ciphers and keying".

Press ctrl+x, enter Y to save the file and enter to complete.

You can use
sudo sshd -t
to verify the changes. If there are no mistakes in the configuration file nothing will be displayed. If there are errors you get a message with the line number where the error occurred.

You can use
sshd -T
to dump the current ssh configuration.

Now we just need to restart the ssh daemon;
sudo systemctl restart sshd


Check your work

Refresh the sshcheck page or rerun nmap. You should see just the cipher suites you entered. Here is nmap against my server:
nmap --script ssh2-enum-algos -sV -p22 hubbardonnetworking.com

Starting Nmap 7.70 ( https://nmap.org ) at 2020-06-24 22:15 PDT
Nmap scan report for hubbardonnetworking.com (107.170.203.230)
Host is up (0.026s latency).

PORT   STATE SERVICE VERSION
22/tcp open  ssh     OpenSSH 7.4 (protocol 2.0)
| ssh2-enum-algos:
|   kex_algorithms: (6)
|       diffie-hellman-group14-sha256
|       diffie-hellman-group16-sha512
|       diffie-hellman-group18-sha512
|       diffie-hellman-group-exchange-sha256
|       curve25519-sha256
|       curve25519-sha256@libssh.org
|   server_host_key_algorithms: (5)
|       ssh-rsa
|       rsa-sha2-512
|       rsa-sha2-256
|       ecdsa-sha2-nistp256
|       ssh-ed25519
|   encryption_algorithms: (6)
|       chacha20-poly1305@openssh.com
|       aes256-gcm@openssh.com
|       aes128-gcm@openssh.com
|       aes256-ctr
|       aes192-ctr
|       aes128-ctr
|   mac_algorithms: (5)
|       umac-128-etm@openssh.com
|       hmac-sha2-256-etm@openssh.com
|       hmac-sha2-512-etm@openssh.com
|       umac-128@openssh.com
|       hmac-sha2-256
|   compression_algorithms: (2)
|       none
|_      zlib@openssh.com

Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 0.70 seconds
 
I have a Python script that presents a menu with several nmap security scripts. If you haven't used nmap much it is worth a look.

Python tool for nmap scripts

References

Putty SSH V2
SSH Algorithms for Common Criteria Certification
Cisco IOS HTTP Services Command Reference

Sunday, March 8, 2020

Southern California Linux Expo SCALE 18x

I missed last year's Linux Expo because I had the flu. This year I was healthy but several speakers canceled and attendance was down because of the Covid-19 travel restrictions that companies have instituted! 
 
Microsoft was the title sponsor and there were some comments that "it's funny that Microsoft is the title sponsor and now we have virus problems" going around.

Several talks were canceled but the organizers did a fantastic job of finding new speakers and pulling off a great event! They had hand sanitizer everywhere and provided sanitizing wipes if you held the microphone to ask a question after a talk.

I got to catch up with a former co-worker and even a current customer. I have found that the "Hallway" track is one of the best parts of a conference and it was true this year!

If you have never attended SCALE I highly recommend it. The cost for all four days is only $85 and that includes as many talks as you want to attend, a party on Friday night, a CTF event all three days and a decent sized Expo floor.

The last two years DC Darknet has been there. They are the group that does the badges for Defcon and their booth is interactive - you can try your hand at lock picking, buy a kit for a Defcon badge and do the soldering at a table beside their booth. If you have never soldered, they will train you!

There is plenty of SWAG at the vendor booths, I scored "Red Hat" and "Salt stack" ball caps along with lots of stickers and Tee shirts. Even with the low entry fee, SCALE provides a T-shirt, lanyard, coffee mug and a bag to carry it in!

Saturday's keynote by Paul Vixie was eye-opening! He discussed DNS over HTTPS and why it will cause more problems than it solves! If they post a transcript I will add a link to it.

Even with the cancellations, there were still a lot of talks to attend! I managed to sit in on a Security Onion talk that was one of the best talks I have attended anywhere. The founder of the project, Doug Burks, gave the talk and his enthusiasm for the project was contagious! I plan to download the ISO and give it a good look. I think for smaller companies it is a great tool to secure their network.

I also attended a Security for Noobs talks. It was very good and generated some interesting questions at the end of the talk. The speaker posted the presentation here

I had planned to sit in on several embedded Linux talks put on by the Core Embedded Linux Project of the Linux Foundation. I attend a couple two years ago and they were great! You brought a Beagle Bone Black, compiled a Linux kernel and built a fully functional embedded device. Unfortunately, the entire track was canceled because of travel restrictions.

I had also planned to attend the Linux Professionals Institute LPIC-1 Preparation Session on Friday but got tied up at work and missed it. I was hoping to take the exam on Sunday as they offer it for $99! But I was too tired to cram on Friday night and after missing the prep session decided to wait. 

Sunday's keynote was from a man named Sha that had been convicted of assault with a firearm and sentenced to 27 years to life. A woman named Jessica McKellar volunteers at San Quentin Prison and teaches Python to the inmates. When Sha's sentence was commuted, she hired him. Sha's presentation was amazing. When the Q&A came up I told Sha that I have heard hundreds for presentations and his was one of the best. I asked him to start a YouTube channel and keep us updated on his progress. I got a large round of applause!

All in all, it was a good SCALE!

Sunday, December 8, 2019

2019 IT Blog Awards Finalists

Cisco is hosting the 2019 IT Blog Awards. Here are the finalists. You can vote by going to 2019 IT Blog Awards Finalists Voting!

I have only used mrnCCIEW and Clear to Send, and recommend both, but if they made this list they are probably worth a look.

Best Analysis - Does this blog provide insightful discussions?
Houman Asefi
How Does Internet Work
mrnCCIEW
Network Defense Blog
Wireless Nerd
The WLAN
vMiss

Best Cert Study Journey - Provides useful insights into the need-to-knows throughout a certification study journey.
Bruno Wollmann
Cisco Redes
IP Cisco
Lab Every Day
Network Fun TimesNo Blinky Blinky
Packit Forwarding

Best Newcomer - A great new blog / podcast / etc. started in the last(ish) year.
Cisco Worker Bee
Dmitry Golovach
Gifted Lane
Micheline Murphy
Network Freestyle
Never the Network
Plugins Blog

Best Podcast or Video Series - Best in content and creativity delivered in the format of videos or podcasts.
CIO In The Know
Clear to Send
Madrasa Tech
Mario Salinas
Network Bruh
Network Chuck
The Ask Anson Channel

Most Educational  - Shares great tips, tricks, or how-to’s to help you learn something new or further your understanding on a topic.
About Networks
Craig Waters
Ferenc Kuris
Jorge De La Cruz










Wednesday, September 18, 2019

Signature Verification failed while upgrading a Cisco IOS-XE Based Switch

While upgrading some Cisco 3850 switches from 3.6.1 to 3.6.10 I ran into this message:

test-sw#software install file flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin new verbose
Preparing install operation ...
[1]: Starting install operation
[1]: Expanding bundle flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin
[1]: % Signature Verification failed on cat3k_caa-base.SPA.03.06.10E.pkg. Operation aborted.
[1]: % Failed to extract consolidated content. Operation aborted.
[1]: % An internal error was encountered. Operation aborted.

I had verified the .bin file using the “verify /md5 <filename>” command before starting so I knew the file was good. I reloaded the switch and tried again but got the same results.

I opened a TAC case and they knew what to do! There is a bug that requires booting into bundle mode, and then manually expanding the .bin file. It’s actually very easy and doesn’t take much longer than the straight upgrade.

The bug has been fixed in 3.6.4 so you should only run into this on 3.6.0 to 3.6.3.

Check the current boot variable

test-sw#sh boot
---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable does not exist

Boot Variables on next reload:
BOOT variable = flash:packages.conf

Obviously the switch is running install mode because it’s booting from “packages.conf.”

List the files in flash

test-sw#dir
Directory of flash:/

24242  -rwx     2097152  Sep 11 2019 23:00:03 -07:00  nvram_config
24243  -rw-         804  Jun 18 2015 14:28:13 -07:00  vlan.dat
24244  -rw-   302112348   Sep 9 2019 21:01:50 -07:00  cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin
24251  -rw-        1236  Jun 16 2015 06:51:25 -07:00  packages.conf
56561  drwx        4096  Jun 16 2015 06:51:53 -07:00  mnt
24252  -rw-    82653508  Jun 16 2015 06:51:13 -07:00  cat3k_caa-base.SPA.03.06.01E.pkg
24253  -rw-     6625980  Jun 16 2015 06:51:13 -07:00  cat3k_caa-drivers.SPA.03.06.01E.pkg
24254  -rw-    33749996  Jun 16 2015 06:51:13 -07:00  cat3k_caa-infra.SPA.03.06.01E.pkg
24255  -rw-    42827072  Jun 16 2015 06:51:13 -07:00  cat3k_caa-iosd-universalk9.SPA.152-2.E1.pkg
24256  -rw-    25727884  Jun 16 2015 06:51:13 -07:00  cat3k_caa-platform.SPA.03.06.01E.pkg
24257  -rw-    99240768  Jun 16 2015 06:51:14 -07:00  cat3k_caa-wcm.SPA.10.2.111.0.pkg
56562  drwx        4096  Aug 14 2019 18:18:34 -07:00  dc_profile_dir

1562509312 bytes total (919986176 bytes free)



Verify the .bin file's md5 hash

test-sw#verify /md5 cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin
..........................................................................................................................…
Done!
verify /md5 (flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin) = ac15e03a732a23e894d3943d667ec168

The hash is correct so we can continue.


Set the boot variable to boot the .bin file

test-sw#conf t
test-sw(config)#no boot system
test-sw(config)#boot system sw all flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin
test-sw(config)#end
test-sw#wr mem
Building configuration...
Compressed configuration from 33642 bytes to 11246 bytes[OK]


Check the boot variable

test-sw#sh boot
---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable = flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin;

Boot Variables on next reload:
BOOT variable = flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin;

Note: If you check the boot variable immediately after the wr mem it may still show

Boot Variables on next reload:
BOOT variable = flash:packages.conf

If it does, just wait 10 seconds and try again.

Boot into bundle mode

test-sw#reload
Reload command is being issued on Active unit, this will reload the whole stack
Proceed with reload? [confirm]

----------------------After the reload-----------------------------


Expand the .bin file

test-sw#software expand file flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin
Preparing expand operation ...
[1]: Expanding bundle flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin
[1]: Copying package files
[1]: A different version of provisioning file packages.conf already exists in flash:.
    The provisioning file from the expanded bundle will be saved as
    flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.conf
[1]: Package files copied
[1]: Finished expanding bundle flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin

Update the packages.conf file

test-sw#delete flash:packages.conf
Delete filename [packages.conf]?
Delete flash:/packages.conf? [confirm]
test-sw#rename flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.conf packages.conf
Destination filename [packages.conf]?

Verify that packages.conf contains the 3.6.10 files

test-sw#more flash:packages.conf
#! /usr/binos/bin/packages_conf.sh

sha1sum: b844cd3af271a3dd0fcfd90d38bb4f58f8b4f531
iso   rp 0 0   rp_base       cat3k_caa-base.SPA.03.06.10E.pkg
iso   rp 0 0   rp_infra       cat3k_caa-infra.SPA.03.06.10E.pkg
iso   rp 0 0   rp_platform       cat3k_caa-platform.SPA.03.06.10E.pkg
iso   rp 0 0   rp_iosd       cat3k_caa-iosd-universalk9.SPA.152-2.E10.pkg
iso   rp 0 0   rp_wcm       cat3k_caa-wcm.SPA.10.2.200.0.pkg
iso   rp 0 0   drivers       cat3k_caa-drivers.SPA.03.06.10E.pkg


Update the Boot Variable for install mode

test-sw#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
test-sw(config)#no boot system
test-sw(config)#boot system flash:packages.conf
test-sw(config)#end
test-sw#wr mem
Building configuration...
Compressed configuration from 33566 bytes to 10974 bytes[OK]
test-sw#sh boot
---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable = flash:packages.conf;

Boot Variables on next reload:
BOOT variable = flash:packages.conf;
Allow Dev Key = yes
Manual Boot = no
Enable Break = no

Reload the switch to boot into 3.6.10

test-sw#reload


Here are the commands ready to paste in:

sh boot
dir
verify /md5  cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin
conf t
no boot system
boot system sw all flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin
end
wr mem
sh boot
reload

software expand file flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin
delete flash:packages.conf
rename flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.conf packages.conf

more flash:packages.conf
conf t
no boot system
boot system flash:packages.conf
end
wr mem
sh boot

Tuesday, August 27, 2019

Configuring Cisco Smart Licensing on the Catalyst Platform

According to Cisco Smart Licensing is the future. From the Cisco website:

"Smart Licensing is a cloud-based, software license management solution that allows you to manage and track the status of your license and hardware and software usage trends. Smart Licensing also enables you to automate time-consuming, manual licensing tasks."

The important statement on the website is this statement:

"Licenses are managed as smart licenses from Cisco IOS XE Fuji 16.9.1 and later."

That means that as soon as you upgrade to 16.9.1 or later your Right to Use licensing will no longer work. So how do you set up Smart Licensing on the switch?

It's actually pretty easy but it does require Internet access. You can use a proxy but I am not going to cover that today. If you are building the switches at your office for a customer you can configure the management interface and use your internal network. That is what I am doing in this example. If your switch already has internet access you can skip the management interface configuration.

Set the correct license level

There will be a printed card in the box with the swiitch that lists the type and quantity of licenses that were purchased. Notice in the picture below that the quantity is 17. That is because there were 17 switches on the sales order.

Near the bottom is a bar code label "Order #". This is what Cisco calls the Sales Order (SO) number. You will need this if you contact TAC or the licensing team.



My switches were purchased with network-advantage licensing so I entered the code listed below. If you don't do this, the call-home service will try to register DNA Advantage and Network Advantage licenses. It works, but you get an Alert on the CSSM portal and you will see this in the output of "show license all"

License Usage
==============

C9300 48P DNA Advantage (C9300-48 DNA Advantage):
  Description: C9300 48P DNA Advantage
  Count: 4
  Version: 1.0
  Status: OUT OF COMPLIANCE
  Export status: NOT RESTRICTED


Enter the following to set the license level to network-advantage. Note that a reload is required.


test#Conf t
test(config)#license boot level network-advantage 
test(config)#exit 
test#write memory 
test#show version (showing just the relevant output)

Technology Package License Information:

------------------------------------------------------------------------------
Technology-package                                     Technology-package
Current                        Type                       Next reboot
------------------------------------------------------------------------------
network-advantage       Smart License                    network-advantage
dna-advantage           Subscription Smart License       None
AIR License Level: AIR DNA Advantage
Next reload AIR license Level: AIR DNA Advantage


reload


If you purchased additional add ons such as DNA your boot level command will look like this:

license boot level network-advantage addon dna-advantage 


Configure the Management interface

I was connecting to a LAN with a DHCP server so I used DHCP. If you don't have access to a DHCP server use a valid static IP address.

conf t
test(config)#interface GigabitEthernet0/0
test(config-if)#ip address dhcp
test(config-if)#no shut
exit

Configure routing and host resolution

Smart Licensing uses the "Call-Home" service and must be able to reach tools.cisco.com. I added a host entry since this customer doesn't allow name servers.

Since I am using the management interface I had to use the Mgmt-vrf for the host. Once the switch is installed, the management interface will be down so I added a host outside the Mgmt-vrf

test(config)#ip host vrf Mgmt-vrf tools.cisco.com 72.163.4.38
test(config)#ip host tools.cisco.com 72.163.4.38
test(config)#ip route vrf Mgmt-vrf 72.163.4.38 255.255.255.255 10.253.4.1
test(config)#ip http client source-interface g0/0

The "ip http client" statement must be in the configuration while using the management interface. You must remove if you don't use the management interface after install.

Make sure you can ping tools.cisco.com:

ping vrf Mgmt-vrf tools.cisco.com
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 72.163.4.38, timeout is 2 seconds:
!!!!!

Configure the Smart Call-Home service


test(config)#service call-home
test(config)#license smart transport callhome
test(config)#call-home
test(cfg-call-home)#no http secure server-identity-check
test(cfg-call-home)#profile CiscoTAC-1
test(cfg-call-home-profile)#reporting all
test(cfg-call-home-profile)#destination transport-method http
test(cfg-call-home-profile)#no destination transport-method email
test(cfg-call-home-profile)active
test(cfg-call-home-profile)#end
test#wr mem

Use show commands to review:
sh run | sec call-home
service call-home
call-home
 ! If contact email address in call-home is configured as sch-smart-licensing@cisco.com
 ! the email address configured in Cisco Smart License Portal will be used as contact email address to send SCH notifications.
 contact-email-addr sch-smart-licensing@cisco.com
 no http secure server-identity-check
 profile "CiscoTAC-1"
  active
  destination transport-method http
  no destination transport-method email


sh run | i call
service call-home
call-home
 ! If contact email address in call-home is configured as sch-smart-licensing@cisco.com
license smart transport callhome



Create the Smart Account

Smart licensing requires a "Smart" account before you can activate the license on the switch. To create your account go to CSSM.

When the page loads you will see 5 sections. Look for Administration:


Click on "Request a Smart Account" and follow the instructions.

Create Token

You will need a "token" that gets pasted into the switch. Once you have your account, log in and click on Smart Licensing in the "License" section:




When the page loads click on the Inventory tab.

Click the General tab, click "New Token..."



In the dialog that opens, enter a description and set the number of uses. Cisco recommends 30 and that is what I have been using.



Click "Create Token". You will return to the previous screen. Click on the blue diagonal arrow on the right of the token to copy it to the clipboard. Save the token for use on the switch.

Enter the Token on the switch
From enable mode, not configuration mode, enter:
license smart register idtoken <your token>

and press enter.

You will see "Registration process is in progress. Use the 'show license status' command to check the progress and result" in the CLI.

You can use "show license status" to check on the progress. If everything worked you will see:

Registration:
  Status: REGISTERED
  <Account name>
  Virtual Account: DEFAULT
  Export-Controlled Functionality: ALLOWED
  Initial Registration: SUCCEEDED on Aug 27 13:58:49 2019 PDT
  Last Renewal Attempt: None
  Next Renewal Attempt: Feb 23 13:58:48 2020 PDT
  Registration Expires: Aug 26 13:52:59 2020 PDT

What if it didn't work?

As you can see, there are a lot of things that must go right for this to work. One thing I have run into is the licenses have to be associated with the switch serial number. If they aren't you will see:

License Authorization:
  Status: OUT OF COMPLIANCE on Aug 27 13:58:53 2019 PDT
  Last Communication Attempt: SUCCEEDED on Aug 27 13:58:53 2019 PDT

On the switch. Back on the portal, click on the "Alerts" tab and you will see:


You will need to contact TAC and get the liscense associated with the switch.


If you receive the message
"Operation not supported because the agent is running in Permanent License Reservation mode"

Run the following
(config)#no license smart reservation 



test#sh call-home profile all


Profile Name: CiscoTAC-1
    Profile status: ACTIVE
    Profile mode: Full Reporting
    Reporting Data: Smart Call Home, Smart Licensing
    Preferred Message Format: xml
    Message Size Limit: 3145728 Bytes
    Transport Method: http
    HTTP  address(es): https://tools.cisco.com/its/service/oddce/services/DDCEService
    Other address(es): default

    Periodic configuration info message is scheduled every 1 day of the month at 09:15

    Periodic inventory info message is scheduled every 1 day of the month at 09:00

    Alert-group               Severity
    ------------------------  ------------
    crash                     debug
    diagnostic                minor
    environment               warning
    inventory                 normal

    Syslog-Pattern            Severity
    ------------------------  ------------
    APF-.-WLC_.*              warning
    .*                        major



Show full license status
You can use "show license all" to see the complete license status


test#sh license all
Smart Licensing Status
======================

Smart Licensing is ENABLED

Registration:
  Status: REGISTERED
  Smart Account: The perfect customer
  Virtual Account: DEFAULT
  Export-Controlled Functionality: ALLOWED
  Initial Registration: SUCCEEDED on Aug 27 13:58:49 2019 PDT
  Last Renewal Attempt: SUCCEEDED on Aug 28 09:01:22 2019 PDT
  Next Renewal Attempt: Feb 24 09:01:21 2020 PDT
  Registration Expires: Aug 27 08:55:36 2020 PDT

License Authorization:
  Status: OUT OF COMPLIANCE on Aug 27 13:58:53 2019 PDT
  Last Communication Attempt: SUCCEEDED on Aug 28 09:01:28 2019 PDT
  Next Communication Attempt: Aug 28 21:01:28 2019 PDT
  Communication Deadline: Nov 26 08:55:44 2019 PDT

Export Authorization Key:
  Features Authorized:
    <none>

Utility:
  Status: DISABLED

Data Privacy:
  Sending Hostname: yes
    Callhome hostname privacy: DISABLED
    Smart Licensing hostname privacy: DISABLED
  Version privacy: DISABLED

Transport:
  Type: Callhome

License Usage
==============

C9300 48P DNA Advantage (C9300-48 DNA Advantage):
  Description: C9300 48P DNA Advantage
  Count: 4
  Version: 1.0
  Status: OUT OF COMPLIANCE
  Export status: NOT RESTRICTED

C9300 48P NW Advantage (C9300-48 Network Advantage):
  Description: C9300 48P NW Advantage
  Count: 4
  Version: 1.0
  Status: AUTHORIZED
  Export status: NOT RESTRICTED

Product Information
===================
UDI: PID:C9300-48UXM,SN:FJC2324S042

HA UDI List:
    Active:PID:C9300-48UXM,SN:XXXXXXXS042
    Standby:PID:C9300-48UXM,SN:XXXXXXXE014
    Member:PID:C9300-48UXM,SN:XXXXXXXE04M
    Member:PID:C9300-48UXM,SN:XXXXXXXB02D

Agent Version
=============
Smart Agent for Licensing: 4.8.5.1_rel/8

Reservation Info
================
License reservation: DISABLED


References

Configuring Smart Licensing