Hubbard on Networking: February 2019

Sunday, February 17, 2019

Locate IP devices on the wrong vlan

When replacing switches, sometimes a device like a Building Automation Control (BACnet) controllers or fire alarm quits working. This is usually because they got patched into the wrong port and now the IP address doesn’t work.

Since these devices normally have static IP addresses and seldom send a packet outbound, their MAC addresses don't register on the switch. This can lead to lost time and aggravation trying to get them back online, especially if they are in a remote cabinet or a switch that is physically hard to reach. This blog will show a process to reduce the time needed to locate the port the device is connected to, so that the problem can be resolved.

Here are the steps to take before the cutover

On the core switch

First, ping all host addresses for subnets of interest on the core switch. This will refresh the arp cache so that devices like BACnet controllers and alarms that have timed out will be in the cache. To do that:

Execute show run | i ^_ip address - The i means include, the ^ means start at the first character, the _ means look for one space, and ip address is the string to look for. This regex returns just IP addresses from the SVIs, not every instance of the string ip address.
Copy the output to a text file named vlans.txt
Run the python3 script pinger.py - You can download the script and read its documentation here. This script converts the subnet address into hosts and pings each host.
Execute show ip arp
Copy the output to a text file named arp.txt
Run the python3 script arp.py - You can download the script and read its documentation here. This script creates a json database of the MAC address/IP address mappings and is used with the next script.

On each edge switch

Execute show mac add int g1/0/1 | i Gi for each edge port on the switch.
Copy the output to a text file named mac-addr.txt. I created a spreadsheet with the necessary commands for several Cisco models and Rukus (Brocade) switches. You can download it here.
Run the python3 script macaddr.py - You can download the script and read its documentation here.

The last step creates a listing of switch ports with the IP address, MAC address, Port and MAC Manufacture:

Device Name: Test.MDF
Vlan   IP Address       MAC Address       Type       Interface   Vendor
--------------------------------------------------------------------------------
16   172.16.16.9        0020.4adb.3e21    DYNAMIC    Gi1/0/5     Pronet
--------------------------------------------------------------------------------
26   172.16.26.94       54ee.7505.86b5    DYNAMIC    Gi1/0/14    WistronI
--------------------------------------------------------------------------------
23   172.16.23.117      5442.49a1.06c6    DYNAMIC    Gi1/0/47    Sony
--------------------------------------------------------------------------------
20   172.16.20.153      000c.820d.007e    DYNAMIC    Gi2/0/24    NetworkT
--------------------------------------------------------------------------------
20   172.16.20.96       0040.9d97.a3aa    DYNAMIC    Gi2/0/26    Digiboar
--------------------------------------------------------------------------------
20   172.16.21.96       cc72.0fff.f6a5    DYNAMIC    Gi2/0/27    Viscount
--------------------------------------------------------------------------------
20   172.16.25.96       0024.7900.095b    DYNAMIC    Gi2/0/30    OptecDis
--------------------------------------------------------------------------------

For this example, I have trimmed the list so it only shows devices that I know are static IP devices like Pronet which is a serial to IP device server, Sony, which in this case is a Surveillance camera, Digiboard which is a serial to IP console server, etc. These are all devices that are critical to the company's operation and I want to make sure they work after the cut over.

The macaddr.py script also creates a section with just the IP address and MAC address. Here is a snippet of it:

Device Name: Test.MDF
PingInfo Data
172.16.16.9 0020.4adb.3e21
172.16.26.94 54ee.7505.86b5

I take that information and save it to a text file named PingInfo-xxx, where xxx is something meaningful for the site. I use the text file with a free program PingInfoView from www.nirsoft.net to create a dashboard of live ping results. Below is an example of a PingInfo dashboard. PingInfo is Windows only!

PingInfoView continuously pings the addresses and any address that stops responding turns red. Obviously, we want all addresses to be green after the cut over!

After the cut over

The PingInfo dashboard should be all green. But what do you do if one of the static ip devices isn't live in the dashboard? Since many of these devices are Operational Technology (OT) versus IT they seldom send any traffic onto the network so immediately after a cut over the port they are connected to won't have a MAC address. If it got connected to a port on the wrong vlan you won't be able to ping it to populate the MAC address table.

There is a Linux tool called arp-scan by Roy Hill that you can use to send arp requests to a device. It can also send arp requests with an 802.1q vlan tag. This is the key to finding the lost device.

How to use arp-scan

Here is the network diagram for our example:

In my lab, I used a Kali VM to simulate an OT device. At a real customer site, you obviously wouldn't know what port the device is connected to.

First, you will need to install arp-scan on the Ubuntu box:

sudo apt install arp-scan

You can use arp-scan --help to see all of the options for arp-scan. It has a lot of uses, if you Google arp-scan examples, you will find a lot of good ones. I wrote a pyhton script that uses arp-scan to find open IP addresses on a subnet. It's useful when you need to connect to a subnet that doesn't have DHCP but you don't want to cause an IP address conflict. You can download it here.

To use arp-scan in our case, we need to know the interface on the Ubuntu box, the MAC address of the device (Destination MAC), the vlan ID of the switch port and the ip network the device is on.

For the vlan ID, we select one from the vlans configured on the switch. You can use show vlan brief to see a list of vlans assigned to ports. If the switch has a lot of vlans defined you may have to run the command several times, iterating through different vlans.

In this example:

Linux interface - enp2s0f1
vlan ID - 46 (10.112.46.0/23)
Device MAC – 00:90:9e:9a:b5:3d
Device IP – 10.112.100.1
Device Subnet - 10.112.100.0/24

As you can see in the diagram, the simulated device is on port g1/0/48 which is configured as an access port on vlan 46. The Linux box with arp-scan is on a trunk port. The requirement here is that the trunk can pass vlan tags on vlan 46.

It doesn't matter if the trunk has a native vlan with DHCP. Security best practices dictates that the native vlan nk on a trunk be an unused vlan. For our purposes, we don’t care if the interface has an IP address since we are using arp. You will see in the output that the IP address on the interface isn’t set.

Interface configuration for the Ubuntu laptop

sh run int g1/0/47
interface GigabitEthernet1/0/47
switchport trunk encapsulation dot1q
switchport mode trunk
end

On theUbuntu laptop connected to port g1/0/47

ip addr show enp2s0f1

2: enp2s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 80:fa:5b:31:de:85 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::6d35:51a0:cef:4475/64 scope link noprefixroute
       valid_lft forever preferred_lft forever

You can see that it doesn't have an IP address.

Interface configuration for the simulated device

sh run int g1/0/48
interface GigabitEthernet1/0/48
switchport access vlan 46
switchport mode access
end

SVI Interfaces

sh run int vl 46
interface Vlan46
ip address 10.112.47.254 255.255.254.0
ip helper-address 192.168.10.221
no ip redirects
end

sh run int vl 100
interface Vlan100
ip address 10.112.100.254 255.255.255.0
ip helper-address 192.168.10.221
no ip redirects
end

Run arp-scan

sudo arp-scan -I enp2s0f1 -Q 46 --destaddr=00:90:9e:9a:b5:3d 10.112.100.0/24
WARNING: Could not obtain IP address for interface enp2s0f1. Using 0.0.0.0 for
the source address, which is probably not what you want.
Either configure enp2s0f1 with an IP address, or manually specify the address
with the --arpspa option.
Interface: enp2s0f1, datalink type: EN10MB (Ethernet)
Starting arp-scan 1.9 with 256 hosts (http://www.nta-monitor.com/tools/arp-scan/)
10.112.100.1 00:90:9e:9a:b5:3d Critical IO, LLC (802.1Q VLAN=46)

1 packets received by filter, 0 packets dropped by kernel
Ending arp-scan 1.9: 256 hosts scanned in 2.556 seconds (100.16 hosts/sec). 1 responded

You can see that arp-scan found the device by mac address and the device replied with its IP address.

Here is what it looked like in Wireshark on the Ubuntu box:

You can see that arp-scan sent 802.1q tag 46 even though I hadn't configured a subinterface on the Ubuntu box. I have a blog showing how to configure Ubuntu to use vlan tags. Here is a link to the blog - Bypass VTY access lists with Linux and Yersinia

Now on the switch, look for the mac address.

sh mac add | i 0090.9e9a.b53d
46 0090.9e9a.b53d DYNAMIC Gi1/0/48

You can either move the device to the correct port or reconfigure the port for the correct vlan. If the customer allows it, I like to label this type of device's switch port like this:

des < BACnet 10.112.100.1 0090.9e9a.b53d>

That gives me the MAC and IP for future troubleshooting.

Let's set up a lab to learn how to use arp-scan.

It's a simple setup, but there are few gotcha's if you are new to Linux or arp-scan. You will need a Linux machine, physical or virtual, and a Kali Linux virtual machine. The Kali VM will simulate our OT device because it is designed to be quiet on the network. The kali motto is "The quieter you are, the more you can hear".

We will use the same network layout as above so you can refer to that diagram.

On the Kali VM

Configure Kali with a static ip address. In this example:

10.112.100.1/24
255.255.255.0
NO GATEWAY - If you assign a gateway the Kali box will send ARP requests to the gateway and populate the switch's mac address table.

Connect the Kali VM to switch port g1/0/48

Check to see if the Kali box sent traffic that caused the switch to record it's MAC address:

sh mac add int g1/0/48
          Mac Address Table
-------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
46    0090.9e9a.b53d    DYNAMIC     Gi1/0/48
Total Mac Addresses for this criterion: 1

Kali did send some traffic and that populated the mac address table. For our purposes, we need to make sure the Kali box isn't sending any traffic. Normally, you do not want your pen test box to send traffic you didn't explicitly request. That is why Kali has DHCP and other services disabled by default.

In this case, I had been using DHCP before switching to static so the dhclient service was still running.

On Kali run

ps -ef | grep dh
root 2500 2125 0 13:18 pts/0 00:00:00 dhclient -v

If you see the dhclient service, then run
dhclient -r

to stop the service. If you don't stop it, the dhclient service will do a discover which will populate the MAC-address table on the switch and invalidates the test we are about to try.

Next, run the following to clear the mac address on the switch port. Sometimes you need to run it a few times before the address is cleared.

clear mac address-table dynamic interface g1/0/48

Finally, you should see an empty mac table for interface g1/0/48

sh mac add int g1/0/48
          Mac Address Table
-------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----

Try to ping the device

ping 10.112.100.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.112.100.1, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

Connect your laptop to port 1/0/47. I run the lldpd daemon (sudo apt install lldpd) so I can verify that the laptop is connected using the following.

sh lldp ne
Capability codes:
    (R) Router, (B) Bridge, (T) Telephone, (C) DOCSIS Cable Device
    (W) WLAN Access Point, (P) Repeater, (S) Station, (O) Other

Device ID           Local Intf     Hold-time Capability      Port ID
PROCURVE J9450A     Gi1/1/4        120        B               2
1S1K-SYS76          Gi1/0/47       120        B,W,R           80fa.5b31.de85

Total entries displayed: 2

Look at the interface that the laptop is connected to:

sh run int g1/0/47

interface GigabitEthernet1/0/47
switchport trunk encapsulation dot1q
switchport mode trunk
end

Look at the interface on the laptop after it's connected to the switch

ip addr show enp2s0f1

You can see that the interface is UP/UP but doesn't have an IP address.

Look at the SVIs

sh run int vl 46

interface Vlan46
ip address 10.112.47.254 255.255.254.0
ip helper-address 192.168.10.221
no ip redirects
end

sh run int vl 100

interface Vlan100
ip address 10.112.100.254 255.255.255.0
ip helper-address 192.168.10.221
no ip redirects
end

Now we are ready to run arp-scan and find the device:

sudo arp-scan -I enp2s0f1 -Q 46 --destaddr=00:90:9e:9a:b5:3d 10.112.100.0/24

WARNING: Could not obtain IP address for interface enp2s0f1. Using 0.0.0.0 for
the source address, which is probably not what you want.
Either configure enp2s0f1 with an IP address, or manually specify the address
with the --arpspa option.
Interface: enp2s0f1, datalink type: EN10MB (Ethernet)
Starting arp-scan 1.9 with 256 hosts (http://www.nta-monitor.com/tools/arp-scan/)
10.112.100.1 00:90:9e:9a:b5:3d Critical IO, LLC (802.1Q VLAN=46)

1 packets received by filter, 0 packets dropped by kernel
Ending arp-scan 1.9: 256 hosts scanned in 2.556 seconds (100.16 hosts/sec). 1 responded

Verify that the mac-address table on the switch has been updated

sh mac add | i 0090.9e9a.b53d
46 0090.9e9a.b53d DYNAMIC Gi1/0/48

@rikosintie

Michael earned his first IT certification, a Novell Certified Netware Engineer (CNE) in 1993. Michael then focused on the Microsoft MCSE credential, studied database design, and wrote several Visual Basic programs to automate production at a manufacturing company. In 2000 he earned the Cisco CCNA. Since then, Michael has earned Cisco’s security certification, Aruba’s switching certification, and the LPI’s Linux Essentials certification.

Friday, February 8, 2019

Ubiquiti Discovery Protocol - Find the firmware version-SSID-Model

Ubiquiti devices use UDP on port 10001 for autodiscover of other Ubiquiti devices. If you are hard coding IPs and managing the devices individually, you can disable this discovery process. The references below have two links to Ubiquiti KBs on the issue. Here are the instructions:

SSH into the device

ssh <username>@IP-Address
configure
set service ubnt-discover interface <interface-name> disable
commit
save

On a router, you will definitely want to disable it on the WAN interface. If you are using Ubiquiti's cloud management you will need to verify that it works after making this change.

SSH into the device

ssh <username>@IP-Address
configure
set service ubnt-discover interface <interface-name> disable
commit
save

To look at what the discover sends out, you can use the following commands. No authentication is required to get the output. As with all network devices, you should use a dedicated management vlan and ACL it off so that only authorized stations can access the management interface.

This is from a NanoStation 5 AC loco in my lab. It has the following settings:

SSID - death2all
firmware version - WA.v8.5.11.39842.190109.1449.bin
Device name - Office

echo -ne "\x01\x00\x00\x00" | socat -t 1 udp:192.168.10.50:10001 - | hexdump -C

echo -ne "\x01\x00\x00\x00" | socat -t 1 udp:192.168.10.50:10001 - | hexdump -C
00000000  01 00 00 9b 01 00 06 fc  ec da c4 6e 55 02 00 0a  |...........nU...|
00000010  fc ec da c4 6e 55 c0 a8  0a 32 02 00 0a fc ec da  |....nU...2......|
00000020  c4 6e 55 a9 fe 6e 55 03  00 23 57 41 2e 61 72 39  |.nU..nU..#WA.ar9|
00000030  33 34 78 2e 76 38 2e 35  2e 31 31 2e 33 39 38 34  |34x.v8.5.11.3984|
00000040  32 2e 31 39 30 31 30 39  2e 31 34 34 39 0a 00 04  |2.190109.1449...|
00000050  00 00 38 69 0b 00 06 4f  66 66 69 63 65 0c 00 03  |..8i...Office...|
00000060  4e 35 4c 0d 00 09 64 65  61 74 68 32 61 6c 6c 0e  |N5L...death2all.|
00000070  00 01 03 10 00 02 e7 fa  13 00 06 fc ec da c4 6e  |...............n|
00000080  55 14 00 14 4e 61 6e 6f  53 74 61 74 69 6f 6e 20  |U...NanoStation |
00000090  35 41 43 20 6c 6f 63 6f  18 00 04 00 00 00 00     |5AC loco.......|

Explanation of the Linux commands

Echo
echo the STRING(s) to standard output.
-n do not output the trailing newline
-e enable interpretation of backslash escapes

socat
Socat is like the cat command but it can transfer data between two locations instead of just from a file to stdout.
-t Delay
- Write to stdout

hexdump
From the man page - The hexdump utility is a filter which displays the specified files, or the standard input, if no files are specified, in a user specified format.
-C Canonical hex+ASCII display. Display the input offset in hexadecimal, followed by sixteen space-separated, two column, hexadecimal bytes, followed by the same sixteen bytes in %_p format enclosed in ``|'' characters.

To look for multiple devices, you can use this simple loop. Change the IP to match your network.

for ip in 192.168.10.{50..51}
 do
         echo "------ ${ip} ------" 
         echo -ne "\x01\x00\x00\x00" | socat -t 1 udp:$ip:10001 - | hexdump -C
done

------ 192.168.10.50 ------
00000000  01 00 00 9a 01 00 06 fc  ec da c4 6e 55 02 00 0a  |...........nU...|
00000010  fc ec da c4 6e 55 c0 a8  0a 32 02 00 0a fc ec da  |....nU...2......|
00000020  c4 6e 55 a9 fe 6e 55 03  00 22 57 41 2e 61 72 39  |.nU..nU.."WA.ar9|
00000030  33 34 78 2e 76 38 2e 35  2e 38 2e 33 38 38 34 35  |34x.v8.5.8.38845|
00000040  2e 31 38 30 39 31 38 2e  31 30 31 36 0a 00 04 00  |.180918.1016....|
00000050  af 11 86 0b 00 06 4f 66  66 69 63 65 0c 00 03 4e  |......Office...N|
00000060  35 4c 0d 00 09 64 65 61  74 68 32 61 6c 6c 0e 00  |5L...death2all..|
00000070  01 03 10 00 02 e7 fa 13  00 06 fc ec da c4 6e 55  |..............nU|
00000080  14 00 14 4e 61 6e 6f 53  74 61 74 69 6f 6e 20 35  |...NanoStation 5|
00000090  41 43 20 6c 6f 63 6f 18  00 04 00 00 00 00        |AC loco.......|
0000009e

------ 192.168.10.51 ------
00000000  01 00 00 9b 01 00 06 fc  ec da c4 77 0b 02 00 0a  |...........w....|
00000010  fc ec da c4 77 0b c0 a8  0a 33 02 00 0a fc ec da  |....w....3......|
00000020  c4 77 0b a9 fe 77 0b 03  00 23 57 41 2e 61 72 39  |.w...w...#WA.ar9|
00000030  33 34 78 2e 76 38 2e 35  2e 31 31 2e 33 39 38 34  |34x.v8.5.11.3984|
00000040  32 2e 31 39 30 31 30 39  2e 31 34 34 39 0a 00 04  |2.190109.1449...|
00000050  00 00 08 29 0b 00 06 47  61 72 61 67 65 0c 00 03  |...)...Garage...|
00000060  4e 35 4c 0d 00 09 64 65  61 74 68 32 61 6c 6c 0e  |N5L...death2all.|
00000070  00 01 02 10 00 02 e7 fa  13 00 06 fc ec da c4 77  |...............w|
00000080  0b 14 00 14 4e 61 6e 6f  53 74 61 74 69 6f 6e 20  |....NanoStation |
00000090  35 41 43 20 6c 6f 63 6f  18 00 04 00 00 00 00     |5AC loco.......|
0000009f

Look for outdated or mismatched firmware

I find this useful on customer networks. It lets me quickly check for outdated or mismatched firmware versions.

for ip in 192.168.10.{50..51}
 do
         echo "------ ${ip} ------" 
         echo -ne "\x01\x00\x00\x00" | socat -t 1 udp:$ip:10001 - | hexdump -C | grep v
done
------ 192.168.10.50 ------
00000030  33 34 78 2e 76 38 2e 35  2e 38 2e 33 38 38 34 35  |34x.v8.5.8.38845|
------ 192.168.10.51 ------
00000030  33 34 78 2e 76 38 2e 35  2e 31 31 2e 33 39 38 34  |34x.v8.5.11.3984|

Nmap
There is an Nmap script for Ubiquiti Discovery - ubiquiti-discovery.nse. It pulls down more information than the bash script and will work on Windows. The home page for the script is here.

You will need to download two files from the nmap repository:

tableaux.lua

ubiquiti-discovery.nse

On Windows

Save tableaux.lua to c:\Program Files (x86)\nselib

Save ubiquiti-discovery.nse to c:\Program Files (x86)\scripts

On Linux, as root

Save tableaux.lua to /usr/share/nmap/nselib

Save ubiquiti-discovery.nse to /usr/share/nmap/scripts

sudo nmap -sU -p 10001 --script ubiquiti-discovery.nse -oG ubnt 192.168.10.50

Starting Nmap 7.60 ( https://nmap.org ) at 2019-02-10 22:16 PST
Nmap scan report for 192.168.10.50
Host is up (0.0027s latency).

PORT      STATE SERVICE
10001/udp open  ubiquiti-discovery
| ubiquiti-discovery: 
|   protocol: v1
|   firmware: WA.ar934x.v8.5.11.39842.190109.1449
|   version: v8.5.11
|   uptime_seconds: 196320
|   uptime: 2 days 06:32:00
|   hostname: Office
|   product: N5L
|   essid: death2all
|   model: NanoStation 5AC loco
|   interface_to_ip: 
|     fc:ec:da:c4:6e:55: 
|       192.168.10.50
|       169.254.110.85
|   mac_addresses: 
|_    fc:ec:da:c4:6e:55
MAC Address: FC:EC:DA:C4:6E:55 (Ubiquiti Networks)
Service Info: OS: Linux

Nmap done: 1 IP address (1 host up) scanned in 0.68 seconds

On Linux, If you want to compare the firmware of more than one device:

sudo nmap -sU -p 10001 --script ubiquiti-discovery.nse -oG ubnt 192.168.10.50-51 | grep firmware
|   firmware: WA.ar934x.v8.5.11.39842.190109.1449
|   firmware: WA.ar934x.v8.5.11.39842.190109.1449

Remember, as always, only run discovery scripts on networks you have explicit permission on.

References

Understanding Ubiquiti Discovery Service Exposures
Rapid7 Sonar Project
Understanding UDP Amplification Vulnerabilities
Add Metasploit module to discover Ubiquiti devices
UDP broadcasts on port 10001 - Ubiquiti KB on disabling discovery protocol
EdgeRouter - Ubiquiti Device Discovery - Ubiquiti KB on disabling discovery protocol on routers
Security Now show notes - Search for ubiquiti
hexdump command in Linux with examples
socat: The General Bidirectional Pipe Handler
Source Code Beautifier - Used to create the code blocks in this blog
Shodan - Search Engine for the Internet of Things

@rikosintie