Locate IP devices on the wrong vlan

When replacing switches, sometimes a device like a Building Automation Control (BACnet) controllers or fire alarm quits working. This is usually because they got patched into the wrong port and now the IP address doesn’t work.

Since these devices normally have static IP addresses and seldom send a packet outbound, their MAC addresses don't register on the switch. This can lead to lost time and aggravation trying to get them back online, especially if they are in a remote cabinet or a switch that is physically hard to reach. This blog will show a process to reduce the time needed to locate the port the device is connected to, so that the problem can be resolved.

Here are the steps to take before the cutover

On the core switch

 

First, ping all host addresses for subnets of interest on the core switch. This will refresh the arp cache so that devices like BACnet controllers and alarms that have timed out will be in the cache. To do that:

• Execute show run | i ^_ip address - The i means include, the ^ means start at the first character, the _ means look for one space, and ip address is the string to look for. This regex returns just IP addresses from the SVIs, not every instance of the string ip address.
• Copy the output to a text file named vlans.txt
• Run the python3 script pinger.py - You can download the script and read its documentation here. This script converts the subnet address into hosts and pings each host.
• Execute show ip arp
• Copy the output to a text file named arp.txt
• Run the python3 script arp.py - You can download the script and read its documentation here. This script creates a json database of the MAC address/IP address mappings and is used with the next script.

On each edge switch

• Execute show mac add int g1/0/1 | i Gi for each edge port on the switch.
• Copy the output to a text file named mac-addr.txt. I created a spreadsheet with the necessary commands for several Cisco models and Rukus (Brocade) switches. You can download it here.
• Run the python3 script macaddr.py  - You can download the script and read its documentation here.

The last step creates a listing of switch ports with the IP address, MAC address, Port and MAC Manufacture: 

Device Name: Test.MDF
Vlan   IP Address       MAC Address       Type       Interface   Vendor
--------------------------------------------------------------------------------
  16   172.16.16.9        0020.4adb.3e21    DYNAMIC    Gi1/0/5     Pronet
--------------------------------------------------------------------------------
  26   172.16.26.94       54ee.7505.86b5    DYNAMIC    Gi1/0/14    WistronI
--------------------------------------------------------------------------------
  23   172.16.23.117      5442.49a1.06c6    DYNAMIC    Gi1/0/47    Sony
--------------------------------------------------------------------------------
  20   172.16.20.153      000c.820d.007e    DYNAMIC    Gi2/0/24    NetworkT
--------------------------------------------------------------------------------
  20   172.16.20.96       0040.9d97.a3aa    DYNAMIC    Gi2/0/26    Digiboar
--------------------------------------------------------------------------------
  20   172.16.21.96       cc72.0fff.f6a5    DYNAMIC    Gi2/0/27    Viscount
--------------------------------------------------------------------------------
  20   172.16.25.96       0024.7900.095b    DYNAMIC    Gi2/0/30    OptecDis
--------------------------------------------------------------------------------

For this example, I have trimmed the list so it only shows devices that I know are static IP devices like Pronet which is a serial to IP device server, Sony, which in this case is a Surveillance camera, Digiboard which is a serial to IP console server, etc.  These are all devices that are critical to the company's operation and I want to make sure they work after the cut over. 

The macaddr.py script also creates a section with just the IP address and MAC address. Here is a snippet of it:

Device Name: Test.MDF
PingInfo Data
172.16.16.9 0020.4adb.3e21
172.16.26.94 54ee.7505.86b5

I take that information and save it to a text file named PingInfo-xxx, where xxx is something meaningful for the site. I use the text file with a free program PingInfoView from www.nirsoft.net to create a dashboard of live ping results. Below is an example of a PingInfo dashboard. PingInfo is Windows only!

PingInfoView continuously pings the addresses and any address that stops responding turns red. Obviously, we want all addresses to be green after the cut over!

After the cut over

The PingInfo dashboard should be all green. But what do you do if one of the static ip devices isn't live in the dashboard? Since many of these devices are Operational Technology (OT) versus IT they seldom send any traffic onto the network so immediately after a cut over the port they are connected to won't have a MAC address. If it got connected to a port on the wrong vlan you won't be able to ping it to populate the MAC address table.

There is a Linux tool called arp-scan by Roy Hill that you can use to send arp requests to a device. It can also send arp requests with an 802.1q vlan tag. This is the key to finding the lost device.

How to use arp-scan

 

Here is the network diagram for our example:

In my lab, I used a Kali VM to simulate an OT device. At a real customer site, you obviously wouldn't know what port the device is connected to.

First, you will need to install arp-scan on the Ubuntu box:

sudo apt install arp-scan

You can use arp-scan --help to see all of the options for arp-scan. It has a lot of uses, if you Google arp-scan examples, you will find a lot of good ones. I wrote a pyhton script that uses arp-scan to find open IP addresses on a subnet. It's useful when you need to connect to a subnet that doesn't have DHCP but you don't want to cause an IP address conflict. You can download it here.

To use arp-scan in our case, we need to know the interface on the Ubuntu box, the MAC address of the device (Destination MAC), the vlan ID of the switch port and the ip network the device is on. 

For the vlan ID, we select one from the vlans configured on the switch. You can use show vlan brief to see a list of vlans assigned to ports. If the switch has a lot of vlans defined you may have to run the command several times, iterating through different vlans.

In this example:

Linux interface - enp2s0f1
vlan ID - 46 (10.112.46.0/23)
Device MAC – 00:90:9e:9a:b5:3d
Device IP – 10.112.100.1
Device Subnet - 10.112.100.0/24

As you can see in the diagram, the simulated device is on port g1/0/48 which is configured as an access port on vlan 46. The Linux box with arp-scan is on a trunk port. The requirement here is that the trunk can pass vlan tags on vlan 46. 

It doesn't matter if the trunk has a native vlan with DHCP. Security best practices dictates that the native vlan nk on a trunk be an unused vlan. For our purposes, we don’t care if the interface has an IP address since we are using arp. You will see in the output that the IP address on the interface isn’t set.

Interface configuration for the Ubuntu laptop

sh run int g1/0/47
interface GigabitEthernet1/0/47
 switchport trunk encapsulation dot1q
 switchport mode trunk
end

On theUbuntu laptop connected to port g1/0/47

ip addr show enp2s0f1

 
2: enp2s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 80:fa:5b:31:de:85 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::6d35:51a0:cef:4475/64 scope link noprefixroute
       valid_lft forever preferred_lft forever

You can see that it doesn't have an IP address.

Interface configuration for the simulated device

sh run int g1/0/48
interface GigabitEthernet1/0/48
 switchport access vlan 46
 switchport mode access
end

SVI Interfaces

sh run int vl 46
interface Vlan46
 ip address 10.112.47.254 255.255.254.0
 ip helper-address 192.168.10.221
  no ip redirects
end

sh run int vl 100
interface Vlan100
 ip address 10.112.100.254 255.255.255.0
 ip helper-address 192.168.10.221
 no ip redirects
end

Run arp-scan

sudo arp-scan -I enp2s0f1 -Q 46 --destaddr=00:90:9e:9a:b5:3d 10.112.100.0/24
WARNING: Could not obtain IP address for interface enp2s0f1. Using 0.0.0.0 for
the source address, which is probably not what you want.
Either configure enp2s0f1 with an IP address, or manually specify the address
with the --arpspa option.
Interface: enp2s0f1, datalink type: EN10MB (Ethernet)
Starting arp-scan 1.9 with 256 hosts (http://www.nta-monitor.com/tools/arp-scan/)
10.112.100.1    00:90:9e:9a:b5:3d    Critical IO, LLC (802.1Q VLAN=46)

1 packets received by filter, 0 packets dropped by kernel
Ending arp-scan 1.9: 256 hosts scanned in 2.556 seconds (100.16 hosts/sec). 1 responded

You can see that arp-scan found the device by mac address and the device replied with its IP address.

Here is what it looked like in Wireshark on the Ubuntu box:

You can see that arp-scan sent 802.1q tag 46 even though I hadn't configured a subinterface on the Ubuntu box. I have a blog showing how to configure Ubuntu to use vlan tags. Here is a link to the blog - Bypass VTY access lists with Linux and Yersinia

Now on the switch, look for the mac address. 

sh mac add | i 0090.9e9a.b53d   
  46    0090.9e9a.b53d    DYNAMIC     Gi1/0/48

You can either move the device to the correct port or reconfigure the port for the correct vlan. If the customer allows it, I like to label this type of device's switch port like this:

des < BACnet 10.112.100.1 0090.9e9a.b53d>

That gives me the MAC and IP for future troubleshooting. 

Let's set up a lab to learn how to use arp-scan.

It's a simple setup, but there are few gotcha's if you are new to Linux or arp-scan. You will need a Linux machine, physical or virtual, and a Kali Linux virtual machine. The Kali VM will simulate our OT device because it is designed to be quiet on the network. The kali motto is "The quieter you are, the more you can hear".

We will use the same network layout as above so you can refer to that diagram.

On the Kali VM

Configure Kali with a static ip address. In this example:

10.112.100.1/24
255.255.255.0
NO GATEWAY - If you assign a gateway the Kali box will send ARP requests to the gateway and populate the switch's mac address table.

Connect the Kali VM to switch port g1/0/48

Check to see if the Kali box sent traffic that caused the switch to record it's MAC address:

sh mac add int g1/0/48                                               
          Mac Address Table                                                    
-------------------------------------------                                    
                                                                               
Vlan    Mac Address       Type        Ports                                    
----    -----------       --------    -----                                    
  46    0090.9e9a.b53d    DYNAMIC     Gi1/0/48                                 
Total Mac Addresses for this criterion: 1               

Kali did send some traffic and that populated the mac address table. For our purposes, we need to make sure the Kali box isn't sending any traffic. Normally, you do not want your pen test box to send traffic you didn't explicitly request. That is why Kali has DHCP and other services disabled by default. 

In this case, I had been using DHCP before switching to static so the dhclient service was still running.

On Kali run

ps -ef | grep dh
root      2500  2125  0 13:18 pts/0    00:00:00 dhclient -v

If you see the dhclient service, then run
dhclient -r


to stop the service. If you don't stop it, the dhclient service will do a discover which will populate the MAC-address table on the switch and invalidates the test we are about to try.

Next, run the following to clear the mac address on the switch port. Sometimes you need to run it a few times before the address is cleared.

clear mac address-table dynamic interface g1/0/48

Finally, you should see an empty mac table for interface g1/0/48

sh mac add int g1/0/48                        
          Mac Address Table
-------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----

Try to ping the device

ping 10.112.100.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.112.100.1, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

Connect your laptop to port 1/0/47. I run the lldpd daemon (sudo apt install lldpd) so I can verify that the laptop is connected using the following.

sh lldp ne
Capability codes:
    (R) Router, (B) Bridge, (T) Telephone, (C) DOCSIS Cable Device
    (W) WLAN Access Point, (P) Repeater, (S) Station, (O) Other

Device ID           Local Intf     Hold-time  Capability      Port ID
PROCURVE J9450A     Gi1/1/4        120        B               2
1S1K-SYS76          Gi1/0/47       120        B,W,R           80fa.5b31.de85

Total entries displayed: 2

Look at the interface that the laptop is connected to:

sh run int g1/0/47

interface GigabitEthernet1/0/47
 switchport trunk encapsulation dot1q
 switchport mode trunk
end

Look at the interface on the laptop after it's connected to the switch

ip addr show enp2s0f1 

2: enp2s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 80:fa:5b:31:de:85 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::6d35:51a0:cef:4475/64 scope link noprefixroute
       valid_lft forever preferred_lft forever

You can see that the interface is UP/UP but doesn't have an IP address.

Look at the SVIs

sh run int vl 46

interface Vlan46
 ip address 10.112.47.254 255.255.254.0
 ip helper-address 192.168.10.221
 no ip redirects
end

sh run int vl 100

interface Vlan100
 ip address 10.112.100.254 255.255.255.0
 ip helper-address 192.168.10.221
 no ip redirects
end

Now we are ready to run arp-scan and find the device:

sudo arp-scan -I enp2s0f1 -Q 46 --destaddr=00:90:9e:9a:b5:3d 10.112.100.0/24

WARNING: Could not obtain IP address for interface enp2s0f1. Using 0.0.0.0 for
the source address, which is probably not what you want.
Either configure enp2s0f1 with an IP address, or manually specify the address
with the --arpspa option.
Interface: enp2s0f1, datalink type: EN10MB (Ethernet)
Starting arp-scan 1.9 with 256 hosts (http://www.nta-monitor.com/tools/arp-scan/)
10.112.100.1    00:90:9e:9a:b5:3d    Critical IO, LLC (802.1Q VLAN=46)

1 packets received by filter, 0 packets dropped by kernel
Ending arp-scan 1.9: 256 hosts scanned in 2.556 seconds (100.16 hosts/sec). 1 responded

Verify that the mac-address table on the switch has been updated

sh mac add | i 0090.9e9a.b53d   
  46    0090.9e9a.b53d    DYNAMIC     Gi1/0/48