Using iPerf3 to Test 2.5Gb/5Gb and 10Gb Links

I am a big fan of the iPerf3 tool written by ESnet, a part of the US Department of Energy. Here is a definition of iPerf from their official github page:

"iperf is a tool for active measurements of the maximum achievable bandwidth on IP networks.  For each test it reports the measured throughput / bitrate, loss, and other parameters."

I have previously blogged about iPerf and how to use it on Windows, Mac OSX, IOS, Android and Linux. You can find that blog here -  Using iPerf3 to verify Link Quality

Introduction

IPerf can be used to test\verify any IP based link. Here are examples of what I have tested using iperf3:

• Remote access VPNs - When a user complains that his home Internet connection is 60Mbps but using VPN back to the office is "slow" you can verify the connection with iPerf. A lot of business Internet connections are asymmetric, for example 60Mbps down and 5Mbps up. When the user connects to the office they are on the 5Mbps upload side, not the 60Mbps download side!
• Site to site VPNs - If you are experiencing a slow connection on a site to site VPN it could be the Internet connection at either site, the firewall at either site or the protocol being used to transfer data. With iPerf you can determine the root cause. 
• MPLS links - If you are having performance issues with an MPLS circuit the carrier will always say that their circuit is working correctly. An iPerf test will give you the data you need to push back. 
• Wireless access points - Anytime I deploy a new AP I set my laptop up in the MDF, connect wirelessly to the AP with a second laptop and verify the bandwidth. I have found problems with fibre connections, structured CAT cabling and even the carriers NID using iPerf.
• Data center to central office - Depending on the carrier you may be able to use iPerf between the data center and the central office. That was actually my introduction to iPerf years ago when I worked for a carrier services group.
• Virtual machine to virtual machine - Find bottlenecks in your virtual infrastructure.

 

Testing 10Gb site to site links

A lot of customers are moving services to the data center and eliminating servers at remote sites. This presents a problem when you are asked to test the site to site link. 

It's easy enough spin up a CentOS box on the virtual infrastructure at the data center as an end point but what to do at the remote site? I purchased an HP z420 workstation off lease on ebay for under $300. I also purchased an HP (Mellanox) 10Gb fiber card off ebay for under $30. 

I installed Ubuntu on the Z420 and Mellanox had the correct driver on thier website. I purchased a single port card but with hindsight I should have purchased a dual port card. That would allow me to test from virtual machine to virtual machine over the 10Gb link (not the Z420's backplane) without needing two 10G capable Z420s.

I recently got to test new 10Gb links at a customer with four remote sites. The customer had HPE switches and luckily he had a 3m HP DAC cable so connecting the Z420 to the switches was easy. On ebay you can purchase 3m DAC cables for under $50. It's best to have a DAC cable made by the switch manufacturer to avoid compatibility issues. You can also find 10Gb optics for under $50 on ebay.

The Z420 worked great and I was able to verify that each site was performing correctly. But it was 110°F (43°C) outside and carrying the Z420, monitor and keyboard to each site wasn't ideal. What to do?

Test MultiGig, NBASE-T and 10Gb with a Laptop?

Laptops have started shipping with Thunderbolt 3 connections. Thunderbolt has a 40Gbps interface to 10Gb is well within its capability. A quick Google search turned up the following Thunderbolt 3 to 10Gb adapters:

Sonnet Solo 10Gbase-T - This Thunderbolt 3 to 10Gb copper adapter also supports 2.5Gb/5Gb Ethernet so you can test the new MultiGig and NBASE-T switches. The webpage only shows Mac/Windows but the 10Gbe controller is an AQC-107S and there are Linux drivers for it. You have to build from source but there are detailed instructions in the readme. The cost is only $199 so it's within my budget!

Sonnet Twin 10G SFP+ - This Thunderbolt 3 to 10Gb adapter has two standard SFP+ ports. It uses the Intel 82599 controller so there are Linux/Mac/Windows drivers. The cost is $499 so it's outside the budget for my personal toolkit but is reasonable for a company.

Now, I just need to buy a new laptop with a Thunderbolt 3 port! The 17" System76 Oryx Pro is the model on my short list! It has Thunderbolt 3, nvidia 1060 (or 1070) and two m.2 NVME slots.

References

iperf3: A TCP, UDP, and SCTP network bandwidth measurement tool 
perfSonar -  A bandwidth testing suite of tools. Available in ISO format in four different toolkits. You can build a complete distributed link quality system with web based dashboard using perfSONAR.
perfSONAR Project YouTube Channel 
perfSONAR Powered - Podcast on the Research Computing and Engineering (RCE) podcast network

DNS Rebinding attacks

As we all know, DNS is used to translate Domain names into IP addresses. DNS uses UDP so it has had a long history of being abused by hackers for DoS. To make matters worse it doesn't have authentication or encryption so Man in the Middle (MiTM) attacks are possible.

Since DNS is used everytime you use the Internet it is hard to overstate the importance of a good DNS service. Companies like OpenDNS (Now Cisco Umbrella) and Quad 9 (www.quad9.net) have added security features like Malware detection and malicious site protection. These services are free for home use and paid for businesses.

Recently an old type of attack using DNS has become popular again - DNS Rebinding. Tripwire has a good explanation of what a DNS rebinding attack is - Practical Attacks with DNS Rebinding.

Armis.com gives this definition for DNS Rebinding. See the references for the link to Armis.com's DNS Rebinding Exposes Half a Billion Devices in the Enterprise. There is a link in the reference section to a youtube video on how it works.

**************************************************
DNS Rebinding Attacks Explained

DNS rebinding takes advantage of a nearly decade-old flaw in web browsers that allows a remote attacker to bypass a victim’s network firewall and use their web browser as a proxy to communicate directly with vulnerable devices on the local network. An example of a vulnerable device is one that is running an unauthenticated protocol like Universal Plug and Play (UPnP) or HTTP (used on unencrypted web servers). These protocols are commonly used to host administrative consoles (for routers, printers, IP cameras) or to allow easy access to the device’s services (for example, streaming video players), and are pervasive in businesses.
**************************************************

Preventing the Attack

There are several things you should do on your home network to prevent attacks:

• Change default credentials - A lot of script based attacks work because the default credentials weren't changed.
• Change the internal network IP address scheme - The scripts work by trying to log into common IP addresses used by network devices like 192.168.1.1 or 192.168.0.1 
• Disable uPnP - Universal Plug and Play can be abused by attackers. If you are a gamer there are plenty of sites that will explain how to port forward once you turn off uPnP.
• Update the firmware on you network devices - This is a MUST DO and is overlooked my most home users
• Install DD-WRT on your SOHO router - There is a link in the reference section below.
• Use OpenDNS 

Configuring OpenDNS to block rebinding attacks

I had been using Quad9 recently because it's fast, new and supports DNS over TLS along with DNSSec but decided to switch to OpenDNS because they offer rebinding filtering. Here is their explanation:

**************************************************
Block internal IP addresses

When enabled, DNS responses containing IP addresses listed in RFC1918 will be filtered out. This helps to prevent DNS Rebinding attacks. For example, if badstuff.attacker.com points to 192.168.1.1, this option would filter out that response.

The three blocks of IP addresses filtered in responses are:

10.0.0.0     - 10.255.255.255  (10/8)
172.16.0.0   - 172.31.255.255  (172.16/12)
192.168.0.0  - 192.168.255.255 (192.168/16)
**************************************************

To take advantage of this feature you need to create an OpenDNS account at https://login.opendns.com. Once you have an account, login and click on the Settings tab. At the bottom you will see a link "Keep your network's IP up-to-date with our free software." It says Mac and Windows but there is a Linux client also.

When you click the link it start the download. Once it finishes, run the program. It will ask you to log into OpenDNS. The updater will show the public IP address of your router.

Once you do that, go back to OpenDNS in the browser and click settings again. You should see the public IP address of your router listed under Add a network. Click Add This Network.

If you look at the updater now, you will see your public IP address listed. Back on the OpenDNS page click down arrow next to --Select a Network-- and select your network. On the dialog that opens, click on security and put a check in the box next to "Block Internal IP addresses"

Now, if a script tries to use an RFC1918 address to spoof a domain it will get filtered by OpenDNS. This isn't a silver bullet but just one more layer of defense.

Testing the Filter

Steve Gibson of Gibson Research wrote a DNS benchmark way back in 2010 that is free and works well for benchmarking DNS performance. He also created some DNS addresses for testing rebinding. There is a link to the original 2010 podcast (episode 260) and the July 24, 2018 update in the reference section.

To test if your DNS server filters RFC1918 addresses, open a terminal or cmd window and enter the following:

nslookup net4.rebindtest.com
nslookup net10.rebindtest.com
nslookup net127.rebindtest.com
nslookup net172.rebindtest.com
nslookup net192.rebindtest.com

Below is the output before I configured the OpenDNS filter. Notice that the address returned for net172.rebindtest.com is 172.16.0.1 which would allow a malicious script to bypass the Same Origin Policy of the browser.

nslookup net172.rebindtest.com
Server:  10.208.0.1
Address: 10.208.0.1#53

Non-authoritative answer:
Name: net172.rebindtest.com
Address: 172.16.0.1
Name: net172.rebindtest.com
Address: ::ffff:172.16.0.1

Below is the output after the filter was enabled. Notice that the address returned is 146.112.61.109.

nslookup net172.rebindtest.com
Server:  127.0.0.53
Address: 127.0.0.53#53

Non-authoritative answer:
Name: net172.rebindtest.com
Address: 146.112.61.109
Name: net172.rebindtest.com
Address: ::ffff:146.112.61.109

What is 146.112.61.109 you ask? You could use nslookup to find out but I wanted to show the dig (DNS Information Groper) command. It's built into Linux/Mac and you can install it on Windows. Here is a blog I wrote on installing dig DNS Information Groper for Windows

146.112.61.109 is the address OpenDNS uses for hit-block.opendns.com so the filter is working!

dig -x 146.112.61.109

; <<>> DiG 9.11.3-1ubuntu1.1-Ubuntu <<>> -x 146.112.61.109
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1651
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;109.61.112.146.in-addr.arpa.    IN    PTR

;; ANSWER SECTION:
109.61.112.146.in-addr.arpa. 3600 IN    PTR    hit-block.opendns.com.

;; Query time: 18 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Sat Aug 04 23:38:43 PDT 2018
;; MSG SIZE  rcvd: 91

References
Dear developers, beware of DNS Rebinding
How DNS Rebinding Attacks Impacts The Enterprise - youtube video
Half a billion smart devices vulnerable to decade-old DNS rebinding attacks 
DNS Rebinding Exposes Half a Billion Devices in the Enterprise
DNS Rebinding - Security Now podcast from 2010. Still applicable today.
Security Now 673 - Show notes
GRC DNS Benchmark - Windows only
DNS Information Groper for Windows - How to install dig on Windows
DD-WRT
Convert code samples into HTML for blogger