Using iPerf3 to Test 2.5Gb/5Gb and 10Gb Links

I am a big fan of the iPerf3 tool written by ESnet, a part of the US Department of Energy. Here is a definition of iPerf from their official github page:

"iperf is a tool for active measurements of the maximum achievable bandwidth on IP networks.  For each test it reports the measured throughput / bitrate, loss, and other parameters."

I have previously blogged about iPerf and how to use it on Windows, Mac OSX, IOS, Android and Linux. You can find that blog here -  Using iPerf3 to verify Link Quality

Introduction

IPerf can be used to test\verify any IP based link. Here are examples of what I have tested using iperf3:

• Remote access VPNs - When a user complains that his home Internet connection is 60Mbps but using VPN back to the office is "slow" you can verify the connection with iPerf. A lot of business Internet connections are asymmetric, for example 60Mbps down and 5Mbps up. When the user connects to the office they are on the 5Mbps upload side, not the 60Mbps download side!
• Site to site VPNs - If you are experiencing a slow connection on a site to site VPN it could be the Internet connection at either site, the firewall at either site or the protocol being used to transfer data. With iPerf you can determine the root cause. 
• MPLS links - If you are having performance issues with an MPLS circuit the carrier will always say that their circuit is working correctly. An iPerf test will give you the data you need to push back. 
• Wireless access points - Anytime I deploy a new AP I set my laptop up in the MDF, connect wirelessly to the AP with a second laptop and verify the bandwidth. I have found problems with fibre connections, structured CAT cabling and even the carriers NID using iPerf.
• Data center to central office - Depending on the carrier you may be able to use iPerf between the data center and the central office. That was actually my introduction to iPerf years ago when I worked for a carrier services group.
• Virtual machine to virtual machine - Find bottlenecks in your virtual infrastructure.

 

Testing 10Gb site to site links

A lot of customers are moving services to the data center and eliminating servers at remote sites. This presents a problem when you are asked to test the site to site link. 

It's easy enough spin up a CentOS box on the virtual infrastructure at the data center as an end point but what to do at the remote site? I purchased an HP z420 workstation off lease on ebay for under $300. I also purchased an HP (Mellanox) 10Gb fiber card off ebay for under $30. 

I installed Ubuntu on the Z420 and Mellanox had the correct driver on thier website. I purchased a single port card but with hindsight I should have purchased a dual port card. That would allow me to test from virtual machine to virtual machine over the 10Gb link (not the Z420's backplane) without needing two 10G capable Z420s.

I recently got to test new 10Gb links at a customer with four remote sites. The customer had HPE switches and luckily he had a 3m HP DAC cable so connecting the Z420 to the switches was easy. On ebay you can purchase 3m DAC cables for under $50. It's best to have a DAC cable made by the switch manufacturer to avoid compatibility issues. You can also find 10Gb optics for under $50 on ebay.

The Z420 worked great and I was able to verify that each site was performing correctly. But it was 110°F (43°C) outside and carrying the Z420, monitor and keyboard to each site wasn't ideal. What to do?

Test MultiGig, NBASE-T and 10Gb with a Laptop?

Laptops have started shipping with Thunderbolt 3 connections. Thunderbolt has a 40Gbps interface to 10Gb is well within its capability. A quick Google search turned up the following Thunderbolt 3 to 10Gb adapters:

Sonnet Solo 10Gbase-T - This Thunderbolt 3 to 10Gb copper adapter also supports 2.5Gb/5Gb Ethernet so you can test the new MultiGig and NBASE-T switches. The webpage only shows Mac/Windows but the 10Gbe controller is an AQC-107S and there are Linux drivers for it. You have to build from source but there are detailed instructions in the readme. The cost is only $199 so it's within my budget!

Sonnet Twin 10G SFP+ - This Thunderbolt 3 to 10Gb adapter has two standard SFP+ ports. It uses the Intel 82599 controller so there are Linux/Mac/Windows drivers. The cost is $499 so it's outside the budget for my personal toolkit but is reasonable for a company.

Now, I just need to buy a new laptop with a Thunderbolt 3 port! The 17" System76 Oryx Pro is the model on my short list! It has Thunderbolt 3, nvidia 1060 (or 1070) and two m.2 NVME slots.

References

iperf3: A TCP, UDP, and SCTP network bandwidth measurement tool 
perfSonar -  A bandwidth testing suite of tools. Available in ISO format in four different toolkits. You can build a complete distributed link quality system with web based dashboard using perfSONAR.
perfSONAR Project YouTube Channel 
perfSONAR Powered - Podcast on the Research Computing and Engineering (RCE) podcast network