Sunday, December 21, 2014

Using iPerf3 to verify Link Quality


  • Update August 11, 2018 - I wrote a new blog on iPerf testing 2.5/5Gb and 10Gb links. You can find it here
  • Update February 11, 2018 - More detail on the Hurricane Electric tools for IOS/Android
  • Update September 20, 2017 - Updated the VMware Player link to point to version 12.5.
  • Update April 8, 2017 - The https://iperf.fr/ site has a Windows version of iPerf3!  I ran it on Windows 7 and connected to iPerf3 on CentOS no problem. I have install instruction in the Install section below.
  • Update October 22, 2015 - ESNET has released iPerf3.1!!! The installation is the same. If you have already installed iPerf3 just use git clone https://github.com/esnet/iperf.git to upgrade.


From the iPerf3 site "iperf is a tool for active measurements of the maximum achievable bandwidth on IP networks."  It is maintained by the Department of Energy’s Dedicated Science network.

If you are responsible for a network, iPerf is a tool you should be familiar with. It can be used to test maximum bandwidth of any link - T1, Ethernet, VPN, etc. I use it to test switch ports by connecting the server and client to the same switch. It can also be used to test link quality using UDP.

When testing VPN connections you can use the -r switch to send data from the server. Useful when you have a typicall asymmetric connection at home I.E. 30Mbps down and 5Mbps up.

iPerf3 is developed on CentOS and Mac OSX but I have used it Kali Linux and Ubuntu. Instructions to install follow below. I have found iPerf3 on Linux/Mac to be very stable. There is an option to output in JSON format so it can be piped to a monitoring package.

There is a lot of good information on network troubleshooting on the DoE site Fasterdata.es.net. Here are some links to their site and a Public iPerf server.

References
iPerf3 Documentation
Disk Testing with iPerf3
iPerf3 Homepage
iPerf3 Documentation Update site
iPerf3 Dev List on Google Groups
Public iPerf3 Server
Autologin to Kali
Hey, Scripting Guy! Tell Me About PowerShell Community Extensions

If you are a Windows user you can still run iPerf using the free VMware Player application and Kali Linux. Kali is the benchmark in Penetration Testing Linux distro. You will find a lot of uses for Kali once you start using it. Download VMware player from the link below.
VMware Player Download

Once you have VMware Player installed download the Kali VMware image from
The Kali Download page

The image is compressed with 7Zip. Extract it and open it in VMware player.
Kali uses root / toor as the default credentials. Once logged into Kali open a terminal and use “passwd” to change the root password. Don’t run Kali with the default password! It would be embarrassing to get PWNED on your pentest box.

Installing iPerf on Kali Linux
Open terminal
1. git clone https://github.com/esnet/iperf.git
2. cd iperf
3. ./configure && make && make install
4. ldconfig (only needed is iPerf doesn’t start)
5. Execute iPerf3 as a server - /usr/local/bin/iperf3 -s

Installing on Ubuntu 16.04 LTS
Ubuntu 16.04 includes the latest iPerf build in the universe repository so install is  a snap.
1. sudo apt-get install iperf3
The following NEW packages will be installed:
  iperf3 libiperf0
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 58.5 kB of archives.
After this operation, 238 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://us.archive.ubuntu.com/ubuntu xenial/universe amd64 libiperf0 amd64 3.0.11-1 [50.4 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu xenial/universe amd64 iperf3 amd64 3.0.11-1 [8,090 B]
To run iPerf3 as a server:
Open a terminal and type iperf3 -s

Installing iPerf3 on Mac OSX 10.9
1. Install the Mac command line tools - Xcode
1. git clone https://github.com/esnet/iperf.git
2. cd iperf
3. ls /usr/local - If you get "No such file or directory" you will need to create the directory structure using "sudo mkdir /usr/local" and "sudo mkdir /usr/local/bin".
4. sudo chmod 777 /usr/local
5. sudo chmod 777 /usr/local/bin
6. ./configure && make && make install

Installing iPerf3 on Windows
1. Go to https://iperf.fr/iperf-download.php#windows and download the appropriate version. They provide hashes to verify the download. On windows run this:

***********************************************************

C:\temp>certutil -hashfile C:\temp\iperf-3.1.3-win64.zip SHA256
SHA256 hash of file C:\temp\iperf-3.1.3-win64.zip:
3c 3d b6 93 c1 bd cc 90 2c a9 19 8f c7 16 33 93 73 65 82 33 b3 39 2f fe 3d 46 7f 76 95 76 2c d1
CertUtil: -hashfile command completed successfully.

or with powershell 4 or above and the community extensions installed. See link in the references.
$PSVersionTable.PSVersion
PS C:\Users\mhubbard> get-hash C:\temp\iperf-3.1.3-win64\iperf-3.1.3-win64.zip -algorithm SHA256

Algorithm: SHA256

Path       : C:\Users\mhubbard\Downloads\iperf-3.1.3-win64\iperf-3.1.3-win64.zip
HashString : 3C3DB693C1BDCC902CA9198FC716339373658233B3392FFE3D467F7695762CD1


************************************************************

Compare that to the hash listed on the site:
3c3db693c1bdcc902ca9198fc716339373658233b3392ffe3d467f7695762cd1

Now unzip the files. You will have iperf3.exe and cygwin1.dll. Open a command line where you unpacked the files. You can then use iperf3 on Windows just like you do on Linux or MAC (minus the Linux/BSD only features)!

iPerf3 on IOS or Android
Hurricane Electric (https://networktools.he.net/) has an app that includes iPerf3 (and iPerf2) along with a lot of utilities including a MAC address browser. Just go to the App Store or Google Play store and search for he.net network tools.

The tool works great and it's useful haveing iPerf on you mobile phone or tablet. You can choose TCP or UDP tests and IPv4 or IPv6 addresses. The one odd thing about the tool is that you don't tell it how long you want to test for, you tell it how much data to send. This makes sense when you consider HE.NET is an ISP and you might be testing over a link that you are paying data rates for.

To use the iPerf test in the app:
Open the HE Netwrok Tools app
Select Iperf2 or Iperf3 at the top of the app
Enter an IP address in serach window
Select IPv4 or IPv6
Select TCp or UDP
In the "Bytes" field enter the number of bytes to send.

This will determine the length of the test and will require some trial and error. In the screeshot below I was testing from an iPhone 6s to an Aruba 225 AP. I picked 200M (200 Megabytes) and the test ran for about 4 second.





The Verbose switch
iPerf3 added a -V switch for the client. You don't need to run it very often but it will display:
iPerf3 version
The Linux version
The date/time the test was started
Maximum Segment Size (MSS) used
The CPU utilization on the client and the sever

*****************************************************************************
mhubbard@1S1K-SYS76:~/Dropbox/nmap-scripts$ iperf3 -c 192.168.10.161 -V
iperf 3.0.7
Linux 1S1K-SYS76 4.4.0-71-generic #92~14.04.1-Ubuntu SMP Fri Mar 24 15:22:50 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
Time: Sun, 09 Apr 2017 03:34:37 GMT
Connecting to host 192.168.10.161, port 5201
      Cookie: 1S1K-SYS76.1491708877.346884.34c4b94
      TCP MSS: 1448 (default)

(Testing)

CPU Utilization: local/sender 0.9% (0.1%u/0.8%s), remote/receiver 1.5% (0.2%u/1.4%s)

*****************************************************************************


Examples

Test for 5 seconds and use TCP - TCP is the default for iPerf
/usr/local/bin/iperf3 -c 192.168.10.142 -t 5

Don't include the first 2 seconds in the BW calculator to allow TCP slow start to finish
/usr/local/bin/iperf3 -c 192.168.10.142 -O 2

Label the test - This is useful when testing in several different rooms or to compare a 2.4Ghz connection to a 5Ghz connection
/usr/local/bin/iperf3 -c 192.168.10.142 -T 2.4GHz

Test using more than one stream 
Use 5 parallel streams - I have found this to be a good number to use
/usr/local/bin/iperf3 -c 192.168.10.142 -P 5 -T 2.4GHz

Reverse the test direction
This is useful when you are testing a VPN connection and have an asymmetric connection - 10Mbps download and 2Mbps Upload for example. You can run the test in each direction to verify.
/usr/local/bin/iperf3 -c 192.168.10.142 -P 5 -T VPNtoServer /usr/local/bin/iperf3 -c 192.168.10.142 -P 5 -R -T VPNfromServer

Test using UDP
/usr/local/bin/iperf3 -c 192.168.10.142 -u

Sample Output
Kali Linux running on VMware with a Linksys WUSB600N v1 Dual-Band Wireless-N Network Adapter [Ralink RT2870]

UDP test
Notice bandwidth is 1Mbps (Default for UDP) and the Jitter measurement. If you are having VoIP issues iPerf can verify the jitter on the link.
root@kali–32:/iperf/examples# /usr/local/bin/iperf3 -c 192.168.10.142 -u -T Wireless
Wireless: Connecting to host 192.168.10.142, port 5201
Wireless: [ 4] local 192.168.10.121 port 49089 connected to 192.168.10.142 port 5201
Wireless: [ ID] Interval      Transfer   Bandwidth        Total Datagrams
Wireless: [ 4] 0.00–1.00 sec  120 KBytes 983 Kbits/sec    15
Wireless: [ 4] 1.00–2.00 sec  128 KBytes 1.05 Mbits/sec   16
Wireless: [ 4] 2.00–3.00 sec  128 KBytes 1.05 Mbits/sec   16
Wireless: [ 4] 3.00–4.00 sec  128 KBytes 1.05 Mbits/sec   16
Wireless: [ 4] 4.00–5.00 sec  128 KBytes 1.05 Mbits/sec   16
Wireless: [ 4] 5.00–6.00 sec  128 KBytes 1.05 Mbits/sec   16
Wireless: [ 4] 6.00–7.00 sec  128 KBytes 1.05 Mbits/sec   16
Wireless: [ 4] 7.00–8.00 sec  128 KBytes 1.05 Mbits/sec   16
Wireless: [ 4] 8.00–9.00 sec  128 KBytes 1.05 Mbits/sec   16
Wireless: [ 4] 9.00–10.00 sec 128 KBytes 1.05 Mbits/sec  16
Wireless: [ ID] Interval      Transfer    Bandwidth      Jitter   Lost/Total Datagrams
Wireless: [ 4] 0.00–10.00 sec 1.24 MBytes 1.04 Mbits/sec 0.273 ms 0/159 (0%)
Wireless: [ 4] Sent 159 datagrams Wireless:
Wireless: iperf Done.
Kali Linux running on VMware with a Linksys WUSB600N v1 Dual-Band Wireless-N Network Adapter [Ralink RT2870]

TCP test
Notice the retries and Congestion Window (Cwnd) data. TCP was adjusting to the drops in the wireless network.
root@kali–32:/iperf/examples# /usr/local/bin/iperf3 -c 192.168.10.142 -P 5 -T Wireless
Wireless: Connecting to host 192.168.10.142, port 5201
Wireless: [ 4] local 192.168.10.121 port 44897 connected to 192.168.10.142 port 5201
Wireless: [ 6] local 192.168.10.121 port 44898 connected to 192.168.10.142 port 5201
Wireless: [ 8] local 192.168.10.121 port 44899 connected to 192.168.10.142 port 5201
Wireless: [ 10] local 192.168.10.121 port 44900 connected to 192.168.10.142 port 5201
Wireless: [ 12] local 192.168.10.121 port 44901 connected to 192.168.10.142 port 5201
Wireless: [ ID] Interval       Transfer   Bandwidth       Retr Cwnd
Wireless: [ 4] 0.00–1.00 sec  1.11 MBytes 9.29 Mbits/sec  20   26.9 KBytes
Wireless: [ 6] 0.00–1.00 sec  1.76 MBytes 14.7 Mbits/sec  43   33.9 KBytes
Wireless: [ 8] 0.00–1.00 sec  959 KBytes  7.85 Mbits/sec  13   24.0 KBytes
Wireless: [ 10] 0.00–1.00 sec 1.05 MBytes 8.78 Mbits/sec  1    31.1 KBytes
Wireless: [ 12] 0.00–1.00 sec 2.95 MBytes 24.7 Mbits/sec  101  109 KBytes
Wireless: [SUM] 0.00–1.00 sec 7.80 MBytes 65.4 Mbits/sec  178
Wireless: - - - - - - - - - - - - - - - - - - - - - - - - -
Wireless: [ 4] 1.00–2.00 sec  993 KBytes  8.13 Mbits/sec  6    1.41 KBytes
Wireless: [ 6] 1.00–2.00 sec  1.44 MBytes 12.0 Mbits/sec  47   1.41 KBytes
Wireless: [ 8] 1.00–2.00 sec  889 KBytes  7.28 Mbits/sec  24   1.41 KBytes
Wireless: [ 10] 1.00–2.00 sec 1.03 MBytes 8.67 Mbits/sec  6    1.41 KBytes
Wireless: [ 12] 1.00–2.00 sec 3.25 MBytes 27.3 Mbits/sec  100  1.41 KBytes
Wireless: [SUM] 1.00–2.00 sec 7.56 MBytes 63.4 Mbits/sec  183
Wireless: - - - - - - - - - - - - - - - - - - - - - - - - -
Wireless: [ 4] 2.00–3.00 sec  731 KBytes  5.99 Mbits/sec  45   18.4 KBytes
Wireless: [ 6] 2.00–3.00 sec  1.61 MBytes 13.5 Mbits/sec  25   29.7 KBytes
Wireless: [ 8] 2.00–3.00 sec  1.33 MBytes 11.2 Mbits/sec  58   60.8 KBytes
Wireless: [ 10] 2.00–3.00 sec 2.09 MBytes 17.6 Mbits/sec  31   89.1 KBytes
Wireless: [ 12] 2.00–3.00 sec 2.62 MBytes 22.0 Mbits/sec  92   82.0 KBytes
Wireless: [SUM] 2.00–3.00 sec 8.38 MBytes 70.3 Mbits/sec  251
Wireless: - - - - - - - - - - - - - - - - - - - - - - - - -
Wireless: [ 4] 3.00–4.00 sec  1.39 MBytes 11.7 Mbits/sec  3    31.1 KBytes
Wireless: [ 6] 3.00–4.00 sec  1.91 MBytes 16.0 Mbits/sec  7    38.2 KBytes
Wireless: [ 8] 3.00–4.00 sec  4.28 MBytes 35.9 Mbits/sec  72   48.1 KBytes
Wireless: [ 10] 3.00–4.00 sec 2.12 MBytes 17.8 Mbits/sec  61   32.5 KBytes
Wireless: [ 12] 3.00–4.00 sec 2.97 MBytes 24.9 Mbits/sec  22   48.1 KBytes
Wireless: [SUM] 3.00–4.00 sec 12.7 MBytes 106 Mbits/sec   165
Wireless: - - - - - - - - - - - - - - - - - - - - - - - - -
Wireless: [ 4] 4.00–5.00 sec  2.58 MBytes 21.6 Mbits/sec  25   24.0 KBytes
Wireless: [ 6] 4.00–5.00 sec  2.72 MBytes 22.8 Mbits/sec  42   22.6 KBytes
Wireless: [ 8] 4.00–5.00 sec  4.26 MBytes 35.7 Mbits/sec  89   74.9 KBytes
Wireless: [ 10] 4.00–5.00 sec 2.63 MBytes 22.0 Mbits/sec  23   29.7 KBytes
Wireless: [ 12] 4.00–5.00 sec 2.76 MBytes 23.2 Mbits/sec  20   62.2 KBytes
Wireless: [SUM] 4.00–5.00 sec 15.0 MBytes 125 Mbits/sec   199
Wireless: - - - - - - - - - - - - - - - - - - - - - - - - -
Wireless: [ 4] 5.00–6.00 sec  1.80 MBytes 15.1 Mbits/sec  11   32.5 KBytes
Wireless: [ 6] 5.00–6.00 sec  1.62 MBytes 13.6 Mbits/sec  11   32.5 KBytes
Wireless: [ 8] 5.00–6.00 sec  3.14 MBytes 26.3 Mbits/sec  4    50.9 KBytes
Wireless: [ 10] 5.00–6.00 sec 1.94 MBytes 16.3 Mbits/sec  9    36.8 KBytes
Wireless: [ 12] 5.00–6.00 sec 4.70 MBytes 39.5 Mbits/sec  67   67.9 KBytes
Wireless: [SUM] 5.00–6.00 sec 13.2 MBytes 111 Mbits/sec   102
Wireless: - - - - - - - - - - - - - - - - - - - - - - - - -
Wireless: [ 4] 6.00–7.00 sec  1.72 MBytes 14.5 Mbits/sec  23   25.5 KBytes
Wireless: [ 6] 6.00–7.00 sec  2.18 MBytes 18.3 Mbits/sec  1    33.9 KBytes
Wireless: [ 8] 6.00–7.00 sec  2.83 MBytes 23.7 Mbits/sec  5    39.6 KBytes
Wireless: [ 10] 6.00–7.00 sec 2.86 MBytes 24.0 Mbits/sec  56   56.6 KBytes
Wireless: [ 12] 6.00–7.00 sec 4.04 MBytes 33.9 Mbits/sec  83   9.5 KBytes
Wireless: [SUM] 6.00–7.00 sec 13.6 MBytes 114 Mbits/sec   168
Wireless: - - - - - - - - - - - - - - - - - - - - - - - - -
Wireless: [ 4] 7.00–8.00 sec  1.40 MBytes 11.8 Mbits/sec  34   31.1 KBytes
Wireless: [ 6] 7.00–8.00 sec  2.05 MBytes 17.2 Mbits/sec  27   38.2 KBytes
Wireless: [ 8] 7.00–8.00 sec  2.04 MBytes 17.1 Mbits/sec  31   38.2 KBytes
Wireless: [ 10] 7.00–8.00 sec 2.93 MBytes 24.6 Mbits/sec  44   42.4 KBytes
Wireless: [ 12] 7.00–8.00 sec 3.23 MBytes 27.1 Mbits/sec  196  77.8 KBytes
Wireless: [SUM] 7.00–8.00 sec 11.7 MBytes 97.8 Mbits/sec  332
Wireless: - - - - - - - - - - - - - - - - - - - - - - - - -
Wireless: [ 4] 8.00–9.00 sec  1.78 MBytes 14.9 Mbits/sec  43   18.4 KBytes
Wireless: [ 6] 8.00–9.00 sec  2.64 MBytes 22.1 Mbits/sec  53   38.2 KBytes
Wireless: [ 8] 8.00–9.00 sec  2.45 MBytes 20.5 Mbits/sec  2    42.4 KBytes
Wireless: [ 10] 8.00–9.00 sec 2.20 MBytes 18.4 Mbits/sec  23   19.8 KBytes
Wireless: [ 12] 8.00–9.00 sec 4.64 MBytes 38.9 Mbits/sec  238  105 KBytes
Wireless: [SUM] 8.00–9.00 sec 13.7 MBytes 115 Mbits/sec   359
Wireless: - - - - - - - - - - - - - - - - - - - - - - - - -
Wireless: [ 4] 9.00–10.00 sec  2.34 MBytes 19.6 Mbits/sec 14  43.8 KBytes
Wireless: [ 6] 9.00–10.00 sec  1.87 MBytes 15.7 Mbits/sec 17  35.4 KBytes
Wireless: [ 8] 9.00–10.00 sec  1.43 MBytes 12.0 Mbits/sec 33  21.2 KBytes
Wireless: [ 10] 9.00–10.00 sec 979 KBytes 8.02 Mbits/sec  12  25.5 KBytes
Wireless: [ 12] 9.00–10.00 sec 5.57 MBytes 46.7 Mbits/sec 187 93.3 KBytes
Wireless: [SUM] 9.00–10.00 sec 12.2 MBytes 102 Mbits/sec  263
Wireless: - - - - - - - - - - - - - - - - - - - - - - - - -
Wireless: [ ID] Interval Transfer Bandwidth Retr
Wireless: [ 4] 0.00–10.00 sec 15.8 MBytes 13.3 Mbits/sec  224 sender
Wireless: [ 4] 0.00–10.00 sec 15.7 MBytes 13.2 Mbits/sec      receiver
Wireless: [ 6] 0.00–10.00 sec 19.8 MBytes 16.6 Mbits/sec  273 sender
Wireless: [ 6] 0.00–10.00 sec 19.7 MBytes 16.5 Mbits/sec      receiver
Wireless: [ 8] 0.00–10.00 sec 23.6 MBytes 19.8 Mbits/sec  331 sender
Wireless: [ 8] 0.00–10.00 sec 23.4 MBytes 19.6 Mbits/sec      receiver
Wireless: [ 10] 0.00–10.00 sec 19.8 MBytes 16.6 Mbits/sec 266 sender
Wireless: [ 10] 0.00–10.00 sec 19.7 MBytes 16.5 Mbits/sec     receiver
Wireless: [ 12] 0.00–10.00 sec 36.7 MBytes 30.8 Mbits/sec 1106 sender
Wireless: [ 12] 0.00–10.00 sec 36.5 MBytes 30.6 Mbits/sec     receiver
Wireless: [SUM] 0.00–10.00 sec 116 MBytes 97.1 Mbits/sec  2200 sender
Wireless: [SUM] 0.00–10.00 sec 115 MBytes 96.5 Mbits/sec       receiver
Wireless: 
Wireless: iperf Done.

Kali Linux running on VMware with a wired Ethernet connection
Notice that there are no retries.
root@kali–32:/iperf/examples# /usr/local/bin/iperf3 -c 192.168.10.129 -P 5 -t 5 -T Wired
Wired: Connecting to host 192.168.10.129, port 5201
Wired: [ 4] local 192.168.10.121 port 58734 connected to 192.168.10.129 port 5201
Wired: [ 6] local 192.168.10.121 port 58735 connected to 192.168.10.129 port 5201
Wired: [ 8] local 192.168.10.121 port 58736 connected to 192.168.10.129 port 5201
Wired: [ 10] local 192.168.10.121 port 58737 connected to 192.168.10.129 port 5201
Wired: [ 12] local 192.168.10.121 port 58738 connected to 192.168.10.129 port 5201
Wired: [ ID] Interval     Transfer      Bandwidth        Retr Cwnd
Wired: [ 4] 0.00–1.24 sec  3.75 MBytes  25.3 Mbits/sec   0    106 KBytes
Wired: [ 6] 0.00–1.24 sec  3.75 MBytes  25.3 Mbits/sec   0    106 KBytes
Wired: [ 8] 0.00–1.24 sec  3.75 MBytes  25.3 Mbits/sec   0    106 KBytes
Wired: [ 10] 0.00–1.24 sec 3.75 MBytes  25.3 Mbits/sec   0    105 KBytes
Wired: [ 12] 0.00–1.24 sec 3.75 MBytes  25.3 Mbits/sec   0    107 KBytes
Wired: [SUM] 0.00–1.24 sec 18.8 MBytes  127 Mbits/sec    0
Wired: - - - - - - - - - - - - - - - - - - - - - - - - -
Wired: [ 4] 1.24–2.17 sec  2.50 MBytes  22.5 Mbits/sec   0    130 KBytes
Wired: [ 6] 1.24–2.17 sec  2.50 MBytes  22.5 Mbits/sec   0    130 KBytes
Wired: [ 8] 1.24–2.17 sec  2.50 MBytes  22.5 Mbits/sec   0    130 KBytes
Wired: [ 10] 1.24–2.17 sec 2.50 MBytes  22.5 Mbits/sec   0    129 KBytes
Wired: [ 12] 1.24–2.17 sec 2.50 MBytes  22.5 Mbits/sec   0    132 KBytes
Wired: [SUM] 1.24–2.17 sec 12.5 MBytes  113 Mbits/sec    0
Wired: - - - - - - - - - - - - - - - - - - - - - - - - -
Wired: [ 4] 2.17–3.13 sec  2.50 MBytes  22.0 Mbits/sec   0    134 KBytes
Wired: [ 6] 2.17–3.13 sec  2.50 MBytes  22.0 Mbits/sec   0    134 KBytes
Wired: [ 8] 2.17–3.13 sec  2.50 MBytes  22.0 Mbits/sec   0    134 KBytes
Wired: [ 10] 2.17–3.13 sec 2.50 MBytes  22.0 Mbits/sec   0    134 KBytes
Wired: [ 12] 2.17–3.13 sec 2.50 MBytes  22.0 Mbits/sec   0    134 KBytes
Wired: [SUM] 2.17–3.13 sec 12.5 MBytes  110 Mbits/sec    0
Wired: - - - - - - - - - - - - - - - - - - - - - - - - -
Wired: [ 4] 3.13–4.13 sec  2.50 MBytes  20.9 Mbits/sec   0    134 KBytes
Wired: [ 6] 3.13–4.13 sec  2.50 MBytes  20.9 Mbits/sec   0    134 KBytes
Wired: [ 8] 3.13–4.13 sec  2.50 MBytes  20.9 Mbits/sec   0    134 KBytes
Wired: [ 10] 3.13–4.13 sec 2.50 MBytes  20.9 Mbits/sec   0    134 KBytes
Wired: [ 12] 3.13–4.13 sec 2.50 MBytes  20.9 Mbits/sec   0    134 KBytes
Wired: [SUM] 3.13–4.13 sec 12.5 MBytes  105 Mbits/sec    0
Wired: - - - - - - - - - - - - - - - - - - - - - - - - -
Wired: [ 4] 4.13–5.12 sec  2.50 MBytes  21.2 Mbits/sec   0    134 KBytes
Wired: [ 6] 4.13–5.12 sec  2.50 MBytes  21.2 Mbits/sec   0    134 KBytes
Wired: [ 8] 4.13–5.12 sec  2.50 MBytes  21.2 Mbits/sec   0    134 KBytes
Wired: [ 10] 4.13–5.12 sec 2.50 MBytes  21.2 Mbits/sec   0    134 KBytes
Wired: [ 12] 4.13–5.12 sec 2.50 MBytes  21.2 Mbits/sec   0    134 KBytes
Wired: [SUM] 4.13–5.12 sec 12.5 MBytes  106 Mbits/sec    0
Wired: - - - - - - - - - - - - - - - - - - - - - - - - -
Wired: [ ID]  Interval     Transfer    Bandwidth         Retr
Wired: [ 4] 0.00–5.12 sec  13.8 MBytes 22.5 Mbits/sec    0 sender
Wired: [ 4] 0.00–5.12 sec  13.8 MBytes 22.5 Mbits/sec     receiver
Wired: [ 6] 0.00–5.12 sec  13.8 MBytes 22.5 Mbits/sec    0 sender
Wired: [ 6] 0.00–5.12 sec  13.8 MBytes 22.5 Mbits/sec     receiver
Wired: [ 8] 0.00–5.12 sec  13.8 MBytes 22.5 Mbits/sec    0 sender
Wired: [ 8] 0.00–5.12 sec  13.8 MBytes 22.5 Mbits/sec     receiver
Wired: [ 10] 0.00–5.12 sec 13.8 MBytes 22.5 Mbits/sec    0 sender
Wired: [ 10] 0.00–5.12 sec 13.8 MBytes 22.5 Mbits/sec     receiver
Wired: [ 12] 0.00–5.12 sec 13.8 MBytes 22.5 Mbits/sec    0 sender
Wired: [ 12] 0.00–5.12 sec 13.8 MBytes 22.5 Mbits/sec     receiver
Wired: [SUM] 0.00–5.12 sec 68.8 MBytes 113 Mbits/sec     0 sender
Wired: [SUM] 0.00–5.12 sec 68.8 MBytes 113 Mbits/sec      receiver
Wired:
Wired: iperf Done.

An Example using iPerf to Verify a Switching Problem
A user reported that the wireless network was very slow. I connected to the wireless network and ran iPerf. At first throughput looked good but within a few seconds it dropped to 0Bps for several seconds and then went back up. This repeated for as long as I ran iPerf. But the connection to the AP didn’t drop and there were no unusual log entries on the AP.

I connected to Ethernet, reran iPerf and observed the same behavior only at Ethernet speeds. Obviously it wasn’t a Wireless issue. Running “sh proc cpu sorted” on the switch revealed cpu up around 95%. Obviously something was wrong on the switch but that’s a blog for another day. Bottom line is that 5 minutes with iPerf revealed that the problem wasn’t the wireless network and I moved on to find the root cause was a bug in the switch firmware.

1S1K–873:iperf mhubbard$ /usr/local/bin/iperf3 -c 10.140.44.149 -t 1000
Connecting to host 10.140.44.149, port 5201
[ 4] local 10.141.1.217 port 57242 connected to 10.140.44.149 port 5201
[ ID]  Interval      Transfer    Bandwidth
[ 4] 0.00–1.00 sec   16.1 MBytes 135 Mbits/sec
[ 4] 1.00–2.00 sec   14.4 MBytes 121 Mbits/sec
[ 4] 2.00–3.00 sec   15.8 MBytes 132 Mbits/sec
[ 4] 3.00–4.00 sec   17.5 MBytes 147 Mbits/sec
[ 4] 4.00–5.00 sec   15.8 MBytes 132 Mbits/sec
[ 4] 5.00–6.00 sec   13.8 MBytes 116 Mbits/sec
[ 4] 6.00–7.00 sec   13.9 MBytes 116 Mbits/sec
[ 4] 7.00–8.00 sec   15.4 MBytes 129 Mbits/sec
[ 4] 8.00–9.00 sec   16.3 MBytes 136 Mbits/sec
[ 4] 9.00–10.00 sec  14.1 MBytes 118 Mbits/sec
[ 4] 10.00–11.00 sec 0.00 Bytes  0.00 bits/sec
[ 4] 11.00–12.00 sec 0.00 Bytes  0.00 bits/sec
[ 4] 12.00–13.00 sec 0.00 Bytes  0.00 bits/sec
[ 4] 13.00–14.00 sec 0.00 Bytes  0.00 bits/sec
[ 4] 14.00–15.00 sec 0.00 Bytes  0.00 bits/sec
[ 4] 15.00–16.00 sec 0.00 Bytes  0.00 bits/sec
[ 4] 16.00–17.00 sec 0.00 Bytes  0.00 bits/sec
[ 4] 17.00–18.00 sec 0.00 Bytes  0.00 bits/sec
[ 4] 18.00–19.00 sec 0.00 Bytes  0.00 bits/sec
[ 4] 19.00–20.00 sec 0.00 Bytes  0.00 bits/sec
[ 4] 20.00–21.00 sec 6.60 MBytes 55.3 Mbits/sec
[ 4] 21.00–22.00 sec 13.6 MBytes 114 Mbits/sec
[ 4] 22.00–23.00 sec 14.3 MBytes 120 Mbits/sec
[ 4] 23.00–24.00 sec 14.6 MBytes 122 Mbits/sec
[ 4] 24.00–25.00 sec 16.4 MBytes 137 Mbits/sec
[ 4] 25.00–26.00 sec 16.5 MBytes 138 Mbits/sec
[ 4] 26.00–27.00 sec 14.4 MBytes 121 Mbits/sec
[ 4] 27.00–28.00 sec 16.9 MBytes 142 Mbits/sec
[ 4] 28.00–29.00 sec 16.0 MBytes 135 Mbits/sec
[ 4] 29.00–30.00 sec 16.0 MBytes 134 Mbits/sec
[ 4] 30.00–31.00 sec 16.7 MBytes 140 Mbits/sec
[ 4] 31.00–32.00 sec 16.8 MBytes 141 Mbits/sec
[ 4] 32.00–33.00 sec 15.8 MBytes 132 Mbits/sec
[ 4] 33.00–34.00 sec 14.6 MBytes 122 Mbits/sec
[ 4] 34.00–35.00 sec 15.0 MBytes 126 Mbits/sec
[ 4] 35.00–36.00 sec 14.5 MBytes 121 Mbits/sec
[ 4] 36.00–37.00 sec 16.4 MBytes 138 Mbits/sec
[ 4] 37.00–38.00 sec 16.5 MBytes 138 Mbits/sec
[ 4] 38.00–39.00 sec 15.7 MBytes 132 Mbits/sec
[ 4] 39.00–40.00 sec 16.9 MBytes 142 Mbits/sec
[ 4] 40.00–41.00 sec 14.7 MBytes 123 Mbits/sec
[ 4] 41.00–42.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 42.00–43.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 43.00–44.00 sec 0.00 Bytes 0.00 bits/sec

Running iPerf for an extended period
Recently a coworker needed to run iPerf over and over while logging the output with a time stamp. I wrote a simple bash shell script to do it. It uses a while do loop to call iPerf. A simple counter is used to control how many times to loop.

I added some formatting to make the report look good and a time stamp to show when the tests were run.

#!/bin/bash
#Start iperf with a time stamp and redirect to a file.
# Define a timestamp function
timestamp() {
  date +"%T"
}
c="5"
t="1"
echo "*************************************************************************" >> iperf.log
echo " " >> iperf.log
while [ $c -gt 0 ]
do
echo "$(timestamp): iPerf Test $t"  >> iperf.log
echo " " >> iperf.log
echo "*************************************************************************" >> iperf.log
iperf3 -c 192.168.10.161 -P 4 -t 5 -T iPerf-test >> iperf.log
echo " " >> iperf.log
echo "*************************************************************************" >> iperf.log
echo " " >> iperf.log
echo "iPerf Test $t complete"
sleep 1
c=$[$c-1]
t=$[$t+1]
done

To run it save the above to iperfloop.sh. Then make it executable using:
chmod +x iperfloop.sh

then ./iperfloop.sh to run it.

Here is what the output looks like:
mhubbard@1S1K-SYS76:~/michael.hubbard999@gmail.com/02_ceh/bashbunny$ ./iperfloop.sh
iPerf Test 1 complete
iPerf Test 2 complete
iPerf Test 3 complete
iPerf Test 4 complete
iPerf Test 5 complete

The script creates iperf.log in the folder where the script was executed. Here is what the log looks like:
*************************************************************************

22:08:48: iPerf Test 1

*************************************************************************
iPerf-test:  Connecting to host 192.168.10.161, port 5201
iPerf-test:  [  4] local 192.168.10.152 port 59086 connected to 192.168.10.161 port 5201
iPerf-test:  [  6] local 192.168.10.152 port 59088 connected to 192.168.10.161 port 5201
iPerf-test:  [  8] local 192.168.10.152 port 59090 connected to 192.168.10.161 port 5201
iPerf-test:  [ 10] local 192.168.10.152 port 59092 connected to 192.168.10.161 port 5201
iPerf-test:  [ ID] Interval           Transfer     Bandwidth       Retr  Cwnd


The second request was to monitor the iperf server and make sure if it failed that it restarted. I found a post here that showed how to do it.

Here is the script listing. I ran this on my CentOS server.

#!/bin/bash
myserver="iperf3 -s"
until myserver; do
    echo "Server 'myserver' crashed with exit code $?.  Respawning.." >&2
    sleep 1
done

Same process as above - Save it, chmod +x and run it. It worked perfectly. I never had iPerf3 on CentOS die but I used ps ef | grep iperf to find the PID and then used kill "PID" to kill the process. The script started it back each time and the loop on the client just kept going.


Use SSH to run iPerf
I carry a Mac Book Air and can use it as the iPerf3 client as I move around a site. But iPerf3 on the Mac doesn't show the Cwnd or Retries. To get around this I installed SSH on Kali and then I SSH into the Linux box and use the Mac as the iPerf server. Here is a link to the Blackmore site with instructions to install SSH www.blackmoreops.com

Having Kali running an SSH server also allows me to access the Kali terminal from my iPhone and iPad. This is very useful especially if I am offsite and don't have LAN connectivity. I can connect over VPN and then shell into Kali.

There are many SSH clients for IOS but my favorite is  get console. It has full Dropbox integration, scripting and supports Serial, SSH and Telnet. How does it do Serial on an IOS device you ask? They sell a 30 Pin or Lightning serial cable but they also have a device called AirConsole.

Airconsole is a great device with WiFi, USB and RJ45 ports. It has a DHCP server built in so you can console into a device while providing DHCP over Ethernet to run TFTP for firmware upgrades. I use it a lot for setting up HP's iLo by connecting Ethernet to the server's ilo port, iLo gets a DHCP address, then I connect to the Airconsole's WiFi network, open a browser and configure iLo.

Sunday, December 14, 2014

Preventing a User from Causing a Loop with an Unmanaged Switch

I bet you have been in this situation - A user brings in a Linksys or Netgear unmanaged switch and plug it in. Then they connect 5 devices and think they are network engineers.

It’s no problem, at least as far as loops go, until they unplug a couple devices and leave the cables laying there. Then someone else comes along and plugs the cable back into the switch. Now you have a loop and everything grinds to a halt while you start trouble shooting.

A better way
Use the interface command “spanning-tree bpduguard enable” to shut the port down as soon as it sees the BPDU it sent returned through the unmanaged switch.
You can enter “show interfaces status err” to see any ports that are in the Error-Disabled state.

Automatic recovery
You can enable automatic recovery from spanning-tree bpduguard enable using the global command “errdisable recovery cause bpduguard”. With this configured a timer will start as soon as the port is disabled. The default for the timer is 300 seconds. You can override this using “errdisable recovery interval xxx” to change the timer to a new value, for example “errdisable recovery interval 500” to make it 500 seconds.

A Cautionary Note
One thing to think about before you enable automatic recovery for BPDUguard is whether the user will notice the problem and remove the cable.

If they do the port will be re-enabled when the timer expires. The switch will now be working again and no call to the helpdesk is needed.

But what happens if the user just leaves the cable attached and leaves? The counter will expire, the port will be enabled and the loop will start again. This will repeat every “timer interval” until the cable is removed. It will create havoc on a schedule.

Here are messages from a switch with a looped unmanaged switch and auto recovery enabled:
* 000044: Dec 14 05:08:56.632 PST: %SPANTREE–2-BLOCK_BPDUGUARD: Received BPDU on port Gi1/0/48 with BPDU Guard enabled. Disabling port.
* 000049: Dec 14 05:09:26.628 PST: %PM–4-ERR_RECOVER: Attempting to recover from bpduguard err-disable state on Gi1/0/48
* 000050: *Dec 14 05:09:29.888 PST: %SPANTREE–2-BLOCK_BPDUGUARD: Received BPDU on port Gi1/0/48 with BPDU Guard enabled. Disabling port.

My recommendation is NOT to use auto-recovery for BPDUguard.

Related commands
* show errdisable recovery - Lists all ErrDisable reasons and their state.
* show interfaces status err - Lists any interfaces that are in the Error Disabled state.

Reference
Cisco 3850 switch manual