Sunday, December 21, 2014

Using iPerf3 to verify Link Quality


  • Update August 11, 2018 - I wrote a new blog on iPerf testing 2.5/5Gb and 10Gb links. You can find it here
  • Update February 11, 2018 - More detail on the Hurricane Electric tools for IOS/Android
  • Update September 20, 2017 - Updated the VMware Player link to point to version 12.5.
  • Update April 8, 2017 - The https://iperf.fr/ site has a Windows version of iPerf3!  I ran it on Windows 7 and connected to iPerf3 on CentOS no problem. I have install instruction in the Install section below.
  • Update October 22, 2015 - ESNET has released iPerf3.1!!! The installation is the same. If you have already installed iPerf3 just use git clone https://github.com/esnet/iperf.git to upgrade.


From the iPerf3 site "iperf is a tool for active measurements of the maximum achievable bandwidth on IP networks."  It is maintained by the Department of Energy’s Dedicated Science network.

If you are responsible for a network, iPerf is a tool you should be familiar with. It can be used to test maximum bandwidth of any link - T1, Ethernet, VPN, etc. I use it to test switch ports by connecting the server and client to the same switch. It can also be used to test link quality using UDP.

When testing VPN connections you can use the -r switch to send data from the server. Useful when you have a typicall asymmetric connection at home I.E. 30Mbps down and 5Mbps up.

iPerf3 is developed on CentOS and Mac OSX but I have used it Kali Linux and Ubuntu. Instructions to install follow below. I have found iPerf3 on Linux/Mac to be very stable. There is an option to output in JSON format so it can be piped to a monitoring package.

There is a lot of good information on network troubleshooting on the DoE site Fasterdata.es.net. Here are some links to their site and a Public iPerf server.

References
iPerf3 Documentation
Disk Testing with iPerf3
iPerf3 Homepage
iPerf3 Documentation Update site
iPerf3 Dev List on Google Groups
Public iPerf3 Server
Autologin to Kali
Hey, Scripting Guy! Tell Me About PowerShell Community Extensions

If you are a Windows user you can still run iPerf using the free VMware Player application and Kali Linux. Kali is the benchmark in Penetration Testing Linux distro. You will find a lot of uses for Kali once you start using it. Download VMware player from the link below.
VMware Player Download

Once you have VMware Player installed download the Kali VMware image from
The Kali Download page

The image is compressed with 7Zip. Extract it and open it in VMware player.
Kali uses root / toor as the default credentials. Once logged into Kali open a terminal and use “passwd” to change the root password. Don’t run Kali with the default password! It would be embarrassing to get PWNED on your pentest box.

Installing iPerf on Kali Linux
Open terminal
1. git clone https://github.com/esnet/iperf.git
2. cd iperf
3. ./configure && make && make install
4. ldconfig (only needed is iPerf doesn’t start)
5. Execute iPerf3 as a server - /usr/local/bin/iperf3 -s

Installing on Ubuntu 16.04 LTS
Ubuntu 16.04 includes the latest iPerf build in the universe repository so install is  a snap.
1. sudo apt-get install iperf3
The following NEW packages will be installed:
  iperf3 libiperf0
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 58.5 kB of archives.
After this operation, 238 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://us.archive.ubuntu.com/ubuntu xenial/universe amd64 libiperf0 amd64 3.0.11-1 [50.4 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu xenial/universe amd64 iperf3 amd64 3.0.11-1 [8,090 B]
To run iPerf3 as a server:
Open a terminal and type iperf3 -s

Installing iPerf3 on Mac OSX 10.9
1. Install the Mac command line tools - Xcode
1. git clone https://github.com/esnet/iperf.git
2. cd iperf
3. ls /usr/local - If you get "No such file or directory" you will need to create the directory structure using "sudo mkdir /usr/local" and "sudo mkdir /usr/local/bin".
4. sudo chmod 777 /usr/local
5. sudo chmod 777 /usr/local/bin
6. ./configure && make && make install

Installing iPerf3 on Windows
1. Go to https://iperf.fr/iperf-download.php#windows and download the appropriate version. They provide hashes to verify the download. On windows run this:

***********************************************************

C:\temp>certutil -hashfile C:\temp\iperf-3.1.3-win64.zip SHA256
SHA256 hash of file C:\temp\iperf-3.1.3-win64.zip:
3c 3d b6 93 c1 bd cc 90 2c a9 19 8f c7 16 33 93 73 65 82 33 b3 39 2f fe 3d 46 7f 76 95 76 2c d1
CertUtil: -hashfile command completed successfully.

or with powershell 4 or above and the community extensions installed. See link in the references.
$PSVersionTable.PSVersion
PS C:\Users\mhubbard> get-hash C:\temp\iperf-3.1.3-win64\iperf-3.1.3-win64.zip -algorithm SHA256

Algorithm: SHA256

Path       : C:\Users\mhubbard\Downloads\iperf-3.1.3-win64\iperf-3.1.3-win64.zip
HashString : 3C3DB693C1BDCC902CA9198FC716339373658233B3392FFE3D467F7695762CD1


************************************************************

Compare that to the hash listed on the site:
3c3db693c1bdcc902ca9198fc716339373658233b3392ffe3d467f7695762cd1

Now unzip the files. You will have iperf3.exe and cygwin1.dll. Open a command line where you unpacked the files. You can then use iperf3 on Windows just like you do on Linux or MAC (minus the Linux/BSD only features)!

iPerf3 on IOS or Android
Hurricane Electric (https://networktools.he.net/) has an app that includes iPerf3 (and iPerf2) along with a lot of utilities including a MAC address browser. Just go to the App Store or Google Play store and search for he.net network tools.

The tool works great and it's useful haveing iPerf on you mobile phone or tablet. You can choose TCP or UDP tests and IPv4 or IPv6 addresses. The one odd thing about the tool is that you don't tell it how long you want to test for, you tell it how much data to send. This makes sense when you consider HE.NET is an ISP and you might be testing over a link that you are paying data rates for.

To use the iPerf test in the app:
Open the HE Netwrok Tools app
Select Iperf2 or Iperf3 at the top of the app
Enter an IP address in serach window
Select IPv4 or IPv6
Select TCp or UDP
In the "Bytes" field enter the number of bytes to send.

This will determine the length of the test and will require some trial and error. In the screeshot below I was testing from an iPhone 6s to an Aruba 225 AP. I picked 200M (200 Megabytes) and the test ran for about 4 second.





The Verbose switch
iPerf3 added a -V switch for the client. You don't need to run it very often but it will display:
iPerf3 version
The Linux version
The date/time the test was started
Maximum Segment Size (MSS) used
The CPU utilization on the client and the sever

*****************************************************************************
mhubbard@1S1K-SYS76:~/Dropbox/nmap-scripts$ iperf3 -c 192.168.10.161 -V
iperf 3.0.7
Linux 1S1K-SYS76 4.4.0-71-generic #92~14.04.1-Ubuntu SMP Fri Mar 24 15:22:50 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
Time: Sun, 09 Apr 2017 03:34:37 GMT
Connecting to host 192.168.10.161, port 5201
      Cookie: 1S1K-SYS76.1491708877.346884.34c4b94
      TCP MSS: 1448 (default)

(Testing)

CPU Utilization: local/sender 0.9% (0.1%u/0.8%s), remote/receiver 1.5% (0.2%u/1.4%s)

*****************************************************************************


Examples

Test for 5 seconds and use TCP - TCP is the default for iPerf
/usr/local/bin/iperf3 -c 192.168.10.142 -t 5

Don't include the first 2 seconds in the BW calculator to allow TCP slow start to finish
/usr/local/bin/iperf3 -c 192.168.10.142 -O 2

Label the test - This is useful when testing in several different rooms or to compare a 2.4Ghz connection to a 5Ghz connection
/usr/local/bin/iperf3 -c 192.168.10.142 -T 2.4GHz

Test using more than one stream 
Use 5 parallel streams - I have found this to be a good number to use
/usr/local/bin/iperf3 -c 192.168.10.142 -P 5 -T 2.4GHz

Reverse the test direction
This is useful when you are testing a VPN connection and have an asymmetric connection - 10Mbps download and 2Mbps Upload for example. You can run the test in each direction to verify.
/usr/local/bin/iperf3 -c 192.168.10.142 -P 5 -T VPNtoServer /usr/local/bin/iperf3 -c 192.168.10.142 -P 5 -R -T VPNfromServer

Test using UDP
/usr/local/bin/iperf3 -c 192.168.10.142 -u

Sample Output
Kali Linux running on VMware with a Linksys WUSB600N v1 Dual-Band Wireless-N Network Adapter [Ralink RT2870]

UDP test
Notice bandwidth is 1Mbps (Default for UDP) and the Jitter measurement. If you are having VoIP issues iPerf can verify the jitter on the link.
root@kali–32:/iperf/examples# /usr/local/bin/iperf3 -c 192.168.10.142 -u -T Wireless
Wireless: Connecting to host 192.168.10.142, port 5201
Wireless: [ 4] local 192.168.10.121 port 49089 connected to 192.168.10.142 port 5201
Wireless: [ ID] Interval      Transfer   Bandwidth        Total Datagrams
Wireless: [ 4] 0.00–1.00 sec  120 KBytes 983 Kbits/sec    15
Wireless: [ 4] 1.00–2.00 sec  128 KBytes 1.05 Mbits/sec   16
Wireless: [ 4] 2.00–3.00 sec  128 KBytes 1.05 Mbits/sec   16
Wireless: [ 4] 3.00–4.00 sec  128 KBytes 1.05 Mbits/sec   16
Wireless: [ 4] 4.00–5.00 sec  128 KBytes 1.05 Mbits/sec   16
Wireless: [ 4] 5.00–6.00 sec  128 KBytes 1.05 Mbits/sec   16
Wireless: [ 4] 6.00–7.00 sec  128 KBytes 1.05 Mbits/sec   16
Wireless: [ 4] 7.00–8.00 sec  128 KBytes 1.05 Mbits/sec   16
Wireless: [ 4] 8.00–9.00 sec  128 KBytes 1.05 Mbits/sec   16
Wireless: [ 4] 9.00–10.00 sec 128 KBytes 1.05 Mbits/sec  16
Wireless: [ ID] Interval      Transfer    Bandwidth      Jitter   Lost/Total Datagrams
Wireless: [ 4] 0.00–10.00 sec 1.24 MBytes 1.04 Mbits/sec 0.273 ms 0/159 (0%)
Wireless: [ 4] Sent 159 datagrams Wireless:
Wireless: iperf Done.
Kali Linux running on VMware with a Linksys WUSB600N v1 Dual-Band Wireless-N Network Adapter [Ralink RT2870]

TCP test
Notice the retries and Congestion Window (Cwnd) data. TCP was adjusting to the drops in the wireless network.
root@kali–32:/iperf/examples# /usr/local/bin/iperf3 -c 192.168.10.142 -P 5 -T Wireless
Wireless: Connecting to host 192.168.10.142, port 5201
Wireless: [ 4] local 192.168.10.121 port 44897 connected to 192.168.10.142 port 5201
Wireless: [ 6] local 192.168.10.121 port 44898 connected to 192.168.10.142 port 5201
Wireless: [ 8] local 192.168.10.121 port 44899 connected to 192.168.10.142 port 5201
Wireless: [ 10] local 192.168.10.121 port 44900 connected to 192.168.10.142 port 5201
Wireless: [ 12] local 192.168.10.121 port 44901 connected to 192.168.10.142 port 5201
Wireless: [ ID] Interval       Transfer   Bandwidth       Retr Cwnd
Wireless: [ 4] 0.00–1.00 sec  1.11 MBytes 9.29 Mbits/sec  20   26.9 KBytes
Wireless: [ 6] 0.00–1.00 sec  1.76 MBytes 14.7 Mbits/sec  43   33.9 KBytes
Wireless: [ 8] 0.00–1.00 sec  959 KBytes  7.85 Mbits/sec  13   24.0 KBytes
Wireless: [ 10] 0.00–1.00 sec 1.05 MBytes 8.78 Mbits/sec  1    31.1 KBytes
Wireless: [ 12] 0.00–1.00 sec 2.95 MBytes 24.7 Mbits/sec  101  109 KBytes
Wireless: [SUM] 0.00–1.00 sec 7.80 MBytes 65.4 Mbits/sec  178
Wireless: - - - - - - - - - - - - - - - - - - - - - - - - -
Wireless: [ 4] 1.00–2.00 sec  993 KBytes  8.13 Mbits/sec  6    1.41 KBytes
Wireless: [ 6] 1.00–2.00 sec  1.44 MBytes 12.0 Mbits/sec  47   1.41 KBytes
Wireless: [ 8] 1.00–2.00 sec  889 KBytes  7.28 Mbits/sec  24   1.41 KBytes
Wireless: [ 10] 1.00–2.00 sec 1.03 MBytes 8.67 Mbits/sec  6    1.41 KBytes
Wireless: [ 12] 1.00–2.00 sec 3.25 MBytes 27.3 Mbits/sec  100  1.41 KBytes
Wireless: [SUM] 1.00–2.00 sec 7.56 MBytes 63.4 Mbits/sec  183
Wireless: - - - - - - - - - - - - - - - - - - - - - - - - -
Wireless: [ 4] 2.00–3.00 sec  731 KBytes  5.99 Mbits/sec  45   18.4 KBytes
Wireless: [ 6] 2.00–3.00 sec  1.61 MBytes 13.5 Mbits/sec  25   29.7 KBytes
Wireless: [ 8] 2.00–3.00 sec  1.33 MBytes 11.2 Mbits/sec  58   60.8 KBytes
Wireless: [ 10] 2.00–3.00 sec 2.09 MBytes 17.6 Mbits/sec  31   89.1 KBytes
Wireless: [ 12] 2.00–3.00 sec 2.62 MBytes 22.0 Mbits/sec  92   82.0 KBytes
Wireless: [SUM] 2.00–3.00 sec 8.38 MBytes 70.3 Mbits/sec  251
Wireless: - - - - - - - - - - - - - - - - - - - - - - - - -
Wireless: [ 4] 3.00–4.00 sec  1.39 MBytes 11.7 Mbits/sec  3    31.1 KBytes
Wireless: [ 6] 3.00–4.00 sec  1.91 MBytes 16.0 Mbits/sec  7    38.2 KBytes
Wireless: [ 8] 3.00–4.00 sec  4.28 MBytes 35.9 Mbits/sec  72   48.1 KBytes
Wireless: [ 10] 3.00–4.00 sec 2.12 MBytes 17.8 Mbits/sec  61   32.5 KBytes
Wireless: [ 12] 3.00–4.00 sec 2.97 MBytes 24.9 Mbits/sec  22   48.1 KBytes
Wireless: [SUM] 3.00–4.00 sec 12.7 MBytes 106 Mbits/sec   165
Wireless: - - - - - - - - - - - - - - - - - - - - - - - - -
Wireless: [ 4] 4.00–5.00 sec  2.58 MBytes 21.6 Mbits/sec  25   24.0 KBytes
Wireless: [ 6] 4.00–5.00 sec  2.72 MBytes 22.8 Mbits/sec  42   22.6 KBytes
Wireless: [ 8] 4.00–5.00 sec  4.26 MBytes 35.7 Mbits/sec  89   74.9 KBytes
Wireless: [ 10] 4.00–5.00 sec 2.63 MBytes 22.0 Mbits/sec  23   29.7 KBytes
Wireless: [ 12] 4.00–5.00 sec 2.76 MBytes 23.2 Mbits/sec  20   62.2 KBytes
Wireless: [SUM] 4.00–5.00 sec 15.0 MBytes 125 Mbits/sec   199
Wireless: - - - - - - - - - - - - - - - - - - - - - - - - -
Wireless: [ 4] 5.00–6.00 sec  1.80 MBytes 15.1 Mbits/sec  11   32.5 KBytes
Wireless: [ 6] 5.00–6.00 sec  1.62 MBytes 13.6 Mbits/sec  11   32.5 KBytes
Wireless: [ 8] 5.00–6.00 sec  3.14 MBytes 26.3 Mbits/sec  4    50.9 KBytes
Wireless: [ 10] 5.00–6.00 sec 1.94 MBytes 16.3 Mbits/sec  9    36.8 KBytes
Wireless: [ 12] 5.00–6.00 sec 4.70 MBytes 39.5 Mbits/sec  67   67.9 KBytes
Wireless: [SUM] 5.00–6.00 sec 13.2 MBytes 111 Mbits/sec   102
Wireless: - - - - - - - - - - - - - - - - - - - - - - - - -
Wireless: [ 4] 6.00–7.00 sec  1.72 MBytes 14.5 Mbits/sec  23   25.5 KBytes
Wireless: [ 6] 6.00–7.00 sec  2.18 MBytes 18.3 Mbits/sec  1    33.9 KBytes
Wireless: [ 8] 6.00–7.00 sec  2.83 MBytes 23.7 Mbits/sec  5    39.6 KBytes
Wireless: [ 10] 6.00–7.00 sec 2.86 MBytes 24.0 Mbits/sec  56   56.6 KBytes
Wireless: [ 12] 6.00–7.00 sec 4.04 MBytes 33.9 Mbits/sec  83   9.5 KBytes
Wireless: [SUM] 6.00–7.00 sec 13.6 MBytes 114 Mbits/sec   168
Wireless: - - - - - - - - - - - - - - - - - - - - - - - - -
Wireless: [ 4] 7.00–8.00 sec  1.40 MBytes 11.8 Mbits/sec  34   31.1 KBytes
Wireless: [ 6] 7.00–8.00 sec  2.05 MBytes 17.2 Mbits/sec  27   38.2 KBytes
Wireless: [ 8] 7.00–8.00 sec  2.04 MBytes 17.1 Mbits/sec  31   38.2 KBytes
Wireless: [ 10] 7.00–8.00 sec 2.93 MBytes 24.6 Mbits/sec  44   42.4 KBytes
Wireless: [ 12] 7.00–8.00 sec 3.23 MBytes 27.1 Mbits/sec  196  77.8 KBytes
Wireless: [SUM] 7.00–8.00 sec 11.7 MBytes 97.8 Mbits/sec  332
Wireless: - - - - - - - - - - - - - - - - - - - - - - - - -
Wireless: [ 4] 8.00–9.00 sec  1.78 MBytes 14.9 Mbits/sec  43   18.4 KBytes
Wireless: [ 6] 8.00–9.00 sec  2.64 MBytes 22.1 Mbits/sec  53   38.2 KBytes
Wireless: [ 8] 8.00–9.00 sec  2.45 MBytes 20.5 Mbits/sec  2    42.4 KBytes
Wireless: [ 10] 8.00–9.00 sec 2.20 MBytes 18.4 Mbits/sec  23   19.8 KBytes
Wireless: [ 12] 8.00–9.00 sec 4.64 MBytes 38.9 Mbits/sec  238  105 KBytes
Wireless: [SUM] 8.00–9.00 sec 13.7 MBytes 115 Mbits/sec   359
Wireless: - - - - - - - - - - - - - - - - - - - - - - - - -
Wireless: [ 4] 9.00–10.00 sec  2.34 MBytes 19.6 Mbits/sec 14  43.8 KBytes
Wireless: [ 6] 9.00–10.00 sec  1.87 MBytes 15.7 Mbits/sec 17  35.4 KBytes
Wireless: [ 8] 9.00–10.00 sec  1.43 MBytes 12.0 Mbits/sec 33  21.2 KBytes
Wireless: [ 10] 9.00–10.00 sec 979 KBytes 8.02 Mbits/sec  12  25.5 KBytes
Wireless: [ 12] 9.00–10.00 sec 5.57 MBytes 46.7 Mbits/sec 187 93.3 KBytes
Wireless: [SUM] 9.00–10.00 sec 12.2 MBytes 102 Mbits/sec  263
Wireless: - - - - - - - - - - - - - - - - - - - - - - - - -
Wireless: [ ID] Interval Transfer Bandwidth Retr
Wireless: [ 4] 0.00–10.00 sec 15.8 MBytes 13.3 Mbits/sec  224 sender
Wireless: [ 4] 0.00–10.00 sec 15.7 MBytes 13.2 Mbits/sec      receiver
Wireless: [ 6] 0.00–10.00 sec 19.8 MBytes 16.6 Mbits/sec  273 sender
Wireless: [ 6] 0.00–10.00 sec 19.7 MBytes 16.5 Mbits/sec      receiver
Wireless: [ 8] 0.00–10.00 sec 23.6 MBytes 19.8 Mbits/sec  331 sender
Wireless: [ 8] 0.00–10.00 sec 23.4 MBytes 19.6 Mbits/sec      receiver
Wireless: [ 10] 0.00–10.00 sec 19.8 MBytes 16.6 Mbits/sec 266 sender
Wireless: [ 10] 0.00–10.00 sec 19.7 MBytes 16.5 Mbits/sec     receiver
Wireless: [ 12] 0.00–10.00 sec 36.7 MBytes 30.8 Mbits/sec 1106 sender
Wireless: [ 12] 0.00–10.00 sec 36.5 MBytes 30.6 Mbits/sec     receiver
Wireless: [SUM] 0.00–10.00 sec 116 MBytes 97.1 Mbits/sec  2200 sender
Wireless: [SUM] 0.00–10.00 sec 115 MBytes 96.5 Mbits/sec       receiver
Wireless: 
Wireless: iperf Done.

Kali Linux running on VMware with a wired Ethernet connection
Notice that there are no retries.
root@kali–32:/iperf/examples# /usr/local/bin/iperf3 -c 192.168.10.129 -P 5 -t 5 -T Wired
Wired: Connecting to host 192.168.10.129, port 5201
Wired: [ 4] local 192.168.10.121 port 58734 connected to 192.168.10.129 port 5201
Wired: [ 6] local 192.168.10.121 port 58735 connected to 192.168.10.129 port 5201
Wired: [ 8] local 192.168.10.121 port 58736 connected to 192.168.10.129 port 5201
Wired: [ 10] local 192.168.10.121 port 58737 connected to 192.168.10.129 port 5201
Wired: [ 12] local 192.168.10.121 port 58738 connected to 192.168.10.129 port 5201
Wired: [ ID] Interval     Transfer      Bandwidth        Retr Cwnd
Wired: [ 4] 0.00–1.24 sec  3.75 MBytes  25.3 Mbits/sec   0    106 KBytes
Wired: [ 6] 0.00–1.24 sec  3.75 MBytes  25.3 Mbits/sec   0    106 KBytes
Wired: [ 8] 0.00–1.24 sec  3.75 MBytes  25.3 Mbits/sec   0    106 KBytes
Wired: [ 10] 0.00–1.24 sec 3.75 MBytes  25.3 Mbits/sec   0    105 KBytes
Wired: [ 12] 0.00–1.24 sec 3.75 MBytes  25.3 Mbits/sec   0    107 KBytes
Wired: [SUM] 0.00–1.24 sec 18.8 MBytes  127 Mbits/sec    0
Wired: - - - - - - - - - - - - - - - - - - - - - - - - -
Wired: [ 4] 1.24–2.17 sec  2.50 MBytes  22.5 Mbits/sec   0    130 KBytes
Wired: [ 6] 1.24–2.17 sec  2.50 MBytes  22.5 Mbits/sec   0    130 KBytes
Wired: [ 8] 1.24–2.17 sec  2.50 MBytes  22.5 Mbits/sec   0    130 KBytes
Wired: [ 10] 1.24–2.17 sec 2.50 MBytes  22.5 Mbits/sec   0    129 KBytes
Wired: [ 12] 1.24–2.17 sec 2.50 MBytes  22.5 Mbits/sec   0    132 KBytes
Wired: [SUM] 1.24–2.17 sec 12.5 MBytes  113 Mbits/sec    0
Wired: - - - - - - - - - - - - - - - - - - - - - - - - -
Wired: [ 4] 2.17–3.13 sec  2.50 MBytes  22.0 Mbits/sec   0    134 KBytes
Wired: [ 6] 2.17–3.13 sec  2.50 MBytes  22.0 Mbits/sec   0    134 KBytes
Wired: [ 8] 2.17–3.13 sec  2.50 MBytes  22.0 Mbits/sec   0    134 KBytes
Wired: [ 10] 2.17–3.13 sec 2.50 MBytes  22.0 Mbits/sec   0    134 KBytes
Wired: [ 12] 2.17–3.13 sec 2.50 MBytes  22.0 Mbits/sec   0    134 KBytes
Wired: [SUM] 2.17–3.13 sec 12.5 MBytes  110 Mbits/sec    0
Wired: - - - - - - - - - - - - - - - - - - - - - - - - -
Wired: [ 4] 3.13–4.13 sec  2.50 MBytes  20.9 Mbits/sec   0    134 KBytes
Wired: [ 6] 3.13–4.13 sec  2.50 MBytes  20.9 Mbits/sec   0    134 KBytes
Wired: [ 8] 3.13–4.13 sec  2.50 MBytes  20.9 Mbits/sec   0    134 KBytes
Wired: [ 10] 3.13–4.13 sec 2.50 MBytes  20.9 Mbits/sec   0    134 KBytes
Wired: [ 12] 3.13–4.13 sec 2.50 MBytes  20.9 Mbits/sec   0    134 KBytes
Wired: [SUM] 3.13–4.13 sec 12.5 MBytes  105 Mbits/sec    0
Wired: - - - - - - - - - - - - - - - - - - - - - - - - -
Wired: [ 4] 4.13–5.12 sec  2.50 MBytes  21.2 Mbits/sec   0    134 KBytes
Wired: [ 6] 4.13–5.12 sec  2.50 MBytes  21.2 Mbits/sec   0    134 KBytes
Wired: [ 8] 4.13–5.12 sec  2.50 MBytes  21.2 Mbits/sec   0    134 KBytes
Wired: [ 10] 4.13–5.12 sec 2.50 MBytes  21.2 Mbits/sec   0    134 KBytes
Wired: [ 12] 4.13–5.12 sec 2.50 MBytes  21.2 Mbits/sec   0    134 KBytes
Wired: [SUM] 4.13–5.12 sec 12.5 MBytes  106 Mbits/sec    0
Wired: - - - - - - - - - - - - - - - - - - - - - - - - -
Wired: [ ID]  Interval     Transfer    Bandwidth         Retr
Wired: [ 4] 0.00–5.12 sec  13.8 MBytes 22.5 Mbits/sec    0 sender
Wired: [ 4] 0.00–5.12 sec  13.8 MBytes 22.5 Mbits/sec     receiver
Wired: [ 6] 0.00–5.12 sec  13.8 MBytes 22.5 Mbits/sec    0 sender
Wired: [ 6] 0.00–5.12 sec  13.8 MBytes 22.5 Mbits/sec     receiver
Wired: [ 8] 0.00–5.12 sec  13.8 MBytes 22.5 Mbits/sec    0 sender
Wired: [ 8] 0.00–5.12 sec  13.8 MBytes 22.5 Mbits/sec     receiver
Wired: [ 10] 0.00–5.12 sec 13.8 MBytes 22.5 Mbits/sec    0 sender
Wired: [ 10] 0.00–5.12 sec 13.8 MBytes 22.5 Mbits/sec     receiver
Wired: [ 12] 0.00–5.12 sec 13.8 MBytes 22.5 Mbits/sec    0 sender
Wired: [ 12] 0.00–5.12 sec 13.8 MBytes 22.5 Mbits/sec     receiver
Wired: [SUM] 0.00–5.12 sec 68.8 MBytes 113 Mbits/sec     0 sender
Wired: [SUM] 0.00–5.12 sec 68.8 MBytes 113 Mbits/sec      receiver
Wired:
Wired: iperf Done.

An Example using iPerf to Verify a Switching Problem
A user reported that the wireless network was very slow. I connected to the wireless network and ran iPerf. At first throughput looked good but within a few seconds it dropped to 0Bps for several seconds and then went back up. This repeated for as long as I ran iPerf. But the connection to the AP didn’t drop and there were no unusual log entries on the AP.

I connected to Ethernet, reran iPerf and observed the same behavior only at Ethernet speeds. Obviously it wasn’t a Wireless issue. Running “sh proc cpu sorted” on the switch revealed cpu up around 95%. Obviously something was wrong on the switch but that’s a blog for another day. Bottom line is that 5 minutes with iPerf revealed that the problem wasn’t the wireless network and I moved on to find the root cause was a bug in the switch firmware.

1S1K–873:iperf mhubbard$ /usr/local/bin/iperf3 -c 10.140.44.149 -t 1000
Connecting to host 10.140.44.149, port 5201
[ 4] local 10.141.1.217 port 57242 connected to 10.140.44.149 port 5201
[ ID]  Interval      Transfer    Bandwidth
[ 4] 0.00–1.00 sec   16.1 MBytes 135 Mbits/sec
[ 4] 1.00–2.00 sec   14.4 MBytes 121 Mbits/sec
[ 4] 2.00–3.00 sec   15.8 MBytes 132 Mbits/sec
[ 4] 3.00–4.00 sec   17.5 MBytes 147 Mbits/sec
[ 4] 4.00–5.00 sec   15.8 MBytes 132 Mbits/sec
[ 4] 5.00–6.00 sec   13.8 MBytes 116 Mbits/sec
[ 4] 6.00–7.00 sec   13.9 MBytes 116 Mbits/sec
[ 4] 7.00–8.00 sec   15.4 MBytes 129 Mbits/sec
[ 4] 8.00–9.00 sec   16.3 MBytes 136 Mbits/sec
[ 4] 9.00–10.00 sec  14.1 MBytes 118 Mbits/sec
[ 4] 10.00–11.00 sec 0.00 Bytes  0.00 bits/sec
[ 4] 11.00–12.00 sec 0.00 Bytes  0.00 bits/sec
[ 4] 12.00–13.00 sec 0.00 Bytes  0.00 bits/sec
[ 4] 13.00–14.00 sec 0.00 Bytes  0.00 bits/sec
[ 4] 14.00–15.00 sec 0.00 Bytes  0.00 bits/sec
[ 4] 15.00–16.00 sec 0.00 Bytes  0.00 bits/sec
[ 4] 16.00–17.00 sec 0.00 Bytes  0.00 bits/sec
[ 4] 17.00–18.00 sec 0.00 Bytes  0.00 bits/sec
[ 4] 18.00–19.00 sec 0.00 Bytes  0.00 bits/sec
[ 4] 19.00–20.00 sec 0.00 Bytes  0.00 bits/sec
[ 4] 20.00–21.00 sec 6.60 MBytes 55.3 Mbits/sec
[ 4] 21.00–22.00 sec 13.6 MBytes 114 Mbits/sec
[ 4] 22.00–23.00 sec 14.3 MBytes 120 Mbits/sec
[ 4] 23.00–24.00 sec 14.6 MBytes 122 Mbits/sec
[ 4] 24.00–25.00 sec 16.4 MBytes 137 Mbits/sec
[ 4] 25.00–26.00 sec 16.5 MBytes 138 Mbits/sec
[ 4] 26.00–27.00 sec 14.4 MBytes 121 Mbits/sec
[ 4] 27.00–28.00 sec 16.9 MBytes 142 Mbits/sec
[ 4] 28.00–29.00 sec 16.0 MBytes 135 Mbits/sec
[ 4] 29.00–30.00 sec 16.0 MBytes 134 Mbits/sec
[ 4] 30.00–31.00 sec 16.7 MBytes 140 Mbits/sec
[ 4] 31.00–32.00 sec 16.8 MBytes 141 Mbits/sec
[ 4] 32.00–33.00 sec 15.8 MBytes 132 Mbits/sec
[ 4] 33.00–34.00 sec 14.6 MBytes 122 Mbits/sec
[ 4] 34.00–35.00 sec 15.0 MBytes 126 Mbits/sec
[ 4] 35.00–36.00 sec 14.5 MBytes 121 Mbits/sec
[ 4] 36.00–37.00 sec 16.4 MBytes 138 Mbits/sec
[ 4] 37.00–38.00 sec 16.5 MBytes 138 Mbits/sec
[ 4] 38.00–39.00 sec 15.7 MBytes 132 Mbits/sec
[ 4] 39.00–40.00 sec 16.9 MBytes 142 Mbits/sec
[ 4] 40.00–41.00 sec 14.7 MBytes 123 Mbits/sec
[ 4] 41.00–42.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 42.00–43.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 43.00–44.00 sec 0.00 Bytes 0.00 bits/sec

Running iPerf for an extended period
Recently a coworker needed to run iPerf over and over while logging the output with a time stamp. I wrote a simple bash shell script to do it. It uses a while do loop to call iPerf. A simple counter is used to control how many times to loop.

I added some formatting to make the report look good and a time stamp to show when the tests were run.

#!/bin/bash
#Start iperf with a time stamp and redirect to a file.
# Define a timestamp function
timestamp() {
  date +"%T"
}
c="5"
t="1"
echo "*************************************************************************" >> iperf.log
echo " " >> iperf.log
while [ $c -gt 0 ]
do
echo "$(timestamp): iPerf Test $t"  >> iperf.log
echo " " >> iperf.log
echo "*************************************************************************" >> iperf.log
iperf3 -c 192.168.10.161 -P 4 -t 5 -T iPerf-test >> iperf.log
echo " " >> iperf.log
echo "*************************************************************************" >> iperf.log
echo " " >> iperf.log
echo "iPerf Test $t complete"
sleep 1
c=$[$c-1]
t=$[$t+1]
done

To run it save the above to iperfloop.sh. Then make it executable using:
chmod +x iperfloop.sh

then ./iperfloop.sh to run it.

Here is what the output looks like:
mhubbard@1S1K-SYS76:~/michael.hubbard999@gmail.com/02_ceh/bashbunny$ ./iperfloop.sh
iPerf Test 1 complete
iPerf Test 2 complete
iPerf Test 3 complete
iPerf Test 4 complete
iPerf Test 5 complete

The script creates iperf.log in the folder where the script was executed. Here is what the log looks like:
*************************************************************************

22:08:48: iPerf Test 1

*************************************************************************
iPerf-test:  Connecting to host 192.168.10.161, port 5201
iPerf-test:  [  4] local 192.168.10.152 port 59086 connected to 192.168.10.161 port 5201
iPerf-test:  [  6] local 192.168.10.152 port 59088 connected to 192.168.10.161 port 5201
iPerf-test:  [  8] local 192.168.10.152 port 59090 connected to 192.168.10.161 port 5201
iPerf-test:  [ 10] local 192.168.10.152 port 59092 connected to 192.168.10.161 port 5201
iPerf-test:  [ ID] Interval           Transfer     Bandwidth       Retr  Cwnd


The second request was to monitor the iperf server and make sure if it failed that it restarted. I found a post here that showed how to do it.

Here is the script listing. I ran this on my CentOS server.

#!/bin/bash
myserver="iperf3 -s"
until myserver; do
    echo "Server 'myserver' crashed with exit code $?.  Respawning.." >&2
    sleep 1
done

Same process as above - Save it, chmod +x and run it. It worked perfectly. I never had iPerf3 on CentOS die but I used ps ef | grep iperf to find the PID and then used kill "PID" to kill the process. The script started it back each time and the loop on the client just kept going.


Use SSH to run iPerf
I carry a Mac Book Air and can use it as the iPerf3 client as I move around a site. But iPerf3 on the Mac doesn't show the Cwnd or Retries. To get around this I installed SSH on Kali and then I SSH into the Linux box and use the Mac as the iPerf server. Here is a link to the Blackmore site with instructions to install SSH www.blackmoreops.com

Having Kali running an SSH server also allows me to access the Kali terminal from my iPhone and iPad. This is very useful especially if I am offsite and don't have LAN connectivity. I can connect over VPN and then shell into Kali.

There are many SSH clients for IOS but my favorite is  get console. It has full Dropbox integration, scripting and supports Serial, SSH and Telnet. How does it do Serial on an IOS device you ask? They sell a 30 Pin or Lightning serial cable but they also have a device called AirConsole.

Airconsole is a great device with WiFi, USB and RJ45 ports. It has a DHCP server built in so you can console into a device while providing DHCP over Ethernet to run TFTP for firmware upgrades. I use it a lot for setting up HP's iLo by connecting Ethernet to the server's ilo port, iLo gets a DHCP address, then I connect to the Airconsole's WiFi network, open a browser and configure iLo.

Sunday, December 14, 2014

Preventing a User from Causing a Loop with an Unmanaged Switch

I bet you have been in this situation - A user brings in a Linksys or Netgear unmanaged switch and plug it in. Then they connect 5 devices and think they are network engineers.

It’s no problem, at least as far as loops go, until they unplug a couple devices and leave the cables laying there. Then someone else comes along and plugs the cable back into the switch. Now you have a loop and everything grinds to a halt while you start trouble shooting.

A better way
Use the interface command “spanning-tree bpduguard enable” to shut the port down as soon as it sees the BPDU it sent returned through the unmanaged switch.
You can enter “show interfaces status err” to see any ports that are in the Error-Disabled state.

Automatic recovery
You can enable automatic recovery from spanning-tree bpduguard enable using the global command “errdisable recovery cause bpduguard”. With this configured a timer will start as soon as the port is disabled. The default for the timer is 300 seconds. You can override this using “errdisable recovery interval xxx” to change the timer to a new value, for example “errdisable recovery interval 500” to make it 500 seconds.

A Cautionary Note
One thing to think about before you enable automatic recovery for BPDUguard is whether the user will notice the problem and remove the cable.

If they do the port will be re-enabled when the timer expires. The switch will now be working again and no call to the helpdesk is needed.

But what happens if the user just leaves the cable attached and leaves? The counter will expire, the port will be enabled and the loop will start again. This will repeat every “timer interval” until the cable is removed. It will create havoc on a schedule.

Here are messages from a switch with a looped unmanaged switch and auto recovery enabled:
* 000044: Dec 14 05:08:56.632 PST: %SPANTREE–2-BLOCK_BPDUGUARD: Received BPDU on port Gi1/0/48 with BPDU Guard enabled. Disabling port.
* 000049: Dec 14 05:09:26.628 PST: %PM–4-ERR_RECOVER: Attempting to recover from bpduguard err-disable state on Gi1/0/48
* 000050: *Dec 14 05:09:29.888 PST: %SPANTREE–2-BLOCK_BPDUGUARD: Received BPDU on port Gi1/0/48 with BPDU Guard enabled. Disabling port.

My recommendation is NOT to use auto-recovery for BPDUguard.

Related commands
* show errdisable recovery - Lists all ErrDisable reasons and their state.
* show interfaces status err - Lists any interfaces that are in the Error Disabled state.

Reference
Cisco 3850 switch manual

Sunday, November 9, 2014

Debugging Cisco Device Authentication to a Microsoft NPS Server

You have configured a Microsoft Network Policy Server to use as a RADIUS server for your Cisco devices. But for some reason your logins aren't successful. Use the following to trouble shoot the NPS Server:

Enable Logging on the NPS Server
Open a cmd window and enter:
netsh ras diagnostics set rastracing * enabled


Be sure to disable afterward (netsh ras diagnostics set rastracing * disabled).

These logs are saved to %windir%\tracing. You can find more detailed infromation on logging at Enabling RAS Tracing in Vista/Longhorn Server 


Accounting logs are saved to C:\Windows\system32\LogFiles by default. You can change this by clicking on “Accounting” in the NPS console.

Click Start, Run and enter GPEDIT.MSC
Enable both success/failure at Computer Configuration -> Windows Settings -> Security Settings -> Advanced Audit Policy Configuration -> System Audit Policies -> Logon/Logoff -> Audit Network Policy Server.

In the command window  run “netsh nps show eventlog” and “netsh nps show filelog” on the NPS server, check that the results are:

C:\Windows\system32>netsh nps show filelog

File log configuration:
---------------------------------------------------------
Accounting                     = Enabled
Authentication                 = Enabled
Periodic accounting status     = Enabled
Periodic authentication status = Enabled
Directory                      = C:\Windows\system32\LogFiles
Format                         = ODBC formatting
Delete old logs                = Enabled
Frequency                      = Monthly logs
Max size                       = 10 MB

C:\Windows\system32>netsh nps show eventlog

Event log configuration:
---------------------------------------------------------
Accepted authentication requests = Enabled
Rejected authentication requests = Enabled

Viewing NPS logs

Open Server Manager and expand Diagnostics, Event Viewer, Custom Views, Server Roles, Network Policy and Access Server

Request Not Matching the Network Policy


If you see the following on the client you are using to log in 'Line has invalid autocommand " ppp negotiate  "' it probably means that the request isn't matching the network policy you created. Putty will close the session before you can see the message. If you try to log in and it looks successful but the session immediately closes try using a different client.

You can verify by looking at the NPS logs. Find the failed attempt, double click, and select the Detail tab. Scroll down until you can see the ProxyPolicyName. Right below that will be the Network Policy name that was matched. You can see in the example below that it's blank.

The reason this causes the " ppp negotiate " is that the default policy in NPS has a service-type of PPP.

From the "Details" tab of the NPS server log viewer:
ProxyPolicyName CISCO-Radius
NetworkPolicyName -
AuthenticationProvider Windows
AuthenticationServer NPS.LAB.PRI
AuthenticationType PAP

In this example I had moved the NPS server to a new AD domain and the policy just wouldn't match. I deleted it, created a new one and all was good.


Invalid Radius Client

It’s possible the client is using a different interface to communicate with the NPS server than the one you configured. The event log entry for an invalid client is 13. You can filter the log to see if your device is using the address you expected.


You can use "ip radius source-interface InterfaceName" to force radius to use a specific interface. For example, if the management interface is vlan 18 use:

ip radius source-interface Vlan18


Invalid Username/Password

If the user entered an invalid username or password event ID 6273 will be recorded. Filter the log on 6273 to find the details. The log records the “friendly Client name”, the Client IP address and message about the failed credentials.



Note: If the RADIUS keys don't match between the device and the client entry in the NPS server you will also get the same message in the log!

NPS Logs show "Access Granted" but the switch doesn't get a response

This was probably a fluke but in one instance I had changed the hostname AND IP address of a switch. The NPS server showed Access Granted but when I looked at the detail in the log the NASIPv4Address was correct but the ClientName and ClientIPAddress were the old values so NPS was sending the response to the wrong IP. I reloaded the switch and it worked correctly.



No Log Entries

There are a few reasons that the log doesn’t contain any entries from the device. The device will accept the first method that has a successful username/password match. I once entered the “Local” keyword before the RADIUS group in the authentication string. I had a local username that was the same as my AD username but with a different password.

The device would see the local username and incorrect password and terminate the login attempt. No log entries were recorded because the device stopped the login attempt as soon as the “Local” method failed.

You may also have put in the wrong IP address of the NPS server in your RADIUS definition. The commands below will help you trouble shoot that.

Finally, if you don't authorize the NPS server in AD it won't show any log entries.

Verifying IOS RADIUS configuration
sh aaa servers – lists the IP of the server and the ports it is using. Make sure that you used the same ports on the NPS server and the device.
#sh aaa server
RADIUS: id 1, priority 1, host 10.140.47.205, auth-port 1645, acct-port 1646


User successfully authenticated
USER ATTRIBUTES
service-type         0   1 [Login]
priv-lvl             0   15 (0xF)

There are many debugs you can enable for trouble shooting RADIUS. The six listed below seem to be enough to solve most issues.

debug aaa authentication
debug aaa sg-server-selection
debug aaa protocol radius
debug aaa author
debug radius elog
debug radius verbose

With these enabled, you will see the following for a wrong password and a successful login:

*******************************************
User entered wrong password
024847: Sep 29 21:03:57.710 PDT: AAA/BIND(00001047): Bind i/f
024848: Sep 29 21:03:57.711 PDT: AAA/AUTHEN/LOGIN (00001047): Pick method list 'UseRadius'
024849: Sep 29 21:03:57.711 PDT: RADIUS/ENCODE(00001047): ask "Please enter your username:"
024850: Sep 29 21:03:57.711 PDT: RADIUS/ENCODE(00001047): send packet; GET_USER
024851: Sep 29 21:04:01.992 PDT: RADIUS/ENCODE(00001047): ask "Please enter your password:"
024852: Sep 29 21:04:01.992 PDT: RADIUS/ENCODE(00001047): send packet; GET_PASSWORD
024853: Sep 29 21:04:06.781 PDT: RADIUS/ENCODE(00001047):Orig. component type = Exec
024854: Sep 29 21:04:06.781 PDT: RADIUS(00001047): Config NAS IP: 0.0.0.0
024855: Sep 29 21:04:06.781 PDT: RADIUS(00001047): Config NAS IPv6: ::
024856: Sep 29 21:04:06.781 PDT: RADIUS/ENCODE: Best Local IP-Address 10.140.119.207 for Radius-Server 10.140.47.205
024857: Sep 29 21:04:06.781 PDT: RADIUS(00001047): Sending a IPv4 Radius Packet
024858: Sep 29 21:04:06.782 PDT: RADIUS(00001047): Started 5 sec timeout
024859: Sep 29 21:04:06.877 PDT: RADIUS: Received from id 1645/10 10.140.47.205:1645, 
Access-Reject, len 20

User entered correct password
024865: Sep 29 21:06:12.137 PDT: AAA/BIND(00001048): Bind i/f
024866: Sep 29 21:06:12.137 PDT: AAA/AUTHEN/LOGIN (00001048): Pick method list 'UseRadius'
024867: Sep 29 21:06:12.138 PDT: RADIUS/ENCODE(00001048): ask "Please enter your username:"
024868: Sep 29 21:06:12.138 PDT: RADIUS/ENCODE(00001048): send packet; GET_USER
024869: Sep 29 21:06:14.844 PDT: RADIUS/ENCODE(00001048): ask "Please enter your pasword:"
024870: Sep 29 21:06:14.844 PDT: RADIUS/ENCODE(00001048): send packet; GET_PASSWORD
024871: Sep 29 21:06:19.677 PDT: RADIUS/ENCODE(00001048):Orig. component type = Exec
024872: Sep 29 21:06:19.677 PDT: RADIUS(00001048): Config NAS IP: 0.0.0.0
024873: Sep 29 21:06:19.677 PDT: RADIUS(00001048): Config NAS IPv6: ::
024874: Sep 29 21:06:19.678 PDT: RADIUS/ENCODE: Best Local IP-Address 10.140.119.207 for Radius-Server 10.140.47.205
024875: Sep 29 21:06:19.678 PDT: RADIUS(00001048): Sending a IPv4 Radius Packet
024876: Sep 29 21:06:19.678 PDT: RADIUS(00001048): Started 5 sec timeout
024877: Sep 29 21:06:19.692 PDT: RADIUS: Received from id 1645/11 10.140.47.205:1645, 
Access-Accept, len 152
024878: Sep 29 21:06:19.692 PDT: AAA/AUTHOR/EXEC(00001048): processing AV priv-lvl=15
024879: Sep 29 21:06:19.692 PDT: AAA/AUTHOR/EXEC(00001048): processing AV priv-lvl=15
024880: Sep 29 21:06:19.692 PDT: AAA/AUTHOR/EXEC(00001048): processing AV service-type=1
024881: Sep 29 21:06:19.693 PDT: AAA/AUTHOR/EXEC(00001048): 
Authorization successful


In this debug I forgot to enter the key in the RADIUS section

001939: Sep 24 13:04:48 PST: RADIUS/ENCODE(00000040):Orig. component type = Exec
001940: Sep 24 13:04:48 PST: RADIUS:  AAA Unsupported Attr: interface         [221] 4   125716940
001941: Sep 24 13:04:48 PST: RADIUS/ENCODE(00000040): Unsupported AAA attribute clid
001942: Sep 24 13:04:48 PST: RADIUS/ENCODE(00000040): dropping service type, "radius-server attribute 6 on-for-login-auth" is off
001943: Sep 24 13:04:48 PST: RADIUS(00000040): Config NAS IP: 10.200.18.36
001944: Sep 24 13:04:48 PST: RADIUS(00000040): Config NAS IPv6: ::
001945: Sep 24 13:04:48 PST: RADIUS/ENCODE(00000040): acct_session_id: 54
001946: Sep 24 13:04:48 PST: RADIUS(00000040): sending
001947: Sep 24 13:04:48 PST: RADIUS: No secret to encode request (rctx:0x7A46DCC)
001948: Sep 24 13:04:48 PST: RADIUS: Unable to encrypt (rctx:0x7A46DCC)
001949: Sep 24 13:04:48 PST: RADIUS: No secret to encode request (rctx:0x7A46DCC)
001950: Sep 24 13:04:48 PST: RADIUS: Unable to encrypt (rctx:0x7A46DCC)
001951: Sep 24 13:04:48 PST: %RADIUS-4-RADIUS_DEAD: RADIUS server 10.1.52.59:1645,1646 is not responding.
001952: Sep 24 13:04:48 PST: %RADIUS-4-RADIUS_ALIVE: RADIUS server 10.1.52.59:1645,1646 is being marked alive.
001953: Sep 24 13:04:48 PST: RADIUS: No secret to encode request (rctx:0x7A46DCC)
001954: Sep 24 13:04:48 PST: RADIUS: Unable to encrypt (rctx:0x7A46DCC)
001955: Sep 24 13:04:48 PST: RADIUS: No secret to encode request (rctx:0x7A46DCC)
001956: Sep 24 13:04:48 PST: RADIUS: Unable to encrypt (rctx:0x7A46DCC)
001957: Sep 24 13:04:48 PST: RADIUS: No secret to encode request (rctx:0x7A46DCC)
001958: Sep 24 13:04:48 PST: RADIUS: Unable to encrypt (rctx:0x7A46DCC)
001959: Sep 24 13:04:48 PST: RADIUS/DECODE: No response from radius-server; parse response; FAIL

001960: Sep 24 13:04:48 PST: RADIUS/DECODE: Case error(no response/ bad packet/ op decode);parse response; FAIL