Saturday, April 6, 2019

Troubleshooting Multicast Routing

Mulitcast is used for applications like paging and bell systems (Bogen and Valcom for example) and Apple Airplay for screen sharing and Airprint for printing as well as many others. On home networks with one VLAN there are seldom any issues getting the multicast applications to work.

In an enterprise network with multiple VLANs, the switch must be configured for multicast to be routed correctly. Troubleshooting multicast issues can be time-consuming and without some tools almost impossible.

Linux has several terminal commands for checking Multicast status of an interface:

ip maddr show - list all multicast interfaces

Example:
ip maddr show
1: lo
inet  224.0.0.251
inet  224.0.0.1
2: wlp0s20f3
inet  224.0.0.251 users 4
inet  224.0.0.1

ip addr show <interface> - Look for MULTICAST

Example:
ip addr show wlp0s20f3 
3: wlp0s20f3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000

ip link show <interface> | grep MULTICAST

Example:
ip link show wlp0s20f3 | grep MULTICAST
3: wlp0s20f3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DORMANT group default qlen 1000

Enable/Disable Multicast
sudo ip link set dev <interface> multicast [on|off]

netstat -g - Lists all multicast interfaces

Example:
netstat -g
IPv6/IPv4 Group Memberships
Interface       RefCnt Group
--------------- ------ ---------------------
lo              1      224.0.0.251
lo              1      all-systems.mcast.net
enp60s0         1      all-systems.mcast.net
wlp0s20f3       4      224.0.0.251
wlp0s20f3       1      all-systems.mcast.net

Pinging Multicast groups

ping -r -I wlp0s20f3 -t 1 -c 2 224.0.0.1
PING 224.0.0.1 (224.0.0.1) from 192.168.10.183 wlp0s20f3: 56(84) bytes of data.
64 bytes from 192.168.10.52: icmp_seq=1 ttl=255 time=3.89 ms
64 bytes from 192.168.10.50: icmp_seq=1 ttl=64 time=4.20 ms (DUP!)
64 bytes from 192.168.10.51: icmp_seq=1 ttl=64 time=5.50 ms (DUP!)
64 bytes from 192.168.10.50: icmp_seq=2 ttl=64 time=1.55 ms

-r - Bypass the normal routing tables and send directly to a host on an attached interface.  If the host is not on a directly-attached network, an error is returned.  This option can be used to ping a local host through an interface that has no route through it provided the option -I is also used.

-I - interface is either an address, or an interface name.  If interface is an address, it sets source address to specified interface address.  If interface is an interface name, it sets source interface to specified interface.

-t - ttl ping only.  Set the IP Time to Live.

-c - Count


On Linux, there are several free open source tools available for testing multicast. In this blog, I am going to demonstrate two tools
mcjoin - Simple multicast testing application for UNIX
omping - Open Multicast ping

The lab consists of:
Cisco 3750x switch running c3750e-universalk9-mz.152-3.E1 with an IP Services license
Ubuntu 18.04 laptop with IP Address 10.112.40.1/23
Ubuntu 18.04 laptop with IP Address 192.168.10.183/24

Vlan 10 - 192.168.10.0/24
Vlan 46 - 10.112.40.0.23

m c j o i n - tiny multicast testing tool

mcjoin is a very simple and easy-to-use tool to test IPv4 and IPv6 multicast. it features:

    an optional multicast generator (server)
    an end device that can act as a data sink (client)
    supports joining one or more groups:
        ASM (*,G) support
        SSM (S,G) support
    IPv4
    IPv6

Installing mcjoin

Download mcjoin_2.4_amd64.deb from mcjoin releases and double click to install.

The manual page for mcjoin is very good. Open it by running
man mcjoin

Simple usage example

Sender - 192.160.10.183/24 Vlan 10
Receiver - 10.112.40.1/23 Vlan46

sender$ mcjoin -s

receiver$ mcjoin
joined group 225.1.2.3 on eth0 ...
..................................................................
Received total: 66 packets
receiver$

In this example, you start mcjoin on the sender laptop with the -s switch and mcjoin with no options on the receiver laptop. After 30 seconds or so press ctrl+c to stop the process. If multicast was successful you see the number of packets that were received.

A more advanced example

Say you want to verify that your topology can forward 5 consecutive groups in the MCAST_TEST_NET, as defined in RFC5771.  Simply add the following as a standalone argument to both the receiver and the sender: 233.252.0.1+5.

On the sender
./mcjoin -s 233.252.0.1+5

On the Receiver
./mcjoin -t3 -i wlp0s20f3 233.252.0.1+5
joined group 233.252.0.1 on wlp0s20f3 ...
joined group 233.252.0.2 on wlp0s20f3 ...
joined group 233.252.0.3 on wlp0s20f3 ...
joined group 233.252.0.4 on wlp0s20f3 ...
joined group 233.252.0.5 on wlp0s20f3 ...
..................................^C
Group 233.252.0.1 received 40 packets
Group 233.252.0.2 received 40 packets
Group 233.252.0.3 received 40 packets
Group 233.252.0.4 received 40 packets
Group 233.252.0.5 received 40 packets
Received total: 200 packets


Using omping (open multicast ping)

omping is available from the same github site as mcjoin.

Install omping
https://github.com/troglobit/omping/

Simple example with two hosts. You can use more than two hosts.

Run the following on both laptops

./omping 10.112.40.1 192.168.10.183
10.112.40.1 : waiting for response msg
10.112.40.1 : joined (S,G) = (*, 232.43.211.234), pinging
10.112.40.1 :   unicast, seq=1, size=69 bytes, dist=1, time=1.669ms
10.112.40.1 :   unicast, seq=2, size=69 bytes, dist=1, time=3.906ms
10.112.40.1 : multicast, seq=2, size=69 bytes, dist=1, time=4.177ms
10.112.40.1 :   unicast, seq=3, size=69 bytes, dist=1, time=3.564ms
10.112.40.1 : multicast, seq=3, size=69 bytes, dist=1, time=3.860ms
^C
10.112.40.1 :   unicast, xmt/rcv/%loss = 3/3/0%, min/avg/max/std-dev = 1.669/3.046/3.906/1.205
10.112.40.1 : multicast, xmt/rcv/%loss = 3/2/33% (seq>=2 0%), min/avg/max/std-dev = 3.860/4.019/4.177/0.224

On the Switch


sh ip igmp groups 
IGMP Connected Group Membership
Group Address    Interface                Uptime    Expires   Last Reporter   Group Accounted
233.89.188.1     Vlan10                   02:12:24  00:02:37  192.168.10.50   
239.255.255.254  Vlan10                   02:12:24  00:02:41  192.168.10.221  
239.255.255.250  Vlan10                   02:12:25  00:02:40  192.168.10.239  
232.43.211.234   Vlan10                   00:00:02  00:02:57  192.168.10.183  
232.43.211.234   Vlan46                   00:00:13  00:02:46  10.112.40.1     
224.0.1.60       Vlan10                   02:12:24  00:02:43  192.168.10.239  
224.0.1.40       Vlan10                   01:52:28  00:02:36  192.168.10.52   
224.0.1.140      Vlan10                   02:12:24  00:02:39  192.168.10.254  

Using omping to test multicast with a specified multicast IP and port

From 192.168.10.183


./omping -m 233.252.0.1 -p 9106 10.112.40.1 192.168.10.183
10.112.40.1 : waiting for response msg
10.112.40.1 : waiting for response msg
10.112.40.1 : joined (S,G) = (*, 233.252.0.1), pinging
10.112.40.1 :   unicast, seq=1, size=69 bytes, dist=1, time=1.574ms
10.112.40.1 : multicast, seq=1, size=69 bytes, dist=1, time=53.798ms
10.112.40.1 :   unicast, seq=2, size=69 bytes, dist=1, time=2.608ms
10.112.40.1 : multicast, seq=2, size=69 bytes, dist=1, time=2.608ms
10.112.40.1 :   unicast, seq=3, size=69 bytes, dist=1, time=1.679ms
10.112.40.1 : multicast, seq=3, size=69 bytes, dist=1, time=1.809ms
10.112.40.1 :   unicast, seq=4, size=69 bytes, dist=1, time=1.595ms
10.112.40.1 : multicast, seq=4, size=69 bytes, dist=1, time=1.901ms
10.112.40.1 :   unicast, seq=5, size=69 bytes, dist=1, time=1.811ms
10.112.40.1 : multicast, seq=5, size=69 bytes, dist=1, time=2.043ms
10.112.40.1 : waiting for response msg
10.112.40.1 : server told us to stop

10.112.40.1 :   unicast, xmt/rcv/%loss = 5/5/0%, min/avg/max/std-dev = 1.574/1.853/2.608/0.432
10.112.40.1 : multicast, xmt/rcv/%loss = 5/5/0%, min/avg/max/std-dev = 1.809/12.432/53.798/23.126


On the switch

sh ip igmp groups
IGMP Connected Group Membership
Group Address    Interface                Uptime    Expires   Last Reporter   Group Accounted
233.89.188.1     Vlan10                   02:13:26  00:02:34  192.168.10.50   
239.255.255.254  Vlan10                   02:13:26  00:02:33  192.168.10.221  
239.255.255.250  Vlan10                   02:13:27  00:02:40  192.168.10.239  
233.252.0.1      Vlan10                   00:00:09  00:00:01  192.168.10.183  
233.252.0.1      Vlan46                   00:00:13  00:02:46  10.112.40.1     
224.0.1.60       Vlan10                   02:13:27  00:02:37  192.168.10.239  
224.0.1.40       Vlan10                   01:53:31  00:02:41  192.168.10.52   
224.0.1.140      Vlan10                   02:13:26  00:02:36  192.168.10.254  


Switch Configuration

3750x(config)#ip multicast-routing distributed 

interface vl 10
 ip address 192.168.10.52 255.255.255.0
 no ip redirects
  ip pim sparse-mode
end

interface Vlan46
 ip address 10.112.41.254 255.255.254.0
 ip helper-address 192.168.10.221
  ip pim sparse-mode
end

Create a loopback to use as the rendezvous point (RP)
interface Loopback0
 ip address 10.10.10.10 255.255.255.255
end

Create the Rendevouz point (RP)
ip pim rp-address 10.10.10.10

Show the rendezvous point
sh ip pim rp            
Group: 233.89.188.1, RP: 10.10.10.10, next RP-reachable never
Group: 239.255.255.254, RP: 10.10.10.10, next RP-reachable never
Group: 239.255.255.250, RP: 10.10.10.10, next RP-reachable never
Group: 233.252.0.1, RP: 10.10.10.10, next RP-reachable never
Group: 224.0.1.60, RP: 10.10.10.10, next RP-reachable never
Group: 224.0.1.40, RP: 10.10.10.10, next RP-reachable never
Group: 224.0.1.140, RP: 10.10.10.10, next RP-reachable never

sh ip multicast 
  Multicast Routing: enabled
  Multicast Multipath: disabled
  Multicast Route limit: No limit
  Multicast Fallback group mode: Dense
  Number of multicast boundaries configured with filter-autorp option: 0
  MoFRR: Disabled

From the Cisco multicast manual - Mulitcast configuration on an SVI
An SVI—A VLAN interface created by using the interface vlan vlan-id global configuration command. You will also need to enable IP PIM sparse-dense-mode on the VLAN, join the VLAN as a statically connected member to an IGMP static group, and then enable IGMP snooping on the VLAN, the IGMP static group, and physical interface. These interfaces must have IP addresses assigned to them. 

sparse-mode - Enables sparse mode of operation. If you configure sparse mode, you must also configure an RP. 

sh ip igmp snooping 
Global IGMP Snooping configuration:
-------------------------------------------
IGMP snooping                : Enabled
IGMPv3 snooping (minimal)    : Enabled
Report suppression           : Enabled
TCN solicit query            : Disabled
TCN flood PortFast           : Disabled
TCN flood query count        : 2
Robustness variable          : 2
Last member query count      : 2
Last member query interval   : 1000

Vlan 10:
--------
IGMP snooping                       : Enabled
IGMPv2 immediate leave              : Disabled
Multicast router learning mode      : pim-dvmrp
CGMP interoperability mode          : IGMP_ONLY
Robustness variable                 : 2
Last member query count             : 2
Last member query interval          : 1000

Vlan 46:
--------
IGMP snooping                       : Enabled
IGMPv2 immediate leave              : Disabled
Multicast router learning mode      : pim-dvmrp
CGMP interoperability mode          : IGMP_ONLY
Robustness variable                 : 2
Last member query count             : 2
Last member query interval          : 1000


References