ESXi vSwitch Configuration for Cisco Cross Stack Etherchannel

There are a lot of examples of how to set up an ESXi vSwitch to work with Cisco Etherchannel available on the Internet. But I still get asked quite often how to do it so I decided to document how I do it.

References

Best Practices for Virtual Networking
Host requirements for link aggregation for ESXi and ESX (1001938)
Sample configuration of EtherChannel / Link Aggregation Control Protocol (LACP) with ESXi/ESX and Cisco/HP switches (1004048)
VMware KB: Understanding IP Hash load balancing

The Hardware

In this example there are two Dell servers running ESXi.

There are two Cisco 2960x switches with stacking modules but this example will work with most Cisco Catalyst switches that are configured as a stack. 

Server Connections

Each server has a four built in Gb interfaces and a four port add in Gb adapter. We will use one port from each adapter for VMware management and one port from each adapter for Guest traffic. You can easily scale the example to use all eight NICs.

Two ports for VM management

• Dell Built in port 1 (vmnic0)
• Dell Add on Port far Left (vmnic7)

vSwitch0 used for Management

vmk0, vlan 54

Click on Properties… and select vSwitch, Edit.

vSwitch0

Click the NIC Teaming tab

Set Load Balancing to "Route based on IP hash"

Set Network Failover Detection to "Link status only"

Set Notify Switches to "Yes"

Set Fallback to "Yes"

Make sure vnmic0 and vmnic7 are the Active Adapters. You may have to select and then use the "Move Up" button if one of them is in standby.

Click Ok and select the Management Network, Edit.

Select the NIC Teaming tab

Set Load Balancing to "Route based on IP hash"

Set Network Failover Detection to "Link status only"

Set Notify Switches to "Yes"

Set Fallback to "Yes"

Make sure vnmic0 and vmnic7 are the Active Adapters. You may have to select and then use the "Move Up" button if one of them is in standby. Click Ok.

Two Ports for Guest Traffic

• Dell Built in port 4 (vmnic3)
• Dell Add on port Far right (vmnic4)

vSwitch1 used for Guest traffic

vmk1, vlan 50

Click on Properties… and select vSwitch, Edit.

Select the NIC Teaming tab

Set Load Balancing to "Route based on IP hash"

Set Network Failover Detection to "Link status only"

Set Notify Switches to "Yes"

Set Fallback to "Yes"

Make sure vnmic3 and vmnic4 are the Active Adapters. You may have to select and then use the "Move Up" button if one of them is in standby.

Click Ok and select the Management Network, Edit.

Select the NIC Teaming tab

Set Load Balancing to "Route based on IP hash"

Set Network Failover Detection to "Link status only"

Set Notify Switches to "Yes"

Set Fallback to "Yes"

Make sure vnmic3 and vmnic4 are the Active Adapters. You may have to select and then use the "Move Up" button if one of them is in standby.

Here's a look at the physical network adapters

That's it for the server. 

Cisco Switch Configuration

From VMware KB 1001938
The switch must be set to perform 802.3ad link aggregation in static mode ON and the virtual switch must have its load balancing method set to Route based on IP hash.
Ensure that the participating NICs are connected to the ports configured on the same physical switch or stacked switch.

If this is a new deployment you can configure the switch to use src-dst-ip for Etherchannel and connect the ESXi servers.

But if you are adding ESXi servers to an existing switch that already has Etherchannels defined you should check the current port-channel load balancing mode before making any changes (and make a backup of the current switch configuration).

Use Show Etherchannel load-balance to see the current setting. For a 2960x the default is src-mac.

TEST#sh etherchannel load-balance
EtherChannel Load-Balancing Configuration:
        src-mac

EtherChannel Load-Balancing Addresses Used Per-Protocol:
Non-IP: Source MAC address
  IPv4: Source MAC address
  IPv6: Source MAC address

You can see that the default uses the Source MAC address instead the IP address. I haven't had any problems connecting a 2960x to a 4507R+E using Etherchannel after making the change but you should be aware of what is in use before making a change.

In global configuration mode:

port-channel load-balance src-dst-ip

This sets the Etherchannel load balancing to match the ESXi vSwitch mode.

In the interface configuration we will add:

channel-group X mode on

On is used instead of Active because VMware doesn’t support negotiation. 

Switch configurations

port-channel load-balance src-dst-ip

interface Port-channel1

 description < Uplink to Core >

 switchport mode trunk

no shut

!

interface Port-channel2

 description < ESX02 management >

 switchport trunk allowed vlan 54

 switchport mode trunk

no shut

!

interface Port-channel3

 description < ESX02 VM >

 switchport mode trunk

no shut

!

interface Port-channel4

 description < ESX01 management>

 switchport trunk allowed vlan 54

 switchport mode trunk

 no shut

!

interface Port-channel5

 description < ESX01 VM >

switchport mode trunk

no shut

!

interface GigabitEthernet1/0/45

 description < ESX01-MG1 >

 switchport trunk allowed vlan 54

 switchport mode trunk

 channel-group 4 mode on

!

interface GigabitEthernet1/0/46

 description < ESX01-VM1 >

 switchport trunk allowed vlan 50

 switchport mode trunk

 channel-group 5 mode on

!

interface GigabitEthernet1/0/47

 description < ESX02-MG1 >

 switchport trunk allowed vlan 54

 switchport mode trunk

 channel-group 2 mode on

!

interface GigabitEthernet1/0/48

 description < ESX02-VM1 >

 switchport trunk allowed vlan 50

 switchport mode trunk

 channel-group 3 mode on

!

interface GigabitEthernet2/0/45

 description <  ESX01-MG2 >

 switchport trunk allowed vlan 54

 switchport mode trunk

 channel-group 4 mode on

!

interface GigabitEthernet2/0/46

 description < ESX01-VM2 >

 switchport trunk allowed vlan 50

 switchport mode trunk

 channel-group 5 mode on

!

interface GigabitEthernet2/0/47

 description <  ESX02-MG2 >

 switchport trunk allowed vlan 54

 switchport mode trunk

 channel-group 2 mode on

!

interface GigabitEthernet2/0/48

 description < ESX02-VM2 >

 switchport trunk allowed vlan 50

 switchport mode trunk

 channel-group 3 mode on

!

Show commands

• Show Etherchannel Sum – Shows status of the port channel and each port
• Show Etherchannel load-balance – Shows load balance mode. Should be src-dst-ip.
• Show Etherchannel detail – Shows detailed information about the port channel.

Show Etherchannel Sum

TEST#Show Etherchannel Sum

Flags:  D – down P - bundled in port-channel

I - stand-alone s - suspended

H - Hot-standby (LACP only)

R - Layer3      S - Layer2

U - in use      f - failed to allocate aggregator

M - not in use, minimum links not met

u - unsuitable for bundling

w - waiting to be aggregated

d - default port

Number of channel-groups in use: 5

Number of aggregators:           5

| Group | Port-channel | Protocol | Ports                   |

|:-----: |:------------: |:--------: |------------------------- |

|   1   |    Po1(SU)   |   LACP   | Te1/0/1(P)  Te2/0/1(P)   |

|   2   |    Po2(SU)   |     -     | Gi1/0/47(P) Gi2/0/47(P) |

|   3   |    Po3(SU)   |     -     | Gi1/0/48(P) Gi2/0/48(P) |

|   4   |    Po4(SU)   |     -     | Gi1/0/45(P) Gi2/0/45(P) |

|   5   |    Po5(SU)   |     -     | Gi1/0/46(P) Gi2/0/46(P) |

Show Etherchannel load-balance

TEST#Show Etherchannel load-balance

EtherChannel Load-Balancing Configuration:

        src-dst-ip

EtherChannel Load-Balancing Addresses Used Per-Protocol:

Non-IP: Source XOR Destination MAC address

  IPv4: Source XOR Destination IP address

  IPv6: Source XOR Destination IP address

Test the load balancing 

test etherchannel load-balance interface port-channel 3 <source IP> <dest IP>

TEST#test etherchannel load-balance interface port-channel 3 ip 10.26.50.100 10.26.50.101

Would select Gi1/0/48 of Po3

TEST#

TEST#test etherchannel load-balance interface port-channel 3 ip 10.26.50.100 10.26.54.102

Would select Gi2/0/47 of Po3

TEST#

Trouble Shooting

The reference link "IP Hash Load Balancing" lists the following under "Disadvantages" of using Port-channels:

Beacon probing is not supported with IP Hash. Only link status can be used as a failure detection method. If a link fails without the link state going down, there is no way to avoid network communication issues on the vSwitch.

This is a serious problem if one of the links is UP but the protocol is down. Symptoms are some servers are reachable and some aren't. The first time you run into this it can be a challenge to figure out, especially if you are remote and depending on someone on site to give you information. Keep this in the back of your mind.