References
Best Practices for Virtual NetworkingHost requirements for link aggregation for ESXi and ESX (1001938)
Sample configuration of EtherChannel / Link Aggregation Control Protocol (LACP) with ESXi/ESX and Cisco/HP switches (1004048)
VMware KB: Understanding IP Hash load balancing
The Hardware
In this example there are two Dell servers running ESXi.
There are two Cisco 2960x switches with stacking modules but this example will work with most Cisco Catalyst switches that are configured as a stack.
Server Connections
Each server has a four built in Gb interfaces and a four port add in Gb adapter. We will use one port from each adapter for VMware management and one port from each adapter for Guest traffic. You can easily scale the example to use all eight NICs.
Two ports for VM management
- Dell Built in port 1 (vmnic0)
- Dell Add on Port far Left (vmnic7)
vSwitch0 used for Management
vmk0, vlan 54
Click on Properties… and select vSwitch, Edit.
 |
| vSwitch0 |
Click the NIC Teaming tab
Set Load Balancing to "Route based on IP hash"
Set Network Failover Detection to "Link status only"
Set Notify Switches to "Yes"
Set Fallback to "Yes"
Make sure vnmic0 and vmnic7 are the Active Adapters. You may have to select and then use the "Move Up" button if one of them is in standby.
Click Ok and select the Management Network, Edit.
Select the NIC Teaming tab
Set Load Balancing to "Route based on IP hash"
Set Network Failover Detection to "Link status only"
Set Notify Switches to "Yes"
Set Fallback to "Yes"
Make sure vnmic0 and vmnic7 are the Active Adapters. You may have to select and then use the "Move Up" button if one of them is in standby. Click Ok.
Two Ports for Guest Traffic
- Dell Built in port 4 (vmnic3)
- Dell Add on port Far right (vmnic4)
vSwitch1 used for Guest traffic
vmk1, vlan 50
Click on Properties… and select vSwitch, Edit.
Select the NIC Teaming tab
Set Load Balancing to "Route based on IP hash"
Set Network Failover Detection to "Link status only"
Set Notify Switches to "Yes"
Set Fallback to "Yes"
Make sure vnmic3 and vmnic4 are the Active Adapters. You may have to select and then use the "Move Up" button if one of them is in standby.
Click Ok and select the Management Network, Edit.
Select the NIC Teaming tab
Set Load Balancing to "Route based on IP hash"
Set Network Failover Detection to "Link status only"
Set Notify Switches to "Yes"
Set Fallback to "Yes"
Make sure vnmic3 and vmnic4 are the Active Adapters. You may have to select and then use the "Move Up" button if one of them is in standby.
Here's a look at the physical network adapters
That's it for the server.
Cisco Switch Configuration
From VMware KB 1001938
The switch must be set to perform 802.3ad link aggregation in static mode ON and the virtual switch must have its load balancing method set to Route based on IP hash.
Ensure that the participating NICs are connected to the ports configured on the same physical switch or stacked switch.
If this is a new deployment you can configure the switch to use src-dst-ip for Etherchannel and connect the ESXi servers.
But if you are adding ESXi servers to an existing switch that already has Etherchannels defined you should check the current port-channel load balancing mode before making any changes (and make a backup of the current switch configuration).
Use Show Etherchannel load-balance to see the current setting. For a 2960x the default is src-mac.
TEST#sh etherchannel load-balance
EtherChannel Load-Balancing Configuration:
src-mac
EtherChannel Load-Balancing Addresses Used Per-Protocol:
Non-IP: Source MAC address
IPv4: Source MAC address
IPv6: Source MAC address
In global configuration mode:
The switch must be set to perform 802.3ad link aggregation in static mode ON and the virtual switch must have its load balancing method set to Route based on IP hash.
Ensure that the participating NICs are connected to the ports configured on the same physical switch or stacked switch.
If this is a new deployment you can configure the switch to use src-dst-ip for Etherchannel and connect the ESXi servers.
But if you are adding ESXi servers to an existing switch that already has Etherchannels defined you should check the current port-channel load balancing mode before making any changes (and make a backup of the current switch configuration).
Use Show Etherchannel load-balance to see the current setting. For a 2960x the default is src-mac.
TEST#sh etherchannel load-balance
EtherChannel Load-Balancing Configuration:
src-mac
EtherChannel Load-Balancing Addresses Used Per-Protocol:
Non-IP: Source MAC address
IPv4: Source MAC address
IPv6: Source MAC address
You can see that the default uses the Source MAC address instead the IP address. I haven't had any problems connecting a 2960x to a 4507R+E using Etherchannel after making the change but you should be aware of what is in use before making a change.
In global configuration mode:
port-channel load-balance src-dst-ip
This sets the Etherchannel load balancing to match the ESXi vSwitch mode.
In the interface configuration we will add:
channel-group X mode on
On is used instead of Active because VMware doesn’t support negotiation.
Switch configurations
port-channel load-balance src-dst-ip
interface Port-channel1
description < Uplink to Core >
switchport mode trunk
no shut
!
interface Port-channel2
description < ESX02 management >
switchport trunk allowed vlan 54
switchport mode trunk
no shut
!
interface Port-channel3
description < ESX02 VM >
switchport mode trunk
no shut
!
interface Port-channel4
description < ESX01 management>
switchport trunk allowed vlan 54
switchport mode trunk
no shut
!
interface Port-channel5
description < ESX01 VM >
switchport mode trunk
no shut
!
interface GigabitEthernet1/0/45
description < ESX01-MG1 >
switchport trunk allowed vlan 54
switchport mode trunk
channel-group 4 mode on
!
interface GigabitEthernet1/0/46
description < ESX01-VM1 >
switchport trunk allowed vlan 50
switchport mode trunk
channel-group 5 mode on
!
interface GigabitEthernet1/0/47
description < ESX02-MG1 >
switchport trunk allowed vlan 54
switchport mode trunk
channel-group 2 mode on
!
interface GigabitEthernet1/0/48
description < ESX02-VM1 >
switchport trunk allowed vlan 50
switchport mode trunk
channel-group 3 mode on
!
interface GigabitEthernet2/0/45
description < ESX01-MG2 >
switchport trunk allowed vlan 54
switchport mode trunk
channel-group 4 mode on
!
interface GigabitEthernet2/0/46
description < ESX01-VM2 >
switchport trunk allowed vlan 50
switchport mode trunk
channel-group 5 mode on
!
interface GigabitEthernet2/0/47
description < ESX02-MG2 >
switchport trunk allowed vlan 54
switchport mode trunk
channel-group 2 mode on
!
interface GigabitEthernet2/0/48
description < ESX02-VM2 >
switchport trunk allowed vlan 50
switchport mode trunk
channel-group 3 mode on
!
Show commands
- Show Etherchannel Sum – Shows status of the port channel and each port
- Show Etherchannel load-balance – Shows load balance mode. Should be src-dst-ip.
- Show Etherchannel detail – Shows detailed information about the port channel.
Show Etherchannel Sum
TEST#Show Etherchannel Sum
Flags: D – down P - bundled in port-channel
I - stand-alone s - suspended
H - Hot-standby (LACP only)
R - Layer3 S - Layer2
U - in use f - failed to allocate aggregator
M - not in use, minimum links not met
u - unsuitable for bundling
w - waiting to be aggregated
d - default port
Number of channel-groups in use: 5
Number of aggregators: 5
| Group | Port-channel | Protocol | Ports |
|:-----: |:------------: |:--------: |------------------------- |
| 1 | Po1(SU) | LACP | Te1/0/1(P) Te2/0/1(P) |
| 2 | Po2(SU) | - | Gi1/0/47(P) Gi2/0/47(P) |
| 3 | Po3(SU) | - | Gi1/0/48(P) Gi2/0/48(P) |
| 4 | Po4(SU) | - | Gi1/0/45(P) Gi2/0/45(P) |
| 5 | Po5(SU) | - | Gi1/0/46(P) Gi2/0/46(P) |
Show Etherchannel load-balance
TEST#Show Etherchannel load-balance
EtherChannel Load-Balancing Configuration:
src-dst-ip
EtherChannel Load-Balancing Addresses Used Per-Protocol:
Non-IP: Source XOR Destination MAC address
IPv4: Source XOR Destination IP address
IPv6: Source XOR Destination IP address
Test the load balancing
test etherchannel load-balance interface port-channel 3 <source IP> <dest IP>
TEST#test etherchannel load-balance interface port-channel 3 ip 10.26.50.100 10.26.50.101
Would select Gi1/0/48 of Po3
TEST#
TEST#test etherchannel load-balance interface port-channel 3 ip 10.26.50.100 10.26.54.102
Would select Gi2/0/47 of Po3
TEST#
Trouble Shooting
The reference link "IP Hash Load Balancing" lists the following under "Disadvantages" of using Port-channels:Beacon probing is not supported with IP Hash. Only link status can be used as a failure detection method. If a link fails without the link state going down, there is no way to avoid network communication issues on the vSwitch.
This is a serious problem if one of the links is UP but the protocol is down. Symptoms are some servers are reachable and some aren't. The first time you run into this it can be a challenge to figure out, especially if you are remote and depending on someone on site to give you information. Keep this in the back of your mind.
