Possible Bug: ARP Flux with HA/VMWare

Hi Guys,

I have a hot standby HA setup in VMware ESXi. The topology is as follows (sorry for the crude diagram)

Basically I have 2 ESXi hosts, each host has many guests [We're mainly interested in two ASGs and a couple Ubuntu boxes]. The HA link is a direct physical connection and HA itself is stable.

The problem is that when we turn on the firewall2 node (on ESX2) all guests on esx2 can no longer communicate with the ASG pair once it becomes Active (as seen with the hs command) and therefore can't get to the Internet. All LAN communication is fine, though - even between esx hosts.

To hammer home the ARP Flux thought, look to section 2.1.4 here. This is very close to the issue that I see. The only difference is that after the first reply to arping, the ASG is sending a single reply with eth4's MAC (instead of eth0). I find this odd.

Things that I've tried:

1. disabled virtual MAC addressing via "cc set ha advanced virtual_mac 0"
2. set arp_ignore to 1 via sysctl
3. set arp_announce to 2 via sysctl
4. set arp_filter to 1 via sysctl
5. enabled proxy ARP (hey why not)

Here is what I've still been seeing as symptoms:

1. With firewall2 down, I get 3 ARP replies from firewall1, the first reply is correct, each subsequent reply is incorrect
2. Ubuntu on esx2 has full connectivity to all nodes and the Internet
3. once firewall2 is active, connectivity to esx1 is normal, but esx2 is down
4. There are still 3 replies from firewall1, but the first MAC is now incorrect
5. Ubuntu on esx2 now caches an incorrect MAC address for firewall1

See the bottom of this post for helpful debug output.

So the short question is How do I get firewall1 to ONLY respond to ARP requests with the MAC if target interface?


SOME COMMAND OUTPUT: