For the ICM class there are 12 different lab environments with 2 virtual ESXi host each. To deploy these ESXi hosts I clone them from a master ESXi template that is preconfigured and has a local datastore with all the necessary class files and software. After I deploy each virtual machine I test connectivity and before last nights class everything seemed fine.
Once the class started accessing the lab environments we started seeing some network issues. Connections to the lab ESXi hosts were being dropped every few seconds, restarting the management networks would bring back connectivity but only for a short time. The issue turned out to be that each of the 24 hosts had the same MAC address assigned to vmk0 – oops!
The vmk0 is a virtual interface and a MAC address is generated for it when it is created. When the ESXi template is cloned the vmknic and its configuration (Name, MAC, MTU, and IP settings) are also cloned. Accessing ESXi Shell and running esxcfg-vmknic -l will list the configured vmknics and the configurations (including the MAC) of each:
There are 2 ways to correct this in the cloned ESXi hosts. One way is to delete and recreate the vmk interface. The other is to set the vmknics to use the hardware MAC address of their physical uplink NIC. The process for both of these options can be found in this kb article.
For the lab environment I set the vmknic MAC to follow the hardware MAC.
esxcfg-advcfg -s 1 /Net/FollowHardwareMac
Not sure why this issue did not show up during testing but it should be fixed now.