Yesterday I re-setup the VMware environment for my client. There was a weird problem we were facing on the port group from standard vswitch for VMkernel. Here is the step how I create it.

Firstly, I created a new vSwitch, which is vSwitch 1 is this case, and I created 2 extra port group which is NFS1 and NFS2 at the same time with correct ip, subnet and etc. Once I done that, I click ok to apply the configuration. Funny thing happen now, the NFS1 created was working as I can mount the datastore as I need. NFS2 was giving problem which it suppose to point to different storage unit for NFS, it could not reach the storage unit. I try to perform vmkping on the ESX host, and it could not reach the storage host.

After I troubleshoot for 30 mins, I decided to remove the NFS2 and re-create again. This round, it works. This happen to 2 of the ESX hosts we rebuilt, and I had confirmed this is the bug on the ESX 4.0 update 2. All the configuration is perform via vCenter GUI. For the additional 2 more host, I tend to create 1 VMkernel at 1 time, and this problem is gone. This bugs had cost us time to figure out, but it could harm your environment if both port group created are pointing to the same storage. You may want to recheck your redundancy configuration while you setup your environment in the future.

The proper way of doing this, you should consider to provision at least 1 physical connection back to the standard vswitch, and migrate all the management port and vmkernel port back to it from the existing DVSwitch. After that, you should disconnect the ESX host from the previous DVSwitch. Now you can remove the ESX host from the previous vCenter, and reconnect it to the new vCenter which you would like to migrate to. From here on wards, you should able to connect the ESX host to the new DVswitch you would like to after the ESX host presented to the new vCenter. Have fun!!!!

Benchmark Test Configuration
2 x VMware vSphere 4 hosts
2 x Dell PE2950 Hardware
– Intel(R) Xeon(R) CPU E5410 @ 2.33GHz
– 16GB RAM
– Qlogic QLE8042 10GbE Mercury Converged Network Adapter(CNA) – connected to PCIe 8x slot

Updated: PERC 6/i version. 6.2.0-0013 & BOIS version. 2.6.1

VMware vSphere vCenter
– running as virtual machine
– 4vCPU
– 6GB RAM
– Microsoft Windows Server 2003 Standard edition

2 x Microsoft Windows Server 2003 Standard edition virtual machines
– 4vCPU
– 4GB RAM

Software:
Netperf

MTU 9000 setup
VMware vSphere host
esxcfg-vswitch -m 9000 vSwitch2

Windows Server 2003 standard edition virtual machine
Device Manager -> Network Adapters -> VMXNET3 Ethernet Adapter -> Properties -> Advanced -> Jumbo Packet -> Jumbo 9000 and Speed / Duplex -> 10Gbps Full Duplex

Cisco Nexus 5000K – Enable MTU 9000 and Jumbo Frame

VMware vSphere hosts
name: esx05
name: esx06

Windows Server 2003 standard edition virtual machine
name: test1 (running on esx06)
name: test2 (running on esx05)

Virtual Switch
Port Group name – test262 (Connected with single vNICs detected as Intel 82598EB 10 Gigabit AF Dual Port Network) with VLAN ID 1.

Note: vSphere auto detect as ISP8432 4Gb FCoE PCI Express HBA & Intel 82598EB 10 Gigabit

And the result:

click to enlarge.

Updated: 10 July 2009
CNA connected back to back on 2 VMware vSphere servers.

Probably you may refer to VMware documentation as link below:
- 10GugE Performance
- ESX Networking Planning

Another tricky part is, I’m managed to get total of 6G out of 10G if I running 10 VMs instances with 8192 Message Size and 163840 Socket Size as suggested in documentation as above. Again, single netperf session will get about 2.7G for Linux and 1.5G for Windows either Nexus 5K nor back to back connection. And you may get better result with 1 CPU compare to 4 CPUs or 8 CPUs which I believed a limitation on netperf itself.

Updated: 27 July 2009

Windows 2008 Standard Edition with 1vCPU, E1000 vNIC and 5G single file transfer.

Windows 2008 Standard Edition with 1vCPU, E1000 vNIC and netperf.

Windows 2008 Standard Edition with 1vCPU, VMXNET3 vNIC, Internet Download Manager HTTP multiple sessions download.

Windows 2008 Data Center Edition with 1vCPU, VMXNET3 vNIC, 5G single file transfer.

Windows 2008 Data Center Edition with 8vCPU, VMXNET3 vNIC, 5G single file transfer with default TCP setup.

Windows 2008 Data Center Edition with 8vCPU, VMXNET3 vNIC, 5G single file transfer with TCP tuning enabled.

Summary:
The result are not consistent and I believed they may have some limitation on VMware or Microsoft Windows Operating System or Qlogic CNA card driver.

Updated:
Thanks to Maurizio & Craig comments. I’m fully agreed with you guys that the Cisco Nexus 5000 is not the bottleneck but other factors.