Benchmark Test Configuration
2 x VMware vSphere 4 hosts
2 x Dell PE2950 Hardware
– Intel(R) Xeon(R) CPU E5410 @ 2.33GHz
– 16GB RAM
– Qlogic QLE8042 10GbE Mercury Converged Network Adapter(CNA) – connected to PCIe 8x slot

Updated: PERC 6/i version. 6.2.0-0013 & BOIS version. 2.6.1

VMware vSphere vCenter
– running as virtual machine
– 4vCPU
– 6GB RAM
– Microsoft Windows Server 2003 Standard edition

2 x Microsoft Windows Server 2003 Standard edition virtual machines
– 4vCPU
– 4GB RAM

Software:
Netperf

MTU 9000 setup
VMware vSphere host
esxcfg-vswitch -m 9000 vSwitch2

Windows Server 2003 standard edition virtual machine
Device Manager -> Network Adapters -> VMXNET3 Ethernet Adapter -> Properties -> Advanced -> Jumbo Packet -> Jumbo 9000 and Speed / Duplex -> 10Gbps Full Duplex

Cisco Nexus 5000K – Enable MTU 9000 and Jumbo Frame

VMware vSphere hosts
name: esx05
name: esx06

Windows Server 2003 standard edition virtual machine
name: test1 (running on esx06)
name: test2 (running on esx05)

Virtual Switch
Port Group name – test262 (Connected with single vNICs detected as Intel 82598EB 10 Gigabit AF Dual Port Network) with VLAN ID 1.

Note: vSphere auto detect as ISP8432 4Gb FCoE PCI Express HBA & Intel 82598EB 10 Gigabit

And the result:

click to enlarge.

Updated: 10 July 2009
CNA connected back to back on 2 VMware vSphere servers.

Probably you may refer to VMware documentation as link below:
- 10GugE Performance
- ESX Networking Planning

Another tricky part is, I’m managed to get total of 6G out of 10G if I running 10 VMs instances with 8192 Message Size and 163840 Socket Size as suggested in documentation as above. Again, single netperf session will get about 2.7G for Linux and 1.5G for Windows either Nexus 5K nor back to back connection. And you may get better result with 1 CPU compare to 4 CPUs or 8 CPUs which I believed a limitation on netperf itself.

Updated: 27 July 2009

Windows 2008 Standard Edition with 1vCPU, E1000 vNIC and 5G single file transfer.

Windows 2008 Standard Edition with 1vCPU, E1000 vNIC and netperf.

Windows 2008 Standard Edition with 1vCPU, VMXNET3 vNIC, Internet Download Manager HTTP multiple sessions download.

Windows 2008 Data Center Edition with 1vCPU, VMXNET3 vNIC, 5G single file transfer.

Windows 2008 Data Center Edition with 8vCPU, VMXNET3 vNIC, 5G single file transfer with default TCP setup.

Windows 2008 Data Center Edition with 8vCPU, VMXNET3 vNIC, 5G single file transfer with TCP tuning enabled.

Summary:
The result are not consistent and I believed they may have some limitation on VMware or Microsoft Windows Operating System or Qlogic CNA card driver.

Updated:
Thanks to Maurizio & Craig comments. I’m fully agreed with you guys that the Cisco Nexus 5000 is not the bottleneck but other factors.