VMware over ISCSI storage – Equal Logic
Posted by craig
- on August 22nd, 2008 in Virtualization | 14 Comments »

Recently there had been strong grow and push for the 2nd tier ISCSI storages from the SAN Storage company like Netapps, EMC, DELL, HP, IBM and etc. A lot of the white paper and marketing brochure had been publish and been communicated to the public in worldwide, which claim the VM is out perform with the ISCSI VS Fiber Channel SAN.
I would like to share couple of my finding in the real Prove of concept and test that myself had personally setup and experience here. VMware do support hardware base ISCSI HBA and software base ISCSI through its VMkernel. Couple of thing that you may want to consider before we really decide to proceed further with the ISCSI. Networking become more and more important since the last decade to be operate most of the important pieces in the Production Data Center. Networking had also become the major issue for most of the time especially in VM environment which may need a bigger bandwidth to support the multiple VMs that been consolidated to single Physical Host.
I had been invited by the Equal Logic vendor to run the real test with the demo in my datacenter. Below is the major finding I would want to share
Test Equipment
Server – DELL PE 2950 8GB and 2 x Quad Core 2.0 Ghz
Storage Switches – DELL Gigabit Switches
Storages – Equal Logic PS 5000 with SAS HDD
Operatin System – SUSE Linux, Windows 2003, ESX 3.5
Impressive
- High Processing speed due to the processor build in for each storage bay. It can be scale up to 12 storage bay in the cluster basis. Each enclosure contains 16 physical spindle drives with SAS technology
- Simplify management – the entire process to configure the storage to be useable is less than 15 Mins. Its all web base and able to run on the open source browser such as firefox
- Maximize through put through the software ISCSI initiator been tested on servers and my personal laptop. It able to suck up 97% of the gigabit through put from my laptop gigabit connection and the server gigabit connection as well.
- High redundancy with global hotspare configuration recommended
- Impressive load balance feature which able to scale from storage and perfomance perspective. Additional enclosure will provide additional spindle power and processing power from the stroage bay. Compare to Clariion series currently, which always provide slow perfomance when the storage are only 60% populated.
- Shorter commissioning process and reduce provisioning time line. almost plug and play
- RAID 50 available as an option to be configure during the provision process
Disadvantages
- Now is time to think about long term strategy in Data Center before it become another mess as old days which we manage the Fiber Channel without director switch with fabric port. In order to gain the perfomance and scalability, each of the network connection, uplink and downstream will direct impact to the through put for each storage box. In Fiber Channel world, as if you run in 4Gb FC switches and cascaded, you will at least have 8Gb redundancy and load balance trough put from switch to switch perspective. In ethernet, we may have to look at 10Gbps uplink from switches perspective. Conclusion, the switch uplink will direct impact the entire storage throughput to the client
- As you may start to manage the ISCSI with distributed or seperate from the normal ethernet in DC, this could generate extra workload from management perspective as well as support operational overhead
- ISCSI over 10GbE are in roadmap but yet to be release, as the current gigabit ISCSI price seems to be little bit too high to be compareable with the fiber channel SAN.
- Software ISCSI had higher overhead in the Server resources compare to FC HBA storage
Summary
My opinion here is to wait for the 10GbE release on the ISCSI before to really decide for ISCSI to replace the major FC SAN Storage for High Availability purpose. As for now, it may be suitable to be deploy in Disaster Recovery Solution, Development VM Farm and Unmanage location example factory, branch office which do not have the complete Data Center solution. This is specifically for solution related to ISCSI in VMware.

14 Responses
Yes Sir,
You’re right for your info, I’d like to share my hard to believe experience in configuring my iSCSI SAN with you here:
http://img38.imageshack.us/img38/1397/deployment.jpg
Dell Power Vault MD3000i is just a small entry level SAN device which can only use one single cable to access the iSCSI target, so no matter how complex the configuration is, the I/O performance will not be as great as the adding managed switch to perform VLAN trunking.
According to the following blog:
http://virtualgeek.typepad.com/virtual_geek/2009/01/a-multivendor-post-to-help-our-mutual-iscsi-customers-using-vmware.html –> the last question #4 is the eye opener
so by using the deployment diagram that i supplied on top, i have to accept that it is not possible to achieve high performance greater than single cable connection
due to the limitation of the ESX Sofware iSCSI initiator. Even by using the Intel Pro 1000 TOE enabled pNIC it’s all the same slow result.
hope that helps you in the future,
I feel bad after spending this much money without any greater performance of my Local Server RAID-5 SATA drive (in terms of Linear Read Speed and Cached Speed / Throughput).
Had you try to have SAS Disk running on your MD 3000i?
Thanks for the reply,
Yes, this is a 15k RPM SAS
here is the performance comparison of both
Local PE 2950-III – 4×500 GB SATA RAID-5 7200 rpm
http://img193.imageshack.us/img193/3681/vmlocalsata.jpg
PV MD3000i Dual Controller – 14×300 GB SAS RAID-5 15000 rpm
http://img193.imageshack.us/img193/7605/vmsansas.jpg
and the result is so slow.
That result seems similar as what we getting on MD3000i too. But to be honest, if you are looking to higher I/O ISCSI, you will need to go for the next range of storage like FAS 2050, Equallogic PS 5000 series, HDS and etc. Those will cost you double the price to increase the performance.
In my experience, we run on FC on cx3 series EMC storage, and I able to achieve 190MB/s, but when we calculate the amount of investment, it is easily 3 to 4 times more expensive if you compare to md3000i and normal gigabits LAN. Performance always bind to amount of investment I believe
you can also try the IO meter to run your I/O test on Virtual Machine
ok, now that does sounds very great testimonial from the Storage expert like you Craig
total full performance comes with a price tag.
therefore I can confidently conclude that with the MD3000i dual controllers, only one that is actually used for data transfer (not both to boost performance).
Combined with the limitation of iSCSI software initiator from the ESXi, by default MPIO is used here for providing link redundancy without the implementation of managed switch to perform VLAN trunking (LACP)
That is all my conclusion to maximize the current investment that i’ve made.
hope this can be a helpful resources for you.
Cheers,
Albert.
I heard the vsphere had improved from the Software ISCSI initiator point of view. Should available for download sometimes next week. you may want to try it out
Hi Albert Widjaja,
you may want to read my latest post, which regarding the md3000i performance tuning, that will help your case
I have almost the same hardware config and did turn on all Jumbo frames. Since the performance tab on the VI is aggregation, I setup the following test instead:
1. install the free VM Explorer (from trilead.com), install it on a Win2003 VM.
2. copy a file from R710 local storage (which is a SAS 15k here) to MD3000i a simple 20GB LUN.
VM Explorer has a run time speed meter while copying files across ESX hosts. And it shows.. 20MB/s the max… :_(
I tried every combination I can think of like: MRU, fixed path, round robin. No improvement and sometimes degration.
I really want to know your MD3000i to ESX 4 hosts on Dell R610/710 using the on board broadcom GbE nics..
If 20Mb is normal.. I think I bought the wrong SAN then ;p
20MB.s is not normal. When you talk about jumbo frames enable, you need to ensure it turn on from the physical switch level, virtual switch level and storage controller itself too. We are using Intel gigabits NIC and broadcom mix in the case we mentioned. Beside that, when you use trilead, if I not mistaken, it will transfer the images through Network, that will depend how was your bandwidth can go from each ESX host on the LAN itself. you can try to perform a cold migration again and capture the performance from the vcenter again.
Yeah I am sure that jumbo frames are enabled on physical, virtual, and md3000i, ‘cos I can do a vmkping -s 8991 to the private network the md3000i from any of my ESX hosts.
When I conduct the test, the ESX hosts have no uplink. i.e. on the switch, there is almost zero traffic except the ESX host itself. And there is only one VM running: the ms2003 server with Trilead installed.
I also tried IOMeter. With 32k, 100% read, sequencial, IOMeter reports around 100Mb/s throughput. With 32k, 50% read 50%write, 100% random (which is more likely a file server traffic looks like), the thru’put down to like 10Mb/s.
I suspect that there is something I can tweak on the MD3000i CLI about the write buffer?
I really want to know the Trilead speed of others.. 20Mb/s copying files is nightmare…
Which raid configuration you are using?
Raid 5? or Raid 6? or Raid 10?
Raid 5 + 1hotspare.. I am kind of giving up already.. researched and tried over 50 different settings. The best cross esx hosts file transfer rate is less than 20Mb/s. I even bought a new Netgear GbE managed switch for the SAN subnet..
When you bought a switch about this, you are talking about 2 different layer of switching. 1 is for the normal ethernet switching which serve the network file transfer purpose. Another layer of switching is specified for ISCSI SAN which connected to your MD3000i. For the ISCSI switch, you will need to go for the ISCSI optimize switch, which you can consider DELL PC5424 and you need to enable all the functionality and jumbo frame. For the ESX host file transfer, it is depending the Ethernet transfer rate as well as the ISCSI SAN performance both.