One of the key area they highlighted was the stoage protocol they choosed, which is NFS rather than iscsi and FC SAN, which was mainly due to the flexibility of volume shrink or resize that could support on the fly. I am agreed with their statement as personal thought, for large scale deployment, NFS are much easier to manage compare to Iscsi and FC environment.

NFS is pure ethernet ip base, which certified and proven in many of the large scale deployments worldwide. With 10G ethernet today, performance throughput on the NFS will no longer concerns over the performance. If you had not evaluated NFS as an option in your deployment currently, you may want to take a try on it.

Firstly, I created a new vSwitch, which is vSwitch 1 is this case, and I created 2 extra port group which is NFS1 and NFS2 at the same time with correct ip, subnet and etc. Once I done that, I click ok to apply the configuration. Funny thing happen now, the NFS1 created was working as I can mount the datastore as I need. NFS2 was giving problem which it suppose to point to different storage unit for NFS, it could not reach the storage unit. I try to perform vmkping on the ESX host, and it could not reach the storage host.

After I troubleshoot for 30 mins, I decided to remove the NFS2 and re-create again. This round, it works. This happen to 2 of the ESX hosts we rebuilt, and I had confirmed this is the bug on the ESX 4.0 update 2. All the configuration is perform via vCenter GUI. For the additional 2 more host, I tend to create 1 VMkernel at 1 time, and this problem is gone. This bugs had cost us time to figure out, but it could harm your environment if both port group created are pointing to the same storage. You may want to recheck your redundancy configuration while you setup your environment in the future.

We had been running Vmware in our environment for more than 2 years now in production mode and found that 80% of our production virtual machine should easily could be handled on either NFS and ISCSI. I will suggest to have FC and IP SAN in place within the virtual infrastructure. This will allow the administrator to select the right storage option to support the IO requirement for each system been hosted in VMware. The mission critical and high IO requirements systems will be deployed on  the FC SAN, but the low IO requirement virtual machines will be migrated to IP SAN. With the storage vMotion option in vSphere 4, this will able to be done with zero down time. At cost perspective, it will make the hosting package to be more competitive to support the customer need.

Personally, I had 3 of it running in my environment now and of course, I had spent a huge number of times to really test out all the solution and compare apple to apple to further study of my requirement. Here are some reading I get from my test for 3 solution above.

For I/O performance, I am using the IO meter as to generate the max output to the storage. I manage to get 190MB/s for my FC storage on 100 % read, and I only managed to get the output of 170MB/s on ISCSI, but on NFS, I only managed to get at 125MB/s. If we look at the number here, and compare to my high load R900 ESX host, actually it is more than enough to handle the number of VM I have in the ESX host. There are a lot of article introduce NFS, and the features of Netapps, but remember, ESX itself had improved from time to time. Thin Provisioning is no longer new thing, as you may get it with many vendor today by free as well as volume snap. But Dedupe will still be an interesting technology in netapps. If we look at FC, you will mostly lose all this technology, but it provide a stable performance as you need to virtualize some heavy environment moving forward. I am currently still test out the storage solution before I decide to go with a final choice. I will suggest to keep an eye on the technology trend as we may also remember that the VMware ESX 4 is on the way, which may change the game of storage strategy significantly.

Here I would like to specify talk about ESX storage planning. As many of the cases I read and experience before, more and more users are really heading to the stage of choosing specify storage for specify solutions. Example you may think off for using Netapps on NFS for ESX, which claim better performance, dedupe and protocol different. Somehow, the users had forgotten that they had not run any of netapps storage in their existing environment. When we look at the management perspective, the operation management for IT support is not SIMPLIFY. Everyone know how important to standardize our environment as well as simplify out IT environment. The more complex you have, the more pain you will feel.

ESX had been design to run on multiple different range, different brand, as well as different type of protocol of storage in the market today. The reason of this is able to provide users a number of choices to be choose for deployment which may really fit the existing environment. Most users are now been looking at the storage specifically for Virtualization, that was really due to the marketing strategy or brochure been always claim the specify products will gain better performance. If you are the storage guys in your IT department, maybe you should consider what will be the best fit for your environment VS what is the coolest technology in the world for VMware.

Let talk about 1 scenario here, a user who had existing running with cx3-80 EMC SAN storage with all fiber switches ready. The storage is only 50% populated. The storage guys had raised the request to bring in the latest ISCSI technology from netapps and Equal logic. When it does the price comparison, the TCO and implementation cost of ISCSI become too huge different VS extending the enclosure on cx3-80. Back to real world, I am really curious that if SAN Fiber storage will be poor perform and inconsistent compare to ISCSI if you have everything setup and configure correctly in your fiber SAN. Most of the case is NO. As in my POC test, I can see that the performances on the Fiber are still better and stable.

ISCSI is really simple and easier manage, but somehow, there are many technical details that require redeploying the entire environment to gain the peak performance you expected. That is definitely an advice at the end you will get from the vendor. Therefore, a best fit storage for your own environment is more important always. You should look at CAPEX vs OPEX when you plan the storage for virtualization.

As for the today, more users are actually deployed the mission critical and high I/O servers on the virtualization environment, but we do see some I/O bottle neck which cause by the storage performance always. VMDK do provide flexibility from management perspective, but it does sacrifice the performance you may require for your databases, files transfers and disk performance. I had run a couple of test with real case scenerio instead of I/O meter that been always use widely, and here is the summarize result I would like to share.

In disk perfomance, we always split it to 2 categories as sequential and random I/O. in sequential mode, you will see the huge different while you try to perform the file transfer locally or through network. My test environment is running with SAN storage from fiber channel with same LUN size and raid group which are created from the Storage Level. The only differences is VMFS Vs Raw.

Raid Group design 7+1 raid 5 configuration and run on MetaLun configuration

Each LUN size is 300GB

Performance monitoring tools = Virtual Center Performance Chart

VM Test Machine = 4 Vcpu, 8GB Memory

Operating System = SLES 10 x32, x64 ; Windows Server 2003 x32, x64

Sequential : RDM is out perform VS VMFS as it able to achieve > 2 times higher through put during the file transfer locally on the VM

Random I/O : The Raw Device Mapping is still out perform the VMFS and getting the similar through put with sequential file transfer. Multi session with random database query is been executed in the test

for NFS file transfer from VMFS to VMFS, I do see the bottle neck happen much more earlier than RDM.

Highest I/O rate for RDM = 180MB/s as during sequential files copy and DB query

Highest I/O rate for VMDK = 100MB/s during sequential files copy and DB query