Just did some test today on building the Windows 2008 R2 Cluster on top of VMware vSphere 4.1. This is my 1st time to configure a Microsoft cluster with windows 2008R2. Read through some best practices guide from VMWare and configure it.  Couple of key things you may wanna take note while building a Microsoft cluster on VMware.

1st is the placement for the 2 node cluster virtual machine, which you should consider split the 2 virtual machine in different host and disable the DRS or vMotion for the cluster node. Now is about the quorum disk and data disk you would like to put in the cluster. You may consider using RDM for either physical or virtual mode, and provide separate SCSI controller for the share disk. You will further require configure the SCSI controller as physical mode allow the RDM to be access by the 2nd node later.

You may not able to add the RDM to the 2nd node as you usually did in this case. You will need to browse through the existing virtual disk, to point it to the vmdk configuration files which own the RDM path from the virtual machine folders.  Once you have done this, you can configure your Microsoft cluster as usual. Not complicated, it just few important steps you need to take note here.

Referring to NTPRO Download Chrome OS VMware image post, I’m managed to get Chrome OS running on vSphere update 1 as screen below:

Please make sure you have selected Other 2.6x Linux 32bit as per your guest OS.

Download Chrome OS VMware Image at http://gdgt.com/google/chrome-os/download/

In order to recover the virtual machine, I had to remove all the snapshots on the virtual machine, but the process hang up after 3 to 4 times I tried. You may able to perform this with command and if you are lucky enough, you should able to remove the snapshot by using command. Alternatively, you can also clone a new virtual disk image contain the delta file changes to a whole new virtual disk by using vmkfstools command. This will require quite some times for you to recover the situation mostly the business will not able to wait and it is no guarantee that the recovery will fully work as we expected.

For my case, the quickest way to recover this is to clone the entire virtual machine to a new datastore that have the disk space you require and power it up immediate after the clone. We double checked the entire virtual machines mount point and services, everything require and last file transaction is available without any issue and back to production. Now I will have my sweet time to slowly investigate the failure to remove snapshots on the virtual machines. Well, is not necessary the method I am using here is the best, but I would like to share my personal experience in the critical production time, you may wanna consider the best option to recover the production virtual machine as soon as possible.

In a test yesterday, we found that a offline VM migration will auto convert the RDM which attach to the virtual machine to the VMDK format when I selected the data store to be moved in offline mode. This was really surprise myself and simplify my work actually, as I do not require some manual file transfer from the RDM to the new virtual disk.

I tried to SSH to the ESX host and browse to the specified datastore, and I found the snapshot file which end with file extension .vmsn were available in the correct location. No matter how many times I tried and rebooted my Virtual center, the snapshot were not visible to the snapshot manager still.

I read through some articles and forums which suggested to clone the snapshot by using vmkfstools -i option, but it didn’t success in my case here, and I continue my research and here I found a useful blog post from 1 of the blogger  Oliver O’Boyle who experienced similar issue previously.

After I read through his article, which explained the chain within the CID and parent CID, it does help me to resolve my issues. I found that the root cause of my VM was due to the snapshot problem & vmdk config file corruption. For snapshot issues, we can recreate a new snapshots and we select to delete all snapshot afterward, it should force the vmdk flat files and delta files to be committed. In 1 of the virtual hard disk, we experience difficulty as the ESX servers will force the virtual HDD to be detached from the VM. The root cause of that was caused by the file missing on the parent file which should be VMxxxx.vmdk.

During this troubleshooting, you should ensure that the delta files and flat files are always retained and not overwritten. There are 2 delta files which end with VMxxxxx-000001.vmdk and VMxxxxx-000001-delta.vmdk. Your flat file should end with VMxxxxx-flat.vmdk. The 1st thing I did, was to ensure the virtual disk was able to re-attached the vm. I had manually created a new vmdk config file follow the guide from the Oliver O’Boyle, and I copy the parent CID and virtual disk value number require.  I had manually configured the link within .vmdk and the flat file. After that, I was able to attach the virtual disk back to the VM from virtual center. Please take note that the virtual center will not see the flat files as the attachable virtual disk, as vCenter recognize the virtual disk base on the location of .vmdk. Recommended to keep the .vmdk and flat file within same datastore. You can also relocate the vmdk files to different datastore if you wish to do so.

Once the virtual disk had been attached to the VM, boot up the VM immediately. Please log in to the system and ensure everything is in normal and functioning correctly. The data I contained now, wasn’t the latest data I needed as the result of the missing snapshot which was not committed by the system. Now, I take a new snapshot for my entire VM. Once I had done that,  datastore in SSH showed up with plenty of delta files and newly created VMDK files which end with VMxxxxxxx-000003.vmdk and so on.

Here are the steps been taken to commit the snapshots manually

1. Power off the VM
2. Right click the VM and select edit settings from vCenter and select the virtual disk that you are trying to recover. The system will show which vmdk files this virtual disk is pointing to
3. Copy down the file names and go back to your SSH screen
4. Replace the VMDK and delta files that you previous retain from your original snapshots which you are recovering with the FILE NAMES that you copy on step 2
5. Open up the snapshot manager for the VM, and select delete all snapshots option. This process will take time as it depend the size of your delta files require to be committed.
6. It should stuck at 95 % or time out, but the system will still continue to commit the delta files back to the flat files. In my case, it took more than 2 hours to delete the snapshot
7. I noticed the ESX server load and disks activity increased from the performance chart
8. Once it completed, all the delta files will be deleted and everything should be back to normal
9. Power on the VM and double check all the data and mount point and I found the system was back to normal

Below is step by step guide with screen captured from the VM (my test machine is running on SLES10 SP2):

Tip: It is best to do a snapshot before you start this task, incase anything goes wrong.

1) This is my test VM machine. It is having one disk at beginning.

2) I am adding a 2GB volume into the VM. Click OK to continue.

3) Rescan to detect the new volume just added in VM Properties Screen.

Tip: Run following as root.

In this screen I actually run following commands:

a)# cat /proc/partitions

This will show the current volume in the server, which is the first portion in the screen capture. What we have on the server is only /dev/sda.

b)# rescan-scsi-bus.sh

This command will do a scan for all the /dev/sdx you have on the server. The new volume that we just add will get detected and show as /dev/sdb in my case.

c)# cat /proc/partitions

This is just to show the volume on the server again to confirm I got the new volume detected.

4) run fdisk to create the new Linux volume.

5.1) Format the volume into EXT3 File System.

Or you are using reiserfs? Please refer to following screen capture:

5.2) Format the volume into ReiserFS.

6) modify /etc/fstab to make sure the server will boot up with the new volume.

Modify /etc/fstab by adding a line which point /dev/sdb1 (the new volume) to moint into /sles (your mount point). The screen capture is for ReiserFS. If you are using EXT3, you may replace the “reiserfs” to “ext3” in the same line.

And, your job is done. No reboot required. Have fun.

After that, I try to check most of the setting and log file from putty and virtual center. Here is my finding. Due to some unknown reason, the .vmx file setting and the configuration on the virtual center show really different. I had to force restart the virtual center service, and reconfigure some of the resources setting in virtual center to ensure it had not reserve or limit any resources on CPU and memory, and I power up the VM again. This round, the VM able to power up and functioning. Originally, It shouldn’t reserve any resource as I had not configured that. For some unknown reason, the resources limit had been configured by the ESX itself. This had happened in the pass but it had not cause any issue.

The root cause here is clear again that the virtual center information is not tally with the information from the command. I will say this is not something new and is well known and really causing so much issue sometimes. A lesson really had been learnt today.

As for the today, more users are actually deployed the mission critical and high I/O servers on the virtualization environment, but we do see some I/O bottle neck which cause by the storage performance always. VMDK do provide flexibility from management perspective, but it does sacrifice the performance you may require for your databases, files transfers and disk performance. I had run a couple of test with real case scenerio instead of I/O meter that been always use widely, and here is the summarize result I would like to share.

In disk perfomance, we always split it to 2 categories as sequential and random I/O. in sequential mode, you will see the huge different while you try to perform the file transfer locally or through network. My test environment is running with SAN storage from fiber channel with same LUN size and raid group which are created from the Storage Level. The only differences is VMFS Vs Raw.

Raid Group design 7+1 raid 5 configuration and run on MetaLun configuration

Each LUN size is 300GB

Performance monitoring tools = Virtual Center Performance Chart

VM Test Machine = 4 Vcpu, 8GB Memory

Operating System = SLES 10 x32, x64 ; Windows Server 2003 x32, x64

Sequential : RDM is out perform VS VMFS as it able to achieve > 2 times higher through put during the file transfer locally on the VM

Random I/O : The Raw Device Mapping is still out perform the VMFS and getting the similar through put with sequential file transfer. Multi session with random database query is been executed in the test

for NFS file transfer from VMFS to VMFS, I do see the bottle neck happen much more earlier than RDM.

Highest I/O rate for RDM = 180MB/s as during sequential files copy and DB query

Highest I/O rate for VMDK = 100MB/s during sequential files copy and DB query