Then we found some interesting finding here. 1 of the Windows 2000 VM was not able to perform the vmotion manually. We was quite lucky that the machine are not production yet. We powered down the VM, and relocate it to another host immediately. Suddenly the DRS error gone. we had not even put the host to maintainence to perform the mgmt-vmware restart command. It proved that the previous error was caused by just 1 single VM. Now we bring back the VM that giving problem and check on it again, the vmtools is shown updated and status OK. We tried 1 more round of vmotion, it gave us the same issue. We decided to perform a re-installation on the vmtools and retry the vmotion, and now it works like a charm.

We were quite lucky as this is not a fully production virtual machine and allow reboot to be done. If you hit to the same situation as we stated here, please try the same step as we did here and hopefully it works for you.

Related posts:

1. Major Changes on Nexus 1000V Version 1.3 to 1.4
2. VMware High Availability Behavior
3. PSOD on HP Proliant G7 with AMD 6100 Series

You may have 30 to 50 VMs into single host today due to high density server with more CPU core or more memory per single system. In the next second, you may face the hardware failure on 1 of the host, there will be around 50 VMs down at 1 time and require another 20 mins before all the virtual machine could be successful restarted on the surviving host. Some of them may consider this is high impact, therefore you decide to restrict the number of virtual machine in single host around 10 to 20 VM per ESX. What happen next, the TCO is high, and ROI is not efficient.  Is a tough point for most administrators to choose in this scenario. I will urge you to backward a little bit and look at the scenerio again. Before Virtualization, all the business system that only invested with standalone server without physical clustering, they do not entitle any HA as the aware off. If they want a HA in physical system, they will had to invest extra CAPEX and OPEX to maintain a same set of hardware and operating system, just for failover purpose. Again, even the operating system clustering does not provide 100% uptime.

When they adopted to Virtualization today, they do know how the HA will work, as I even personally demo to the business and explain how the HA work in ESX servers. Well, they pay less to the system cost VS Standalone servers but gain. They are happy and acceptable with the 20 mins recovery time if the host are failed due to the hardware failure on the ESX server.  Well, if 20 mins are not acceptable for them today, what about the users who even refuse to host their system in the virtual infrastructure today? There are still many of the systems are currently running on the standalone host without physical clustering. VMware had done the great job by providing ESX cluster and Virtual machine heartbeat monitoring. Users will not get this if they are still on standalone host today.

If you would like to minimize the system down time due to ESX host failure, you can always build the Microsoft cluster or Linux cluster on top of the virtual machines. Of course, this will require extra efforts to manage and maintain just like 2 physical system.  Another choice you may think of, is the Fault Tolerance function from VMware. Of course, you should always configure the policies to control the DRS activities to ensure all your same functionalities system are always split to multiple ESX hosts in the same cluster.

From my experience on deploying multi tier application system in x86 platform, I will say the application design are much critical to improve the uptime from time to time. Those intelligent application today, are able to scale dynamically from multi-tier perspective with auto failover and load balance mode enable. In the event of any application server are down, the users will be auto redirect to the available application server automatically. Well, even we had planned everything in place, there are still much more thing to be involved to prevent the single point of failure such as, SAN storage and Networking.

My opinion on high consolidation ratio in virtualization today, does not mean the risks are increasing. It is well depend on the architecture planning, consideration, design and implementation by the team.

But I do want to point out some of weakness on the solutions proposed by the deployment team or vendors that suggest to start with 1 blade chassis to reduce the CAPEX require. To achieve the fully redundancy on the virtualization, we may require NIC teaming, redundant FC connection, redundant ISCSI HBA and etc. Some of the guys may had forgotten about the chassis issues on the blade today. If I would like to run the virtualization hosts or clustering servers on blade, I will not choose to go with only single blade chassis in this case. I will not believe there is 100 % guarantee of no failure on the chassis of the blade. I will definitely suggest to split the blade across to 2 different chassis as minimum to minimize the risk. But of course, we may still have a single point of failure on the SAN which require huge amount of capex investment to keep the production storage to be redundant. Imagine if you run 5 ESX hosts on single blade chassis today, you will easily achieve 15:1 per blade for consolidation, and if you are UNLUCKY enough and your chassis just burn or malfunction for any failure, you will have 75 VMs go down at 1 time and your HA and DRS is not worked at all. You will definitely screw and whack by your business or boss in this case. If we do not have enough capex to start with 2 blade chassis for HA in the virtualization, I will suggest to stick with 1U or 2U rack mount servers to minimize the risk of the business. I am impressed with the blade technology today, but just to start with 1 single chassis for HA and redundancy purpose, will not be my call.

I am able to get the ESX 3.5 to be manage by the latest vcenter. A vmotion from ESX 3.5 to vsphere 4 had been successes too, but the latest version of virtual machine which built from vSphere 4 might not compatible to vmotion back to the ESX 3.5 hosts. At the same time, if you have different processors chipset in the environment and require EVC to be turned on, it may be a little challenge to do so. You may want to ensure the EVC to be done with no down time.  You may need to refer to my previous post about how to enable EVC with no down time.

This trick is workable in the case. The new cluster created in vSphere 4 environment, may temporally disable the HA and DRS due to the incompatibility within ESX 3.5 and vSphere 4 for VMotion, which may happen on the latest virtual hardware version which provided by vSphere 4. Anyway, my test will still going on to further analyze the best upgrade method to be used to our environment due to the challenge of different region, time zone and business function we currently serve with our VMware environment.

More review will be publish after my test on this. If the success of this products is true, we should able to save some money for some of the cases which may not require big SAN box to their environment to entitle the HA, DRS and Vmotion features. Stay tune.

You can have a look about the technology from here

Here is what I did to reconfigure my ESX host and re-join it back to the HA and DRS cluster in my production farm.

Disable the HA and DRS features from the cluster, and totally remove the ESX host from the inventory on Virtual Center server. Follow by that, I SSH in to the ESX host with su -, then I path to the /etc/init.d and look for the services mgmt-VMware status command

It show the services is running. Then I issue the command services mgmt-vmware restart. This will take couple of minutes to get the service fully restarted. At the same time I had actually Remote log on to 1 of the VM to ensure no impact on the VM guest which sit on the ESX host. The result is perfectly work without any downtime on the VM guests, and should credit to the ability from VMware technology.

Once the services restarted, you can easily add host to the virtual center and reconfigure the HA and DRS cluster mode again. The ESX host is back to normal now and work perfectly as usual.

there is an incident for myself that the HA keep disconnect and reconnect automatically for every couple of minutes. The work around is you need to make sure the hostname that you configure for your esx server is still with lower case or upper cases. If you do have mix environment with upper case and lower case in the physical cluster for ESX, you will experience this technical issue due to the way of the HA communication had been tie to the DNS name. In linux, case sensitive is always apply

For DRS, is not recommend to configure to aggressive level as during the vmotion, it do generate the overhead and network traffic for vmkernel port for both original host and destination host. 1 of the test I had done previously, I Vmotion 20 VM to another host in 1 time, and I do face the CPU bottleneck which stop the vmotion activity. Advice here is to Vmotion 1 by 1 if you are not in rush as it will generate less overhead for the vmotion activity and of course higher success rate.

The HA in ESX will only take effect when the Host is having hardware failure or Down. If the VM itself is hang, it will not able to force the VM to be restarted. Latest release virtual center come with features to monitor the status of VM and will force it to be restarted if the VM is hang.