This post is to shared 1 of the incident happened to the previous deployment. There were 2 types of Network adapters running on the physical host, which are NetXen and Broadcom in this case. For the Broadcom adapter, it come with TOE, therefore we had configured the VMkernal on the adapters to handle the IP base datastore for the ESX host.

There was a case that the ESX host become isolated and unresponsive due to the Netxen Driver failure, which we will still able to type command from the server console. While this happen, as the isolation respond option was set to leave power on, the VMware HA will not kick in to force the virtual machine for the fail over process.

There are pros and cons while doing these, but according to my Local VMware friend, the system recognize the lock session on the specified virtual machines were detected in the datastore, therefore the HA clustering will not allow the surviving systems to take over the session. The explanation fall exactly the same with our incident as the VMkernel traffic was still alive during the host network outage happen on the Netxen NICs.

I was playing around with VMware vSphere ESXi 4.1 recently and I found the warning message “HA initiated a failover action in cluster your-cluster-name in datacenter your-datacenter-name” keep show on my vCenter. Nothing has been changed, HA working like a charm, no VMs have been reboot and the warning message just annoying.

The solution are pretty simple.

Option 1:
1. Turn OFF VMware HA on Cluster Feature setting.
2. Enable VMware HA setting again.

Option 2:
1. Enter maintenance mode
2. Remove ESXi hosts from cluster
3. Add ESXi hosts back to cluster
4. Exit maintenance mode

Done! The annoying warning message will go away.

You may have 30 to 50 VMs into single host today due to high density server with more CPU core or more memory per single system. In the next second, you may face the hardware failure on 1 of the host, there will be around 50 VMs down at 1 time and require another 20 mins before all the virtual machine could be successful restarted on the surviving host. Some of them may consider this is high impact, therefore you decide to restrict the number of virtual machine in single host around 10 to 20 VM per ESX. What happen next, the TCO is high, and ROI is not efficient.  Is a tough point for most administrators to choose in this scenario. I will urge you to backward a little bit and look at the scenerio again. Before Virtualization, all the business system that only invested with standalone server without physical clustering, they do not entitle any HA as the aware off. If they want a HA in physical system, they will had to invest extra CAPEX and OPEX to maintain a same set of hardware and operating system, just for failover purpose. Again, even the operating system clustering does not provide 100% uptime.

When they adopted to Virtualization today, they do know how the HA will work, as I even personally demo to the business and explain how the HA work in ESX servers. Well, they pay less to the system cost VS Standalone servers but gain. They are happy and acceptable with the 20 mins recovery time if the host are failed due to the hardware failure on the ESX server.  Well, if 20 mins are not acceptable for them today, what about the users who even refuse to host their system in the virtual infrastructure today? There are still many of the systems are currently running on the standalone host without physical clustering. VMware had done the great job by providing ESX cluster and Virtual machine heartbeat monitoring. Users will not get this if they are still on standalone host today.

If you would like to minimize the system down time due to ESX host failure, you can always build the Microsoft cluster or Linux cluster on top of the virtual machines. Of course, this will require extra efforts to manage and maintain just like 2 physical system.  Another choice you may think of, is the Fault Tolerance function from VMware. Of course, you should always configure the policies to control the DRS activities to ensure all your same functionalities system are always split to multiple ESX hosts in the same cluster.

From my experience on deploying multi tier application system in x86 platform, I will say the application design are much critical to improve the uptime from time to time. Those intelligent application today, are able to scale dynamically from multi-tier perspective with auto failover and load balance mode enable. In the event of any application server are down, the users will be auto redirect to the available application server automatically. Well, even we had planned everything in place, there are still much more thing to be involved to prevent the single point of failure such as, SAN storage and Networking.

My opinion on high consolidation ratio in virtualization today, does not mean the risks are increasing. It is well depend on the architecture planning, consideration, design and implementation by the team.

But I do want to point out some of weakness on the solutions proposed by the deployment team or vendors that suggest to start with 1 blade chassis to reduce the CAPEX require. To achieve the fully redundancy on the virtualization, we may require NIC teaming, redundant FC connection, redundant ISCSI HBA and etc. Some of the guys may had forgotten about the chassis issues on the blade today. If I would like to run the virtualization hosts or clustering servers on blade, I will not choose to go with only single blade chassis in this case. I will not believe there is 100 % guarantee of no failure on the chassis of the blade. I will definitely suggest to split the blade across to 2 different chassis as minimum to minimize the risk. But of course, we may still have a single point of failure on the SAN which require huge amount of capex investment to keep the production storage to be redundant. Imagine if you run 5 ESX hosts on single blade chassis today, you will easily achieve 15:1 per blade for consolidation, and if you are UNLUCKY enough and your chassis just burn or malfunction for any failure, you will have 75 VMs go down at 1 time and your HA and DRS is not worked at all. You will definitely screw and whack by your business or boss in this case. If we do not have enough capex to start with 2 blade chassis for HA in the virtualization, I will suggest to stick with 1U or 2U rack mount servers to minimize the risk of the business. I am impressed with the blade technology today, but just to start with 1 single chassis for HA and redundancy purpose, will not be my call.

I am able to get the ESX 3.5 to be manage by the latest vcenter. A vmotion from ESX 3.5 to vsphere 4 had been successes too, but the latest version of virtual machine which built from vSphere 4 might not compatible to vmotion back to the ESX 3.5 hosts. At the same time, if you have different processors chipset in the environment and require EVC to be turned on, it may be a little challenge to do so. You may want to ensure the EVC to be done with no down time.  You may need to refer to my previous post about how to enable EVC with no down time.

This trick is workable in the case. The new cluster created in vSphere 4 environment, may temporally disable the HA and DRS due to the incompatibility within ESX 3.5 and vSphere 4 for VMotion, which may happen on the latest virtual hardware version which provided by vSphere 4. Anyway, my test will still going on to further analyze the best upgrade method to be used to our environment due to the challenge of different region, time zone and business function we currently serve with our VMware environment.

More review will be publish after my test on this. If the success of this products is true, we should able to save some money for some of the cases which may not require big SAN box to their environment to entitle the HA, DRS and Vmotion features. Stay tune.

You can have a look about the technology from here

• Login to Virtual Center
• Right click on your Cluster
• Click Edit Settings
• Click VMware HA
• Click ‘Enable virtual machine monitoring‘

If you don’t see any ‘VMware HA’ column available, please make sure you have VMware HA enable.

• Login to Virtual Center
• Right click on your Cluster
• Click Edit Settings
• Click General
• Click Features
• Click Enable VMware HA

Hope this is useful.

1st, create two SuSE Linux(I’m running SLES 10 SP2) on VMware running on same network said cluster1(192.168.1.1) and cluster2(192.168.1.2).

2nd, add new hard disk on VMware infrastructure client. Please remember to choose ‘Use an existing virtual disk‘.

Click here for bigger image.

3rd, install and configure Linux cluster heartbeat.

Make sure you have heartbeat RPM install
cluster1:~ # rpm -qa|grep heartbeat
sles-heartbeat_en-10.1-0.20
heartbeat-pils-2.1.3-0.9
heartbeat-stonith-2.1.3-0.9
yast2-heartbeat-2.13.13-0.3
heartbeat-2.1.3-0.9

cluster2:~ # rpm -qa|grep heartbeat
sles-heartbeat_en-10.1-0.20
heartbeat-pils-2.1.3-0.9
heartbeat-stonith-2.1.3-0.9
yast2-heartbeat-2.13.13-0.3
heartbeat-2.1.3-0.9

Install Linux cluster heartbeat RPM package if you haven’t install
cluster1:~ # rpm -ivh heartbeat-xxxxxx
cluster1:~ # rpm -ivh heartbeat-stonith-xxxxxx
and etc

Edit Authentication file (same for cluster1 and cluster2)
cluster1:~ #vi /etc/ha.d/authkeys
auth 2
2 sha1 MVM_CLUS2!

Edit HA configuration file
cluster1
cluster1:~ #vi /etc/ha.d/ha.cf
logfile /var/log/ha-log
logfacility local0
node cluster1
node cluster2
keepalive 1
warntime 30
deadtime 60
initdead 120
udpport 694
ucast eth0 192.168.1.1
ucast eth0 192.168.1.2
auto_failback off
ping_group group1 192.168.1.254
respawn hacluster /usr/lib/heartbeat/ipfail
watchdog /dev/watchdog

cluster2
cluster2:~ #vi /etc/ha.d/ha.cf
logfile /var/log/ha-log
logfacility local0
node cluster2
node cluster1
keepalive 1
warntime 30
deadtime 60
initdead 120
udpport 694
ucast eth0 192.168.1.2
ucast eth0 192.168.1.1
auto_failback off
ping_group group1 192.168.1.254
respawn hacluster /usr/lib/heartbeat/ipfail
watchdog /dev/watchdog

Edit HA Resource file (same for cluster1 and cluster2)
cluster1:~ #vi /etc/ha.d/haresources
cluster1 192.168.1.100 \
Filesystem::/dev/sdb1::/db::ext3

Make sure you can see hard disk partition on both servers
cluster1:~ # cat /proc/partitions
8 17 20964793 sdb1
cluster2:~ # cat /proc/partitions
8 17 20964793 sdb1

Configure and format hard disk partition
cluster1:~ # fdisk /dev/sdb1
press n add a new partition
press p primary partition
press 1 partition number
First cylinder, press ENTER
Last cylinder or +size or +sizeM or +sizeK, press ENTER
press p print the partition table
press w write table to disk and exit

cluster1:~ # mkfs.ext3 /dev/sdb1

Create new folder on both servers
cluster1:~ # mkdir /db
cluster2:~ # mkdir /db

Startup heartbeat services
cluster1:~ # chkconfig heartbeat on
cluster1:~ # /etc/init.d/heartbeat start
cluster2:~ # chkconfig heartbeat on
cluster2:~ # /etc/init.d/heartbeat start

You should be able to see /db mounted and IP address 192.168.1.100 is up on cluster1(active). If you turn OFF cluster1 server and you should see cluster2 running on ACTIVE mode.

Congratulations! You’re done! Enjoy!

This had been really impressed and simplify the day to day task for our operation perspective. We know the snapshot manager is the option do help when we apply changes to the VM, but some how for critical machine, we do need to have clone another set of VM in case for roll back purpose.

Previously, we had to shutdown the VM and do a cold clone, that is really time consuming and provide down time to the business. In order to persuade with the 30 mins down time is not as easier as we thought due to the business operation perspective. Now, we can do a Live Clone without shutting down the production, and that will really help us during the test environment as well as patching activity. This features is really benefit for us as we run the ESX and VMs on the High Availability environment which host critical application to our business.