How to recover a VM hung in the ESX Farm

Posted on September 24th, 2008 in Virtualization | 4 Comments »

Today, I experience with a VM hung which is not able to be reset, power off or remove from the ESX farm in our production environment. During the troubleshooting, of course we start from the virtual center, which didn’t work. Follow by that, I start to using the vmware-cmd command to do a stop & stop hard, and it still do not work. At the same time, I also restarted the management service from ESX host. Once I had done that, the VM show poweroff, but in the esxtop, it will still show the VM are running. I try to register the vm to another host, and issue to power on, but it failed, due to the resources hold up by the ESX host for the specify problem VM.

Read more »

Running SuSE Linux Cluster on VMware

Posted on September 12th, 2008 in Operating Systems, Tips | 14 Comments »

What? Running SuSE Linux Cluster on VMware ESX? Basically if you have VMware HA and DRS enable, your virtual machine is running on cluster mode. Then, why running Linux cluster on VMware? Who care? It’s just for fun and for personal testing purpose only.

1st, create two SuSE Linux(I’m running SLES 10 SP2) on VMware running on same network said cluster1(192.168.1.1) and cluster2(192.168.1.2).

2nd, add new hard disk on VMware infrastructure client. Please remember to choose ‘Use an existing virtual disk‘.

VMware
Click here for bigger image.

3rd, install and configure Linux cluster heartbeat.

Make sure you have heartbeat RPM install
cluster1:~ # rpm -qa|grep heartbeat
sles-heartbeat_en-10.1-0.20
heartbeat-pils-2.1.3-0.9
heartbeat-stonith-2.1.3-0.9
yast2-heartbeat-2.13.13-0.3
heartbeat-2.1.3-0.9

cluster2:~ # rpm -qa|grep heartbeat
sles-heartbeat_en-10.1-0.20
heartbeat-pils-2.1.3-0.9
heartbeat-stonith-2.1.3-0.9
yast2-heartbeat-2.13.13-0.3
heartbeat-2.1.3-0.9

Install Linux cluster heartbeat RPM package if you haven’t install
cluster1:~ # rpm -ivh heartbeat-xxxxxx
cluster1:~ # rpm -ivh heartbeat-stonith-xxxxxx
and etc

Edit Authentication file (same for cluster1 and cluster2)
cluster1:~ #vi /etc/ha.d/authkeys
auth 2
2 sha1 MVM_CLUS2!

Read more »

Tips: VM Network Problem

Posted on September 10th, 2008 in Tips | No Comments »

For some reason, when you require to clone or deploy a new VM guest using a template from one ESX host to another ESX host that having different network setup, you may have network problem.

Said for example, I have a SuSE Linux VM template sitting on ESX 1 which is only using “VMNetwork1″ network setting. Whenever I clone or deploy this template to ESX 2 which is only using “VMNetwork2″ network setting, I’m sure you will getting annoying error when power on the virtual machine.

VM network
You can’t change using “Edit Settings”.

Read more »

ESX & VM Networking Concepts

Posted on September 4th, 2008 in Data Center, Virtualization | 4 Comments »

This topic is specifically talk about the networking concept in VM infrastructure. In most of the cases we discuss and Virtualization and Consolidation, we always thought about number of servers we reduce in the data center, the powers we save as well as others facts. Some how, I could see most of the users today who may had already in the production for little while, and will start realize about some performance issues from the network, storage and servers perspective. WIth that particular challenges and reason, we start to hear these groups are trying to tell the customers or users, if you do want to run it on VM, it meant there is performance scarification.

I am strongly disagreed with these comments as most of us know that the reason of consolidate and virtualization, is not for performance reduce, is really to improve efficiency and utilization of the hardware that we purchased. Networking play a big parts in VM infrastructure and most of the time, it did become the performance bottle neck for most users. Let me talk about some example below.

1 of the case i saw here, which the engineer configure it’s ESX in to a server which only have 2 physical NIC connected for VMnetwork interface to allow VM to be connected to the production network. There is more than 10 VMs on the ESX servers which connected to the 2 gigabits NIC and share among each others. In physical environment, for 10 physical servers, they always get 1gb per servers and is not in shared condition. But now, since virtualized, they need to share 2gb with 10 VM. Guess what, the users start complaining slow performance on the network file transfer, the backup through Network is slow as well as any reason that the network slowing down due to the high peak bandwidth utilization from any of the VM which shared on the 2 NICs. Not only the NIC causing the performance issues, and the switches uplink to the DC switches had also another thing you may need to keep an eye on. No matter how many gigabits connection you connected your servers through, it will still depend of the total uplink for your switches to route the traffic to the DC.

In this case, they not really figure out the performance issues as they won’t notice this performance bottleneck in the performance chart from Virtual Center. Most of the time, only the network guys will able to identified these issues. It really hit hard to some of the engineer which push hard on the virtualization, but it did become performance sacrification to the customers at the end. I will not want this to happen for myself, as we had invested SAN storage & High capacity servers which is not cheap solution.

Read more »

Live Clone on Virtual Center 2.5 update 2

Posted on September 4th, 2008 in Virtualization | 1 Comment »

With the latest release from VMware on the virtual center, it does provide the LIVE CLONE features which previously I had also suggested to the local VMware representative. What is Live Clone, Live clone meants you can now clone the VM in online status.

This had been really impressed and simplify the day to day task for our operation perspective. We know the snapshot manager is the option do help when we apply changes to the VM, but some how for critical machine, we do need to have clone another set of VM in case for roll back purpose.

Read more »