My Client had a weird issue recently on 1 of the production vSphere 4.1 host. Error message shown on summary tab as ” DRS is not able to function normally cause by insufficient resource”.This is not the exact message as I can’t remember the full error message, but it represent the similar meaning of the error. This message had been alerted for more than a week as we thought the mgmt-ware restart command should able to fix this.  The system is currently in production, therefore we are suggesting to vmotion out all the production VMs, and put the host to maintenance before we issue the command.

Then we found some interesting finding here. 1 of the Windows 2000 VM was not able to perform the vmotion manually. We was quite lucky that the machine are not production yet. We powered down the VM, and relocate it to another host immediately. Suddenly the DRS error gone. we had not even put the host to maintainence to perform the mgmt-vmware restart command. It proved that the previous error was caused by just 1 single VM. Now we bring back the VM that giving problem and check on it again, the vmtools is shown updated and status OK. We tried 1 more round of vmotion, it gave us the same issue. We decided to perform a re-installation on the vmtools and retry the vmotion, and now it works like a charm.

We were quite lucky as this is not a fully production virtual machine and allow reboot to be done. If you hit to the same situation as we stated here, please try the same step as we did here and hopefully it works for you.

Related posts:

1. Major Changes on Nexus 1000V Version 1.3 to 1.4
2. VMware High Availability Behavior
3. PSOD on HP Proliant G7 with AMD 6100 Series

Well, since we had achieved this in virtual environment, what about physical environment?

Most of the server vendor out there today, they are stateful, because the information require for the system such as UUID, Bios, Firmware, MAC, WWN ID are tie down to the physical hardware. There are others vendor able to virtualize the WWN and MAC address out there but still there are challenges to remain the UUID, Bios and firmware when they move the system state from 1 physical hardware to another. Now with Cisco Service profiles technology, we able to migrate our system state without changes, without reconfiguration and the entire process could be completed within 15 to 20 mins. All you need to do is to disassociate the service profile from the blade and apply the service profiles across another available blade. This is similar with the cold migrate features we are using in Virtualize environment. Of course, it does not provide online migration as vmotion does today. Therefore, now you can have vmotion on vmware provide online migration, cold migration for offline migration on virtual machine. With Cisco UCS Service Profile features, you will able to achieve the cold migration of the physical system within short down time. It meant a lot to the system administrator who manage the 24 x 7 operation.

VMware’s Technology Day is a deep-dive seminar designed toward individuals responsible for maximizing technology infrastructure efficiency and optimization within their organization. VMware Technology Day 2010 will focus on Desktop Virtualization, Infrastructure Management & Automation and Cloud Computing.

Explore and discover the latest virtualization and cloud computing technology and solutions in this complimentary one-day event that will showcase how VMware and our partner organizations are revolutionizing the next generation of computing.

Your Opportunity

Technology Day will provide you with the opportunity to find out how to

• Transform Your Technology Infrastructure and Deliver IT as a Service
• Automate, Simplify & Control Your Desktop Infrastructure Environment
• Control & Manage Application Migration (focusing on Win 7) With Ease
• Build and Manage Security Within A Virtual Environment
• Guarantee SLA Performance Management & Delivery
• Automate Capacity Measurement, Analysis and Management
• Integrate Services For Seamless Project Plan, Design and Implementation

Who Should Attend

Technology Day 2010 has been designed for end-users, implementers, influencers and decision makers focused on Cloud, Desktop, Datacentre and Application infrastructure within their organization. Your attendance will ensure that you receive the latest and most relevant virtualization technology update to bring back to your office.

Venue:
The Westin
199 Jalan Bukit Bintang
Kuala Lumpur
Malaysia.

Date:
Thursday, 20 May 2010

Time:
8:00 am to 4:00 pm

Register Now and Win a Nitendo Wii!

I tried to select thin provision option when I perform the next virtual machine. The thin provision will be maintained as usual after the storage vMotion completed. Well, looks like there are some minor issue on the option to be selected from the migration process on vcenter and vsphere 4. Not a major issue for me, but it may confused others user.

If you plan to have VMotion over long distance, you may need to full fill the requirements as listed above.  Virtual Geek had blog it in details about the long distance vmotion supported on vmware. Feel free to take a look here.

I am able to get the ESX 3.5 to be manage by the latest vcenter. A vmotion from ESX 3.5 to vsphere 4 had been successes too, but the latest version of virtual machine which built from vSphere 4 might not compatible to vmotion back to the ESX 3.5 hosts. At the same time, if you have different processors chipset in the environment and require EVC to be turned on, it may be a little challenge to do so. You may want to ensure the EVC to be done with no down time.  You may need to refer to my previous post about how to enable EVC with no down time.

This trick is workable in the case. The new cluster created in vSphere 4 environment, may temporally disable the HA and DRS due to the incompatibility within ESX 3.5 and vSphere 4 for VMotion, which may happen on the latest virtual hardware version which provided by vSphere 4. Anyway, my test will still going on to further analyze the best upgrade method to be used to our environment due to the challenge of different region, time zone and business function we currently serve with our VMware environment.

More review will be publish after my test on this. If the success of this products is true, we should able to save some money for some of the cases which may not require big SAN box to their environment to entitle the HA, DRS and Vmotion features. Stay tune.

You can have a look about the technology from here

The 2 simple way doesn’t work and I have to proceed further with the kill -9 option by doing the ps -ef about the PID for the VM, and it show -1 as PID, which consider abnormal.

grep VMNAME /proc/vmware/vm/*/*

This command will show the PID as well.

In normal case, you can just run the command kill -9 (PID Number)

My case, it doesn’t work. The only option now to go is to run the vm-support -X (VMID)

Please take note that this process will become a pain for you. It took me more than 25 mins for the entire process, and some how, the VM are still hung.

At the end, I Vmotion all the vm to the rest of the host and rebooted the ESX, and my problem is solved and back to normal. Somehow, it has been too much time consuming to troubleshoot this as the VM is consider critical. I will not suggest to spend too much time on troubleshoot on command if we can fix the thing faster.

Here I would like to share my experience on how I get this enable without interrupt my existing production environment. By default, the features will tell you to power off all your VM before you can enable this features. Here is the tweak around solution you may want to try. Most of the time, we run critical application on VM which able to minimize the down time for us.

To have Enhanced VMotion Compatibility(EVC) working, 1st you need to create a new cluster. This cluster is not necessary to have HA/DRS running itself, because is to really allow you to move the existing production VM to the temp servers. At the same time, you can setup 1 or 2 temporally ESX Server with the evaluation edition, which have similiar configuration with your production ESX. This is to allow Vmotion happen between the ESX host.

Once you had those ready, start to VMotion all the VM out from the production ESX to the Newly build temporally ESX, until your existing production ESX cluster is empty with any of the VM power on and running.

Your cluster is now ready to configure for EVC. Right click your cluster and choose Edit Setting, and click on the VMware EVC. Select the option Enable EVC for Intel or AMD ( This will depend which processor you are using in your environment). Click Ok after that and the cluster will enable the EVC for you.

Once finish, you have EVC ready for your environment and say goodbye for processor incompatible on the Vmotion. Now you can vmotion back all the VM into this HA/DRS production cluster, and decommission your temp ESX servers you just built.

Please note that the 2 temporally ESX servers are not compatible for Vmotion before the EVC enable.

Update Manager had simplified the life of the system engineers who manage the VM farm with the huge number of VM guests and ESX hosts which may require a frequent patch update. Before the Update Manager released, most of the time we had patched the server by using satellite servers, Altiris, SMS and others patching tools. That will require additional cost required to be implemented on the VM guests or ESX host due to the licensing agreement from the vendor.

Update Manager is fully compatible with VMware ESX patches update for ESX 3.0, 3.5 and ESX 3i. From the Host level, you will able to get all the patches downloaded by the update manager schedule task once the VMware had officially release their patch on their official system. Update Manager had also integrated well with Microsoft patches as well as others famous software patches like Red Hat, Adobe and etc. It even allow us to patch the template image which we store for deployment purpose, without manual interaction to convert the template back to virtual machine. If you try to patch a windows 2003 template image, the entire process is fully automated. This is really impressive. I had also patch my DR servers which is 30 miles away from my major Data Center, and we had 30 Mb MPLS across the WAN, it worked perfectly without any issue at all, and of course, the patching timing will be slightly longer due to the location of the DR servers.

To get the update manager deployed in your environment, here are couple of steps you may need to configure before it could fully function.

A dedicated DB for update manager in the SQL or Oracle – Depend on the choice of database servers you are using. This Database will store all the information and patches to be used for patching purpose. If you have proxy server in your environment, you need to configure the proxy address and port number in the virtual center configuration for Update Manager. Schedule task to refresh and check the latest patches release from the official site, recommend to run the schedule task at least once in a week. I do schedule it to be run on weekly basis, to ensure you getting the latest patches when you try to patch you VM guest or ESX host.

Baseline – baseline is been use to define the patches required for specify product or platform by the update manager. ESX host baseline is been built in by default and categorize under Critical and Non Critical. You are also require to create you own baseline for specify OS and software you are using.

Please make sure you had update manager plug-in install on your virtual infrastructure client. To attach the baseline to the ESX or VMs you would like to deploy, you need to switch the view mode to Virtual machines and template mode, then select the system you would like to patch, and click on the update manager tab on it, and start attach the suitable baseline on it.

After you attach the baseline, right click the virtual machine or ESX host and select Scan. Scan will not actually apply the patches, this is allow the update manager to compare the current patch level for the ESX hosts and Virtual Machines and preview of the number of patches needed to be applied to be compliance. After the scan result display, right click the machine and select remmediate. This will start to apply the patches automatically.

For hosts, you need to Vmotion all the VM guests to another ESX host. This will provide 0 down time during the maintenance, thanks to the cool technology by VMware on Vmotion. This had worked for me all the time. Once the ESX hosts is ready, is recommend to send the ESX host to maintenance mode, then start the remediation after that. Once the patch is completed, it will show the ESX host at a different patch level or update code by VMware release. You can verify this with the VMware website information easily.

For VM guests patching, down time will be required as usual, due to the reboot require from the operating system perspective. Again, this tools is bundle together with the Virtual Infrastructure by VMware, is really useful for the VMware engineers to patch their VM guests.

The only disadvantage at this moment, SUSE Linux is not supported by update manager. According to the VMware, they will soon release the next version of update manager to support patch activities on SUSE Linux VMs.

# Configure Proxy setting if required

# Schedule update