Manual commit snapshots delta file to vmdk flat file
Posted by craig 
- on March 4th, 2009 in Tips, Virtualization | 15 Comments »
I had a tough time this week to deal with the snapshot issue with one of the VM. The VM is containing an important snapshot that previously taken for system restoration. When I browsed through the snapshot manager from vCenter, the system show my VM was running without any snapshots. Here was the kicked start of my problem and excited journey until I managed to recover it this morning.
I tried to SSH to the ESX host and browse to the specified datastore, and I found the snapshot file which end with file extension .vmsn were available in the correct location. No matter how many times I tried and rebooted my Virtual center, the snapshot were not visible to the snapshot manager still.
I read through some articles and forums which suggested to clone the snapshot by using vmkfstools -i option, but it didn’t success in my case here, and I continue my research and here I found a useful blog post from 1 of the blogger Oliver O’Boyle who experienced similar issue previously.
After I read through his article, which explained the chain within the CID and parent CID, it does help me to resolve my issues. I found that the root cause of my VM was due to the snapshot problem & vmdk config file corruption. For snapshot issues, we can recreate a new snapshots and we select to delete all snapshot afterward, it should force the vmdk flat files and delta files to be committed. In 1 of the virtual hard disk, we experience difficulty as the ESX servers will force the virtual HDD to be detached from the VM. The root cause of that was caused by the file missing on the parent file which should be VMxxxx.vmdk.
During this troubleshooting, you should ensure that the delta files and flat files are always retained and not overwritten. There are 2 delta files which end with VMxxxxx-000001.vmdk and VMxxxxx-000001-delta.vmdk. Your flat file should end with VMxxxxx-flat.vmdk. The 1st thing I did, was to ensure the virtual disk was able to re-attached the vm. I had manually created a new vmdk config file follow the guide from the Oliver O’Boyle, and I copy the parent CID and virtual disk value number require. I had manually configured the link within .vmdk and the flat file. After that, I was able to attach the virtual disk back to the VM from virtual center. Please take note that the virtual center will not see the flat files as the attachable virtual disk, as vCenter recognize the virtual disk base on the location of .vmdk. Recommended to keep the .vmdk and flat file within same datastore. You can also relocate the vmdk files to different datastore if you wish to do so.
Once the virtual disk had been attached to the VM, boot up the VM immediately. Please log in to the system and ensure everything is in normal and functioning correctly. The data I contained now, wasn’t the latest data I needed as the result of the missing snapshot which was not committed by the system. Now, I take a new snapshot for my entire VM. Once I had done that, datastore in SSH showed up with plenty of delta files and newly created VMDK files which end with VMxxxxxxx-000003.vmdk and so on.
Here are the steps been taken to commit the snapshots manually
- Power off the VM
- Right click the VM and select edit settings from vCenter and select the virtual disk that you are trying to recover. The system will show which vmdk files this virtual disk is pointing to
- Copy down the file names and go back to your SSH screen
- Replace the VMDK and delta files that you previous retain from your original snapshots which you are recovering with the FILE NAMES that you copy on step 2
- Open up the snapshot manager for the VM, and select delete all snapshots option. This process will take time as it depend the size of your delta files require to be committed.
- It should stuck at 95 % or time out, but the system will still continue to commit the delta files back to the flat files. In my case, it took more than 2 hours to delete the snapshot
- I noticed the ESX server load and disks activity increased from the performance chart
- Once it completed, all the delta files will be deleted and everything should be back to normal
- Power on the VM and double check all the data and mount point and I found the system was back to normal
15 Responses
I’ve got absolutely no idea what you just said, but thanks anyway.
Hi VMguy, this post is explaining the manual way you can recover a snapshot delta file and force it to be committed to the flat file. In normal case, when you remove the snapshot from snapshot manager, the system will commit the delta and emerge back with the flat file. Delta file is appear after the snapshot happen, as it contains the changes of the virtual disk after the snapshot.
In my case, my snapshots disappear from the snapshot manager, and 1 of my virtual disk had been forced to detach from the VM. To reattach the virtual disk with the latest data in there, I was required to merge the delta file with the flat file, and a rebuilt on the .vmdk file was required.
[...] process. So you might just need this link to recreate the vmdk. You may find yourself in need of this process to commit the snapshot as well. Or, if you’re really lucky, you’ll find yourself with a [...]
What command did you use to merge delta disks to base disk ? What order do we to follow ? (Tail to Head or Head to Tail)
The method I used is to delete all the snapshot after I rebuilt the chain, therefore, I do not have any command to do so. The method you need to rebuild should be tail to head. Once the chain rebuilt and you select to delete all snapshots, it will force to commit all the delta file back to single -flat.vmdk
Check out this KB @ http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1003743
Ya….so it kinda worked. I think what he was trying to say was:
- Verify you have the two files associated with the original snapshot (computer-000001-delta.vmdk and computer-000001.vmdk)
or more if you have multiple snapshots as in my case, I had 2 snapshots. This can be done through the console of your ESX server or if you SSH to the ESX server. (cd /vmfs/volumes/storage/computer
- Create a new snap shot or 2, 3, 4, etc….
- Ensure the VM is powered off.
- Go back to your ESX console or SSH to ESX server
- Drill down to the folder containing the config and HD files again
- delete the original .vmdk not -delta.vmdk for snapshot 1, 2, etc… ( I just renamed mine, computer-000001-original.vmdk.)
- Rename the latest snapshot .vmdk file you just took (For me computer-000003.vmdk) to the original .vmdk file name. (For me computer-000001.vmdk)
- Edit the file (computer-000001.vmdk) to point to the orignal -delta.vmdk.
- do the same for any other snapshots. (for me it was change snapshot 4′s .vmdk to snapshot 2′s….so on and so forth.)
- turn on the VM and yo should be good to go.
Just my 10c
Background (ESXi 4.1):
After deleting an unsuccessful VEEAM Snapshot I was left with a delta file that was taking up the entire disk space. This caused me not to be able to bring my VM up.
Procedure to fix:
1. Shut VM down.
2. Take snapshot (created a tiny snapshot)
3. Delete all snapshots.
All delta files got deleted and disk space released.
Cheers
cyber7
the procedure you use might work when the chain of the snapshot are not corrupted. The solution I posted here is mainly meant for those situation while the snapshot chain corrupted which you will need to rebuild manually.
But not works for me, snapshots isn’t consolidated
Now i’v got from 01 to 05 delta files.
Please, advice how to consolidate it!
ESXi 4.1u1
The concept is the same, you need to rebuild the chain follow the description in the post. .vmdk is the config file that point to the contain data flat file. Easiest way is full clone to new machine.
Dear mr Craig,
What about if i forgot how many snapshots file that i have done, and i just want to commit only the lates snapshots (ignoring all the snapshot before – i have 2 disk attach as c & d, so i must have 2 file delta and vmdk at least).
Can i just re-create 1 snapshots, and then change the original delta that will be commited with the latest snapshot, and run delete all snapshot…and will it be succeed to commit to the flat vmdk??
thanks a lot for your help
You can check in CLI how many delta file created in the datastore, that will tell you how many snapshot you had created earlier. In the scenario you mentioned, you can try to create a new snapshot and delete all snapshot after that. It should able to commit all existing snapshot. If failed, you will need to go through the method that I mentioned here to perform a manual commit on broken chain snapshot for VM level.
Thank for your answer, one more things, what about the flat vmdk was modified before i merge the delta files?? Can i still merge the delta to modified Flat files?? It is happen becoz the backup of the original was damage, and we have no chance to merge at first, but we already run the vm with only the flat vmdk, and that vmdk already access/modified by user…Could we still merge the lost data on delta file to the Flat??
It will disaster if we cannot get the data back
Thanks for your great help
to access back the existing flat vmdk, you need to rebuild the chain of the .vmdk configuration file. Please refer to my pose with external URL link, which explain in details how to rebuild the chain. Once you rebuild it, you should able to access or commit the snapshot.