How to backup live VMs in Xenserver free

Ever wanted to do full backups of your vms running in xenserver free edition?
Sure you can export them as backup, but in order to do so you would need to power them down.
After some digging around in the command and reading several forum threads etc…
Now I’ve been using this method for quite a while and it has proven working and successful.

First of all we will need to make the USB disk useable to the system, in order to do so connect the disk to the server and log on to the server using the local console or remote shell

To view the disks connected to the system enter
fdisk -l (yes that’s an L)

Then see if your disk is listed , probably something like /dev/sda(but could be sdb/c/etc…)

Now we must partition the disk.
fdisk -l (yes L) /dev/sda
if there are any partitions , delete them by pressing d

Now press n for new partition, then p for primary
Then press w to save changes

So now we must format the partition so there is a valid filesystem for the system to use.
mkfs.ext3 /dev/sda1  (this may take a while depending on the disk size)

Now that we have a valid and useable disk to perform our backups on, it’s time to make the disk available to the system.

to do so:
mkdir /mnt/<xenbackupfoldernameyoulike>
mount /dev/sda1 /mnt/<xednbackupfolderyoucreated>
From here on the disk is ready to be used, so we’ll skip to the fun part, the actual backups.

To fully backup(export) a live VM we need to snapshot–>convert snapshot to “vm”–>export the “vm” –>cleanup

1 Snapshot the machine, this can be done by the shell or xencenter, whatever you prefer.

The following is all SHELL!!!

2 Identify the snapshot you just made:
xe snapshot-list
this will give you a list of snapshots, note the UUID for the needed snapshot.

3 Marking the snapshot as a “vm”
xe snapshot-param-set is-a-template=false uuid=<UUID of the snapshot>

4 Exporting the vm
xe vm-export vm=<name of the snapshot you converted> filename=/mnt/xenbackupfolder/filenameyoulike.xva

The export will take a while, depending on the size of the vm.
You can monitor the progress in xencenter by selecting your xen host in the left pane and then clicking the “log” tab.

after the export is complete you may delete the snapshot.

How to recover deleted or corrupt VMX file

In case you find yourself in the position where you have a vmware virtual machine, that for some reason lost it’s configuration file (.vmx), but you still have the vmdk files which go with the vm. Then you’re in luck , there is an easy way to recover the vm!!

Just start the “new vm” wizard and during the wizard, click edit hardware.Now delete the pre-added disk from your new vm, and click add disk.
Then choose existing disk. And browse your datastore for the vmdk from the broken vm.
Do this for all the disks that were a part of the vm.
Now just boot your newly created vm, and you will have your machine back!!!!

Outlook certificate error on hosted exchange

When you want to deploy hosted Exchange/multi domain Exchange, there are several things to take in consideration.
One of the things is certificates. To be honest the only thing needed is that your certificate is valid for YOUR domain, so not the client’s. When that is the case the client will be able to connect using outlook anywhere(rpc over http) however , for the clients to be able to use the “out of office assistant” and other funtions. You need to set up an autodiscover record for THEIR domain on their dns servers.
You’d think that you’ll just create an A record of Cname pointing autodiscover.clientmaildomain.com to your mailserver, no it’s not that easy.

If you do this, then autodiscover will work, as well as all the OOOF functions etc… But your clients will receive an annoying certificate mismatch popup everytime they start outlook!

This can’t be helped, since the client will lookup autodiscover.clientmaildomain.com but will be retargeted to autodiscover.YOURdomain.com , this is why there is a certificate mismatch.
Installing the certificate won’t help and nothing else will.

The only VALID solution to this, is to make an SRV record on the client’s dns as follows:

Service: _autodiscover
Protocol: _tcp
Name: @
Priority: 0
Weight: 0
Port: 443
Target: autodiscover.YOURDOMAIN.com

Then remove all the A and CNAME records for autodiscover!

Bootmgr is missing

When you come across the error “Bootmgr is missing” in windows 7/Server 2008.
the answer is:
Load a windows cd(7/2008) for the apropriate OS and boot it, then choose “reapair my computer” , then choose command promt.(in case of HW raid, load the proper drivers first)
Type: cd recovery
Then type: startrep.exe
If you’re lucky this will solve your problems automaticaly.

In my case it didn’t so i had to do it manually, first of all when in the commandpromt enter:
bootrec /fixmbr
if this doesn’t resolve it, make sure the right partition/disk is set as “Active”.
it has to be the small partition that goes before your main drive(C:)

go back to the command prompt and enter diskpart
now enter list volume
look for the said volume (it’s about 100mb)
select volume
type: Active

then make sure none of the others are marked active, select them by using
select volume “.
Then type: Inactive

now you shoud be able to boot without problems.

Switch full screen of Terminal server client in ubuntu on VMWare Workstation/Player/server

A stupid thing but could be frustrating:
When you want to switch out of fullscreen in ubuntu’s terminal server client you have to enter:
CTRL-ALT-ENTER

but doing this in an ubuntu VM will result in full screen switching of vmware(the vm) and thus not the terminal server client running in the ubuntu vm.

to solve this enter
CTRL-ALT-SPACE this will passthrough all key combinations to the vm instead of applying them to vmware ws/server/player.

after this you can enter ctrl-alt-enter to switch full screen in TS client.

Recovering phantom VMrs

Ok this will be a post about something that had me baffled for 3 days…
We had an esxi server with 2 live vm’s which were a windows 2008r2 DC and a 2008r2 Exchange 2010 server.
Someone at work somehow managed to remove both the vm’s vmdk,vmx and snapshots.
So you would think this was a genuine eipc fail. The vms were no longer listed in vsphere, and in the datastore there were only Flat.vmdks(luckily!) and some delta files.

However upon investigation it turned out that the vms were still running???!!!
They were phantom vm’s. the domain was still functioning and so was exchange, but there was nothing to see in esxi.

You don’t need to be a rocketscientist to realize that this is both good and bad , if the vm is restarted for any reason then the vm and domain is toast. But it did give me the chance to think of the best recovery option. Cloning was not an option since it wasn’t listed in vsphere, nor was VeeAm or anything else.Also there were about 4 snapshots per machine, which were also deleted as a result so was the snapshot chain.

So here’s how i did it.
I’ve tested 2 options and both were successfull.

Option 1 = Take backup with backup exec, or any other decent software. Then recreate vmdk(descriptor file) and recreate vm , then restore the backup taken with BE. (Did this with the 2008 DC)

Option 2 = do a V2V conversion of the VM resulting in a new vm, with all your settings saved. (Did this with the Exchange Server 2010)

Option 1:

TAKE A BACKUP FIRST OF ALL YOUR NEEDED FILES!!! there is no guarantee that you will have a working vm after this!!

-Log into the ESX using ssh/console
-CD into the right folder (/vmfs/volumes/yourdatastore/yourphantomVM)
-to recreate a vmdk( disk descriptor file):
first you need to have the size of the FLAT file in bytes. You can check this by issueing: “du ” this will output the size in bytes.
-then enter this command to create the files:
vmkfstools -c -youramountofbytes- -a lsilogic -d thin temp.vmdk
This will create a TEMP.vmdk disk , with an lsilogic controller (this is the default one, but in your case could be buslogic or anything else, please check.)
And it will mark the disk as thin to save space. if your FLAT.vmdk is thick provisioned, then you will need to change this later.
-Now we have the needed .vmdk file (temp.vmdk) so now we can remove the temp-flat.vmdk since we don’t need it:
rm temp-flat.vmdk
-Next we need to rename the new temp.vmdk to the same name as your FLAT file.
to do so issue: “mv temp.vmdk -yourfilenamereflectingtheFLATfile-.vmdk
-Now we need to edit the descriptor file to point to the right disk.
vi yourfile.vmdk
-In case you’re not familiar with vi editor, you need to press the “insert” button to be able to edit the file and escape to exit edit mode. to save your chanes enter “:w” to exit VI enter “:q“.
-Now edit the file so it resembles:

# Disk DescriptorFile
version=1
CID=fb183c20
parentCID=ffffffff
createType=”vmfs”

# Extent description
RW 8388608 VMFS “YOUR FLAT FILE.vmdk”

# The Disk Data Base
#DDB

ddb.virtualHWVersion = “4″
ddb.geometry.cylinders = “522″
ddb.geometry.heads = “255″
ddb.geometry.sectors = “63″
ddb.adapterType = “lsilogic”
ddb.thinProvisioned = “1″

-NOTE THE ddb.thinProvisioned = “1″ LINE, IF THE DISK WAS THICK PROVISIONED, THEN CHANGE THIS TO 0!!

-Now we have our new disk descriptor file, we can recreate the VM.
-Knowing you have a backup of the phantom, you can power it down.
-Log into vsphere/vcenter and create a new vm, before completion choose to edit the hardware.
-Remove the disk attached to the vm and click “add” add a new hard disk and browse to the new .vmdk you made.
-Hopefully this will boot your vm without any problems and all is well.
-If you have the same issue I had , which was you realize you could only recover the Base disk and see that the VM is in a state it was before any snapshot made. This is because the snapshots are toast as well. Therefor I hope you listened and made a backup using Backup Exec or something so you can restore the AD/system state etc.
-Now you only need to clean up a bit, because now you will have a new VM with disks residing in another folder etc, so move them to the new folder (carefully!)

Option 2:

AGAIN, TAKE A FULL BACKUP !!!!!

-Remote desktop to the vm, or use another pc for the conversion
-Download and install VMware converter stand anlone client.
-Follow the wizard, choose “convert this powered on machine
-Choose your ESX as destination
-Select the apropriate options for the VM to be created (default is fine normal), only check the disks, and mark them as “Thin” if needed.
-Then run the converter.
-When it is done, log into your esx and you will see a fresh VM.
edit the vm’s settings and disconnect the network adapter(to prevent conflicts)
-Boot the VM, and check everything using the Vsphere console. When all looks swell, you can power off your Phantom VM and start using the new one!

BES Express Server stops working after installing outlook

Last I came across an issue which looking back at it was obvious enough, but not at the time:
I needed to export some mailboxes to pst files on exchange 2010 NOT SP1 using the EMS (Shell), so as you might (not) know you need Outlook x64 installed on the server where you want to do the export from.

No problem, I just installed outlook and proceeded with my exports. But then it turbed out that my BES Express which was on the same server stopped working. The BES Services wouldn’t start anymore.
So after some digging in the event logs it turns out that outlook overwrites the mapi32.dll file that you installed as a BES Prerequisite (MAPI CDO 1.2.1 pack).

The solution:

-Uninstall outlook , then reboot the server.
-Then go to control panel –> Add/Remove programs –> select the MAPI/CDO pack from the list and remove it.
-Download the pack again from here
-Reinstall the pack
-That’s it, the BES Services should start now and you should be up and running again.

Blackberry Administration Service keeps stopping with event ID 7031

I just had a BES Server where the Administration service – Application service would keep starting and then stopping.
Turns out that the service was competing with M$ DNS Server for the ports.
To fix this open the registry editor and navigate to:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

-Right-click ReservedPorts, and then click Modify.

Paste these ports in the list:
48855-48858
45588-45588
49955-49955
45599-45599

How to reset HP ILO password

Here’s how to easily reset the HP Integrated Lights Out password without needing to access the console.(tested only with ILO2!!)
Please note this will only work if the server is online and if the hp management tools are installed in the OS. If they are not then refer to this:

1. Install SNMP on the windows server (neede by HP Insight Management Agents.)
2. Ddowloaded and install the HP Insight Management Agents.
3. Download and install HP Lights-Out Online Configuration Utility.

Ok now that everything is in place, open a notepad file and enter the following :

<ribcl VERSION="2.0">
<login USER_LOGIN="Administrator" PASSWORD="somerandompassword">
<user_INFO MODE="write">
<mod_USER USER_LOGIN="Administrator">
<password value="yournewpassword"/>
</mod_USER>
</user_INFO>
</login>
</ribcl>

Now save the file as yourfile.XML , and copy it to your hp\hponcfg directory (normally under prog. files)
When done that open a command prompt , navigate to c:\program files\hp\hponcfg directory and issue the following command:

HPONCFG.exe /f yourfile.xml
Now it should successfully reset the password, but please note that the min. password length is 8 characters.


V-79-57344-34108 – An unexpected error occurred when cleaning up snapshot volumes. Confirm that all snapped volumes are correctly resynchronized with the original volumes.

Last week I had another interesting issue with backup exec.
As you can see from the title the job ended with a :

V-79-57344-34108 – An unexpected error occurred when cleaning up snapshot volumes. Confirm that all snapped volumes are correctly resynchronized with the original volumes.

and another :

V-79-57344-65033 – Directory not found. Cannot backup directory D:\HOMEDIRS\BLA\Desktop and its subdirectories.

Needless to say that the directory ofcourse was available and backup ran fine like forever until then…

But first a little scenario of the problem:

-ESX 4 Host
-Windows 2008 R2 SP1 with shares on drive “D:”
-Backup of C: has no problem what so ever
-VSS errors filling up event log during backup of D:

So to isolate the problem I had checked every goddamn Symantec KB, every MS KB… But all to no avail.
When checking out the VCenter , nothing unusual!

But then when I went clueless, and just for the fun of it connected directly to the ESX Host instead of Vcenter, I saw a Snapshot task of the said VM running and hung at 95% for like 1 week already!!(from the backup exec VMWare backup)

So I cancelled this snapshot, but still not any luck with my D: Disk.

Then my collegue spotted something I didn’t see, for some reason we had 2 VSS providers:
-Backup Exec VSS Provider
-BEVSS Provider

As it turns out, when making backups of whole VMs, Backup exec generates an extra VSS Provider (BEVSS Provider) in the services console of windows, and when backups are complete removes it again. But because the snapshot hung, the service didn’t get removed and somehow backupexec messed up!

Now to solve this:

There is a script that get’s called after a VM Backup Job ( Post-Thawte-script.bat)
located in “c:\windows”

Run this script manually and reboot, this should fix the issue and remove the extra VSS Provider!!

more info here: http://www.symantec.com/connect/forums/backupexec-2010-bevss-provider-installs-inself