Veeam: The specified network resource or device is no longer available

I had a problem with Veeam in combination with Hyper-V cluster and an Equallogic SAN. All backups would run fine using storage snapshots , except for one of them. Veeam would run the job but one of the VHDs for the vm could not be backed up and Veeam would fail with the shady error:

18/11/2013 17:00:24 :: Processing ‘VM’ Error: Client error: The specified network resource or device is no longer available. Failed to read data from the file [\\?\Volume{f0c726da-445d-11e3-8b10-91d33815ae43}\VM.vhd].
Failed to read data from the file [\\?\Volume{f0c726da-445d-11e3-8b10-91d33815ae43}\VM.vhd].”Exception from server: The specified network resource or device is no longer available.

Now when looking at the Event log of the Hyper-V host which hosts the VM I would see these errors regarding VSS with IDs: 12329 &13230  during the job:

This is the 12329 error:

Volume Shadow Copy Warning: A plug-and-play devnode could not be found for the volume while attempting to restart or remove the associated device.  The function GetHiddenVolumeDevinfo failed with 0×0000000000000002. The operation will be continue, however some state associated with this volume may not be cleaned up.

And then for the 12330:

Volume Shadow Copy Warning: The hardware shadow-copy volume \\?\Volume{f0c725f5-445d-11e3-8b10-91d33815ae43} disappeared unexpectedly. Check for any event logs reported by Plug And Play, by the hardware provider, or by volsnap

This last one gives us a clue about what’s going. I got a feeling that the Equallogic was in it since it talks about HW Snapshots disappearing. And sure enough:

Here we can see that the snapshot reserved space is insufficient for the backup to run a storage snapshot. The Equallogic deletes it because of it’s policy and that’s why Windows complains it disappeared unexpectedly.

Conclusion:

You need to either extend the snapshot reserve on the san or run backup without storage snapshotting.

V-79-57344-34108 – An unexpected error occurred when cleaning up snapshot volumes. Confirm that all snapped volumes are correctly resynchronized with the original volumes.

Last week I had another interesting issue with backup exec.
As you can see from the title the job ended with a :

V-79-57344-34108 – An unexpected error occurred when cleaning up snapshot volumes. Confirm that all snapped volumes are correctly resynchronized with the original volumes.

and another :

V-79-57344-65033 – Directory not found. Cannot backup directory D:\HOMEDIRS\BLA\Desktop and its subdirectories.

Needless to say that the directory ofcourse was available and backup ran fine like forever until then…

But first a little scenario of the problem:

-ESX 4 Host
-Windows 2008 R2 SP1 with shares on drive “D:”
-Backup of C: has no problem what so ever
-VSS errors filling up event log during backup of D:

So to isolate the problem I had checked every goddamn Symantec KB, every MS KB… But all to no avail.
When checking out the VCenter , nothing unusual!

But then when I went clueless, and just for the fun of it connected directly to the ESX Host instead of Vcenter, I saw a Snapshot task of the said VM running and hung at 95% for like 1 week already!!(from the backup exec VMWare backup)

So I cancelled this snapshot, but still not any luck with my D: Disk.

Then my collegue spotted something I didn’t see, for some reason we had 2 VSS providers:
-Backup Exec VSS Provider
-BEVSS Provider

As it turns out, when making backups of whole VMs, Backup exec generates an extra VSS Provider (BEVSS Provider) in the services console of windows, and when backups are complete removes it again. But because the snapshot hung, the service didn’t get removed and somehow backupexec messed up!

Now to solve this:

There is a script that get’s called after a VM Backup Job ( Post-Thawte-script.bat)
located in “c:\windows”

Run this script manually and reboot, this should fix the issue and remove the extra VSS Provider!!

more info here: http://www.symantec.com/connect/forums/backupexec-2010-bevss-provider-installs-inself