Virtual USB-disks on VMware Guests

As anyone who has tried to use a USB licensing key or test USB devices in a VMware/vSphere cluster environment knows that using a USB can be problematic. You may have dozens or hundreds of guests on multiple hosts in you data center/cluster; and trying to map a USB port to just one VM gets silly (and unsecure)

Apparently there is a VMware feature available that I didn’t know about or haven’t read about.

I cobbled the info below from multiple sources after a lot of research. Hope it helps some. If you have anything to add or need credit let me know.

See https://communities.vmware.com/thread/580985
We are used to the VMware virtual disk format (vmdk-files)
This VMDK-files can be attached to a VM so that the guestOS perceives them as:
– IDE-device
– SCSI-device
– SATA-device
– NVME-device
Until today I was not aware of the fact that there is one more option:
– USB-device

This feature is not exposed in the GUI but using it is quite easy and straight forward.
To define an existing VMDK as USB-device you have to edit the vmx-file.
First of all make sure that you have this line:
ehci.present = “TRUE”
You need this line as a main switch for USB 2 ports.
You should also see a line like
ehci.pciSlotNumber = “35”
Do not edit this line – instead simply delete it if you want to reset the port.
If you assign a “bad” port you will get obscure follow-up problems – so don’t do it.
Now to assign a VMDK as USB-device set this parameters:

ehci:0.present = “TRUE”
ehci:0.deviceType = “disk”
ehci:0.fileName = “usb-vmdk.vmdk”

ehci:1.present = “TRUE”
ehci:1.deviceType = “disk”
ehci:1.fileName = “usb2-vmdk.vmdk”

Using this appears to be possible for more than one VMDK – so it maybe possible that the full range from ehci0 – ehci5 is allowed.
This is just a first guess – I need to do more research here ….

I said that this feature is not exposed in the GUI – that is not entirely correct.
Once you created the required vmx-parameters and start the VM you will see the disks appear in the list of removable devices:
clip_image001

Inside a guest both vmdks appear like this: (using my Linux-LiveCD with Ubuntu 14)

clip_image002
During my experiments I noticed that the USB-vmdks may appear in a write protected mode.
At the moment I cant claim to have completely understood in which constellation the vmdks are write-protected.
This will require further research ….

Anyway – even at the moment I would call this an extremly useful “new” feature.

1. for all those guys that develope USB-bootable tools

If the USB-vmdks are created with the monolithicFlat VMDK-format the USB-images can be easily transferred to real USB-devices with a simple dd-command.

2. for all users who are looking for a way to assign VMDKS  as “optional”

A USB-vmdk is allowed to be temporarily unavailable !!!
All other options to assign VMDKs will fail if the file is not present.
With USB-vmdks the VM will start even if the file is not available.
This will open new paths to acchieve obscure constellations that were impossible until now.

3. for all users that missed the option to assign single-partition images as a VMDK.

All other options to assign VMDKs usually require a partitioned image including a valid MBR or GPT.

How to Access Local and USB Hard Drives In Your Hyper-V Virtual Machine

There are dozens of scenarios where you would want to access a physical hard drive from inside your Hyper-V virtual machine ranging from accessing and importing data, dumping data from the virtual machine into the external drive, and importing or exporting disc images.

Whatever reason you have for wanting to access an existing local hard drive in your Hyper-V machine it’s, unfortunately, a bit complicated giving the drives access to the virtual machine. Unlike other virtualization solutions (VMware and VirtualBox) there is no easy way to just share resources between the host machine and the Hyper-V virtual machine.

To use a drive within a Hyper-V machine you need to take the drive, while physically attached and mounted to the host machine, offline and then reroute the offline-to-the-host disk into the virtual machine. Let’s take a look at how to do that now.

A quick note before we proceed: this trick only works with hard drives (magnetic hard drives, solid-state hard drives, and removable USB hard drives) but does not work, alas, with removable media like flash drives.

Drive Preparation and Mounting In Hyper-V

There are two primary steps to the procedure. First we need to manage the disk from within Windows on the host machine to bring it offline (but still powered and attached to the host computer) and then we need to tell Hyper-V to take control of the now-offline disk.

Preparing the Drive

The first step is to bring the disk offline. To do so open up the Disk Management application by searching for the application name or typing “diskmgmt.msc” in the search box and running the app. Within Disk Management look for the drive you wish to pass through to the virtual machine.

Although it’s probably safe to assume, since you’re playing around with Hyper-V and concerned with matters like physical-disk-pass-through, that you’re already familiar with Disk Management it would be remiss of us not to issue a standard warning: Disk Management is a powerful tool and you can do terrible things to your system by mucking about within the management system. Double check each step and make sure you’re working with the right disk(s).

clip_image001

In our case we want to pass Disk 10, a USB backup HDD, through to the virtual machine so we can dump some backup files from our virtual machine onto the disk. To prepare the disk we must first locate it in the list of drives within Disk Management, right click on it, and select “Offline” from the right-click context menu as seen in the screenshot above.

clip_image002

Confirm the disk is offline. If you wish to bring the disk back online, after you’re done using it within the virtual machine, simply return to this menu, right click on the disk, and select “Online” to bring the disk back online for the host operating system.

Adding the Drive to the Virtual Machine

Once the disk is offline to the host operating system it’s time to add the disk to the virtual machine within Hyper-V. Launch Hyper-V and, from your Virtual Machines list, select the machine you wish to pass the hard disk through to.

clip_image003

Right click on the machine and select “Settings…”

clip_image004

From within the Settings menu for that specific virtual machine, select “SCSI Controller” from the left hand navigation pane. Select “Hard Drive” and click the “Add” button.

Note: We’ve presumed you wish to pass the hard drive through to the virtualized operating system which means you already have a SCSI controller and virtual disk for the OS. If you do not yet have a SCSI controller you will need to first select “Add Hardware” at the top of the navigation pane and add a “SCSI Controller” to your virtual machine.

clip_image005

Select “Physical hard disk” and then, from the drop down menu, select the disk you brought offline in the previous section of the tutorial. Don’t worry there’s no chance you’ll accidentally select a disk from the host machine that you didn’t intend to use. The only disks you can select in the physical hard disk menu are disks that are powered and attached to the host machine but in an offline state. If you didn’t put it offline it isn’t even an option.

Confirm the disk selected is the disk you want. Press the “Apply” button and then “OK”. Boot up your virtual machine.

Accessing Your Pass-through Hard Disk

In the vast majority of cases the disk should be automatically detected by the host operating system as if it was simply a physical disk attached to the real machine the virtual machine is emulating.

If the disk isn’t immediately accessible to the virtual machine operating system you may need to open up, in the case of Windows, Disk Management and bring the disk online. When using other operating systems you’ll need to perform an equivalent action to access the disk through the virtual OS.

A very important word of caution here. The OS in the virtual machine has no idea that this disk actually belongs to your host machine and it will act upon the disk however you tell it to act upon the disk. If the disk does not automatically mount in your virtual machine OS you need to use a mounting procedure (refer to documentation for the particular OS in question) to mount the disk in the non-destructive way so that it can be returned to the host operating system later on.

When you no longer need the disk in the virtual machine simply boot down the virtual machine. Do not attempt to make changes to the drive state via the host machine until you’ve turned off the virtual machine the drive is connected to.

You can outright remove the hard drive from the virtual SCSI controller in your virtual machine if you aren’t going to use it again or you can leave the entry. Any time the physical drive is disconnected from the host machine or you’ve made it online for the host operating system the drive will simply appear as unavailable to the virtual machine.

Moving Virtual Machines in ESXi from older hardware to new server.

If you have outgrown you current ESXi hardware and need to move to another physical machine and you don’t have vSphere you can still easily move all your VMs to the new server. Without additional software costs.

First install the ESXi Host software on the machine and configure it for access via the management console.

Take note of the IP address and credentials you used for you new installation.

Then login to your existing VMware installation via your management computer/console. And you can then do one of the following.

Simple way?

Use the outstanding free product(s) by VEEAM!
Veeam Backup Free Edition. The free version that will let you backup, replicate/move and restore entire machines from one ESX server to another on the same subnet. The free version limits the backup/migrate tasks to one machine at a time but it works! If you have a VMware environment I highly recommend their product(s) for backup and disaster recovery. You can automate just about everything you need for backing up and restoration. Check out more here http://www.veeam.com .

Older not so quick but still simple old fashioned way.

Open the management console. Connect to the original server/Host. Navigate to the Server (vCenter) containing the VMs you wish to move.
Commit any snapshots in the VM you are going to move.
Power off the VM.
Click the Configuration tab.
Click Storage.
Open/Browser to the Datastore where the VMs are located and find the folder containing the entire Virtual Machine (the one with the disks (vmdk), .vmx file, nvram, .log files etc.).
Copy that entire machine to a drive (a folder on the management station or other attached/network available storage)

Connect to the new host via the management console.
Open the Datastore where you wish to place your new machine(s)
Upload (import) the entire folder – containing the Virtual Machine, you previously copied/downloaded.
Then open the that new folder and right-click on the .vmx file and select ‘Add to Inventory’.
It will then show up in the list of machines in your selected Server/Cluster. Then power off all the old machines and shut down that Host.

When you power on the machines on the new Host you will be walked through a process that will ask you if the machine was copied or cloned. Since you are moving the VM choose that –  move and NOT clone or it will generate a new machine ID/SID and jack up the VMs usability.

Do that for any and all that you moved and test connectivity from workstations.

I’ve done this with virtualized Domain Controllers, SQL servers and even an Exchange server!

That’s it.

Peace out.

Installing SQL Server 2008R2 to Windows 7 VM

While installing an instance of SQL2008R2 to a Windows 7 Workstation VM (used by my programmers) on my VMWare vSphere cluster I ran into a problem.

While the SQL installation seemed to go OK, it hung up toward the end during ‘Windows Installation Final Tasks – Clean up processes…” or something at “Install_sqlncli_Cpu64_Action”
The install just hung forever – let it actually try to run over night and it never completed.

The solution is simple, and kind of a ‘duh’.
SQL2008R2 needs 4GB RAM and minimum of 4 cores/processors.
My Production SQL VM’s easily met this criteria but not so with the workstations.
So simply cancelled the installation. Shut down the VM. Edited the VM – upped RAM to 6GB and processors to 4.
Restart VM then let it boot and let Win7 reconfigure itself. It will require another reboot.
I then completely uninstalled the failed installation. Then rebooted.
Also remember that this is REALLY IMPORTANT!! – Open the ports on the Windows Firewall for SQL2008

This can be done by copying the following (everything in between the break lines including the last ‘return’ at the end) into a text file and renaming it with a .bat extension. something like ‘opensqlfirewallports.bat’ or what ever and running it with administrative permissions.
————————–
@echo =========  SQL Server Ports  ===================
@echo Enabling SQLServer default instance port 1433
netsh firewall set portopening TCP 1433 “SQLServer”
@echo Enabling Dedicated Admin Connection port 1434
netsh firewall set portopening TCP 1434 “SQL Admin Connection”
@echo Enabling conventional SQL Server Service Broker port 4022
netsh firewall set portopening TCP 4022 “SQL Service Broker”
@echo Enabling Transact-SQL Debugger/RPC port 135
netsh firewall set portopening TCP 135 “SQL Debugger/RPC”
@echo =========  Analysis Services Ports  ==============
@echo Enabling SSAS Default Instance port 2383
netsh firewall set portopening TCP 2383 “Analysis Services”
@echo Enabling SQL Server Browser Service port 2382
netsh firewall set portopening TCP 2382 “SQL Browser”
@echo =========  Misc Applications  ==============
@echo Enabling HTTP port 80
netsh firewall set portopening TCP 80 “HTTP”
@echo Enabling SSL port 443
netsh firewall set portopening TCP 443 “SSL”
@echo Enabling port for SQL Server Browser Service’s ‘Browse’ Button
netsh firewall set portopening UDP 1434 “SQL Browser”
@echo Allowing multicast broadcast response on UDP (Browser Service Enumerations OK)
netsh firewall set multicastbroadcastresponse ENABLE

———————————————–

I also added this registry key below. You can copy everything between the breaks and save as a .reg file if you’d like and run that ‘as administrator’ as well.

—————-
Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSSQLServer\Client\SuperSocketNetLib\tcp]
“DefaultPort”=dword:00000599
—————-

After that I was able to very quickly install SQL2008R2.
During server configuration, under Account Name I selected “NT AUTHORITY\SYSTEM”, leaving password blank. In addition I also changed its startup type from Automatic to Manual.

Once installation was complete, I opened SQL Server Configuration Manager. The current setting was that I was logging on as built-in account “Local System”. I started the service without making any changes and this time it started without any error messages.

Just thought I’d put this up for any of those who’ve had the same issue and as a reference for myself.

To connect to the Server make sure you specify the machine name AND the server instance name: Windows7PC1\SQL2008dev or whatever

Well, until I hear differently from my Devs 😛

Backups, system failures and peace of mind

Another week in the trenches. I had a primary server at our organization have a major failure. The SAS controller (which provides access to SAS type HDDs) died OR the motherboard to the server itself has an issue. Either way without another ‘like’ system that I can put the SAS card into to see if the issues is just the card or the motherboard I cannot access my drives – and they too may be very corrupted. The only machine I have capable of putting the card into is in production. And the cost of a replacement Dell Perc5i SAS card is nearly $200.00 US and could take days to get here. Plus I needed to have this system back up and running very quickly – the server in question runs all or our company financial, shipping and reporting software applications!

Since I have all my ‘data’ backed up to a server drive every night I was secure in the knowledge that we at least had the financial databases and ‘files’ available. But how to get a system back into production? Disk Imaging to the rescue!! I had a fairly recent full system image, created with my favorite backup software – Acronis, available. Yay! Just need a place to restore it to.

Since my organization now has a VMware ESXi/vSphere SAN and cluster running I was easily able to create/import a new ‘Virtual Machine’ from the Acronis disk image very quickly and then just copy over the backed up data files from the night/early morning before. WORKED LIKE A CHARM! If I’d had an available server (Hardware wise) I could also have restored that image to it too.

I’m telling this to you to remind you – I believe in Images(Clones) for my backups, alone with periodic ‘file backups’. That way I’m protected against full drive failures/loses AND stupidity – accidentally erasing or overwriting files. :)
[Imaging or cloning is the procedure by which you create a backup that is identical to a bootable system either to another internal or external drive. This is the ultimate backup! Should your drive fail you can just ‘pop in’ your cloned drive or ‘restore’ that clone to a new drive and your are up and running.]

If you are not regularly creating full image backups you WILL be sorry! I have written numerous articles about cloning and back up.

PLEASE read here if you any kind of concern for you data.

For Mac images and cloning go here.

So of course this weekend I created two new images on separate drives for my home system(s). I can’t tell you the peace of mind you will get from knowing that the worst that could happen to your system is that you might lose a couple of days or a weeks worth of information. If your drive gets corrupted or fails or you get trashed by some virus, you could be back up and running within a very short period of time! No re-installing your Operating System and programs and ‘trying’ to find you data files. Just restore the image and BAM, you up!

What prompted me to start on this rant is that Apple has finally acknowledged it is having some major issues with some of the hard drives in some of their newer systems they have been selling. Looks like some of the drives just ‘fail’. OUCH! You can read about that here.

And although you can have your drive replaced – YOU WILL LOSE YOUR DATA! The Apple folk and/or kids the the ‘Genius’ bar will NOT re-install your system software or clone your drive for you!! Unless you have an image to restore you will have to re-install you System and applications. And unless you had at least some kind of backup to another drive (Time Machine type) your data (read pictures and music!) will be gone!

So folks, backup, backup and then backup again.

The cost of a couple of extra external drives and a little program setup is minuscule to the cost of losing you ‘digital life’. Right now Acronis has a special – only $29.00 US for their home product!! With Apple’s you can even get a way with out purchasing any software!

Be safe, be secure and gain some peace of mind.