vSphere Storage Appliance cleanup.bat

April 1, 2013 in Storage, vHersey, VMware

I have been doing some home lab work with the vSphere Storage Appliance (VSA). I have had a few request recently from customers with very small environments looking to leverage virtualization and a couple of large environments looking for solutions for branch offices.

I have been doing a lot of testing which included destroying and rebuilding the environment (multiple times). I just want to be familiar with how failures are handled and more importantly to recover from a failure. This is pretty well covered in the vSphere Storage Appliance Installation and Configuration Documentation.

One thing that was helpful in the lab environment was the section on “Deleting a VSA Cluster

There is a script called cleanup.bat on the VSA Manager Server, in the case of my lab this is installed on the vCenter Server, located in C:\Program Files\VMware\Infrastructure\tomcat\webapps\VSAManager\WEB-INF\test\tool\. To completely delete a VSA Cluster simply run this script passing your vCenter Administrator username, password, and Datacenter the VSA Cluster is configured on as parameters.

cleanup.bat vi-admin@lab.local MySuperSecretPassword LabDC

Not only does this delete the VSA Cluster, it also destroys all data in the VSA Cluster associated with the Datacenter LabDC – so be careful!!!

Just a note that I tried this several times without success. I found that the cleanup.bat script had to be run from an elevated (Run as Administrator) command prompt for it to work.

As the document says once the cleanup.bat completes you will need to restart the VMware Virtual Center Management Webservices.

I know the VSA is marketed for the SMB and I can see where that would be a good use case for it but I think for use in a branch office, where it is managed by a vCenter in an HQ, is really the where the VSA can shine.

Curious if anyone has deployed the VMware vSphere VSA in a production environment? Interested in hearing your thoughts or comments.

vSphere 5.1 Announced!

August 28, 2012 in vHersey, VMware

Lot of great announcements this morning during the VMworld keynote. One of these being the official announcement of vSphere 5.1 and there are already a number of blogs out there posting information on many of the new features and enhancements. Here are just a few of the new features and enhancements I am looking forward to:

  • Enhanced Web Client
  • Enhanced vMotion (Shared storage no longer necessary < face melting stuff!)
  • Lots of virtual Distributed Switch enhancements (LACP, vDS backup and restore)
  • Virtual Data Protection (VDR replacement based on EMC AVAMAR technology)
  • Reboots no longer necessary after guest VMware Tool upgrades (makes this VM admin very much happy!)
  • Licensed by physical CPU socket – No vRAM licensing!
  • and more…

You can find a good overview of the new features of vSPhere 5.1 here http://www.vmware.com/files/pdf/products/vsphere/vmware-what-is-new-vsphere51.pdf and vCenter Server 5.1 here http://www.vmware.com/files/pdf/techpaper/Whats-New-VMware-vCenter-Server-51-Technical-Whitepaper.pdf

I have had the vSphere 5.1 beta installed in my Home Lab for some time now but have not had a lot of time mess with it much. The new Web Client is pretty slick and it seems to have all the functionality of the Windows vSphere Client. The vDS backup and restore is also a great feature. Going to have to make some time to work with it a bit more.

Of course once the NDA curtain was lifted the blog post started flying. Here are a few post that dive a bit deeper into the 5.1 release.

There will be many more post in the near future so keep an eye out for them!

Some really exciting stuff! What’s your favorite new 5.1 feature?

vSphere HA Protection: Protected? or Not?

June 6, 2012 in vHersey, VMware

Update from VMware Support – 6/12/2012

Just received this from VMware Support:

“We were able to reproduce this issue in our lab and agree that discrepancy in the reporting is misleading. In our testing, we verified that although the reporting is incorrect currently, the manual action is still taking place. We have reported our findings to engineering to address this issue and a knowledge base article is being published to explain the issue. Once the patch/fix becomes available from engineering, the KB article will be updated with a link to it.”

I will update with the KB link once I get it.

Update from VMware Support – 6/11/2012

I submitted a support request on this and here is what I received:

“This is expected behavior in the environment. HA will still be monitoring all of the hosts and VM’s in the environment, so it will still label it as Protected. In the event of a failover, it will read its setting, and follow whatever you have defined for the VM, specifically. If the restart priority is set to disabled, it will not restart the VM in any way. This does not affect things like VM Monitoring, each of these settings can be configured individually as well.”

Not sure I agree with that since it clearly does not meet the required conditions as shown on the screenshot (Restart priority has not been disabled for this VM in the cluster), but it looks like it is what it is. I emailed support for more clarification but have not heard back yet (case is still open).

I would expect “Protected” to mean that a VM would be restarted in an HA event but doesn’t look like that is the case.


There was a recent discussion in the VMware Communities about using PowerCLI to get the VM HA protection status of a VM.

I think there may be a problem with how the protection status is determined.

According to the call out on the VM summary page the following conditions must be met for a VM to be protected:

- VM is in a vSphere HA enabled Cluster
- VM powered on successfully after a user-initiated power on
- vSphere HA has recorded the power state for this VM is on
- Restart priority has not been disabled for this VM in the cluster

So I set the restart priority of one of my VMs to disabled. HA reconfigured and the VM still showed protected. Hmmmm, according to the conditions that must be met the VM shouldn’t show protected anymore, right?

I disabled HA on the cluster and then re-enabled it. Still the VM shows protected. I tried this test in both my lab environment and my production environment.

I also posted a message to twitter and received a response from @TheJasonNash that his quick lab test produced the same results. Disabling the HA restart priority for a VM does appear to change the protection status.

I have a bunch of non-critical VMs that I have set to the HA restart priority set to disabled in my production environment, never really paid attention to the fact that they still show protected until I started messing with the PowerCLI to report on it.

What would cause vSphere HA Protection to display unprotected?

Searched the VMware KB but did not find anything. Anyone know if this is a known issue or if there is a fix?

vSphere Auto Deploy – Image Profile and Booting ESXi

April 24, 2012 in DCA, vHersey, VMware

The VCAP5-DCA Blueprint has objectives on working with ESXi Image Profiles and configuring VMware Auto Deploy which are both new features that were released with vSphere 5. As part of studying for the VCAP5-DCA beta exam I have been working on becoming more familiar with these features in my home lab. The vSphere Installation and Setup Documentation does a good job walking through the process of preparing a ESXi Software Image and setting up Auto Deploy.

Enable Auto Deploy on vCenter Server Virtual Appliance

If you are running vCenter on a Windows Server you will need to install the Auto Deploy server on that server. If you are running the vCenter Server Appliance Auto Deploy needs to be enabled by logging in to the web management interface and navigating to Services -> Autodeploy.

To use the default port and repository size just click “Test Settings”, then click “Save Settings.” Then you must stop and start the ESXi services from the Services -> Status page. ESXi Autodeploy status should change to “Running”.

Once Auto Deploy is running it can be accessed from the Home menu in the vSphere Client. From here download the TFTP Boot Zip and note the BIOS DHCP File Name.

Set up DHCP and TFTP

I am using this DHCP server running on an XP VM to serve up DHCP for my lab environment. This DHCP server also has a built in TFTP server.

Configure the DHCP scope and add the BIOS DHCP File Name and the TFTP server IP address to the boot file parameter (this is done in the advanced options of the DHCP wizard if you use this DHCP Server). Unzip the Auto Deploy TFTP Boot File into the TFTP server root directory.

ESXi Image Profile

Adding an ESXi Software Image Profile is done using PowerCLI cmdlets. I downloaded the ESXi 5 zipped distribution from VMware.com and placed it in a directory on my C:\ drive.

The zip is then added as a Software Depot using PowerCLI.

Add-ESXSoftwareDepot c:\pathto\ESXi5.zip

Use Get-ESXImageProfile to display the available images in the depot.

Name                           Vendor          Last Modified   Acceptance Level
----                           ------          -------------   ----------------
ESXi-5.0.0-20111104001-stan... VMware, Inc.    10/13/2011 3... PartnerSupported
ESXi-5.0.0-20111104001-no-t... VMware, Inc.    10/13/2011 3... PartnerSupported

Then create a new deploy rule to use the Image Profile you want and apply it based on a host pattern. I am checking that the IP address (ipv4) is within the DHCP range set up for the hosts.

New-DeployRule -Name “ESXiDeployProfile” -Item ESXi-5.0.0-20111104001-standard -Pattern “ipv4=192.168.1.230-192.168.1.239″

The software will unzip and the packages will be uploaded to the Auto Deploy server.

Once the deploy rule is created it has to be activated.

Add-DeployRule -DeployRule ESXiDeployProfile

Auto Deploying ESXi

Since I am running my lab in Workstation I created an ESXi VM with no hard drives and added the necessary NICs for my environment (connections to management, storage, vMotion, and VM networks). The VM boots straight to network boot. The VM receives a DHCP address, the boot file from the TFTP server, and then begins booting from the Auto Deploy repository.

After the initial PXE boot the ESXi VM begins booting ESXi from the Image Profile based on the Deploy Rule.

When the host finishes booting it is added to the vCenter server inventory in maintenance mode.

Now a Host Profile and Answer File will need to be applied to configure the ESXi host for use in the environment. I will cover this in another post in the next couple days… and here it is Auto Deploy: Host Profiles and Answer Files.

oneVirt Viewer Version 1.4.0 Released

April 18, 2012 in VMware

oneVirt Viewer is a handy “cross platform” FREE configuration data collection and reporting tool for vSphere 4 and 5. It is written in Java and can be used on Windows, Mac, and Linux as long as Java virtual machine is installed. Very handy if you are in need of a tool that works on a Linux workstation.

Once collected the configuration data can then be exported to csv or displayed in an HTML report.

The new version was suppose to fix an issue with opening reports on Windows but I am still experiencing it. If a report does not open in your browser it can be found in the programs install directory (c:\Program Files\oneVirt Corp\onevirtviewer-1.4.0 by default) in the reports folder.

The oneVirt Viewer homepage is here and oneVirt Viewer 1.4.0 can be downloaded here. Great tool that is worth taking some time to check out.

vSphere 5 host stops sending syslog messages

October 25, 2011 in DCA, vHersey, VMware

I had some maintenance to do on my central syslog server and the server was down for about 30 minutes or so. When the syslog server was back online it was no longer logging syslog messages from my vSphere hosts. First thing I did was check to see if I was receiving syslog messages from the hosts using tcpdump.

tcpdump -i eth0 -A -s 0 udp port 514 and host ESXiHostIpAddress

Nothing.

I thought maybe something changed with the maintenance/upgrade that caused the issue but a quick search resulted in this vSphere 4.x article in the VMware KB. I did not find anything on this issue in vSphere 5 but the 4.x article pointed me in the right direction.

I checked the logging in the Syslog.global.logDir for the hosts and logging had stopped there also.

To correct this you simply have to reload the ESXi syslog service using esxcli.

esxcli –server HostIpAddress –username UserName –password PassWord system syslog reload

Once syslog is reloaded run tcpdump on the system again and you should see syslog messages from the host. I also verified that syslog messages were being logged to Syslog.global.logDir.

I did some quick testing and if I take the syslog server down even briefly (no more than a minute) I have to reload syslog on each of the hosts before log messages a generated.

All is well now. Just have to remember to reload syslog on the ESXi hosts anytime I take down the syslog server.

Anyone know why this happens or if there is a way to prevent it from happening (other than never taking down the syslog server)?

Another vSphere 5 Upgrade Gotcha – Syslog

October 5, 2011 in vHersey, VMware

I had my vSphere 4.1 ESXi hosts set up to send logs to a central syslog server. After the upgrade to vSphere 5 this was no longer working. Did some checking and the syslog configurations that were set in 4.1 are gone and had to be reconfigured.

In vSphere 5 the remote syslog host is configured by setting Syslog.global.logHost

This setting is found in the Host Configuration -> Advanced Settings -> Syslog

The remote server for Syslog.global.logHost is specified in this format protocol://remotesyslogip:port (udp://10.10.10.10:514 for syslog using udp on port 514 to host 10.10.10.10).

It can also be set from the VMA using “esxcli system syslog config set –loghost=protocol://remoteipaddress:port”

Here is the VMwareKB on Configuring syslog for ESXi 5.0.

But wait – there’s more!

After setting up the remote syslog host you have to enable the firewall rule to allow syslog traffic outbound (port 514 tcp or udp). This can be done on the host from the vSphere client in Configuration -> Security Profile -> Firewall Properties.

Good luck.

vCenter 5 Upgrade Gotcha (well me anyway) – DB Recovery Model Changed to Bulk-logged.

September 29, 2011 in vHersey, VMware

Last night my vCenter server ran out of disk space. The transaction log for the vCenter DB filled until there was no more space on disk available for vCenter server services to write to so the the service stopped.

I received an email alarm early in the AM – “The transaction log for database ‘vCenter’ is full.” A quick check of the server showed the drive where the DB is stored was completely full and the DB transaction log was giant. Digging a little deeper I discovered the Recovery Model for the vCenter DB was no longer set to Simple, it had been changed to Bulk-logged.

It appears that when upgrading from vCenter 4.1 to vCenter 5 the recovery model for the vCenter database is changed to Bulk-logged as part of the upgrade process. Changing this back to simple and recovering the disk space is fairly easy and instructions can be found in the VMware KB article Troubleshooting transaction logs on a Microsoft SQL database server.

Searching through the KB it looks like this has been happening with upgrades since version 2.x. The Known issues when installing or upgrading vCenter Server KB article only makes mentions through version 4.x. It looks like there may be some of the same issues when updating to vCenter 5, though I could find anything in the KB that specifically mentioned these issues with 5.

Just a heads up for anyone that may be upgrading to vCenter 5. After you upgrade you may want to check the recovery model settings for you DB.

Awesome HA and DRS Audit Script Based on the HA/DRS Technical Deepdive

May 17, 2011 in VMware

If you do not have a copy of VMware vSphere 4.1 HA and DRS Technical Deepdive by @DuncanYB and @FrankDenneman then go here, buy it, and read it cover to cover. The book covers everything VMware High Availability (HA) and the Distributed Resource Scheduler (DRS) (and then some!).

Alan Renouf, co-author of another must have book – VMware vSphere PowerCLI Reference: Automating vSphere Administration, has put together a nice HA and DRS Audit PowerCLI script that can be used to gather information and check for best practices on your HA/DRS cluster based on the information from the Technical Deepdive book.

The script is run against your vCenter and creates a nicely formatted HTML report with all the HA/DRS related details of each cluster managed by vCenter. This gives you a detailed overview of your environment and the configuration of HA and DRS.

As of right now the script is based on the information contained in just the first 50 pages of the HA and DRS Deepdive. Alan Renouf is promising updates to the script as he works his way through the book.

Go here to download the HA and DRS Audit script.

Free Compliance Checker for vSphere Test Drive

May 16, 2011 in VMware

The free Compliance Checker for vSphere checks your environment against the security guidelines from the vSphere 4.0 Hardening Guide.

It checks up to 5 ESX/ESXi host for free.  Just enter your vCenter server name or the IP address of an individual ESX/ESXi host and it performs the assessment.  If you have more than 5 ESX/ESXi hosts in your environment the compliance checker only performs the assessment on the first 5.

The Compliance Checker generates a nice HTML report that displays the compliance rule that was checked and whether the host passed or failed the check according to best practices.

Each rule references a code from the Hardening Guide so that you can look up the recommendations and remediation steps.

Nice free utility, very useful in checking small (<5 host) environments for security hardening best practices.

Download the Free Compliance Checker for vSphere here.