When upgrading vCenter Server it displays a warning that it is going to remove some users that the VMware SSO Server is not aware of and writes a text file containing the deleted users and groups called deleted_vc_users.txt into the system’s temp directory. Any local server users or groups that have vCenter permissions are deleted.

During the upgrade the pop up warned me, but I was sure that the Domain Admins group had been configured with the Administrator role (it was not – Domain Admins had been added to the local Administrators group -ugh!). The upgrade finished successfully but it had deleted the local Administrators group from the vCenter permissions. Oh no!!!

Again I had been warned, but thought for sure the Domain Admins group had the permissions. If the upgrade is going to delete all the users with the Administrator role it allows you add a group during the upgrade, since there was a single Domain User that did have the role I did not receive this prompt.

I could have tracked down or changed the password for the user that still had Administrator access but since I had some time and there was no production impact I decided to do some poking around.

***DISCLAIMER*** ***WARNING*** ***DANGER***
Make sure you have a good backup of your vCenter database and know how to get yourself out of a jam before messing around in it. You have been warned!

I came across a forum thread about exporting the original VPX_ACCESS table from the vCenter Database and then importing it over the new VPX_ACCESS table, this gave me the idea to just try and manually add the Domain Admins group to the VPX_ACCESS table.

A little more digging and I found this VMware KB article that walks you through the process of changing a user from the Read Only role to the Administrator role by updating the VPX_ACCESS table. From the information in the KB I was able to add the Domain Admins group and regain access to the vCenter Server.

First stop the vCenter Server Service. Then use Microsoft SQL Server Management Studio to access the vCenter Database. Navigate to the Tables folder of the vCenter database and find the table dbo.VPX_ACCESS. Right click on the table and choose “Edit Top 200 Rows”

The table should display users configured with permissions in the environment.

Select the Null row to add a new entry in the table where the ID is a unique ID (just add a number that does not already exist), PRINCIPAL is the user or group (LAB\Domain Admins for this example), ROLE_ID is set to -1 for Administrator role (-2 is Read Only), ENTITY_ID is the object the user is granted permission to which is 1 for vCenter, and FLAG is 3 for a group (or 1 for a user).

I had found a post that explained the fields but I am not able to find the link again. Will update this post when/if I come across it again.

Once the row has been added to the table start the vCenter Server Service.

You should now be able to login and administer the vCenter environment with a user from the group that was added. The new permissions should also display.

TA-DA!

Of course the better option would had been to verify the permissions and heed the warnings that popped up during the upgrade but it is good to know there is another option.

Note that for this to work the Identity Source for the domain must be configured correctly in VMware SSO Sign-On and Discovery. If the domain is not set up correctly you will not be able to authenticate any users in the domain.

The vExpert awards for 2013 were announced yesterday afternoon. This is my second year as a vExpert! I am excited to be a part of the vExpert community and it is humbling to be included along side of so many great VMware community contributors.

I want give a shout out to ProfessionalVMware.com for providing the outstanding vBrownBag Program. The vBrownBag provides a fantastic opportunity to contribute to the VMware community. My vExpert application included many links to the vBrownBags I have presented. Thanks for a great program and I look forward to presenting many more in the future.

A huge thanks to John Troyer and the VMware Community for having such a great program.

It is AWESOME to be part of such a fantastic community!

Looking forward to the future….

I downloaded the offline bundle from My VMware.

Using the Datastore Browser I uploaded the offline bundle to a datastore that is accessible by the host I am upgrading.

To list out the image profiles that are contained in the offline bundle using esxcli.

esxcli software sources profile list -d /vmfs/volumes/datastore1/update-from-esxi5.1-5.1_update01.zip

Now apply the image profile update ESXi-5.1.0-20130402001-standard from the offline bundle with esxcli.

esxcli software profile update -d /vmfs/volumes/datastore1/update-from-esxi5.1-5.1_update01.zip -p ESXi-5.1.0-20130402001-standard

Once the update completes a host reboot is required.

After I rebooted the host the build number changed to 1065491 which corresponds to ESXi 5.1 Update 1.

Now to update VMware Tools on all my lab VMs. Not a big deal though since the tools update does not require a reboot! Nice!

Here is a great post on performing the update by downloading the bundle from the VMware update repository directly to the host – http://vninja.net/vmware-2/quick-dirty-esxi-5-1u1-upgrade/ – great write up.

Good to know the update can be applied a number of different ways; using VUM if your host is managed by vCenter, using the offline bundle, or by connecting directly to update repository from the host.

SSH access has to be enabled on the host you are copying to (and on the host you are copying from if you are accessing it via SSH).

scp path_to_file_to_copy root@FQDN_or_remote_host_ip:path_to_new_file_location

For example if you have an ISO named Win.iso on the local datastore1 of ESXi-1 (192.168.1.25) and you want to copy it to the local datastore1 of ESXi-2 (192.168.1.201):

scp /vmfs/volumes/datastore1/Win.iso root@192.168.1.201:/vmfs/volumes/datastore1/Win.iso

You will be prompted for the root password of the remote host and once you have successfully authenticated the file will copy.

I know this works on ESXi 5.x hosts, not sure about earlier versions. Again make sure SSH is enabled on the remote host.

Happy Friday!!!

I created a backup of the original vCenter database and for the size of the environment the database was ginormous, a little over 15GB. I thought that was odd but for whatever reason I did not really dig into it much (until this would cause the upgrade to fail – keep reading). The database was detached from the original physical vCenter Server, the db and log files were copied to the new vCenter virtual machine, and the database was then attached to the the new SQL server. The vCenter ODBC connection to the new database was created and tested, all looked fine.

The vCenter Server installation was started and the database upgrade option was selected. The upgrading database part of the process appeared to hang and the transaction log for the vCenter database started to grow. It started to look like it would fill up the disk space on the drive set up to store the databases. After giving it about an hour I went ahead and canceled the upgrade process.

Some quick googling led me to the VMware KB article http://kb.vmware.com/kb/1007453. The VPX_HIST_STAT tables were over 12GB in size! I then noticed that the SQL Agent was not running on the old vCenter Server. The SQL Agent Service had been disabled and because of this the vCenter performance data roll up jobs were not being run. When SQL is installed the SQL Agent service is set to Manual start up by default, this should be changed to Automatic since this service is required for the vCenter database roll up jobs to run.

Due to the size of the tables and time constraints instead of trying to get the roll up jobs to run, I followed the process outlined in VMware KB 1007453 to truncate the VPX_HIST_STAT(1-4) tables and the VPX_SAMPLE_TIME(1-4) tables. Be aware that this process removes all historical performance data. To keep the performance data the roll up jobs could have been run, but because of the size of the tables this probably would have taken a lot of time and I doubt it would have completed successfully.

Once these tables were truncated the upgrade completed without issue. On the new vCenter Server I set the SQL Agent Service to Automatic and after a few days checked the environment to ensure the roll up jobs were being processed (VMware KB article – http://kb.vmware.com/kb/2007388).

Once the vCenter upgrade was completed the upgrading/migrating the host from ESX 4.0 using VUM went without issue.

Checking the roll up jobs is now part of my pre- and post- upgrade process.
(I know, I know it should have been part of the process in the first place)

One of these environments is going through a network redesign and I am helping out with that. The network changes are going to require that some changes be made to the network configurations on the XenServers and this is what I was asked to help out with. I have had minimal exposure to XenServer and it was back a few years ago so I needed a quick refresh. I spun up XenServer in the lab for some quick testing.

Good news is the changes that need to be made are fairly simple and things should go without issue. Bad news is all VMs without XenServer Tools installed (a majority of them) will have to be powered down before the network connection can be changed. What!!! Powered down to disconnect a VM from one network and reconnect it to a different one, yep. Crap!

One feature of XenCenter that I do find extremely neat (and useful) is the host Console access through XenCenter. Just select the host and then the Console Tab and you are presented with a command line console for the select XenServer host.

To start up the console GUI (similar to DCUI on ESXi) run xsconsole from the command line.

XenServer was easy to install and XenCenter management is fairly intuitive. The host console access is a neat feature but overall I am not impressed (now I know why I stopped messing with this way back when). I am going to keep poking around in it just so I can be familiar with the ins and outs but as I tweeted yesterday – “XenServer installed in the home lab. So I tried it and it is OK, now give me back my vSphere.”

BTW my lab XenServer environment is running on ESXi. To do this you prepare and configure the VM just as you would to run nested ESXi (VM HW version 9, vhv.enabled = “TRUE”, and Guest OS: VMware ESXi 5.x).

I <3 vSphere!

I have been doing a lot of testing which included destroying and rebuilding the environment (multiple times). I just want to be familiar with how failures are handled and more importantly to recover from a failure. This is pretty well covered in the vSphere Storage Appliance Installation and Configuration Documentation.

One thing that was helpful in the lab environment was the section on “Deleting a VSA Cluster”

There is a script called cleanup.bat on the VSA Manager Server, in the case of my lab this is installed on the vCenter Server, located in C:\Program Files\VMware\Infrastructure\tomcat\webapps\VSAManager\WEB-INF\test\tool\. To completely delete a VSA Cluster simply run this script passing your vCenter Administrator username, password, and Datacenter the VSA Cluster is configured on as parameters.

cleanup.bat vi-admin@lab.local MySuperSecretPassword LabDC

Not only does this delete the VSA Cluster, it also destroys all data in the VSA Cluster associated with the Datacenter LabDC – so be careful!!!

Just a note that I tried this several times without success. I found that the cleanup.bat script had to be run from an elevated (Run as Administrator) command prompt for it to work.

As the document says once the cleanup.bat completes you will need to restart the VMware Virtual Center Management Webservices.

I know the VSA is marketed for the SMB and I can see where that would be a good use case for it but I think for use in a branch office, where it is managed by a vCenter in an HQ, is really the where the VSA can shine.

Curious if anyone has deployed the VMware vSphere VSA in a production environment? Interested in hearing your thoughts or comments.

There are several Toastmaster clubs in my area but I chose the ComUnity Speakers Toastmasters Club #5037 located in Chesapeake. This club meets on the 2nd and 4th Monday of each month at 7:00 PM and the meeting location is on my way home from work. I decided to check out this club specifically because of the activity posted on social media. The club seems to be very active with good participation from the members.

I enjoyed the meeting. The meeting follows an agenda that is very similar to the one on the ComUnity Getting Started page. The club President called the meeting to order and there were two prepared speeches by members. One was a member’s first Icebreaker speech and the other was a member’s speech project to demonstrate using props while speaking.

After the first two prepared speeches there were what are called “table topics” sessions. This is where you are given a topic and asked to speak “off the cuff” about it for one to two minutes. Everyone was invited to participate and I think everyone that was in attendance did. I even got up and talked about my cat for a minute and 18 seconds.

After the prepared speeches and table topics evaluations are given. The evaluator for each speech gives an oral assessment of the speech offering praise, tips, and suggestions. The “ah” counter and timer (everything is timed) also give their assessments. The general evaluator then gives an evaluation of the meeting overall. All evaluations were very constructive.

At the close of the meeting they asked guest to speak about how they found out about the club, what brought them to the meeting, and what they thought of the meeting. Neat process and a good group of folks. I can definitely see where this could be a benefit to anyone looking to become more comfortable with public speaking or just communicating in general.

Looking forward to the next meeting.

Had some really great sessions my last day at PEX. The Virtualizing Business Critical Applications for Maximum Performance session with @vmMarkA was excellent. Some really good performance best practices and a ton of great performance troubleshooting information. Specifically the information on the key metrics to monitor when trouble shooting performance issues. A lot of information packed into this one hour session and I have a few pages of notes on it. These will most certainly be summarized in another post in the very near future.

I also attended two other fantastic sessions on my final day, one on Exchange Virtualization Best Practices by @afontana_vmw and another which was a HA Deep Dive for SQL Server Virtualization by Scott Salyer of VMware. The Exchange session expanded a bit on the earlier performance session offering up some Exchange specific performance tuning options. The HA Deep Dive for SQL dove deep into the different HA options for SQL. Again, once I organize my notes there will probably be a separate post with a bit more detail on these session.

Finally got a chance to meet up with @nfritsch. Great guy! Had a great discussion with him about Citrix VDI and we chatted a bit about VMUG. Great to meet you Nick and look forward to chatting with you again soon.

The only thing I was disappointed with at PEX were the General Session Keynotes. Not sure what I was expecting but even with the stage setup and elaborate production these were anticlimactic (how awesome can a tech keynote be, don’t know but I was expecting more) and uninspiring. I can’t say that keynotes were very beneficial to me.

Where the keynotes were pretty blah, the guest speaker, Shawn Achor who gave a talk during the General Session on the second day was fantastic. Will probably end up buying his book, The Happiness Advantage: The Seven Principles of Positive Psychology That Fuel Success and Performance at Work, hope it is as interesting and entertaining as his presentation was.

I had a great time, learned a bunch of new stuff, and met a lot of really cool people. Look forward to the next one.

Nick @nfritsch

Damian (@sixfootdad) doing some work on AutoLab in the Hang Space.

Hey look it’s me! (This picture won me an iPad in the Brocade/EMC twitter contest!!!)

Fingers crossed for @timantz and @kennega and everyone else that defended for VCDX at PEX. You guys are awesome and inspiring!

The reception was fun and it was awesome to meet a few folks that I only know by twitter handle. Good to see @Bacon_Is_King, @sixfootdad, @networkingnerd, and @jasemccarty If you are not all ready you should follow these guys, great folks.

Also ran in to @timantz and @kennega at the party. Both of these guys did their VCDX defense earlier in the day. It was great chatting with them about the experience.

Here are a couple few (not so great) pics from the reception.

See it’s not just me. @Bacon_Is_King and @sixfootdad tweeting.

@Bacon_Is_King, @sixfootdad, and @networkingnerd

@kennega why so serious.

Hey it’s @timantz and that is @jasemccarty in the foreground.

Hey look. It’s me!

Had a great time and it was great to be able to put a face to a twitter handle.