I was asked today to figure out a quick (Implemented in less than 6 weeks from today 10th December!) short term (< 12 months before being replaced) solution to provide between 20 and 50 additional remote desktops for our existing VMware Horizon View environment. I have some constraints and options to consider. Continue reading
VMware Horizon View Persistent Desktops on Local SSD Storage (A back of the envelope design exercise)
Filed under Sysadmin
Real World Write Performance with PernixData FVP
I have been running PernixData FVP software in my production datacenter for a little over a month now and I wanted to share some of the results we’ve seen from deploying this amazing software.
Filed under PernixData FVP
Safely Backing Up a Linux HA Cluster with Veeam
I recently deployed a Linux HA Cluster for NFS in an active/passive mode using drbd, running on VMware and using vCentre for fencing.
Whilst I was testing this deployment I noted that during our nightly backup process Veeam would (as is normal) request a snapshot of the cluster VMs and then release the snapshot after the backup was complete.
The VMs appeared to stop responding to Cluster heartbeats briefly whilst the snapshots where taken and released; this is something that Veeam customers are occasionally familiar with and usually it’s not an issue. However because of this brief dropout in heartbeats the passive node would then begin to promote itself to active status. In order to avoid a split brain scenario and because I had configured the cluster to use vCentre for fencing the previously active node would immediately be reset as soon as this promotion began.
Generally speaking Linux VMs prefer to be shutdown cleanly and safely; regular hard resets soon caused problems in my cluster and on several days of testing I’d come into the office to find that the cluster had just given up and both nodes were in a passive state, unable to decide which of them should be active and which should be passive.
The solution I’ve implemented is to use a Veeam feature called “Enable VMware Tools quiescence”, 
this allows me to use VMware tools to run a script prior to snapshot creation which correctly and cleanly forces the VM being backed up into passive mode (if it was active to begin with) allowing the other node to take over the cluster without heartbeat issues.
Having ticked the appropriate box in the Veeam backup job the necessary scripts needed to be deployed on the VMs themselves, there are two scripts I’ve hacked up that work together to make this work.
In my cluster I have two hosts LinuxNFS-1 and LinuxNFS-2, I’ve hardcoded the names of the hosts into my scrips, I’m pretty sure there’s probably a more elegant way of pulling the other hostname out of the cluster cdm tool but this works well enough for my purposes.
Thoughts on passing the VCAP5-DCD Exam (or why Fibre Channel might be my Achilles heel)
So I sat and passed VMware’s Advanced Professional exam for Data Centre Design today and whilst I’m super happy that I passed I only just scrapped a passing score and I had some serious concerns whilst sitting the exam that I had some holes in my knowledge base.
Having had a few hours to think and re-consider I’ve concluded that actually I’ve got one large, gaping hole in my knowledge base, Fibre Channel; it’s the one VMware storage technology that I’ve only got a theoretical knowledge of – I don’t have any practical hands on experience with FC and it shows.
I found that every time an FC question came up on the DCD exam (and there were quite a few) I dropped out of the flow of the exam; I had to think harder about what the question wanted and what the answer was – I found these questions much more difficult than I should have.
So – what do I do about this? I need a practical way to gain some hands-on experience with FC equipment. I’m thinking maybe I need to add some second hand FC HBAs, switches and storage to my test environment. Of course it can’t be too expensive but it also can’t be too old and no longer representative of the current state of typical FC installations.
I’m going to look for advice from folks I know who have lots of FC experience (look out twitter…) and then I guess I’ll be sending a good bit of time on e-bay….
Filed under VCAP-DCA
Recruitment Spam By Postcard!
Those of use who have spent any time in a technology role will be familiar with the quantity of spam generated by recruitment agencies, either trying to sell us on a new role or to convince us of how many really great candidates they are currently representing.
I know plenty of IT folk whose voice mail and e-mail filters are constantly under pressure from the efforts of recruiters but today I experienced an all new form of spam, one that actually got my attention, even if only briefly…
Postcard Spam!
Yep that’s right – I got an actual hard copy postcard delivered to me at work, individually hand written and stamped so that it wouldn’t look like commercial mail.
The message on the back of the postcard read “Hi, I’ve joined [name of recruiter] and it’s great! I’m still available but might not be for long as there’s a financial services technology company interested in my 7 years experience in IT project management. Get in touch with me through [name of recruiter] I’m candidate #99999 if you want to see my CV.”
I’m impressed that this recruitment agency went out of it’s way to get my attention, I’m also impressed that they were willing to spend real money on doing so (sending out e-mail spam is very low cost, this particular post card had an Irish 60 cent stamp on it.) I’m also impressed by the timing; summer is typically when postcards show up in offices from staff who want everyone to know what a great time they are having by the beach and how much they don’t miss everyone back at HQ.
However much I was impressed by this I do hope it doesn’t catch on – or if it does maybe Google will launch a new Bayesian filter service for postal networks, I think maybe Postini might be a good name for it!
Filed under Sysadmin
How To – Fix Packet Drops on Nested ESXi
Edit:
Did you ever wish you could just forget the last hour or so and start again?
I thought I’d found something useful today – unfortunately it turns out that even though the %DRPRX counter was running comfortably at zero I had also managed to completely break all connectivity to my nested ESXi hosts – so it’s back to the drawing board for this problem – I’ll leave the post here so that anyone who’s interested can see what doesn’t work!
I was getting some unexpected connectivity problems with some nested ESXi hosts I was running in my test lab. I had a look at esxtop on the real ESXi host and noticed in the network screen that I was getting unexpected values in %DRPRX – a quick dig online found this VMware KB ‘http://kb.vmware.com/kb/1010071′.
This article suggests that this problem is caused by not having sufficient receive buffers in the guest, and it provides instructions for increasing the number of receive buffers for Windows and Linux guests – unfortunately that’s not going to work with an ESXi guest.
The same article however also points out that the default number of receive buffers is much greater when using vmxnet3 compared to e1000. Which got me thinking… Can you use a VMXNET3 nic with ESXi? Well it didn’t take long to find this post from William Lam ‘http://www.virtuallyghetto.com/2012/09/nested-esxi-51-supports-vmxnet3-network.html’ William discovered that ESXi 5.1 does allow the use of vmxnet3 nice, it’s unsupported but it works.
Anyway a quick modification of my virtual machines and I’ve now got my nested ESXi hosts working with vmxnet3 nics and suddenly %DRPRX is a nice healthy 0.00
vCentre 5.1 SSO – Changing the Master Password (the right way and the wrong way)
Following on from my post yesterday http://vcdxorbust.com/2013/05/29/upgrading-vcentre-5-1-sso-password-confusion/ I have since learned that there is a supported way to change the master password. It’s documented in http://kb.vmware.com/kb/2038826
The tool used to change this master password however is in my opinion arguably broken. It appears to allow two mechanisms to change the master password
- Change the master password using the existing master password
- Change the master password using the credentials of the current admin user
Obviously the second option sounds pretty attractive if you find yourself in the position of needing this master password but not remembering what you originally set it to. Don’t try it – you’ll break your SSO installation!
Here’s the right way to do it:
C:\Program Files\VMware\Infrastructure\SSOServer\utils>rsautil manage-secrets -a change -m ********** -N ********** Master password changed successfully.
And you can use the same tool to confirm that the change worked….
C:\Program Files\VMware\Infrastructure\SSOServer\utils>rsautil manage-secrets -a list Enter Master password: ********* Secrets stored in C:\Program Files\VMware\Infrastructure\SSOServer\webapps\ims\W EB-INF\classes\systemfields.properties. Command API Client User ID ............................: CmdClient Command API Client User Password ......................: ********** SSL Client Identity Certificate Private Key Password ..: ********** SSL Client Identity Certificate Keystore File Password : ********** SSL Server Identity Certificate Private Key Password ..: ********** SSL Server Identity Certificate Keystore File Password : ********** Root Certificate Private Key Password .................: ********** Root Certificate Keystore File Password ...............: ********** SSL Trust Store File Password .........................: ********** The "listkeys" action displays the key names to use when setting the values.
And for educational purposes only here’s the wrong way… Please don’t try this at home…
C:\Program Files\VMware\Infrastructure\SSOServer\utils>rsautil manage-secrets -a change -u admin -p ********** -N ********** Master password changed successfully.
This time if you try again to see if it worked, you’ll soon find it didn’t – even though it said it had.
C:\Program Files\VMware\Infrastructure\SSOServer\utils>rsautil manage-secrets -a list Enter Master password: ********* Error: Failed to decrypt field system.encryption.key
