VMware View 5.1, SSL Certificates, and Mac OSX

Posted by Kenneth Ristau on August 4, 2012
Posted in: Mac OSX, Technology, Virualization.

The Setup

The VMware View 5.1 Installation Guide recommends replacing the default self-signed SSL certificates on all servers (Connection, Security, and Composer) with a certificate signed by a Certificate Authority (CA). For the externally-facing Security server role, you should purchase a signed cert from an established CA provider. For your internal Connection and Composer servers, however, it makes more sense to deploy an internal CA.

The other day, myself and a co-worker ran into a situation where we had configured Microsoft’s CA server on a Windows Server 2008 Enterprise server, but were having issues getting the Connection servers to connect to it and generate a certificate request (CSR). After spending too much time trying to get past the RPC error, I decided to by-pass that process by using openssl on an ancient MacBook Pro to generate the CSR.

The following is an account of the process I used, noting some of the pitfalls that hung me up along the way and providing references to Web sites which were helpful.

The Procedure

  1. Generate the CSR on the Mac:
    1. Generate an RSA key by issuing: openssl genrsa -aes128 -out server1.key 2048
    2. Generate the CSR using that key: openssl req -new -key server1.key -out server1.csr
    3. Answer the questions during the CSR generation, making sure to enter the FQDN of the connection server in the Common Name field.
  2. Sign the CSR using the Microsoft CA’s Web interface:
    1. Connect to http://<CA Server fqdn>/certsrv
    2. Select “Request a certificate”
    3. Select “advanced certificate request”
    4. Select “Submit a certificate request using a base-64-encoded CMC. . .”
    5. The next form will allow you to copy and past the text of the server1.csr file into it.
    6. You can use the Web Server Certificate Template, or create a custom template earlier on the CA.
    7. Click Submit.
    8. Download the Base 64 encoded certificate (don’t need the whole chain).
  3. Generate a .pfx file on the Mac:
    1. Combine the .csr and .crt files into a .pfx: openssl pkcs12 -export -in server1.crg -inkey server1.key -name vdm -passout pass:<password> -out server1.pfx
    2. The key here is the ‘-name vdm‘ option which sets the friendly name so that View will use this certificate.
  4. Install the .pfx file on the View Connection server:
    1. Transfer the .pfx file from the Mac to the View Connection server. smbclient on the Mac works well for this.
    2. Open the Certificates (Local Computer) -> Personal -> Certificates snap-in in the mmc.
    3. Import the .pfx certificate. It will prompt you for the password you gave during generation of the .pfx.
    4. Make sure to check “Mark this key as exportable. . .”
    5. Also, make sure the internal Microsoft CA server is imported as a Trusted Root Certification Authority.
    6. If the self-signed certificate with the Friendly name of vdm is still present, change its Friendly name to something else so that the View server only sees one cert with this Friendly name.
  5. Reboot the View Connection server. If you just restart the services, the new certificates may not get picked up by View. I’ve had better success simply rebooting.

The Pitfalls

Here are some of the ways I messed up along the way, causing myself more grief than was necessary:

  1. Spent too much time troubleshooting the RPC issue. While using the CA server Web interface would have made generating the .csr file easier, it wasn’t that much more difficult to create the .csr on my Mac. I still need to fix the RPC issue, but this work-around helped to make progress.
  2. At first, I skipped the encapsulation of the signed certificate and the private key into a .pfx. After reviewing some of the other blogs which step through this process, I realized I was missing the prompt during import for the private key password.
  3. Perhaps I was just impatient, but simply re-starting the VMwareVDMDS service didn’t result in a recognized, valid certificate. Rebooting the View server resulted in the certificate being recognized as soon as the services came up.

The References

  1. Start with this site, as it gives a very good step-by-step process.
  2. This site shows the steps to generate the CSR using openssl for Windows. The commands don’t translate to a Mac, but the rest of the steps are spot on. The openssl commands for generating the .pfx file, however, do work on the Mac version of openssl.
  3. This site has the proper options for generating the CSR on a Mac with openssl. Note that I used -aes128 instead.
  4. VMware View 5.1 documentation on generating the certificates was also helpful in steering me in the right direction.