Upgrading ESXi 6.5 to Update 2

In one of my test labs I have a stand-alone ESXi 6.5 VM server and it was time to make sure it was upgraded and patched.

Since this particular server is not managed by a vCenter, I could not use the Update Manager. I also didn’t have a burner handy, so I decided not to go the bootable ISO method.

The method I went with was using the Offline Bundle.

I found these instructions on Mike Tabor’s blog here. So props to Mike!

First download the VMware vSphere Hypervisor (ESXi) 6.5 Update 2 Offline Bundle from VMware. These instructions should work with future updates as well – so if they have released U3 or later I would use the newer release.

Now onto the actual update:

  1. Upload the ZIP file to a datastore the ESX host can reach – if possible, I would recommend a local datastore but that is not a requirement.
  2. Enable SSH on your ESXi host
  3. Put your ESXi host into Maintenance Mode
  4. Shutdown or migrate all of your active VMs to ensure the host is quiet
  5. SSH into your ESX host
  6. Run the following command. Be sure the path references your datastore and the ZIP file you put there
    esxcli software vib update -d /vmfs/volumes/DATASTORE/update-from-esxi6.5-6.5_update02.zip
  7. Once that completes, you just need to reboot the box
  8. After the reboot, exit Maintenance Mode and power on the needed VMs.

You are now running the latest update.

I hope this guide helps. I know it will help me the next time I need to do an upgrade.

OpenVPN Server on Ubiquity EdgeMax

We had been using pptp and l2tp for VPN services on our corporate router to allow remote users to access our network.  This works quite well on Mac OSX, but is a bit annoying on Windows and Linux workstations.  What to do?  OpenVPN.  It is built in to the EdgeMax, it just needs to be configured.

I used these two guides to give me a majority of the information I needed ForShee OpenVPN on EdgeMax and EdgeMAX – OpenVPN Server with TLS

Here are my deployment steps:

Create a CA
ssh admin@router
sudo -i
cd /usr/lib/ssl/misc
CA.sh -newca

Fill out the requested information including the signing passphrase.  You will need this passphrase to sign certificates later on.  It will create cakey.pem and cacert.pem files for the CA.

Edit /usr/lib/ssl/openssl.cnf ; locate default_days, the default is 365, I upped this to 1095 to match the CA because I don’t want to deal with renewing keys next year.

Create a Server Cert for the VPN server
sudo -i
cd /usr/lib/ssl/misc
CA.sh -newreq
CA.sh -sign

The request will ask a bunch of questions including subject data, hostname, and for a certificate passphrase. You will end up with a request file in the ssl/misc directory named newreq.pem.

The sign command will ask for the CA passphrase from and will confirm that you want to sign this key. Because of the edit you made to the config file, the key should expire in about 3 years.

Create a Diffie-Helman file
openssl dhparam -out /config/auth/dhp.pem -2 2048

Note: this will take some time. It also puts it directly into the config directory so we don’t have to move it later.

Copy the certs to the needed locations
cp demoCA/cacert.pem /config/auth/openvpn.cacert.pem
cp demoCA/private/cakey.pem /config/auth/openvpn.cakey.pem
mv newcert.pem /config/auth/vpn.host.pem
mv newkey.pem /config/auth/vpn.host.key
Update the Server Key

You need to remove the passphrase from the Server Cert so the OpenVPN service can start non-interactively:

openssl rsa -in /config/auth/vpn.host.key -out /config/auth/vpn.host-rmpass.key
mv /config/auth/vpn.host-rmpass.key /config/auth/vpn.host.key
Configure the Router
edit interfaces openvpn vtun0
set description OpenVPN
set hash sha256
set mode server
set openvpn-option "--port 1194"
set openvpn-option --tls-server
set openvpn-option "--comp-lzo yes"
set openvpn-option --persist-key
set openvpn-option --persist-tun
set openvpn-option "--keepalive 10 120"
set server name-server
set server push-route
set server subnet
set tls ca-cert-file /config/auth/openvpn.cacert.pem
set tls cert-file /config/auth/vpn.host.pem
set tls dh-file /config/auth/dhp.pem
set tls key-file /config/auth/vpn.host.key
Configure the Firewall

Only one additional firewall rule is needed:

edit firewall name WAN_LOCAL rule <#>
set description OpenVPN
set action accept
set destination port 1194
set log disable
set protocol udp
Create a Client Key

You can use the built in CA.sh script to create client keys, or you can run the openssl command yourself. Since I am not a big fan of re-entering the same data multiple times, I have opted to use the openssl command so it can be automated. The process of creating a client certificate and key is basically the same as the server key. First you need to create a request, then you need to sign the request.

sudo -i
cd /usr/lib/ssl/misc
openssl req -new \
    -days 1095 \
    -subj "/C=US/ST=MyState/L=MyCity/O=MyCompanyName/CN=Full Username" \
    -keyout full.username.key \
    -out full.username.cert

openssl ca -policy policy_anything -out full.username.pem -infiles full.username.cert

During the request you will have to provide a certificate passphrase. This is the passphrase is what the end-use will type in when the connect to the VPN. We are not going to be removing that passphrase. During the signing part, you will have to provide the CA signing key. Otherwise, you should not have to provide any additional information.

Create an OpenVPN content bundle

When a client installs the OpenVPN client and starts configuration they will need some files including the ovpn config file and the needed keys and certs. Here is my example ovpn client configuration file (note there are customizations required here for each company and each client):

dev tun
proto udp
auth SHA256
resolv-retry infinite
remote vpn.mycompany.example.com 1194
ca openvpn.cacert.pem
cert full.username.pem
key full.username.key
verb 3

My distribution model is to create a directory with the ovpn configuration file, the cacert, and the two user specific files. I then create a zip file of the directory and its contents.

Save and Backup

You can make sure openvpn is running by checking the process list (ps -ef | grep openvpn). Then test to make sure a client can connect and can access your internal services. Your VPN is ready to go. Be sure to commit your changes:


Also, now that it is not just /config/config.boot (which you are backing up right?) You need to make sure to save your certs and keys. I recommend getting the files in here: /config/auth and in /usr/lib/ssl/misc/demoCA. Archive them off the router and put them someplace that gets backed up.

Revoke a Client

What happens when a client is no longer valid? You need to revoke their cert. These steps are bit incomplete and I am trying to get them as accurate as possible. The revocation is fairly easy, locate the correct certificate ID and issue the revoke command.

sudo -i
cd /usr/lib/ssl/misc
less demoCA/index.txt [ locate the key you want to revoke, it will have an ID that looks something like: A24C7101DACBCF83 ]
openssl ca -revoke /usr/lib/ssl/misc/demoCA/newcerts/[certificate id].pem

Even though this particular cert is technically revoked, OpenVPN will not understand or respect that. Getting OpenVPN to understand and respect the revocation list is the missing piece.

Doing a Factory Reset on a Lexmark E260dn

I found incomplete and mildly conflicting instructions on how to do a Factory Reset (including the network) on a Lexmark E260dn that I picked up used.

Here is some, hopefully, helpful info on working with that printer that will save you some time.

To get a printout of Device Statistics and Current network configuration, press the right facing triangle button twice.  This is called the “Continue” button.  It will print 4 or 5 pages.  The last page has the network settings.  In my case, the former owner setup a static IP and a user name and password that were lost to history.

In order to reset the printer, you will need to get it into what Lexmark calls home mode.  Here are those steps:

  1. Turn the printer off
  2. Open the front door, as if you were going to remove the toner cartridge
  3. Press and hold the green arrow continue button as you turn the printer on.  All of the lights will cycle.
  4. Release the button.
  5. Close the front door.  The front panel will now show the home light sequence.  This has the first two lights out, the next three lit – along with the green arrow continue button (four lights total).
  6. Press the green arrow continue button until all of the lights cycle, a printer settings configuration sheet will print.

Now you are ready to do a factory reset.  Press and release the cancel button once.  This will change the lights to only the green arrow continue button.  Press and hold the continue button until all of the lights cycle.  This will perform the factory reset.  Power off and on the printer to access it again.

If you need to reset the network settings (such as changing away from a static IP) then you need to reset the Network NVRAM.

Follow the steps 1 to 5 from above.  If you have already printed out the Configuration guide, there is no need for two.

To reset the NVRAM, you need to cycle through a bunch of menu items.  You do this with the cancel button, one at a time.  On my guide it is the last menu item, when you get there the continue button and the Load Paper button will be blinking – no other lights will be on.  Then press and hold the continue button until all of the lights cycle.  When complete, power the printer off and on again.

After the printer came up, I pressed the continue button twice to print out a new copy of the Device Statistics and Current network configuration.  The last page now shows it having DHCP on my network.  I then browsed to the web portal of the printer and could access the menus that were previously blocked by passwords.

I can now configure this printer how I need to for my network.

Big thanks to Lexmark support who answered my email request for reset information.  They got me very close, I just needed to reset the NVRAM to complete the process.

Hope this helps and good luck with your printer!

Manually Updating/Removing RAID groups on a LSI RAID Controller

We had a semi unique situation where we wanted to remove a bunch of unused hard drives from some LSI server.

Luckily there were two RAID groups (a RAID-1 for OS and a RAID-10 for data) and only wanted to remove the RAID-10 and associated drives.

First up, make sure there is nothing mounted or using those drives. We had a volume group on them so I needed to run commands like these:

root@linux # vgdisplay
root@linux # vgremove data-store
root@linux # vgdisplay

Then I needed to remove the phyical volume:

root@linux # vgdisplay
root@linux # vgremove db-store
root@linux # vgdisplay

Note – for all of my commands below, the controller number is “0” and the EID is “252”. Yours may vary, but this is a pretty common configuration.

Once I was sure nothing was using /dev/sdb – It is time to identify the RAID Volume Group to remove:

root@linux # storcli /c0 show

The Volume Group ID is listed as the DG/VD – be sure to pick the correct one.

root@linux # storcli /c0/v[ID] del
root@linux # storcli /c0 show

Now you can physically remove all of the drives you no longer need. I removed drives 3 to 7 leaving the RAID-1 with drives 0 and 1 and a spare in drive 2.

Making drive 2 a Global Hot Spare is also pretty easy:

root@linux # storcli /c0 /e252 /s2 add hotsparedrive
root@linux # storcli /c0 show all

Password Reminders on Windows Domain

I setup a small domain here at the office recently so we could use it for Single Sign On. Over time, individual services have been converted to use that domain for authentication including things like VPN.

However, most of my users do not actually have workstations or laptops on the domain.  This is because we a a majority Mac shop and the Windows PCs that are in use are mostly using a Home edition.

The first step was to get an email reminder out to everyone. I wrote a scheduled task that runs on my domain controller every morning at 6AM. The script I settled on was written and distributed by Justin Shin at 4sysops.com. It is called PowerPasswordNotify.ps1. It works quite well and only took minor tweaks for our environment.  To fire off the script, I created a batch script that gets run by the Scheduled Task.  The contents of that script are exactly:

"C:Windowssyswow64Windowspowershellv1.0powershell.exe" -executionpolicy Unrestricted -file "C:ApplicationsPPNPowerPasswordNotify.ps1"

The only AD related thing I had to do was make sure that everyone’s email was part of their AD account.

For those pesky Windows workstation users (of which I am one), I wanted to make sure they got a reminder as well.  This can all be done with GPO.

The first GPO I set was the default password reminder. I edited the default policy for my whole domain and navigated in the Group Policy Management Editor to: Computer Configuration -> Policies -> Windows Settings -> Security Settings -> Local Policies -> Security Options .

I then found the “Interactive Logon: Prompt user to change password before expiration” policy.  Double click to edit the policy. I then checked the define this policy setting and set the days to 14 days.

That works, you get a small pop-up in your task bar with a “Your password will expire in X days”. However, it is easily missed. I was looking for it – and it was not easy to see.  So it is time to be more assertive. Time for a pop-up.

The best way to do this is put a VBS script that runs on logon.

I found a good script on the SpiceWorks Community for doing this password reminder pop-up.  On my DC, I created a folder called C:DomainScripts.  I then shared that folder as “DomainScripts$” so it is a hidden share. The permissions were to allow read-only to Everyone.  I could access the share via \mydomain-dc1DomainScripts$ and run my password reminder script.

In order to get it to run on logon, I updated a Default Domain Group Policy.  In the GPO Editor, I navigated to Computer Configuration -> Policies -> Administrative Templates -> System -> Login. Then I found the “Run these programs at user logon” setting and double clicked it. In the dialog, I enabled the policy, then clicked on the “Show…” dialog.  I added the value pointing at the share and script above: “\mydomain-dc1.mydomain.privDomainScripts$password_reminder.vbs”.

Once this policy propagates, users will get a pop-up on logon from the running of the Password Reminder script (assuming they are in the reminder window).

VBS is not a trusted thing – even when it is shared off of the Domain Controller.  The user will get a prompt asking if it is OK to run the VBS script.  We need to set the DC to be a trusted source.

In the GPO Editor, navigate to Computer Configuration ->  Policies -> Administrative Templates -> Windows Components -> Internet Explorer -> Internet Control Panel -> Security Page.  Locate the “Site to Zone Assignment List” policy and double click it.

Enable the policy, then click the “Show…” button. It will be empty at this time. I put two shares in that dialog: “\mydomain-dc1” and “\mydomain-dc1.mydomain.priv”.  Both have a value of “1” which means they are Intranet sites.

Now when the script runs, it will be trusted – because it came from the domain controller – and the user won’t get a warning pop-up before the actual password expire pop-up.

Now I just need to figure out how to get more workstations onto the domain so they will get the reminders so I don’t have to reset people’s accounts. all the time.

Installing ESX 5.1 via a Network Install

My lab was recently given a fairly decent modern server to expand the ESX environment we use for development and QA. The only problem with the box is that it had a broken CD drive. Rather than try and dig out an external CD drive to install ESX, I decided to spend the time to figure out how to use PXE to network install it.

First up is to extract the ISO. I copied the ISO to my network boot server and then mounted it as a loop back device.

mount -tiso9660 -o loop,ro VMware-VMvisor-Installer-5.1.0-799733.x86_64.iso /mnt-loop

Then I used rsync to copy the contents of that ISO to my tftp directory. Unlike the network boots of Ubuntu or CentOS which can use HTTP or NFS – the ESX installer will use tftp to get all of the needed files for the install.

rsync -a --include ".*" /mnt-loop/ /srv/tftp/vmware-5.1.0-799733

I then had to add a menu item into the PXE menus. I already had a Utilities sub-menu, so I added this to that menu:

LABEL ESX-5-1-0-799733 Unattended
MENU LABLE ESX 5.1.0-799733 Unattended
KERNEL vmware-5.1.0-799733/mboot.c32
APPEND -c vmware-5.1.0-799733/boot.cfg ks=
VMware ESX 5.1.0-799733 Unattended Install

You will see a reference to a kickstart (ks) file in the APPEND section. This is a simple kickstart file to make the install an unattended one. It needs to be hosted on a web server the server can get to while it is booting. The contents of my file look like this:

install --firstdisk --overwritevmfs
rootpw password
network --bootproto=dhcp --device=vmnic0

There is one last step. In the extracted ISO directory (/srv/tftp/vmware-5.1.0-799733 in this example), you need to modify the boot.cfg file. The paths of the on-disk version need to be updated. A leading “/” needs to be removed. This sed command will do that for you:

mv boot.cfg boot.cfg.orig
cat boot.cfg.orig | sed -e "s#/##g" > boot.cfg

In addition, the on-disk version of that file is missing one line. Add this line below the “title=” line:


That needs to match the directory name that the ISO was extracted into.

That is your network install server configuration. All that is left to to is to kick off a network install on your new VM server, pick the ESX installer from the menu and let it finish.

When you are done, you have a server using DHCP with the ESX root password of “password”. Do your post installation configuration- setting a static IP, reset the password, and connect up your additional network cables. Then join it to your vCenter and you are ready to start using VMs.

Scripting the Re-Index of JIRA4

The JIRA instance I am in charge of is quite large. Well north of 800,000 issues large.

This means when it is time to re-index that instance it takes quite a while – several hours.  Because of the 24 hour nature of the business, I have a need to re-index it at odd hours – like 1AM on a Sunday. As it turns out, I like to sleep at those same odd hours.

In order to facilitate that, I wrote a script to kick off that re-index automatically. I can then run that script via a crontab.

Here is the script I wrote – you will need to season this to your environment. The USER needs to be a jira-administrator, I use a local JIRA account (versus an LDAP based one). I also connect directly to the tomcat port rather than the Apache re-director running on port 80.


### SETTINGS ###

### COMMANDS ###

GREP=$(grep 'HTTP Status 401 - Basic Authentication Failure - Reason : AUTHENTICATION_DENIED' $LOG_FILE)
if [ -z "$GREP" ];then
  curl -s --cookie $COOKIE_FILE_LOCATION -d "webSudoPassword=$PASSWORD" $WEBSUDO_PAGE_URL --output $LOG_FILE
  curl --cookie $COOKIE_FILE_LOCATION --header "X-Atlassian-Token: no-check" -d "indexPathOption=DEFAULT" -d "Re-Index=Re-Index" -d "indexPath=" $INDEX_PAGE_URL --output $LOG_FILE
  echo "Password is outdated."
  echo "JIRA response: $GREP"


echo $DATE >> /usr/local/jira/bin/reindex.log

Once the script is setup, it can be manually run or can be scheduled via  a cron.

Installing iTunes as light as possible

iTunes is a beast. It does a lot of things- but sometimes you just want a music player.

On my work box I wanted to install iTunes as light as possible.

The first step is to download install file from Apple. Save it to a known location.

Use a program like 7-Zip to unpack the downloaded EXE. It will contain more than half a dozen sub-installers.

Copy the following installers to a different directory:

  • AppleApplicationSupport.msi
  • QuickTime.msi
  • iTunes.msi (or iTunes64.msi) if you are using the 64 bit version
  • Start a command prompt and change to the directory where those three installers are.

    Run the installers in the above order this way:

  • msiexec /i AppleApplicationSupport.msi /passive
  • msiexec /i QuickTime.msi /passive
  • msiexec /i iTunes64.msi /passive
  • The installers will put a QuickTime and an iTunes shortcut on your desktop. Unfortunately, there is no way that I have found to run iTunes without QuickTime.

    The first time you run iTunes it will complain about not having Bonjour and being able to share music. I think this is a good thing in the work environment. It will not complain after that.

    There you go, an install of iTunes that is as light as possible. I can download podcasts, add music to my library, and create play lists.

    Changing User Names in Confluence

    At the office we use Atlassian Confluence as our internal Wiki system. I do like it, but it has some idiosyncrasies. Similar to JIRA there is no internal way to change user names. Here is the SQL needed to update Confuence.

    This is pretty clean SQL because you really only need to update the first two lines.

    SET @oldusername = "OLD_USER_NAME";
    SET @newusername = "NEW_USER_NAME";
    SET @tildedoldusername = CONCAT('~', @oldusername);
    SET @tildednewusername = CONCAT('~', @newusername);
    SET @locoldusername = CONCAT('LOC_', @oldusername);
    SET @locnewusername = CONCAT('LOC_', @newusername);
    -- Attachments
    update ATTACHMENTS set creator = @newusername where creator = @oldusername;
    update ATTACHMENTS set lastmodifier = @newusername where lastmodifier = @oldusername;
    -- Bandana
    update BANDANA set bandanacontext = @newusername where bandanacontext = @oldusername;
    -- Content
    update CONTENT set creator = @newusername where creator = @oldusername;
    update CONTENT set lastmodifier = @newusername where lastmodifier = @oldusername;
    update CONTENT set username = @newusername where username = @oldusername;
    update CONTENT set draftspacekey = @tildednewusername where draftspacekey = @tildeoldusername;
    -- content_label
    update CONTENT_LABEL set owner = @newusername where owner = @oldusername;
    -- update CONTENT_LABEL set spacekey = @tildednewusername where owner = @tildedoldusername;
    -- content_perl
    update CONTENT_PERM set creator = @newusername where creator = @oldusername;
    update CONTENT_PERM set lastmodifier = @newusername where lastmodifier = @oldusername;
    update CONTENT_PERM set username = @newusername where username = @oldusername;
    -- contentlock
    update CONTENTLOCK set creator = @newusername where creator = @oldusername;
    update CONTENTLOCK set lastmodifier = @newusername where lastmodifier = @oldusername;
    -- decorator
    update DECORATOR set SPACEKEY = @tildednewusername where SPACEKEY = @tildedoldusername;
    -- extrnlnks
    update EXTRNLNKS set creator = @newusername where creator = @oldusername;
    update EXTRNLNKS set lastmodifier = @newusername where lastmodifier = @oldusername;
    -- label
    update LABEL set owner = @newusername where owner = @oldusername;
    -- links
    update LINKS set creator = @newusername where creator = @oldusername;
    update LINKS set lastmodifier = @newusername where lastmodifier = @oldusername;
    update LINKS set destspacekey = @tildednewusername where destspacekey = @tildedoldusername;
    update LINKS set destpagetitle = @tildednewusername where destpagetitle = @tildedoldusername;
    -- notifications
    update NOTIFICATIONS set creator = @newusername where creator = @oldusername;
    update NOTIFICATIONS set lastmodifier = @newusername where lastmodifier = @oldusername;
    update NOTIFICATIONS set username = @newusername where username = @oldusername;
    -- os_propertyEntry
    update OS_PROPERTYENTRY set entity_name = @locnewusername where entity_name = @locoldusername;
    update OS_PROPERTYENTRY set string_val = @tildednewusername where entity_name = @tildedoldusername;
    -- pagetemplates
    update PAGETEMPLATES set creator = @newusername where creator = @oldusername;
    update PAGETEMPLATES set lastmodifier = @newusername where lastmodifier = @oldusername;
    -- spacegrouppermissions
    update SPACEGROUPPERMISSIONS set permusername = @newusername where permusername = @oldusername;
    -- spacegroups
    update SPACEGROUPS set creator = @newusername where creator = @oldusername;
    update SPACEGROUPS set lastmodifier = @newusername where lastmodifier = @oldusername;
    -- spacepermissions
    update SPACEPERMISSIONS set creator = @newusername where creator = @oldusername;
    update SPACEPERMISSIONS set lastmodifier = @newusername where lastmodifier = @oldusername;
    update SPACEPERMISSIONS set permusername = @newusername where permusername = @oldusername;
    -- spaces
    update SPACES set creator = @newusername where creator = @oldusername;
    update SPACES set lastmodifier = @newusername where lastmodifier = @oldusername;
    update SPACES set spacekey = @tildednewusername where lastmodifier = @tildedoldusername;
    -- trackbacklinks
    update TRACKBACKLINKS set creator = @newusername where creator = @oldusername;
    update TRACKBACKLINKS set lastmodifier = @newusername where lastmodifier = @oldusername;
    -- os_user and users
    update os_user set username = @newusername where username = @oldusername;
    update users set name = @newusername where name = @oldusername;

    I used this code on about 50 users recently and did not have any problems.

    Here are the steps I did:

    • I took the template above and made copies for each user that needed to be changed. Each was edited as needed.
    • I shutdown Confluence
    • I used the mysql command line utility to connect to my confluence database
    • I loaded each SQL file by hand, which will update the database
    • I relocated the cache which is in the data directory and called index.
    • I restarted confluence and rebuilt the cache

    Everything went pretty well. After I did the user migration, I then integrated LDAP to our AD and every user now has one less password to remember.

    Hope this helps someone.

    Merging PDFs

    I recently had the need to merge some PDF scans into a single document. Since I use Ubuntu as my primary workstation and network storage machine- I wanted to work out a solution there (plus I could script it for future use if needed.

    The tool to use is Ghostscript. On my fairly stock install, I did not have to install anything.

    Here is a command example to combine three PDF files into one:

    docs> gs -q -sPAPERSIZE=letter -dNOPAUSE -dBATCH -sDEVICE=pdfwrite -sOutputFile=outfile.pdf infile01.pdf infile02.pdf infile03.pdf

    The combined file is even smaller than the combined size of the input file, which is great.