[author:alex]
Read part 2 of this blog post series.
Update: Read part 3 of this blog post series.
One of the questions I’m often asked is:
“How do I update a virtual appliance in the wild, if it’s behind a firewall and doesn’t have internet access?”.
This is a very good question!!
In this post, I plan to give some technical tips and tricks for doing so, and our recommended best practices to facilitate your life, and your customer’s life down the road.
Naturally, the first reflex is to create a completely new virtual appliance and offer that to your customers. Reality is that’s not always the best approach. The problems with providing a new appliance are:
Immutable?
This is the new term on the block: immutable servers. The idea is to have a server which can be quickly deleted and replaced by a new server, with the ability to move data between them in a very short amount of time (through shared storage or external mounted devices), thus minimizing downtime and foregoing the need to perform “updates” or “upgrades”. I strongly recommend building immutable virtual appliances and packaging your SaaS application to allow for this, but I’ll save those details for another post.
Updating an existing virtual appliance
I believe the best approach is to provide an update package, which could be in the form of a simple tar file, containing everything necessary to update the virtual appliance.
Your update scripts should not make any calls to external servers or services, as this defeats the purpose of running “behind the firewall”. It also poses a security risk for your customers, so we strongly recommend you don’t do it.
If your tar file contains anything sensitive (source code, private packages or scripts), it’s a good idea to encrypt the tar file, and only allow the virtual appliance to decrypt the file.
ProTip: When you first provision the virtual appliance, make sure it contains a decryption key for future update packages.
If you’re using Jidoteki, chances are you probably use a configuration management tool such as Ansible, Puppet or Chef to provision your virtual appliance.
The provisioning is either done using Ansible’s “local” mode, “puppet apply”, “chef-solo”.
Whichever tool you use, make sure it remains installed in the virtual appliance.
Setting up repos
During an update, you will often want to install new .deb or .rpm packages. I strongly recommend configuring a local CentOS or Debian package repository. Since it’s only accessed by the instance, you don’t even need a web server to access the packages.
For CentOS, you will need to create a reference to a local repo, similar to this, in a file “/etc/yum.repos.d/myapp.repo”:
[myapp] name=CentOS-$releasever - MyApp baseurl=file:/opt/myapp/CentOS/$releasever/os/$basearch/ gpgcheck=0 enabled=1
For Debian (Wheezy), you can create a repo, similar to this, in a file “/etc/apt/sources.list.d/myapp.list”
deb file:///opt/myapp/debian wheezy main
Of course, there’s more to it than that when it comes to creating CentOS and Debian repositories, some tutorials can be found here:
With a local repo, you can then use your configuration management tool to add the packages to the repos, and then install them regularly using “yum install <package>” or “apt-get install <package>”.
Bootstrapping the update
When you distribute a virtual appliance, it should already contain a bootstrap script for updating itself. This can be a very generic shell script that runs a command to extract the package, and run the configuration management tool. Here’s a horrible example (don’t copy this):
#!/bin/bash cd /tmp/updates gpg --output update-package.tar --decrypt update-package.tar.gpg --passphrase something tar -xvf update-package.tar
reprepro includedeb wheezy update-package/nginx.deb
reprepro includedeb wheezy update-package/myapp.deb
apt-get install nginx myapp
chef-solo -c update-package/solo.rb -j update-package/solo.json
service nginx restart
exit 0
I know this is a bad example, but you get the idea (I hope?).
Idempotency and rollbacks
Since you’re using a configuration management tool, you should make sure the runs works perfectly each time. If it fails, it should exit gracefully and log the failure somewhere. At no time should the update leave the virtual appliance in an inconsistent or failed state. This will only give you angry customers.
Of course, building idempotent update scripts and ensuring proper rollback is not an easy task, but we’re experts in this, so contact us if you need help. Our best suggestion is to keep your scripts as simple as possible, and add some built-in methods for reverting everything back to its original state (as much as possible)
Jidoteki allows you to test your update scripts multiple times against multiple virtual appliances, very quickly, so make sure you get successful runs prior to distributing an update package to your customers.
Ensuring things go smoothly
When you provide update packages to customers, ensure they understand the risks involved, and make sure they perform backups of their data.
I like the idea of rebooting a virtual appliance after a failed update, and having it fully revert to its previous state prior to the update. This requires a bit of scripting magic, but it will prevent headaches for everyone.
Backdoors
I’m deeply against creating backdoors in software. On the other hand, if all hell breaks loose, the most important thing is for your customers to have access to their data. After all, it’s THEIR data.
With that in mind, a safe backdoor would be to create an unprivileged user which can SSH to the virtual appliance and perform certain administrative tasks, as well as retrieve all their data. This SSH access should only be allowed from within the local network, and can be given on demand (through a private key or password held by you only). This key can also be unique per-customer.
Uploading the update package
If you look at GitHub Enterprise, they use a custom control panel and allow customers to upload their update package through a web UI. This is a wonderful feature, but possibly overkill for your needs.
If you don’t have the luxury of creating a custom control panel UI, you can always revert to uploading and updating an appliance using SCP and SSH. This is the simplest approach, but might be slightly difficult for less advanced users. Make sure you give customers proper documentation on how to perform updates.
With Jidoteki, we decided to create an Open Source control panel, which is accessible via a REST API or web UI. Unfortunately we haven’t had time to implement the web UI yet, and it’s still missing some features, so we didn’t release the code yet. Once the control panel is reasonably useful and functional, we’ll release it to the public so you can include it in your virtual appliance for your customers.
Final thoughts
Updating a virtual appliance in the wild can be very tricky, but as usual this is part of our area of expertise, so feel free to join our chat room #Jidoteki on FreeNode, send us an email or tweet me directly @alexandermensa if you need help.