Read part 1 of this blog post series.
Read part 2 of this blog post series.
We’ve improved our update process once more, and want to share with you our latest Software Update Flow for virtual appliances.
To be or not to be idempotent
Previously, a software update package contained one set of scripts and files for updating a virtual appliance. It worked flawlessly because the Ansible scripts were written to be idempotent. ex: if the /random.data file already existed, the Ansible task would not try to re-create it.
This is fine, but it requires a lot of thought and care when modifying the scripts. You had to rely on tags (a lovely feature of Ansible btw), and be meticulous enough to ensure all the old tags were changed to the newer version (in Part 2 we showed this change from 1.1 to 1.2). This also forced you to match the tags to the version of the update package. That means you had know the version number BEFORE actually creating the package. What?
These requirements were problematic, and they also prevented deviations such as beta updates and test releases. Moreover, tags are not a common feature for Chef or Puppet.
Releases
We’ve discussed versioning and releasing in the past, but we’ve decided to improve on that slightly.
Our concept of releasing a virtual appliance or software update package remains the same. The only difference is the version number system.
Semantic Versioning
Semantic Versioning works and is familiar to most people. Since we’re not specifically dealing with software versions, we had to redefine the meaning of MAJOR, MINOR and PATCH.
An example released appliance might have v1.1.0, and its update package would be v1.1.1. A newly provisioned/released appliance would get v1.2.0, and update packages for that appliance would be v1.2.1, v1.2.2, etc.
If you install a new appliance from scratch and decide to release it, we won’t automatically increment its MAJOR version for you, but you definitely should if it’s for the same application. This means you can start with whatever version you want, ex: v10.
We’ll always increment the MINOR number for each provisioned/released virtual appliance. You can provision all you want but you will only see the MINOR number increase by 1 for an actual release. We think that’s sane.
We’ll always increment the PATCH number for each software update package. This means you might release an update package v1.1.1 and then v1.1.20. Whatever happened in between, well.. nobody needs to know ;)
Why auto-increment PATCH numbers only?
Why do we do this? There’s a high probability you will want to create update packages without necessarily releasing them. Ensuring their PATCH number always increases will help track changes and “what’s out there”. You might also want to create special updates for certain customers. These will all have their own unique version numbers.
Provisioned virtual appliances work differently. You can always send an unreleased provisioned virtual appliance to a customer. The appliance itself has a unique identifier, but the appliance will most likely be useless without a software update package. For this reason, we only increment the MINOR for released appliances.
We only increment the MINOR for released appliances.
Unreleased appliances and updates
We don’t allow you to release a software update package for an unreleased appliance. That wouldn’t make sense and could become really messy. We know there’s a way to do it, but it’s not clean, or simple, and will be a management nightmare. You will hate yourself. Let’s just avoid that rabbit hole altogether.
In the event you create a software update package for an unreleased appliance, we CHANGE the version number to be: 1.0.<PATCH>. This allows you to easily differentiate update packages which are released/unreleased and is great for testing or sending out private beta update packages.
Bundling all the updates (redux)
Our new process automatically grabs the latest released update packages for each MINOR version, and bundles them all into one encrypted tar file.
For example, if you’re creating a software update package for appliance v1.3.0, we’ll bundle the latest released v1.1.x update with the latest released v1.2.x update, and then we’ll create a v1.3.1 update containing all that good stuff.
What about one-off updates?
We’ll let you select exactly which updates to include, but the default IS NOT “latest and greatest”. We aim to keep Jidoteki flexible with intelligent defaults. We know you will be building appliances, provisioning, and testing software update packages. We know the reason is to ensure your customers have a flawless update process. For this reason, Jidoteki will guide you with creating the proper updates for released appliances. That’s what you should care about most. If you have a one-off and decide to create an update package for it, we’ll still allow you to do that though ;)
Jidoteki Admin
If you install the Jidoteki Admin during provisioning, you get the decryption, extraction, and sequential management of these files for free.
This means when a customer uploads the software update package to the appliance, and runs the update, the Jidoteki Admin will handle applying all the updates sequentially, starting with the oldest one.
It seems like a bit of yak shaving, but the process is a common concept, similar to DB migrations, without the ability to skip anything (potentially important) in the middle.
Testing the software update packages
We’ve designed Jidoteki to TEST these update packages on different appliances. This helps you be more confident that a software update package won’t break a customer’s virtual appliance, thus avoiding emergency recovery / data migration.
We’re in the process of releasing this feature soon, and we’ll make sure to let everyone know once it’s ready.
In any case, it’s still important for your customers to perform a full backup or snapshot of their virtual appliance before performing an update.
Your new responsibility
Since you don’t need to worry about tags and version numbers anymore (yes we wrote 1000 words about versioning just to tell you not to worry about it), you have one simple job: Make sure your update scripts can cleanly transition / migrate a customer from one version to the next.
Make sure your update scripts can cleanly transition / migrate a customer from one version to the next.
This should be its own blog post..
To give an example: if you provision a v1.1.0 appliance with Apache, but you decide to provision a v1.2.0 appliance with Nginx instead, then make sure your update package v1.2.1 has the ability to REMOVE Apache before installing Nginx. You should assume the package will be run on a v1.2.0 appliance as well as a v1.1.x appliance. If the update runs on a v1.1.x appliance, you need to remove old artifacts and ensure the updating customer will have an identical (or extremely similar) setup as someone starting fresh with v1.2.0. After applying the update, both customers will be at v1.2.1. Wouldn’t it be nice if they had identical setups? Yes, so make sure to remove Apache (example only).
That is your only job. In fact, that was always a requirement for creating software update packages, but we never specifically mentioned it. DB migrations and removal of artifacts. Remember that.
Sequential updates after reboot
The problem with running sequential updates is the possibility of a server reboot occuring after one update, but before the others. How do you proceed? Do you setup the server to continue running the updates? Do you make it a manual process? This can become quite tricky.
In fact, we’ve somewhat solved that problem by including an init script in the Jidoteki Admin. The init script should be the last to run on boot. It’s something like S99zzzjidoteki. The goal of the init script is to verify the status of the last update, to update the log entry, cleanup old files and continue the update process if needed. This might not always work, so we didn’t include the init script in the Jidoteki Admin just yet. We want to make sure it runs flawlessly on all our supported operating systems rather than releasing a half-baked script which could cause more damage than anything, so please be patient until then.
In the meantime, simply ensure your software update package DOES NOT REBOOT the virtual appliance ;)
Final remarks
We want to make the creation and management of software update packages extremely easy with Jidoteki. If you have ideas on how we can improve our process, please let us know.
If you have difficulty with creating software update scripts to update your appliance, we’ll be more than happy to help you. Feel free to contact us by email, or on Twitter, or even in our chat room #Jidoteki on Freenode IRC.