Updating a virtual appliance in the wild (part 4)

Read part 1 of this blog post series.

Read part 2 of this blog post series.

Read part 3 of this blog post series.

We’ve improved our update process again, but this time we aimed to make it bullet-proof.

Jidoteki prototype builds

For the last year, we’ve been building prototype virtual appliances using a completely different technique. Some of our customers have shipped these appliances in production, and the results are in: it’s AMAZING.

Today, we’ll highlight the changes we’ve incorporated, and our latest bullet-proof update process.

Goodbye idempotent, hello immutable

Idempotent is so last year. It’s a known fact that we’re Ansible fanatics, as it allows us to provision and update appliances idempotently - with a guaranteed final result.

The problem which plagued us was “what happens when an update fails due to circumstances outside of our control?” ex: offline/on-premises appliance crashes halfway through an update?

We needed a better solution.

In 2015, we designed a system which is completely immutable, and allows fail-proof updates.

Immutable means all system changes are reverted upon reboot.

Fail-proof means the update is pretty much atomic with a tiny 0.002ms “do not reboot” window.

Moreover, we’ve improved our update scripts and packages to only allow signed update packages (for security), and to contain binary diffs (deltas) instead of full update files. An update package which used to be 200MB can be reduced to just a few KB.

Here’s how it works:

Explanation

• On boot, the OS gets extracted to memory. To achieve this, we’ve standardized our builds on a very small footprint (12MB) base operating system
• The OS is a CPIO archive, and resides on the first (boot) partition
• Custom configuration files are stored on the second partition (data), for persistence
• The update package is decrypted, its signature is verified, and the contents are extracted to a temporary location
• The update script sorts and parses a manifest file, which lists all the files to be updated
• A loop validates whether each file in the manifest is OK to be updated or should be SKIPPED, based on the appliance’s version
• Each file to be updated is merged (binary diff) into the original source, and then copied to the first partition
• At the end of all the merging, a rename of the updated files into the original source is made (Linux’s mv is atomic if both files are on the same filesystem)
• A reboot of the system will boot the OS from the updated CPIO archive

Watch the video

Major win

The process went from taking nearly 20 minutes, to just a few seconds, with a guarantee of success.

Of course, there are many more details to our update process, but you’ll need to be our customer to find out ;)

Finally, we’ve also deployed this system and our update process on Raspberry Pi, and Beagle Bone Black devices.

We feel confident to say that our on-premises virtual appliances and hardware appliances are the most solid, robust, flexible, secure, and bullet-proof in the industry - and we can prove it.

If you’re a SaaS provider with a great cloud-based solution, please contact us so we can help you build and ship your on-prem product as quickly as possible.