Over the years, we’ve be able to compile a list of the most frequently asked questions regarding our service. This blog post has recently been summarized on our website, so I’ll provide more details here.
Business questions
Q: Why did you get rid of the Jidoteki SaaS?
A: The Jidoteki SaaS we were offering was shutdown sometime in 2015, when we decided to focus on providing a Managed Service for building virtual appliances. The first and most obvious reason, was that we followed demand. Most of our customers didn’t want to, or were unable to do all the “provisioning” work on their own - that is, the step of actually configuring the appliance with Puppet/Chef/Ansible. One of the issues of hosting a SaaS for Jidoteki was that virtual appliance builds required a large amount of memory, disk space, bandwidth, and CPU. The cost was high for us, and would end up being higher for our customers. Downloading large files from our servers in Tokyo introduced latency issues as well. Finally, being a small bootstrapped company meant we needed to avoid supporting servers 24/7 - aka being on-call.
Q: Your services are not exactly cheap. Why?
A: The primary reason to use Jidoteki as a managed service is to offload the most difficult part of going on-prem: creating and updating the appliance, to a team of experts. In 5 years of running this business, we’ve accumulated a plethora of knowledge which we can transfer directly to our customers, in the form of solutions to the most difficult edge-cases, and with an unbelievably fast turnaround time-to-ship. With everything we’ve built so far, depending on the complexity of a customer’s setup, we can typically have a beta virtual appliance ready in less than one week, containing all the features of a professional updateable and secure on-prem virtual appliance. It takes (and took) time to do this work, and so we charge for that.
Q: What’s included in the Setup plan?
A: Other than what’s already listed on the site, the Setup plan is just a consulting contract allowing us to build your first virtual appliance (OVA) from scratch, with all the features required for you to ship to production. It also includes the Ansible scripts to build the custom rootfs (see Technical Questions below). The scripts are openly licensed to you, so you’re free to modify and re-use them as you see fit. Since the Setup plan is only for the initial OVA, you will be left to manage the updates of that OVA on your own, along with the creation of new OVAs. We can provide suggestions on how to create updates and new OVAs on your own, but it’s also a lot of work, which is why we offer the Support and Meta plans.
Q: Are the Support and Meta plans really necessary?
A: A full project is typically completed in one month - from the day we start work, to the day you’re shipping to your customers. During that time, we constantly iterate on the appliance until it’s built exactly as needed. The Support and Meta plans are completely optional, and only really needed once the appliance has been released. The Support plan includes access to our team for help with updating the appliance (creating Update Packages), as well as updating dependencies such as OpenSSL, handling weird edge-cases, and adding new features such as NFS support, etc. We’ll do all that work for you under our Support plan. The Meta plan is a self-serve DIY solution, also known as Jidoteki v3. If you’re comfortable with updating the appliance dependencies, and want to host everything internally, the Jidoteki Meta appliance is the best option. It runs entirely offline, and can easily be integrated into a Continuous Integration system such as Jenkins, to automate the creation of your Update Packages and OVAs. This is a somewhat repackaged version of the original Jidoteki SaaS, although it’s quite different behind the scenes (we got rid of Ruby and NodeJS).
Technical questions
Q: What is the difference between a rootfs and an OVA?
A: The rootfs is a single cpio archive built from scratch, which contains custom init scripts, custom config files, boot scripts, and other things which we build specifically for your virtual appliance. The rootfs also contains all of your app’s software dependencies (ex: nodejs and the node_modules). The rootfs is the only thing that’s really “custom” about your OVA, other than the OVF metadata file - but that’s just a detail. The OVA is just a standard tar archive which contains the OVF file, and 2 vmdk disk images. The first disk image is the bootable disk, which contains among other things: the rootfs, the unmodified base OS (TinyCore Linux), and a vanilla Linux kernel. To update an OVA, you can boot it and simply replace the existing rootfs, but that process is manual and error-prone.
Q: How is Jidoteki better than InstallAnywhere, Bitnami, Replicated, Gravitational?
A: InstallAnywhere is garbage. Garbage in, garbage out. If you enjoy employing (and paying) a full-time engineer just to build your Installer or OVAs, then that software is the way to go, but you’ll deeply regret it in the short and long-term. Bitnami is focused more on creating Open Source stacks - which is admirable since we love Open Source - but their process is not aimed at building secure, hardened, on-prem, small-footprint, easy-to-update appliances the way ours are built. Case in point: their base OS is Ubuntu. Replicated and Gravitational are not really in the same category as us, since they chose to redefine the term “on-premises” to mean “on Amazon’s or Google’s cloud”. They do seem to provide some nice features for management of appliances, and we respect them for that, but their solutions include a hidden “feature”: vendor lock-in. Choosing to work with those closed-source SaaS platforms locks you into their technology, and their vendor platforms, and their pricing models. Finally, they both focus on Docker at the base of the system. If you’re not using Docker and don’t care for it, then you will be forced to use it with them. What we provide - which is a much more sane solution - is a 100% open source platform for your appliance, without Docker, and with a very tiny OS to boot. Every single tool, script and software included in your appliances is open sourced. This allows you to validate that it does what it’s supposed to do (no telemetry or analytics phoning home, no timebombs or illegal data logging). It also provides complete freedom to modify your solution as you see fit, as opposed to being locked to our way of doing things. That in itself is the most valuable tool for any business. Our Jidoteki Meta appliance does contain some closed-source software, but none of it is obfuscated (we distribute the sources), and as our customer, you’re more than welcome to look at the code and suggest improvements/bug fixes if you want. Again this benefit is a long-term one, where even if you don’t “want” to modify our code, if we happen to stop working on Jidoteki Meta, you “can” continue using it.
Q: Why do you dislike Docker and systemd?
A: Garbage in, garbage out.
Q: How do you handle updates when there’s no internet?
A: I don’t want to say it’s our secret sauce, because we’ve blogged about it here, here, here, and here. Essentially, by running the OS entirely in memory, it allows us to atomically update the system’s OS, kernel, and custom rootfs, without affecting the running system. A reboot is required to “activate” the changes, but once that’s done, they are up and running with a fully updated system. To handle offline updates, we package the update files into an encrypted tar package, and distribute that. The package can be used to fully update the system to the latest version, without requiring internet access. If an update happens to fail, it is automatically reverted and “nothing changes” to the running system.
Q: How do you handle database migrations?
A: Since the OS is only “updated” on reboot, all DB migrations should run on boot. We use shmig to handle the DB migrations after a reboot, and it has worked flawlessly for us since the start. It’s your job to ensure the DB migration scripts don’t fail when run. Unfortunately that’s not our area of expertise, but the following answer might help.
Q: Can you revert a failed migration?
A: Of course we can! By default, all our appliances ship with a second hard disk for storing the database and other persistent data. We ensure to include LVM, to allow you to perform a database snapshot right before running the database migration. If the migration fails, you can simply revert the snapshot and go back to a working state. In any case, if a migration fails and leaves the DB in a bad state where your application won’t start, the included (open source) Jidoteki Admin tools will allow your customers to either upload a new Update Package, to debug the appliance, and even login via SSH to perform manual system administration.
Q: Can you do hardware builds?
A: Yes! We also have a set of Ansible scripts to build a system for the Beagle Bone Black and Raspberry Pi family. The final system is almost identical to a virtual appliance, the only difference is rather than having a custom rootfs layered on top of the unmodified OS, we’re forced to merge them all into one rootfs (because the ARM bootloader can’t chainload them). Of course, the binaries are all compiled for the armv7 architecture instead of x86 or x86_64.
Q: What’s the typical size of a final OVA and Update Package?
A: Our latest Jidoteki Meta appliance is 57MB, and Update Packages can vary wildly, but typically between 100KB and 5MB. The Update packages contain binary diffs of the previous OVA files, which allows them to be much smaller than a full OVA. Of course, if you’re updating a Java application and constantly making changes to the file/directory structure, adding new dependencies, etc, then Update Packages can grow quite large - to multiple hundred MBs or more. There’s no rule of thumb for the OVA size, but our default appliance with all the base dependencies, kernel, etc end up at roughly 35MB. Everything else will depend on your application size, and its dependencies.
Misc questions
Q: What does Jidoteki mean?
A: In Japanese it’s written 自動的（じどうてき）which means “automatically”. Our plan has always been to automate the build of virtual appliances, through scripting, continuous integration, etc, so we thought this name was apt even though it sounds a bit weird. We’re also based in Japan.
Q: Is Jidoteki Meta going to be open sourced?
A: Yes, we plan to fully open source it eventually. For the moment we’re releasing parts of it at a time, but we haven’t written full documentation or tests for all the tools yet. It would be irresponsible of us to open source something with no tests and no documentation ;) We would like to fund development of Jidoteki Meta by selling it first, as a “partially” closed source solution, and within the next year or two once the product is mature, we would be happy to release it publicly for the community to enjoy and use/maintain collectively.
Q: What other benefit(s) do you provide?
A: The greatest benefit is that you can focus exclusively on your software, while we (experts) handle your on-prem builds and setup. This has an important impact on your engineering team, since you don’t need to dedicate any resources to something that’s not core to your business. This translates to huge cost savings. Moreover, with your newfound ability to quickly iterate on your appliance and updates, you can now quickly provide updates for your customers, new versions, and new features - which translates to increased revenues. In the end, the cost of working with us will have an important positive financial impact on your business, and as long as that’s true, it’s win-win for both of us.
I hope this long and detailed FAQ answers questions which many of you may already have regarding building on-premise virtual appliances. If you’re interested in getting started with Jidoteki as a managed service, please feel free to contact us, and we’ll be happy to discuss your requirements and see how we can help. Thanks for reading!