Cloud: Sync, Storage, Bandwidth, Throughput, and more

[author: alex]

These days, the line between Cloud and non-Cloud stuff is quite blurred. Buzzwords confuse people. Furthermore, products which are significantly different tend to group together under the same category, because they employ that magic keyword: Cloud.

Our storage problems & solutions

At Unscramble, we have a huge storage problem.

1. With our Jidoteki™ web service, we find ourselves hosting a large amount of small files (install screenshots: 5 to 10 per install @ 100KB each), and a large amount of big files (OS images: 1 to 4 per install @ 300MB to 3000MB each).
2. We work remotely, so we need to share data in a central location. This can range from not-so-private things such as pictures of a company event, to confidential information such as accounting data.
3. We want to spend the least amount of money as possible.

I think most startups doing new things have similar problems as us, so perhaps I can shed some light on how we solved these things.

For the most part, we own our hardware. We have some pretty big (and redundant) disks. We also don’t have enough users to destroy our bandwidth (yet).

In planning for growth, we quickly realized our solution won’t scale. Mainly because running servers from our living room is not optimal ;) For that reason, we chose to outsource our data storage to places which are better suited for serving data to our users and customers.

Outsourcing the Cloud

At first we put all our files on Dropbox, and that was fine until we recognized the need to maintain full control of our confidential information.

• Solution: Switch to ownCloud. Pat did most of the legwork to get us up and running on there. So far it does a good job of syncing our data and keeping it in a safe location which we have full control of. The fact that we can run our own cloud storage solution on our own hardware is quite nice.

For Jidoteki, we started by uploading files to Linode, which was good and bad for a few reasons:

1. Good: 2TB of bandwidth per Linode (pooled to 10TB since we have more than one Linode)
2. Good: 50Mbit of throughput, which means we can quickly upload large files to each Linode.
3. Bad: Limited disk space, 24GB per Linode hardly enough for the amount of large files we have. They recently upgraded our Linodes to 48GB, and additional disk space can be added, but the max we found was 24GB for $24/mo. That’s crazy expensive for such a small amount of extra space. Bleh.

The limited disk space on Linode was such a huge issue, so we looked into other options: Amazon S3, Dropbox, Mediafire.

Our requirements

Our alternative solutions, although they all claim to be “cloud storage” providers, are all extremely different.

Here is what we were looking for:

• 5TB of bandwidth per month
• 500GB of storage space
• 10Mbit of throughput

We also needed the ability to quickly and easily scale those resources, by either doubling storage space, bandwidth or throughput at any given time.

Cloud offerings

• Amazon S3: Fast throughput, decent data storage prices, but ridiculous bandwidth costs. If you’re doing a lot of high volume data transfer, then S3 is crowned THE MOST EXPENSIVE popular cloud storage platform on earth. I wouldn’t recommend it for that.
• Dropbox: Sure, they have an API, but it’s mostly for sync, not actual storing/serving files to people. In any case, if we were to utilize the API for serving Jidoteki’s OS images, we would do fairly well for a decent price. Unfortunately they don’t have an SLA in regards to throughput, so we could easily be throttled to less than what we need.
• Mediafire: Almost perfect, the price for storage space and bandwidth was just too good to be true. Unfortunately uploading a file to Mediafire through the API would not go over 1.5Mbit/sec. That’s actually quite horrible, a deal-breaker.

Non-Cloud solutions

Dedicated servers. That’s where it’s at. If you take iWeb for example, they offer:

• 12TB of monthly bandwidth
• 2 x 500GB disks
• 100Mbit/sec throughput

This is with the most inexpensive dedicated server offering. Not only that, but since it’s a DEDICATED server, you get all the IO for yourself and don’t have to worry about sharing resources or storing data in the USA (iWeb is Canadian, disclaimer: I used to work there and have many friends still working there. They are great, but I’m a bit biased, sorry).

There are many hosting providers which offer actual hardware, dedicated servers. Their prices are all competitive, but slightly more than most Cloud services.

Why Cloud?

With dedicated servers, you get what you pay for: high speed, large storage space, higher bandwidth limits, dedicated resources. With a bit of planning, the advantages are clear. It’s just a matter of knowing what you need.

Cloud storage hosting providers are great for certain applications, but you need to evaluate them very carefully, and compare costs before you decide to move in that direction.