My name is Doug Headley and I am a technical consultant at Radically Distinct. As a CISSP certified software engineer, I build software systems focused on security. I'm excited to write about my current side project.

Currently, I am building a home VPS ( Virtual Private Server and sometimes called a cluster ) and I want to write about what it takes to set one up and why you would want to.

This is probably the least technical blog post about the subject.

When trying to figure out where to start first with this home cloud experiment, I decided that I should start by carefully documenting my rationale for embarking on this journey. This is the beginning of a series of blog posts that will focus on making your own cloud service at home for fun and profit.

In this blog post, I will be going over two main points. The first is that there is nothing special about the cloud. The second is that building your own cloud has an important constraint: between the properties of good, fast, cheap.. a project may only have two of them.

When finished this “garage cloud” will be the equivalent to $6k/month of rented cloud infrastructure.

Let’s get started.

There is no such thing as cloud

It used to be that companies would need to have their own servers to host their online presence, create business software, serve data resources for their employees. "The cloud" changed all that. Instead of racks of servers, which required massive cost overhead, a business could use the cloud rent those resources. The cloud is pretty spectacular.

The cloud allowed business to access computing power without investing in IT infrastructure. This is the important part, because a business is renting someone else's infrastructure, the cloud removes the barrier to accessing high end computational power.

Take an example of a business looking to become a franchise. In the past, that company would need 10s of thousands of dollars for specialized computers called servers. A non-trivial amount of office square footage would need to be allocated to house these giant computers. And that’s just the beginning.

Installing giant computers aren’t the end of the story. An IT staff would needed to maintain it all. Developers would need to design and implement the original business goal. Not to mention that the payoff needs to meet the projected business needs. The risk of buying to much or too little can lead to paralysis. It takes a certain amount of savvy to be confident turn a business into a software company. At least, until the cloud.

The cloud made the costs of IT overhead trivial. It also diminished the risk for participating. For that reason it had legitimate hype for many years. It's also why Jeff Bezos' AWS made him the richest man in the world. My goal is not to get rich, my goal is to not give cloud providers all my money.

Fast, cheap, or good… pick two.

I’d like to frame this home cloud project in the context of, “fast, cheap or good… pick two”. The saying means

If it’s good and cheap, it will require lots and lots of time
If it’s fast and good, it’s going to be expensive
If it’s cheap and fast, the quality will be poor

So why do I want to build my own? There are many reasons. For me, it's fun, cheap, private, educational, and skill-building. The cloud providers are good and they are fast. They are cheap compared to creating a in-office VPS, but not for businesses who grow into dependency of vendor lock-in. In contrast, my garage cloud will be cheap and good because I will use best practices and take my time doing so.

Good

Good can’t go fast in the paradigm of “fast, good, cheep.. pick two.” Building a private cloud in the garage is slow and difficult — at least implementing it with best practices. A 5 minute hair cut is fast, it’s not good. A home cluster is "good" if it mirrors a professional business cluster. A professional VPS is good if it:

scales up to meet demand
doesn't require maintenance downtime
secures data from unauthorized users
has snappy performance

The actual technique of how to accomplish this is for another blog post.

Fast

Even with a multitude of do-it-yourself articles, every accomplishment is hard-earned. That is because technology is difficult and I am not trained in cloud, but I do spend most of my time there.

Every single thing I want to build are offered as "services" in any of the cloud providers. It is in these services where fast replaces cheap. Take for instance AWS

Each one of these services is designed to diminish the requirements of an IT professional for a business. However, it's also designed to promote vendor lock-in to their own services. For example, I have never purchased a domain name without being offered to host it for $25/month by that same provider. It costs a fraction of that $25 in operational costs. I'll speak more on this later.

There are many articles out there on how to develop your own cloud. Those same articles only write about their specific itch to scratch. Because the work is complex at any skill level, googling to completion is a long process that involves bridging the needs of the home cluster with the solution the blogger is writing for.

The needs of any business is vastly different from one another. One might simply want a simple online presence. Another might want to securely decimate tax documentation to their employees. Another still might want to track their seed-planting drones in real-time. Every use case is different and so is the complexity in which to create it.

Cloud providers provide a shortcut around complexity by offering services or pre-built infrastructure. Often times it’s necessary to buy both and that’s costs begin to stack up. Generally speaking, the "compute" icon (highlighted in orange in the screen grab above) is capable of doing all the rest of the services listed around it. Compute can host containers, compute can run media services, compute can accomplish block-chain. The reason those other services exist is because they represent another layer of convenience — at a higher premium.

Another reason why my cloud development is not “fast” is that I'm approaching a home cluster from a developer background and not as a Developer Operations professional. In my career, it’s not uncommon to build a piece of software and hand it off to a DevOps team set up the infrastructure in order to make it all work.

Hypothetically, if I want my website to have a search feature that can figure out if a user is misspelling in their search term I have two options: I could buy it or build it. To build it would cost me less money but more time. Let’s examine what that would look like.

To make a user’s search result understand mistyped or misspelled words, I will need a certain kind of database to store a copy of my web content to search against. I will also need a way to keep that database up-to-date with the ever changing content on my site so that the search results reflect the entirety of web content. In this example, I would build out a tool that pushes new content to a database. A separate DevOps team would make sure that database is in working order for me on a server somewhere fixing misspelled words at a snappy pace. While we see teams work this way, there is often room for overlap.

Cheap

According to my calculations, my cluster is (or will be) worth about $6k/month in cloud costs. This is based on the fantastic article on coddinghorror.com. Roughly 32 CPUs on demand; is worth $6k/month? That computation power is roughly the equivalent of 4 powerful macbooks. Any coffee shop in Seattle will have more computing power than that connected to its wifi at virtually any time of day. So why is it so expensive?

The cloud is expensive because data has less distance to travel and travel at faster speed. Also, because their is a premium for setting everything up. Finally, because of the expertise put into the setup, they can offer service level agreements (SLAs) to guarantee quality of service.

A system in the cloud doesn’t connect to a consumer Internet Service Provider (ISP). Cloud providers are hooked into the fastest connections and make less hops to get to where they are connecting to.

I’m using my ISP to serve my content. I hate my ISP, but I hate them less than its competitor.

My garage cloud solution is cheap because it:

is being done by myself as a hobby project
does not have monthly fee outside consumer internet costs
utilizes inexpensive hardware

Cutting the right corners

The Raspberry Pi is a $25 computer and I've been addicted to them for some time. The pi is another example of cheap and good, but not fast. The cost can be kept low because the typical interfaces to interact with the computer. There is no operating system, no hard drive, no monitor or keyboard. It's a computer with the niceties stripped out and is perfect for creating a DIY cloud.

Everything is free of charge.

Open source is free software. Free as in freedom, not free as in, "free beer." Everything I want to accomplish is community driven and available without charge. A non-comprehensive list can be found here on github under the title, awesome-selfhosted.

Is free software good? I argue it's better. Linux is an open source operating system that runs the cloud. The cloud isn't Windows or Mac OS, it's Ubuntu. All cloud providers make money off of free community products.

The fact is, capitalism doesn't promote competition, competition is stomped out by capitalism. Did market competition make Zoom better than Skype? I believe Skype used its anti-competitive practices to remain an awful product long enough that the world was happy to receive an alternative. I'd rant about this further, but it's a digression for another time.

The short lesson is it can be more expensive to have decisions made for you. If a person doesn’t agree with Youtube’s terms of service (TOS), they could host their own peertube instance. Another example similar to Skype is Jitsi. Jitsi is a free alternative to video conferencing — if you have the time and knowledge to set it up.

Experience counts

When architecting complex systems in a business, a group discussion inevitably happens to discuss whether to build or outsource a service. Building takes man hours, outsourcing takes money. This is the essential logic behind fast or cheap.

When building out this garage VPS of Raspberry Pi’s, I am going to lean on my experience to help me figure out the parts I don’t understand yet. DIY means that I’m not being charged for DevOps labor.

Working as a developer at various startups, I’ve ended up wearing a lot of hats. In addition to writing software, it’s been my responsibility to keep the servers online, debug problems in the cloud (including ones I wasn’t at fault for), and striping out systems in favor of completely different systems. The very things that one must do to maintain their own network.

DIY means that I am going to make mistakes. I am certain that I’ll mess something up and have to start all over again with some or all of the systems. Because this project is a hobby to learn on, I’m not worried about falling into gumption traps over mistakes.

Having maintained systems in the past, it is a certainty that I will be adding processes that will keep data-loss at a minimum. Everyone learns the hard way that backing up data first will save on headaches later. I have been using software to automate some processes already. I’ve also been keeping track of best practices in the industry.

Conclusion

This journey is destined to become more technical. I hope you will want to come along.

Normally, I would just, “get to work” and start putting all the pieces together like a mad scientist. In this moment however, I enjoyed making myself write. And not just write, but write out the high level concepts to forge a raison d'être.

Next steps are going to be to enumerate exactly what I want to accomplish. Afterwards, I will set about creating tasks that I can consume to make the work load less daunting. Thank you for reading and claim your cloud!

DIY cloud at home