In this article, we will learn how cloud computing has changed the way we scale IT infrastructure, why automation is so important, and what role Cloud-Init plays within the big picture.
Hope you enjoy!
Introduction to Cloud Computing and Automation
Brief overview of cloud computing
In the not-so-distant past, the process of scaling IT infrastructure was not only time-consuming, but also cost-intensive.
Teams would have to discuss the needs in compute power, storage and networking connectivity to figure out what hardware to order, that can run their application.
They would then wait, sometimes weeks or months, until the hardware arrived. Admins had to set the machine up, configure the network connection, storage, etc., and could then proceed to install the software that the developers had built.
The process of maintenance and updates was rather slow too, requiring administrators and developers to discuss update strategies and find maintenance windows.
Transitioning to cloud computing has fundamentally changed how building and scaling IT infrastructure works.
Businesses can now subscribe to one or more cloud providers and have next to unlimited access to compute power, storage, and networking bandwidth as a service, enabling teams to start out small and scale as they need.
Upfront investments are not needed anymore, because cloud providers follow the pay-as-you-go model, sometimes breaking down billing periods to seconds, further reducing costs.
Virtual machines get deployed within seconds, networks are attached on the fly, and you can set up a server that is accessible from the internet in a fraction of the time and cost it took a few years ago.
Additionally, you can deploy your servers in any datacenter worldwide to place your services closer to your customers.
Cloud computing also eliminates the work that was necessary to replace failed hard drives, configure network switches, routers, etc.
By making use of cloud computing, businesses are not constrained by the limitations of physical infrastructure, but can operate more agile and cost efficient.
The role of automation in cloud environments
Now, all these benefits don't come without some requirements.
As we've just discussed, scaling IT infrastructure is now an easy task that can be performed through the use of a web UI, as is reconfiguring network routes and expanding storage, among other things.
The problem is that this still creates a bottleneck: Someone would still need to log in, spin up new VMs, configure networking and routing, install software on maybe tens or hundreds of machines, and check that everything works.
This is where infrastructure automation enters the stage.
Running repetitive tasks is more often than not a cause for human error.
We get distracted, miss one line in an installation instruction, forget a parameter added to a command-line script, and problems start piling up.
That's not quite an improvement, compared to how infrastructure was provisioned before cloud computing.
Infrastructure automation takes away the boring, repetitive work. It reduces the chance for human error and generally improves the speed of deployment and configuration.
The first step on the way to automation is where Cloud-Init shines: Provisioning cloud instances in a repeatable and programmatic way.
What is Cloud-Init
Origins and Purpose of Cloud-Init
In an article about Cloud-Init, Soren Hansen writes about the history of Cloud-Init, which started as EC2-init (for Amazons AWS ec2, yes!) as a byproduct of Canonical's work on adding virtualization support to Ubuntu, the famous debian-based Linux operating system, back in 2007/2008.
What started as ec2-init, a way to apply EC2 "user-data" scripts to a VM at boot time, quickly developed into a widely adopted framework.
ec2-init was thus renamed to Cloud-Init in January 2010.
Cloud-Init is now supported by all major cloud providers and even as a way to provision virtual machines on private cloud solutions like VMware, Proxmox or more generally, KVM and LXD.
In short, Cloud-Init is the leading industry-standard for applying configuration to cloud-instances at boot time.
How Cloud-Init works
As briefly mentioned, Cloud-Init uses what's called "user-data" to configure a virtual machine at boot time.
This user-data can be provided by you, the user, or an automation tool that requests the new instances.
Cloud-Init gets started as a systemd Service during the boot process.
It then figures out, what kind of configuration it has to apply, by looking for it in a few common places, that are predefined in Cloud-Init itself.
Once a configuration is found, several different modules get started in a predefined order, each applying a configuration with a very narrow scope.
Benefits of Using Cloud-Init
Looking back, we can now see, how Cloud-Init can immensely speed up the process of setting up a base operating system for us, taking away the necessity of logging in, changing configuration and installing software. This is even more impressive, considering that it can be applied to tens or hundreds of machines at the same time.
But speed is not the only improvement. As was already mentioned, this approach also makes the provisioning of cloud instances repeatable and predictable. A configuration will always be applied based on the exact same user-data, hence will always produce the same machine configuration to start with.
Manual error is reduced to a minimum (given that your user-data script is correct, that is), and manual intervention is only necessary should things go wrong. However, this isn't the case very often.
Common Use Cases
Now, even though Cloud-Init brings with it quite a lot of modules to use, only a few of them are usually used by a broader user base:
Initial System Setup
At the top of the list are probably changing the hostname and network connectivity. Cloud-Init will also generate new host keys, if none are found, and generate a unique machine-id.
Also commonly used (at least from what I know about how I and others use Cloud-Init) is the package update functionality. Nothing is quite as nice as starting on a server instance that is up-to-date with the latest packages installed.
Application installation and configuration
Another neat feature is the installation and configuration of software.
I usually tend to have ag (aka "the silver searcher"), tmux, vim and some other tools installed on every machine I provision.
This could definitely also be handled by Chef, Salt, or Ansible. But it's nice to know, that these things are available to me should I need them before the machine is properly configured.
User account setup
This, again, could be easily done by another configuration management software, but there are pros and cons to each approach.
Let's just keep in mind that Cloud-Init is capable of adding users to your instance and configuring their password, groups, permissions, etc.
Summary
In the last few minutes, we have learned how infrastructure was provisioned and scaled before the rise of cloud computing as we know it today. We discussed the improvements that cloud infrastructure brings for us as system administrators / DevOps Engineers, but also how much businesses can benefit from shorter infrastructure lifecycles, reduced upfront costs and better maintainability.
We have also established the role of automation in modern IT infrastructure and why this principle finds more and more application to the way we work.
Not only that, but we have established Cloud-Init as an integral part of cloud instance provisioning with its now mature approach to instance configuration at boot time and found out how we can benefit from its core strengths in providing repeatable and predictable instance configuration.
In the next article, we will get our hands dirty and take a look at the initial steps of using Cloud-Init, when we deploy a basic web server instance and write our first user-data to configure the instance.
Comments