Category Archives: Ansible

Ansible copying content from one remote system to another

Just a quick tip of you’re trying to do the same thing I was trying to do.

The Problem

I am generating some content on Server A. I want to replicate this content onto Servers B and C.

The Solution

TL;DR: Read files content from Server A. Write files on Servers B and C from those contents.

Ansible provides a couple modules that make this possible. The first one we’ll look at is the slurp module. This module allows you to read in contents of a file from a remote system. Here is an task to read the content, utilizing the run_once mechanism:

This will read each of those files in and save the results in the pki_certs variable. Ansible will only do this once, presumably on the host where this content was generated with a previous run_once task. However the variable data will be assigned to every host to make it easily accessible.

Next we need to write out the content on our other systems. There are a couple things to consider. First, because the files were read via a with_items loop, the registered content is in a list, specifically in pki_certs.results. This is easy enough to deal with because the results list is a list of dictionaries, and the name of the file is part of that dictionary. The filename resides in the item key, while the content resides in the content key. This allows us to template out both the path to be written as well as the content to be written at that path.

The next thing we need to consider is that the slurp module stores content in base 64 encoding. That means when we write it back out, we need to decode it from base 64, otherwise Ansible will happily write out some long strings that look nothing like your file. To decode from base 64, simply use the b64decode filter on the content variable.

The last thing to consider has to do with yaml and whitespace and ansible. This may not come into play with every file, but these files have multiple lines. A somewhat recent change in Ansible means that if the “short form” of task description is used (with key=value parameters) your written out file will have double linefeeds. The simple solution is to use “long form” task syntax as you’ll see below:

Because every host has access to the pki_certs variable this task can run across all of them. You might see a change registered for the first host in the loop, even though it was the source of the content, due to permissions or ownership changes, however subsequent runs will be nice and clean.

Hopefully this helps you out and saves you from spending an afternoon poking around at it like I just did!

SSH Key Rotation with Ansible

Introduction to SSH Keys

SSH keys are fantastic things. They provide a 2-part blob of data, a private part and a public part, that can be used to authenticate ssh connections. You keep the private part private, often with a passphrase to “unlock” it, while you can hand out the public part to things like GitHub, compute cloudsother systems that you might wish to connect to via SSH, and remote servers you will ssh to. The public part of your SSH key pair gets stored in a special file that SSH servers on remote systems read, the authorized_keys file. When you connect, your ssh client will provide details about your private key that the remote end can validate against your public key to authenticate you. This is a great convenience over having to provide a password every single time.

This convenience for users is also a necessity for infrastructure administration. SSH is ubiquitous in the Linux world, and the vast majority of administration is accomplished over SSH. Without the ability to use SSH Keys (or similar auth mechanisms) one would not be able to automate actions across many systems easily.

With convenience comes responsibility though. Having a key that an automated process can use to manipulate your fleet of systems is great, but it’s also a pretty juicy attack vector. For that reason it is good practice to rotate your keys often. Rotating keys is the act of replacing the keys you’re currently using with new keys, and removing the ability for old keys to be used to log into your systems.

Rotating keys requires a new key. Creating a new key is fairly simple. Getting the public part of this key out into your fleet, and removing existing public keys is a bit harder. Thankfully we have orchestration and automation tools such as Ansible. The rest of this blog post will discuss how to use Ansible to automate rotating your ssh credentials across your fleet.

Orchestrating SSH Key Rotation

Lets consider the steps necessary to rotate a key:

  1. Create a new key
  2. Add new key to authorized_keys files on your fleet
  3. Test new key
  4. Remove previous keys from authorized_keys files

As stated before, step 1 is simple, and for the sake of this post we’ll assume that this has been completed, and there is a new key-pair, located at ~/.ssh/id_rsa_new and ~/.ssh/id_rsa_new.pub. The private key part is id_rsa_new, the public is id_rsa_new.pub. It’s the pub we need to distribute. For now, we’ll also assume that this key has not yet replaced the existing key, and we can still use the existing key to reach our fleet.

Step 2 is adding the new key to the authorized_keys file. This is where our Ansible playbook will begin. First we need a play header and a couple variables defined to reference the public and private parts of our new key-pair.

Next we’ll need a task to copy the public part of our new key-pair to the remote hosts. For this we will use the authorized_key module. This module allows us to provide a key to add, which we will do.

Now for step 3, we will want to test this new key, to make sure that our new key addition is working. To do this, we will need to direct Ansible to use our new private key when connecting to our servers. We can use a set_fact task to set ansible_ssh_private_key variable to our new private key.

Our next task will make use of this new key when creating the connection (provided ControlPersist is not at play).

The next task is step 4, removing previous keys. Because of our previous task, this step will make use of the new key, and accomplish step 3 along the way.

Currently, the authorized_key Ansible module does not have a method to remove all but the specified ssh key. However I have sent a pull request to accomplish this, by way of the exclusive keyword. The task here will assume that this pull request has merged.

This task looks just like the first task, but with the addition of exclusive=yes. If you don’t want to use the modified authorized_key module, you could make use of the copy module which could get content from the new_pub_key file similar to how authorized_key gets content from the file.

If all has gone well, all that should be left in the authorized_keys file is the public part of our new key-pair. Our new key has been successfully rotated in and the old key is no longer allowed to log in.

Next Steps

There are more things we could do with our playbook. We could automate the creation of the key itself, which would look something like this:

The when conditional here makes sure that only one key is generated, by only running on the first  host. Delegation is also used to make the action happen on the system calling ansible, rather than a remote host.

We could also move the private key file into a location that our local ssh config is prepared to use by default:

Any number of other tasks could be added around these, or specific options to the existing tasks. This blog post is just enough to get you started.

Conclusion

SSH keys are awesome. Anybody using ssh should be using keys. Keys are powerful, and thus need care. Rotate keys frequently and make sure to invalidate old keys. Automation can make this process a lot easier and more reliable.

For convenience, here is a complete playbook code block:

And lastly here is a horn-less unicorn pooping a rainbow I found on photobucket, because this post has been far too serious.

Wheeeeee!