Tag Archives: ssh

Persistent SSH connections with context!

SSH, the Secure Shell, is an awesome tool. Rather indispensable for somebody like me who has to operate on remote systems. I use it constantly to either run code from a privileged host or log into systems to diagnose problems. My entire cloud of servers is just a terminal session away.

I’m also a huge fan of laptops. I really like being portable with my computer. Partly because I work from home, which means I often work from a coffee shop, or various parts of my home. I don’t have a “workstation” that I’m tied to, and I haven’t for years. I fell in love with the ease in just closing up my laptop and walking outside, or riding my bike to the cafe and opening it back up to continue work right where I left off.

Unfortunately, over time, the ease of transport has lessened, and for good reasons. First up is the VPN, or Virtual Private Network. VPNs allow me as a remote person to securely log into my employer’s network in order to access resources, or SSH into systems. VPNs are ubiquitous now for remote workers. In the good days, my VPN was automatic. If I closed my laptop and relocated within my house, upon opening my laptop the VPN would re-establish itself without my interaction. SSH, with it’s built in ability to re-establish communication would often come back fine, and whatever I was working on, i.e. my context, would be saved. But as time went on, automatic VPNs began to be viewed as insecure. They required stored credentials on my laptop, and it mean that whomever had my laptop had access to these credentials. To combat this, VPNs started using “One Time Passwords“, or OTPs. OTPs come in many flavors, but essentially they combine a Thing You Have (like a number generating physical device) with a Thing You Know (a passphrase only you know) into a unique string of characters. The numbers from the device plus your passphrase. This combo could be used only once to authenticate and after that it was invalid. More secure, but this ended the days of automatically established VPNs, and it often meant that the time it took me to re-establish my VPN went beyond SSH’s ability to recover a connection. Because of this I’d often find myself walking around my house with my laptop open rather than closed, to keep my connections running. Not nearly as cool and convenient of just closing it and walking around.

Of course, this doesn’t consider transitions from my home to a coffee shop. Two problems there, length of time to get to my destination exceeds SSH recovery time, and the local network details will have changed, preventing SSH recovery completely. This means whenever I go somewhere not my home, I have to re-establish my SSH session(s) and recover my context.

Keeping context is a solved problem. There are tools out there that help with this. GNU Screen and Tmux are very popular options. These utilities essentially create a terminal session that is insulated from disconnections. When you reconnect to wherever a screen or tmux session is running, you can re-attach to the session and all your context is back. These tools have been around for a while and work really well, when you remember to do your work inside one of them. However getting to them is still a manual process. I have to wait for my SSH session to finally realize it can’t re-establish my connection, then I have to re-issue the SSH connection command on my local laptop, and once connected I have to re-attach to whatever session I was working on. Not a lot of work, but certainly an annoyance.

What I want is something that will keep my SSH connections persistent. Persistent across network outages or even network relocations. Not only do I want the connection itself persistent, but I want the context within that connection to be persistent as well. I don’t just want my ssh connection to re-establish itself should it timeout, I want to be re-attached to whatever session I was working in.

Thankfully there are a few tools out there that help with this! Mosh and autossh.

Mosh is kind of the new kid on the block, and is rather interesting. It does a few more things than just keep a persistent connection with context. It also does some things which really help with performance (perceived and actual) over slow connections. When you start a mosh session, it uses ssh to connect to the target and starts some software there, software that your local mosh client will use to communicate with. When the network dies or changes, mosh will quickly re-establish communication with the remote software and your terminal acts as if nothing has changed.

I played around a bit with mosh when it first came out and discovered some things I didn’t like about the setup. First, mosh requires new software be installed on  your connection target. This can either be extremely easy, or a nightmare depending on the target, corporate policy, etc… The other thing I really didn’t like about mosh is what it does to your local terminal window. I currently use OSX as my operating system, and within it I use iTerm2 as my terminal emulator. Often I use the built in search function of iTerm2 to find things in scrollback, or I just simply use the touchpad to scroll back my iTerm2 window to read things that have “scrolled off” my screen. These things are quick and natural and useful. Unfortunately the way mosh works, neither of those things are possible. Scrolling back will only show you the things on your terminal from BEFORE you started your mosh session. All that has happened within your mosh session and has scrolled off your screen is lost. Mosh says to use screen or tmux to capture that, and use the scrollback capability of screen or tmux to review or search it. Because of these reasons, I don’t use mosh, although I will say it is really neat, and does feel extremely fast. If I worked more on very laggy connections I may feel different about it.

The other option I mentioned is AutoSSH. AutoSSH is similar to mosh, in that it attempts to re-establish a broken connection, but it is different in a few key ways. First, it’s a pure ssh implementation. It does not require additional software to be installed on the remote host, and it does not attempt any communication over anything other than ssh. It does not however attempt to keep context. All it will do by itself is re-establish an ssh connection to a given remote host. In order to retain context, screen or tmux are needed. Thankfully it is trivial to use screen or tmux in a way that automatically (re)connects to a session. In my case, I use screen. Screen has one important feature over Tmux for me, and that feature is the way it does scrollback. When using screen in a iTerm2 window, anything that scrolls off the screen is still in the “history” of iTerm2, which means I can scroll up with the touch pad, or use iTerm2’s search feature to find things. This does not work when using Tmux, so I have gone with screen.

Screen has the ability to with one action either create a new session, or if the session named already exists, disconnect that session from wherever it may be connected and reconnect it to where you are now. That is accomplished via $ screen -D -R session_name  . This can be added to an execution of autossh, so that when autossh initially establishes your connection, or ever re-establishes your connection, the execution will run:

This is nearly perfect, but it doesn’t seem to react as fast as mosh does to network disconnects and reconnects. This is due to some defaults in autossh, namely how frequently it polls the monitoring port for activity. The default poll time is 600 seconds, which can be quite a long time. I’ve found that a poll time of 5 seconds seems to keep things feeling fast. To adjust this, it’s as simple as adding an environment variable when launching autossh. Also due to a bug one needs to also adjust the time autossh will wait to first start polling a connection.

Now autossh will start monitoring my connection after 5 seconds, and monitor it every 5 seconds for changes. When it reconnects, it will automatically reattach my screen session for context. Any scrollback is still in my terminal window so my local native terminal actions still work, which means I can roam at will without losing my work! Granted, this does require that screen is installed on the remote host, but screen is nearly ubiquitous these days, and hardly ever contentious to get installed if it isn’t already on your remote host.

This setup has made my life more awesome, and I hope it will make your life more awesome too, dear reader. If you have anything to add, or other tricks for this style of work life you’d like to share, please use the comments boxes. They require my approval but I’ll get to them quite quickly!

SSH Key Rotation with Ansible

Introduction to SSH Keys

SSH keys are fantastic things. They provide a 2-part blob of data, a private part and a public part, that can be used to authenticate ssh connections. You keep the private part private, often with a passphrase to “unlock” it, while you can hand out the public part to things like GitHub, compute cloudsother systems that you might wish to connect to via SSH, and remote servers you will ssh to. The public part of your SSH key pair gets stored in a special file that SSH servers on remote systems read, the authorized_keys file. When you connect, your ssh client will provide details about your private key that the remote end can validate against your public key to authenticate you. This is a great convenience over having to provide a password every single time.

This convenience for users is also a necessity for infrastructure administration. SSH is ubiquitous in the Linux world, and the vast majority of administration is accomplished over SSH. Without the ability to use SSH Keys (or similar auth mechanisms) one would not be able to automate actions across many systems easily.

With convenience comes responsibility though. Having a key that an automated process can use to manipulate your fleet of systems is great, but it’s also a pretty juicy attack vector. For that reason it is good practice to rotate your keys often. Rotating keys is the act of replacing the keys you’re currently using with new keys, and removing the ability for old keys to be used to log into your systems.

Rotating keys requires a new key. Creating a new key is fairly simple. Getting the public part of this key out into your fleet, and removing existing public keys is a bit harder. Thankfully we have orchestration and automation tools such as Ansible. The rest of this blog post will discuss how to use Ansible to automate rotating your ssh credentials across your fleet.

Orchestrating SSH Key Rotation

Lets consider the steps necessary to rotate a key:

  1. Create a new key
  2. Add new key to authorized_keys files on your fleet
  3. Test new key
  4. Remove previous keys from authorized_keys files

As stated before, step 1 is simple, and for the sake of this post we’ll assume that this has been completed, and there is a new key-pair, located at ~/.ssh/id_rsa_new and ~/.ssh/id_rsa_new.pub. The private key part is id_rsa_new, the public is id_rsa_new.pub. It’s the pub we need to distribute. For now, we’ll also assume that this key has not yet replaced the existing key, and we can still use the existing key to reach our fleet.

Step 2 is adding the new key to the authorized_keys file. This is where our Ansible playbook will begin. First we need a play header and a couple variables defined to reference the public and private parts of our new key-pair.

Next we’ll need a task to copy the public part of our new key-pair to the remote hosts. For this we will use the authorized_key module. This module allows us to provide a key to add, which we will do.

Now for step 3, we will want to test this new key, to make sure that our new key addition is working. To do this, we will need to direct Ansible to use our new private key when connecting to our servers. We can use a set_fact task to set ansible_ssh_private_key variable to our new private key.

Our next task will make use of this new key when creating the connection (provided ControlPersist is not at play).

The next task is step 4, removing previous keys. Because of our previous task, this step will make use of the new key, and accomplish step 3 along the way.

Currently, the authorized_key Ansible module does not have a method to remove all but the specified ssh key. However I have sent a pull request to accomplish this, by way of the exclusive keyword. The task here will assume that this pull request has merged.

This task looks just like the first task, but with the addition of exclusive=yes. If you don’t want to use the modified authorized_key module, you could make use of the copy module which could get content from the new_pub_key file similar to how authorized_key gets content from the file.

If all has gone well, all that should be left in the authorized_keys file is the public part of our new key-pair. Our new key has been successfully rotated in and the old key is no longer allowed to log in.

Next Steps

There are more things we could do with our playbook. We could automate the creation of the key itself, which would look something like this:

The when conditional here makes sure that only one key is generated, by only running on the first  host. Delegation is also used to make the action happen on the system calling ansible, rather than a remote host.

We could also move the private key file into a location that our local ssh config is prepared to use by default:

Any number of other tasks could be added around these, or specific options to the existing tasks. This blog post is just enough to get you started.


SSH keys are awesome. Anybody using ssh should be using keys. Keys are powerful, and thus need care. Rotate keys frequently and make sure to invalidate old keys. Automation can make this process a lot easier and more reliable.

For convenience, here is a complete playbook code block:

And lastly here is a horn-less unicorn pooping a rainbow I found on photobucket, because this post has been far too serious.