Derp Ops – The blog of Jesse Keating

Kicking the tires of the Brigade service for Kubernetes

2017-11-14T21:57:24+00:00

Recently, the Azure team announced a new open source service called Brigade. The point of this service is to provide a neat way to execute pipelines of job executions via containers within a Kubernetes system, in response to events from the likes of GitHub or Dockerhub. I thought I'd take it for a test drive on the IBM Cloud Containers Service, which exposes Kubernetes.

Background Knowledge

This post assumes some level of understanding of Kubernetes, (Docker) Containers, and GitHub pull requests.

What's the point?

Why would somebody want a Brigade like service? There are many reasons, but for me the reason that stood out was an easy way to set up a Continuous Integration and Continuous Delivery pipeline. That is, I wanted proposed changes to code I have hosted on GitHub to be tested before I merge it, and once merged, to be deployed somewhere. There are many options available to provide CI/CD services in the (growing) GitHub Marketplace, however being in control of your own service and infrastructure, as well as being able to potentially tie into other event streams, is an appealing thought. Full disclosure, I was recently working on a project to add another option to the GitHub Marketplace for CI/CD, based on the open source Zuul project from OpenStack.

Getting Started

The installation process for Brigade was very easy, which is to be expected from the same team that created Helm. It should be no surprise that the installation is managed via Helm. Pretty much any Kubernetes system will suffice, and for my experimentation I made use of the IBM Containers Service. I could have just as easily used Minikube, however one aspect of the Brigade service is ingesting GitHub webhook events. Having a Kubernetes cluster available that supported LoadBalancer services makes things a bit easier.

Installing the Brigade Service

The installation documentation from Brigade is very straight forward. All I had to do was make sure I had Helm installed on my laptop and my cluster, and that my kubectl client was properly configured to reach my cluster. There are two Helm charts to deal with. One chart defines a project to interact with, such as a GitHub repository. This chart is expected to be used multiple times, once for each repository to be interacted with. As such, there is an example set of values that need to be copied and defined specifically for the repository. In my case, I populated the `myvalues.js` file with details for a GitHub repository I specifically created for this experiment, `j2sol/demobrigade`. I defined a secret phrase that will be used to validate incoming events as having come from GitHub, and supplied a GitHub personal access token so that the Brigade install can act on my behalf when interfacing with the GitHub API. I left the rest of the values alone, and installed the chart:

> helm install --name demobrigade-project ./chart/brigade-project -f myvalues.yaml
NAME:   demobrigade-project
LAST DEPLOYED: Mon Nov 13 10:13:36 2017
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/Secret
NAME                                                            TYPE    DATA  AGE
brigade-463662f2bc06a89c2464d52d1b248602a1085ed2ccf2953df6a8fa  Opaque  6     0s


NOTES:
Your new Brigade project has been created.

This is a brigade project for github.com/j2sol/demobrigade

It will clone the public repo

To check it's state, run:

  helm status demobrigade-project

The other Helm chart provided in the Brigade project repository is the chart for the Brigade services themselves. This is expected to be installed only once. (They advocate one Brigade install per tenant, where a tenant is one or more people responsible for one or more associated repositories. I'd take that a bit further and advocate for independent Kubernetes clusters per Tenant.) For my installation, I needed to tweak one feature, which is the rbac feature. IBM Cloud Container Service's Kubernetes supports Role Base Access Control, and thus my Brigade install needed to account for that. I did not need to edit any files to account for this, I just used an extra option during the helm install:

> helm install --name brigade ./chart/brigade --set rbac.enabled=true
NAME:   brigade
LAST DEPLOYED: Mon Nov 13 10:14:09 2017
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1beta1/RoleBinding
NAME                  AGE
brigade-brigade-api   2s
brigade-brigade-ctrl  2s
brigade-brigade-gw    2s
brigade-brigade-wrk   2s

==> v1/Service
NAME                 TYPE          CLUSTER-IP      EXTERNAL-IP     PORT(S)         AGE
brigade-brigade-api  ClusterIP     172.21.13.101             7745/TCP        2s
brigade-brigade-gw   LoadBalancer  172.21.219.206  169.48.217.138  7744:32586/TCP  2s

==> v1beta1/Deployment
NAME                  DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
brigade-brigade-api   1        1        1           0          2s
brigade-brigade-ctrl  1        1        1           0          2s
brigade-brigade-gw    1        1        1           0          2s

==> v1/Pod(related)
NAME                                   READY  STATUS             RESTARTS  AGE
brigade-brigade-api-57d56ff7f8-rppkh   0/1    ContainerCreating  0         2s
brigade-brigade-ctrl-69ff6bd48f-h49mc  0/1    ContainerCreating  0         2s
brigade-brigade-gw-5bc49bd866-jgd79    0/1    ContainerCreating  0         2s

==> v1/ServiceAccount
NAME                  SECRETS  AGE
brigade-brigade-api   1        2s
brigade-brigade-ctrl  1        2s
brigade-brigade-gw    1        2s
brigade-worker        1        2s

==> v1beta1/Role
NAME                  AGE
brigade-brigade-api   2s
brigade-brigade-ctrl  2s
brigade-brigade-gw    2s
brigade-brigade-wrk   2s


NOTES:
Brigade is now installed!

To find out about your newly configured system, run:

  $ helm status brigade

At this point, all the Brigade pods had been created and the services had also been created. For the next step I needed information about the Brigade Gateway service, easily obtainable from the above output, or by using kubectl:

> kubectl get services
NAME                  CLUSTER-IP       EXTERNAL-IP      PORT(S)          AGE
brigade-brigade-api   172.21.13.101               7745/TCP         1h
brigade-brigade-gw    172.21.219.206   169.48.217.138   7744:32586/TCP   1h
kubernetes            172.21.0.1                  443/TCP          2d

The information we need is the EXTERNAL-IP and the external port, which in this case is `169.48.217.138 32586`. With this information I can moved on to the GitHub repository setup.

Configuring GitHub

My running Brigade service is rather useless without events to act upon. The events I wanted to react to come from GitHub, in the form of webhooks. As certain things happen to a repository on GitHub, it can send data about those events to a set URL. Setting this up requires administrative rights to the repository within GitHub. To set this up, I followed the provided instructions, replacing the Payload URL with data discovered in the above section, and the Secret with the string I added to myvalues.yaml earlier. Since my demo repository is public, I did not go through the steps to set up a private repository.

Upon completing the webhook configuration, GitHub will send an event to the webhook, a ping event. This is visible in the Recent Deliveries section of the webhook configuration page. It should have a little green checkmark next to the UUID of the delivery. This lets me know that my GitHub project is able to communicate with my Brigade service as expected.

Create a brigade.js file

With my Brigade install reacting to GitHub events, it's time to direct it to do something when those events occur. The Brigade architecture relies on a special file within the git repository in question to inform the system what to do. This file is expected to be JavaScript. To start with, a simple statement to log a can be used:

console.log("hello from Brigade")

I created this file and pushed it to a branch. Because I pushed this branch to the repository that is generating webhooks, Brigade got events for the branch creation and push of a commit to the branch. I was able to see this by checking the logs of the gateway pod's container:

> kubectl get pods
NAME                                    READY     STATUS    RESTARTS   AGE
brigade-brigade-api-57d56ff7f8-rppkh    1/1       Running   0          4h
brigade-brigade-ctrl-69ff6bd48f-h49mc   1/1       Running   0          4h
brigade-brigade-gw-5bc49bd866-jgd79     1/1       Running   0          4h

> kubectl logs brigade-brigade-gw-5bc49bd866-jgd79

[GIN] 2017/11/13 - 21:55:45 | 400 |       77.33µs |  10.184.120.244 |  POST     /events/github
Expected event push, got create
[GIN] 2017/11/13 - 21:58:43 | 400 |     451.036µs |  10.184.120.244 |  POST     /events/github

I can also look at the logs of the Brigade controller to get some more information:

> kubectl logs brigade-brigade-ctrl-69ff6bd48f-h49mc

handler.go:16: EventHandler: type=push provider=github commit=c967a7e629167d7672f31b2061e12add0ff6cc9f
handler.go:43: Started brigade-worker-01byvp8jqfn9vggv0exbe24wbx-c967a7e6 for "push" [c967a7e629167d7672f31b2061e12add0ff6cc9f] at -62135596800

If I follow the clues, I can get the logs from the worker pod that was started:

> kubectl logs brigade-worker-01byvp8jqfn9vggv0exbe24wbx-c967a7e6
yarn run v1.3.2
$ node prestart.js
prestart: src/brigade.js written
$ node --no-deprecation ./dist/src/index.js
hello from Brigade
Creating PVC named brigade-worker-01byvp8jqfn9vggv0exbe24wbx-c967a7e6
after: default event fired
beforeExit(2): destroying storage
Destroying PVC named brigade-worker-01byvp8jqfn9vggv0exbe24wbx-c967a7e6
Done in 0.87s.

With this output I can see the string I asked to be logged to the console, hello from Brigade. All the basics are set up and functioning as expected!

Beyond the basics

Simply logging words to the console of a container isn't very useful. One of the basic capabilities of Brigade is to launch containers to do actions upon certain events. For instance, I'd like to validate that my demoapp code passes pycodestyle. To achieve this, I can follow the scripting guide in order to update my brigade.js file to run pycodestyle within a container using the python:alpine image on pull_request events. I've created such a script and opened a pull request. The content of brigade.js is:

const { events, Job } = require("brigadier")

events.on("pull_request", () => {
    var job = new Job("pycodestyle", "python:alpine")
    job.tasks = [
      "pip install pycodestyle",
      "cd /src",
      "pycodestyle demoapp"
    ]
    job.run()
})

Don't worry if you don't know JavaScript, I don't either. I'm just using details from the scripting guide to create my file content. I'm making use of the /src/ directory, as that's where Brigade has made the code from my repository available. The version of code that is checked out is the tip of the pull request. Once again, I can log the controller pod to discover the worker pod that was created in reaction to the pull request event:

> kubectl logs brigade-brigade-ctrl-69ff6bd48f-h49mc

handler.go:16: EventHandler: type=pull_request provider=github commit=4023f592ff7a4afb24d667c720a4b21b4c78b6a4
handler.go:43: Started brigade-worker-01byvsvswfc6j0g9bmebn29963-4023f592 for "pull_request" [4023f592ff7a4afb24d667c720a4b21b4c78b6a4] at -62135596800

The logs for the worker pod show something interesting:

> kubectl logs brigade-worker-01byvsvswfc6j0g9bmebn29963-4023f592
yarn run v1.3.2
$ node prestart.js
prestart: src/brigade.js written
$ node --no-deprecation ./dist/src/index.js
Creating PVC named brigade-worker-01byvsvswfc6j0g9bmebn29963-4023f592
looking up default/github-com-j2sol-demobrigade-pycodestyle
Creating secret pycodestyle-1510614103251-4023f592
Creating Job Cache PVC github-com-j2sol-demobrigade-pycodestyle
undefined
Creating pod pycodestyle-1510614103251-4023f592
Timeout set at 900000
default/pycodestyle-1510614103251-4023f592 phase Pending
default/pycodestyle-1510614103251-4023f592 phase Pending
default/pycodestyle-1510614103251-4023f592 phase Pending
default/pycodestyle-1510614103251-4023f592 phase Pending
default/pycodestyle-1510614103251-4023f592 phase Running
default/pycodestyle-1510614103251-4023f592 phase Failed
FATAL: job pycodestyle(pycodestyle-1510614103251-4023f592): Pod pycodestyle-1510614103251-4023f592 failed to run to completion (rejection)
error handler is cleaning up
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

It appears that the pod created for my job has failed. I'll look at the logs of the failed pod:

> kubectl logs pycodestyle-1510614103251-4023f592
Collecting pycodestyle
  Downloading pycodestyle-2.3.1-py2.py3-none-any.whl (45kB)
Installing collected packages: pycodestyle
Successfully installed pycodestyle-2.3.1
demoapp/__init__.py:4:1: E302 expected 2 blank lines, found 1
demoapp/__init__.py:9:17: E225 missing whitespace around operator
demoapp/__init__.py:12:1: E302 expected 2 blank lines, found 1
demoapp/__init__.py:17:1: E305 expected 2 blank lines after class or function definition, found 1

Looks like my code does not currently pass pycodestyle! Normally I'd fix those as part of the pull request to add the testing, but I want to leave this pull request open for the sake of the blog. I'll open a new one that has the fixed contents on a new branch. This time the worker pod logs show a successful run:

> kubectl logs brigade-worker-01byxzk5nm3vfxvw763a1743xc-c2347081
yarn run v1.3.2
$ node prestart.js
prestart: src/brigade.js written
$ node --no-deprecation ./dist/src/index.js
Creating PVC named brigade-worker-01byxzk5nm3vfxvw763a1743xc-c2347081
looking up default/github-com-j2sol-demobrigade-pycodestyle
Creating secret pycodestyle-1510687220035-c2347081
Creating Job Cache PVC github-com-j2sol-demobrigade-pycodestyle
undefined
Creating pod pycodestyle-1510687220035-c2347081
Timeout set at 900000
default/pycodestyle-1510687220035-c2347081 phase Pending
default/pycodestyle-1510687220035-c2347081 phase Running
default/pycodestyle-1510687220035-c2347081 phase Succeeded
after: default event fired
beforeExit(2): destroying storage
Destroying PVC named brigade-worker-01byxzk5nm3vfxvw763a1743xc-c2347081
Done in 6.98s.

Pipelines

Reacting to events to run tests is a pretty common thing. There's nothing special about it. What really makes Brigade interesting is the ability to run multiple jobs in multiple containers, and to create groups of those jobs which can either run in parallel or serially. There is also the ability to use content and output generated as part of a job in a later job.

For my simple demo repository, I want to set up a functional test of my application, and alter the brigade.js file to run the functional test and the style test in parallel via two containers. Another pull request shows this change. The content of the brigade.js file is now:

const { events, Job, Group } = require("brigadier")

events.on("pull_request", () => {
    var style = new Job("pycodestyle", "python:alpine")
    style.tasks = [
      "pip install pycodestyle",
      "cd /src",
      "pycodestyle demoapp"
    ]

    var functional = new Job("functional", "python:alpine")
    functional.tasks = [
      "apk update && apk add curl",
      "pip install /src/",
      "/usr/local/bin/demoapp &",
      "sleep 2",
      "curl http://127.0.0.1:8000"
    ]

    Group.runAll([style, functional])
})

Viewing the logs from the worker pod that was created for the pull_request event shows the new job names and the pods used. The output also shows how they're running in parallel:

> kubectl logs brigade-worker-01byy57yadgr1khq86q5z6k07g-96842797
yarn run v1.3.2
$ node prestart.js
prestart: src/brigade.js written
$ node --no-deprecation ./dist/src/index.js
Creating PVC named brigade-worker-01byy57yadgr1khq86q5z6k07g-96842797
looking up default/github-com-j2sol-demobrigade-pycodestyle
looking up default/github-com-j2sol-demobrigade-functional
Creating secret pycodestyle-1510693143841-96842797
Creating secret functional-1510693143844-96842797
Creating Job Cache PVC github-com-j2sol-demobrigade-pycodestyle
undefined
undefined
Creating Job Cache PVC github-com-j2sol-demobrigade-functional
Creating pod pycodestyle-1510693143841-96842797
Creating pod functional-1510693143844-96842797
Timeout set at 900000
Timeout set at 900000
default/pycodestyle-1510693143841-96842797 phase Pending
default/functional-1510693143844-96842797 phase Pending
default/functional-1510693143844-96842797 phase Running
default/pycodestyle-1510693143841-96842797 phase Running
default/pycodestyle-1510693143841-96842797 phase Succeeded
default/functional-1510693143844-96842797 phase Running
default/functional-1510693143844-96842797 phase Running
default/functional-1510693143844-96842797 phase Running
default/functional-1510693143844-96842797 phase Succeeded
after: default event fired
beforeExit(2): destroying storage
Destroying PVC named brigade-worker-01byy57yadgr1khq86q5z6k07g-96842797
Done in 13.03s.

I can imagine creating complex sets of jobs within groups to run some tests in parallel, some in serial, more in parallel, while sharing content and output between the sets.

Conclusion

Brigade is an interesting project. It has the capability to create complex workflows in reaction to specific events, and the set up is pretty easy. It may be a better solution than creating your own webhook handler and launching native Kubernetes Jobs. As a project, I think it's in the early stages of development. I ran into a few issues as I was playing around. The project developers have been quite responsive and eager for input. I wouldn't recommend taking Brigade into production just yet, but I would encourage exploring the capability and thinking about potential use cases. I'm sure the developers would love more input on how people would like to use Brigade. The project is open source, so if you want to get your hands dirty I'm sure they'd welcome the help too!

Interactive debugging python code in (mini) Kubernetes

2017-10-26T23:20:10+00:00

Lately I've been playing around with Kubernetes. If you don't know what Kubernetes (k8s) is, then the rest of this post is going to be very confusing to you.

Needed concepts

I'm going to talk about a few things. Here are some links to places to get up to speed should any of this not make any sense:

Background

I'm somewhat late to the k8s game, and I'm still trying to get my bearings. One task I set out to figure out is how I can replicate my development workflow I had built up with Docker Compose to launch containers of my application locally for testing. The application I've been developing on consists of a Zookeeper service and three Python services. I have the Python services broken out into three separate containers based on the same image, but with different launch commands. When testing things locally, I often want to introduce code that I haven't yet committed to a repository, and instead of building new images every time, I make use of a volume mount to bring my code into the container at runtime. This works well with Python thanks to a feature in pip, the tool for installing python packages. I can tell pip to perform an editable install. This type of install makes use of a symlink in the installation target path which links to the source directory of the install. In my Dockerfile I clone the source code to /zuul, and then perform the pip install from there. What this means is that after the install, I could simply alter the files in the original checkout in /zuul and restart the process and the changes will take effect. To expand on that further, this gives me the ability to attach a volume mount from my laptop's zuul checkout directory (where I have edited files) to the /zuul path within the container at start time, so that I can make use of edited files without rebuilding the image.

On to Kubernetes!

My workflow worked great in docker-compose, but now I want to do this with k8s. To demonstrate how this works, I've created a simple Python web server in a demo app. I've put the source for this on GitHub for convenience.

First, I need to install and run minikube, a tool to run a Kubernetes cluster locally on my laptop. Minikube will download some data and launch a virtual machine in which to run the k8s services, including its own Docker daemon.

Build Docker image

With minikube running, I next need to configure my Docker client to make use of the Docker Engine within minikube. A simple command `eval $(minikube docker-env)` will set up my client. Now I can build my Docker image so that it'll be available for use within k8s. I've cloned my repository into a src/derpops/demoapp directory relative to my homer. From there I just need a simple Docker build command to build my image. NOTE! I'm using a tag other than latest so that Kubernetes will not try to pull the latest version of my image. My image will only exist locally, so a pull would fail.

$ docker build -t demoapp:demo .
Sending build context to Docker daemon  129.5kB
Step 1 : FROM python:alpine
 ---> 83da41380580
Step 2 : RUN apk --no-cache add --update     git
 ---> Using cache
 ---> bf6e5c3fd23c
Step 3 : RUN git clone https://github.com/j2sol/demoapp.git /demoapp
 ---> Using cache
 ---> eacba0083289
Step 4 : RUN pip install -e /demoapp
 ---> Using cache
 ---> 3f881178557d
Step 5 : RUN pip install rpdb
 ---> Using cache
 ---> 81845e2506d2
Step 6 : RUN apk del git
 ---> Using cache
 ---> 19d2d2bb91d1
Step 7 : CMD demoapp
 ---> Using cache
 ---> 9eddb5c0209a
Successfully built 9eddb5c0209a

Creating a Deployment

With the image in place, I can now create a k8s Deployment. The Deployment lets me define a container to launch, with the image I built above, and a command to run within the container. I can do this with a simple kubectl command:

$ kubectl create deployment demoapp --image demoapp:demo
deployment "demoapp" created

$ kubectl get pods
NAME                       READY     STATUS    RESTARTS   AGE
demoapp-3399253556-ngkv0   1/1       Running   0          1s

If my deployment is created successfully, a new pod will show up, which will be running my code.

Creating a Service

To see if my code is working properly, I need to be able to reach the web server. A k8s Service is necessary to set up the networks appropriately. Just like with the Deployment, a simple kubectl command will suffice to create the service:

$ kubectl create service nodeport demoapp --tcp=8000
service "demoapp" created

$ kubectl get services
NAME         CLUSTER-IP   EXTERNAL-IP   PORT(S)          AGE
demoapp      10.0.0.232          8000:32369/TCP   1m
kubernetes   10.0.0.1             443/TCP          23h

This command created a service that will allow me to reach port 8000 of the container running the application. My code specifically tells the python library to use port 8000 and to listen on all addresses. This type of service is a nodeport, which makes this port reachable from every node. However, since this is minikube, I only have one node, and I can ask minikube to tell me what the IP address of the node is with the service command. I'll ask it to just display the URL, instead of opening the URL in my browser, and then use curl to access the URL:

$ minikube service demoapp --url
http://192.168.64.3:32369

$ curl http://192.168.64.3:32369
Hi there!%

Injecting new code

My application works, but now I want to alter the code. To get new code into my container, I need to add a volume mount to my deployment.

VolumeMounts are used to expose content into a container. A VolumeMount definition combines the name of a volume and a path to mount it within the container. The name matches a defined Volume, of which many types are supported. The type we're interested in, the hostPath type, exposes a file or a directory from the node (the machine a container is running on). The use of minikube automatically exposes a folder from the host machine minikube runs on into the VM where k8s is running, which is the node. In my case, the /Users directory on my laptop is exposed as /Users on the node, and thus I can make use of this in a hostPath volume.

To add a volume to my Deployment, I'll need to create a yaml file describing my Deployment spec. I can do this quickly by repeating the earlier command but adding arguments to print out YAML:

$ kubectl create deployment demoapp --image demoapp:demo --dry-run -o yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: demoapp
  name: demoapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: demoapp
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: demoapp
    spec:
      containers:
      - image: demoapp:demo
        name: demoapp
        resources: {}
status: {}

I saved this output to a new file, demoapp-deployment.yaml, where I can make adjustments as needed. I need to alter the containers spec for the demoapp container to define a volumeMount:

        resources: {}
        volumeMounts:
          - name: demosource
            mountPath: /demoapp/demoapp
status: {}

This references a volume by the name of demosource which I also need to define in the spec as a new key:

            mountPath: /demoapp/demoapp
      volumes:
        - name: demosource
          hostPath:
            path: /Users/jkeating/src/derpops/demoapp/demoapp
status: {}

These additions will cause the directory /Users/jkeating/src/derpops/demoapp/demoapp to be mounted to the path /demoapp/demoapp within the container when it launches. This will overlay the version of code on my laptop on top of the code that already exists in the container.

To get the new definition of my Deployment in use, I'll delete the existing Deployment and create a new one from the YAML file:

$ kubectl delete deployment demoapp
deployment "demoapp" deleted

$ kubectl create -f demoapp-deployment.yaml
deployment "demoapp" created

$ kubectl get pods
NAME                       READY     STATUS        RESTARTS   AGE
demoapp-2344086337-x6697   1/1       Running       0          1s
demoapp-3399253556-ngkv0   1/1       Terminating   0          30m

The existing pod is being terminated while a new pod is running. If all went well, I should still be able to use curl to reach the web server:

$ curl http://192.168.64.3:32369
Hi there!%

The output is the same as earlier, as I haven't changed any code. But what if I change the value of the variable RESPONSE to something new by editing the file demoapp/__init__.py on my laptop.

        self.end_headers()
        RESPONSE='''Edited code'''
        self.wfile.write(bytes(RESPONSE, 'UTF-8'))

To get my new code in use, I can simply delete the Pod my Deployment created. A Deployment uses a ReplicaSet to control how many Pods are active. Deleting the pod will trigger the creation of a new one, which should pick up my new code. To determine which pod to delete, I'll use the get pods command via kubectl. This will list all the running pods. Then I'll use the delete pod command to delete the `demoapp` pod, and then make sure a new one is created:

$ kubectl get pods
NAME                       READY     STATUS    RESTARTS   AGE
demoapp-2344086337-x6697   1/1       Running   0          11m

$ kubectl delete pod demoapp-2344086337-x6697
pod "demoapp-2344086337-x6697" deleted

$ kubectl get pods
NAME                       READY     STATUS        RESTARTS   AGE
demoapp-2344086337-c6qth   1/1       Running       0          1s
demoapp-2344086337-x6697   1/1       Terminating   0          11m

The new pod should have my new code, which I'll verify with curl once more:

$ curl http://192.168.64.3:32369
Edited code%

Interactive debugging

Getting new code used in the container is fun, but what is even more useful is being able to interactively debug this new code. Python developers should be familiar with the use of pdb, the python debugger. This utility can be used to insert a break point into the source code in order to interactively debug the code at that point in the execution. Pdb is fantastic when you are able to execute the code directly, but requires being attached to the tty that started the python process. That's a difficult feat inside of a system like k8s. Thankfully there is a wrapper around pdb specifically for connecting to remote python processes, called rpdb. When using rpdb, the wrapper will redirect stdout/stdin to a socket handler, which can be accessed over TCP. This will allow me to define a breakpoint in the code and then connect to the socket remotely in order to attach to the debugger to interact with the process. (One downside of rpdb is that it is not part of the standard library, and thus the library will need to be explicitly installed into the python environment. I've done this in my Dockerfile.)

To debug my code, I first have to add the breakpoint to one of my source files. At the appropriate line, I need to insert import rpdb; rpdb.set_trace("0.0.0.0"). I specifically need to tell rpdb to listen on 0.0.0.0 instead of the default 127.0.0.0. This will make rpdb listen on all addresses rather than just localhost. This is less secure, but required in order to reach it through k8s. Once again, I'll edit the demoapp/__init__.py file on my laptop:

        self.end_headers()
        RESPONSE='''Edited code'''
        import rpdb; rpdb.set_trace('0.0.0.0')
        self.wfile.write(bytes(RESPONSE, 'UTF-8'))

I also need to update my k8s Service to expose the new port. Rpdb will listen on port 4444 by default, so I'll use that in my service. First I'll delete the existing demoapp service and then re-create it adding a second port:

$ kubectl delete service demoapp
service "demoapp" deleted

$ kubectl create service nodeport demoapp --tcp=8000 --tcp=4444
service "demoapp" created

$ kubectl get services
NAME         CLUSTER-IP   EXTERNAL-IP   PORT(S)                         AGE
demoapp      10.0.0.145          8000:31661/TCP,4444:30611/TCP   0s
kubernetes   10.0.0.1             443/TCP                         1d

I can test that the new service works by using curl to the new port:

$ curl http://192.168.64.3:31661
Edited code%

Now that the code is edited and the service is created to forward ports through, I can restart the container. Once again I'll delete the running pod to trigger the creation of a replacement.

$ kubectl get pods
NAME                       READY     STATUS    RESTARTS   AGE
demoapp-2344086337-c6qth   1/1       Running   0          17m

$ kubectl delete pod demoapp-2344086337-c6qth
pod "demoapp-2344086337-c6qth" deleted

$ kubectl get pods
NAME                       READY     STATUS        RESTARTS   AGE
demoapp-2344086337-c6qth   1/1       Terminating   0          18m
demoapp-2344086337-mvh0g   1/1       Running       0          0s

Since my breakpoint is inside the do_GET function, I'll need to use curl to initiate a GET request. This will seem to hang in the terminal, as the break point has been reached and execution is waiting. At this point, I should be able to connect to the waiting debugger, using the Service information to determine which port to connect to. In another terminal I can use nc to connect to the debugger! From this point on, it's debugging as usual.

$ curl http://192.168.64.3:31661
_
---------------------------------------

$ nc 192.168.64.3 30611
> /demoapp/demoapp/__init__.py(11)do_GET()
-> self.wfile.write(bytes(RESPONSE, 'UTF-8'))
(Pdb) l
  6  	        self.send_response(200)
  7  	        self.send_header('Content-type', 'text/html')
  8  	        self.end_headers()
  9  	        RESPONSE='''Edited code'''
 10  	        import rpdb; rpdb.set_trace('0.0.0.0')
 11  ->	        self.wfile.write(bytes(RESPONSE, 'UTF-8'))
 12
 13  	def run(server_class=http.server.HTTPServer, handler_class=myHandlers):
 14  	    server_address = ('', 8000)
 15  	    myserver = server_class(server_address, handler_class)
 16  	    myserver.serve_forever()
(Pdb)

From here I can change the value of RESPONSE once more, and then continue execution, which should cause my curl command to return with my new message:

$ curl http://192.168.64.3:31661
Live debugging sure is fun!%

----------------------------------

 16  	    myserver.serve_forever()
(Pdb) RESPONSE='''Live debugging sure is fun!'''
(Pdb) c
_

Conclusion

Kuberetes is a pretty huge leap forward in container orchestration. With that advancement comes some complexity, and a whole lot of new concepts to learn. However, the basic building blocks are there to continue using workflows that have been useful in the past. This workflow to debug code live is just a small example of what is possible with k8s, minikube, and containers in general.

I've added a complete demoapp-deployment.yaml file to the git repository, including a Service definition. Hopefully this example will be useful! As always, comment here or on Twitter should you have any thoughts to share.

Happy kubing!

OpenStack Nova and Hypervisor disk consumption

2016-10-07T23:32:50+00:00

Recently I found myself in a situation at $DAYJOB where I needed to account for the local disk consumption of a nova-compute node, a hypervisor. I had a heck of a time gathering all the information I needed to figure out why the space was being consumed the way it was, and since I couldn't find a single source for all this information I felt it was best to write up a post about it (so that I can find it next time I'm in the same scenario).

This post explores the various ways Nova consumes hypervisor disk space with regard to instance images and booting.

A Nova setup with libvirt/kvm as the hypervisor, ephemeral disk space being provided by a filesystem directory is assumed, and instances booting with ephemeral disk as opposed to Block Storage volumes.

There are two main ways Nova consumes the underlying hypervisor disk space: cached images downloaded from the image service (Glance) and instance ephemeral disk files. All of this content is stored in Nova's state path, which by default is configured for `/var/lib/nova/`. In our setup, we mount a filesystem to Nova's sate path so that we can contain the disk usage to the images and instances that are booted on the system, without risk of filling up `/` or other critical filesystems.

Image cache

Nova make use of an image cache on each hypervisor. This is a place where each image that's used to boot an instance is downloaded to and preserved for a period of time. Each time a new instance is created on a hypervisor, the disk cache is checked to see if the image requested for the instance is already in the cache. If it is, that image is used then as the basis for the instance. If not, the image is downloaded from the image store and placed in the cache. This cache is typically on the same filesystem as where Nova stores the instance data (the state path), and thus the images in the cache will account for some amount of overall disk usage and availability for instances. Images in the cache are held for a period of time determined by configuration. Whether to clean unused cache images is a configuration toggle, along with a minimum age for the image before discarding the image.

The amount of space consumed by an image in the cache depends on details from the image itself. While the listed size of the image in the image store may be small, either due to compression or just overall content, the virtual size of the image may be much larger. This virtual size is used as a value to resize the downloaded image to, as Nova will resize each downloaded image to the match the virtual size.

The virtual size of the image depends on how the image was created, the source of the image, and the options used while creating the image. When using the `qemu-img` tool to create an image, a size can be specified. This will become the virtual size of the image. If creating an image of an existing instance, either by way of an image creation or a backup (which uses the same method), the size of the image will be matched to the size of disk for the flavor the instance uses. If the instance's flavor states a 200G disk, then the image virtual size from that instance will be 200G, regardless of how little space is actually consumed within the instance.

During an instance creation, Nova downloads an image from glance, checks the virtual size of the image, and resizes the file to the virtual size of the image. This file is saved in an `instances/_base/` subdirectory within Nova's state path. The resize creates a sparse file, where the apparent size matches the virtual size, when the actual consumed size may be much lower. Use of the `du` utility can show the difference: `du -h --apparent-size ` vs `du -h `.

Instance ephemeral disk

Each server instance that Nova manages will have its own directory to store data. Part of that data is the ephemeral disk data, the data within the instance itself. The amount of space consumed by the ephemeral disk depends on configuration details of Nova, and on the flavor of the instance.

Copy on write

During an instance creation attempt, Nova will download and resize the base image, if the base image doesn't already exist. Then Nova may either create a copy on write file for the instance, linked to the base image, or copy the entire base image for the instance with no linkage. This decision is based on a configuration entry, `use_cow_images`, which defaults to `True`.

A copy on write file is an overlay file that will overlay on top of the base image file and keep track of any changes to the filesystem within the image.

If copy on write is desired, overlays are created from the base image to the `instances//disk` file within the state path. Otherwise a direct copy will be made to the same path.

Preallocation

In either case, Nova may make a call to pre-allocate enough blocks on that file to be able to fill the size of the flavor's disk. This is determined by a configuration entry `preallocate_image`.

If copy on write is used then the file will appear one of two ways. Without image preallocation, the file will only be as large as the amount of change that has occurred in the file since boot, thus it can be quite small to start with, but may expand to the full size of the flavor's disk size. With image preallocation set, both the apparent and actual size will be the full size of the flavor's disk size.

If a direct copy of the image file is used, then the file will appear in one of two ways. Without image preallocation, then the file will appear exactly as it does in the image cache. The apparent size will can be quite large, but the actual size will be relatively smaller. With image preallocation, both the apparent and the actual size will be the full size of the flavor's disk size.

Launching

Qemu will be launched referencing the disk file in `instances//`, which may or may not be linked to the cached image file. This linkage is what determines whether or not an image file in the cache is still "in use", and will prevent Nova from removing the file when it ages out.

Conclusion

The amount of disk space consumed on a hypervisor depends on numerous factors, such as source image virtual size, the number of active unique images used to boot instances on the hypervisor, and configuration settings regarding disk preallocation and copy on write files. The vast majority of overall consumption of space by Nova will be the sum of all the cached images and all the ephemeral disks for all the instances booted on a given hypervisor.

Configuration items that drive decisions

`preallocate_disk`: Can set to `none` or `space`. If `space`, an `fallocate` call is made on the instance (overlay) disk to allocate enough blocks to cover the flavor disk size. Without preallocating, the underlying hypervisor filesystem can become overcommitted, and if an instance causes enough data change to occur to it's disk file, the host filesystem may become exhausted. An operator could prevent exhaustion by relying on the `DiskFilter` scheduling filter to avoid scheduling instances to where disk has been fully committed, but there are defects and drawbacks to this filter (a subject for a future post). The default value is `none`
`use_cow_images`: Can be set to `True` or `False`. If `True`, the instance's disk file is a copy on write file, attached to the base image in nova's image cache (`instances/_base/`). When this happens, the base image for any booted instance is always held open, and cannot be cleaned. This can drive up the storage overhead on a hypervisor. The default value is `True`.
`remove_unused_base_images`: Can be set to `True` or `False`. If `True`, when a cached image is no longer used by an instance on the hypervisor, and has reached a minimum age, the image will be removed from the cache. This can prevent unbound growth of the image cache on a hypervisor. The default is `True`.
`remove_unused_original_minimum_age_seconds`: An integer of seconds to indicate how old an image file must be before it is a candidate for removal if unused. The default value is `86400`.

Why I love open source!

2015-09-05T16:42:31+00:00

The other day I decided it was time I got familiar with Docker. Yes, I know, I'm a bit late to the party, but better late than never. I understood some of the concepts around Docker, just not necessarily the mechanics, so it was time to dive in.

Docker these days has a handy utility named docker-machine. This tool is used to create a target system to create docker containers on. This is really useful if you're on a Mac and don't have a kernel that supports containers natively. By using docker-machine I was able to provision a VM via VirtualBox that was all set up to run containers. From that point in, docker commands ran as expected, and containers showed up inside the VirtualBox VM.

This is all well and good, but being a curious nerd I wanted to see what other drivers there were for docker-machine. Unsurprisingly, there are numerous drivers, many of them cloud based. There is an OpenStack driver as well, which is great! My day job is all about OpenStack and I have numerous clouds at my disposal. The idea of utilizing my cloud to run docker containers just seems natural to me, so that's what I tried to do.

Unfortunately I ran into a problem. Our clouds work in a way that requires the allocation of a "floating IP" address to an instance in order for that instance to be accessible by the outside world. The docker-machine OpenStack driver supports this, by passing in the correct arguments to tell the driver where to allocate the floating IP from, a pool. It turns out that my account on the cloud I was targeting has admin level rights (a scenario many of my customers will be in), and thus was able to see more available floating IP addresses in the pool than a normal user would, many of which had been allocated to a different project (projects, or tenants in OpenStack are a way to segregate groups of users and resources within a cloud). The docker-machine driver simply attempted to use the first address it thought was available for the instance it just created. In my case, this address had already been allocated to a different tenant and the OpenStack API returned an error when the assignment to my instance was attempted.

I understood the problem, and I had a general idea of how to fix it. The driver should filter the floating IP addresses by my project ID when searching for an available floating IP to use. I essentially had three choices at this point:
Door #1: If the project didn't have a public bug tracker I could give up on the tool or write a negative review somewhere or a snarky tweet about it and find something else to play with.
Door #2: If the project had a public bug tracker I could file a bug in the tracker and explain the scenario that led to the error. I would just have to wait and hope somebody at the project cared enough to fix my bug.
Door #3:Because docker-machine is an open source project, I had a third choice. I could pull down the code and try my hand at fixing the problem myself.

Obviously I went with door #3. I've never looked at the source code behind Docker before (written in Go), but I figured I could fudge my way through a small change. Thankfully the Docker project has spent a fair amount of time thinking about how to make contributing to the project an easy process. Numerous documents exist to help guide a first time contributor through setting up a development environment, understanding the code testing tools, and walking through the submission and review process. Following these guides I was able to start making modifications to the docker-machine code and testing them out on my laptop. What I thought was going to be a simple change turned out to be a more involved code addition, which led me to reading the code and developer documents for a supporting library that docker-machine uses.

Through much iteration and testing, I was finally able to create a change that resolved my issue in a satisfactory way. Being a good open source citizen, I then submitted this change back up to the project in hopes of inclusion in a future release. I myself am not currently blocked in using this tool, but I'd like my customers to be able to use this tool as well, in a way that doesn't require me to distribute a modified binary to them.

This is the real joy of Open Source to me. I found a tool I want to make use of, I discovered a way in which the tool doesn't quite work right for me, I have access to the code to debug the problem, I have access to the documentation and supporting code to develop a solution, and I have the opportunity to contribute a change back to the tool. This process feels so natural to me now that any other way just seems broken. Open Source has enabled me to make my life better, as well as potentially making the life of other users of the software better too, and that gets me right in the feels.

To Do Journaling

2015-08-05T17:37:01+00:00

A co-worker noticed me taking some notes in my notebook the other day, and asked about my To Do journaling (yes I'm making that word up). There was genuine interest from a few other people, so I decided to make a blog post about it.

To Do journaling, to me, is writing down a list of things to accomplish for the day. This is a pretty common practice, and there is nothing really revelatory about the process, simply write the things you want to do down. I started doing this when I couldn't keep track of the things I needed to do in my head any more, and I was letting things fall through the crack. There are many ways in which one could enumerate a to do list; there is a whole industry around tools to help one accomplish this. However, given how much of my life is ruled by electronic means and apps and tools, I have eschewed the "smart" tools, and instead have adopted a more traditional model. I find it helps keep me grounded.

For my journaling, I prefer to use pen and paper. A physical notebook I carry with me in which I can document my to dos for the day, and any sort of notes I need to write down. Writing things down by hand seems to trigger some bit of memory goo in my brain and helps me to remember it. If I type it out, it seems to slip through the synapses and fall out the other side, but writing it down gives it a bit of "stickiness". Because I will spend a fair amount of time touching and feeling and writing in my notebook, I splurge on a high quality notebook. One that's the right size, the right paper feel, and a nice cover. My current favorite is the Rhodia Webnotebook, in black. This notebook is hardcover, which helps me write on a variety of surfaces, but it covered in a very comfortable leatherette. The paper is very nice as well. A good notebook deserves a good pen too. I've gone through a multitude of pens in the past, with various favorites along the way. My current favorite is the Sharpie Grip Pen with a fine black point. These aren't super fancy, but they write nicely (particularly on the Rhodia paper) and feel good in the hand. Additionally they're so cheap that I don't fret if I lose one. To keep my pen and my note pad together, I got a Quiver pen holder. This actually cost more than the notebook and pens combined, but unlike the notebook and pen which are consumables, the Quiver will transfer from note book to note book. This keeps my pen always with my notebook, and adds a nice touch of style to the setup.

Enough about the gear, lets talk about the process. Every day I create a new to do list. Every day is a fresh list, so that I don't have to keep paging back through history to find the list of things to do. I make a To Do XX/XX header to indicate the date, and then start listing things prefixed with a dash.

This is just a simple, unordered list. I typically leave room to the right of the list so that later, after taking various notes below it, if I need to add another item I have room. Having the items unordered also means I don't stress if I have to add an item later, it can go wherever and not upset any ordering. As I start working on something, I'll make a tilde mark next to the list item.

Later, as I finish an item, I'll check it off with a check mark. I do not cross the item out, because I want to be able to go back through my history to remember the things I've done. Crossing an item out makes it that much harder to read (my handwriting is challenging enough).

If at the end of the day I haven't accomplished everything on my list, I draw a square around the incomplete items. This square is a visual cue that lets me know I need to carry the item over into the next day's list. In this way I don't lost track of the things I want to accomplish, but I don't have to flip back to previous days. I just carry over the unfinished items to the next day.

This very simple process has kept me on top of the things I need to accomplish for the past few years. I've filled a number of notebooks this way, and that feels like a great accomplishment. Every time I feel like nothing is happening and my wheels are spinning I can page through them and realize that yes, I am actually getting things done. This process also helps come review time, looking back at past accomplishments. I haven't done this yet, but I could star the items that would be good to highlight in any future self review scenario.

Thanks for reading, and feel free to share your own to do process in the comments, or on twitter.

Ansible copying content from one remote system to another

2014-12-25T00:10:00+00:00

Just a quick tip of you're trying to do the same thing I was trying to do.

The Problem

I am generating some content on Server A. I want to replicate this content onto Servers B and C.

The Solution

TL;DR: Read files content from Server A. Write files on Servers B and C from those contents.

Ansible provides a couple modules that make this possible. The first one we'll look at is the slurp module. This module allows you to read in contents of a file from a remote system. Here is an task to read the content, utilizing the run_once mechanism:

- name: slurp keystone certs
  slurp: src=
  with_items:
    - /etc/keystone/ssl/certs/ca.pem
    - /etc/keystone/ssl/certs/signing_cert.pem
    - /etc/keystone/ssl/private/cakey.pem
    - /etc/keystone/ssl/private/signing_key.pem
  register: pki_certs
  run_once: true

This will read each of those files in and save the results in the pki_certs variable. Ansible will only do this once, presumably on the host where this content was generated with a previous run_once task. However the variable data will be assigned to every host to make it easily accessible.

Next we need to write out the content on our other systems. There are a couple things to consider. First, because the files were read via a with_items loop, the registered content is in a list, specifically in pki_certs.results. This is easy enough to deal with because the results list is a list of dictionaries, and the name of the file is part of that dictionary. The filename resides in the item key, while the content resides in the content key. This allows us to template out both the path to be written as well as the content to be written at that path.

The next thing we need to consider is that the slurp module stores content in base 64 encoding. That means when we write it back out, we need to decode it from base 64, otherwise Ansible will happily write out some long strings that look nothing like your file. To decode from base 64, simply use the b64decode filter on the content variable.

The last thing to consider has to do with yaml and whitespace and ansible. This may not come into play with every file, but these files have multiple lines. A somewhat recent change in Ansible means that if the "short form" of task description is used (with key=value parameters) your written out file will have double linefeeds. The simple solution is to use "long form" task syntax as you'll see below:

- name: write out keystone certs
  copy:
    dest: ""
    content: ""
    owner: keystone
    group: keystone
    mode: 0700
  with_items: pki_certs.results

Because every host has access to the pki_certs variable this task can run across all of them. You might see a change registered for the first host in the loop, even though it was the source of the content, due to permissions or ownership changes, however subsequent runs will be nice and clean.

Hopefully this helps you out and saves you from spending an afternoon poking around at it like I just did!

On next chapters

2014-10-27T17:53:18+00:00

I haven't gotten a lot of sleep lately, I've had quite a lot on my mind.

I've been a "Racker" for almost two years now. I've helped build an amazing thing. The way we expand, deploy, upgrade, operate, and otherwise manipulate OpenStack and the other infrastructure behind the Rackspace Public Cloud is awesome, and getting better all the time. As much as I would love to stay on and continue to intensify the awesomeness, it was time to say good bye. While I've done a lot of the work, I wasn't the only one, and I know that there are a lot of good people still plugged into the mission and it will go on.

What's next? Well, I've been a working remotely for over 6 years now. It has been fantastic to be able to be home and around my family as we've grown. I've tried hard to strike a balance between work travels and home life, but sometimes that balance falls out of whack, particularly when the social side of me that craves in person interaction overrules the empathy side of me that keeps me in check when I'm putting too much on the shoulders of those I leave behind. With our boys growing older, spending quality time with them is more important than ever, which is making it harder and harder to make frequent visits to my Rackspace offices. At the same time, my boys are both now in full time school, which opens up my day time hours, which means it's a great time to think about re-joining the "work from office" folks. I've also been working for large publicly traded companies for nearly 10 years, and I really felt like it was time to get back into the small startup game.

I'm very excited to be joining the Blue Box team as an OpenStack Engineer. I get to keep working on OpenStack stuff, and Ansible stuff, and other fun open source things. They're in Seattle, which gives me an excuse to get on my bike a few days a week and multi-modal commute in, and still strike a good balance with my home life. I also get to plug more firmly into the Seattle tech scene, which exploded with awesomeness since I was last a part of it.

Parting ways is never easy, but the relationships I've built feel strong enough to survive. We'll still see each other at various conferences and meet ups, and we'll all keep working to make OpenStack even better.

Persistent SSH connections with context!

2014-09-05T20:00:04+00:00

SSH, the Secure Shell, is an awesome tool. Rather indispensable for somebody like me who has to operate on remote systems. I use it constantly to either run code from a privileged host or log into systems to diagnose problems. My entire cloud of servers is just a terminal session away.

I'm also a huge fan of laptops. I really like being portable with my computer. Partly because I work from home, which means I often work from a coffee shop, or various parts of my home. I don't have a "workstation" that I'm tied to, and I haven't for years. I fell in love with the ease in just closing up my laptop and walking outside, or riding my bike to the cafe and opening it back up to continue work right where I left off.

Unfortunately, over time, the ease of transport has lessened, and for good reasons. First up is the VPN, or Virtual Private Network. VPNs allow me as a remote person to securely log into my employer's network in order to access resources, or SSH into systems. VPNs are ubiquitous now for remote workers. In the good days, my VPN was automatic. If I closed my laptop and relocated within my house, upon opening my laptop the VPN would re-establish itself without my interaction. SSH, with it's built in ability to re-establish communication would often come back fine, and whatever I was working on, i.e. my context, would be saved. But as time went on, automatic VPNs began to be viewed as insecure. They required stored credentials on my laptop, and it mean that whomever had my laptop had access to these credentials. To combat this, VPNs started using "One Time Passwords", or OTPs. OTPs come in many flavors, but essentially they combine a Thing You Have (like a number generating physical device) with a Thing You Know (a passphrase only you know) into a unique string of characters. The numbers from the device plus your passphrase. This combo could be used only once to authenticate and after that it was invalid. More secure, but this ended the days of automatically established VPNs, and it often meant that the time it took me to re-establish my VPN went beyond SSH's ability to recover a connection. Because of this I'd often find myself walking around my house with my laptop open rather than closed, to keep my connections running. Not nearly as cool and convenient of just closing it and walking around.

Of course, this doesn't consider transitions from my home to a coffee shop. Two problems there, length of time to get to my destination exceeds SSH recovery time, and the local network details will have changed, preventing SSH recovery completely. This means whenever I go somewhere not my home, I have to re-establish my SSH session(s) and recover my context.

Keeping context is a solved problem. There are tools out there that help with this. GNU Screen and Tmux are very popular options. These utilities essentially create a terminal session that is insulated from disconnections. When you reconnect to wherever a screen or tmux session is running, you can re-attach to the session and all your context is back. These tools have been around for a while and work really well, when you remember to do your work inside one of them. However getting to them is still a manual process. I have to wait for my SSH session to finally realize it can't re-establish my connection, then I have to re-issue the SSH connection command on my local laptop, and once connected I have to re-attach to whatever session I was working on. Not a lot of work, but certainly an annoyance.

What I want is something that will keep my SSH connections persistent. Persistent across network outages or even network relocations. Not only do I want the connection itself persistent, but I want the context within that connection to be persistent as well. I don't just want my ssh connection to re-establish itself should it timeout, I want to be re-attached to whatever session I was working in.

Thankfully there are a few tools out there that help with this! Mosh and autossh.

Mosh is kind of the new kid on the block, and is rather interesting. It does a few more things than just keep a persistent connection with context. It also does some things which really help with performance (perceived and actual) over slow connections. When you start a mosh session, it uses ssh to connect to the target and starts some software there, software that your local mosh client will use to communicate with. When the network dies or changes, mosh will quickly re-establish communication with the remote software and your terminal acts as if nothing has changed.

I played around a bit with mosh when it first came out and discovered some things I didn't like about the setup. First, mosh requires new software be installed on your connection target. This can either be extremely easy, or a nightmare depending on the target, corporate policy, etc... The other thing I really didn't like about mosh is what it does to your local terminal window. I currently use OSX as my operating system, and within it I use iTerm2 as my terminal emulator. Often I use the built in search function of iTerm2 to find things in scrollback, or I just simply use the touchpad to scroll back my iTerm2 window to read things that have "scrolled off" my screen. These things are quick and natural and useful. Unfortunately the way mosh works, neither of those things are possible. Scrolling back will only show you the things on your terminal from BEFORE you started your mosh session. All that has happened within your mosh session and has scrolled off your screen is lost. Mosh says to use screen or tmux to capture that, and use the scrollback capability of screen or tmux to review or search it. Because of these reasons, I don't use mosh, although I will say it is really neat, and does feel extremely fast. If I worked more on very laggy connections I may feel different about it.

The other option I mentioned is AutoSSH. AutoSSH is similar to mosh, in that it attempts to re-establish a broken connection, but it is different in a few key ways. First, it's a pure ssh implementation. It does not require additional software to be installed on the remote host, and it does not attempt any communication over anything other than ssh. It does not however attempt to keep context. All it will do by itself is re-establish an ssh connection to a given remote host. In order to retain context, screen or tmux are needed. Thankfully it is trivial to use screen or tmux in a way that automatically (re)connects to a session. In my case, I use screen. Screen has one important feature over Tmux for me, and that feature is the way it does scrollback. When using screen in a iTerm2 window, anything that scrolls off the screen is still in the "history" of iTerm2, which means I can scroll up with the touch pad, or use iTerm2's search feature to find things. This does not work when using Tmux, so I have gone with screen.

Screen has the ability to with one action either create a new session, or if the session named already exists, disconnect that session from wherever it may be connected and reconnect it to where you are now. That is accomplished via $ screen -D -R session_name . This can be added to an execution of autossh, so that when autossh initially establishes your connection, or ever re-establishes your connection, the execution will run:

$ autossh -M 20000 -t host.name 'screen -D -R session_name'

This is nearly perfect, but it doesn't seem to react as fast as mosh does to network disconnects and reconnects. This is due to some defaults in autossh, namely how frequently it polls the monitoring port for activity. The default poll time is 600 seconds, which can be quite a long time. I've found that a poll time of 5 seconds seems to keep things feeling fast. To adjust this, it's as simple as adding an environment variable when launching autossh. Also due to a bug one needs to also adjust the time autossh will wait to first start polling a connection.

$ AUTOSSH_FIRST_POLL=5 AUTOSSH_POLL=5 autossh -M 20000 -t host.name 'screen -D -R session_name'

Now autossh will start monitoring my connection after 5 seconds, and monitor it every 5 seconds for changes. When it reconnects, it will automatically reattach my screen session for context. Any scrollback is still in my terminal window so my local native terminal actions still work, which means I can roam at will without losing my work! Granted, this does require that screen is installed on the remote host, but screen is nearly ubiquitous these days, and hardly ever contentious to get installed if it isn't already on your remote host.

This setup has made my life more awesome, and I hope it will make your life more awesome too, dear reader. If you have anything to add, or other tricks for this style of work life you'd like to share, please use the comments boxes. They require my approval but I'll get to them quite quickly!

SSH Key Rotation with Ansible

2014-06-07T12:14:08+00:00

Introduction to SSH Keys

SSH keys are fantastic things. They provide a 2-part blob of data, a private part and a public part, that can be used to authenticate ssh connections. You keep the private part private, often with a passphrase to "unlock" it, while you can hand out the public part to things like GitHub, compute clouds, other systems that you might wish to connect to via SSH, and remote servers you will ssh to. The public part of your SSH key pair gets stored in a special file that SSH servers on remote systems read, the authorized_keys file. When you connect, your ssh client will provide details about your private key that the remote end can validate against your public key to authenticate you. This is a great convenience over having to provide a password every single time.

This convenience for users is also a necessity for infrastructure administration. SSH is ubiquitous in the Linux world, and the vast majority of administration is accomplished over SSH. Without the ability to use SSH Keys (or similar auth mechanisms) one would not be able to automate actions across many systems easily.

With convenience comes responsibility though. Having a key that an automated process can use to manipulate your fleet of systems is great, but it's also a pretty juicy attack vector. For that reason it is good practice to rotate your keys often. Rotating keys is the act of replacing the keys you're currently using with new keys, and removing the ability for old keys to be used to log into your systems.

Rotating keys requires a new key. Creating a new key is fairly simple. Getting the public part of this key out into your fleet, and removing existing public keys is a bit harder. Thankfully we have orchestration and automation tools such as Ansible. The rest of this blog post will discuss how to use Ansible to automate rotating your ssh credentials across your fleet.

Orchestrating SSH Key Rotation

Lets consider the steps necessary to rotate a key:

Create a new key
Add new key to authorized_keys files on your fleet
Test new key
Remove previous keys from authorized_keys files

As stated before, step 1 is simple, and for the sake of this post we'll assume that this has been completed, and there is a new key-pair, located at ~/.ssh/id_rsa_new and ~/.ssh/id_rsa_new.pub. The private key part is id_rsa_new, the public is id_rsa_new.pub. It's the pub we need to distribute. For now, we'll also assume that this key has not yet replaced the existing key, and we can still use the existing key to reach our fleet.

Step 2 is adding the new key to the authorized_keys file. This is where our Ansible playbook will begin. First we need a play header and a couple variables defined to reference the public and private parts of our new key-pair.

- name: Rotate SSH Keys
  hosts: all
  vars:
    - new_pub_key: ~/.ssh/id_rsa_new.pub
    - new_priv_key: ~/.ssh/id_rsa_new

  tasks:

Next we'll need a task to copy the public part of our new key-pair to the remote hosts. For this we will use the authorized_key module. This module allows us to provide a key to add, which we will do.

    - name: Add new key to authorized_keys
      authorized_key: key=""
                      user=root state=present

Now for step 3, we will want to test this new key, to make sure that our new key addition is working. To do this, we will need to direct Ansible to use our new private key when connecting to our servers. We can use a set_fact task to set ansible_ssh_private_key variable to our new private key.

    - name: Make use of new private key when connecting
      set_fact: ansible_private_ssh_key=

Our next task will make use of this new key when creating the connection (provided ControlPersist is not at play).

The next task is step 4, removing previous keys. Because of our previous task, this step will make use of the new key, and accomplish step 3 along the way.

Currently, the authorized_key Ansible module does not have a method to remove all but the specified ssh key. However I have sent a pull request to accomplish this, by way of the exclusive keyword. The task here will assume that this pull request has merged.

    - name: Make our new key exclusive
      authorized_key: key=""
                      user=root state=present exclusive=yes

This task looks just like the first task, but with the addition of exclusive=yes. If you don't want to use the modified authorized_key module, you could make use of the copy module which could get content from the new_pub_key file similar to how authorized_key gets content from the file.

If all has gone well, all that should be left in the authorized_keys file is the public part of our new key-pair. Our new key has been successfully rotated in and the old key is no longer allowed to log in.

Next Steps

There are more things we could do with our playbook. We could automate the creation of the key itself, which would look something like this:

    - name: Create new ssh key-pair
      command: ssh-keygen -t rsa -N "" -q
               -f 
      when: inventory_hostname == play_hosts[0]
      delegate_to: localhost

The when conditional here makes sure that only one key is generated, by only running on the first host. Delegation is also used to make the action happen on the system calling ansible, rather than a remote host.

We could also move the private key file into a location that our local ssh config is prepared to use by default:

    - name: Move new key into default location
      command: mv  ~/.ssh/id_rsa
      when: inventory_hostname == play_hosts[0]
      delegate_to: localhost

Any number of other tasks could be added around these, or specific options to the existing tasks. This blog post is just enough to get you started.

Conclusion

SSH keys are awesome. Anybody using ssh should be using keys. Keys are powerful, and thus need care. Rotate keys frequently and make sure to invalidate old keys. Automation can make this process a lot easier and more reliable.

For convenience, here is a complete playbook code block:

---
- name: Rotate SSH Keys
  hosts: all
  vars:
    - new_pub_key: ~/.ssh/id_rsa_new.pub
    - new_priv_key: ~/.ssh/id_rsa_new

  tasks:
    - name: Create new ssh key-pair
      command: ssh-keygen -t rsa -N "" -q
               -f 
      when: inventory_hostname == play_hosts[0]
      delegate_to: localhost

    - name: Add new key to authorized_keys
      authorized_key: key=""
                      user=root state=present

    - name: Make use of new private key when connecting
      set_fact: ansible_private_ssh_key=

    - name: Make our new key exclusive
      authorized_key: key=""
                      user=root state=present exclusive=yes

    - name: Move new key into default location
      command: mv  ~/.ssh/id_rsa
      when: inventory_hostname == play_hosts[0]
      delegate_to: localhost

And lastly here is a horn-less unicorn pooping a rainbow I found on photobucket, because this post has been far too serious.

Breaking the cycle - It’s okay to say “I don’t know”.

2014-06-03T17:18:55+00:00

While listening to a recent Freakonomics podcast, I learned that from childhood on there is programming to teach us that it is better to have an answer, even if it is wrong, than to not have an answer at all. Why this happens is not fully understood, but the consequences are becoming more apparent. Maybe my upbringing was different, but I've never seemed to have a problem with stating that "I don't know". But I have ran into people who always seem to have the answer, even if it is wrong.

People in the business world are no exception. There often can be an expectation set, particularly for people in more senior positions, to be experts in our fields, which to some means always having an answer.

Far too often though, we don't have an answer. Our skill isn't in knowing all the answers, our skill is in being able to discover the answer. It's knowing when to ask for help, where to get that help, and how to sort through data to find a solution.

Stating that "I don't know" is not an admittance of failure. Instead it is an invitation for collaboration and discovery. It is the very nature of science and the Scientific Method; starting with an unknown, research to form a hypothesis, test that hypothesis by experimentation, analyze the data and form a conclusion, then communicate the results. Science works when we start with the unknown and journey to the known. The door is open for others to provide ideas and research to the table. Differing opinions are welcome and debated.

When we skip passed the "I don't know" stage and jump straight to an answer, particularly when practicing "fake it until you make it", any differing opinion is viewed as adversarial. This creates conflict rather than collaboration. More effort will be spent to entrench on the wrong answer just to hold up the illusion that one has all the answers. Any evidence that does not support the already chosen answer will be ignored or discounted.

In these situations, a wrong answer asserted with authority, is far more expensive than the answer of "I don't know" and the subsequent period of discovery. Good money and time will be thrown after bad, just to make the wrong answer work, or to find ways to blame other things from preventing the wrong answer from working. This creates a very hostile work environment and does not foster collaboration.

Speaking of discovery, the only method to to learn new things is through feedback. Feedback often comes from the process of trail and error, or experimentation. When it is okay to be wrong, it is also okay to make mistakes and errors. This leads to much clearer data and better understanding of the problem at hand. The ability and encouragement to try new things, even if (and especially if) they fail is a fantastic tool for learning, and ultimately for discovering the best answers to a given question.

In many ways, the ability to admit not knowing the answer to something is a valuable trait, one that should be celebrated, encouraged, and cultivated. An environment should be created and maintained that fosters this behavior. Encourage trail and error. Celebrate failures as data about what doesn't work and use it as data to measure what does work against. The rewards will be many.

I won't claim that this is the best way, I just don't know. What I do know is that organizations I've been a part of that have embraced the culture of discovery have been more successful and more pleasant to be a part of than those which haven't. I encourage those I work with to not be afraid to state "I don't know". It's impossible to know everything about everything. Together we work to find answers to the unknown and the journey is just as important as the solution.