Derp Ops -- The blog of Jesse Keating

A computer, bike, and climb nerd living in the Pacific Northwest. I do cloudy things with computers, currently at GitHub.

View on GitHub
14 November 2017

Kicking the tires of the Brigade service for Kubernetes

by Jesse Keating

Recently, the Azure team announced a new open source service called Brigade. The point of this service is to provide a neat way to execute pipelines of job executions via containers within a Kubernetes system, in response to events from the likes of GitHub or Dockerhub. I thought I'd take it for a test drive on the IBM Cloud Containers Service, which exposes Kubernetes.

Background Knowledge

This post assumes some level of understanding of Kubernetes, (Docker) Containers, and GitHub pull requests.

What's the point?

Why would somebody want a Brigade like service? There are many reasons, but for me the reason that stood out was an easy way to set up a Continuous Integration and Continuous Delivery pipeline. That is, I wanted proposed changes to code I have hosted on GitHub to be tested before I merge it, and once merged, to be deployed somewhere. There are many options available to provide CI/CD services in the (growing) GitHub Marketplace, however being in control of your own service and infrastructure, as well as being able to potentially tie into other event streams, is an appealing thought. Full disclosure, I was recently working on a project to add another option to the GitHub Marketplace for CI/CD, based on the open source Zuul project from OpenStack.

Getting Started

The installation process for Brigade was very easy, which is to be expected from the same team that created Helm. It should be no surprise that the installation is managed via Helm. Pretty much any Kubernetes system will suffice, and for my experimentation I made use of the IBM Containers Service. I could have just as easily used Minikube, however one aspect of the Brigade service is ingesting GitHub webhook events. Having a Kubernetes cluster available that supported LoadBalancer services makes things a bit easier.

Installing the Brigade Service

The installation documentation from Brigade is very straight forward. All I had to do was make sure I had Helm installed on my laptop and my cluster, and that my kubectl client was properly configured to reach my cluster. There are two Helm charts to deal with. One chart defines a project to interact with, such as a GitHub repository. This chart is expected to be used multiple times, once for each repository to be interacted with. As such, there is an example set of values that need to be copied and defined specifically for the repository. In my case, I populated the `myvalues.js` file with details for a GitHub repository I specifically created for this experiment, `j2sol/demobrigade`. I defined a secret phrase that will be used to validate incoming events as having come from GitHub, and supplied a GitHub personal access token so that the Brigade install can act on my behalf when interfacing with the GitHub API. I left the rest of the values alone, and installed the chart:

> helm install --name demobrigade-project ./chart/brigade-project -f myvalues.yaml
NAME:   demobrigade-project
LAST DEPLOYED: Mon Nov 13 10:13:36 2017
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/Secret
NAME                                                            TYPE    DATA  AGE
brigade-463662f2bc06a89c2464d52d1b248602a1085ed2ccf2953df6a8fa  Opaque  6     0s


NOTES:
Your new Brigade project has been created.

This is a brigade project for github.com/j2sol/demobrigade

It will clone the public repo

To check it's state, run:

  helm status demobrigade-project

The other Helm chart provided in the Brigade project repository is the chart for the Brigade services themselves. This is expected to be installed only once. (They advocate one Brigade install per tenant, where a tenant is one or more people responsible for one or more associated repositories. I'd take that a bit further and advocate for independent Kubernetes clusters per Tenant.) For my installation, I needed to tweak one feature, which is the rbac feature. IBM Cloud Container Service's Kubernetes supports Role Base Access Control, and thus my Brigade install needed to account for that. I did not need to edit any files to account for this, I just used an extra option during the helm install:

> helm install --name brigade ./chart/brigade --set rbac.enabled=true
NAME:   brigade
LAST DEPLOYED: Mon Nov 13 10:14:09 2017
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1beta1/RoleBinding
NAME                  AGE
brigade-brigade-api   2s
brigade-brigade-ctrl  2s
brigade-brigade-gw    2s
brigade-brigade-wrk   2s

==> v1/Service
NAME                 TYPE          CLUSTER-IP      EXTERNAL-IP     PORT(S)         AGE
brigade-brigade-api  ClusterIP     172.21.13.101             7745/TCP        2s
brigade-brigade-gw   LoadBalancer  172.21.219.206  169.48.217.138  7744:32586/TCP  2s

==> v1beta1/Deployment
NAME                  DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
brigade-brigade-api   1        1        1           0          2s
brigade-brigade-ctrl  1        1        1           0          2s
brigade-brigade-gw    1        1        1           0          2s

==> v1/Pod(related)
NAME                                   READY  STATUS             RESTARTS  AGE
brigade-brigade-api-57d56ff7f8-rppkh   0/1    ContainerCreating  0         2s
brigade-brigade-ctrl-69ff6bd48f-h49mc  0/1    ContainerCreating  0         2s
brigade-brigade-gw-5bc49bd866-jgd79    0/1    ContainerCreating  0         2s

==> v1/ServiceAccount
NAME                  SECRETS  AGE
brigade-brigade-api   1        2s
brigade-brigade-ctrl  1        2s
brigade-brigade-gw    1        2s
brigade-worker        1        2s

==> v1beta1/Role
NAME                  AGE
brigade-brigade-api   2s
brigade-brigade-ctrl  2s
brigade-brigade-gw    2s
brigade-brigade-wrk   2s


NOTES:
Brigade is now installed!

To find out about your newly configured system, run:

  $ helm status brigade

At this point, all the Brigade pods had been created and the services had also been created. For the next step I needed information about the Brigade Gateway service, easily obtainable from the above output, or by using kubectl:

> kubectl get services
NAME                  CLUSTER-IP       EXTERNAL-IP      PORT(S)          AGE
brigade-brigade-api   172.21.13.101               7745/TCP         1h
brigade-brigade-gw    172.21.219.206   169.48.217.138   7744:32586/TCP   1h
kubernetes            172.21.0.1                  443/TCP          2d

The information we need is the EXTERNAL-IP and the external port, which in this case is `169.48.217.138 32586`. With this information I can moved on to the GitHub repository setup.

Configuring GitHub

My running Brigade service is rather useless without events to act upon. The events I wanted to react to come from GitHub, in the form of webhooks. As certain things happen to a repository on GitHub, it can send data about those events to a set URL. Setting this up requires administrative rights to the repository within GitHub. To set this up, I followed the provided instructions, replacing the Payload URL with data discovered in the above section, and the Secret with the string I added to myvalues.yaml earlier. Since my demo repository is public, I did not go through the steps to set up a private repository.

Upon completing the webhook configuration, GitHub will send an event to the webhook, a ping event. This is visible in the Recent Deliveries section of the webhook configuration page. It should have a little green checkmark next to the UUID of the delivery. This lets me know that my GitHub project is able to communicate with my Brigade service as expected.

Create a brigade.js file

With my Brigade install reacting to GitHub events, it's time to direct it to do something when those events occur. The Brigade architecture relies on a special file within the git repository in question to inform the system what to do. This file is expected to be JavaScript. To start with, a simple statement to log a can be used:

console.log("hello from Brigade")

I created this file and pushed it to a branch. Because I pushed this branch to the repository that is generating webhooks, Brigade got events for the branch creation and push of a commit to the branch. I was able to see this by checking the logs of the gateway pod's container:

> kubectl get pods
NAME                                    READY     STATUS    RESTARTS   AGE
brigade-brigade-api-57d56ff7f8-rppkh    1/1       Running   0          4h
brigade-brigade-ctrl-69ff6bd48f-h49mc   1/1       Running   0          4h
brigade-brigade-gw-5bc49bd866-jgd79     1/1       Running   0          4h

> kubectl logs brigade-brigade-gw-5bc49bd866-jgd79

[GIN] 2017/11/13 - 21:55:45 | 400 |       77.33µs |  10.184.120.244 |  POST     /events/github
Expected event push, got create
[GIN] 2017/11/13 - 21:58:43 | 400 |     451.036µs |  10.184.120.244 |  POST     /events/github

I can also look at the logs of the Brigade controller to get some more information:

> kubectl logs brigade-brigade-ctrl-69ff6bd48f-h49mc

handler.go:16: EventHandler: type=push provider=github commit=c967a7e629167d7672f31b2061e12add0ff6cc9f
handler.go:43: Started brigade-worker-01byvp8jqfn9vggv0exbe24wbx-c967a7e6 for "push" [c967a7e629167d7672f31b2061e12add0ff6cc9f] at -62135596800

If I follow the clues, I can get the logs from the worker pod that was started:

> kubectl logs brigade-worker-01byvp8jqfn9vggv0exbe24wbx-c967a7e6
yarn run v1.3.2
$ node prestart.js
prestart: src/brigade.js written
$ node --no-deprecation ./dist/src/index.js
hello from Brigade
Creating PVC named brigade-worker-01byvp8jqfn9vggv0exbe24wbx-c967a7e6
after: default event fired
beforeExit(2): destroying storage
Destroying PVC named brigade-worker-01byvp8jqfn9vggv0exbe24wbx-c967a7e6
Done in 0.87s.

With this output I can see the string I asked to be logged to the console, hello from Brigade. All the basics are set up and functioning as expected!

Beyond the basics

Simply logging words to the console of a container isn't very useful. One of the basic capabilities of Brigade is to launch containers to do actions upon certain events. For instance, I'd like to validate that my demoapp code passes pycodestyle. To achieve this, I can follow the scripting guide in order to update my brigade.js file to run pycodestyle within a container using the python:alpine image on pull_request events. I've created such a script and opened a pull request. The content of brigade.js is:

const { events, Job } = require("brigadier")

events.on("pull_request", () => {
    var job = new Job("pycodestyle", "python:alpine")
    job.tasks = [
      "pip install pycodestyle",
      "cd /src",
      "pycodestyle demoapp"
    ]
    job.run()
})

Don't worry if you don't know JavaScript, I don't either. I'm just using details from the scripting guide to create my file content. I'm making use of the /src/ directory, as that's where Brigade has made the code from my repository available. The version of code that is checked out is the tip of the pull request. Once again, I can log the controller pod to discover the worker pod that was created in reaction to the pull request event:

> kubectl logs brigade-brigade-ctrl-69ff6bd48f-h49mc

handler.go:16: EventHandler: type=pull_request provider=github commit=4023f592ff7a4afb24d667c720a4b21b4c78b6a4
handler.go:43: Started brigade-worker-01byvsvswfc6j0g9bmebn29963-4023f592 for "pull_request" [4023f592ff7a4afb24d667c720a4b21b4c78b6a4] at -62135596800

The logs for the worker pod show something interesting:

> kubectl logs brigade-worker-01byvsvswfc6j0g9bmebn29963-4023f592
yarn run v1.3.2
$ node prestart.js
prestart: src/brigade.js written
$ node --no-deprecation ./dist/src/index.js
Creating PVC named brigade-worker-01byvsvswfc6j0g9bmebn29963-4023f592
looking up default/github-com-j2sol-demobrigade-pycodestyle
Creating secret pycodestyle-1510614103251-4023f592
Creating Job Cache PVC github-com-j2sol-demobrigade-pycodestyle
undefined
Creating pod pycodestyle-1510614103251-4023f592
Timeout set at 900000
default/pycodestyle-1510614103251-4023f592 phase Pending
default/pycodestyle-1510614103251-4023f592 phase Pending
default/pycodestyle-1510614103251-4023f592 phase Pending
default/pycodestyle-1510614103251-4023f592 phase Pending
default/pycodestyle-1510614103251-4023f592 phase Running
default/pycodestyle-1510614103251-4023f592 phase Failed
FATAL: job pycodestyle(pycodestyle-1510614103251-4023f592): Pod pycodestyle-1510614103251-4023f592 failed to run to completion (rejection)
error handler is cleaning up
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

It appears that the pod created for my job has failed. I'll look at the logs of the failed pod:

> kubectl logs pycodestyle-1510614103251-4023f592
Collecting pycodestyle
  Downloading pycodestyle-2.3.1-py2.py3-none-any.whl (45kB)
Installing collected packages: pycodestyle
Successfully installed pycodestyle-2.3.1
demoapp/__init__.py:4:1: E302 expected 2 blank lines, found 1
demoapp/__init__.py:9:17: E225 missing whitespace around operator
demoapp/__init__.py:12:1: E302 expected 2 blank lines, found 1
demoapp/__init__.py:17:1: E305 expected 2 blank lines after class or function definition, found 1

Looks like my code does not currently pass pycodestyle! Normally I'd fix those as part of the pull request to add the testing, but I want to leave this pull request open for the sake of the blog. I'll open a new one that has the fixed contents on a new branch. This time the worker pod logs show a successful run:

> kubectl logs brigade-worker-01byxzk5nm3vfxvw763a1743xc-c2347081
yarn run v1.3.2
$ node prestart.js
prestart: src/brigade.js written
$ node --no-deprecation ./dist/src/index.js
Creating PVC named brigade-worker-01byxzk5nm3vfxvw763a1743xc-c2347081
looking up default/github-com-j2sol-demobrigade-pycodestyle
Creating secret pycodestyle-1510687220035-c2347081
Creating Job Cache PVC github-com-j2sol-demobrigade-pycodestyle
undefined
Creating pod pycodestyle-1510687220035-c2347081
Timeout set at 900000
default/pycodestyle-1510687220035-c2347081 phase Pending
default/pycodestyle-1510687220035-c2347081 phase Running
default/pycodestyle-1510687220035-c2347081 phase Succeeded
after: default event fired
beforeExit(2): destroying storage
Destroying PVC named brigade-worker-01byxzk5nm3vfxvw763a1743xc-c2347081
Done in 6.98s.

Pipelines

Reacting to events to run tests is a pretty common thing. There's nothing special about it. What really makes Brigade interesting is the ability to run multiple jobs in multiple containers, and to create groups of those jobs which can either run in parallel or serially. There is also the ability to use content and output generated as part of a job in a later job.

For my simple demo repository, I want to set up a functional test of my application, and alter the brigade.js file to run the functional test and the style test in parallel via two containers. Another pull request shows this change. The content of the brigade.js file is now:

const { events, Job, Group } = require("brigadier")

events.on("pull_request", () => {
    var style = new Job("pycodestyle", "python:alpine")
    style.tasks = [
      "pip install pycodestyle",
      "cd /src",
      "pycodestyle demoapp"
    ]

    var functional = new Job("functional", "python:alpine")
    functional.tasks = [
      "apk update && apk add curl",
      "pip install /src/",
      "/usr/local/bin/demoapp &",
      "sleep 2",
      "curl http://127.0.0.1:8000"
    ]

    Group.runAll([style, functional])
})

Viewing the logs from the worker pod that was created for the pull_request event shows the new job names and the pods used. The output also shows how they're running in parallel:

> kubectl logs brigade-worker-01byy57yadgr1khq86q5z6k07g-96842797
yarn run v1.3.2
$ node prestart.js
prestart: src/brigade.js written
$ node --no-deprecation ./dist/src/index.js
Creating PVC named brigade-worker-01byy57yadgr1khq86q5z6k07g-96842797
looking up default/github-com-j2sol-demobrigade-pycodestyle
looking up default/github-com-j2sol-demobrigade-functional
Creating secret pycodestyle-1510693143841-96842797
Creating secret functional-1510693143844-96842797
Creating Job Cache PVC github-com-j2sol-demobrigade-pycodestyle
undefined
undefined
Creating Job Cache PVC github-com-j2sol-demobrigade-functional
Creating pod pycodestyle-1510693143841-96842797
Creating pod functional-1510693143844-96842797
Timeout set at 900000
Timeout set at 900000
default/pycodestyle-1510693143841-96842797 phase Pending
default/functional-1510693143844-96842797 phase Pending
default/functional-1510693143844-96842797 phase Running
default/pycodestyle-1510693143841-96842797 phase Running
default/pycodestyle-1510693143841-96842797 phase Succeeded
default/functional-1510693143844-96842797 phase Running
default/functional-1510693143844-96842797 phase Running
default/functional-1510693143844-96842797 phase Running
default/functional-1510693143844-96842797 phase Succeeded
after: default event fired
beforeExit(2): destroying storage
Destroying PVC named brigade-worker-01byy57yadgr1khq86q5z6k07g-96842797
Done in 13.03s.

I can imagine creating complex sets of jobs within groups to run some tests in parallel, some in serial, more in parallel, while sharing content and output between the sets.

Conclusion

Brigade is an interesting project. It has the capability to create complex workflows in reaction to specific events, and the set up is pretty easy. It may be a better solution than creating your own webhook handler and launching native Kubernetes Jobs. As a project, I think it's in the early stages of development. I ran into a few issues as I was playing around. The project developers have been quite responsive and eager for input. I wouldn't recommend taking Brigade into production just yet, but I would encourage exploring the capability and thinking about potential use cases. I'm sure the developers would love more input on how people would like to use Brigade. The project is open source, so if you want to get your hands dirty I'm sure they'd welcome the help too!

tags: k8s - kubernetes - python - GitHub