Category Archives: python

Kicking the tires of the Brigade service for Kubernetes

Recently, the Azure team announced a new open source service called Brigade. The point of this service is to provide a neat way to execute pipelines of job executions via containers within a Kubernetes system, in response to events from the likes of GitHub or Dockerhub. I thought I’d take it for a test drive on the IBM Cloud Containers Service, which exposes Kubernetes.

Background Knowledge

This post assumes some level of understanding of Kubernetes, (Docker) Containers, and GitHub pull requests.

What’s the point?

Why would somebody want a Brigade like service? There are many reasons, but for me the reason that stood out was an easy way to set up a Continuous Integration and Continuous Delivery pipeline. That is, I wanted proposed changes to code I have hosted on GitHub to be tested before I merge it, and once merged, to be deployed somewhere. There are many options available to provide CI/CD services in the (growing) GitHub Marketplace, however being in control of your own service and infrastructure, as well as being able to potentially tie into other event streams, is an appealing thought. Full disclosure, I was recently working on a project to add another option to the GitHub Marketplace for CI/CD, based on the open source Zuul project from OpenStack.

Getting Started

The installation process for Brigade was very easy, which is to be expected from the same team that created Helm. It should be no surprise that the installation is managed via Helm. Pretty much any Kubernetes system will suffice, and for my experimentation I made use of the IBM Containers Service. I could have just as easily used Minikube, however one aspect of the Brigade service is ingesting GitHub webhook events. Having a Kubernetes cluster available that supported LoadBalancer services makes things a bit easier.

Installing the Brigade Service

The installation documentation from Brigade is very straight forward. All I had to do was make sure I had Helm installed on my laptop and my cluster, and that my kubectl client was properly configured to reach my cluster. There are two Helm charts to deal with. One chart defines a project to interact with, such as a GitHub repository. This chart is expected to be used multiple times, once for each repository to be interacted with. As such, there is an example set of values that need to be copied and defined specifically for the repository. In my case, I populated the myvalues.js file with details for a GitHub repository I specifically created for this experiment, j2sol/demobrigade. I defined a secret phrase that will be used to validate incoming events as having come from GitHub, and supplied a GitHub personal access token so that the Brigade install can act on my behalf when interfacing with the GitHub API. I left the rest of the values alone, and installed the chart:

The other Helm chart provided in the Brigade project repository is the chart for the Brigade services themselves. This is expected to be installed only once. (They advocate one Brigade install per tenant, where a tenant is one or more people responsible for one or more associated repositories. I’d take that a bit further and advocate for independent Kubernetes clusters per Tenant.) For my installation, I needed to tweak one feature, which is the rbac feature. IBM Cloud Container Service’s Kubernetes supports Role Base Access Control, and thus my Brigade install needed to account for that. I did not need to edit any files to account for this, I just used an extra option during the helm install:

At this point, all the Brigade pods had been created and the services had also been created. For the next step I needed information about the Brigade Gateway service, easily obtainable from the above output, or by using kubectl:

The information we need is the EXTERNAL-IP and the external port, which in this case is 169.48.217.138 32586. With this information I can moved on to the GitHub repository setup.

Configuring GitHub

My running Brigade service is rather useless without events to act upon. The events I wanted to react to come from GitHub, in the form of webhooks. As certain things happen to a repository on GitHub, it can send data about those events to a set URL. Setting this up requires administrative rights to the repository within GitHub. To set this up, I followed the provided instructions, replacing the Payload URL with data discovered in the above section, and the Secret with the string I added to myvalues.yaml earlier. Since my demo repository is public, I did not go through the steps to set up a private repository.

Upon completing the webhook configuration, GitHub will send an event to the webhook, a ping event. This is visible in the Recent Deliveries section of the webhook configuration page. It should have a little green checkmark next to the UUID of the delivery. This lets me know that my GitHub project is able to communicate with my Brigade service as expected.

Create a brigade.js file

With my Brigade install reacting to GitHub events, it’s time to direct it to do something when those events occur. The Brigade architecture relies on a special file within the git repository in question to inform the system what to do. This file is expected to be JavaScript. To start with, a simple statement to log a can be used:

I created this file and pushed it to a branch. Because I pushed this branch to the repository that is generating webhooks, Brigade got events for the branch creation and push of a commit to the branch. I was able to see this by checking the logs of the gateway pod’s container:

I can also look at the logs of the Brigade controller to get some more information:

If I follow the clues, I can get the logs from the worker pod that was started:

With this output I can see the string I asked to be logged to the console, hello from Brigade. All the basics are set up and functioning as expected!

Beyond the basics

Simply logging words to the console of a container isn’t very useful. One of the basic capabilities of Brigade is to launch containers to do actions upon certain events. For instance, I’d like to validate that my demoapp code passes pycodestyle. To achieve this, I can follow the scripting guide in order to update my brigade.js file to run pycodestyle within a container using the python:alpine image on pull_request events. I’ve created such a script and opened a pull request. The content of brigade.js is:

Don’t worry if you don’t know JavaScript, I don’t either. I’m just using details from the scripting guide to create my file content. I’m making use of the /src/ directory, as that’s where Brigade has made the code from my repository available. The version of code that is checked out is the tip of the pull request. Once again, I can log the controller pod to discover the worker pod that was created in reaction to the pull request event:

The logs for the worker pod show something interesting:

It appears that the pod created for my job has failed. I’ll look at the logs of the failed pod:

Looks like my code does not currently pass pycodestyle! Normally I’d fix those as part of the pull request to add the testing, but I want to leave this pull request open for the sake of the blog. I’ll open a new one that has the fixed contents on a new branch. This time the worker pod logs show a successful run:

Pipelines

Reacting to events to run tests is a pretty common thing. There’s nothing special about it. What really makes Brigade interesting is the ability to run multiple jobs in multiple containers, and to create groups of those jobs which can either run in parallel or serially. There is also the ability to use content and output generated as part of a job in a later job.

For my simple demo repository, I want to set up a functional test of my application, and alter the brigade.js file to run the functional test and the style test in parallel via two containers. Another pull request shows this change. The content of the brigade.js file is now:

Viewing the logs from the worker pod that was created for the pull_request event shows the new job names and the pods used. The output also shows how they’re running in parallel:

I can imagine creating complex sets of jobs within groups to run some tests in parallel, some in serial, more in parallel, while sharing content and output between the sets.

Conclusion

Brigade is an interesting project. It has the capability to create complex workflows in reaction to specific events, and the set up is pretty easy. It may be a better solution than creating your own webhook handler and launching native Kubernetes Jobs. As a project, I think it’s in the early stages of development. I ran into a few issues as I was playing around. The project developers have been quite responsive and eager for input. I wouldn’t recommend taking Brigade into production just yet, but I would encourage exploring the capability and thinking about potential use cases. I’m sure the developers would love more input on how people would like to use Brigade. The project is open source, so if you want to get your hands dirty I’m sure they’d welcome the help too!

Interactive debugging python code in (mini) Kubernetes

Lately I’ve been playing around with Kubernetes. If you don’t know what Kubernetes (k8s) is, then the rest of this post is going to be very confusing to you.

Needed concepts

I’m going to talk about a few things. Here are some links to places to get up to speed should any of this not make any sense:

Background

I’m somewhat late to the k8s game, and I’m still trying to get my bearings. One task I set out to figure out is how I can replicate my development workflow I had built up with Docker Compose to launch containers of my application locally for testing. The application I’ve been developing on consists of a Zookeeper service and three Python services. I have the Python services broken out into three separate containers based on the same image, but with different launch commands. When testing things locally, I often want to introduce code that I haven’t yet committed to a repository, and instead of building new images every time, I make use of a volume mount to bring my code into the container at runtime. This works well with Python thanks to a feature in pip, the tool for installing python packages. I can tell pip to perform an editable install. This type of install makes use of a symlink in the installation target path which links to the source directory of the install. In my Dockerfile I clone the source code to /zuul, and then perform the pip install from there. What this means is that after the install, I could simply alter the files in the original checkout in /zuul and restart the process and the changes will take effect. To expand on that further, this gives me the ability to attach a volume mount from my laptop’s zuul checkout directory (where I have edited files) to the /zuul path within the container at start time, so that I can make use of edited files without rebuilding the image.

On to Kubernetes!

My workflow worked great in docker-compose, but now I want to do this with k8s. To demonstrate how this works, I’ve created a simple Python web server in a demo app. I’ve put the source for this on GitHub for convenience.

First, I need to install and run minikube, a tool to run a Kubernetes cluster locally on my laptop. Minikube will download some data and launch a virtual machine in which to run the k8s services, including its own Docker daemon.

Build Docker image

With minikube running, I next need to configure my Docker client to make use of the Docker Engine within minikube. A simple command eval $(minikube docker-env) will set up my client. Now I can build my Docker image so that it’ll be available for use within k8s. I’ve cloned my repository into a src/derpops/demoapp directory relative to my homer. From there I just need a simple Docker build command to build my image. NOTE! I’m using a tag other than latest so that Kubernetes will not try to pull the latest version of my image. My image will only exist locally, so a pull would fail.

Creating a Deployment

With the image in place, I can now create a k8s Deployment. The Deployment lets me define a container to launch, with the image I built above, and a command to run within the container. I can do this with a simple kubectl command:

If my deployment is created successfully, a new pod will show up, which will be running my code.

Creating a Service

To see if my code is working properly, I need to be able to reach the web server. A k8s Service is necessary to set up the networks appropriately. Just like with the Deployment, a simple kubectl command will suffice to create the service:

This command created a service that will allow me to reach port 8000 of the container running the application. My code specifically tells the python library to use port 8000 and to listen on all addresses. This type of service is a nodeport, which makes this port reachable from every node. However, since this is minikube, I only have one node, and I can ask minikube to tell me what the IP address of the node is with the service command. I’ll ask it to just display the URL, instead of opening the URL in my browser, and then use curl to access the URL:

Injecting new code

My application works, but now I want to alter the code. To get new code into my container, I need to add a volume mount to my deployment.

VolumeMounts are used to expose content into a container. A VolumeMount definition combines the name of a volume and a path to mount it within the container. The name matches a defined Volume, of which many types are supported. The type we’re interested in, the hostPath type, exposes a file or a directory from the node (the machine a container is running on). The use of minikube automatically exposes a folder from the host machine minikube runs on into the VM where k8s is running, which is the node. In my case, the /Users directory on my laptop is exposed as /Users on the node, and thus I can make use of this in a hostPath volume.

To add a volume to my Deployment, I’ll need to create a yaml file describing my Deployment spec. I can do this quickly by repeating the earlier command but adding arguments to print out YAML:

I saved this output to a new file,  demoapp-deployment.yaml, where I can make adjustments as needed. I need to alter the containers spec for the demoapp container to define a volumeMount:

This references a volume by the name of demosource which I also need to define in the spec as a new key:

These additions will cause the directory /Users/jkeating/src/derpops/demoapp/demoapp to be mounted to the path /demoapp/demoapp within the container when it launches. This will overlay the version of code on my laptop on top of the code that already exists in the container.

To get the new definition of my Deployment in use, I’ll delete the existing Deployment and create a new one from the YAML file:

The existing pod is being terminated while a new pod is running. If all went well, I should still be able to use curl to reach the web server:

The output is the same as earlier, as I haven’t changed any code. But what if I change the value of the variable RESPONSE to something new by editing the file demoapp/__init__.py on my laptop.

To get my new code in use, I can simply delete the Pod my Deployment created. A Deployment uses a ReplicaSet to control how many Pods are active. Deleting the pod will trigger the creation of a new one, which should pick up my new code. To determine which pod to delete, I’ll use the get pods command via kubectl. This will list all the running pods. Then I’ll use the delete pod command to delete the demoapp pod, and then make sure a new one is created:

The new pod should have my new code, which I’ll verify with curl once more:

Interactive debugging

Getting new code used in the container is fun, but what is even more useful is being able to interactively debug this new code. Python developers should be familiar with the use of pdb, the python debugger. This utility can be used to insert a break point into the source code in order to interactively debug the code at that point in the execution. Pdb is fantastic when you are able to execute the code directly, but requires being attached to the tty that started the python process. That’s a difficult feat inside of a system like k8s. Thankfully there is a wrapper around pdb specifically for connecting to remote python processes, called rpdb. When using rpdb, the wrapper will redirect stdout/stdin to a socket handler, which can be accessed over TCP. This will allow me to define a breakpoint in the code and then connect to the socket remotely in order to attach to the debugger to interact with the process. (One downside of rpdb is that it is not part of the standard library, and thus the library will need to be explicitly installed into the python environment. I’ve done this in my Dockerfile.)

To debug my code, I first have to add the breakpoint to one of my source files. At the appropriate line, I need to insert import rpdb; rpdb.set_trace("0.0.0.0"). I specifically need to tell rpdb to listen on 0.0.0.0 instead of the default 127.0.0.0. This will make rpdb listen on all addresses rather than just localhost. This is less secure, but required in order to reach it through k8s. Once again, I’ll edit the demoapp/__init__.py file on my laptop:

I also need to update my k8s Service to expose the new port. Rpdb will listen on port 4444 by default, so I’ll use that in my service. First I’ll delete the existing demoapp service and then re-create it adding a second port:

I can test that the new service works by using curl to the new port:

Now that the code is edited and the service is created to forward ports through, I can restart the container. Once again I’ll delete the running pod to trigger the creation of a replacement.

Since my breakpoint is inside the do_GET function, I’ll need to use curl to initiate a GET request. This will seem to hang in the terminal, as the break point has been reached and execution is waiting. At this point, I should be able to connect to the waiting debugger,  using the Service information to determine which port to connect to. In another terminal I can use nc to connect to the debugger! From this point on, it’s debugging as usual.

From here I can change the value of RESPONSE once more, and then continue execution, which should cause my curl command to return with my new message:

Conclusion

Kuberetes is a pretty huge leap forward in container orchestration. With that advancement comes some complexity, and a whole lot of new concepts to learn. However, the basic building blocks are there to continue using workflows that have been useful in the past. This workflow to debug code live is just a small example of what is possible with k8s, minikube, and containers in general.

I’ve added a complete demoapp-deployment.yaml file to the git repository, including a Service definition. Hopefully this example will be useful! As always, comment here or on Twitter should you have any thoughts to share.

Happy kubing!

Documenting python code using sphinx and github

Documentation is good right? Doesn’t everybody like to have a good source of documents when working with a piece of software? I know I sure do. But creating documentation can be a drag, and creating pretty documentation can be even more of a drag, more time consuming than writing the code in the first place. Ain’t nobody got time for that!

Thankfully there are utilities out there that will help with creating documentation. My language of choice is Python and for generating documentation I like to use Sphinx. Sphinx appeals to me because much of the documentation can be auto-generated based on existing code. It works with python docstrings in a way that can be PEP257 compatible. It takes very little setup to get from nothing to decent looking documentation. For an example of what sphinx output can look like, see my pyendeavor project.

The first step is to make frequent use of docstrings in the code. Not only will this help to generate useful documentation later, but it is also really handy for anybody working with the code to understand what the code is doing. A docstring is a string literal that occurs as the first statement in a module, function, class, or method definition. It’s more than just a comment, because it will become an attribute of the object itself. When I look at a function or a class init I ask 3 main questions:

  1. What does this code do?
  2. What are the inputs to it?
  3. What will I get in return?

These questions can be broken down into the docstring very easily:

Those three lines answer the question pretty well, and if we were just going to look at the code and not try to generate html/pdf/whatever documentation we could be done. Instead lets try to give it a little more structure, structure that sphinx will appreciate:

A human reading this docstring is still going to know what’s going on pretty well, and sphinx is going to read it even better:


The python source files themselves service as input to the documentation creation tool.  Just go about the business of writing code and keep the docstrings flowing and updated with changes and very useful documentation can be produced.

How does one generate the documentation though? How does one use sphinx? I’m glad you asked! Sphinx has a utility that can help get started — sphinx-apidoc. First make a docs/ subdir of the project (or whatever you want to call it). This is where some sphinx control files will go, although the rendered output doesn’t necessarily have to go there. For my example software pyendeavor I have a single python package, pyendeavor, located in the src/pyendeavor directory. To get started with sphinx, I would issue the command:

The F causes a full setup to happen, the -o docs tells sphinx to direct it’s output to the newly created docs directory, and the src/ tells sphinx to look in src/ for my modules.

Sphinx uses reStructuredText as the input format, which is pretty easy to work with.  And if we look at the files it generated for us, we’ll see that there isn’t much there:

And if we look in the pyendevor file we’ll see more, but here is a snippit:

This just tells sphinx to read the module files and generate content for module members, undocumented members, and to follow inheritance. These are all just commands that sphinx understands, but you don’t really have to.

There is a conf.py file that will need some attention. Sphinx will need to know how to import the code, so a system path entry to where the code can be found needs to be added.  There is a helpful comment near the top, just clear the hash and update the path:

Now we are ready to make some content! Make sure to be in the docs/ dir and run make. A list of possible make targets will be displayed, but the we care about is ‘html’, so run that one:

Lots of output, but the end result is some html pages in _build/html/ :

A browser can be used to view index.html and all the linked docs. How awesome! Useful documentation without having to do much more than just use docstrings in the code (which should be done anyway).

Now that docs can be generated, they should be put somewhere useful. That’s where the github part of this post comes in. A lot of projects are posted on github and I’ve started using it for more of mine too. One nice feature is a way to create a webspace for a project by pushing content to a ‘gh-pages’ branch of a project. These following steps will help setup a repo to have a place to publish the html content of sphinx. They are based on the directions I found here, however instead of using a directory outside our project space we’re going to make use of a git workdir so that we never have to leave the project directory to get things done.

First lets create a directory to hold our new branch from the top level of our source.

Next create a new workdir named html within that directory:

(More information about git-new-workdir can be found here but essentially it is a way to create a subdirectory that can be checked out to a new branch, but all the git content will be linked. A git fetch in the topdir of the clone will also update the git content in the workdir path. No need for multiple pulls.)

Now we have to prepare the workdir for sphinx content. To do this we need to create an empty gh-pages branch within the html directory:

Initially there will be a copy of the source tree in the html directory that can be blown away with:

Back in the docs/ directory a change needs to be made to the Makefile to tell it to output content to where we want it, the gh-pages/html/ directory. Look for:

and change it to

Now from the docs/ directory, run make html again. This time you’ll notice that the output goes to ../gh-pages/html/

Switching back to that directory the files can be added and committed with:

Now the branch can be pushed to github:

After upwards to 10 minutes later the pages site for the repo can be visited, like mine for pyendeavor. Github also has a feature for README files in the base of your repo, supporting/rendering markdown and reStructuredText. This README.rst file can also be included in your pages output with a simple tweak to the index.rst file in docs/. The ..include directive will tell sphinx to include the content from the README.rst file when generating html output:

The README.rst file source can be seen here. The same content of the README file will display in the generated docs, updating this content only has to happen once.

It is a good idea to add the control files in docs/ to the repository and keep them under source control:

If new modules or packages are added to the source tree then a new run of sphinx-apidoc is necessary. The -f flag will tell sphinx that it is OK to overwrite the existing files:

When functions change or new docstrings are added to the code new html content needs to be generated by running make html and committing/pushing the new content in the gh-pages/html/ directory.

That’s all there is to it, sphinx generated docs rendered by github via the gh-pages branch. All within a single directory tree. Have fun, and get to documenting!

Python magic and remote APIs

I’m a pretty big fan of Python as a programming language. It allows me to program by discovery, that is poke and prod at things until the work. Not having to compile an entire program every time I change something is pretty fantastic, as is the ability to insert a debug statement and be able to break a program at that point, then run arbitrary python code within that context. Pretty indispensable to how I write software.

Another thing I like about Python, which some may not, is the ability to do magic things. Not quite so magic as xkcd would like us to believe, but fun stuff indeed.

Recently one of the services at work grew a json API to bang against, and for fun I thought I’d whip up some python to play with it. My team had a few utility scripts that would bang on the old xmlrpc interface to get some data, I wanted to see how much faster it was with json.

First, if you have to do anything web stuff, you really should be using the Requests module. It is so, so much better than using urllib(2) directly.

The API I wanted to program against had an Auth end point that would return to you a token string. This string could be included with later API calls to provide authentication. Requests lets you create a session object that can have attributes that carry on to all web calls, such as a custom auth header with a token.

Now the session object can be used just like requests itself, and it’ll include the new header we’ve added. While this was neat at first, I quickly realized that I wanted to make this session object an attribute of a more generic object for working with the API. Each time you use session or requests you have to fill in a url and that’s tedious, so I made a python class to handle that for me. One bit of magic I used here was a python property.

A python property is a way populate a class attribute on the fly / as needed without the code using your object needing to know that it’s happening behind the scenes. It’s a getter/setter without having to get and set, and it caches the value for future getting.  My class sets some data during the init process, and creates a property for the session attribute, which can then be used in later functions, like a login or query function.

With this structure we can do things like:

We get back a json blob that has what the API returned to us. What happened was that the query function built up the information for the requests bit, which was passed into self.session.post(). Since this was the first time trying to access self.session we went through the @property tagged session() function. That function determined that self._session was not populated yet and called _auth(). _auth() in turn did the login dance to generate the token, built up a requests.Session object, tweaked the header and stuffed it into self._session. session() then returned that to the caller thus delivering the actual session object. Magic!  The next time session is accessed it will quickly return the value of self._session. Properties are awesome and useful.

A CServ() object is okay, but not useful on its own. If I wanted to get a bit of data about a computer and use that data numerous times I’d either have to store a copy of the data in a local variable, or do queries each time I wanted the data. Neither are efficient. What I really want is to be able to create a Computer object, and from that object access attributes directly, like say Computer.name. Here is where some more magic comes in. We already know we can create properties to back attributes. We could go find out what all possible things we could look up about a computer in our CServ service, then write out properties for each of those. That… doesn’t sound fun. Particularly if you think about this CServ having Computer items, Switch items, Account items, etc… That would be a lot of typing!

What if instead there was a way to do dynamic properties?  What if when we tried to access Computer.primary_ip the code would just know to do a query to look up the ‘primary_ip’ attribute of a Computer.Computer cserv API class?  Well we’re in luck, because there is a way!

First we’re going to create a subclass of CServ, CServOBJ. This class will be a base class for any number of objects, like Computers, Accounts, etc.. We can save a lot of code duplication by putting the shared bits in CServOBJ.

Right now we don’t need to overload the __init__ method, so we can dive right into the magic. In python, when you attempt to access an object’s attribute, behind the scenes an object’s __getattr__(attribute) method is called. Normally you don’t see it because it’s all built in, but we can override it to make attribute access do something different. In our case, we want to do an API look up to get the value if we don’t already have it, so we’ll overload the function:

Objects in python also have a built in __dict__ that keeps track of all the attributes. Our simple little bit of code will try to return the value for the name of the attribute the function gets. If that attribute doesn’t exist in the built in dict, a keyerror would happen. We catch that error and call our _setAttrib() function. This function is where the look up is built up using some other class attributes we’ll get to later. A session call is made and the value is fed into the setattr python built-in. All this work happens behind the scenes to the bit of code just trying to access the attribute, and the lookup only happens once.  That’s all we really need for now in the base class, lets create a Computer class.

That’s all there is to it.  _qclass is defined as a class attribute, it does not change per-object.  It is the class name passed into the remote API.  The object creation takes a number, which is the identifier for computers in our system.  It assigns that to the number attribute so that if we reference computer.number we don’t make another API call.  _qval is the place holder that will be common across all the objects for what do use as a look up key.  The parent class’s init is called (which skips all the way up to CServ) to complete the object creation.

With this setup, we can program against it very easily to access and cache data:

MAGIC!

Now if you are like me, you spend a lot of time in things like ipython or bpython to interactively program stuff and play with objects and whatnot.  These environments provide tab completion, help functions, etc…  With our current code though, we couldn’t tab complete the available attributes.  Only the name attribute and the functions we’ve created would show up.  To fix that, we need to overload the built in __dir__ function.  This function is used when getting a listing of what is available to an object, dir(object).  A useful exploratory tool.  ipython/bpython use this method to see what tab completion options to provide you.  Luckily our internal service provides an API call to get a listing of possible attributes, so we can hook that into __dir__.  But of course we only want to do this API call once (per object) so we will want to make it a property.  Since there is nothing API class specific we can put the code into the CServOBJ class:

Since we are creating a property at this level we will grow an __init__ function to prep for that. Then we define the attribs() function. A short-cut is taken here, instead of calling out to some other function to load the attributes the load is done directly.  Any time a Computer object gets a dir() call our overloaded function will return a sorted list that is the combination of the built in functions/attributes, anything that has been added to the specific object, and the available API attributes.  Tab-completion achieved!

This has been a quick look into some of the magic you can do with Python.  It’s not quite antigravity, but it is useful, and food for thought for anybody that’s programming against remote APIs.  Objects are good, objects with dynamic attributes are better, and tab completion is icing on the cake.  Enjoy!