Category Archives: github

Why I love open source!

The other day I decided it was time I got familiar with Docker. Yes, I know, I’m a bit late to the party, but better late than never. I understood some of the concepts around Docker, just not necessarily the mechanics, so it was time to dive in.

Docker these days has a handy utility named docker-machine. This tool is used to create a target system to create docker containers on. This is really useful if you’re on a Mac and don’t have a kernel that supports containers natively. By using docker-machine I was able to provision a VM via VirtualBox that was all set up to run containers. From that point in, docker commands ran as expected, and containers showed up inside the VirtualBox VM.

This is all well and good, but being a curious nerd I wanted to see what other drivers there were for docker-machine. Unsurprisingly, there are numerous drivers, many of them cloud based. There is an OpenStack driver as well, which is great! My day job is all about OpenStack and I have numerous clouds at my disposal. The idea of utilizing my cloud to run docker containers just seems natural to me, so that’s what I tried to do.

Unfortunately I ran into a problem. Our clouds work in a way that requires the allocation of a “floating IP” address to an instance in order for that instance to be accessible by the outside world. The docker-machine OpenStack driver supports this, by passing in the correct arguments to tell the driver where to allocate the floating IP from, a pool. It turns out that my account on the cloud I was targeting has admin level rights (a scenario many of my customers will be in), and thus was able to see more available floating IP addresses in the pool than a normal user would, many of which had been allocated to a different project (projects, or tenants in OpenStack are a way to segregate groups of users and resources within a cloud). The docker-machine driver simply attempted to use the first address it thought was available for the instance it just created. In my case, this address had already been allocated to a different tenant and the OpenStack API returned an error when the assignment to my instance was attempted.

I understood the problem, and I had a general idea of how to fix it. The driver should filter the floating IP addresses by my project ID when searching for an available floating IP to use. I essentially had three choices at this point:
Door #1: If the project didn’t have a public bug tracker I could give up on the tool or write a negative review somewhere or a snarky tweet about it and find something else to play with.
Door #2: If the project had a public bug tracker I could file a bug in the tracker and explain the scenario that led to the error. I would just have to wait and hope somebody at the project cared enough to fix my bug.
Door #3:Because docker-machine is an open source project, I had a third choice. I could pull down the code and try my hand at fixing the problem myself.

Obviously I went with door #3. I’ve never looked at the source code behind Docker before (written in Go), but I figured I could fudge my way through a small change. Thankfully the Docker project has spent a fair amount of time thinking about how to make contributing to the project an easy process. Numerous documents exist to help guide a first time contributor through setting up a development environment, understanding the code testing tools, and walking through the submission and review process. Following these guides I was able to start making modifications to the docker-machine code and testing them out on my laptop. What I thought was going to be a simple change turned out to be a more involved code addition, which led me to reading the code and developer documents for a supporting library that docker-machine uses.

Through much iteration and testing, I was finally able to create a change that resolved my issue in a satisfactory way. Being a good open source citizen, I then submitted this change back up to the project in hopes of inclusion in a future release. I myself am not currently blocked in using this tool, but I’d like my customers to be able to use this tool as well, in a way that doesn’t require me to distribute a modified binary to them.

This is the real joy of Open Source to me. I found a tool I want to make use of, I discovered a way in which the tool doesn’t quite work right for me, I have access to the code to debug the problem, I have access to the documentation and supporting code to develop a solution, and I have the opportunity to contribute a change back to the tool. This process feels so natural to me now that any other way just seems broken. Open Source has enabled me to make my life better, as well as potentially making the life of other users of the software better too, and that gets me right in the feels.

Documenting python code using sphinx and github

Documentation is good right? Doesn’t everybody like to have a good source of documents when working with a piece of software? I know I sure do. But creating documentation can be a drag, and creating pretty documentation can be even more of a drag, more time consuming than writing the code in the first place. Ain’t nobody got time for that!

Thankfully there are utilities out there that will help with creating documentation. My language of choice is Python and for generating documentation I like to use Sphinx. Sphinx appeals to me because much of the documentation can be auto-generated based on existing code. It works with python docstrings in a way that can be PEP257 compatible. It takes very little setup to get from nothing to decent looking documentation. For an example of what sphinx output can look like, see my pyendeavor project.

The first step is to make frequent use of docstrings in the code. Not only will this help to generate useful documentation later, but it is also really handy for anybody working with the code to understand what the code is doing. A docstring is a string literal that occurs as the first statement in a module, function, class, or method definition. It’s more than just a comment, because it will become an attribute of the object itself. When I look at a function or a class init I ask 3 main questions:

  1. What does this code do?
  2. What are the inputs to it?
  3. What will I get in return?

These questions can be broken down into the docstring very easily:

Those three lines answer the question pretty well, and if we were just going to look at the code and not try to generate html/pdf/whatever documentation we could be done. Instead lets try to give it a little more structure, structure that sphinx will appreciate:

A human reading this docstring is still going to know what’s going on pretty well, and sphinx is going to read it even better:


The python source files themselves service as input to the documentation creation tool.  Just go about the business of writing code and keep the docstrings flowing and updated with changes and very useful documentation can be produced.

How does one generate the documentation though? How does one use sphinx? I’m glad you asked! Sphinx has a utility that can help get started — sphinx-apidoc. First make a docs/ subdir of the project (or whatever you want to call it). This is where some sphinx control files will go, although the rendered output doesn’t necessarily have to go there. For my example software pyendeavor I have a single python package, pyendeavor, located in the src/pyendeavor directory. To get started with sphinx, I would issue the command:

The F causes a full setup to happen, the -o docs tells sphinx to direct it’s output to the newly created docs directory, and the src/ tells sphinx to look in src/ for my modules.

Sphinx uses reStructuredText as the input format, which is pretty easy to work with.  And if we look at the files it generated for us, we’ll see that there isn’t much there:

And if we look in the pyendevor file we’ll see more, but here is a snippit:

This just tells sphinx to read the module files and generate content for module members, undocumented members, and to follow inheritance. These are all just commands that sphinx understands, but you don’t really have to.

There is a conf.py file that will need some attention. Sphinx will need to know how to import the code, so a system path entry to where the code can be found needs to be added.  There is a helpful comment near the top, just clear the hash and update the path:

Now we are ready to make some content! Make sure to be in the docs/ dir and run make. A list of possible make targets will be displayed, but the we care about is ‘html’, so run that one:

Lots of output, but the end result is some html pages in _build/html/ :

A browser can be used to view index.html and all the linked docs. How awesome! Useful documentation without having to do much more than just use docstrings in the code (which should be done anyway).

Now that docs can be generated, they should be put somewhere useful. That’s where the github part of this post comes in. A lot of projects are posted on github and I’ve started using it for more of mine too. One nice feature is a way to create a webspace for a project by pushing content to a ‘gh-pages’ branch of a project. These following steps will help setup a repo to have a place to publish the html content of sphinx. They are based on the directions I found here, however instead of using a directory outside our project space we’re going to make use of a git workdir so that we never have to leave the project directory to get things done.

First lets create a directory to hold our new branch from the top level of our source.

Next create a new workdir named html within that directory:

(More information about git-new-workdir can be found here but essentially it is a way to create a subdirectory that can be checked out to a new branch, but all the git content will be linked. A git fetch in the topdir of the clone will also update the git content in the workdir path. No need for multiple pulls.)

Now we have to prepare the workdir for sphinx content. To do this we need to create an empty gh-pages branch within the html directory:

Initially there will be a copy of the source tree in the html directory that can be blown away with:

Back in the docs/ directory a change needs to be made to the Makefile to tell it to output content to where we want it, the gh-pages/html/ directory. Look for:

and change it to

Now from the docs/ directory, run make html again. This time you’ll notice that the output goes to ../gh-pages/html/

Switching back to that directory the files can be added and committed with:

Now the branch can be pushed to github:

After upwards to 10 minutes later the pages site for the repo can be visited, like mine for pyendeavor. Github also has a feature for README files in the base of your repo, supporting/rendering markdown and reStructuredText. This README.rst file can also be included in your pages output with a simple tweak to the index.rst file in docs/. The ..include directive will tell sphinx to include the content from the README.rst file when generating html output:

The README.rst file source can be seen here. The same content of the README file will display in the generated docs, updating this content only has to happen once.

It is a good idea to add the control files in docs/ to the repository and keep them under source control:

If new modules or packages are added to the source tree then a new run of sphinx-apidoc is necessary. The -f flag will tell sphinx that it is OK to overwrite the existing files:

When functions change or new docstrings are added to the code new html content needs to be generated by running make html and committing/pushing the new content in the gh-pages/html/ directory.

That’s all there is to it, sphinx generated docs rendered by github via the gh-pages branch. All within a single directory tree. Have fun, and get to documenting!