Indispensable, Disposable Jenkins

August 15, 2017 - by Mandy Hubbard, Software Engineer, HomePay

Imagine this: It’s 4:30pm on a Friday, you have a major release on Monday, and your Jenkins server goes down. It doesn’t matter if it experienced a hardware failure, fell victim to a catastrophic fat-finger error, or just got hit by a meteor — your Jenkins server is toast. How long did it take to perfect your Pipeline, all your Continuous Delivery jobs, plugins, and credentials? Hopefully you at least have a recent backup of your Jenkins home directory, but you’re still going to have to work over the weekend with IT to procure a new server, install it, and do full regression testing to be up and running by Monday morning. Go ahead and take a moment, go to your car and just scream. It will help…a little.

But what if you could have a Jenkins environment that is completely disposable? Using Docker and Joyent’s ContainerPilot, the team at HomePay has created a production Jenkins environment that is completely software-defined. Everything required to set up a new Jenkins environment is stored in source control, versioned, and released just like any other software.

First, ContainerPilot is added to our Jenkins image by including it in the Dockerfile.

## ContainerPilot

ENV CONTAINERPILOT_SHA256 3cf91aabd3d3651613942d65359be9af0f6a25a1df9ec9bd9ea94d980724ee13
ENV CONTAINERPILOT file:///etc/containerpilot/containerpilot.json

RUN curl -Lso /tmp/containerpilot.tar.gz${CONTAINERPILOT_VERSION}/containerpilot-${CONTAINERPILOT_VERSION}.tar.gz && \
    echo "${CONTAINERPILOT_SHA256}  /tmp/containerpilot.tar.gz" | sha256sum -c && \
    tar zxf /tmp/containerpilot.tar.gz -C /bin && \
rm /tmp/containerpilot.tar.gz

Then we specify ‘containerpilot’ as the Docker command in the docker-compose.yml and pass the Jenkins startup script as an argument. This allows ContainerPilot to perform our preStart business before starting the Jenkins server.

    image: devmandy/auto-jenkins:latest
    restart: always
    mem_limit: 8g
      - 80
      - 22
    env_file: _env
      - /var/run/docker.sock:/var/run/docker.sock
      - CONSUL=consul
      - consul:consul
      - "8080:80"
      - "2222:22"
    command: >

Configuration data is read from a Docker Compose _env file, as specified in the docker-compose.yml file, and stored in environment variables inside the container. This is an example of our _env file:

GITHUB_TOKEN=<my Github user token>
DOCKERHUB_PASSWORD=<my Dockerhub password>
DOCKER_HOST=<my Docker host, or localhost>
SLACK_TOKEN=<my Slack token>
BASIC_AUTH=<my basic auth token>
AD_NAME=<my AD domain>
AD_SERVER=<my AD server>
PRIVATE_KEY=<my ssh private key, munged by a setup script>

As you may know, Jenkins stores its credentials and plugin information in various xml files. The preStart script modifies the relevant files, substituting the environment variables as appropriate, using a set of command line utilities called xmlstarlet. Here is an example method from our preStart script that configures Github credentials:

github_credentials_setup() {
    ## Setting Up Github username in credentials.xml file
    echo -e "Adding Github username to credentials.xml file for SSH key"
    xmlstarlet \
        ed \
        --inplace \
        -u '//com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey[id="github"]/username' \
        -v ${GITHUB_USERNAME} \

    echo -e "Adding Github username to credentials.xml file for Github token"
    xmlstarlet \
        ed \
         --inplace \
        -u '//com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl[id="github_token"]/username' \
        -v ${GITHUB_USERNAME} \

    echo -e "Adding Github token to credentials.xml"
    xmlstarlet \
        ed \
        --inplace \
        -u '//com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl[id="github_token"]/password' \
        -v ${PASSWORD} \

This approach can be used to automate all things Jenkins. These are some of the things our team has automated:

  1. Creation of credentials sets for interacting with third party services like Github, Docker Hub and Slack
  2. Configuration of the Active Directory plugin and setup of matrix-based security
  3. Configuration of the Github Organization plugin, which results in the automatic creation of all Jenkins pipeline jobs by scanning the organization for all repositories containing a Jenkinsfile
  4. Configuration of the Docker plugin, including creating templates for all custom build slaves
  5. Configuration of the Global Pipeline Libraries plugin
  6. Configuration of the Global Slack Notifier plugin

With software-defined Jenkins we can make the development pipeline as flexible and resilient as the rest of the development process. If we decide to change our Jenkins configuration in any way — for example installing a new plugin or upgrading an existing one, adding a new global library, or adding new Docker images for build slaves — we simply edit our preStart script to include these changes, build a new Docker image, and the Jenkins environment is automatically reconfigured when we start a new container. Because all this configuration specification lives in a Github repository, changes are merged to the master branch using pull requests, and our Jenkins Docker image is tagged using semantic versioning just like any other software. Because everything is driven from that repo, we can recover from failure just by deploying it again and know that we can restore this critical aspect of our development pipeline. Jenkins can be both indispensable and completely disposable at the same time.