The problem

Ideally a developer will run all the tests before they check into a central repository. But sometimes this is not practical. For example the product under development may need to run on many platforms. Running all the tests on all the possible environments would be prohibitively slow.

On Cruise we have Servers and Agents, each of which are supported on Linux (Ubuntu, Redhat and CentOS), Solaris, MacOSX and Windows. The Web UI is supported on Firefox (2/3) Safari and IE 7/8 (not 6 thankfully).

All of this is tested of course. We have unit and integration tests which we run on Linux and Windows, and regression tests that run in Twist on both Linux and Windows. But all of this can take a lot of time. This is particularly so for us since as a continuous integration product we do a lot of spawning external commands on the various platforms and SCMs and waiting for them to complete.

We parallelize our test suites using test load balancer and the whole stage takes less than 10 minutes, but even this feels very slow when you are in the middle of something interesting.

We like checking in often and mercurial makes it very easy for us to do this. Mercurial is a distributed source control management system (DSCM) which means that every developer has their own local repository. We can check in locally many times a day, merge changes from the central repository, and later on decide to commit a set of changes (possibly more than one changeset) back to the central repository. This is particularly useful for those of us who are working in San Francisco, since our central repository is in Beijing, and non-distributed SCMs are painfully slow in this scenario.

It is useful to run tests very often, and indeed we do that. But when you have a long or complicated test suite it becomes painful to have to wait, and the effect is that we batch up larger changes, or run the tests less often. Even a 10 minute build is too long if you want to check in every few minutes.

The personal pipeline

In this scenario it would be nice to have a build that could run comprehensive tests in the background while we keep developing the next feature. If the comprehensive tests expose a bug then we can go back and fix that bug quickly before checking the changes into the central repository. (It is possible to do this using mercurial queues)

Since we are using a DSCM this is actually very easy to implement. Mercurial can push changes from a repository to any other related repository. there are no ‘privileged’ repositories. It is a convention that we always push to the central repository. In fact we often push or pull changes to other developer repositories if there are changes that we want to work on that we don’t yet want to send to the central repository.

So to set up a personal build, all a developer needs to do is create a local clone of the repository, start an hg server, and create a pipeline in Cruise that monitors that repository. When the developer wants to run the tests on their current set of changes they push their changes to that local clone.

Cruise automatically picks up the changes and runs the pipeline. If the tests fail we go back and fix the problem, and then repeat the process. Once our work is complete we push the full set of changes to the central repository, and the main Cruise pipeline runs.

Using Cruise Free Edition to run personal builds

The new 1.3 release of Cruise allows you to run more than one Agent on a single machine. The Cruise free edition license allows you to run as many of these local Agents as you like (but no remote Agents). So it is possible to set up a personal build server with a few Agents using the Cruise free edition.

This is actually what we do in the Cruise team (partly to dogfood Cruise itself). We have a 4 core Ubuntu machine that runs 8 local Agents and a Cruise Server with a free license. There is a pipeline for each pairing station (yes we pair on everything) and we use this regularly to test our changes before we check in to the central repository.

Example

Create a local clone of the repository:

$ hg clone cruise cruise-personal

Run hg serve:

$ cd cruise-personal
$ hg serve -v
listening at http://X.X.X.X:8000/ (bound to *:8000)

Set up a pipeline with the new personal build: 

The personal build config looks like this:

<pipeline name="CCEDev05-X.X.X.X">
  <materials>
    <hg url="http://X.X.X.X:8000" />
  </materials>
  <stage name="fast">
    <jobs>
      <job name="fast-1">
        <tasks>
          <ant target="-Drun=all -Divy.default.ivy.user.dir=target/ivy 
            -Dhelp.skip=true clean ut ft">
          </ant>
        </tasks>
        <resources>
          <resource>personal</resource>
        </resources>
      </job>
     </jobs>
  </stage>
</pipeline>

Acknowledgements

Thanks to WPC and the Mingle team for their work on their build grid (which deserves a post of its own) and informed a lot of what we do with our personal builds.