Go Blog
What the Go team is thinking about
Posts
- 1-6 of 6
Blog Entry posted June 18, 2009 by Chris Stevenson , tagged distributed-source-control-management, Go, version-control-systemsPersonal Builds with Go (Cruise) and MercurialA Personal Build allows a developer to send their changes to a continuous integration server to find out if all the tests are passing before they commit them to the central repository. Personal builds can increase developer productivity by allowing them to continue developing while a large or slow test suite is running. It can also improve the productivity of the team by ensuring that when code changes are checked in they do not break the build.
This post talks about how we use Mercurial and Go (Cruise) to implement personal builds.
The problem
Ideally a developer will run all the tests before they check into a central repository. But sometimes this is not practical. For example the product under development may need to run on many platforms. Running all the tests on all the possible environments would be prohibitively slow.
On Cruise we have Servers and Agents, each of which are supported on Linux (Ubuntu, Redhat and CentOS), Solaris, MacOSX and Windows. The Web UI is supported on Firefox (2/3) Safari and IE 7/8 (not 6 thankfully).
All of this is tested of course. We have unit and integration tests which we run on Linux and Windows, and regression tests that run in Twist on both Linux and Windows. But all of this can take a lot of time. This is particularly so for us since as a continuous integration product we do a lot of spawning external commands on the various platforms and SCMs and waiting for them to complete.
We parallelize our test suites using test load balancer and the whole stage takes less than 10 minutes, but even this feels very slow when you are in the middle of something interesting.
We like checking in often and mercurial makes it very easy for us to do this. Mercurial is a distributed source control management system (DSCM) which means that every developer has their own local repository. We can check in locally many times a day, merge changes from the central repository, and later on decide to commit a set of changes (possibly more than one changeset) back to the central repository. This is particularly useful for those of us who are working in San Francisco, since our central repository is in Beijing, and non-distributed SCMs are painfully slow in this scenario.
It is useful to run tests very often, and indeed we do that. But when you have a long or complicated test suite it becomes painful to have to wait, and the effect is that we batch up larger changes, or run the tests less often. Even a 10 minute build is too long if you want to check in every few minutes.
The personal pipeline
In this scenario it would be nice to have a build that could run comprehensive tests in the background while we keep developing the next feature. If the comprehensive tests expose a bug then we can go back and fix that bug quickly before checking the changes into the central repository. (It is possible to do this using mercurial queues)
Since we are using a DSCM this is actually very easy to implement. Mercurial can push changes from a repository to any other related repository. there are no ‘privileged’ repositories. It is a convention that we always push to the central repository. In fact we often push or pull changes to other developer repositories if there are changes that we want to work on that we don’t yet want to send to the central repository.
So to set up a personal build, all a developer needs to do is create a local clone of the repository, start an hg server, and create a pipeline in Cruise that monitors that repository. When the developer wants to run the tests on their current set of changes they push their changes to that local clone.
Cruise automatically picks up the changes and runs the pipeline. If the tests fail we go back and fix the problem, and then repeat the process. Once our work is complete we push the full set of changes to the central repository, and the main Cruise pipeline runs.
Using Cruise Free Edition to run personal builds
The new 1.3 release of Cruise allows you to run more than one Agent on a single machine. The Cruise free edition license allows you to run as many of these local Agents as you like (but no remote Agents). So it is possible to set up a personal build server with a few Agents using the Cruise free edition.
This is actually what we do in the Cruise team (partly to dogfood Cruise itself). We have a 4 core Ubuntu machine that runs 8 local Agents and a Cruise Server with a free license. There is a pipeline for each pairing station (yes we pair on everything) and we use this regularly to test our changes before we check in to the central repository.
Example
Create a local clone of the repository:
$ hg clone cruise cruise-personal
Run hg serve:
$ cd cruise-personal $ hg serve -v listening at http://X.X.X.X:8000/ (bound to *:8000)
Set up a pipeline with the new personal build:
The personal build config looks like this:
<pipeline name="CCEDev05-X.X.X.X"> <materials> <hg url="http://X.X.X.X:8000" /> </materials> <stage name="fast"> <jobs> <job name="fast-1"> <tasks> <ant target="-Drun=all -Divy.default.ivy.user.dir=target/ivy -Dhelp.skip=true clean ut ft"> </ant> </tasks> <resources> <resource>personal</resource> </resources> </job> </jobs> </stage> </pipeline>Acknowledgements
Thanks to WPC and the Mingle team for their work on their build grid (which deserves a post of its own) and informed a lot of what we do with our personal builds.
a simple tool to manage artifacts repository with Go (Cruise)Li Jian (TWer in China office) took few days to write ruby code to clean the artifacts repository of cruise automatically.
It is ruby code on google code (svn checkout http://cruisetools.googlecode.com/svn/trunk/ ). Anyone can check out to use.
You can run it to clean the artifacts under the artifacts folder. If some files/folders need to be kept, just put the relative path of files/folders in the file 'foldersToBeKept.txt' (One path in each line)
Then run the script 'rake dirname=/var/lib/cruise-server/logs/pipelines time=1w', it will delete all folders which are generated 1 week ago except for the folders in the kept list.
the param 'dirname' is the absolute path for the repository,
the param 'time' could be 1w, 2w, .... (means to delete the folders n weeks ago)
1m, 2m, .... (means to delete the folders n months ago)
2009-05-01 ... (meams to delete the folders before 2009-05-01)
you can run cron task to clean it. And our approach is following.We setup a pipeline called 'cleanArtifacts' to clean the folder which is generated 1 week ago except for the ones in the list of folderToBeKept.txt.
And the job in this pipeline will be run by the agent which is running on the same box of Cruise server.
We only want to keep the important artifacts and get rid of others. So we setup a subversion repository say 'http://xx.xx.xx.xx:8080/svn/artifactsmanagement'. And only one file 'foldersToBeKept.txt' in the repository.
<pipeline name="cleanArtifacts">
<materials>
<svn url="http://cruisetools.googlecode.com/svn/trunk/artifactsmanagement/lib" dest="app" />
<svn url="http://xx.xx.xx.xx:8080/svn/artifactsmanagement/trunk" username="yyy" password="***" dest="keptfolder" />
</materials>
<stage name="defaultStage">
<jobs>
<job name="clean">
<tasks>
<exec command="/bin/bash" args="-c " cp foldersToBeKept.txt ../app"" workingdir="keptfolder" />
<rake target="dirname=/var/lib/cruise-server/logs/pipelines time=1w" workingdir="app" />
</tasks>
</job>
</jobs>
</stages>
</pipeline>
- Blog Entry posted May 26, 2009 by Qiao Liang 乔梁 , tagged distributed-source-control-management, Go, version-control-systems使用Go (Cruise) 和Mercurial实现PersonalBuild
什么是个人预提交（Personal build）?
Personal build简单来说，就是开发人员在代码提交之前，先要自己在本地运行一次构建和测试代码，保证本地没有测试失败后，再将其提交到中央代码仓库。
Personal build的痛处在哪里？“提交代码之前，必须在本地运行并通过单元测试”是敏捷团队的原则之一。而随着新功能的增加，我们的单元测试越来越多，运行时间当然也就越来越长，那么开发人员等待的时间就长。
因此，正如在InfoQ 上的一篇关于持续集成实践的文章 所 述，我们在每台PairStation上都安装了一个虚拟机来运行自己提交前的单元测试。然而，每台虚拟机都运行自己的单元测试，也就是说，在运行时间上没有什么大的改进，还是需要三十分钟左右（因为还是在一台机器上运行），其好处是执行本地单元测试的同时，开发人员可以继续他们自己的工作，但本地单元测 试的时间并未缩短，反馈时间长的问题还是存在。如何解决这个问题？另外，在目前尚未发布的Cruise R1.3 中，有个新功能就是：多个Agent可以运行于同一个机器上。因此，在那台安装有CruiseServer的PersonalBuild服务器上同时又启动了四个Agent。个人构建持续集成服务器就有至少八台Agent可以同时工作啦。如图1所示。图1由于团队使用的是Mercurial （分布式版本控制服务器），所以开发人员所要做的准备工作就是：（１）在自己的机器上再Clone一个本地的Repository，运行hg serve 启动该Repository的http服务。（２） 在Personalbuild Server上建立自己的一个Pipeline，该Pipeline的materials指向这个新建的Repository。且Pipeline上只有 一个Stage，同时利用Test-load-balancer将单元测试分成多个Job加入到该Stage中。这样，利用Cruise的Job并行化和Agent的分布化，以及Mercurial的特点，使得每个人都相当于拥有八台Agent为其工作。而建立PersonialBuild后的开发流程大致为如下步骤(见图2)：（１）开发人员对本地代码进行修改；（２）开发人员将代码放到本地预提交仓库中；（３）个人构建服务器发现本地预提交仓库的代码变化，触发个人的Pipeline进行构建；（４）开发人员重复（１）和（２），并关注自己的个人构建Pipeline的运行结果；（５）个人构建服务器重复（３）；（６）开发人员使自己的最后一次提交在个人构建服务器上通过所有测试；（７）开发人员将所有修改提交到中央代码仓库中；（８）团队集成构建服务器发现这些变化的代码，触发团队Pipeline的构建。图2其好处在于：(1) 不需要太多的准备工作(原本就使用Mecurial)；(2) Cruise的分布式加快了个人构建与测试的反馈速度；(3) 更大的灵活性(开发人员可以有选择性的向中央代码库提交任意指定的本地版本)；(4) 促进开发人员频繁提交，运行测试；(5) 在团队持续集成环境中，失败的次数会大大减少；具体准备工作：(1) 建立本地代码库的复本
hg clone cruise cruise-personal
cd cruise-personal
hg serve
(2) 启动一个agent，指向PersonalBuild 服务器
mkdir cruise-personal-agent
cp cruise/target/cruise-agent-1.3/* cruise-personal-agent
cd cruise-personal-agent
java -jar agent-bootstrapper.jar ${your personal build IP address}
(3) 在建立一个pipeline，使其Materials指向第1步中建立的本地代码库复本具体工作过程：(1) 频繁向本地checkin代码；(2) 需要个人构建时，使用命令'hg push ../cruise-personal'(3)个人构建服务器会发现代码的改变而构建。(4) 当需要向中心代码库提交时，使用命令'hg pull --rebase'注意：
当 我们想要运行个人构建时，如果使用命令 'hg push ../cruise-personal'，你会得到信息说‘creating multiple heads’。假如你使用命令 'hg push -f ../cruise-personal' (注意那个 -f)，就会强制Push并创建multiple heads。在中心代码库上建立多个head是非常不好的。但是，对于个人的本地复本来说，无所谓。实际上，Cruise会发现并更新到最新的head上，而忽略旧的head。 - Use resource-matching to get stage grouped by Go (Cruise) APIs
A team wants two stages in one pipeline to run as group. The reason is:
They have three stages in one pipeline, and the first stage for unit test, the second one for server deployment, and the last one for functional test which will hit the servers that deployed in the previous stage 'deployServer'.
The unit test stage runs faster than the other two, so it is possable to deploy a new version of the application at the time the functional test is running if checkin frequently.
It could handle this case by cruise-resource-matching and ruby script with cruise Json APIs:
Setup a job in 'FunctionTest' stage called 'checkJob' and it needs the same resource with the job in 'deployServer' stage and will treat the script as a task.
When the third stage is running, 'checkJob' will occupy the resource and run a task which is a ruby script to check whether the other jobs in 'FunctionTest' stage are finished or not. If other jobs are finished, 'checkJob' is going will finish immidately. Once 'checkJob' finishes, the resource will release, and the new version could start to run on 'deployServer' stage agent.
The attachment is an example, including a rake file and a section of cruise-config.xml. However You have to modify them to feed your pipeline and check in the rake file to your repository if you want to use them. Also the script needs 'json-1.1.3.gem'.
Notes: It works with 1.3.
Go (Cruise) in the newsIn an exciting milestone for us, we had our first blog review from somebody who has never worked at ThoughtWorks.
Also Julian Simpson, who runs a blog called The Build Doctor, has put up a video interview (with transcription) he did with me a couple of weeks back. We discuss continuous deployment, the value proposition of Cruise, the competition, and a bit about Cruise's history and our future plans. It's in two parts.
Parallelize your JUnit tests with the Test Load BalancerOne of Cruise's most valuable features is that it runs jobs in parallel. To make long running test suites go faster, you can simply split your long running test suite into multiple suites and create jobs for each one. Given enough build agents, Cruise can run all jobs within a specified stage simultaneously, which means your long running testing can be done in a snap.
But wait, why do I need to manually split my test suites? Enter the test load balancer.
For example, I have a unit-test target in my ant build script which includes more than 100 tests. Those tests are organized by corresponding packages, so it's hard to split them in an easy way. It's a burden to users of Cruise.
What is the test load balancer?
The TLB (test load balancer) is a custom ant task that can split test suites written in junit for you. With TLB, you can simply specify in Cruise how many batches you want to split your test suite into. TLB talks to Cruise to work out how many batches your suits is split into and which of these batches the current job should do.
How do I use it?
First, download test load balancer jar file and add it to the classpath of ant build script.
Then, change your build.xml as following:
<typedef name="filter-fileset" classname="com.googlecode.tlb.support.junit.FilterFileSet" classpathref="classpath"/>
<junit>
<batchtest todir="target/test-results">
<filter-fileset dir="target/test-classes" includes="**/*Test.class"/>
</batchtest>
<classpath refid="classpath" />
</junit>
Third, change your Cruise configuration. You will need to create a job for each batch of tests you want to run. Cruise makes sure these jobs will run in parallel. You need to make each job run the same ant task - the one you put the typedef above into. But you use a naming convention so that the TLB can recognize them.For example, say you have a unit test job running on Linux environment named 'ut-linux' that has 100 tests and takes 10 minutes. You want to split it to three batches. You simply replace the old 'ut-linux' job with three jobs: 'ut-linux-1', 'ut-linux-2', 'ut-linux-3'. Each job should do exactly the same thing as the original ut-linux job.
That's it - you're done! Now you can get the benefits of faster, parallelized test running with no changes to your junit test suite.
Further information about the test load balancer.