Cruise > Cruise Blog

Making Cruise Work With TFS (3 Comments)

Thu, 03 Sep 2009 07:18:08 +0000

Entry

Cruise doesn't support TFS directly. But if you are using TFS as your source control system, and you want to try Cruise, there is a good solution: SvnBridge.

Basically, SvnBridge acts as a bridge between a Subversion client and a TFS server, translating subversion commands to TFS ones. In this way you can use Cruise's built-in Subversion support to access TFS repositories.

There are two ways to setup SvnBridge:

standalone utility

This is a simple way to try SvnBridge. What you need to do is just download the utility and run it. Choose a listening port as the svn server port. That's it. Now you can use the following svn url to access your TFS repository:
http://localhost:[port]/[TFS-hostname]/[your-project-name]

Note: with this approach you can only use it on localhost. The SvnBridge exposed svn repository is not available for remote access. That means you can't use remote agents on Cruise to access your TFS repository.

application service running in IIS

This can be your production choice. In this case the SvnBridge runs in IIS on the same host with your TFS server. So download the SvnBridge server application and install it following the instruction in the website. Make sure you have .Net 2 (or better) framework installed before that.

Two things, which are a little different with the official instruction:
- the location of aspnet_isapi.dll is in your %windir%, not always c:\winnt, depends on your OS.
- for the authentication methods instruction, the recommended digest authentication doesn't work with svn command line, but it works with basic authentication.
After installation, you can use the following url to access the exposed svn repository from anywhere: http://[server-host]:[port]/[your-project-name]

So, once you have setup SvnBridge successfully, you got an svn interface for the TFS repository. You can just use it as an svn repository in Cruise. Following the Cruise instruction you can easily setup an svn pipeline to work with your TFS repository.

Keywords

TFS,svnbridge

Use resource-matching to get stage grouped by Cruise APIs (1 Comment)

Mon, 18 May 2009 02:00:54 +0000

Entry by Qiao Liang 乔梁

Entry

A team wants two stages in one pipeline to run as group. The reason is:

They have three stages in one pipeline, and the first stage for unit test, the second one for server deployment, and the last one for functional test which will hit the servers that deployed in the previous stage 'deployServer'.

The unit test stage runs faster than the other two, so it is possable to deploy a new version of the application at the time the functional test is running if checkin frequently.

It could handle this case by cruise-resource-matching and ruby script with cruise Json APIs:

Setup a job in 'FunctionTest' stage called 'checkJob' and it needs the same resource with the job in 'deployServer' stage and will treat the script as a task.

When the third stage is running, 'checkJob' will occupy the resource and run a task which is a ruby script to check whether the other jobs in 'FunctionTest' stage are finished or not. If other jobs are finished, 'checkJob' is going will finish immidately. Once 'checkJob' finishes, the resource will release, and the new version could start to run on 'deployServer' stage agent.

The attachment is an example, including a rake file and a section of cruise-config.xml. However You have to modify them to feed your pipeline and check in the rake file to your repository if you want to use them. Also the script needs 'json-1.1.3.gem'.

Notes: It works with 1.3.

Keywords

stage group, resource matcher

File

stage-group.rar (1KB)

Personal Builds with Cruise and Mercurial

Thu, 18 Jun 2009 23:57:40 +0000

Entry by Chris Stevenson

Entry

A Personal Build allows a developer to send their changes to a continuous integration server to find out if all the tests are passing before they commit them to the central repository. Personal builds can increase developer productivity by allowing them to continue developing while a large or slow test suite is running. It can also improve the productivity of the team by ensuring that when code changes are checked in they do not break the build.

This post talks about how we use Mercurial and Cruise to implement personal builds.

Image

The problem

Ideally a developer will run all the tests before they check into a central repository. But sometimes this is not practical. For example the product under development may need to run on many platforms. Running all the tests on all the possible environments would be prohibitively slow.

On Cruise we have Servers and Agents, each of which are supported on Linux (Ubuntu, Redhat and CentOS), Solaris, MacOSX and Windows. The Web UI is supported on Firefox (2/3) Safari and IE 7/8 (not 6 thankfully).

All of this is tested of course. We have unit and integration tests which we run on Linux and Windows, and regression tests that run in Twist on both Linux and Windows. But all of this can take a lot of time. This is particularly so for us since as a continuous integration product we do a lot of spawning external commands on the various platforms and SCMs and waiting for them to complete.

We parallelize our test suites using test load balancer and the whole stage takes less than 10 minutes, but even this feels very slow when you are in the middle of something interesting.

We like checking in often and mercurial makes it very easy for us to do this. Mercurial is a distributed source control management system (DSCM) which means that every developer has their own local repository. We can check in locally many times a day, merge changes from the central repository, and later on decide to commit a set of changes (possibly more than one changeset) back to the central repository. This is particularly useful for those of us who are working in San Francisco, since our central repository is in Beijing, and non-distributed SCMs are painfully slow in this scenario.

It is useful to run tests very often, and indeed we do that. But when you have a long or complicated test suite it becomes painful to have to wait, and the effect is that we batch up larger changes, or run the tests less often. Even a 10 minute build is too long if you want to check in every few minutes.

The personal pipeline

In this scenario it would be nice to have a build that could run comprehensive tests in the background while we keep developing the next feature. If the comprehensive tests expose a bug then we can go back and fix that bug quickly before checking the changes into the central repository. (It is possible to do this using mercurial queues)

Since we are using a DSCM this is actually very easy to implement. Mercurial can push changes from a repository to any other related repository. there are no ‘privileged’ repositories. It is a convention that we always push to the central repository. In fact we often push or pull changes to other developer repositories if there are changes that we want to work on that we don’t yet want to send to the central repository.

So to set up a personal build, all a developer needs to do is create a local clone of the repository, start an hg server, and create a pipeline in Cruise that monitors that repository. When the developer wants to run the tests on their current set of changes they push their changes to that local clone.

Cruise automatically picks up the changes and runs the pipeline. If the tests fail we go back and fix the problem, and then repeat the process. Once our work is complete we push the full set of changes to the central repository, and the main Cruise pipeline runs.

Using Cruise Free Edition to run personal builds

The new 1.3 release of Cruise allows you to run more than one Agent on a single machine. The Cruise free edition license allows you to run as many of these local Agents as you like (but no remote Agents). So it is possible to set up a personal build server with a few Agents using the Cruise free edition.

This is actually what we do in the Cruise team (partly to dogfood Cruise itself). We have a 4 core Ubuntu machine that runs 8 local Agents and a Cruise Server with a free license. There is a pipeline for each pairing station (yes we pair on everything) and we use this regularly to test our changes before we check in to the central repository.

Example

Create a local clone of the repository:

$ hg clone cruise cruise-personal

Run hg serve:

$ cd cruise-personal
$ hg serve -v
listening at http://X.X.X.X:8000/ (bound to *:8000)

Set up a pipeline with the new personal build:

The personal build config looks like this:

<pipeline name="CCEDev05-X.X.X.X">
  <materials>
    <hg url="http://X.X.X.X:8000" />
  </materials>
  <stage name="fast">
    <jobs>
      <job name="fast-1">
        <tasks>
          <ant target="-Drun=all -Divy.default.ivy.user.dir=target/ivy 
            -Dhelp.skip=true clean ut ft">
          </ant>
        </tasks>
        <resources>
          <resource>personal</resource>
        </resources>
      </job>
     </jobs>
  </stage>
</pipeline>

Acknowledgements

Thanks to WPC and the Mingle team for their work on their build grid (which deserves a post of its own) and informed a lot of what we do with our personal builds.

Keywords

cruise mercurial personal

File

personal-build.png (40KB)

a simple tool to manage artifacts repository with Cruise

Wed, 17 Jun 2009 13:41:07 +0000

Entry by Qiao Liang 乔梁

Entry

   Li Jian (TWer in China office) took few days to write ruby code to clean the artifacts repository of cruise automatically.

   It is ruby code on google code (svn checkout http://cruisetools.googlecode.com/svn/trunk/ ). Anyone can check out to use.

   You can run it to clean the artifacts under the artifacts folder. If some files/folders need to be kept, just put the relative path of files/folders in the file 'foldersToBeKept.txt' (One path in each line)

   Then run the script 'rake dirname=/var/lib/cruise-server/logs/pipelines time=1w', it will delete all folders which are generated 1 week ago except for the folders in the kept list.

   the param 'dirname' is the absolute path for the repository,
   the param 'time' could be 1w, 2w, .... (means to delete the folders n weeks ago)
                             1m, 2m, .... (means to delete the folders n months ago)
                             2009-05-01 ... (meams to delete the folders before 2009-05-01)

   you can run cron task to clean it. And our approach is following.

We setup a pipeline called 'cleanArtifacts' to clean the folder which is generated 1 week ago except for the ones in the list of folderToBeKept.txt.

And the job in this pipeline will be run by the agent which is running on the same box of Cruise server.

We only want to keep the important artifacts and get rid of others. So we setup a subversion repository say 'http://xx.xx.xx.xx:8080/svn/artifactsmanagement'. And only one file 'foldersToBeKept.txt' in the repository.

使用Cruise和Mercurial实现PersonalBuild

Wed, 27 May 2009 05:11:06 +0000

Entry by Qiao Liang 乔梁

Entry

什么是个人预提交（Personal build）?

Personal build简单来说，就是开发人员在代码提交之前，先要自己在本地运行一次构建和测试代码，保证本地没有测试失败后，再将其提交到中央代码仓库。

Personal build的痛处在哪里？

“提交代码之前，必须在本地运行并通过单元测试”是敏捷团队的原则之一。而随着新功能的增加，我们的单元测试越来越多，运行时间当然也就越来越长，那么开发人员等待的时间就长。

因此，正如在InfoQ 上的一篇关于持续集成实践的文章所述，我们在每台PairStation上都安装了一个虚拟机来运行自己提交前的单元测试。然而，每台虚拟机都运行自己的单元测试，也就是说，在运行时间上没有什么大的改进，还是需要三十分钟左右（因为还是在一台机器上运行），其好处是执行本地单元测试的同时，开发人员可以继续他们自己的工作，但本地单元测试的时间并未缩短，反馈时间长的问题还是存在。

如何解决这个问题？

在一台机器上新安装一个Cruise ，并在Pairstation的虚拟机上安装了一个Cruise Agent ，让它连上这个Cruise，这样就成了个人构建持续集成服务器。

另外，在目前尚未发布的Cruise R1.3 中，有个新功能就是：多个Agent可以运行于同一个机器上。因此，在那台安装有CruiseServer的PersonalBuild服务器上同时又启动了四个Agent。

个人构建持续集成服务器就有至少八台Agent可以同时工作啦。如图1所示。

图1

由于团队使用的是Mercurial （分布式版本控制服务器），所以开发人员所要做的准备工作就是：

（１）在自己的机器上再Clone一个本地的Repository，运行hg serve 启动该Repository的http服务。

（２）在Personalbuild Server上建立自己的一个Pipeline，该Pipeline的materials指向这个新建的Repository。且Pipeline上只有一个Stage，同时利用Test-load-balancer将单元测试分成多个Job加入到该Stage中。

这样，利用Cruise的Job并行化和Agent的分布化，以及Mercurial的特点，使得每个人都相当于拥有八台Agent为其工作。

而建立PersonialBuild后的开发流程大致为如下步骤(见图2)：

（１）开发人员对本地代码进行修改；

（２）开发人员将代码放到本地预提交仓库中；

（３）个人构建服务器发现本地预提交仓库的代码变化，触发个人的Pipeline进行构建；

（４）开发人员重复（１）和（２），并关注自己的个人构建Pipeline的运行结果；

（５）个人构建服务器重复（３）；

（６）开发人员使自己的最后一次提交在个人构建服务器上通过所有测试；

（７）开发人员将所有修改提交到中央代码仓库中；

（８）团队集成构建服务器发现这些变化的代码，触发团队Pipeline的构建。

图2

其好处在于：

(1) 不需要太多的准备工作(原本就使用Mecurial)；

(2) Cruise的分布式加快了个人构建与测试的反馈速度；

(3) 更大的灵活性(开发人员可以有选择性的向中央代码库提交任意指定的本地版本)；

(4) 促进开发人员频繁提交，运行测试；

(5) 在团队持续集成环境中，失败的次数会大大减少；

具体准备工作：

(1) 建立本地代码库的复本
        hg clone cruise cruise-personal
        cd cruise-personal
        hg serve
(2) 启动一个agent，指向PersonalBuild 服务器
        mkdir cruise-personal-agent
        cp cruise/target/cruise-agent-1.3/* cruise-personal-agent
        cd cruise-personal-agent
        java -jar agent-bootstrapper.jar ${your personal build IP address}

(3) 在建立一个pipeline，使其Materials指向第1步中建立的本地代码库复本

具体工作过程：

(1) 频繁向本地checkin代码；

(2) 需要个人构建时，使用命令'hg push ../cruise-personal'

(3)个人构建服务器会发现代码的改变而构建。

(4) 当需要向中心代码库提交时，使用命令'hg pull --rebase'

注意：

当我们想要运行个人构建时，如果使用命令 'hg push ../cruise-personal'，你会得到信息说‘creating multiple heads’。假如你使用命令 'hg push -f ../cruise-personal' (注意那个 -f)，就会强制Push并创建multiple heads。

在中心代码库上建立多个head是非常不好的。但是，对于个人的本地复本来说，无所谓。实际上，Cruise会发现并更新到最新的head上，而忽略旧的head。

Cruise in the news

Tue, 28 Apr 2009 20:57:17 +0000

Entry by Jez Humble

In an exciting milestone for us, we had our first blog review from somebody who has never worked at ThoughtWorks.

Also Julian Simpson, who runs a blog called The Build Doctor, has put up a video interview (with transcription) he did with me a couple of weeks back. We discuss continuous deployment, the value proposition of Cruise, the competition, and a bit about Cruise's history and our future plans. It's in two parts.

Parallelize your junit tests with the test load balancer

Tue, 14 Apr 2009 04:46:07 +0000

Entry by Li Yanhui

Entry

One of Cruise's most valuable features is that it runs jobs in parallel. To make long running test suites go faster, you can simply split your long running test suite into multiple suites and create jobs for each one. Given enough build agents, Cruise can run all jobs within a specified stage simultaneously, which means your long running testing can be done in a snap.

But wait, why do I need to manually split my test suites? Enter the test load balancer.

For example, I have a unit-test target in my ant build script which includes more than 100 tests. Those tests are organized by corresponding packages, so it's hard to split them in an easy way. It's a burden to users of Cruise.

What is the test load balancer?

The TLB (test load balancer) is a custom ant task that can split test suites written in junit for you. With TLB, you can simply specify in Cruise how many batches you want to split your test suite into. TLB talks to Cruise to work out how many batches your suits is split into and which of these batches the current job should do.

How do I use it?

First, download test load balancer jar file and add it to the classpath of ant build script.

Then, change your build.xml as following:

<typedef name="filter-fileset" classname="com.googlecode.tlb.support.junit.FilterFileSet" classpathref="classpath"/>

<junit>
<batchtest todir="target/test-results">
<filter-fileset dir="target/test-classes" includes="**/*Test.class"/>
</batchtest>
<classpath refid="classpath" />
</junit>

Third, change your Cruise configuration. You will need to create a job for each batch of tests you want to run. Cruise makes sure these jobs will run in parallel. You need to make each job run the same ant task - the one you put the typedef above into. But you use a naming convention so that the TLB can recognize them.

For example, say you have a unit test job running on Linux environment named 'ut-linux' that has 100 tests and takes 10 minutes. You want to split it to three batches. You simply replace the old 'ut-linux' job with three jobs: 'ut-linux-1', 'ut-linux-2', 'ut-linux-3'. Each job should do exactly the same thing as the original ut-linux job.

That's it - you're done! Now you can get the benefits of faster, parallelized test running with no changes to your junit test suite.

Further information about the test load balancer.

Cruise > Cruise Blog

Making Cruise Work With TFS (3 Comments)

Entry

More

Keywords

Use resource-matching to get stage grouped by Cruise APIs (1 Comment)

Entry

Keywords

File

Personal Builds with Cruise and Mercurial

Entry

Image

More

The problem

The personal pipeline

Using Cruise Free Edition to run personal builds

Example

Acknowledgements

Keywords

File

a simple tool to manage artifacts repository with Cruise

Entry

More

使用Cruise和Mercurial实现PersonalBuild

Entry

More

Cruise in the news

Parallelize your junit tests with the test load balancer

Entry

More