Continuous Integration Strategies (Part I)

Continuous integration is a powerful concept, usually associated with only compilation and unit testing. However, there is additional benefit to be had if you look beyond unit testing. I would like to present some strategies that I have tried that allow full suites of tests to be ran in orderly stages, from unit tests, to integration tests to acceptance tests. For this series unit tests are defined as single class tests with no external dependencies on network or container resources. Integration tests are white box testing of class interactions and acceptance tests are black box system tests.

This first post on the subject deals with strategies using maven and cruisecontrol. Later posts will move on to maven 2 and hudson.

First off, a lesson learned. When we first migrated from ant to maven, we were not sure how best to configure cruisecontrol to handle CI. Our code base is a large selection of components that comprise a toolset of capabilities. There are many small projects that build on each other, so there are many dependencies on our on artifacts. In fact, from a maven point of view, we could build our toolset with a single, rather large, multiproject build.

It seemed logical to map each component maven project (itself a multiproject consisting of api + implementations + tests) to a cruisecontrol project. That presented a nice one-to-one view of the system on the build status page. Each project was independently triggered via cvs commits. This seemed to work for awhile, but it became clear that this is highly unstable because commits spanning multiple cruisecontrol projects would trigger the builds in an unpredictable order, causing the build to break or tests to fail.

The lesson learned and correction we took was to not fight maven, but let it determine the build order from start to finish. So we created a single cruisecontrol project, pointed it at the top-most maven project.xml with the goal multiproject:install and the property -Dmaven.test.failure.ignore=true. Then for each component project we wanted test status granularity on, we created a cruisecontrol project that ran a custom maven plugin that scanned the test-results directory and failed that project if test failures were found. Additionally, that cruisecontrol project also used <merge> to aggregate maven's test-results files so our developers could drill down and see which test failed and the details why.

As a quick aside, Hudson has very nice maven integration that mimics (and improves on) this kind of setup automatically.

Our next step was to enable a controlled progression of testing, where all unit test would run first, followed by integration tests only if all unit tests passed. This was accomplished in three steps: 1) one maven multiproject build responsible for compiling and unit testing, 2) a custom test failure check plugin (basically find + grep) serving as the go-no-go gate, then 3) another maven multiproject build running only integration tests. This three step orchestration was handled by a custom maven plugin running a mix of jelly and shell scripting.

The different types of tests were in different directories, as maven subprojects, under the component, so we were able to use maven.multiproject.includes and maven.multiproject.excludes on the directory names to achieve steps 1) and 3) above.

To facilitate the the including and excluding for the unit test pass, the ~/build.properties included these properties:

  maven.multiproject.includes=**/project.xml
 maven.multiproject.excludes=project.xml,*/project.xml,**/inttest/project.xml,**/tck/project.xml

For the integration test pass, the ~/build.properties included these properties:

  gcp.integration.multiproject.includes=**/inttest/project.xml
 gcp.integration.multiproject.excludes=**/tck/project.xml

and the plugin goal to actually run the integration tests looked like this:

  <goal name="gcp:integration-tests">
   <j:set var="usethese" scope="parent" value="${gcp.integration.multiproject.includes}"/>
   <j:set var="notthese" scope="parent" value="${gcp.integration.multiproject.excludes}"/>
   <j:set var="thisgoal" scope="parent" value="test:test"/>
   ${systemScope.put('maven.multiproject.includes', usethese)}
   ${systemScope.put('maven.multiproject.excludes', notthese)}
   ${systemScope.put('goal', thisgoal)}
   <maven:maven descriptor="${CC_HOME}/checkout/gcp/project.xml" goals="multiproject:goal"/>
 </goal>

The crazy jelly maneuvering to get the includes and excludes properties to stick after the call to maven:maven is a story for another day. (If you want a sneek peek, however, start here.) The concept of dynamically using maven properties to properly setup the integration test run should still be clear.

Hopefully, this first post in a series will help you think about ways to get more out of your CI environment. I'm curious to know what you think about the strategy we took and I invite you to share how you accomplish CI for your projects.