Wednesday, October 31, 2007

Continuous Integration Strategies (Part II)

Your CI environment is reporting a broken build. Now what?

I would like to stress that the faster you jump on the problem, the easier it is to solve. The changeset will be smaller and the person who most likely committed the offending code will have the changes he made freshly in mind.
It is a good policy that your team does not commit any additional changes to source control until the build is fixed.
At times, it is too easy to ignore a broken build message, whether it is an email notification or a flashing red light or a lava lamp. Sometimes, the team will assume someone else is working the issue. I will always recommend having immediate and active communication that the problem is being worked so that positive control in maintained.

There were some additional words of wisdom that recently circulated here that I would like to share with you. Props to Chad for nailing the importance of the entire team owning CI and having active communication about its status. The emphasis is mine.

Make sure you're at least running unit tests before you commit. You can also have a buddy immediately update and build if you want feedback before waiting for Cruise Control to build. Also, after you commit, watch your email for a notification that the build has broken. If you're not going to be around, use a check-in buddy as mentioned previously.

If you've just committed a change and receive a build failure notification email, look into what's causing the problem asap. If it's a quick fix, just make the change and re-commit. Optionally reply to the build failure notification so that the team is aware you're putting in the fix. If it's not a quick fix, reply to the build failure notification stating that you're working the issue; again so the team is aware.

...The build is everyone's responsibility.

If you see that the build is remaining broken for a period of time, take it upon yourself to investigate. Find out if anyone is working the issue. If not, try to identify the problem and notify the party responsible so that the build can be fixed quickly. Now if you can't figure out what's wrong and identify who is responsible, take it upon yourself to fix the issue. If you are too busy, find someone that can. If you can't figure it out, ask for help. Use it as an opportunity to learn something. When you do finally find the problem, let the responsible party know what happened. Then they can learn something as well.

To sum it up, there shouldn't be any duration of time where the build is broken but isn't being looked into. While everyone should be watching after they commit to see if they've broken the build, it won't always get caught. You shouldn't be saying to yourself "I didn't break it so it's not my job to fix it".
And that's the word.

1 comment:

Michael Hüttermann said...

Hello Jeff,
your post is very good, thanks for sharing your experience on this important topic.

I like your statements ... yes, CI is a topic for the whole team and "It is a good policy that your team does not commit any additional changes to source control until the build is fixed." In order to accomplish this the team must be aware of the build status. Who do you accomplish this best?

You are also right that it is essential to run tests (e.g. unit tests) before commiting to the CI-system. The absolute minimum!! But what if you check something in that will break the build neverthess (delta in tests on the systems) or you forget to run the tests or you do not run them because they run too long?

Did you ever try the CI-server TeamCity? There you have the feature to do a "remote run" i.e. you can the run the build on the dedicated remote server *but* not as part of the single-point-of-truth build system. It is a private, isolated build. It is done on a distributed build grid .. and combine this with the "delayed commit" feature. If and only if the "remote run" (which does not occupy your personal computer!!! you can continue your work meanwhile) goes through successfully the changes will be commited to the VCS automatically. This is thrilling ... you do not have to cross your fingers and wait for some mail notifications from a central build system or wait for the peer who will come to your office with an angry face. You can use the features out of your IDE having the TeamCity plugin.

Concerning taking the responsiblity .. yes, the whole team is responsible but how do you know who is working on the issue? With TeamCity all stakeholders have access to its appealing Web 2.0 web application (the interface to the core build application) where you can easily take the issue to you or see who "owns the defect" ... You mention CruiseControl -- it is good, but sometimes it is annoying to customize everything via XML. In Teamcity this isn't necessary.

There is also a smooth integration of TeamCity to your IDE. Try out the TeamCity plugin (for IntelliJ, Eclipse) to monitor or trigger builds !! There you can also see who took responsibility of issues ... e.g. you yourself. :-)

You talk about tests ... with TeamCity you have the chance to configure and run test metrics, code coverage ... with implicit support of the JetBrains product family.

It is always the people who use the tools and live the process. So I'm supporter of agile methods which have answers to all the critical points you mentioned. And with TeamCity there is a fantastic tool on the market (even for free in the entry version) which supports being agile !! You should try it out, it is amazing !!