The Supply Chain of Continuous Integration

When I was first introduced to Continuous Integration, I viewed it as a black box with a well defined interface. It was the same kind throw-it-over-the-wall mentality that some people have with testing: When the code is "done", give it to the testing shop and mark the checkbox complete. It may or may not return with feedback attached.

I got a different perspective of how CI should work, however, while working on a CI team. More than a little of this new perspective is probably due to working in a company that believes in and practices agile development and scrum management. With agile, we are much better at involving testers into day-to-day activities rather than the cold baton hand-off as sprint review time rolls near.

What got me thinking about a CI supply chain concept was the story from The Earth is Flat about how UPS has evolved from a package delivery company to an integral part of many companies day-to-day business operations. They started with a core business of picking up and shipping packages. But it turns out, there are inefficiencies to this bolted on approach. In order for the shipping to be efficient for both UPS and the contracting company, the entire supply chain had to be prepped in advance. This includes both pre and post shipment. The visibility into business processes had to be bidirectional.

I think the same is true with continuous integration. It is based on the old computer adage, GIGO, Garbage In Garbage Out. So a team that writes plenty of code, but no tests. What is the value of CI for them? The feedback is minimal. To get the most value out of an automated build system, there must be some forethought into what kind of quality feedback you want to see, and then what teams can do to define and integrate the right reporting tools into the build.

For example, even after automated tests written and incorporated into the build, you may want to get feedback on some quality measures on your code. In our case, we wanted Checkstyle and PMD reporting on our Java code. We use maven as our build tool, so adding the reporting into our builds was simple. But then the question becomes, what coding standards do we want to compare to? What PMD rulesets represent a sane minimum that teams can deal with and learn from at the same time?

So now, a team dedicated to providing CI services is recommending quality reports to both developers and stakeholders AND helping to define the rules that code should be compared against. My first take on this approach, deviating from a traditional CI definition was a sense of invasiveness. Upon further reflection, I have embraced the blurred lines between teams for several reasons. Firstly, it breaks artificial team boundaries and keeps communication lines open. CI is no longer a black box, it is a visible, value contributing component of system development. Secondly, developers won't (at least consistently across an organization) take the time to inject QA into their processes. A CI team, however, does have the time to take the first steps and get the ball rolling. The conversation is more constructive when you have something in place actually working then just a bunch of talking going on about what could be.

I think a simple diagram emphasizes the point. If "A" is a development team, "B" is a CI team, and "C" represents the stakeholders of the solution being developed, then the shaded areas pinpoint a missed opportunity unless you adopt the supply chain analogy. These are not hard control points, but juicy overlap, waiting to be optimized by as AB and BC working together.

So far, I've talked about the pre-shipment benefit leading into CI. I also now believe the supply chain post-CI is equally important. That is, where does the feedback go and how do you make is so easy, there is no reason not to use it. In our case, this encompassed two audiences with two different needs.

1) Developers. They need the low level test results, test coverage, Checkstyle and PMD reports.

2) Stakeholders (Project Managers, Solution Owners, Scrum Masters). They need to know that the developer teams are using the system. What is the source control commit frequency? How long do CI builds stay broken? How long are the automated builds taking? In short, the interest is around are they getting their monies worth on CI investment and knowing if the teams "get it". It provides them with just enough information to start asking the right questions to the right people.

So the supply chain from CI into management visibility, in our case, ended up being a enterprise level portal aggregating project CI metrics together into an at-a-glance view of how well their agile teams are performing. This could be stop light charts on current CI conditions or simple graphs of build times or number of tests as plotted over the last 30 days.

As always seem to be the case, the benefit of this proposed supply chain approach is inceased communication.

Again, this initially seemed to stray from traditional core CI functionality, but in reality it is simply providing the visibility into the process that scrum promises. Warts and all.

Hudson Status Google Gadget on Linux

Google has just announced gadget support for the linux version of Google Desktop. This is exciting news for me as that means the Hudson Status gadget is now cross-platform. For all you developers out there that rely on Hudson, this means you don't have to visit your Hudson site or check your email or your feed reader to get the latest job statuses.

The dots are right there on your desktop!

For you existing users out there, make sure you upgrade to the newest gadget version, which has some important bug fixes. More information about the latest release, v0.9.5, can be found on the Hudson wiki.

Maven SNAPSHOT Traceability

So your scrum team is creating non-unique SNAPSHOT versioned artifacts throughout your sprint. How do trace that SNAPSHOT version back to a baseline that is of any QA relevance (build number, subversion revision, datetime stamp)?

<plugin>
 <groupId>org.apache.maven.plugins</groupId>
 <artifactId>maven-antrun-plugin</artifactId>
 <executions>
   <execution>
     <phase>generate-resources</phase>
     <configuration>
       <tasks>
         <tstamp>
           <format property="now" pattern="MM/dd/yyyy hh:mm" unit="hour" />
         </tstamp>
         <property name="build.version" value="${version} (private-${now}-${user.name})" />
         <property name="hudson.build" value="hudson-${BUILD_NUMBER}, subversion-${SVN_REVISION}" />
                         
         <!-- put the version file -->
         <echo message="The build id is: ${build.version}" />
         <mkdir dir="target/${project.build.finalName}/" />
         <echo file="target/${project.build.finalName}/version.properties">version=${build.version} ${hudson.build}
         </echo>
       </tasks>
     </configuration>
     <goals>
       <goal>run</goal>
     </goals>
   </execution>
 </executions>
</plugin>

...

<profiles>
 <profile>
   <id>release</id>
   <properties>
       <!-- for releases, just use the POM version. -->
       <build.version>${version}</build.version>
       <hudson.build></hudson.build>
   </properties>
 </profile>
</profiles>

This little piece of magic creates a version.properties file that contains some valuable information.

version=1.1.0-SNAPSHOT (private-04/21/2008 02:23-jblack), hudson-453, subversion-1124

When you do a formal release, specify the -Prelease profile to have this file simply hold the pom version.

For the plugin configuration above, this file gets put in a war project root webapp directory, suitable for immediate viewing from your browser!

The mad props for this idea go to Kohsuke. :)

Hudson Gadget For Google Desktop

I took a first stab at creating a google desktop gadget that can display the job names and statuses of any Hudson instance. Simply configure it to the Hudson server url and it will discover all the jobs and start polling the for blue dots.



Hudson Gadget, Version 0.9.0.

Let me know if you have any improvement ideas or other issues.

[update] This gadget has been contributed to the open source Hudson project.

Code Reuse Inside Your Own Company

A short read over on Dr. Dobbs, Where Are the Clients In a SOA? made some good observations about the mindset of creating reusable SOA components, especially when the reuse is likely to occur in your own back yard.

"The best "enterprise architecture" plan will never be realized as long as developers assume the client of a piece of code is another piece of code."

"As software developers we are not very good customers of the work our colleagues do. We are quicker to search the Internet for a utility or example than we are to search our own internal repositories."

"More importantly, we aren't thinking about other developers when we create the code. We don't create interfaces that make it easy for people to use our code in ways we have not considered. We don't provide design-level abstractions that help other programmers and architects fit our solution into theirs. We don't market the code to the rest of the team or to other teams within the organization. We don't do any market research to make sure the nontraditional customers of our code are going to make the most of it. We don't package it in ways that also make it easy to use outside our intended use."

The emphasis is mine, although it should be yours, too.

Jersey... Jetty and Maven style!

I recently created a webapp that due to time constraints, I implemented with plain old servlets which I am comfortable with. In reality, the project was a perfect candidate to implement in RESTful style. I had been loosely following the Jersey project, part of the larger java.net Metro project, so I decided to give Jersey a try.

"Jersey is the open source ... JAX-RS (JSR 311) Reference Implementation for building RESTful Web services. But, it is also more than the Reference Implementation. Jersey provides additional APIs and extension points (SPIs) so that developers may extend Jersey to suite their needs."

My only requirements were really around a quick turn-around dev setup. So I wanted to use maven and the jetty plugin to preview the results quickly.

Jersey is still only available as an early access preview (0.5-ea) , so some of the examples I found were "out of date", as the team is still refactoring the implementation.

So, first step: create a maven war project structure and configure the web.xml to look this this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE web-app PUBLIC "-//Sun Microsystems, Inc.//DTD Web Application 2.3//EN" "http://java.sun.com/dtd/web-app_2_3.dtd">
<web-app>
<servlet>
  <servlet-name>Jersey</servlet-name>
  <servlet-class>com.sun.ws.rest.spi.container.servlet.ServletContainer</servlet-class>
  <init-param>
      <param-name>com.sun.ws.rest.config.property.resourceConfigClass</param-name>
      <param-value>com.sun.ws.rest.api.core.PackagesResourceConfig</param-value>
  </init-param>
  <init-param>
      <param-name>com.sun.ws.rest.config.property.packages</param-name>
      <param-value>org.naiade.tutorials.jersey;com.gestalt.gci.rest.resources</param-value>
  </init-param>
  <load-on-startup>1</load-on-startup>
</servlet>
<servlet-mapping>
  <servlet-name>Jersey</servlet-name>
  <url-pattern>/*</url-pattern>
</servlet-mapping>
</web-app>

The two key things here are:

• use the ServletContainer servlet (not ServletAdapter)
• use the pluggable PackagesResourceConfig parameter value. This allows Jersey to auto find all REST resources based on the package names. This is configured in the additional parameter value. Here we provide a couple of packages to look in, semicolon separated.
  

For the code, I used the ShoeService example I found here.
As I mentioned before, Jersey is still evolving so I ended up making two code changes to still be able to compile under 0.5-ea:

• @HttpMethod() changed to @GET or @POST
• @UriTemplate() changed to @Path()
  

The maven project looks like:

<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.shoe</groupId>
<artifactId>shoeservice</artifactId>
<version>1.0</version>
<name>Shoe Service REST</name>
<packaging>war</packaging>
<dependencies>
   <!-- jersery.dev.java.net -->
   <dependency>
       <groupId>jersey</groupId>
       <artifactId>jersey</artifactId>
       <version>0.5-ea</version>
   </dependency> 
</dependencies>
<build>
   <plugins>
       <plugin>
           <groupId>org.apache.maven.plugins</groupId>
           <artifactId>maven-compiler-plugin</artifactId>
           <version>2.0.2</version>
           <configuration>
               <source>1.5</source>
               <target>1.5</target>
           </configuration>
       </plugin>
       <plugin>
           <groupId>org.mortbay.jetty</groupId>
           <artifactId>maven-jetty-plugin</artifactId>
           <version>6.1.8</version>
           <configuration>
               <scanIntervalSeconds>10</scanIntervalSeconds>
           </configuration>
           <dependencies>
               <dependency>
                   <groupId>org.apache.geronimo.specs</groupId>
                   <artifactId>geronimo-j2ee_1.4_spec</artifactId>
                   <version>1.0</version>
                   <scope>provided</scope>
               </dependency>
           </dependencies>
       </plugin>
   </plugins>
</build>
</project>

To bring it all together, it's as simple as

mvn jetty:run

Open you browser and hit http://localhost:8080/shoeservice/shoe

Easy.

I really see Jersey taking off as it is a good project and easy to work with. If you are looking for a good Java solution to creating a RESTful accessible services, give it a try.

NetBeans 6.0 Maven Integration

As a proponent of continuous integration, I naturally don't endorse one IDE over another. The choice should be free and open for developers to choice what works best for them. The important thing is that the project should not be reliant on an IDE to build. That is, there must be an independent build tool that can function outside of any IDE, whether it be ant or maven or whatever.

I have used several IDEs over the last couple of year to include Eclipse, IDEA and NetBeans . They all have their strengths and weakness and for the projects I have been working on, I ended up switching around quite a bit. With the recent announcement of the new Netbeans 6.0, I wanted to revisit it's capabilities and see what was new, particularly with its maven integration.

While the Netbeans installer was downloading, I started to search for what plugins I would need to do maven->Netbeans or Netbeans ->maven project conversions. Most of what I found looked outdated and didn't look easy to use. With a little stumbling around I found out that the Mevenide project does now support Netbeans 6.0 (this is not apparent on their Netbeans project page). So I installed the Mevenide Maven2 plugin.

And was blown away!

There is no conversion of project types from maven->NetBeans . It just works natively. File > Open Project and browse for your existing maven project. Bingo.

I actually started by opening a parent project, <packaging>pom</packaging>, thinking that the Mavenide wouldn't know what to do with a multi module project, but it just worked, too! It showed me the project files, libraries, test libraries and the modules that this parent included. Too cool.



So then I double clicked my war module name hoping for the best and was not disappointed. It opened the war project in the project pane and I was off and running.



Right click on the project name to run the more common maven goals from right inside NetBeans. To run custom goals, you can create a Custom > Goals... goal that you can use to chain together whatever goals you want for a maven execution. This is the only thing I did that actually created a custom file on my filesystem. Everything else was native.

How's that for a slice of fried gold?

So, here's a big high five to to the Mevenide team on a job well done! (And update your NetBeans project page to highlight that it works in Netbeans 6.0. This is important news to advertise!).

Global Collaboraton In the Second Grade

I just finished reading The World is Flat by Thomas L. Friedman. He had some good insights and specific example about how the world is increasingly connected and how it functions as a global marketplace. The most interesting take away I had was that simply providing the cabling into countries that had not been previously connected was enough of a spark to allow individuals with curiosity and an innovative spirit to engage a global community.

Those of us in the software business, take this interconnectedness for granted now. I find it fascinating, being a father of two, that my children will know no different; the world was always connected. In the span of a single generation the world got very small indeed. I have friends who grew up their entire life in single small town. Any road trip of more than 100 miles was a big deal. But my kids won't have to "go" anywhere to instantly know what is going on on the other side of the planet or to actively collaborate with people on other continents.


So with this in mind, I had an idea for an education software project. My second-grader already has computer lab class once a week where he and his classmates learn and solve problems on a standalone system. What better time to learn about teamwork, on a global scale, then now, while he is in school.

My idea is a collaborative system that lets kids work together to solve and reinforce basic school (and life) skills. As math is a universal language, it would be a good first subject.

Picture this:

• The system is browser-based and has a easy to use, friendly RIA interface.
• One student from the USA joins the "Collabmath" course; One student from India joins the same course and they are paired together to solve 5 math problems.
• Each student will have a friendly avatar with their county's flag next to it. Each student's name can be clicked on to hear it pronounced in the native language. Google maps can show each student's school marked on a world map.
• The math problem will be presented in a large left hand workspace, and a chat window could be in a right-hand pane.
• The chat window can be used by the students to provide encouragement or ask for help. The chat would be I18N capable and feature emoticons that would make the session more personable.
  

• Each student takes turns solving a problem a step at a time. When each step is complete, control transfers to the other student, and so on. So, for a simple example, if the problem is to draw an octagon, the first student will draw the first line segment, then the second student will draw a complementary line segment connected to the first. When the octagon is complete, the students will have worked together to solve the problem.
• At the conclusion of each problem, a practical real world example of the use of that problem in each culture could be presented. In my example, the students would learn that in America, the octagon is the shape of a traffic stop sign; a related symbolic example could be presented that would be relevant in the country of India.

A wide range subjects and grade levels could be taught, of course, but for starters, this could get the ball rolling. I envision my kids being excited one day, coming home from school, "Dad! Guess what? I learned division and fractions with a kid from China today! Her name was Ming Wu! It was so cool!"

I can see it. Can you?