The Marketplace Of Ideas Should Be Brutal

I think this wisdom applies to everything in life, beyond software development. You really don't know if your idea is superior until honest feedback is received and it is compared to competing ideas. I thought this was summarized nicely as:

"Ideas are meant to be attacked, torn apart, and put back together again. You may well want to shield your idea from the harsh sunlight at first, but by the time it’s ready to meet the world, it should also be ready for rain or shine. Bad ideas are supposed to wither under the stress of criticism."

-- Via David's post at 37Signals

Mutual PKI Authentication With a Java-based Application

I recently added certificate based authentication to an application I have been working on for awhile. It took longer to get done than I would have thought primarily because the number of moving pieces and most advice and guidance I found online was incomplete. I hope that documenting the complete setup I used will save someone else some time and frustration.

The requirements were:

• Wiring together two Java applications via SOAP, using CXF framework
• The producing application needed a valid server certificate
• The consuming application needed to present a valid client certificate to the server
• These certificates needed to be signed by a trusted root and share the same certificate authority
• For authentication purposes, no custom code within the application itself
• Use the Java keytool program to create keypairs and generate signing requests

The broad strokes:

1. Create the necessary certificates
2. Setup tomcat on the producing side to accept only https connections and require client authentication
3. Setup the consuming application to present client certs that can satisfy the secure authentication handshake

Creating and signing the certificates

I have previously created a CA setup using strictly openssl, but for this project, I needed to lean more heavily on keytool to originate the keypairs and openssl to do the certificate signing. The keytool program comes with the JDK. I used a windows based openssl binaries from SLP.

Setup a directory structure as recommended by openssl installation and set an environment variable, %SSLDIR%, to point to your CA directory.

Create CA and RA

# Create the CA's keypair and self-signed certificate
#   -x509 means create self-sign cert
#   -keyout means generate keypair
#   -nodes means do not encrypt private key.
#   -set_serial sets the serial number of the certificate
openssl req -verbose -x509 -new -nodes -set_serial 1234 -subj "/CN=codebeneathCA/OU=project/O=company/ST=MO/C=US" -days 7300 -out %SSLDIR%/certs/cacert.pem -keyout %SSLDIR%/private/caprivkey.pem -config %SSLDIR%/openssl.config

# Create the Root Authority's keypair and Certificate Request
#    without -x509, we generate an x509 cert request.
#   -keyout means generate keypair
#   -nodes means do not encrypt private key.
openssl req -verbose -new -nodes -subj "/CN=codebeneathRoot/OU=project/O=company/ST=MO/C=US" -days 7300 -out %SSLDIR%/certs/csrra.pem -keyout %SSLDIR%/private/raprivkey.pem -config %SSLDIR%/openssl.config

# Have the CN=codebeneathCA issue a certificate for the CN=codebeneathRoot
# We need -extfile exts -extenstions x509_extensions to make sure
# CN=TheRA can be a Certificate Authority.
openssl ca -batch -days 7300 -cert %SSLDIR%/certs/cacert.pem -keyfile %SSLDIR%/private/caprivkey.pem -in %SSLDIR%/certs/csrra.pem -out %SSLDIR%/certs/ra-ca-cert.pem -extfile exts -extensions x509_extensions -config %SSLDIR%/openssl.config

Create Server Cert

# Create keypairs and Cert Request for a certificate for server
# This procedure must be done in JKS, because we need to use a JKS keystore.
# The current version of CXF using PCKS12 will not work for a number of
# internal CXF reasons.
keytool -genkey -alias codebeneath-server -dname "CN=10.9.5.210,OU=codebeneath-server,O=company,ST=MO,C=US" -keystore C:/certs/server-keystore.jks -storetype jks -storepass password -keypass password -validity 365
keytool -certreq -alias codebeneath-server -keystore C:/certs/server-keystore.jks -storetype jks -storepass password -keypass password -file codebeneath-server-csr.pem

# Have the CN=codebeneathRoot issue a certificate for server via the Certificate Requests.
openssl ca -batch -days 7300 -cert %SSLDIR%/certs/ra-ca-cert.pem -keyfile %SSLDIR%/private/raprivkey.pem -in %SSLDIR%/requests/codebeneath-server-csr.pem -out %SSLDIR%/requests/codebeneath-server-ra.pem -config %SSLDIR%/openssl.config

# Rewrite the certificates in PEM only format. This allows us to concatenate them into chains.
openssl x509 -in %SSLDIR%/certs/cacert.pem -out %SSLDIR%/certs/cacert.pem -outform PEM
openssl x509 -in %SSLDIR%/certs/ra-ca-cert.pem -out %SSLDIR%/certs/ra-ca-cert.pem -outform PEM
openssl x509 -in %SSLDIR%/requests/codebeneath-server-ra.pem -out %SSLDIR%/requests/codebeneath-server-ra.pem -outform PEM

# Create a chain readable by CertificateFactory.getCertificates.
type %SSLDIR%\requests\codebeneath-server-ra.pem %SSLDIR%\certs\ra-ca-cert.pem %SSLDIR%\certs\cacert.pem > %SSLDIR%\requests\codebeneath-server.chain

# Replace the certificate in the server keystore with their respective full chains.
keytool -import -file codebeneath-server.chain -alias codebeneath-server -keystore C:/certs/server-keystore.jks -storetype jks -storepass password -keypass password -noprompt

# Create the server Truststore file containing the CA cert.
keytool -import -file cacert.pem -alias codebeneathCA -keystore C:/certs/server-truststore.jks -storepass password -noprompt

Create Client Cert

# Create keypairs and Cert Request for a certificate for client (codebeneath)
# This procedure must be done in JKS, because we need to use a JKS keystore.
# The current version of CXF using PCKS12 will not work for a number of
# internal CXF reasons.
keytool -genkey -alias codebeneath -dname "CN=10.9.5.105,OU=codebeneath-client,O=company,ST=MO,C=US" -keystore C:/certs/client-keystore.jks -storetype jks -storepass password -keypass password -validity 365
keytool -certreq -alias codebeneath -keystore C:/certs/client-keystore.jks -storetype jks -storepass password -keypass password -file codebeneath-csr.pem

# Have the CN=codebeneathRoot issue a certificate for client via the Certificate Requests.
openssl ca -batch -days 7300 -cert %SSLDIR%/certs/ra-ca-cert.pem -keyfile %SSLDIR%/private/raprivkey.pem -in %SSLDIR%/requests/codebeneath-csr.pem -out %SSLDIR%/requests/codebeneath-ra.pem -config %SSLDIR%/openssl.config

# Rewrite the certificates in PEM only format. This allows us to concatenate them into chains.
openssl x509 -in %SSLDIR%/requests/codebeneath-ra.pem -out %SSLDIR%/requests/codebeneath-ra.pem -outform PEM

# Create a chain readable by CertificateFactory.getCertificates.
type %SSLDIR%\requests\codebeneath-ra.pem %SSLDIR%\certs\ra-ca-cert.pem %SSLDIR%\certs\cacert.pem > %SSLDIR%\requests\codebeneath.chain

# Replace the certificate in the client keystore with their respective full chains.
keytool -import -file codebeneath.chain -alias codebeneath -keystore C:/certs/client-keystore.jks -storetype jks -storepass password -keypass password -noprompt

# Create the client Truststore file containing the CA cert and server cert.
keytool -import -file cacert.pem -alias codebeneathCA -keystore C:/certs/client-truststore.jks -storepass password -noprompt
keytool -import -file codebeneath-server-ra.pem -alias codebeneath-server -keystore C:/certs/client-truststore.jks -storepass password -noprompt

Create trust for the server to know the client

# Import the client cert into the server Truststore.
keytool -import -file codebeneath-ra.pem -alias codebeneath -keystore C:/certs/client-truststore.jks -storepass password -noprompt

Export client cert from keystore for browser import

The keytool program cannot export private keys in any format, so use portecle to do these steps so that we can verify the setup using a browser without involving the actual client application.

1. Start portecle: java -jar ./portecle.jar
2. File... Open keystore file... select the ./client-keystore.jks file. Type the password.
3. Right-client the "codebeneath" certificate name. Select Export...
4. Export Type: Private Key and Certificates. Export Format: PKCS #12. Click OK. Type the password.
5. For the PKCS #12 password, it must be non-blank value or else the browser will not import it correctly.
6. In the browser, import the certificate as a "Personal" certificate.
7. In the browser, import the CA certificate as a trusted authoritative certificate.

Double check the results so far...

On the server machine:

keytool -list -keystore ./server-keystore.jks

• one signed certificate chain, PrivateKeyEntry, alias="codebeneath-server"

keytool -list -keystore ./server-truststore.jks

• CA certificate, trustedCertEntry, alias="codebeneathca",
• client certificate,trustedCertEntry, alias="codebeneath"

On the client machine:

keytool -list -keystore ./client-keystore.jks

• one signed certificate chain, PrivateKeyEntry, alias="codebeneath"

keytool -list -keystore ./client-truststore.jks

• CA certificate, trustedCertEntry, alias="codebeneathca",
• server certificate, trustedCertEntry, alias="codebeneath-server"

Server setup using tomcat

To accomplish the required mutual authentication (server is trusted https:// and client presents a valid certificate) requires setup within tomcat and the server war web.xml file. There is no code involved, it is strictly the setup of the servlet container to handle the security handshaking.

Tomcat server.xml file

Note that the clientAuth attribute must be "true" for things to work properly, even though the specification says that each war may individually be setup to require/not require client auth.

...
<Connector port=\"8443\" protocol=\"HTTP/1.1\" SSLEnabled=\"true\"
        maxThreads=\"150\" scheme=\"https\" secure=\"true\"
        clientAuth=\"true\" sslProtocol=\"TLS\"
        keystoreFile=\"C:/certs/server-keystore.jks\"
        keystorePass=\"password\"
        keyAlias=\"codebeneath-server\"
        truststoreFile=\"C:/certs/server-truststore.jks\"
        truststorePass=\"password\"
        />
...

Tomcat tomcat-users.xml file

Each username must match a client certificate subject line. Use portecle to open the client cert and cut-and-paste the "Subject:" line to this file's username value. So, even though the client certificate is valid given the truststore requirements, each client must also be specifically listed in this file to have access to the server application.

...
<role rolename=\"codebeneathRole\">
<user username=\"CN=10.9.5.43, OU=project, O=company, ST=MO, C=US\" password=\"password\" roles=\"codebeneathRole\">
...

Server war web.xml file

Both "role-name" elements must match the tomcat-users.xml defined rolename. The "auth-method" value of "CLIENT-CERT" will require a client certificate for authentication. The "transport-guarentee" value of "CONFIDENTIAL" will redirect all requests
...

<login-config>
<auth-method>CLIENT-CERT</auth-method>
</login-config>
<security-role>
<role-name>codebeneathRole</role-name>
</security-role>
<security-constraint>
<display-name>CodeBeneath</display-name>
<web-resource-collection>
 <web-resource-name>CodeBeneath</web-resource-name>
 <description></description>
 <url-pattern>/*</url-pattern>
 <http-method>GET</http-method>
 <http-method>POST</http-method>
 <http-method>HEAD</http-method>
 <http-method>PUT</http-method>
 <http-method>OPTIONS</http-method>
 <http-method>TRACE</http-method>
 <http-method>DELETE</http-method>
</web-resource-collection>
<auth-constraint>
 <description>
 <role-name>codebeneathRole</role-name>
</auth-constraint>
<user-data-constraint>
 <description>
 <transport-guarantee>CONFIDENTIAL</transport-guarantee>
</user-data-constraint>
</security-constraint>
...

Debug SSL

In the event that the mutual authentication fails, we can enable specific debugging on the server side. In the catalina.bat file, add the switch -Djavax.net.debug=ssl,handshake

Client Setup

Since the project used SOAP WS-*, including WS-Security, on the client side we used the CXF project to handle the wsdl-to-java code generation and to create the client port. My project uses CXF version 2.2.11, with the additional "cxf-rt-ws-security" dependency jar.

Client java code

package com.codebeneath;

import org.apache.cxf.endpoint.Client;
import org.apache.cxf.frontend.ClientProxy;
import org.apache.cxf.jaxws.JaxWsProxyFactoryBean;
//import org.apache.cxf.interceptor.LoggingInInterceptor;
import org.apache.cxf.interceptor.LoggingOutInterceptor;
import org.apache.cxf.transport.http.HTTPConduit;
import org.apache.cxf.configuration.jsse.TLSClientParameters;
import org.apache.cxf.binding.soap.saaj.SAAJOutInterceptor;
import org.apache.cxf.ws.security.wss4j.WSS4JOutInterceptor;
import org.apache.ws.security.handler.WSHandlerConstants;

import java.net.URL;
import java.security.GeneralSecurityException;
import java.security.KeyStore;
import java.security.KeyStoreException;
import java.security.NoSuchAlgorithmException;
import java.security.UnrecoverableKeyException;
import java.security.cert.CertificateException;
import java.util.HashMap;
import java.util.Map;
import java.util.logging.Logger;
import javax.net.ssl.KeyManager;
import javax.net.ssl.KeyManagerFactory;
import javax.net.ssl.TrustManager;
import javax.net.ssl.TrustManagerFactory;
import javax.xml.ws.BindingProvider;

/**
* Client a SOAP WS client for the service.  Handles logging support, WS-Security and PKI certificates
*/
public final class Client {

private static final String DEFAULT_PASSWORD = "password";
private static final String KEYSTORE_FILENAME = "/client-keystore.jks";
private static final String TRUSTSTORE_FILENAME = "/client-truststore.jks";

private Client() {
}

/**
 * Create a new WS client
 * @param serviceUrl, the service URL endpoint
 * @param sigPropFile, the properties file for client PKI.  Must be on the classpath at runtime
 * @return client port
 */
public static TroubleTicketServiceSoap createClient(URL serviceUrl, String sigPropFile, String keystoreDir) {

 JaxWsProxyFactoryBean factory = new JaxWsProxyFactoryBean();
 factory.setAddress(serviceUrl.toString());
 factory.setServiceClass(TroubleTicketServiceSoap.class);

 // request and response traces in the server.log
 addLogging(factory);

 // SOAP WS-Security header
 addSecurityHeaders(factory, sigPropFile);

 // create the client
 TroubleTicketServiceSoap port = (TroubleTicketServiceSoap) factory.create();
 ((BindingProvider) port).getRequestContext().put("schema-validation-enabled", Boolean.TRUE.toString());

 // handle trust of server certificates
 addCertificateHandling(port, serviceUrl, keystoreDir);

 return port;
}

private static void addLogging(JaxWsProxyFactoryBean factory) {
 factory.getOutInterceptors().add(new LoggingOutInterceptor());  // SOAP Request
//  factory.getInInterceptors().add(new LoggingInInterceptor());    // SOAP Response
}

private static void addSecurityHeaders(JaxWsProxyFactoryBean factory, String sigPropFile) {

 // setup the properties for WS-Security.  This is PKI signed.  The SIG_PROP_FILE must be on the classpath
 Map wss4jOutProps = new HashMap();
 wss4jOutProps.put(WSHandlerConstants.ACTION, WSHandlerConstants.SIGNATURE);
 wss4jOutProps.put(WSHandlerConstants.USER, "client");
 wss4jOutProps.put(WSHandlerConstants.PW_CALLBACK_CLASS, ClientPasswordCallback.class.getName());
 wss4jOutProps.put(WSHandlerConstants.SIG_PROP_FILE, sigPropFile);
 wss4jOutProps.put(WSHandlerConstants.SIG_KEY_ID, "DirectReference");

 // add required security interceptors
 factory.getOutInterceptors().add(new SAAJOutInterceptor());
 factory.getOutInterceptors().add(new WSS4JOutInterceptor(wss4jOutProps));
}

private static void addCertificateHandling(TroubleTicketServiceSoap port, URL serviceUrl, String keystoreDir) {
 Client proxy = ClientProxy.getClient(port);
 HTTPConduit conduit = (HTTPConduit) proxy.getConduit();
 TLSClientParameters tcp = new TLSClientParameters();
 tcp.setDisableCNCheck(true);

 try {
  KeyStore trustStore = KeyStore.getInstance("JKS");
  KeyStore keyStore = KeyStore.getInstance("JKS");

  // accept all self-signed server certs
//   tcp.setTrustManagers(CertificateAcceptorCXF.acceptAllCerts());

  File truststoreFile = new File(keystoreDir + TRUSTSTORE_FILENAME);
  trustStore.load(new FileInputStream(truststoreFile), DEFAULT_PASSWORD.toCharArray());
  TrustManagerFactory trustFactory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
  trustFactory.init(trustStore);
  TrustManager[] tm = trustFactory.getTrustManagers();
  tcp.setTrustManagers(tm);

  // enable client certs for authentication
  File keystoreFile = new File(keystoreDir + KEYSTORE_FILENAME);
  keyStore.load(new FileInputStream(keystoreFile), DEFAULT_PASSWORD.toCharArray());
  KeyManagerFactory keyFactory = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
  keyFactory.init(keyStore, DEFAULT_PASSWORD.toCharArray());
  KeyManager[] km = keyFactory.getKeyManagers();
  tcp.setKeyManagers(km);

 } catch (KeyStoreException kse) {
  System.out.println("Security configuration failed with the following: " + kse.getCause());
 } catch (NoSuchAlgorithmException nsa) {
  System.out.println("Security configuration failed with the following: " + nsa.getCause());
 } catch (FileNotFoundException fnfe) {
  System.out.println("Security configuration failed with the following: " + fnfe.getCause());
 } catch (UnrecoverableKeyException uke) {
  System.out.println("Security configuration failed with the following: " + uke.getCause());
 } catch (CertificateException ce) {
  System.out.println("Security configuration failed with the following: " + ce.getCause());
 } catch (GeneralSecurityException gse) {
  System.out.println("Security configuration failed with the following: " + gse.getCause());
 } catch (IOException ioe) {
  System.out.println("Security configuration failed with the following: " + ioe.getCause());
 }

 if (serviceUrl.getProtocol().trim().equalsIgnoreCase("https")) {
  conduit.setTlsClientParameters(tcp);
  LOG.info("GSTv3 Web Service protocol is 'https' so " + "set the TLSClientParameters on the "
    + "HTTPConduit. This should cause the" + " request to accept all certificates "
    + "and disable the CN Check.");
 } else {
  LOG.info("GSTv3 Web Service url did was not 'https' so did NOT " + "set the TLSClientParameters "
    + "on the HTTPConduit.");
 }
}
}

Client client-sign.properties

# Do not change these two properties
#
org.apache.ws.security.crypto.provider=org.apache.ws.security.components.crypto.Merlin
org.apache.ws.security.crypto.merlin.keystore.type=jks
# Set your keystore location and password here!
#
org.apache.ws.security.crypto.merlin.file=c:/certs/client-keystore.jks
org.apache.ws.security.crypto.merlin.keystore.password=password
org.apache.ws.security.crypto.merlin.keystore.alias=codebebeath

Conclusion

I hope that this complete picture of mutual PKI certificate authentication helps. If anything is not clear, let me know and I will try to clarify.

"Perfectionism is a crime against humanity"

I wish I had written this...

You could say that perfectionism is a crime against humanity. Adaptability is the characteristic that enables the species to survive-and if there’s one thing perfectionism does, it rigidifies behavior. It constricts people just when the fast-moving world requires more flexibility and comfort with ambiguity than ever. It turns people into success slaves.

There are several people I work with or deal with in my life that don't realize that the return on investment for their efforts is detrimental to success of the project. It is an ongoing battle and the battle-cry is "Moving on...".

Read the rest of the source article here: Is perfectionism killing your career?

"There’s no success quite like failure"

From Accept Defeat: The Neuroscience of Screwing Up:

"This is why other people are so helpful: They shock us out of our cognitive box. “I saw this happen all the time,” Dunbar says. “A scientist would be trying to describe their approach, and they’d be getting a little defensive, and then they’d get this quizzical look on their face. It was like they’d finally understood what was important.”

What turned out to be so important, of course, was the unexpected result, the experimental error that felt like a failure. The answer had been there all along — it was just obscured by the imperfect theory, rendered invisible by our small-minded brain. It’s not until we talk to a colleague or translate our idea into an analogy that we glimpse the meaning in our mistake. Bob Dylan, in other words, was right: There’s no success quite like failure."

Grail Projects in Hudson, Pt 2.

I just recently concluded a first sprints worth of work using Grails for the first time. We delivered.

While cutting the release, I found one additional gotcha in our CI environment. The grails application.properties file holds several key pieces of information including the version number, grails plugin versions and the last build date. This means the file must be checked into source control, but is also modified locally every build by CI (Hudson, nudge, nudge, wink wink). This caused our build to fail while bumping our project version.

1. CI will fail a grails+maven project if you change the version number of the project in the application.properties file.
   • Workaround: 1) Have your CI project perform a clean checkout instead of an update for you build. 2) At the end of your build, add a custom hook to delete the locally modified application.properties file.
   • Fix: I think have the build timestamp in this file may be a mistake as Grails grows more mature. As an improvement, it should be removed.
     

Grail Projects in Hudson

Here is a quick reference for adding a Grails project to Hudson. I'm assuming the webtest plugin is installed becuase there is no reason not to use it. It rocks. Also note that for integration into my existing maven build, I'm using the built-in Grails v1.1.1 maven integration. The project structure was created via the maven archetype:

mvn org.apache.maven.plugins:maven-archetype-plugin:2.0-alpha-4:generate \
-DarchetypeGroupId=org.grails \
-DarchetypeArtifactId=grails-maven-archetype \
-DarchetypeVersion=1.1 \
-DgroupId=com.black -DartifactId=grails-project

With that, here is what works so far:

General sequence (these are also the steps for keeping a development team in synch):

1. svn up

2. grails upgrade --non-interactive

3. mvn clean install -DnonInteractive=true

Notes:

• Step 2 above: This upgrades the grails project to include all core grails plugins and also creates some important files under web-app/WEB-INF (applicationContext.xml, ...). We would rather run mvn grails:exec -Dcommand=upgrade -DnonInteractive, however, this fails with an error: Embedded error: java.lang.reflect.InvocationTargetException
/home/black/dev/branches/parent/grails-project/null/src/war not found. Anybody have an idea on this?
• Step 3 above: This step will upgrade all non-core plugins added/modified/removed since the last checkout. Since grails will ask the "are you sure" question, make sure in CI to add the non-interactive switch as above, which will have the effect of cheerfully answering "Y" to all of grails questions.
  

Problems you may run into:

1. As part of a big maven multiproject build, grails may consume enough memory to produce an out of memory error.
   • Fix: Set MAVEN_OPTS=-Xmx512m -XX:MaxPermSize=150m
   
   


2. Adding new grails plugins to the build will freeze the CI build due to the "are you sure" prompt. The same will happen when removing a grails plugin.
   • Fix: Add the non-interactive switch to the build (mvn -DnonInteractive=true or grails --non-interactive)
   
   


3. Webtest launches a window during the test run that needs a display, so CI may need to run the webtests headless.
   • Fix: Add non-interactive switch to the build (mvn -DnonInteractive=true or grails --non-interactive)
   
   


4. Webtest starts the jetty server on port 8080 by default which may clash with other services on your CI box using that port.
   • Fix: Use the switch mvn -Dserver.port=8183. Note: The webtest.properties file property wt.config.port seems to be used if the code under test were already hosted on another running server besides the provided jetty.
   
   


5. A maven multiproject build that includes a grails project with webtests will look for /test/webtest/conf/webtest.properties in the wrong directory. That is, it will not look in ${basedir}, but always in the root project where you invoked maven to start with.
   
   • Workaround: Create a hudson job that only builds the grails war/webtest project. Create a downstream dependency on your main CI hudson job to trigger this one to build.
   
   
   • Fix: ?? Anyone have an answer for this one?
   
   

I hope this helps. For the few items above that have lingering questions, I would be interested in hearing if you have solved these. Let me know.

"Shipping is a feature."

Another good variation on Perfect is the Enemy of Very Good:

"A 50%-good solution that people actually have solves more problems and survives longer than a 99% solution that nobody has because it’s in your lab where you’re endlessly polishing the damn thing. Shipping is a feature. A really important feature. Your product must have it."
-- http://www.joelonsoftware.com/items/2009/09/23.html