Hudson and Simpletest
We've recently set up a Hudson Continuous Integration build server to help us with testing our sites. We got selenium up and running fairly quickly, it installs and tests all the sites and Hudson has aplugin to report the results from selenium tests. But we also wanted to run simpletests and report those results too. This article explains how I've set it up on our server, and will hopefully give you a head start on the way to getting it set up on yours.
1. Run simpletests from the command line.
2. Integrate the results with the Hudson's interface.
#1 Is actually fairly easy to do, recent versions of Drupal's simpletest come with a backport of the run-tests.sh script that can run tests from the command line, however, getting the results from the script into a format that Hudson will understand is a bit more challenging. Since Hudson started out as a Java build tool, it can natively understand the JUnit test result format. Now, I had hoped that it would be really easy to ask the run-tests.sh script to output in a format other than it's default, stdout. It turns out that simpletest does have a plugable reporting mechanism, but it also turns out that relying on an external library was too good for Drupal, and the simpletest Drupal uses is a hack. So I decided to hack the run-tests.sh script to output the results in the format I wanted.
To cut a long story short, it's really hard to find a specification of the JUnit output file format, but after lots of trying, I had a script that could run simpletests and report the results in a JUnit friendly (Hudson friendly) format, yey!
Find the script attached, there's an extra argument from the normal set, --xml which allows you to specify the directory in which to output the xml result files. Lovely.
Finally you'll want to make a bash script that passes in all the right arguments to the script and then run that from Hudson, as part of the normal build process.
Edit: We lost the attachment in a server migration, but actually this in Drupal 7 core anyway!