GSoC 2010 Week 1: Parsing, Building, and Running Testsuites

The first week of the Google Summer of Code 2010 has been a productive one, for me! I've been working on:

• parsing test stanzas from .cabal files,
• building executable-type testsuites, and
• running testsuites and collecting results.

On the first two points, I have patches ready and awaiting approval from Duncan Coutts, Cabal's maintainer; hopefully, they will be available in the repository soon.

As I have worked on the implementation, there has been one notable change to the design of the test stanza in the .cabal file: rather than indicate the testsuite interface version in its own field, it is now indicated in the same field as the interface type in the same way versions are indicated in the build dependencies. If the description I just gave is difficult to understand, then a picture may be worth a thousand words:

Test foo
    test-is: foo.hs
    type: executable == 1

A few notes about this interface:

• Version 1 is the only version of the executable interface at this time.
• No guarantee of compatibility is made between interface versions, so although this style would allow package authors to specify version ranges, they are encouraged to do so carefully. In fact, there is no reason to do so, as newer versions of Cabal will always support older testsuite interface versions.
• Because all versions of the interface will always be available and be mutually incompatible, there is no sane default version; therefore, the version must be specified. This differs from the build-depends field.

In week 2, I will be focusing on running testsuites and collecting their results. The standard output and standard error will be captured, along with the exit code, which will indicate the success or failure of the testsuite. Hopefully, I will have patches submitted and reviewed by the end of the week, and executable testsuite support will be (more or less) complete!