GSoC 2010 Week 1: Parsing, Building, and Running Testsuites

The first week of the Google Summer of Code 2010 has been a productive one, for me! I've been working on:

• parsing test stanzas from .cabal files,
• building executable-type testsuites, and
• running testsuites and collecting results.

On the first two points, I have patches ready and awaiting approval from Duncan Coutts, Cabal's maintainer; hopefully, they will be available in the repository soon.

As I have worked on the implementation, there has been one notable change to the design of the test stanza in the .cabal file: rather than indicate the testsuite interface version in its own field, it is now indicated in the same field as the interface type in the same way versions are indicated in the build dependencies. If the description I just gave is difficult to understand, then a picture may be worth a thousand words:

Test foo
    test-is: foo.hs
    type: executable == 1

A few notes about this interface:

• Version 1 is the only version of the executable interface at this time.
• No guarantee of compatibility is made between interface versions, so although this style would allow package authors to specify version ranges, they are encouraged to do so carefully. In fact, there is no reason to do so, as newer versions of Cabal will always support older testsuite interface versions.
• Because all versions of the interface will always be available and be mutually incompatible, there is no sane default version; therefore, the version must be specified. This differs from the build-depends field.

In week 2, I will be focusing on running testsuites and collecting their results. The standard output and standard error will be captured, along with the exit code, which will indicate the success or failure of the testsuite. Hopefully, I will have patches submitted and reviewed by the end of the week, and executable testsuite support will be (more or less) complete!

A Prototype Test Suite Library Interface

This post is Literate Haskell.

> {-# LANGUAGE ExistentialQuantification #-}
> module Distribution.Testsuite where

This requires base >= 4 for extensible exceptions.

> import Control.Exception ( SomeException, Exception(toException) )
> import Control.Monad ( liftM )
> import Data.Dynamic ( Dynamic() )
> import Data.Function ( on )
> import Data.List ( unionBy )
> import Data.Monoid ( Monoid(..) )

The use of unsafePerformIO is an unfortunate consequence of the method used to catch exceptions below.

> import System.IO.Unsafe ( unsafePerformIO )

Tests are separated based on their purity. Although it appears that any test using the RNG, i.e. all tests using QuickCheck, must be impure, the use of the RNG is actually pure if the starting seed is specified in the Options. Therefore, only tests using other sorts of IO need be impure. Hopefully, with this purity information, test agents can make more informed decisions about which tests can and cannot be run in parallel.

> data Test
>     = forall p. PureTestable p => PureTest p
>     | forall i. ImpureTestable i => ImpureTest i
> 
> class TestOptions t => ImpureTestable t where
>     getResult :: t -> Options -> IO Result
> 
> class TestOptions t => PureTestable t where
>     result :: t -> Options -> Result
> 
> class TestOptions t where
>     name :: t -> Name
>     options :: t -> [String]
>     defaultOptions :: t -> IO Options

The defaultOptions are returned in IO because it may be necessary to generate a random seed for some types of tests.

> type Name = String
> 
> newtype Options = Options [(String, Dynamic)]
>
> data Result = Pass | Fail Reason | Error SomeException
>     deriving Show
> 
> type Reason = String

The instances of the PureTestable, ImpureTestable and TestOptions classes will be left to the test libraries, e.g. HUnit and QuickCheck. It is not our purpose to reinvent the features they already provide, but to specify a uniform interface between them and test agents.

We provide a sensible instance of Monoid to allow the combination of sets of Options with the default options.

> instance Monoid Options where
>     mempty = Options []
> 
>     mappend (Options a) (Options b) =
>         Options $ unionBy ((==) `on` fst) a b

Default options go on the right argument of mappend and get overwritten by Options to the left.

What remains are some helper functions for handling options and exceptions:

> wrapException :: IO Result -> IO Result
> wrapException go = catch go $ return . Error . toException
> 
> mergeOptions :: TestOptions t => t -> Options -> IO Options
> mergeOptions test opts = liftM (mappend opts) $ defaultOptions test
> 
> runImpureTest :: ImpureTestable t => t -> Options -> IO Result
> runImpureTest = (wrapException .) . getResult
> 
> runPureTest :: PureTestable t => t -> Options -> Result
> runPureTest test opts = unsafePerformIO $ wrapException $ return $ result test opts

The use of unsafePerformIO in runPureTest is actually safe, since the function we are catching exceptions from is pure.

Changes to Cabal Command-Line Interface

Configuring and Building

In the configure stage, Cabal will have to recognize the --{en,dis}able-tests options. With tests disabled, the Test sections can be ignored entirely, but with tests enabled, Cabal will need to generate stub executables and do dependency resolution for the Test sections and any stubs.

During the build stage, Cabal will build any test suite executables, test suite libraries, and stub executables. Test suite libraries and executables may depend on the library exported by the package (if any), and therefore must be built after the rest of the package's contents.

Running Tests

The test stage can be invoked by the command cabal test. If the name of a test suite is supplied, then only that test suite will be run. Otherwise, all test suites named in the package description file will be run. For shell-type test suites, Cabal will run the test suite executable, collecting any output and the exit code, and report these to the user, indicating the success or failure of the test suite. For library-type test suites, Cabal will run the stub executable linked to the test suite library which will run each test in the suite and report on the individual results.

Library-type test suites may also be invoked by external agents. In this case, the method of invocation will depend on the test agent. The test agent will be responsible for generating and compiling a stub executable. The behavior of this executable will also be agent-specific, allowing for functionality to be extended beyond what Cabal's basic test runner will support. The choice of test agent is left to the user and the use of one external test agent will not prevent the use of Cabal's basic test runner or of other external agents.

Changes to .cabal File Format

This project will allow for a new type of section in Cabal's package description files: the aptly named Test section. The Test section will contain the field type indicating the test suite interface and the field test-is, the meaning of which will depend on the type of test suite. Test suite types will support versions to maintain backwards compatibility if improvements are made to the interface types.

Test Section (Shell Interface)

If the type field is set to shell-1 (indicating a test suite using the first version of the shell type), the test-is field should refer to a file that compiles into a valid executable. The rest of the section should contain any of the usual build info fields necessary. For example:

Test test-foo
    type: shell-1
    test-is: TestFoo.hs
    ...

Test Section (Library Interface)

If the type field is set to library-1, the test-is field should be a module that exports a symbol tests. The rest of the section should contain any of the necessary build info fields. For example:

Test test-bar
    type: library-1
    test-is: Tests.Bar
    ...

In the file "Tests/Bar.hs" we should find:

module Tests.Bar (tests) where
import Distribution.Testsuite (Test(..))

tests :: [Test]
tests = [ test1, test2, ... ]

Build Systems

For packages using the Simple build system, Cabal will automatically handle building test suite executables or libraries. For library test suites, Cabal will also build a stub executable allowing the user to selectively run tests with results reported to the standard output. (This is the default test agent.) To allow other test agents to use the test suite library, Cabal will install it to a local package database reserved for this purpose. In order to be compatible with this interface, packages with custom build systems must likewise handle test suite executables and libraries, including the installation of the test suite libraries to a local database.

Introductions

Who am I?

I am Thomas Tuegel from Indianapolis, USA. I have just completed my B.S. in physics and mathematics at Butler University. In August, I will begin studying for my Ph.D. in physics at the University of Illinois at Urbana-Champaign. I have been accepted as a student in the 2010 Google Summer of Code to work on testing support in Cabal.

Overview

The immediate goal of this project is to implement testing support in Cabal so that users have access to useful information about package test suite results presented in a uniform format. Initially, this will make it easy for users to run test suites locally, but eventually Hackage will collect test suite results for uploaded packages. By making this information readily available to the public, we hope to incline more authors to provide useful, working test suites for their packages, improving the quality of software available on Hackage.

Cabal will support two test suite interfaces: a shell interface designed for easy integration with existing test suites and a library interface that will be the standard for integrating new test suites. For tests using the shell interface, Cabal will collect the exit code (indicating success or failure of the test suite) and standard output from the test suite executable. The shell interface will allow us to collect meaningful information from existing test suites without modification; existing test suites need only be designated as such in the .cabal file.

The library interface is really the interesting part of this project. A library test suite will export an entry point of a particular type which Cabal (or other test agents) can import into a stub executable. Then, the stub executable can access information about each test in the suite, run selected tests, and report the results. Cabal will specify a uniform interface for library test suites so that test agents can be written which are compatible with the many packages we hope will have test suites implementing the library interface in the future.

Cabal will be one test agent providing the capability to run selected tests from a library test suite and report the results on the standard output. To facilitate other test agents compiling stub executables linking to the test suite library, Cabal will install the library to a local package database reserved for this purpose.