Monday, May 10, 2010

Changes to .cabal File Format

This project will allow for a new type of section in Cabal's package description files: the aptly named Test section. The Test section will contain the field type indicating the test suite interface and the field test-is, the meaning of which will depend on the type of test suite. Test suite types will support versions to maintain backwards compatibility if improvements are made to the interface types.

Test Section (Shell Interface)

If the type field is set to shell-1 (indicating a test suite using the first version of the shell type), the test-is field should refer to a file that compiles into a valid executable. The rest of the section should contain any of the usual build info fields necessary. For example:

Test test-foo
type: shell-1
test-is: TestFoo.hs

Test Section (Library Interface)

If the type field is set to library-1, the test-is field should be a module that exports a symbol tests. The rest of the section should contain any of the necessary build info fields. For example:

Test test-bar
type: library-1
test-is: Tests.Bar

In the file "Tests/Bar.hs" we should find:
module Tests.Bar (tests) where
import Distribution.Testsuite (Test(..))

tests :: [Test]
tests = [ test1, test2, ... ]

Build Systems

For packages using the Simple build system, Cabal will automatically handle building test suite executables or libraries. For library test suites, Cabal will also build a stub executable allowing the user to selectively run tests with results reported to the standard output. (This is the default test agent.) To allow other test agents to use the test suite library, Cabal will install it to a local package database reserved for this purpose. In order to be compatible with this interface, packages with custom build systems must likewise handle test suite executables and libraries, including the installation of the test suite libraries to a local database.


  1. I don't like this shell-1 and library-1 idea. The idea of encoding a numeric version into a name is quite silly.

    Also, you are conflating "shell" with "executable". If you expect a program that can be compiled into an executable, then name it "executable". Otherwise, I'd expect a shell script.

  2. Bryan,

    Would it be better if we kept the interface version separate? Example:

    Test test-foo
    type: library
    interface-version: 1.0
    test-is: TestFoo.hs

    The reason we have an interface version is so that it's possible to compile a stub against the library and run the tests using that. I guess "executable" is clearer than "shell" (I stole the latter from the convention we use at work).

  3. For that matter, why not "legacy" instead of "executable"? We need some way to discourage people from using the executable interface for new test suites.

    As far as versioning: every test must have an explicitly listed interface version; we can't just default to the most recent version because we have no intention of maintaining backwards compatibility between different versions. I suggested encoding the version in the name to keep things concise. Obviously, if the consensus is that a separate field is better, I will go that direction; I just wanted to point out that it will be a required field for every test.

  4. I think I like "executable" better than "legacy" since there are legitimate reasons to have tests that is just a binary (e.g. end-to-end tests, load tests).