NAME

test-manager/ - An automatic unit-testing framework for MIT Scheme or Guile


SYNOPSYS

  (load "test-manager/load.scm")
  (in-test-group
   simple-stuff
   (define-test (arithmetic)
     "Checking that set! and arithmetic work"
     (define foo 5)
     (assert-= 5 foo "Foo should start as five.")
     (set! foo 6)
     (assert-= 36 (* foo foo)))
   ; Each of these will become a separate anonymous one-form test
   (define-each-test
     (assert-= 4 (+ 2 2) "Two and two should make four.")
     (assert-= 6 (+ 2 2 2))
     (assert-= 2147483648 (+ 2147483647 1) "Addition shouldn't overflow.")
     (assert-equal '(1 2 3) (cons 1 '(2 3)))))
  (run-registered-tests)


DESCRIPTION

This test framework defines a language for specifying test suites and a simple set of commands for running them. A test suite is a collection of individual tests grouped into a hierarchy of test groups. The test group hierarchy serves to semantically aggregate the tests, allowing the definition of shared code for set up, tear down, and surround, and also partition the test namespace to avoid collisions.

The individual tests are ordinary procedures, with some associated bookkeeping. A test is considered to pass if it returns normally, and to fail if it raises some condition that it does not handle (tests escaping into continuations leads to unspecified behavior). The framework provides a library of assertions that can be invoked in tests and have the desired behavior of raising an appropriate condition if they fail.

Defining Test Suites

All tests are grouped into a hierarchy of test groups. At any point in the definition of a test suite, there is an implicit ``current test group'', into which tests and subgroups can be added. By default, the current test group is the top-level test group, which is the root of the test group hierarchy.

(define-test (name) expression ... )

Define a test named name that consists of the given expressions, and add it to the current test group. When the test is run, the expressions will be executed in order, just like the body of any procedure. If the test raises any condition that it does not handle, it is considered to have failed. If it returns normally, it is considered to have passed. Usually, tests will contain assertions from the list below, which raise appropriate conditions when they fail. In the spirit of Lisp docstrings, if the first expression in the test body is a literal string, that string will be included in the failure report if the test should fail.

This is the most verbose and most expressive test definition syntax. The three test definition syntaxes provided below are increasingly terse syntactic sugar for common usage patterns of this syntax.

(define-test () expression ... )

Define an explicitly anonymous test. I can't see why you would want to do this, but it is provided for completeness.

(define-test expression)

Define a one-expression anonymous test. The single expression will be printed in the failure report if the test fails. This is a special case of define-each-test, below.

(define-each-test expression ... )

Define a one-expression anonymous test for each of the given expressions. If any of the tests fail, the corresponding expression will be printed in that test's failure report.

If you have many simple independent assertions you need to make and you don't want to invent names for each individual one, this is the test definition syntax for you.

(in-test-group name expression ... )

Locate (or create) a test subgroup called name in the current test group. Then temporarily make this subgroup the current test group, and execute the expressions in the body of in-test-group. This groups any tests and further subgroups defined by those expressions into this test group. Test groups can nest arbitrarily deep. Test groups serve to disambiguate the names of tests, and to group them semantically. In particular, should a test fail, the names of the stack of groups it's in will be displayed along with the test name itself.

(define-set-up expression ...)

Defines a sequence of expressions to be run before every test in the current test group. Clobbers any previously defined set up for this group.

(define-tear-down expression ...)

Defines a sequence of expressions to be run after every test in the current test group. Clobbers any previously defined tear down for this group.

(define-surround expression ...)

Defines a sequence of expressions to be run surrounding every test in the current test group. Inside the define-surround, the identifier run-test is bound to a nullary procedure that actually runs the test. Clobbers any previously defined surround for this group.

(define-group-set-up expression ...)

Defines a sequence of expressions to be run once before running any test in the current test group. Clobbers any previously defined group set up for this group.

(define-group-tear-down expression ...)

Defines a sequence of expressions to be run once after running all tests in the current test group. Clobbers any previously defined group tear down for this group.

(define-group-surround expression ...)

Defines a sequence of expressions to be run once surrounding running the tests in the current test group. Inside the define-group-surround, the identifier run-test is bound to a nullary procedure that actually runs the tests in this group. Clobbers any previously defined group surround for this group.

Running Test Suites

The following procedures are provided for running test suites:

(run-test name-stack)

Looks up the test indicated by name-stack in the current test group, runs it, and prints a report of the results. Returns the number of tests that did not pass. An empty list for a name stack indicates the whole group, a singleton list indicates that immediate descendant, a two-element list indicates a descendant of a descendant, etc. For example, (run-test '(simple-stuff harder)) would run the second test defined in the example at the top of this page.

(run-registered-tests)

Runs all tests registered so far, and prints a report of the results. Returns the number of tests that did not pass. This could have been defined as (run-test '()).

Assertions

The following assertions are provided for writing tests. The message arguments to the assertions are user-specified messages to print to the output if the given assertion fails. The assert-proc assertion requires a message argument because it cannot construct a useful output without one, and because it is not really meant for extensive direct use. The message is optional for the other assertions because they can say something at least mildly informative even without a user-supplied message. In any case, the message can be either a string or a promise (as created by delay) to produce a string. The latter is useful for assertions with dynamically computed messages, because that computation will then only be performed if the test actually fails.

(assert-proc message proc)

Passes iff the given procedure, invoked with no arguments, returns a true value. On failure, arranges for the given message to appear in the failure report. This is a primitive assertion in whose terms other assertions are defined.

(assert-true thing [message])

Passes iff the given value is a true value (to wit, not #f).

(assert-false thing [message])

Passes iff the given value is a false value (to wit, #f).

(assert-equal expected actual [message]) Likewise assert-eqv, assert-eq, and assert-=

Passes iff the given actual value is equivalent, according to the corresponding predicate, to the expected value. Produces a reasonably helpful message on failure, and includes the optional message argument in it if present. When in doubt, use assert-equal to compare most things; use assert-= to compare exact numbers like integers; and use assert-in-delta, below, for inexact numbers like floating points.

assert-equals, assert=

Are aliases for assert-equal and assert-=, respectively.

(assert-equivalent predicate [pred-name])

This is intended as a tool for building custom assertions. Returns an assertion procedure that compares an expected and an actual value with the given predicate and produces a reasonable failure message. assert-equal and company could have been defined in terms of assert-equivalent as, for example, (define assert-equal (assert-equivalent equal? "equal?")).

(assert-matches regexp string [message])

Passes iff the given regular expression matches the given string.

(assert-no-match regexp string [message])

Passes iff the given regular expression does not match the given string.

(assert-in-delta expected actual delta [message])

Passes iff the given actual value differs, in absolute value, from the given expected value by no more than delta. Use this in preference to assert-= to check sameness of inexact numerical values.


BUGS

This unit testing framework is a work in progress. The assertion library is quite impoverished, the test groups do not support as much shared set up code among their tests as I would like, and the language for explicit test group handling is ill-specified and undocumented (peruse test-group.scm if interested). Suggestions are welcome.


AUTHOR

Alexey Radul, axch@mit.edu