(autoconf.info)Writing Testsuites

Next: testsuite Invocation Prev: Using an Autotest Test Suite Up: Using Autotest

18.2 Writing `testsuite.at'

The `testsuite.at' is a Bourne shell script making use of special
Autotest M4 macros.  It often contains a call to `AT_INIT' near its
beginning followed by one call to `m4_include' per source file for
tests.  Each such included file, or the remainder of `testsuite.at' if
include files are not used, contain a sequence of test groups.  Each
test group begins with a call to `AT_SETUP', then an arbitrary number
of shell commands or calls to `AT_CHECK', and then completes with a
call to `AT_CLEANUP'.  Multiple test groups can be categorized by a
call to `AT_BANNER'.

 -- Macro: AT_INIT ([NAME])
     Initialize Autotest.  Giving a NAME to the test suite is
     encouraged if your package includes several test suites.  In any
     case, the test suite always displays the package name and version.
     It also inherits the package bug report address.

     State that, in addition to the Free Software Foundation's
     copyright on the Autotest macros, parts of your test suite are
     covered by COPYRIGHT-NOTICE.

     The COPYRIGHT-NOTICE shows up in both the head of `testsuite' and
     in `testsuite --version'.

     Log the file name and answer to `--version' of each program in
     space-separated list EXECUTABLES.  Several invocations register
     new executables, in other words, don't fear registering one program
     several times.

   Autotest test suites rely on `PATH' to find the tested program.
This avoids the need to generate absolute names of the various tools,
and makes it possible to test installed programs.  Therefore, knowing
which programs are being exercised is crucial to understanding problems
in the test suite itself, or its occasional misuses.  It is a good idea
to also subscribe foreign programs you depend upon, to avoid
incompatible diagnostics.

     This macro identifies the start of a category of related test
     groups.  When the resulting `testsuite' is invoked with more than
     one test group to run, its output will include a banner containing
     TEST-CATEGORY-NAME prior to any tests run from that category.  The
     banner should be no more than about 40 or 50 characters.  A blank
     banner will not print, effectively ending a category and letting
     subsequent test groups behave as though they are uncategorized
     when run in isolation.

     This macro starts a group of related tests, all to be executed in
     the same subshell.  It accepts a single argument, which holds a
     few words (no more than about 30 or 40 characters) quickly
     describing the purpose of the test group being started.
     TEST-GROUP-NAME must not contain unbalanced quotes or parentheses.

     Associate the space-separated list of KEYWORDS to the enclosing
     test group.  This makes it possible to run "slices" of the test
     suite.  For instance, if some of your test groups exercise some
     `foo' feature, then using `AT_KEYWORDS(foo)' lets you run
     `./testsuite -k foo' to run exclusively these test groups.  The
     TITLE of the test group is automatically recorded to `AT_KEYWORDS'.

     Several invocations within a test group accumulate new keywords.
     In other words, don't fear registering the same keyword several
     times in a test group.

     If the current test group fails, log the contents of FILE.
     Several identical calls within one test group have no additional

     Determine whether the test is expected to fail because it is a
     known bug (for unsupported features, you should skip the test).
     SHELL-CONDITION is a shell expression such as a `test' command;
     you can instantiate this macro many times from within the same
     test group, and one of the conditions is enough to turn the test
     into an expected failure.

 -- Macro: AT_CLEANUP
     End the current test group.

     Initialize an input data FILE with given CONTENTS.  Of course, the
     CONTENTS have to be properly quoted between square brackets to
     protect against included commas or spurious M4 expansion.  The
     contents must end with an end of line.  FILE must be a single
     shell word that expands into a single file name.

 -- Macro: AT_CHECK (COMMANDS, [STATUS = `0'], [STDOUT = `'], [STDERR =
          `'], [RUN-IF-FAIL], [RUN-IF-PASS])
     Execute a test by performing given shell COMMANDS.  These commands
     should normally exit with STATUS, while producing expected STDOUT
     and STDERR contents.  If COMMANDS exit with status 77, then the
     whole test group is skipped.  Otherwise, if this test fails, run
     shell commands RUN-IF-FAIL or, if this test passes, run shell
     commands RUN-IF-PASS.

     This macro must be invoked in between `AT_SETUP' and `AT_CLEANUP'.

     If STATUS, or STDOUT, or STDERR is `ignore', then the
     corresponding value is not checked.

     The special value `expout' for STDOUT means the expected output of
     the COMMANDS is the content of the file `expout'.  If STDOUT is
     `stdout', then the standard output of the COMMANDS is available
     for further tests in the file `stdout'.  Similarly for STDERR with
     `experr' and `stderr'.

automatically generated by info2www