320 lines
11 KiB
Groff
320 lines
11 KiB
Groff
|
.\"
|
||
|
|
.\" Automated Testing Framework (atf)
|
||
|
|
.\"
|
||
|
|
.\" Copyright (c) 2007 The NetBSD Foundation, Inc.
|
||
|
|
.\" All rights reserved.
|
||
|
|
.\"
|
||
|
|
.\" Redistribution and use in source and binary forms, with or without
|
||
|
|
.\" modification, are permitted provided that the following conditions
|
||
|
|
.\" are met:
|
||
|
|
.\" 1. Redistributions of source code must retain the above copyright
|
||
|
|
.\" notice, this list of conditions and the following disclaimer.
|
||
|
|
.\" 2. Redistributions in binary form must reproduce the above copyright
|
||
|
|
.\" notice, this list of conditions and the following disclaimer in the
|
||
|
|
.\" documentation and/or other materials provided with the distribution.
|
||
|
|
.\"
|
||
|
|
.\" THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND
|
||
|
|
.\" CONTRIBUTORS ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES,
|
||
|
|
.\" INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
|
||
|
|
.\" MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
|
||
|
|
.\" IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS BE LIABLE FOR ANY
|
||
|
|
.\" DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
||
|
|
.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE
|
||
|
|
.\" GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
|
||
|
|
.\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
|
||
|
|
.\" IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
|
||
|
|
.\" OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN
|
||
|
|
.\" IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
||
|
|
.\"
|
||
|
|
.Dd January 13, 2011
|
||
|
|
.Dt ATF-TEST-CASE 4
|
||
|
|
.Os
|
||
|
|
.Sh NAME
|
||
|
|
.Nm atf-test-case
|
||
|
|
.Nd generic description of test cases
|
||
|
|
.Sh DESCRIPTION
|
||
|
|
A
|
||
|
|
.Em test case
|
||
|
|
is a piece of code that stress-tests a specific feature of the software.
|
||
|
|
This feature is typically self-contained enough, either in the amount of
|
||
|
|
code that implements it or in the general idea that describes it, to
|
||
|
|
warrant its independent testing.
|
||
|
|
Given this, test cases are very fine-grained, but they attempt to group
|
||
|
|
similar smaller tests which are semantically related.
|
||
|
|
.Pp
|
||
|
|
A test case is defined by three components regardless of the language it is
|
||
|
|
implemented in: a header, a body and a cleanup routine.
|
||
|
|
The
|
||
|
|
.Em header
|
||
|
|
is, basically, a declarative piece of code that defines several
|
||
|
|
properties to describe what the test case does and how it behaves.
|
||
|
|
In other words: it defines the test case's
|
||
|
|
.Em meta-data ,
|
||
|
|
further described in the
|
||
|
|
.Sx Meta-data
|
||
|
|
section.
|
||
|
|
The
|
||
|
|
.Em body
|
||
|
|
is the test case itself.
|
||
|
|
It executes all actions needed to reproduce the test, and checks for
|
||
|
|
failures.
|
||
|
|
This body is only executed if the abstract conditions specified by the
|
||
|
|
header are met.
|
||
|
|
The
|
||
|
|
.Em cleanup
|
||
|
|
routine is a piece of code always executed after the body, regardless of
|
||
|
|
the exit status of the test case.
|
||
|
|
It can be used to undo side-effects of the test case.
|
||
|
|
Note that almost all side-effects of a test case are automatically cleaned
|
||
|
|
up by the library; this is explained in more detail in the rest of this
|
||
|
|
document.
|
||
|
|
.Pp
|
||
|
|
It is extremely important to keep the separation between a test case's
|
||
|
|
header and body well-defined, because the header is
|
||
|
|
.Em always
|
||
|
|
parsed, whereas the body is only executed when the conditions defined in
|
||
|
|
the header are met and when the user specifies that test case.
|
||
|
|
.Pp
|
||
|
|
At last, test cases are always contained into test programs.
|
||
|
|
The test programs act as a front-end to them, providing a consistent
|
||
|
|
interface to the user and several APIs to ease their implementation.
|
||
|
|
.Ss Results
|
||
|
|
Upon termination, a test case reports a status and, optionally, a textual
|
||
|
|
reason describing why the test reported such status.
|
||
|
|
The caller must ensure that the test case really performed the task that its
|
||
|
|
status describes, as the test program may be bogus and therefore providing a
|
||
|
|
misleading result (e.g. providing a result that indicates success but the
|
||
|
|
error code of the program says otherwise).
|
||
|
|
.Pp
|
||
|
|
The possible exit status of a test case are one of the following:
|
||
|
|
.Bl -tag -width expectedXfailureXX
|
||
|
|
.It expected_death
|
||
|
|
The test case expects to terminate abruptly.
|
||
|
|
.It expected_exit
|
||
|
|
The test case expects to exit cleanly.
|
||
|
|
.It expected_failure
|
||
|
|
The test case expects to exit with a controller fatal/non-fatal failure.
|
||
|
|
If this happens, the test program exits with a success error code.
|
||
|
|
.It expected_signal
|
||
|
|
The test case expects to receive a signal that makes it terminate.
|
||
|
|
.It expected_timeout
|
||
|
|
The test case expects to execute for longer than its timeout.
|
||
|
|
.It passed
|
||
|
|
The test case was executed successfully.
|
||
|
|
The test program exits with a success error code.
|
||
|
|
.It skipped
|
||
|
|
The test case could not be executed because some preconditions were not
|
||
|
|
met.
|
||
|
|
This is not a failure because it can typically be resolved by adjusting
|
||
|
|
the system to meet the necessary conditions.
|
||
|
|
This is always accompanied by a
|
||
|
|
.Em reason ,
|
||
|
|
a message describing why the test was skipped.
|
||
|
|
The test program exits with a success error code.
|
||
|
|
.It failed
|
||
|
|
An error appeared during the execution of the test case.
|
||
|
|
This is always accompanied by a
|
||
|
|
.Em reason ,
|
||
|
|
a message describing why the test failed.
|
||
|
|
The test program exits with a failure error code.
|
||
|
|
.El
|
||
|
|
.Pp
|
||
|
|
The usefulness of the
|
||
|
|
.Sq expected_*
|
||
|
|
results comes when writing test cases that verify known failures caused,
|
||
|
|
in general, due to programming errors (aka bugs).
|
||
|
|
Whenever the faulty condition that the expectation is trying to convery is
|
||
|
|
fixed, then the test case will be reported as
|
||
|
|
.Sq failed
|
||
|
|
and the developer will have to adjust it to match its new condition.
|
||
|
|
.Pp
|
||
|
|
It is important to note that all
|
||
|
|
.Sq expected_*
|
||
|
|
results are only provided as a
|
||
|
|
.Em hint
|
||
|
|
to the caller; the caller must verify that the test case did actually terminate
|
||
|
|
as the expected condition says.
|
||
|
|
.Ss Input/output
|
||
|
|
Test cases are free to print whatever they want to their
|
||
|
|
.Xr stdout 4
|
||
|
|
and
|
||
|
|
.Xr stderr 4
|
||
|
|
file descriptors.
|
||
|
|
They are, in fact, encouraged to print status information as they execute
|
||
|
|
to keep the user informed of their actions.
|
||
|
|
This is specially important for long test cases.
|
||
|
|
.Pp
|
||
|
|
Test cases will log their results to an auxiliary file, which is then
|
||
|
|
collected by the test program they are contained in.
|
||
|
|
The developer need not care about this as long as he uses the correct
|
||
|
|
APIs to implement the test cases.
|
||
|
|
.Pp
|
||
|
|
The standard input of the test cases is unconditionally connected to
|
||
|
|
.Sq /dev/zero .
|
||
|
|
.Ss Meta-data
|
||
|
|
The following list describes all meta-data properties interpreted
|
||
|
|
internally by ATF.
|
||
|
|
You are free to define new properties in your test cases and use them as
|
||
|
|
you wish, but non-standard properties must be prefixed by
|
||
|
|
.Sq X- .
|
||
|
|
.Bl -tag -width requireXmachineXX
|
||
|
|
.It descr
|
||
|
|
Type: textual.
|
||
|
|
Required.
|
||
|
|
.Pp
|
||
|
|
A brief textual description of the test case's purpose.
|
||
|
|
Will be shown to the user in reports.
|
||
|
|
Also good for documentation purposes.
|
||
|
|
.It has.cleanup
|
||
|
|
Type: boolean.
|
||
|
|
Optional.
|
||
|
|
.Pp
|
||
|
|
If set to true, specifies that the test case has a cleanup routine that has
|
||
|
|
to be executed by
|
||
|
|
.Xr atf-run 1
|
||
|
|
during the cleanup phase of the execution.
|
||
|
|
This property is automatically set by the framework when defining a test case
|
||
|
|
with a cleanup routine, so it should never be set by hand.
|
||
|
|
.It ident
|
||
|
|
Type: textual.
|
||
|
|
Required.
|
||
|
|
.Pp
|
||
|
|
The test case's identifier.
|
||
|
|
Must be unique inside the test program and should be short but descriptive.
|
||
|
|
.It require.arch
|
||
|
|
Type: textual.
|
||
|
|
Optional.
|
||
|
|
.Pp
|
||
|
|
A whitespace separated list of architectures that the test case can be run
|
||
|
|
under without causing errors due to an architecture mismatch.
|
||
|
|
.It require.config
|
||
|
|
Type: textual.
|
||
|
|
Optional.
|
||
|
|
.Pp
|
||
|
|
A whitespace separated list of configuration variables that must be defined
|
||
|
|
to execute the test case.
|
||
|
|
If any of the required variables is not defined, the test case is
|
||
|
|
.Em skipped .
|
||
|
|
.It require.files
|
||
|
|
Type: textual.
|
||
|
|
Optional.
|
||
|
|
.Pp
|
||
|
|
A whitespace separated list of files that must be present to execute the
|
||
|
|
test case.
|
||
|
|
The names of these files must be absolute paths.
|
||
|
|
If any of the required files is not found, the test case is
|
||
|
|
.Em skipped .
|
||
|
|
.It require.machine
|
||
|
|
Type: textual.
|
||
|
|
Optional.
|
||
|
|
.Pp
|
||
|
|
A whitespace separated list of machine types that the test case can be run
|
||
|
|
under without causing errors due to a machine type mismatch.
|
||
|
|
.It require.memory
|
||
|
|
Type: integer.
|
||
|
|
Optional.
|
||
|
|
Specifies the minimum amount of physical memory needed by the test.
|
||
|
|
The value can have a size suffix such as
|
||
|
|
.Sq K ,
|
||
|
|
.Sq M ,
|
||
|
|
.Sq G
|
||
|
|
or
|
||
|
|
.Sq T
|
||
|
|
to make the amount of bytes easier to type and read.
|
||
|
|
.It require.progs
|
||
|
|
Type: textual.
|
||
|
|
Optional.
|
||
|
|
.Pp
|
||
|
|
A whitespace separated list of programs that must be present to execute
|
||
|
|
the test case.
|
||
|
|
These can be given as plain names, in which case they are looked in the
|
||
|
|
user's
|
||
|
|
.Ev PATH ,
|
||
|
|
or as absolute paths.
|
||
|
|
If any of the required programs is not found, the test case is
|
||
|
|
.Em skipped .
|
||
|
|
.It require.user
|
||
|
|
Type: textual.
|
||
|
|
Optional.
|
||
|
|
.Pp
|
||
|
|
The required privileges to execute the test case.
|
||
|
|
Can be one of
|
||
|
|
.Sq root
|
||
|
|
or
|
||
|
|
.Sq unprivileged .
|
||
|
|
.Pp
|
||
|
|
If the test case is running as a regular user and this property is
|
||
|
|
.Sq root ,
|
||
|
|
the test case is
|
||
|
|
.Em skipped .
|
||
|
|
.Pp
|
||
|
|
If the test case is running as root and this property is
|
||
|
|
.Sq unprivileged ,
|
||
|
|
.Xr atf-run 1
|
||
|
|
will automatically drop the privileges if the
|
||
|
|
.Sq unprivileged-user
|
||
|
|
configuration property is set; otherwise the test case is
|
||
|
|
.Em skipped .
|
||
|
|
.It timeout
|
||
|
|
Type: integral.
|
||
|
|
Optional; defaults to
|
||
|
|
.Sq 300 .
|
||
|
|
.Pp
|
||
|
|
Specifies the maximum amount of time the test case can run.
|
||
|
|
This is particularly useful because some tests can stall either because they
|
||
|
|
are incorrectly coded or because they trigger an anomalous behavior of the
|
||
|
|
program.
|
||
|
|
It is not acceptable for these tests to stall the whole execution of the
|
||
|
|
test program.
|
||
|
|
.Pp
|
||
|
|
Can optionally be set to zero, in which case the test case has no run-time
|
||
|
|
limit.
|
||
|
|
This is discouraged.
|
||
|
|
.El
|
||
|
|
.Ss Environment
|
||
|
|
Every time a test case is executed, several environment variables are
|
||
|
|
cleared or reseted to sane values to ensure they do not make the test fail
|
||
|
|
due to unexpected conditions.
|
||
|
|
These variables are:
|
||
|
|
.Bl -tag -width LCXMESSAGESXX
|
||
|
|
.It Ev HOME
|
||
|
|
Set to the work directory's path.
|
||
|
|
.It Ev LANG
|
||
|
|
Undefined.
|
||
|
|
.It Ev LC_ALL
|
||
|
|
Undefined.
|
||
|
|
.It Ev LC_COLLATE
|
||
|
|
Undefined.
|
||
|
|
.It Ev LC_CTYPE
|
||
|
|
Undefined.
|
||
|
|
.It Ev LC_MESSAGES
|
||
|
|
Undefined.
|
||
|
|
.It Ev LC_MONETARY
|
||
|
|
Undefined.
|
||
|
|
.It Ev LC_NUMERIC
|
||
|
|
Undefined.
|
||
|
|
.It Ev LC_TIME
|
||
|
|
Undefined.
|
||
|
|
.It Ev TZ
|
||
|
|
Hardcoded to
|
||
|
|
.Sq UTC .
|
||
|
|
.El
|
||
|
|
.Ss Work directories
|
||
|
|
The test program always creates a temporary directory
|
||
|
|
and switches to it before running the test case's body.
|
||
|
|
This way the test case is free to modify its current directory as it
|
||
|
|
wishes, and the runtime engine will be able to clean it up later on in a
|
||
|
|
safe way, removing any traces of its execution from the system.
|
||
|
|
To do so, the runtime engine will perform a recursive removal of the work
|
||
|
|
directory without crossing mount points; if a mount point is found, the
|
||
|
|
file system will be unmounted (if possible).
|
||
|
|
.Ss File creation mode mask (umask)
|
||
|
|
Test cases are always executed with a file creation mode mask (umask) of
|
||
|
|
.Sq 0022 .
|
||
|
|
The test case's code is free to change this during execution.
|
||
|
|
.Sh SEE ALSO
|
||
|
|
.Xr atf-run 1 ,
|
||
|
|
.Xr atf-test-program 1 ,
|
||
|
|
.Xr atf-formats 5 ,
|
||
|
|
.Xr atf 7
|