To do so, a few dependencies have been imported: * external/bsd/lutok * external/mit/lua * external/public-domain/sqlite * external/public-domain/xz The Kyua framework is the new generation of ATF (Automated Test Framework), it is composed of: * external/bsd/atf * external/bsd/kyua-atf-compat * external/bsd/kyua-cli * external/bsd/kyua-tester * tests Kyua/ATF being written in C++, it depends on libstdc++ which is provided by GCC. As this is not part of the sources, Kyua is only compiled when the native GCC utils are installed. To install Kyua do the following: * In a cross-build enviromnent, add the following to the build.sh commandline: -V MKBINUTILS=yes -V MKGCCCMDS=yes WARNING: At this point the import is still experimental, and not supported on native builds (a.k.a make build). Change-Id: I26aee23c5bbd2d64adcb7c1beb98fe0d479d7ada
205 lines
6.2 KiB
Groff
205 lines
6.2 KiB
Groff
.\" Copyright 2012 Google Inc.
|
|
.\" All rights reserved.
|
|
.\"
|
|
.\" Redistribution and use in source and binary forms, with or without
|
|
.\" modification, are permitted provided that the following conditions are
|
|
.\" met:
|
|
.\"
|
|
.\" * Redistributions of source code must retain the above copyright
|
|
.\" notice, this list of conditions and the following disclaimer.
|
|
.\" * Redistributions in binary form must reproduce the above copyright
|
|
.\" notice, this list of conditions and the following disclaimer in the
|
|
.\" documentation and/or other materials provided with the distribution.
|
|
.\" * Neither the name of Google Inc. nor the names of its contributors
|
|
.\" may be used to endorse or promote products derived from this software
|
|
.\" without specific prior written permission.
|
|
.\"
|
|
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
|
|
.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
|
|
.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
|
|
.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
|
|
.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
|
|
.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
|
|
.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
|
|
.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
|
|
.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
|
|
.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
|
.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
|
.Dd December 26, 2012
|
|
.Dt KYUA-TESTER 1
|
|
.Os
|
|
.Sh NAME
|
|
.Nm kyua-tester
|
|
.Nd Scriptable interface to interact with test programs
|
|
.Sh SYNOPSIS
|
|
.Nm
|
|
.Op Fl g Ar gid
|
|
.Op Fl t Ar timeout
|
|
.Op Fl u Ar uid
|
|
.Ar list
|
|
.Ar test_program
|
|
.Nm
|
|
.Op Fl g Ar gid
|
|
.Op Fl t Ar timeout
|
|
.Op Fl u Ar uid
|
|
.Ar test
|
|
.Op Fl v Ar var=value
|
|
.Ar test_program
|
|
.Ar test_case
|
|
.Ar result_file
|
|
.Sh DESCRIPTION
|
|
This generic manual page does
|
|
.Em not
|
|
correspond to any specific binary. It describes the generic command-line
|
|
interface provided by all Kyua testers shipped by the
|
|
.Sq __PACKAGE__
|
|
package, which all have names of the form
|
|
.Nm kyua-INTERFACE-NAME-tester .
|
|
.Pp
|
|
The main goal of a
|
|
.Nm
|
|
is to provide a scriptable interface to run
|
|
.Em a single test case
|
|
of
|
|
.Em a single test program
|
|
in a generic manner. The specific implementation of how to do this is up to the
|
|
tester, as each
|
|
.Nm
|
|
binary implements a specific test interface. The following binaries are shipped
|
|
with the
|
|
.Sq __PACKAGE__
|
|
package:
|
|
.Bl -tag -width XXXX
|
|
.It Xr kyua-atf-tester 1
|
|
An implementation of
|
|
.Nm
|
|
that wraps tests that follow the
|
|
.Xr kyua-atf-interface 7
|
|
interface.
|
|
.It Xr kyua-plain-tester 1
|
|
An implementation of
|
|
.Nm
|
|
that wraps tests that follow the
|
|
.Xr kyua-plain-interface 7
|
|
interface.
|
|
.El
|
|
.Pp
|
|
.Xr kyua 1
|
|
uses the various
|
|
.Nm
|
|
binaries to execute tests that implement diverse interfaces, without having to
|
|
know about any of its specifics. It is possible for other packages to provide
|
|
new testers that implement other interfaces as long as the tester's command line
|
|
complies to what is described in this manual page.
|
|
.Pp
|
|
A
|
|
.Nm
|
|
implements a typical command-line interface based on subcommands. The tool
|
|
takes a set of generic options first, a subcommand name, a set of any options
|
|
accepted by such subcommand, and a set of arguments to the subcommand.
|
|
.Pp
|
|
The following options apply to all subcommands:
|
|
.Bl -tag -width XtXtimeoutXX
|
|
.It Fl g Ar gid
|
|
GID of the user to switch to before running the test case. Defaults to the
|
|
current GID.
|
|
.It Fl t Ar timeout
|
|
Seconds to allow the test program to run before killing it. Defaults to 60.
|
|
.It Fl u Ar uid
|
|
UID of the user to switch to before running the test case. Defaults to the
|
|
current UID.
|
|
.El
|
|
.Pp
|
|
Killing the tester should always be a safe operation, resulting in the
|
|
subsequent termination of the test program being run and the cleanup of any used
|
|
resources.
|
|
.Ss Workflow
|
|
The way
|
|
.Xr kyua 1
|
|
uses the testers to interact with individual test cases is the following:
|
|
.Bl -enum
|
|
.It
|
|
For every test program listed in a
|
|
.Xr kyuafile 5 ,
|
|
execute the
|
|
.Sq list
|
|
subcommand of the
|
|
.Nm
|
|
that matches the interface name in the test program description.
|
|
.It
|
|
.Xr kyua 1
|
|
collects the test cases returned in the previous step and checks if they are
|
|
runnable according to their metadata.
|
|
.It
|
|
For every test case collected from the previous steps, execute the
|
|
.Sq test
|
|
subcommand of the same
|
|
.Nm
|
|
used in the test case listing.
|
|
.It
|
|
Reinterpret the generic test result and record it as necessary.
|
|
.El
|
|
.Ss The list subcommand
|
|
The
|
|
.Sq list
|
|
subcommand extracts the list of test cases that are part of the given test
|
|
program in a machine-parseable format. The test program should be provided as
|
|
an absolute pathname.
|
|
.Pp
|
|
The format of the output is described in
|
|
.Xr kyua-tester-list 5 .
|
|
.Ss The test subcommand
|
|
The
|
|
.Sq test
|
|
subcommand executes a single test case of a test program in a controlled manner,
|
|
which means that it is placed in its own subprocess and its own work directory
|
|
so that side-effects during execution are minimized.
|
|
.Pp
|
|
If the test is executed successfully, the
|
|
.Ar result_file
|
|
is created following the format described in
|
|
.Xr kyua-tester-result 5 .
|
|
.Pp
|
|
If the test crashes halfway through and dumps core,
|
|
.Nm
|
|
will attempt to gather a stacktrace and print it as part of the test's output.
|
|
.Pp
|
|
The following options are specific to the test command:
|
|
.Bl -tag -width XvXvar=nameXX
|
|
.It Fl v Ar var=name
|
|
Passes a configuration variable to the test case.
|
|
.El
|
|
.Pp
|
|
The following configuration variables have special meaning:
|
|
.Bl -tag -width unprivilegedXuserXX
|
|
.It unprivileged-user
|
|
The name (not UID) of an unprivileged user that the test can use for its own
|
|
purposes.
|
|
.El
|
|
.Sh ENVIRONMENT
|
|
The following variables are recognized and can be freely tuned by the end user:
|
|
.Bl -tag -width TMPDIRXX
|
|
.It Va TMPDIR
|
|
Path to the system-wide temporary directory.
|
|
.Nm
|
|
uses this location to place the work directory of test cases.
|
|
.Pp
|
|
The default value is
|
|
.Pa __TMPDIR__ .
|
|
.El
|
|
.Sh EXIT STATUS
|
|
.Nm
|
|
returns the following exit codes:
|
|
.Bl -tag -compact -width 0XX
|
|
.It 0
|
|
The tester ran correctly and the test passed.
|
|
.It 1
|
|
The tester ran correctly, but the test failed.
|
|
.It 2
|
|
The tester failed. The result of the test itself is irrelevant.
|
|
.It 3
|
|
The user invoked the tester incorrectly.
|
|
.El
|
|
.Sh SEE ALSO
|
|
.Xr kyua-atf-tester 1 ,
|
|
.Xr kyua-plain-tester 1
|