11be35a165
To do so, a few dependencies have been imported: * external/bsd/lutok * external/mit/lua * external/public-domain/sqlite * external/public-domain/xz The Kyua framework is the new generation of ATF (Automated Test Framework), it is composed of: * external/bsd/atf * external/bsd/kyua-atf-compat * external/bsd/kyua-cli * external/bsd/kyua-tester * tests Kyua/ATF being written in C++, it depends on libstdc++ which is provided by GCC. As this is not part of the sources, Kyua is only compiled when the native GCC utils are installed. To install Kyua do the following: * In a cross-build enviromnent, add the following to the build.sh commandline: -V MKBINUTILS=yes -V MKGCCCMDS=yes WARNING: At this point the import is still experimental, and not supported on native builds (a.k.a make build). Change-Id: I26aee23c5bbd2d64adcb7c1beb98fe0d479d7ada
368 lines
11 KiB
Groff
368 lines
11 KiB
Groff
.\" Copyright 2011 Google Inc.
|
|
.\" All rights reserved.
|
|
.\"
|
|
.\" Redistribution and use in source and binary forms, with or without
|
|
.\" modification, are permitted provided that the following conditions are
|
|
.\" met:
|
|
.\"
|
|
.\" * Redistributions of source code must retain the above copyright
|
|
.\" notice, this list of conditions and the following disclaimer.
|
|
.\" * Redistributions in binary form must reproduce the above copyright
|
|
.\" notice, this list of conditions and the following disclaimer in the
|
|
.\" documentation and/or other materials provided with the distribution.
|
|
.\" * Neither the name of Google Inc. nor the names of its contributors
|
|
.\" may be used to endorse or promote products derived from this software
|
|
.\" without specific prior written permission.
|
|
.\"
|
|
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
|
|
.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
|
|
.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
|
|
.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
|
|
.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
|
|
.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
|
|
.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
|
|
.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
|
|
.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
|
|
.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
|
.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
|
.Dd February 7, 2013
|
|
.Dt KYUA 1
|
|
.Os
|
|
.Sh NAME
|
|
.Nm kyua
|
|
.Nd Command-line interface to Kyua quality assurance toolkit
|
|
.Sh SYNOPSIS
|
|
.Nm
|
|
.Op Fl -config Ar file
|
|
.Op Fl -logfile Ar file
|
|
.Op Fl -loglevel Ar level
|
|
.Op Fl -variable Ar name=value
|
|
.Ar command
|
|
.Op Ar command_options
|
|
.Op Ar command_arguments
|
|
.Sh DESCRIPTION
|
|
Kyua is a quality assurance toolkit: a package that provides libraries to
|
|
aid in the creation of test cases, and tools to execute such test cases and
|
|
generate reports of their results.
|
|
.Pp
|
|
This manual page describes
|
|
.Nm
|
|
(note the lowercase), which is the command-line interface to the Kyua
|
|
package.
|
|
.Pp
|
|
If you are here because all you want to know is how to run the tests in
|
|
.Pa __TESTSDIR__ ,
|
|
you most likely want to read the
|
|
.Xr tests 7
|
|
manual page instead if your system provides one.
|
|
.Pp
|
|
As can be observed in the synopsis, the interface of
|
|
.Nm
|
|
implements a common subcommand-based interface. The arguments to the tool
|
|
specify, in this order: a set of common options that all the commands
|
|
accept, a required
|
|
.Ar command
|
|
name that specifies what
|
|
.Nm
|
|
should do, and
|
|
a set of possibly-optional
|
|
.Ar command_options
|
|
and
|
|
.Ar command_arguments
|
|
that are specific to the chosen command.
|
|
.Ss Options supported by all commands
|
|
The following options are recognized by all the commands.
|
|
Keep in mind that these must always be specified before the command name.
|
|
.Bl -tag -width XX
|
|
.It Fl -config Ar path , Fl c Ar path
|
|
Specifies the configuration file to process, which must be in the format
|
|
described in
|
|
.Xr kyua.conf 5 .
|
|
The special value
|
|
.Sq none
|
|
explicitly disables the loading of any configuration file.
|
|
.Pp
|
|
Defaults to
|
|
.Pa ~/.kyua/kyua.conf
|
|
if it exists, otherwise to
|
|
.Pa __CONFDIR__/kyua.conf
|
|
if it exists,
|
|
or else to
|
|
.Sq none .
|
|
.It Fl -logfile Ar path
|
|
Specifies the location of the file to which
|
|
.Nm
|
|
will log run time events useful for postmortem debugging.
|
|
.Pp
|
|
The default depends on different environment variables as described in
|
|
.Sx Logging ,
|
|
but typically the file will be stored within the user's home directory.
|
|
.It Fl -loglevel Ar level
|
|
Specifies the maximum logging level to record in the log file.
|
|
See
|
|
.Sx Logging
|
|
for more details.
|
|
.Pp
|
|
The default is
|
|
.Sq info .
|
|
.It Fl -variable Ar name=value , Fl v Ar name=value
|
|
Sets the
|
|
.Ar name
|
|
configuration variable to
|
|
.Ar value .
|
|
The values set through this option have preference over the values set in the
|
|
configuration file.
|
|
.Pp
|
|
The specified variable can either be a builtin variable or a test-suite
|
|
specific variable. See
|
|
.Xr kyua.conf 5
|
|
for more details.
|
|
.El
|
|
.Ss Available commands
|
|
The following commands are generic and do not have any relation to the execution
|
|
of tests or the inspection of their results:
|
|
.Bl -tag -width reportXhtmlXX -offset indent
|
|
.It Ar about
|
|
Shows general program information.
|
|
See
|
|
.Xr kyua-about 1 .
|
|
.It Ar config
|
|
Inspects the values of the configuration variables.
|
|
See
|
|
.Xr kyua-config 1 .
|
|
.It Ar db-exec
|
|
Executes an arbitrary SQL statement in the store database and prints the
|
|
resulting table.
|
|
See
|
|
.Xr kyua-db-exec 1 .
|
|
.It Ar help
|
|
Shows usage information.
|
|
See
|
|
.Xr kyua-help 1 .
|
|
.El
|
|
.Pp
|
|
The following commands are used to generate reports based on the data previously
|
|
stored in the database:
|
|
.Bl -tag -width reportXhtmlXX -offset indent
|
|
.It Ar report
|
|
Generates a plain-text report.
|
|
See
|
|
.Xr kyua-report 1 .
|
|
.It Ar report-html
|
|
Generates an HTML report.
|
|
See
|
|
.Xr kyua-report-html 1 .
|
|
.El
|
|
.Pp
|
|
The following commands are used to interact with a test suite:
|
|
.Bl -tag -width reportXhtmlXX -offset indent
|
|
.It Ar debug
|
|
Executes a single test case in a controlled environment for debugging purposes.
|
|
See
|
|
.Xr kyua-debug 1 .
|
|
.It Ar list
|
|
Lists test cases and their metadata.
|
|
See
|
|
.Xr kyua-list 1 .
|
|
.It Ar test
|
|
Runs tests.
|
|
See
|
|
.Xr kyua-test 1 .
|
|
.El
|
|
.Ss Logging
|
|
.Nm
|
|
has a logging facility that collects all kinds of events at run time.
|
|
These events are always logged to a file so that the log is available when
|
|
it is most needed: right after a non-reproducible problem happens.
|
|
The only way to disable logging is by sending the log to
|
|
.Pa /dev/null .
|
|
.Pp
|
|
The location of the log file can be manually specified with the
|
|
.Fl -logfile
|
|
option, which applies to all commands. If no file is explicitly specified, the
|
|
location of the log files is chosen in this order:
|
|
.Bl -enum -offset indent
|
|
.It
|
|
.Pa ${HOME}/.kyua/logs/
|
|
if
|
|
.Va HOME
|
|
is defined.
|
|
.It
|
|
.Pa ${TMPDIR}/
|
|
if
|
|
.Va TMPDIR
|
|
is defined.
|
|
.It
|
|
.Pa /tmp/ .
|
|
.El
|
|
.Pp
|
|
And the default naming scheme of the log files is:
|
|
.Sq <progname>.<timestamp>.log .
|
|
.Pp
|
|
The messages stored in the log file have a level (or severity) attached to
|
|
them. These are:
|
|
.Bl -tag -width warningXX -offset indent
|
|
.It error
|
|
Fatal error messages. The program generally terminates after these, either
|
|
in a clean manner or by crashing.
|
|
.It warning
|
|
Non-fatal error messages. These generally report a condition that must be
|
|
addressed but the application can continue to run.
|
|
.It info
|
|
Informational messages. These tell the user what the program was doing at
|
|
a general level of operation.
|
|
.It debug
|
|
Detailed informational messages. These are often useful when debugging
|
|
problems in the application, as they contain lots of internal details.
|
|
.El
|
|
.Pp
|
|
The default log level is
|
|
.Sq info
|
|
unless expicitly overriden with
|
|
.Fl -loglevel .
|
|
.Pp
|
|
The log file is a plain text file containing one line per log record. The
|
|
format of each line is as follows:
|
|
.Bd -literal -offset indent
|
|
timestamp entry_type pid file:line: message
|
|
.Ed
|
|
.Pp
|
|
.Ar entry_type
|
|
can be one of:
|
|
.Sq E
|
|
for an error,
|
|
.Sq W
|
|
for a warning,
|
|
.Sq I
|
|
for an informational message and
|
|
.Sq D
|
|
for a debug message.
|
|
.Ss Bug reporting
|
|
If you think you have encountered a bug in
|
|
.Nm ,
|
|
please take the time to let the developers know about it. This will ensure
|
|
that the bug is addressed and potentially fixed in the next Kyua release.
|
|
.Pp
|
|
The first step in reporting a bug is to check if there already is a similar
|
|
bug in the database. You can check what issues are currently in the
|
|
database by going to:
|
|
.Bd -literal -offset indent
|
|
http://code.google.com/p/kyua/issues/list
|
|
.Ed
|
|
.Pp
|
|
If there is no existing issue that describes an issue similar to the
|
|
one you are experiencing, you can open a new one by visiting:
|
|
.Bd -literal -offset indent
|
|
http://code.google.com/p/kyua/issues/entry
|
|
.Ed
|
|
.Pp
|
|
When doing so, please include as much detail as possible. Among other
|
|
things, explain what operating system and platform you are running
|
|
.Nm
|
|
on, what were you trying to do, what exact messages you saw on the screen,
|
|
how did you expect the program to behave, and any other details that you
|
|
may find relevant.
|
|
.Pp
|
|
Also, please include a copy of the log file corresponding to the problem
|
|
you are experiencing. Unless you have changed the location of the log
|
|
files, you can most likely find them in
|
|
.Pa ~/.kyua/logs/ .
|
|
If the problem is reproducible, it is good idea to regenerate the log file
|
|
with an increased log level so as to provide more information. For
|
|
example:
|
|
.Bd -literal -offset indent
|
|
$ kyua --logfile=problem.log --loglevel=debug \\
|
|
[rest of the command line]
|
|
.Ed
|
|
.Sh ENVIRONMENT
|
|
The following variables are recognized and can be freely tuned by the end user:
|
|
.Bl -tag -width COLUMNSXX
|
|
.It Va COLUMNS
|
|
The width of the screen, in number of characters.
|
|
.Nm
|
|
uses this to wrap long lines.
|
|
If not present, the width of the screen is determined from the terminal
|
|
stdout is connected to, and, if the guessing fails, this defaults to infinity.
|
|
.It Va HOME
|
|
Path to the user's home directory.
|
|
.Nm
|
|
uses this location to determine paths to configuration files and default log
|
|
files.
|
|
.It Va TMPDIR
|
|
Path to the system-wide temporary directory.
|
|
.Nm
|
|
uses this location to place the work directory of test cases, among other
|
|
things.
|
|
.Pp
|
|
The default value of this variable depends on the operating system. In
|
|
general, it is
|
|
.Pa /tmp .
|
|
.El
|
|
.Pp
|
|
The following variables are also recognized, but you should not need to set them
|
|
during normal operation.
|
|
They are only provided to override the value of built-in values, which is useful
|
|
when testing
|
|
.Nm
|
|
itself:
|
|
.Bl -tag -width KYUAXCONFDIRXX
|
|
.It Va KYUA_CONFDIR
|
|
Path to the system-wide configuration files for
|
|
.Nm .
|
|
.Pp
|
|
Defaults to
|
|
.Pa __CONFDIR__ .
|
|
.It Va KYUA_DOCDIR
|
|
Path to the location of installed documentation.
|
|
.Pp
|
|
Defaults to
|
|
.Pa __DOCDIR__ .
|
|
.It Va KYUA_MISCDIR
|
|
Path to the location of the installed miscellaneous scripts and data
|
|
files provided by
|
|
.Nm .
|
|
.Pp
|
|
Defaults to
|
|
.Pa __MISCDIR__ .
|
|
.It Va KYUA_STOREDIR
|
|
Path to the location of the installed store support files; e.g. the
|
|
directory containing the SQL database schema.
|
|
.Pp
|
|
Defaults to
|
|
.Pa __STOREDIR__ .
|
|
.It Va KYUA_TESTERSDIR
|
|
Path to the location of the installed testers; e.g. the directory containing
|
|
programs like
|
|
.Pa plain_tester .
|
|
Defaults to
|
|
.Pa __TESTERSDIR__ .
|
|
.El
|
|
.Sh FILES
|
|
.Bl -tag -width XXXX
|
|
.It Pa ~/.kyua/kyua.conf
|
|
User-specific configuration file.
|
|
.It Pa ~/.kyua/logs/
|
|
Default location for the collected log files.
|
|
.It Pa __CONFDIR__/kyua.conf
|
|
System-wide configuration file.
|
|
.El
|
|
.Sh EXIT STATUS
|
|
.Nm
|
|
returns 0 on success, 1 on a controlled error condition in the given
|
|
subcommand, 2 on a general unexpected error and 3 on a usage error.
|
|
.Pp
|
|
The documentation of the subcommands in the corresponding manual pages only
|
|
details the difference between a successful exit (0) and the detection of a
|
|
controlled error (1). Even though when those manual pages do not describe
|
|
any other exit statuses, codes above 1 can be returned.
|
|
.Sh SEE ALSO
|
|
.Xr atf 7 ,
|
|
.Xr tests 7
|
|
.Sh AUTHORS
|
|
For more details on the people that made
|
|
.Nm
|
|
possible and the license terms, run:
|
|
.Bd -literal -offset indent
|
|
$ kyua about
|
|
.Ed
|