sanchayanmaity
/
minix
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
minix/minix/benchmarks/unixbench-5.1.2/USAGE

395 lines
18 KiB
Plaintext
Raw Blame History

Running the Tests
=================
All the tests are executed using the "Run" script in the top-level directory.
The simplest way to generate results is with the commmand:
./Run
This will run a standard "index" test (see "The BYTE Index" below), and
save the report in the "results" directory, with a filename like
hostname-2007-09-23-01
An HTML version is also saved.
If you want to generate both the basic system index and the graphics index,
then do:
./Run gindex
If your system has more than one CPU, the tests will be run twice -- once
with a single copy of each test running at once, and once with N copies,
where N is the number of CPUs. Some categories of tests, however (currently
the graphics tests) will only run with a single copy.
Since the tests are based on constant time (variable work), a "system"
run usually takes about 29 minutes; the "graphics" part about 18 minutes.
A "gindex" run on a dual-core machine will do 2 "system" passes (single-
and dual-processing) and one "graphics" run, for a total around one and
a quarter hours.
============================================================================
Detailed Usage
==============
The Run script takes a number of options which you can use to customise a
test, and you can specify the names of the tests to run. The full usage
is:
Run [ -q | -v ] [-i <n> ] [-c <n> [-c <n> ...]] [test ...]
The option flags are:
-q Run in quiet mode.
-v Run in verbose mode.
-i <count> Run <count> iterations for each test -- slower tests
use <count> / 3, but at least 1. Defaults to 10 (3 for
slow tests).
-c <n> Run <n> copies of each test in parallel.
The -c option can be given multiple times; for example:
./Run -c 1 -c 4
will run a single-streamed pass, then a 4-streamed pass. Note that some
tests (currently the graphics tests) will only run in a single-streamed pass.
The remaining non-flag arguments are taken to be the names of tests to run.
The default is to run "index". See "Tests" below.
When running the tests, I do *not* recommend switching to single-user mode
("init 1"). This seems to change the results in ways I don't understand,
and it's not realistic (unless your system will actually be running in this
mode, of course). However, if using a windowing system, you may want to
switch to a minimal window setup (for example, log in to a "twm" session),
so that randomly-churning background processes don't randomise the results
too much. This is particularly true for the graphics tests.
============================================================================
Tests
=====
The available tests are organised into categories; when generating index
scores (see "The BYTE Index" below) the results for each category are
produced separately. The categories are:
system The original Unix system tests (not all are actually
in the index)
2d 2D graphics tests (not all are actually in the index)
3d 3D graphics tests
misc Various non-indexed tests
The following individual tests are available:
system:
dhry2reg Dhrystone 2 using register variables
whetstone-double Double-Precision Whetstone
syscall System Call Overhead
pipe Pipe Throughput
context1 Pipe-based Context Switching
spawn Process Creation
execl Execl Throughput
fstime-w File Write 1024 bufsize 2000 maxblocks
fstime-r File Read 1024 bufsize 2000 maxblocks
fstime File Copy 1024 bufsize 2000 maxblocks
fsbuffer-w File Write 256 bufsize 500 maxblocks
fsbuffer-r File Read 256 bufsize 500 maxblocks
fsbuffer File Copy 256 bufsize 500 maxblocks
fsdisk-w File Write 4096 bufsize 8000 maxblocks
fsdisk-r File Read 4096 bufsize 8000 maxblocks
fsdisk File Copy 4096 bufsize 8000 maxblocks
shell1 Shell Scripts (1 concurrent) (runs "looper 60 multi.sh 1")
shell8 Shell Scripts (8 concurrent) (runs "looper 60 multi.sh 8")
shell16 Shell Scripts (8 concurrent) (runs "looper 60 multi.sh 16")
2d:
2d-rects 2D graphics: rectangles
2d-lines 2D graphics: lines
2d-circle 2D graphics: circles
2d-ellipse 2D graphics: ellipses
2d-shapes 2D graphics: polygons
2d-aashapes 2D graphics: aa polygons
2d-polys 2D graphics: complex polygons
2d-text 2D graphics: text
2d-blit 2D graphics: images and blits
2d-window 2D graphics: windows
3d:
ubgears 3D graphics: gears
misc:
C C Compiler Throughput ("looper 60 $cCompiler cctest.c")
arithoh Arithoh (huh?)
short Arithmetic Test (short) (this is arith.c configured for
"short" variables; ditto for the ones below)
int Arithmetic Test (int)
long Arithmetic Test (long)
float Arithmetic Test (float)
double Arithmetic Test (double)
dc Dc: sqrt(2) to 99 decimal places (runs
"looper 30 dc < dc.dat", using your system's copy of "dc")
hanoi Recursion Test -- Tower of Hanoi
grep Grep for a string in a large file, using your system's
copy of "grep"
sysexec Exercise fork() and exec().
The following pseudo-test names are aliases for combinations of other
tests:
arithmetic Runs arithoh, short, int, long, float, double,
and whetstone-double
dhry Alias for dhry2reg
dhrystone Alias for dhry2reg
whets Alias for whetstone-double
whetstone Alias for whetstone-double
load Runs shell1, shell8, and shell16
misc Runs C, dc, and hanoi
speed Runs the arithmetic and system groups
oldsystem Runs execl, fstime, fsbuffer, fsdisk, pipe, context1,
spawn, and syscall
system Runs oldsystem plus shell1, shell8, and shell16
fs Runs fstime-w, fstime-r, fstime, fsbuffer-w,
fsbuffer-r, fsbuffer, fsdisk-w, fsdisk-r, and fsdisk
shell Runs shell1, shell8, and shell16
index Runs the tests which constitute the official index:
the oldsystem group, plus dhry2reg, whetstone-double,
shell1, and shell8
See "The BYTE Index" below for more information.
graphics Runs the tests which constitute the graphics index:
2d-rects, 2d-ellipse, 2d-aashapes, 2d-text, 2d-blit,
2d-window, and ubgears
gindex Runs the index and graphics groups, to generate both
sets of index results
all Runs all tests
============================================================================
The BYTE Index
==============
The purpose of this test is to provide a basic indicator of the performance
of a Unix-like system; hence, multiple tests are used to test various
aspects of the system's performance. These test results are then compared
to the scores from a baseline system to produce an index value, which is
generally easier to handle than the raw sores. The entire set of index
values is then combined to make an overall index for the system.
Since 1995, the baseline system has been "George", a SPARCstation 20-61
with 128 MB RAM, a SPARC Storage Array, and Solaris 2.3, whose ratings
were set at 10.0. (So a system which scores 520 is 52 times faster than
this machine.) Since the numbers are really only useful in a relative
sense, there's no particular reason to update the base system, so for the
sake of consistency it's probably best to leave it alone. George's scores
are in the file "pgms/index.base"; this file is used to calculate the
index scores for any particular run.
Over the years, various changes have been made to the set of tests in the
index. Although there is a desire for a consistent baseline, various tests
have been determined to be misleading, and have been removed; and a few
alternatives have been added. These changes are detailed in the README,
and should be born in mind when looking at old scores.
A number of tests are included in the benchmark suite which are not part of
the index, for various reasons; these tests can of course be run manually.
See "Tests" above.
============================================================================
Graphics Tests
==============
As of version 5.1, UnixBench now contains some graphics benchmarks. These
are intended to give a rough idea of the general graphics performance of
a system.
The graphics tests are in categories "2d" and "3d", so the index scores
for these tests are separate from the basic system index. This seems
like a sensible division, since the graphics performance of a system
depends largely on the graphics adaptor.
The tests currently consist of some 2D "x11perf" tests and "ubgears".
* The 2D tests are a selection of the x11perf tests, using the host
system's x11perf command (which must be installed and in the search
path). Only a few of the x11perf tests are used, in the interests
of completing a test run in a reasonable time; if you want to do
detailed diagnosis of an X server or graphics chip, then use x11perf
directly.
* The 3D test is "ubgears", a modified version of the familiar "glxgears".
This version runs for 5 seconds to "warm up", then performs a timed
run and displays the average frames-per-second.
On multi-CPU systems, the graphics tests will only run in single-processing
mode. This is because the meaning of running two copies of a test at once
is dubious; and the test windows tend to overlay each other, meaning that
the window behind isn't actually doing any work.
============================================================================
Multiple CPUs
=============
If your system has multiple CPUs, the default behaviour is to run the selected
tests twice -- once with one copy of each test program running at a time,
and once with N copies, where N is the number of CPUs. (You can override
this with the "-c" option; see "Detailed Usage" above.) This is designed to
allow you to assess:
- the performance of your system when running a single task
- the performance of your system when running multiple tasks
- the gain from your system's implementation of parallel processing
The results, however, need to be handled with care. Here are the results
of two runs on a dual-processor system, one in single-processing mode, one
dual-processing:
Test Single Dual Gain
-------------------- ------ ------ ----
Dhrystone 2 562.5 1110.3 97%
Double Whetstone 320.0 640.4 100%
Execl Throughput 450.4 880.3 95%
File Copy 1024 759.4 595.9 -22%
File Copy 256 535.8 438.8 -18%
File Copy 4096 1261.8 1043.4 -17%
Pipe Throughput 481.0 979.3 104%
Pipe-based Switching 326.8 1229.0 276%
Process Creation 917.2 1714.1 87%
Shell Scripts (1) 1064.9 1566.3 47%
Shell Scripts (8) 1567.7 1709.9 9%
System Call Overhead 944.2 1445.5 53%
-------------------- ------ ------ ----
Index Score: 678.2 1026.2 51%
As expected, the heavily CPU-dependent tasks -- dhrystone, whetstone,
execl, pipe throughput, process creation -- show close to 100% gain when
running 2 copies in parallel.
The Pipe-based Context Switching test measures context switching overhead
by sending messages back and forth between 2 processes. I don't know why
it shows such a huge gain with 2 copies (ie. 4 processes total) running,
but it seems to be consistent on my system. I think this may be an issue
with the SMP implementation.
The System Call Overhead shows a lesser gain, presumably because it uses a
lot of CPU time in single-threaded kernel code. The shell scripts test with
8 concurrent processes shows no gain -- because the test itself runs 8
scripts in parallel, it's already using both CPUs, even when the benchmark
is run in single-stream mode. The same test with one process per copy
shows a real gain.
The filesystem throughput tests show a loss, instead of a gain, when
multi-processing. That there's no gain is to be expected, since the tests
are presumably constrained by the throughput of the I/O subsystem and the
disk drive itself; the drop in performance is presumably down to the
increased contention for resources, and perhaps greater disk head movement.
So what tests should you use, how many copies should you run, and how should
you interpret the results? Well, that's up to you, since it depends on
what it is you're trying to measure.
Implementation
--------------
The multi-processing mode is implemented at the level of test iterations.
During each iteration of a test, N slave processes are started using fork().
Each of these slaves executes the test program using fork() and exec(),
reads and stores the entire output, times the run, and prints all the
results to a pipe. The Run script reads the pipes for each of the slaves
in turn to get the results and times. The scores are added, and the times
averaged.
The result is that each test program has N copies running at once. They
should all finish at around the same time, since they run for constant time.
If a test program itself starts off K multiple processes (as with the shell8
test), then the effect will be that there are N * K processes running at
once. This is probably not very useful for testing multi-CPU performance.
============================================================================
The Language Setting
====================
The $LANG environment variable determines how programs abnd library
routines interpret text. This can have a big impact on the test results.
If $LANG is set to POSIX, or is left unset, text is treated as ASCII; if
it is set to en_US.UTF-8, foir example, then text is treated as being
encoded in UTF-8, which is more complex and therefore slower. Setting
it to other languages can have varying results.
To ensure consistency between test runs, the Run script now (as of version
5.1.1) sets $LANG to "en_US.utf8".
This setting which is configured with the variable "$language". You
should not change this if you want to share your results to allow
comparisons between systems; however, you may want to change it to see
how different language settings affect performance.
Each test report now includes the language settings in use. The reported
language is what is set in $LANG, and is not necessarily supported by the
system; but we also report the character mapping and collation order which
are actually in use (as reported by "locale").
============================================================================
Interpreting the Results
========================
Interpreting the results of these tests is tricky, and totally depends on
what you're trying to measure.
For example, are you trying to measure how fast your CPU is? Or how good
your compiler is? Because these tests are all recompiled using your host
system's compiler, the performance of the compiler will inevitably impact
the performance of the tests. Is this a problem? If you're choosing a
system, you probably care about its overall speed, which may well depend
on how good its compiler is; so including that in the test results may be
the right answer. But you may want to ensure that the right compiler is
used to build the tests.
On the other hand, with the vast majority of Unix systems being x86 / PC
compatibles, running Linux and the GNU C compiler, the results will tend
to be more dependent on the hardware; but the versions of the compiler and
OS can make a big difference. (I measured a 50% gain between SUSE 10.1
and OpenSUSE 10.2 on the same machine.) So you may want to make sure that
all your test systems are running the same version of the OS; or at least
publish the OS and compuiler versions with your results. Then again, it may
be compiler performance that you're interested in.
The C test is very dubious -- it tests the speed of compilation. If you're
running the exact same compiler on each system, OK; but otherwise, the
results should probably be discarded. A slower compilation doesn't say
anything about the speed of your system, since the compiler may simply be
spending more time to super-optimise the code, which would actually make it
faster.
This will be particularly true on architectures like IA-64 (Itanium etc.)
where the compiler spends huge amounts of effort scheduling instructions
to run in parallel, with a resultant significant gain in execution speed.
Some tests are even more dubious in terms of host-dependency -- for example,
the "dc" test uses the host's version of dc (a calculator program). The
version of this which is available can make a huge difference to the score,
which is why it's not in the index group. Read through the release notes
for more on these kinds of issues.
Another age-old issue is that of the benchmarks being too trivial to be
meaningful. With compilers getting ever smarter, and performing more
wide-ranging flow path analyses, the danger of parts of the benchmarks
simply being optimised out of existance is always present.
All in all, the "index" and "gindex" tests (see above) are designed to
give a reasonable measure of overall system performance; but the results
of any test run should always be used with care.
Reference in New Issue View Git Blame Copy Permalink