Running the Tests ================= All the tests are executed using the "Run" script in the top-level directory. The simplest way to generate results is with the commmand: ./Run This will run a standard "index" test (see "The BYTE Index" below), and save the report in the "results" directory, with a filename like hostname-2007-09-23-01 An HTML version is also saved. If you want to generate both the basic system index and the graphics index, then do: ./Run gindex If your system has more than one CPU, the tests will be run twice -- once with a single copy of each test running at once, and once with N copies, where N is the number of CPUs. Some categories of tests, however (currently the graphics tests) will only run with a single copy. Since the tests are based on constant time (variable work), a "system" run usually takes about 29 minutes; the "graphics" part about 18 minutes. A "gindex" run on a dual-core machine will do 2 "system" passes (single- and dual-processing) and one "graphics" run, for a total around one and a quarter hours. ============================================================================ Detailed Usage ============== The Run script takes a number of options which you can use to customise a test, and you can specify the names of the tests to run. The full usage is: Run [ -q | -v ] [-i ] [-c [-c ...]] [test ...] The option flags are: -q Run in quiet mode. -v Run in verbose mode. -i Run iterations for each test -- slower tests use / 3, but at least 1. Defaults to 10 (3 for slow tests). -c Run copies of each test in parallel. The -c option can be given multiple times; for example: ./Run -c 1 -c 4 will run a single-streamed pass, then a 4-streamed pass. Note that some tests (currently the graphics tests) will only run in a single-streamed pass. The remaining non-flag arguments are taken to be the names of tests to run. The default is to run "index". See "Tests" below. When running the tests, I do *not* recommend switching to single-user mode ("init 1"). This seems to change the results in ways I don't understand, and it's not realistic (unless your system will actually be running in this mode, of course). However, if using a windowing system, you may want to switch to a minimal window setup (for example, log in to a "twm" session), so that randomly-churning background processes don't randomise the results too much. This is particularly true for the graphics tests. ============================================================================ Tests ===== The available tests are organised into categories; when generating index scores (see "The BYTE Index" below) the results for each category are produced separately. The categories are: system The original Unix system tests (not all are actually in the index) 2d 2D graphics tests (not all are actually in the index) 3d 3D graphics tests misc Various non-indexed tests The following individual tests are available: system: dhry2reg Dhrystone 2 using register variables whetstone-double Double-Precision Whetstone syscall System Call Overhead pipe Pipe Throughput context1 Pipe-based Context Switching spawn Process Creation execl Execl Throughput fstime-w File Write 1024 bufsize 2000 maxblocks fstime-r File Read 1024 bufsize 2000 maxblocks fstime File Copy 1024 bufsize 2000 maxblocks fsbuffer-w File Write 256 bufsize 500 maxblocks fsbuffer-r File Read 256 bufsize 500 maxblocks fsbuffer File Copy 256 bufsize 500 maxblocks fsdisk-w File Write 4096 bufsize 8000 maxblocks fsdisk-r File Read 4096 bufsize 8000 maxblocks fsdisk File Copy 4096 bufsize 8000 maxblocks shell1 Shell Scripts (1 concurrent) (runs "looper 60 multi.sh 1") shell8 Shell Scripts (8 concurrent) (runs "looper 60 multi.sh 8") shell16 Shell Scripts (8 concurrent) (runs "looper 60 multi.sh 16") 2d: 2d-rects 2D graphics: rectangles 2d-lines 2D graphics: lines 2d-circle 2D graphics: circles 2d-ellipse 2D graphics: ellipses 2d-shapes 2D graphics: polygons 2d-aashapes 2D graphics: aa polygons 2d-polys 2D graphics: complex polygons 2d-text 2D graphics: text 2d-blit 2D graphics: images and blits 2d-window 2D graphics: windows 3d: ubgears 3D graphics: gears misc: C C Compiler Throughput ("looper 60 $cCompiler cctest.c") arithoh Arithoh (huh?) short Arithmetic Test (short) (this is arith.c configured for "short" variables; ditto for the ones below) int Arithmetic Test (int) long Arithmetic Test (long) float Arithmetic Test (float) double Arithmetic Test (double) dc Dc: sqrt(2) to 99 decimal places (runs "looper 30 dc < dc.dat", using your system's copy of "dc") hanoi Recursion Test -- Tower of Hanoi grep Grep for a string in a large file, using your system's copy of "grep" sysexec Exercise fork() and exec(). The following pseudo-test names are aliases for combinations of other tests: arithmetic Runs arithoh, short, int, long, float, double, and whetstone-double dhry Alias for dhry2reg dhrystone Alias for dhry2reg whets Alias for whetstone-double whetstone Alias for whetstone-double load Runs shell1, shell8, and shell16 misc Runs C, dc, and hanoi speed Runs the arithmetic and system groups oldsystem Runs execl, fstime, fsbuffer, fsdisk, pipe, context1, spawn, and syscall system Runs oldsystem plus shell1, shell8, and shell16 fs Runs fstime-w, fstime-r, fstime, fsbuffer-w, fsbuffer-r, fsbuffer, fsdisk-w, fsdisk-r, and fsdisk shell Runs shell1, shell8, and shell16 index Runs the tests which constitute the official index: the oldsystem group, plus dhry2reg, whetstone-double, shell1, and shell8 See "The BYTE Index" below for more information. graphics Runs the tests which constitute the graphics index: 2d-rects, 2d-ellipse, 2d-aashapes, 2d-text, 2d-blit, 2d-window, and ubgears gindex Runs the index and graphics groups, to generate both sets of index results all Runs all tests ============================================================================ The BYTE Index ============== The purpose of this test is to provide a basic indicator of the performance of a Unix-like system; hence, multiple tests are used to test various aspects of the system's performance. These test results are then compared to the scores from a baseline system to produce an index value, which is generally easier to handle than the raw sores. The entire set of index values is then combined to make an overall index for the system. Since 1995, the baseline system has been "George", a SPARCstation 20-61 with 128 MB RAM, a SPARC Storage Array, and Solaris 2.3, whose ratings were set at 10.0. (So a system which scores 520 is 52 times faster than this machine.) Since the numbers are really only useful in a relative sense, there's no particular reason to update the base system, so for the sake of consistency it's probably best to leave it alone. George's scores are in the file "pgms/index.base"; this file is used to calculate the index scores for any particular run. Over the years, various changes have been made to the set of tests in the index. Although there is a desire for a consistent baseline, various tests have been determined to be misleading, and have been removed; and a few alternatives have been added. These changes are detailed in the README, and should be born in mind when looking at old scores. A number of tests are included in the benchmark suite which are not part of the index, for various reasons; these tests can of course be run manually. See "Tests" above. ============================================================================ Graphics Tests ============== As of version 5.1, UnixBench now contains some graphics benchmarks. These are intended to give a rough idea of the general graphics performance of a system. The graphics tests are in categories "2d" and "3d", so the index scores for these tests are separate from the basic system index. This seems like a sensible division, since the graphics performance of a system depends largely on the graphics adaptor. The tests currently consist of some 2D "x11perf" tests and "ubgears". * The 2D tests are a selection of the x11perf tests, using the host system's x11perf command (which must be installed and in the search path). Only a few of the x11perf tests are used, in the interests of completing a test run in a reasonable time; if you want to do detailed diagnosis of an X server or graphics chip, then use x11perf directly. * The 3D test is "ubgears", a modified version of the familiar "glxgears". This version runs for 5 seconds to "warm up", then performs a timed run and displays the average frames-per-second. On multi-CPU systems, the graphics tests will only run in single-processing mode. This is because the meaning of running two copies of a test at once is dubious; and the test windows tend to overlay each other, meaning that the window behind isn't actually doing any work. ============================================================================ Multiple CPUs ============= If your system has multiple CPUs, the default behaviour is to run the selected tests twice -- once with one copy of each test program running at a time, and once with N copies, where N is the number of CPUs. (You can override this with the "-c" option; see "Detailed Usage" above.) This is designed to allow you to assess: - the performance of your system when running a single task - the performance of your system when running multiple tasks - the gain from your system's implementation of parallel processing The results, however, need to be handled with care. Here are the results of two runs on a dual-processor system, one in single-processing mode, one dual-processing: Test Single Dual Gain -------------------- ------ ------ ---- Dhrystone 2 562.5 1110.3 97% Double Whetstone 320.0 640.4 100% Execl Throughput 450.4 880.3 95% File Copy 1024 759.4 595.9 -22% File Copy 256 535.8 438.8 -18% File Copy 4096 1261.8 1043.4 -17% Pipe Throughput 481.0 979.3 104% Pipe-based Switching 326.8 1229.0 276% Process Creation 917.2 1714.1 87% Shell Scripts (1) 1064.9 1566.3 47% Shell Scripts (8) 1567.7 1709.9 9% System Call Overhead 944.2 1445.5 53% -------------------- ------ ------ ---- Index Score: 678.2 1026.2 51% As expected, the heavily CPU-dependent tasks -- dhrystone, whetstone, execl, pipe throughput, process creation -- show close to 100% gain when running 2 copies in parallel. The Pipe-based Context Switching test measures context switching overhead by sending messages back and forth between 2 processes. I don't know why it shows such a huge gain with 2 copies (ie. 4 processes total) running, but it seems to be consistent on my system. I think this may be an issue with the SMP implementation. The System Call Overhead shows a lesser gain, presumably because it uses a lot of CPU time in single-threaded kernel code. The shell scripts test with 8 concurrent processes shows no gain -- because the test itself runs 8 scripts in parallel, it's already using both CPUs, even when the benchmark is run in single-stream mode. The same test with one process per copy shows a real gain. The filesystem throughput tests show a loss, instead of a gain, when multi-processing. That there's no gain is to be expected, since the tests are presumably constrained by the throughput of the I/O subsystem and the disk drive itself; the drop in performance is presumably down to the increased contention for resources, and perhaps greater disk head movement. So what tests should you use, how many copies should you run, and how should you interpret the results? Well, that's up to you, since it depends on what it is you're trying to measure. Implementation -------------- The multi-processing mode is implemented at the level of test iterations. During each iteration of a test, N slave processes are started using fork(). Each of these slaves executes the test program using fork() and exec(), reads and stores the entire output, times the run, and prints all the results to a pipe. The Run script reads the pipes for each of the slaves in turn to get the results and times. The scores are added, and the times averaged. The result is that each test program has N copies running at once. They should all finish at around the same time, since they run for constant time. If a test program itself starts off K multiple processes (as with the shell8 test), then the effect will be that there are N * K processes running at once. This is probably not very useful for testing multi-CPU performance. ============================================================================ The Language Setting ==================== The $LANG environment variable determines how programs abnd library routines interpret text. This can have a big impact on the test results. If $LANG is set to POSIX, or is left unset, text is treated as ASCII; if it is set to en_US.UTF-8, foir example, then text is treated as being encoded in UTF-8, which is more complex and therefore slower. Setting it to other languages can have varying results. To ensure consistency between test runs, the Run script now (as of version 5.1.1) sets $LANG to "en_US.utf8". This setting which is configured with the variable "$language". You should not change this if you want to share your results to allow comparisons between systems; however, you may want to change it to see how different language settings affect performance. Each test report now includes the language settings in use. The reported language is what is set in $LANG, and is not necessarily supported by the system; but we also report the character mapping and collation order which are actually in use (as reported by "locale"). ============================================================================ Interpreting the Results ======================== Interpreting the results of these tests is tricky, and totally depends on what you're trying to measure. For example, are you trying to measure how fast your CPU is? Or how good your compiler is? Because these tests are all recompiled using your host system's compiler, the performance of the compiler will inevitably impact the performance of the tests. Is this a problem? If you're choosing a system, you probably care about its overall speed, which may well depend on how good its compiler is; so including that in the test results may be the right answer. But you may want to ensure that the right compiler is used to build the tests. On the other hand, with the vast majority of Unix systems being x86 / PC compatibles, running Linux and the GNU C compiler, the results will tend to be more dependent on the hardware; but the versions of the compiler and OS can make a big difference. (I measured a 50% gain between SUSE 10.1 and OpenSUSE 10.2 on the same machine.) So you may want to make sure that all your test systems are running the same version of the OS; or at least publish the OS and compuiler versions with your results. Then again, it may be compiler performance that you're interested in. The C test is very dubious -- it tests the speed of compilation. If you're running the exact same compiler on each system, OK; but otherwise, the results should probably be discarded. A slower compilation doesn't say anything about the speed of your system, since the compiler may simply be spending more time to super-optimise the code, which would actually make it faster. This will be particularly true on architectures like IA-64 (Itanium etc.) where the compiler spends huge amounts of effort scheduling instructions to run in parallel, with a resultant significant gain in execution speed. Some tests are even more dubious in terms of host-dependency -- for example, the "dc" test uses the host's version of dc (a calculator program). The version of this which is available can make a huge difference to the score, which is why it's not in the index group. Read through the release notes for more on these kinds of issues. Another age-old issue is that of the benchmarks being too trivial to be meaningful. With compilers getting ever smarter, and performing more wide-ranging flow path analyses, the danger of parts of the benchmarks simply being optimised out of existance is always present. All in all, the "index" and "gindex" tests (see above) are designed to give a reasonable measure of overall system performance; but the results of any test run should always be used with care.