APARA Validation Suite¶
The APARA engine has two sets of tests. The engine suites test the
engine on its own, in the apara configuration. The system suites
test a SPARC program using the engine, in the sparc_apara
configuration. This page describes what each set contains, how to run
it on both models, and how to add a test. The general format of the
SPARC tests, and the runner validation/run_tests.py, are described in
Validation Suite.
| Set | Folder | Configuration | Runner | Tests |
|---|---|---|---|---|
| Engine suites | validation/apara/ |
apara |
validation/apara/run_tests.py |
108 |
| System suites | validation/sparc_apara/ |
sparc_apara |
validation/run_tests.py --config sparc_apara |
26 |
The commands on this page use $SPARC_HOME, the repository's folder,
as set in step 2 of Installation.
Before running the tests¶
Each runner runs the plain build of its configuration's models. The following commands build the C++ model and the Sitar model of both configurations:
cd $SPARC_HOME/model/system_models/apara/cpp_model
./build.sh
cd $SPARC_HOME/model/system_models/apara/sitar_model
./build.sh
cd $SPARC_HOME/model/system_models/sparc_apara/cpp_model
./build.sh
cd $SPARC_HOME/model/system_models/sparc_apara/sitar_model
./build.sh
The Sitar models need Sitar (step 3 of Installation). The executables of
apara are executable/apara_sim_cpp and executable/apara_sim_sitar,
and those of sparc_apara are executable/sparc_sim_cpp_sparc_apara
and executable/sparc_sim_sitar_sparc_apara.
No cross-toolchain is needed to run the tests. The engine suites are assembled by the runner itself, with the engine's assembler, which is a Python program. The system suites' memory images are already built and committed. The cross-toolchain is needed only to rebuild a system test after a change to its sources (see Rebuilding the memory images).
The engine suites (validation/apara)¶
What they contain¶
validation/apara/ holds the engine's own test suites, one folder each:
| Folder | Tests | What they check |
|---|---|---|
isa/ |
29 | Every instruction, in one subfolder per class (alu_int, alu_float, vector, cast, bits, cmov, memory, control). The tests cover each operation on each type, edge values (wraparound, sign extension, rounding), immediates and increments at their limits, and every exception. |
sequences/ |
7 | Instructions working together. Examples are full 8-slot bundles, loops, a recurrence in one bundle per step, a block copy with auto-increments, threads sharing DMEM, and one thread's fault not stopping the others. |
hazards/ |
10 | The pipeline's handling of dependences between bundles (see Hazards). |
predictor/ |
6 | The branch predictor, with loops, nested loops, two branches sharing a predictor entry, and calls and returns. Each test's mispredict count is derived from the predictor's rules. |
asm_errors/ |
21 | Programs the assembler must reject. Examples are a register written twice in a bundle (also through multi-register writes and increments), too many control transfers, divides or memory operations in a bundle, values out of range, and invalid types. |
timing/ |
10 | Microbenchmarks with exact cycle counts, derived by hand from the timing rules. Its README.md also has each test's counts for other settings of the knobs. |
kernels/ |
6 | Larger hand-scheduled programs. These are a dot product, a vector addition, SAXPY, a reduction, and matrix products on one and four threads. |
c_programs/ |
11 | Programs compiled from C with the C compiler. They cover the language's features, sorting, a prime sieve, and linear algebra. |
deviations/ |
8 | Behavior where the model departs, on purpose, from the engine's reference model (see Deviations and Findings). |
Each folder has a README.md describing its tests. A good way to learn
the instruction set is to read the tests of isa/ next to the
Instruction Set, and then the programs of kernels/.
A test¶
A test is a source file and a .vprj file of the same name. The
.vprj says how to run the program and what to expect, in the same
spirit as the SPARC tests. For example, the timing test
timing/t3_latency is the following program:
// Timing: EXEC latency. An f32 add takes 3 cycles in EXEC (cycles 3-5).
// The $halt behind it enters EXEC in cycle 4 but results leave in order,
// one per cycle: the add's in cycle 5, the halt's in 6; WRITEBACK in 8:
// 9 cycles.
//
|| + $r1 ($f32) $r2 $r3 ;
|| $halt ;
Its .vprj gives the exact cycle count that the timing model must
reach:
The source's header explains where the expected count comes from. A
functional test sets registers and memory in INIT, and lists the
expected results in RESULTS. The following lines are from
isa/control/control.vprj:
The lines of INIT and RESULTS have the following forms:
| Line | Meaning |
|---|---|
r3 = 0x10 |
Thread 0's $r3. |
t1.r3 = 0x10 |
Thread 1's $r3. |
m[0x80] = 0x0102030405060708 |
The DMEM dword (8 bytes) at byte address 0x80. |
m[C+0x8] = 0x2f2 |
In a C test, a dword of the global C. |
t1.pc = 0x9 |
(RESULTS only) Where thread 1 stopped. |
error = 0x2000000 |
(RESULTS only) Thread 0's ERROR register. The value 0x2000000 is a $halt. |
A value in RESULTS may be followed by a mask, which selects the bits
that are compared.
The other keys are the following:
| Key | Meaning |
|---|---|
SOURCES = prog.mcode |
The program. It may be mcode, C (.c, compiled first), or an object file (.obj). |
THREADS = 0x0 worker |
Start a thread at each of these IMEM addresses or labels. The default is one thread at 0x0. |
CYCLES = 12 |
The exact cycle count on the timing model, with the default timing settings. |
MISPREDICTS = 2, BTB_PREDICTIONS = 3 |
The exact branch predictor counts on the timing model. |
HAZARDS_UNLOCKED = 1 |
The hazards that the timing model must count when built without the interlock (see Hazards). |
ASM_ERROR = written twice |
The program must not assemble, and the error message must contain this text. |
DEVIATION = <why> |
The test depends on one of the model's deviations. |
MAX_CYCLES = 100000 |
A larger limit, for a long test. |
APCC_FLAGS = -O0 |
Options for the C compiler. |
Running the tests¶
The engine suites have their own runner, validation/apara/run_tests.py.
The following commands run every suite, first on the C++ model and then
on the Sitar model:
For each test, the runner assembles or compiles the program, writes its
files into validation/apara/build/<test>/, runs it, and prints PASS
or FAIL. It then prints a summary for each suite:
PASS timing/t1_halt.vprj
...
asm_errors 21/21 passed
c_programs 11/11 passed
deviations 8/8 passed
hazards 10/10 passed
isa 29/29 passed
kernels 6/6 passed
predictor 6/6 passed
sequences 7/7 passed
timing 10/10 passed
OVERALL: 108 passed, 0 failed
With --sitar, the runner also checks the timing keys CYCLES,
MISPREDICTS, BTB_PREDICTIONS and HAZARDS_UNLOCKED. The runner can
also be given one suite, or one .vprj:
cd $SPARC_HOME
python3 validation/apara/run_tests.py validation/apara/hazards --sitar
python3 validation/apara/run_tests.py validation/apara/timing/t3_latency.vprj --sitar
A failing test can be run again by hand, with the files the runner left
in validation/apara/build/<test>/, and the options of Running a
program, such as --trace.
Adding a test¶
To add a test, write a source file and its .vprj in a suite's folder,
and run the suite. Nothing needs to be registered, because the runner
finds every .vprj file in the folders it is given. To find the
expected values of a new test, run it once with --dump-results - (see
Running a program). Then check every
value by hand, or with an independent calculation, before writing it
down.
The system suites (validation/sparc_apara)¶
What they contain¶
validation/sparc_apara/ holds the tests of the SPARC using the engine.
Each test is a bare-metal SPARC C program, with one or more engine
programs. The tests are grouped in five folders:
| Folder | Tests | What they check |
|---|---|---|
basic/ |
8 | Each part of the engine's interface on its own. The tests cover IMEM and registers, running and polling, the completion interrupt, exceptions, four threads, the engine's own main-memory accesses, DMA and its interrupt, and a whole offload by DMA (dma_dot_product). |
setup/ |
5 | Bringing the engine up, with one thread. The tests cover its state at power-up, loading and reading back its memories, complete offloads of a program compiled from C (by polling and by interrupt), and loading a second program over the first. |
threads/ |
5 | The same program on 1, 2, 3 and 4 threads, each on its own slice of the data, and threads started and restarted at different times. |
interference/ |
5 | Whether threads, the SPARC and the DMA unit disturb one another (see What may happen while threads run), and the one real interference, a race on shared data. |
linalg/ |
3 | Linear algebra offloaded to the engine. The tests are an 8x8 matrix product on four threads, SAXPY in floating point, and a four-thread dot product compiled from C. |
Each folder has a README.md describing its tests, and
validation/sparc_apara/README.md summarizes the protocol of
Setting up and running the
engine. The
tests also serve as worked examples for new programs, as listed in
Going further.
A test¶
A system test is a folder with a SPARC C program, its engine programs,
and a .vprj file. The folder basic/dma_dot_product/ holds the
following sources:
| File | What |
|---|---|
dma_dot_product.c |
The SPARC program. |
kernel.mcode |
The engine program. |
dma_dot_product.vprj |
How to build the test, and what to expect. |
Its .vprj names the SPARC program, its engine program, and the
expected registers:
The line MCODE = kernel.mcode names an engine program. The build
script writes its C header, kernel.mcode.h, before it compiles the
SPARC program. A test may have several MCODE lines, and an engine
program may also be written in C (MCODE = kernel.c). The line
STARTUP = crt0_interrupts.s links the interrupt startup code, for a
test that takes the engine's interrupt. The other files in the folder,
such as dma_dot_product.hex and kernel.mcode.h, are written by the
build, and are committed so that the tests run without a
cross-toolchain.
Every test reports in the same way as the device tests. %o0 is 1 if
every check passed, and %o1 has one bit per check. The checks are
listed in the header comment of the test's C file. The SPARC program
computes the expected results itself.
The tests share some helpers, in validation/sparc_apara/apara_common.h:
| Helper | Does |
|---|---|
report_and_halt(ok, n) |
Writes 1 to %o0 if all n checks passed, and 0 otherwise, writes ok to %o1, and halts. |
physical_word(p) |
Reads a word of main memory with an MMU-bypass load, past the data cache, for data that the engine has written there. |
dword_equals(o, v) |
Whether the 64-bit word at DMEM offset o equals the 32-bit value v, sign-extended. |
next_random(&state) |
The next number of a fixed pseudo-random sequence, for test data. |
enable_fpu() |
Turns on the SPARC's FPU, which the startup code leaves off. |
HALTED |
The completion word of a thread that ended with $halt. |
Running the tests¶
The system suites run with the general runner, on --config sparc_apara.
The Sitar model needs a larger cycle limit than the default:
cd $SPARC_HOME
python3 validation/run_tests.py validation/sparc_apara --config sparc_apara
python3 validation/run_tests.py validation/sparc_apara --config sparc_apara --sitar --max-cycles 50000000
[PASS] validation/sparc_apara/basic/dma_dot_product/dma_dot_product.vprj
...
[PASS] validation/sparc_apara/threads/threads_4/threads_4.vprj
26/26 tests passed
One folder, or one test's folder, can be given instead:
cd $SPARC_HOME
python3 validation/run_tests.py validation/sparc_apara/basic --config sparc_apara
python3 validation/run_tests.py validation/sparc_apara/basic/dma_dot_product --config sparc_apara
A test's memory image can also be run by hand, as in An example: offloading a dot product.
Rebuilding the memory images¶
This section needs the cross-toolchain (step 4 of Installation).
After a change to a test's C program or engine program, the following command rebuilds the memory images of a folder of tests. It writes the engine programs' headers, then compiles each SPARC program:
The runner then runs the new images. Rebuilt .hex files should be
committed together with the changed sources.
Adding a test¶
To add a test, make a new folder in one of the five folders, with the
SPARC program, its engine programs, and a .vprj that names them. Then
build it and run it:
cd $SPARC_HOME
python3 validation/build_hex.py validation/sparc_apara/basic/my_test
python3 validation/run_tests.py validation/sparc_apara/basic/my_test --config sparc_apara
Writing a new program
builds such a test step by step. The file validation/sparc_apara/REQUIRES
names the components that these tests need, which are the SPARC core,
the devices and the engine. The runner therefore runs a new test in
validation/sparc_apara/ only on the sparc_apara configuration (see
Which tests run on which
configuration).
The whole suite on sparc_apara¶
The sparc_apara configuration also runs the general SPARC tests of
validation/, because its SPARC core is the one of
core_l1cache_mmu_devices. The following command runs every test that
the configuration can run, on the C++ model:
skipped 108 test(s) under validation/apara: sparc_apara has no apara_standalone
291/291 tests passed (108 skipped)
The 291 tests are the SPARC tests and the 26 system tests. The runner
skips the 108 engine tests of validation/apara/, which only their own
runner runs.