Validation Suite¶
Validating a processor model can mean different things: functional
(does each instruction do the right thing), timing or microarchitectural
(does it take the right number of cycles, in the right order), or both.
This project's SPARC suite, in validation/asm/ and validation/C/, is
purely functional. There are no SPARC timing tests here. Both models drive the exact same SparcCore (see
Models), so this one suite validates both, and
catches a regression in either.
It checks the core's instruction-level behavior against the SPARC V8 manual, in two ways:
validation/asm/, hand-written assembly, each test targeting one instruction (or a small family of closely related ones) in isolation.validation/C/, compiled from freestanding C, each test a small self-validating program (a sort, an FFT, a checksum) exercising many instructions together the way a real program would.
It consists of two things: the tests themselves (see "What's in a test"
below), and a script, validation/run_tests.py, that runs them all and
prints a pass/fail summary (see "The validation script" below).
Beside the SPARC suite, further suites test the components of the larger configurations: the MMU, the peripheral devices, the caches, and the APARA engine, both on its own (with timing tests too) and driven by a SPARC program. Each suite runs only on the configurations that have its components (see Which tests run on which configuration).
This page covers how a test is put together, how to run the suite, and how to add a new one. For actually writing a test program (the trap table, the pass-fail convention, structuring a self-validating C test), see Writing and Running Assembly Programs and Writing and Running C Programs.
What's in a test¶
Every test is a matched set of files sharing one name, for example
ADD.s, ADD.hex, ADD.objdump, ADD.vprj, and ADD.expected
(validation/asm/integer_alu/Arithmetic/Add/). Only two of these are
written by hand:
<name>.sor<name>.c, the source program.<name>.vprj, its expected final state, in the human-friendly format described below.
The rest are derived automatically, not hand-written (though .hex and
.objdump are still committed as build artifacts, see "The validation
script" below for why):
<name>.hex, the memory image compiled from the source, loadable directly by either model.<name>.objdump, its disassembly plus full symbol table, useful for finding an address to give gdb, see Examining Core State at Runtime Using GDB.<name>.expected, the.vprj's checks rewritten into the plain format the checker executables themselves take as a direct command-line argument, see Getting Started.
In short, .vprj is what you write, once, by hand. .expected is what
run_tests.py generates from it, fresh, every time the test runs. It's
never edited directly, and isn't committed to git. The one exception is
validation/test_simple_ADD/test_simple_ADD.expected, which is committed
so that Getting Started
works right after cloning.
The .vprj expected-results format¶
SOURCESnames the paired.s/.cfile.- Each
RESULTSline is either a register check (<name>=<hex value>, register mnemonicsg1-g7,o0-o7,l0-l7,i0-i7,f0-f31,psr,fpsr,y,wim,tbr,pc,npc,asr0-asr31) or a memory check (m[<hex addr>]=<hex value>, a word-aligned 32-bit read). - Either kind of line accepts an optional trailing mask
(
m[0x148] = 0x00000021 0x00000021checks only the bits set in the mask. Seevalidation/asm/floating_point/fp_exceptions/accrued_inexact.sfor a real example distinguishing individual FSR bits.) - On the
multicoreconfiguration a register check may name a core, as incore2.o0=0x00000001. A register check without a core prefix checks core 0. asi = ...lines are recognized and ignored.MemCoreis a single flat address space with no ASI distinction.
Before running a test, run_tests.py parses this and normalizes it into
the plain REG <name> <hex value> [mask] / MEM <addr> <hex value>
[mask] format, writing it to a .expected file alongside the test.
Optional lines, used by the device, APARA and multicore tests:
STARTUP = crt0_interrupts.s, beforeRESULTS, links that startup file instead ofcompiler/crt0.s(C tests only). Multicore tests useSTARTUP = crt0_multicore.s.COMPILE_OPTIONS = -fopenmp, beforeRESULTS, adds options to the compiler's command line (C tests only).MCODE = kernel.mcode, beforeRESULTS, names an APARA engine program the test carries, in mcode or in C (MCODE = kernel.c). Before compiling,build_hex.pyassembles or compiles it, and writes it out as a C header,kernel.mcode.h(orkernel.c.h), for the test to#include(C tests only, may be repeated). See Programming the System.MAX_CYCLES = 1000000, beforeRESULTS, raises the cycle limit for a test that legitimately runs long (the runner uses the larger of this and its--max-cycles).RUN_OPTIONS = --coherent, beforeRESULTS, adds options to the simulator's command line, for a test that only makes sense in one mode of a configuration.serial_lines = "text", on the multicore configuration, checks that the lines sent are these, in any order, for cores whose order of printing is not fixed.serial_out = "text", among the results, checks the exact bytes the program sent out through the Serial device (C-style escapes such as\nand\x41work). Only configurations with devices check it.
A test's received serial input, if it needs any, is a file named after
the test with the extension .serial_in, next to its .hex.
The validation script¶
A two-phase pipeline, so the existing suite can be run without a cross-compiler installed at all.
# phase 1 (needs the sparc-elf toolchain): assembles/compiles .s or .c
# into .hex, only needed when you add or edit a test's source
validation/build_hex.py validation
# phase 2: runs the already-built .hex against a model
validation/run_tests.py validation
[PASS] validation/asm/misc/save_restore/SAVE.vprj
[PASS] validation/asm/misc/stbar_unimp_nop_sethi/STBAR_UNIMP_NOP_SETHI.vprj
<passed>/<total> tests passed
Since .hex files are committed to git, phase 2 alone is enough to run
the existing suite. Phase 1 is only needed after adding or editing a
.s/.c source, and also (re)generates that test's .objdump.
You can point run_tests.py at any subset too, not just the
validation root:
It recurses to find .vprj files, so nesting is free. Tests the
configuration cannot run are skipped, and listed at the end (see Which
tests run on which
configuration).
Options¶
| Option | Does |
|---|---|
--config NAME |
Which system_models configuration to run against, default core_only. E.g. --config core_mmu runs against model/system_models/core_mmu/'s own executables instead, and runs only the tests core_mmu can run. |
--sitar |
Run against the chosen configuration's Sitar-timed model instead of its plain C++ model (the default). Same CLI, same .vprj format, same PASS/FAIL/OVERALL output either way. See Getting Started to build it. |
--max-cycles N |
Per-test cycle limit, default 200,000. A test that hasn't halted by then is reported as a failure, rather than hanging forever on a genuine bug. A "cycle" means one complete instruction executed against the plain C++ model, and an actual clock cycle against the Sitar-timed model, see Getting Started. |
-v |
Show every check's result, not just failures. |
validation/run_tests.py validation --config core_mmu --sitar -v
validation/run_tests.py validation/asm/control_transfer --max-cycles 50000
validation/clean.sh [folder] removes generated build byproducts (.o,
.elf, .expected) without touching .hex/.s/.c/.vprj/.objdump.
Which tests run on which configuration¶
Not every test can run on every configuration: the MMU tests need an
MMU, the APARA tests an engine. Two small files decide which tests a
configuration runs, so that you can always point run_tests.py at the
whole of validation/:
COMPONENTS, in each configuration's folder (model/system_models/<config>/COMPONENTS), lists what the configuration contains.REQUIRES, in a suite's folder, lists what its tests need. Requirements add up down the tree: a test needs everything its own folder's and every parent folder'sREQUIRESfile lists.
run_tests.py --config <config> runs the tests whose requirements the
configuration has, and lists the folders it skipped, with the reason:
skipped 13 test(s) under validation/C/caches: core_only has no caches
skipped 9 test(s) under validation/C/devices: core_only has no devices
skipped 16 test(s) under validation/C/mmu: core_only has no mmu
skipped 108 test(s) under validation/apara: core_only has no apara_standalone
skipped 78 test(s) under validation/multicore: core_only has no devices, multicore
skipped 26 test(s) under validation/sparc_apara: core_only has no apara, devices
241/241 tests passed (250 skipped)
The components, and which configuration has which:
| Configuration | COMPONENTS |
|---|---|
core_only |
sparc |
core_mmu |
sparc mmu |
core_mmu_devices |
sparc mmu devices |
core_l1cache_mmu_devices |
sparc mmu devices caches |
sparc_apara |
sparc mmu devices caches apara |
apara |
apara_standalone (the engine alone, no SPARC core) |
And the suites, with what they require, and where they run:
| Folder | REQUIRES |
Runs on |
|---|---|---|
asm/, C/ (the benchmarks), test_simple_ADD/ |
sparc |
every SPARC configuration |
C/mmu/ |
sparc mmu |
core_mmu and later |
C/devices/ |
sparc devices |
core_mmu_devices and later |
C/caches/ |
sparc caches |
core_l1cache_mmu_devices, sparc_apara |
sparc_apara/ |
sparc devices apara |
sparc_apara |
apara/ |
apara_standalone |
apara, with its own runner, validation/apara/run_tests.py |
The engine's own suites, in apara/, are engine programs rather than
SPARC programs, so they have their own format and runner (see APARA
Validation Suite). run_tests.py
--config apara says so, and stops.
A new configuration needs only its COMPONENTS file to work with every
existing suite. A new suite that needs more than the SPARC core gets a
REQUIRES file.
The test collection¶
The SPARC suite has two sub-suites, side by side under validation/,
both using the same .vprj format and the same two-phase pipeline
above. The component suites follow in the next section.
asm/
Hand-written assembly tests, each targeting one instruction (or a small family of closely related ones, such as every branch condition) in isolation: integer ALU ops, control transfer (branches, traps, call, jump,rett), loads and stores (including atomic and coprocessor variants), and floating point. Organized into category subfolders:integer_alu/,floating_point/,control_transfer/,data_transfer/,misc/.compiler/assemble.shbuilds a test's.sinto.hex/.objdump, called automatically bybuild_hex.pyabove.-
C/
Self-validating bare-metal C mini-benchmarks, each compiled program exercising a sequence of C-level operations together (loops, arrays, structs, global variables) the way a real program would, rather than one instruction in isolation.compiler/compile_c.shbuilds a test's.cthe same way. Every test here follows the same shape: compute something, compare it against a golden value computed once on the host machine and hardcoded right there in the source, and report pass (1) or fail (0) in%o0. Its.vprjthen only ever has to checko0=1, instead of re-embedding the golden value a second time. See Writing and Running C Programs for the full convention, andarray_sum/array_sum.cfor a worked example.Test What it computes array_sumSum of a small int array matrix_mul3x3 integer matrix multiply fft4-point integer radix-2 FFT root_findingInteger square root via Newton-Raphson integer_sortBubble sort gcdEuclidean algorithm fibonacciIterative Fibonacci prime_sieveSieve of Eratosthenes checksumByte-array checksum dot_productInteger vector dot product
Alongside them sits test_simple_ADD/, one more .vprj test, but its
main job is as the small worked example used throughout the docs (every
configuration's run_simple_test.sh reads the files here directly, and
log_viewer/ symlinks to the reference trace produced here, rather than
keeping their own copies), see validation/README.md.
Adding an asm test¶
- Pick an existing category folder under
validation/asm/(integer_alu/,floating_point/,control_transfer/,data_transfer/,misc/), or add a new subfolder for a new, focused group of tests. - Write
<name>.s. See Writing and Running Assembly Programs for the full format: the required setup, the trap table, and the pass-fail convention every test shares. - Build and run it directly first, to find the actual final register
values, see
Assembling to generate a memory image
(
compiler/assemble.sh <name>.s, thenmodel/system_models/core_only/cpp_model/executable/sparc_sim_cpp_core_only <name>.hexwith no expected-results argument, prints the final state). - Write
<name>.vprj, naming<name>.sas itsSOURCESand listing the register (and, if needed, memory) values from step 3 that you want checked, see the format above. - Run
validation/build_hex.py <folder>to (re)generate<name>.hexand<name>.objdumpthrough the suite's own tooling (build_hex.pyfinds a folder's.vprjfiles and readsSOURCESto know what to build, so this step needs the.vprjto already exist, from step 4). - Run
validation/run_tests.py <folder>to check it passes.
No registration step anywhere else, run_tests.py and build_hex.py
both recurse automatically.
Adding a C test¶
- Add a new folder under
validation/C/, one per test. - Write
<name>.c, freestanding (no libc, see Writing and Running C Programs), following the same self-validating shape as the existing benchmarks: compute something, compare it against a golden value computed on the host, and report pass (1) or fail (0) in%o0. - Write
<name>.vprj, naming<name>.cas itsSOURCES. Since the test already computed its own pass/fail, itsRESULTSis almost always justo0=1. - Run
validation/build_hex.py <folder>to generate<name>.hexand<name>.objdumpfrom it (pickingcompiler/compile_c.shautomatically, based on the.cextension). As with an asm test, this needs the.vprjto already exist, sincebuild_hex.pyreads itsSOURCESline to know what to build. - Run
validation/run_tests.py <folder>to check it passes.
Component validation suites¶
The suite above is purely functional, and purely for the SPARC core itself: every opcode, exercised in isolation or as part of a larger C program. Alongside it, dedicated test suites validate individual components, each living in its own folder:
validation/C/mmu/
The MMU: register access, page-table walks, TLB hit/miss/fill, faults (including the trap a faulting store must take), probe and flush, bypass. See Model Components.validation/C/devices/
The timer, interrupt controller (including interrupt delivery to a real trap handler, priority, masking, and IPIs), and serial device (output, polled and interrupt-driven input). Needscore_mmu_devices. See Peripheral Devices. On the sitar_model these tests need more cycles than the default limit, so run them with--max-cycles 5000000.validation/C/caches/
The L1 caches: write-through, staleness until a flush, device registers never cached, atomics on a cached line, and synonyms, in every combination of virtual address and context, with the MMU on. Needscore_l1cache_mmu_devices. The whole suite above also runs through the caches on that configuration, and with every ordinary access cached if you setSPARC_ALWAYS_CACHEABLE=1in the environment.validation/multicore/
Programs that run on several cores at once: startup of every core, the run ending only once every core has halted, interrupts to chosen cores, OpenMP programs, and three worked examples. Needsmulticore, built with its default four cores. See the multicore Validation Suite page.validation/sparc_apara/
An APARA engine driven from a SPARC program, in five suites:basic/(each part of the engine's interface),setup/(bringing the engine up, and complete offloads by polling and by interrupt),threads/(one to four engine threads),interference/(whether threads, the SPARC and the DMA unit disturb one another), andlinalg/(linear algebra offloaded to the engine). Needssparc_apara. On the sitar_model, run them with--max-cycles 50000000. See APARA Validation Suite.validation/apara/
The APARA engine on its own, with its own test format and runner, in nine suites: every instruction (isa/), sequences, hazards, the branch predictor, programs the assembler must reject, timing, hand-written kernels, programs compiled from C, and the model's deviations. Unlike the SPARC suite, it also checks timing: cycle and mispredict counts, derived by hand for the default timing settings, which the timing model must match exactly. See APARA Validation Suite.
For example, to run the MMU validation suite on core_mmu, build that
configuration first (see Getting
Started):
then run the suite against it:
This same suite runs unmodified on any other configuration that also
includes an MMU, such as core_mmu_devices. Without --config, the
runner uses core_only, which has no MMU, and skips the whole suite.
Credits: validation suite of the AJIT SPARC V8 processor¶
Most of validation/asm/'s tests, and the scripts that build and run
this suite, are adapted from the AJIT processor project (IIT
Bombay), specifically its ajit32 instruction-level verification suite.
Many more tests have since been added as part of this project (the
quad-precision suite, the validation/C/ benchmarks, and the MMU suite
above, none of which exist in AJIT's own tests). Tests adapted directly
from AJIT carry their original author credit inline, and are listed as
such in validation/asm/README.md and Authors. Tests
developed as part of this project are not AJIT-derived.
A handful of specific, well-understood divergences between this model's
behavior and AJIT's own reference results are documented, with
side-by-side expected-vs-actual data, in docs/compliance/README.md,
rather than silently patched or dropped. These are kept separate from
the maintained suite (docs/compliance/, not validation/), since they
represent understood implementation differences from AJIT's own
hardware, not bugs in this model.