Skip to content

Validation Suite

validation/multicore/ holds the programs that run on several cores at once. They assume the default build, with four cores. This page lists them, shows how each core's results are checked, and how to run the suite on both models. See Validation Suite for the test pipeline that every suite shares.


Checking each core's results

Every core reports for itself, as the device tests do on one core. %o0 is 1 if every one of the core's checks passed, and %o1 has one bit per check. A test's .vprj names a core by prefixing a register with core<i>.. A register line without a prefix checks core 0, so every single-core test keeps working.

RESULTS =
o0=0x00000001
core1.o0=0x00000001
core2.o0=0x00000001
core3.o0=0x00000001
m[0x100000]=0x000000a0

Memory checks (m[...]) read main memory directly, which every core shares. The run ends once every core has halted, and a core that never halts makes the test fail when the cycle limit is reached.

The C tests start from crt0_multicore.s (their .vprj's STARTUP line) and include multicore_common.h (see Routines provided). The tests of the synchronization library also include multicore_sync.h.


The tests

Test Checks
boot/release_flag_asm In assembly, without the startup file. Cores 1 to 3 wait on a release flag in the scratch-pad, set by core 0, then each core writes its own word of memory. Each core's ASR29 is checked too.
boot/crt0_multicore What the startup file promises every core: its ASR29 value, its own stack region, .bss cleared, and the release flag set. Each core records its arrival in shared memory.
boot/staggered_halt Core i counts to (i + 1) * 3000 before halting, so the cores halt far apart, and the run still waits for all of them.
boot/deep_calls_mmu_on A call chain deeper than the register windows, with the MMU on. A window overflow writes above the stack pointer, so the startup file must leave room above its first one.
interrupts/ipi_from_core0 Core 0 sends an inter-processor interrupt to each of cores 1 to 3 in turn. Each takes exactly one, with the right message, and core 0 takes none.
interrupts/timer_to_selected_cores Cores 0 to 2 enable the timer's level and core 3 does not. Cores 0 to 2 each take exactly one timer interrupt, and core 3 takes none.
interrupts/ipi_ping_pong Two pairs of cores pass an IPI back and forth at the same time. Every IPI arrives, from the right core, with the right pointer.
interrupts/ipi_all_to_all Every core sends to every other core at once. All arrive, with message headers and pointers that agree, and each sender's in order.
devices/console_under_lock Every core prints lines at once. They come out whole, in any order.
devices/scratch_pad_across_cores A write by one core is seen by the others, also through the alias, and byte stores by different cores to one word all take effect.
examples/parallel_sum The parallel array sum example. Checks each core's partial sum and the total.
examples/parallel_matmul The parallel matrix multiply example. Checks every element of the product.

Atomic instructions

The tests in atomics/ check LDSTUB and SWAP themselves across cores, with bare instruction loops rather than the library routines. Each is built so that a broken instruction shows up as a wrong final result, however the cores interleave.

Test Checks
atomics/ldstub_counter Every core adds to one counter 100 times under a lock taken with LDSTUB. The total must be exact, or two cores were inside at once.
atomics/ldstub_counter_asm The same test in assembly, without the startup file or the library.
atomics/swap_counter The same with a lock taken with SWAP.
atomics/ldstub_one_winner Every core does LDSTUB on each byte of a row. Exactly one core can read 0 from each byte, so the wins must add up to the number of bytes.
atomics/swap_keeps_every_value Five different values are held by the cores and one shared word, and the cores keep swapping them. Each value must still exist exactly once at the end.
atomics/ldstub_neighbour_bytes Each core uses LDSTUB on its own byte of one shared word, so a model that wrote back more than that byte would disturb a neighbour.
atomics/mixed_instructions One lock, taken with LDSTUB, SWAP, LDSTUBA and SWAPA by different cores. All four must exclude each other.
atomics/lock_in_scratchpad Locks in a device, the scratch-pad, rather than in main memory.
atomics/unmapped_address Atomic instructions to an address that maps to nothing must not leave the bus locked, while other cores carry on.
atomics/counter_without_lock The program ldstub_counter with no lock, showing additions being lost. The total depends on the interleaving, so only its limits are checked.

The synchronization library

The tests in sync_library/ check the routines of multicore_sync.h.

Test Checks
sync_library/mutex_routines The AJIT-named mutexes on two counters, and that trying a mutex this core already holds fails.
sync_library/fetch_and_add Every value of an atomic counter is handed out exactly once.
sync_library/ticket_lock_order A log kept under a ticket lock must read 0, 1, 2 and so on, so cores were served in ticket order.
sync_library/barrier_phases Cores doing different amounts of work meet at one reused barrier. After each pass, every core must have reached that phase.
sync_library/semaphore_limit A semaphore with a count of 2 never lets more than 2 cores in.
sync_library/flush_inside_lock A counter under a mutex, read with a plain load after a flush inside the lock. This matters when run with the caches on.

Coherence

The tests in coherence/ run with coherence on, through a RUN_OPTIONS line in each .vprj, and each core turns caching on for itself. They read shared data with plain cached loads. Each reader first brings the data into its cache, because a line that was never cached proves nothing, since a miss reads memory anyway.

Test Checks
coherence/write_seen_by_plain_load After a core writes a word the others hold in their caches, a plain load sees the new value.
coherence/stale_without_coherence The same program with coherence off. The others go on reading the old value until they flush, which shows the first test really tests something.
coherence/lock_and_counter_with_plain_loads A spinlock and a counter using only plain loads, with an exact total.
coherence/every_kind_of_write A byte store, a doubleword store, LDSTUB, SWAP and a bypass store all remove other cores' copies.
coherence/instruction_cache Other cores run code a core has changed, with no flush, and the writer runs it after FLUSH.
coherence/false_sharing Cores writing different words of one line never see a wrong value.
coherence/fill_race A write during another core's line fill must not leave a stale line. See Coherence.
coherence/snoop_filter With the snoop filter on, invalidates are still delivered to a core that has fetched the line again.

The litmus tests have the cores do a few reads and writes at the same moment, 100 times over with varying delays, and check that an outcome memory ordering forbids never happens.

Test The cores do Forbidden
coherence/litmus_mp Core 0 writes data and then a flag, and others read the flag and then the data. Seeing the flag but not the data.
coherence/litmus_sb Two cores each write one word and read the other. Both reading 0. TSO allows this, but this model has no store buffer.
coherence/litmus_lb Two cores each read one word and write the other. Both reading 1.
coherence/litmus_iriw Two cores write different words, and two others read them in opposite orders. The readers disagreeing about the order of the writes.
coherence/litmus_corr Core 0 writes a word twice, and others read it twice. Seeing the writes go backwards.
coherence/litmus_2plus2w Two cores each write both words, in opposite orders. A final state that needs a cycle in the order of writes.

With the MMU on

In coherence/ the MMU is off, so a virtual and a physical address are the same. The tests in mmu_coherence/ give each core its own view of a shared physical page, so an invalidate has to find its line from the physical address alone. They run with --coherent, and everything is cached.

Test Virtual address Context What else
mmu_coherence/diff_va_across_cores Different per core Shared Each in a different cache set. Every kind of write.
mmu_coherence/same_va_diff_context Same Different per core The tags all differ. Every kind of write.
mmu_coherence/diff_va_diff_context Different per core Different per core Every kind of write.
mmu_coherence/separate_page_tables Different per core A context table per core Every kind of write.
mmu_coherence/same_va_diff_pa Same, different physical pages Different per core Not synonyms, so a write must not disturb another core's line.
mmu_coherence/line_in_another_context Same A reader switches away and back An invalidate goes by physical address, whatever context is running. Runs with two ways, so that the second context's line does not evict the first.
mmu_coherence/code_through_another_address Different per core Shared The instruction cache.
mmu_coherence/lock_through_aliases Different per core Different per core A spinlock and a counter, with plain loads.
mmu_coherence/fill_race Different per core Shared A write during a fill.
mmu_coherence/litmus_mp Different per core Shared Message passing.
mmu_coherence/pte_write_race Same Shared Core 0 rewrites a page table entry while other cores walk to it. The R update must not undo the write.
mmu_coherence/pte_update_through_stale_tlb Same Shared A store through a TLB entry that is out of date must set the M bit in the entry in memory, not write the old copy back.
mmu_coherence/pte_keeps_both_bits Same Shared One core stores and another loads, each walking at once. The entry ends with both R and M set.

On one core, validation/C/caches/synonyms/ covers the same combinations, with the MMU on and on every configuration with caches. Each test maps one physical page at two addresses, and writes through one with a store that misses, a store that hits, a byte store, LDSTUB and SWAP, and reads through the other. The test faulting_access_keeps_line covers a store or atomic that faults with the MMU's NF bit set. Such a fault looks like a store to an uncached page, but it has no physical address, so it must not remove a line.

TLB shootdown

A shootdown needs the page table setups above, with an IPI to each core and an answer from each. tlb_shootdown/ tests tlb_shootdown_page().

Test Checks
tlb_shootdown/stale_without_shootdown Why it is needed. After core 0 changes an entry, a core that empties its caches still reads the old page, until it drops its TLB entry.
tlb_shootdown/remap_page Core 0 keeps moving a page between two physical pages. Each round, every core reads the page chosen.
tlb_shootdown/revoke_write Write permission taken away. A store by every core faults, and does not reach memory.
tlb_shootdown/from_every_core Every core moves a page of its own at once, so shootdowns overlap.
tlb_shootdown/other_context The other cores are running in another context when the entry changes.
tlb_shootdown/waits_for_every_core The other cores hold their interrupts off for a while. The shootdown must not return before they have answered.
tlb_shootdown/core_that_does_not_answer One core holds its interrupts off until two shootdowns have returned. Each must return anyway, and report that core, though the second cannot even send it an IPI.

OpenMP

openmp/ tests the OpenMP runtime with real pragmas, through programs that also build with the host's gcc -fopenmp. The script check_on_host.py in that folder runs the eleven that fit on a host, with four threads, and compares their results with what the .vprj files expect. The four it skips depend on this system. They count its cores, end the run with an error code, run on one core, or print on its console.

Test Checks
openmp/parallel_team Team sizes through num_threads, if and omp_set_num_threads(), the data clauses, many regions in a row, and the settings a worker takes from the thread that started the region.
openmp/loop_schedules Every schedule, on its own and combined with parallel, also collapse, loops with no iterations, and the size of a guided loop's first chunk. Every iteration runs exactly once.
openmp/ordered_loops ordered blocks run in the loop's order under every schedule, also when a chunk skips the block and finishes before an older one.
openmp/sync_constructs critical, named and unnamed, atomic, reduction, barrier, master and flush.
openmp/single_and_nowait single and copyprivate, and one thread held back until the others have filled the runtime's ring of work-shares.
openmp/sections Each section runs once, in any size of team, also with lastprivate and nowait.
openmp/locks omp_ locks and nested locks, also the owner of a nested lock.
openmp/nested_and_orphaned A nested region has a team of one, and work-sharing pragmas work outside the function holding the region.
openmp/nested_too_deep Nesting past the limit ends the run with the code 0xBAD0E571.
openmp/tasks Tasks run, with their own copies of firstprivate data, and taskwait finishes.
openmp/threadprivate Each thread has its own copy, initialized, and copyin works.
openmp/floating_point Every core can use the FPU in a region.
openmp/on_one_core With --run-cores 1 a program finishes, with teams of one.
openmp/num_procs omp_get_num_procs() finds all four cores, and a program's return value reaches %o0.
openmp/console_in_a_region The console routines work from inside a region.
examples/openmp_matmul The worked example, checked against the same product done by core 0 alone.

A hang in the runtime shows up as a run reaching its cycle limit, so these tests are the exception to the bounded waits described below.

Checking that the tests can fail

A test that cannot fail proves nothing, so these were also run on a sitar_model with the bus lock turned off. Eight of the ten atomics/ tests and five of the six sync_library/ tests then fail. counter_without_lock and ldstub_neighbour_bytes pass, as they should, since neither relies on the lock. semaphore_limit also passes in that run, so it is not a sensitive test of the lock underneath it, but it does fail when the semaphore itself is removed. flush_inside_lock fails when its flush is removed and the caches are on, and unmapped_address fails if an access to an unmapped address leaves the bus locked for good.

The coherence/ tests were checked the same way, with deliberate mistakes in the model.

Mistake Tests that then fail
No invalidates sent All but stale_without_coherence, and all of mmu_coherence/ but same_va_diff_pa, which needs none, in both models.
A fill in progress installs its line regardless fill_race and litmus_corr, on the sitar_model only, since the cpp_model has no such race.
The snoop filter never forgets snoop_filter, in both models.
The instruction cache is not invalidated instruction_cache, in both models.
An older copy of a line is not removed on a fill code_aliases and three others in C/caches/synonyms/, in both models.
The search for a line looks only in the set its physical address names Most of C/caches/synonyms/ and mmu_coherence/, in both models.
A line keeps its virtual address in place of its physical address Most of C/caches/synonyms/ and mmu_coherence/, in both models.
An atomic does not remove a copy under another address Six of C/caches/synonyms/, in both models.
A store through an uncached alias does not remove the cached copy uncached_alias, in both models.
A fault suppressed by the NF bit removes a line, as an uncached store does faulting_access_keeps_line, in both models.
An invalidate clears the whole set, not just the matching line same_va_diff_pa, in both models.
Tags ignore the context diff_pa_same_va and line_in_another_context, in both models.
An R or M update writes back the old copy of the entry pte_update_through_stale_tlb in both models, and also pte_write_race and pte_keeps_both_bits on the sitar_model, since the cpp_model has no such race.
An R or M update reads the entry again, without holding the bus pte_write_race and pte_keeps_both_bits, on the sitar_model.
A TLB flush does nothing Every shootdown test but the two about waiting, and also pte_keeps_both_bits, in both models.
A shootdown's handler does not drop the translation, or no IPIs are sent remap_page, revoke_write, from_every_core and other_context, in both models.
The handler flushes the running context, not the one named other_context, in both models.
The starter does not drop its own translation remap_page, revoke_write and from_every_core, in both models.
The starter does not wait for answers waits_for_every_core and core_that_does_not_answer, in both models.
The starter waits without a limit to send an IPI core_that_does_not_answer, which then never ends, on the cpp_model.
No shootdown lock from_every_core, in both models.
No lock around the IPI registers and the console console_under_lock, ipi_ping_pong, ipi_all_to_all, and most shootdown tests, in both models.
Senders wait for a busy destination by retrying under the lock ipi_all_to_all, which then never ends, in both models.
A message is written before checking for an IPI not yet taken ipi_all_to_all, in both models.

The first of these shows why each litmus test also checks that a plain load sees the writes at the end of each round. A stale cache only ever shows old values, which is not an outcome any litmus test forbids.

Every wait for another core is bounded, so a broken model makes a test fail rather than run until its cycle limit.

The OpenMP tests were checked the same way, with mistakes in the runtime. Each mistake below was run on the cpp_model under its default schedule and under rr:1, and on the sitar_model, and each is caught on all three unless the table says otherwise.

Mistake Tests that then fail
A barrier does not wait Most of openmp/.
critical, named or unnamed, takes no lock, or all names share one lock sync_constructs.
atomic takes no lock sync_constructs, parallel_team, nested_and_orphaned and floating_point.
A loop's chunks are taken without the lock loop_schedules, ordered_loops, sections, single_and_nowait, nested_and_orphaned and floating_point.
ordered does not wait, or a chunk passes on a turn it does not hold ordered_loops.
A slot of the work-share ring is taken over too early single_and_nowait.
A thread waits for a ring slot by retrying under the lock single_and_nowait, on the cpp_model only. The cores run in lock step there, and the thread that must leave the slot never gets the lock.
Every thread is first in a work-share, or none is first in single Several tests, among them single_and_nowait.
copyprivate passes no data, or skips its barrier single_and_nowait.
num_threads is ignored parallel_team, nested_and_orphaned and nested_too_deep.
A team is not limited to the cores parallel_team.
The census assumes every core is running on_one_core.
A nested region gets a real team nested_and_orphaned.
A nested region's level is not undone nested_and_orphaned, parallel_team and loop_schedules.
A worker keeps its old settings parallel_team.
Static chunks ignore the thread number loop_schedules and ordered_loops.
Guided ignores the share of what is left loop_schedules.
A dynamic loop overruns on its last chunk loop_schedules and nested_and_orphaned.
A task's data is not copied, or not aligned tasks.
A lock test always takes the lock, or a nested lock does not check its owner or depth locks.
The FPU is left off floating_point and tasks.
Thread-local data is shared, or not initialized threadprivate.
Nesting too deep is not caught nested_too_deep.
Workers are not stopped at the end Nearly every test, which then reach their cycle limit.

Running the suite

validation/run_tests.py validation/multicore --config multicore
validation/run_tests.py validation/multicore --config multicore --sitar

The tests are written to pass whatever the interleaving of the cores, whether or not the caches are in use, and whether or not they are coherent. Running them under several cpp scheduling strategies, with the caches on, and with coherence on, checks all of this. Tests that need one mode ask for it themselves in their .vprj.

SPARC_ALWAYS_CACHEABLE=1 validation/run_tests.py validation/multicore --config multicore --sitar
SPARC_ALWAYS_CACHEABLE=1 SPARC_COHERENT=1 validation/run_tests.py validation/multicore --config multicore --sitar
for s in rr:1 rr:4 rr-fixed:1 random:7:32 random:99:3; do
    SPARC_SCHEDULE=$s validation/run_tests.py validation/multicore --config multicore
done

The single-core suites run on this configuration too, on core 0 alone.

for d in asm C test_simple_ADD; do
    SPARC_RUN_CORES=1 validation/run_tests.py validation/$d --config multicore
done

With one core running, both models reproduce core_l1cache_mmu_devices cycle for cycle, so these runs also check that the multicore models did not change what a single core does.


What the suite does not cover

The tests are thorough in what they check, but some things have not been tested.

  • Other numbers of cores. The tests assume four cores. An OpenMP program was run by hand on builds with one, two and three cores, on both models, and gave the right answer, but the suite does not run on those builds.
  • Features together. OpenMP has not been run with the MMU on, or with TLB shootdown.
  • Long runs. Each test is short. The longest run for about 20 million cycles.
  • Fairness. Only the ticket lock's order is checked. No test measures whether a core can be held off by the others under a plain mutex.
  • The cpp_model against the sitar_model. Each is checked against a test's expected results. Nothing compares their cycle counts, since these differ by design.
  • The real AJIT hardware. The model follows AJIT's documents, and has not been compared with traces from the hardware.

How long the tests run

With the MMU off nothing is cached, so on the sitar_model every instruction fetch of every core goes through the one shared bus, and the cores slow each other down. boot/staggered_halt takes about 1.1 million cycles with four cores, hence the MAX_CYCLES line in its .vprj, and about 150 thousand with SPARC_ALWAYS_CACHEABLE=1. The Execution Model page shows how the cycle count grows with the number of cores.