Skip to content

The Multicore System

The multicore configuration models several SPARC V8 cores that share one main memory and one set of peripheral devices. Each core has its own split L1 instruction and data caches and its own MMU, exactly as in the single-core core_l1cache_mmu_devices configuration. The cores reach memory and the devices over one shared bus, so a program can run on all of them at once and share data through memory. The design follows AJIT's multi-core processors wherever that design is known.


The system at a glance

  • Cores. NUM_CORES cores, 4 by default. Core i has core ID i. It reads 0x50520000 | (i << 8) from ASR29, the same register AJIT software uses to find its own core and thread.
  • Caches and MMU. Each core has its own L1 caches and MMU. The caches are write-through, so main memory always holds every value a core has written.
  • Coherence. A run-time option, --coherent, off by default. When it is off, a core's caches can keep an old copy of a line after another core writes to it, and programs read shared data with uncached loads, or flush their caches first. When it is on, a write removes that line from every other core's caches (see Coherent caches).
  • Atomic instructions. LDSTUB and SWAP are atomic across cores in both models, and a library of locks, barriers and semaphores is built on them.
  • OpenMP. C programs can use real OpenMP pragmas, on a runtime written for this system.
  • Memory and devices. One main memory and one set of devices (Timer, Serial, Interrupt Controller, Scratch-pad), shared by every core. The Interrupt Controller has a slot for each core and delivers each core its own interrupts.
  • Startup. Every core starts at PC 0 and runs the same program, which reads its core ID to decide what to do.
  • End of a run. A run ends when every core has halted.

Status

The configuration is being built in stages.

Stage Contents Status
1 The cores, the shared bus, startup of every core, checking each core's results, per-core interrupts Done
2 Atomic LDSTUB and SWAP across cores, and a library of synchronization routines (locks, barriers) Done
3 Write-invalidate cache coherence, the --coherent option Done
4 Shared devices from several cores (a console under a lock, TLB shootdown through IPIs) Done
5 OpenMP programs, with a bare-metal OpenMP runtime Done
6 Context switching and a scheduler on each core Planned
7 Parallel simulation of the sitar_model on several host threads Done, with speedup measurements on a larger host to follow

These pages describe what is done, and grow with each stage.


A first run

Both models are built like any other configuration. The following commands build the sitar_model, and run a worked example that sums an array on all four cores.

cd $SPARC_HOME/model/system_models/multicore/sitar_model
./build.sh
./executable/sparc_sim_sitar_multicore \
    $SPARC_HOME/validation/multicore/examples/parallel_sum/parallel_sum.hex "" 5000000

With no expected results to check, the simulator prints the final state of each core under its own heading, and then the length of the run.

=== core 0 ===
PROCESSOR STATE: ERROR
...
Simulation halted after 3085726 cycles (all 4 cores halted).

As on a single core, PROCESSOR STATE: ERROR is how a program ends (see Run the program). The third argument is the cycle limit, and its default of one million is too small for this run, as the next section explains.


Three ways to run a program

Whether anything is cached decides both a run's length and what a program must do to share data, so a run is one of three kinds.

Options What is cached The program's shared data Timing
None Nothing, with the MMU off Every load sees main memory Slow on the sitar_model, since every fetch crosses the bus
--always-cacheable All of main memory A plain load can return an old copy, so the program must read with read_shared() or flush first Realistic
--always-cacheable --coherent All of main memory A plain load always sees the latest value The most realistic

Sharing data between cores explains the difference. parallel_sum above takes about 3.1 million cycles the first way, and about 470 thousand the second.


Options

./executable/sparc_sim_cpp_multicore <program.hex> [expected] [max_cycles] [options]

The cache size, serial and timer options are those of core_l1cache_mmu_devices, and they apply to every core. An option the simulator does not know stops the run with an error, so a mistyped option cannot go unnoticed.

  • --always-cacheable makes every access cacheable while the MMU is off. SPARC_ALWAYS_CACHEABLE=1 does the same.
  • --run-cores <m> runs only cores 0 to m-1. The others never leave reset, and the run does not wait for them. With --run-cores 1, any single-core program runs unchanged. The environment variable SPARC_RUN_CORES does the same.
  • --schedule <spec> (cpp_model only) chooses how the cores' instructions are interleaved. See Execution Model.
  • --threads <n> and --thread-map <list> (sitar_model, built with --openmp) run the simulation on several host threads. See Parallel Simulation.
  • --coherent keeps the caches coherent. SPARC_COHERENT=1 does the same, and --no-coherent overrides it. See Coherent caches.
  • --snoop-filter leaves out repeated invalidates. SPARC_SNOOP_FILTER=1 does the same.
  • --stats prints each core's counters, then the shared memory's and devices' counters. The sitar_model also prints the number of bus transactions each core made.

Compile-time knobs

Knob Default Where Description
NUM_CORES 4 cpp_model: MulticoreConfig.h
sitar_model: System's parameter in System.sitar
Number of cores, 1 to 4. The Interrupt Controller has slots for 4 cores. The validation suite assumes 4.

Every Sitar latency knob of the single-core configuration is also in System.sitar, set the same for every core, along with the bus's own bridge.delay. See Model Configuration Settings.


In these pages

To write programs for the system:

  • Programming the Multicore System covers the startup file, the core ID, sharing data with and without coherent caches, locks and other synchronization, interrupts on several cores, shared devices, and the routines provided.
  • OpenMP Programs covers programs that use OpenMP's pragmas, and what the runtime supports.
  • Example Programs walks through a parallel array sum and a parallel matrix multiply.
  • Logging and Debugging covers per-core traces, gdb with several cores, and finding data races.

To understand or change the model:

  • Execution Model describes how the cpp_model and the sitar_model run several cores, and why their results can differ for some programs.
  • Model Structure and Assumptions shows how each model is built, how the shared bus arbitrates between cores, and where the model makes its own assumptions.
  • Validation Suite lists the multicore tests, how to run them, and what they do not cover.
  • Parallel Simulation runs the sitar_model on several host threads, and explains when that is faster.