The Multicore System¶
The multicore configuration models several SPARC V8 cores that share
one main memory and one set of peripheral devices. Each core has its
own split L1 instruction and data caches and its own MMU, exactly as in
the single-core core_l1cache_mmu_devices
configuration. The cores reach memory and the devices over one shared
bus, so a program can run on all of them at once and share data
through memory. The design follows AJIT's multi-core
processors wherever
that design is known.
The system at a glance¶
- Cores.
NUM_COREScores, 4 by default. Coreihas core IDi. It reads0x50520000 | (i << 8)from ASR29, the same register AJIT software uses to find its own core and thread. - Caches and MMU. Each core has its own L1 caches and MMU. The caches are write-through, so main memory always holds every value a core has written.
- Coherence. A run-time option,
--coherent, off by default. When it is off, a core's caches can keep an old copy of a line after another core writes to it, and programs read shared data with uncached loads, or flush their caches first. When it is on, a write removes that line from every other core's caches (see Coherent caches). - Atomic instructions.
LDSTUBandSWAPare atomic across cores in both models, and a library of locks, barriers and semaphores is built on them. - OpenMP. C programs can use real OpenMP pragmas, on a runtime written for this system.
- Memory and devices. One main memory and one set of devices (Timer, Serial, Interrupt Controller, Scratch-pad), shared by every core. The Interrupt Controller has a slot for each core and delivers each core its own interrupts.
- Startup. Every core starts at PC 0 and runs the same program, which reads its core ID to decide what to do.
- End of a run. A run ends when every core has halted.
Status¶
The configuration is being built in stages.
| Stage | Contents | Status |
|---|---|---|
| 1 | The cores, the shared bus, startup of every core, checking each core's results, per-core interrupts | Done |
| 2 | Atomic LDSTUB and SWAP across cores, and a library of synchronization routines (locks, barriers) |
Done |
| 3 | Write-invalidate cache coherence, the --coherent option |
Done |
| 4 | Shared devices from several cores (a console under a lock, TLB shootdown through IPIs) | Done |
| 5 | OpenMP programs, with a bare-metal OpenMP runtime | Done |
| 6 | Context switching and a scheduler on each core | Planned |
| 7 | Parallel simulation of the sitar_model on several host threads | Done, with speedup measurements on a larger host to follow |
These pages describe what is done, and grow with each stage.
A first run¶
Both models are built like any other configuration. The following commands build the sitar_model, and run a worked example that sums an array on all four cores.
cd $SPARC_HOME/model/system_models/multicore/sitar_model
./build.sh
./executable/sparc_sim_sitar_multicore \
$SPARC_HOME/validation/multicore/examples/parallel_sum/parallel_sum.hex "" 5000000
With no expected results to check, the simulator prints the final state of each core under its own heading, and then the length of the run.
=== core 0 ===
PROCESSOR STATE: ERROR
...
Simulation halted after 3085726 cycles (all 4 cores halted).
As on a single core, PROCESSOR STATE: ERROR is how a program ends (see
Run the program). The third
argument is the cycle limit, and its default of one million is too small for
this run, as the next section explains.
Three ways to run a program¶
Whether anything is cached decides both a run's length and what a program must do to share data, so a run is one of three kinds.
| Options | What is cached | The program's shared data | Timing |
|---|---|---|---|
| None | Nothing, with the MMU off | Every load sees main memory | Slow on the sitar_model, since every fetch crosses the bus |
--always-cacheable |
All of main memory | A plain load can return an old copy, so the program must read with read_shared() or flush first |
Realistic |
--always-cacheable --coherent |
All of main memory | A plain load always sees the latest value | The most realistic |
Sharing data between cores
explains the difference. parallel_sum above takes about 3.1 million cycles
the first way, and about 470 thousand the second.
Options¶
The cache size, serial and timer options are those of
core_l1cache_mmu_devices, and they apply to every core. An option the
simulator does not know stops the run with an error, so a mistyped option
cannot go unnoticed.
--always-cacheablemakes every access cacheable while the MMU is off.SPARC_ALWAYS_CACHEABLE=1does the same.--run-cores <m>runs only cores 0 tom-1. The others never leave reset, and the run does not wait for them. With--run-cores 1, any single-core program runs unchanged. The environment variableSPARC_RUN_CORESdoes the same.--schedule <spec>(cpp_model only) chooses how the cores' instructions are interleaved. See Execution Model.--threads <n>and--thread-map <list>(sitar_model, built with--openmp) run the simulation on several host threads. See Parallel Simulation.--coherentkeeps the caches coherent.SPARC_COHERENT=1does the same, and--no-coherentoverrides it. See Coherent caches.--snoop-filterleaves out repeated invalidates.SPARC_SNOOP_FILTER=1does the same.--statsprints each core's counters, then the shared memory's and devices' counters. The sitar_model also prints the number of bus transactions each core made.
Compile-time knobs¶
| Knob | Default | Where | Description |
|---|---|---|---|
NUM_CORES |
4 | cpp_model: MulticoreConfig.hsitar_model: System's parameter in System.sitar |
Number of cores, 1 to 4. The Interrupt Controller has slots for 4 cores. The validation suite assumes 4. |
Every Sitar latency knob of the single-core configuration is also in
System.sitar, set the same for every core, along with the bus's own
bridge.delay. See Model Configuration
Settings.
In these pages¶
To write programs for the system:
- Programming the Multicore System covers the startup file, the core ID, sharing data with and without coherent caches, locks and other synchronization, interrupts on several cores, shared devices, and the routines provided.
- OpenMP Programs covers programs that use OpenMP's pragmas, and what the runtime supports.
- Example Programs walks through a parallel array sum and a parallel matrix multiply.
- Logging and Debugging covers per-core traces, gdb with several cores, and finding data races.
To understand or change the model:
- Execution Model describes how the cpp_model and the sitar_model run several cores, and why their results can differ for some programs.
- Model Structure and Assumptions shows how each model is built, how the shared bus arbitrates between cores, and where the model makes its own assumptions.
- Validation Suite lists the multicore tests, how to run them, and what they do not cover.
- Parallel Simulation runs the sitar_model on several host threads, and explains when that is faster.