Model Structure and Assumptions¶
This page describes how the multicore configuration is put together
in each of its two models, and lists the assumptions the model makes
where AJIT's own design was not followed or is not yet modeled. It
builds on the single-core structure described in Model
Configurations.
The system¶
Every core of the multicore configuration is the core of
core_l1cache_mmu_devices, with its own split L1 caches and its own
MMU. Below the MMUs, one shared bus connects all the cores to one main
memory and one set of peripheral devices.
core 0 core 1 core 2 core 3
L1 I/D caches L1 I/D caches L1 I/D caches L1 I/D caches
MMU MMU MMU MMU
| | | |
==+===============+===============+===============+== shared bus
| |
main memory peripheral devices
(256 MB) (Timer, Serial, Interrupt Controller,
Scratch-pad)
A cache hit stays inside its core. A cache miss, every store (the caches are write-through), every uncached access, and every access to a device goes over the shared bus.
The cpp_model¶
The cpp_model builds NUM_CORES copies of the single-core
configuration's chain of C++ objects, one per core. Each copy holds a
SparcCore, its SparcStateMachine, an L1CachesCore and an
MmuCore. Every core's MmuCore is connected to the same
SystemBus
object, which sends each access to main memory or to the devices by its
address, exactly as in the single-core configurations.
The cpp_model runs on one host thread. A
Scheduler
picks which core executes next and for how many instructions. Every
access completes before the next one starts, so the bus never has two
requests at once and needs no arbitration. The Execution
Model page describes the scheduling strategies.
The sitar_model¶
Modules and nets¶
System holds NUM_CORES copies of the single-core configuration's
Core module, as a Sitar submodule_array. Each Core contains the
SPARC core, its L1 caches and its MMU as procedures, so a cache hit
never leaves the core. Below the MMUs sits a CoherentBridge, the
shared bus, and below it one main memory and one set of peripheral
devices.
Top | +-- System (module) | +-- cores[NUM_CORES] : Core (module, one per core) | | | +-- sparcThread : SparcThread (procedure) | | | +-- caches : L1Caches (procedure) | | | +-- mmu : Mmu (procedure) | +-- bridge : CoherentBridge (module) | +-- mainMemory : PhysicalMainMemory (module) | +-- devices : PeripheralDevices (module)
Each core has four nets of its own, and the bus reaches memory and the devices over four more. Each net carries data from an output port to an input port.
| Net | From | To |
|---|---|---|
coreRequestNet[i] |
cores[i].requestOut |
bridge.coreRequestIn[i] |
coreResponseNet[i] |
bridge.coreResponseOut[i] |
cores[i].responseIn |
interruptLevelNet[i] |
devices.interruptLevelOut[i] |
cores[i].interruptLevelIn |
coreInvalidateNet[i] |
bridge.coreInvalidateOut[i] |
cores[i].invalidateIn |
memoryRequestNet |
bridge.memoryRequestOut |
mainMemory.requestIn |
memoryResponseNet |
mainMemory.responseOut |
bridge.memoryResponseIn |
devicesRequestNet |
bridge.devicesRequestOut |
devices.requestIn |
devicesResponseNet |
devices.responseOut |
bridge.devicesResponseIn |
System sets each core's ID in its init block, so cores[i] is core
i.
The shared bus¶
CoherentBridge does the work of the single-core configuration's
SystemBridge for several cores. It has one request port and one
response port for each core. It takes one request at a time, reads its
physical address, forwards it to main memory or to the devices, and
returns the answer to the core that asked. Its timing is
SystemBridge's, so with one core running, the multicore sitar_model
reproduces core_l1cache_mmu_devices cycle for cycle.
When several cores have requests waiting, the bridge grants them in round-robin order. Starting from the port after the last one granted, it checks each port in turn, skips a port with no request waiting, and grants the first one that has a request.
The bridge is named after AJIT's coherent memory controller, which plays the same role.
The bus lock¶
The bridge makes LDSTUB and SWAP atomic across cores. Each of these
instructions reads a location and then writes it, and in the
sitar_model those are two separate bus transactions, so another core
could otherwise come in between them. The core's MMU marks the read
with a lock bit, as AJIT's does.
A read with the lock bit set locks the bus to the core that made it,
and that core's next request releases it. While the bus is locked, the
bridge looks only at that core's port, so every other core's requests
wait, whichever address they are for. As in AJIT's bridge, there is one
lock for the whole bus, and it belongs to a core. This is coarser than
atomicity strictly needs, since two atomic instructions on unrelated
addresses need not exclude each other, but it is simple and it covers
main memory and the devices alike.
The lock holds no extra time beyond the two transactions, so it changes
no timing when only one core is using the bus. An access to an address
that maps to nothing still takes and releases the lock, so it cannot
leave the bus locked. --stats prints the number of locked read-write
pairs the bridge has served.
Coherence¶
With --coherent, a write to main memory removes that line from every
other core's caches. Only the transport differs between the two models.
The caches themselves are the same L1CachesCore class that every
configuration uses, and in a single-core system nothing ever sends them
an invalidate.
- Finding a line. The caches are virtually indexed and tagged, but an invalidate carries a physical address. Each line therefore also keeps its physical address. A virtual and a physical address agree within a page, so only the sets the rest of the index could select need searching. In a 32 KB cache with 4 KB pages, that is 8 sets.
- The sitar_model. When the bridge forwards a write to main memory, it pushes the line's address to every other core, on a net of its own, in that same phase. Each core polls this net every cycle, whatever it is doing, and again before every cache access, so a hit in that phase cannot see a line that has just been invalidated. The invalidate reaches the other cores in the same cycle that memory takes the write, before the writer gets its response.
- The cpp_model. A small
CoherentBusgives each core a port of its own onto the bus, and a write calls the other cores' caches directly. Nothing else is running at that moment, because the cores take turns.
A fill in progress. A line fill in the sitar_model is 8 separate bus
reads, and another core can write the line between them. The invalidate
then finds nothing to remove, since the line is not installed yet, and
installing it would leave a stale line that nothing would ever remove.
A cache therefore remembers the invalidates that arrive during a fill. If
one of them is for this line, the fill still serves the read that started
it, but the line is not installed. --stats counts these as fills
abandoned.
The writer's own caches get no invalidate, as in AJIT. A store updates
the writer's own data cache, but not its instruction cache, so code that
modifies itself must still execute FLUSH.
Snoop filter. Every write sends an invalidate to every other core,
since the bus cannot see which lines each cache holds. --snoop-filter
leaves one out if that core's last invalidate was for the same line and
it has read nothing from the bus since, because it cannot then hold the
line. The results of a correct program do not change. Timing can change
slightly, because the filter changes which fills in progress hear about a
write, and so which of them are abandoned. --stats shows how many
invalidates were left out. AJIT's bridge has a larger filter with the same
purpose.
Synonyms¶
Two virtual addresses, or one address in two MMU contexts, can map to one
physical line. A cache holds at most one copy of it, in every
configuration. Installing a line first removes any older copy with the
same physical address, so a store through one address is never missed by
a load through the other. A store through an uncached alias and an atomic
instruction never install a line, so they remove the other copy themselves.
A store or atomic whose fault is suppressed by the MMU's NF bit reaches no
memory and has no physical address, so it removes nothing.
--stats counts these as synonyms removed. A bypass-ASI access goes
around the cache on purpose, and leaves this core's own copy alone.
Page table updates¶
A successful translation sets the R bit of the page table entry, and a first write also
sets its M bit. Both are written back to memory. With several cores, what the walk or
the TLB holds may by then be out of date, and writing it back could undo a change
another core has just made. So when more than one core runs, the MMU reads the entry
again and sets the bits on what it finds, and an entry that is no longer a PTE is left
alone. In the sitar_model the second read is locked, so the bus is held until the
write. In the cpp_model each access runs to completion, so nothing can come between
them. The second read costs a memory access, so it is left out with --run-cores 1,
and in the single-core configurations, where nothing else writes the entry.
The end of a run¶
In the single-core configurations, Core stops the simulation as soon
as its core halts. Here a halted core only logs that it has halted,
because the other cores may still be running. System watches every
running core and stops the simulation once all of them have halted. It
checks in the second phase of each cycle, after every core has
finished its work for that cycle, so the cycle on which a run ends does
not depend on the order in which the cores happen to run.
System's second parameter, STOP_WHEN_HALTED, is true by default.
A model that holds several Systems sets it to false, since the first
system to finish would otherwise end the run for all of them, and stops
the run itself once every system has halted (see A larger
model).
Cores beyond --run-cores are held in reset for the whole run, through
SparcThread's reset pin, and the run does not wait for them.
Latency knobs¶
System.sitar's init block holds every latency knob. The per-core
knobs (memory-interface delays, cache hit and fill delays, MMU delays)
are set in a loop, the same for every core. The bus's bridge.delay,
mainMemory.delay and the device knobs follow. All default to 0, as in
the single-core configurations. See Model Configuration
Settings for
what each one does.
Interrupts¶
The Interrupt Controller
has 8 slots, one for each of 2 threads on each of 4 cores. Thread 0 of
core i uses slot 2i. Every core here has one thread, so slots 0,
2, 4 and 6 are in use. Each core's interrupt input is driven from
its own slot.
- cpp_model. At the end of each scheduling round, every core's interrupt input takes the level its slot presents.
- sitar_model.
PeripheralDeviceshas one interrupt-level output for each of the 4 cores the controller supports, andSystemconnects outputito corei. Each core polls its own net once per cycle. The single-core configurations connect output 0 only.
The controller's slot count is the reason NUM_CORES is at most 4.
Assumptions¶
The model follows AJIT's multi-core design wherever that design is known, from the AJIT processor description and its section on cache coherence. The table lists where it makes its own assumptions, and what is not yet modeled.
| Aspect | In this model | Notes |
|---|---|---|
| Reset | Every core starts at PC 0 and runs the same program. The reset PC is SparcCore's resetPC, which a model's main may change before the run. |
As in AJIT. |
| Core ID | ASR29 reads 0x50520000 \| (core << 8) \| thread. |
As in AJIT. One thread per core, so the thread field is 0. |
| Threads per core | 1 | AJIT cores can have 2. |
| Number of cores | 1 to 4, set at compile time | Limited by the Interrupt Controller's 8 slots. The validation suite assumes 4. |
| L1 caches | Each core's own, VIVT, write-through, as in core_l1cache_mmu_devices |
|
| Coherence | Off by default, write-invalidate with --coherent |
See Coherence. Each line keeps its physical address, where AJIT keeps a separate reverse table (its RLUT). Data is the same either way, and only hit and miss counts differ. |
| Synonyms | At most one copy of a physical line per cache | AJIT removes an older copy too, unless both have the same virtual address, in different contexts. Here that older copy is removed as well, since a store in one context would otherwise leave it stale. |
FLUSH |
Empties the caches of the core that executes it, and no other | The other cores' copies go through coherence, or a flush of their own. |
| Memory ordering | Sequentially consistent | No store buffer. SPARC's TSO allows more than this, so correct programs still work. STBAR has no effect. |
Atomic LDSTUB and SWAP across cores |
Both models. | The cpp_model runs each instruction to completion. In the sitar_model, the bridge locks the bus from the read to the write (see The bus lock), by core and for the whole bus, as in AJIT. |
| Page table entry updates | Atomic across cores, by a locked read and write in the sitar_model | The R and M bits are set on the entry in memory, not on a copy. See Page table updates. |
| Bus arbitration (sitar_model) | Round-robin over the cores, one transaction at a time | An assumption, not taken from AJIT. |
| Bus timing (sitar_model) | SystemBridge's, plus the bridge.delay knob |
|
| Main memory and devices | One of each, shared by every core | As in AJIT. |
| Interrupt routing | Core i takes its interrupts from slot 2i |
As in AJIT, for thread 0. |
| End of run | When every running core has halted | A modeling convenience. AJIT has no such notion. |