The APARA C Compiler¶
compiler/apara/apcc.py compiles a small subset of C to mcode, the
engine's assembly language. It lets you write engine programs, such as
linear algebra kernels, in a familiar language, and still reach the
engine's vector instructions where they matter.
It is a teaching compiler, kept small enough to read: one Python file, standard library only. It does not optimize much, and it leaves out the parts of C that need a stack, since the engine has none. Its output is plain mcode, so you can read what it produced, and replace a part with hand-written mcode when speed matters (see inline mcode).
The tool chain, for a program prog.c:
prog.c --apcc.py--> prog.mcode --apara_as.py--> prog.obj --> simulator
\--> prog.data (DMEM's initial contents) --> simulator
A first program¶
A dot product of two 64-element int8_t vectors, from
validation/apara/c_programs/dot_vector.c:
#define N 64
int8_t A[N] = {-43, -9, 13, 117, ...};
int8_t B[N] = {54, -114, -11, -113, ...};
int64_t result;
void main(void)
{
int64_t sum = 0;
for (int64_t i = 0; i < N; i += 8)
sum = __dot_acc("vi8", sum, __ld64(&A[i]), __ld64(&B[i]));
result = sum;
}
The global arrays live in DMEM. Each __ld64 loads 8 elements as one
64-bit value, and __dot_acc multiplies the 8 pairs and adds the
products to sum, in one $dot instruction. Compile, assemble, and
run it on the standalone engine:
python3 compiler/apara/apcc.py dot_vector.c # writes dot_vector.mcode and dot_vector.data
python3 compiler/apara/apara_as.py dot_vector.mcode
model/system_models/apara/cpp_model/executable/apara_sim_cpp \
-i dot_vector.obj -d dot_vector.data --dump-results -
result is DMEM dword 0x10 (byte address 0x80), and holds -56180. The
compiler lists where it put each global at the top of the mcode file:
// DMEM layout (byte offsets):
// A 0x0000 int8_t[64]
// B 0x0040 int8_t[64]
// result 0x0080 int64_t
// Generated by apcc.py (-O1). Edit the C source, not this file.
main:
|| + $r1 ($i64) $r0 0 + $r2 ($i64) $r0 0 ;
|| - $r3 ($i64) $r2 64 ;
|| ? ($i64) $r3 >= $goto L3_done ;
L1_loop:
...
|| $ld ($i64) $r4 [$r5 + 0] ;
|| $dot $accumulate $r1 ($vi8) $r3 $r4 ;
L2_next:
|| + $r2 ($i64) $r2 8 ;
|| - $r3 ($i64) $r2 64 ;
|| ? ($i64) $r3 < $goto L1_loop ;
L3_done:
|| $st ($i64) [$r0 + 128] $r1 $halt ;
Comments in the mcode give the C line each part came from.
Running it¶
python3 compiler/apara/apcc.py prog.c [-o prog.mcode] [--data prog.data] \
[--symbols prog.sym] [-O0] [--dmem-base addr]
| Option | Does |
|---|---|
-o <file> |
The mcode file (default: the source's name with .mcode). |
--data <file> |
DMEM's initial contents, a data file (default: .data), from the globals' initializers. |
--symbols <file> |
Also write each global's DMEM byte offset, one name offset per line. |
-O0 |
One instruction per bundle. The default, -O1, packs independent instructions into bundles (see Code generation). |
--dmem-base <addr> |
The address of DMEM byte 0 as the program sees it: 0 (the default) for the standalone engine, 0x01000000 for the engine in sparc_apara, whose DMEM is a window in the system's address space (see In a system). |
Errors stop the compilation, with the file, the line, and the reason:
apcc.py: error: prog.c:2: recursion (f -> f) is not supported
apcc.py: error: prog.c:7: x is not defined
python3 compiler/apara/test_apcc.py runs the compiler's own unit
tests: small programs whose layout, output and error messages it checks,
and whose mcode must assemble. The programs' results are checked by the
validation/apara/c_programs/ suite.
The language¶
Types¶
| C type | Engine type |
|---|---|
int8_t, int16_t, int32_t, int64_t (and char, short, int, long) |
$i8 .. $i64 |
uint8_t .. uint64_t (and unsigned char ..) |
$u8 .. $u64 |
float, double |
$f32, $f64 |
void |
(functions only) |
Integer expressions are computed in 64 bits (unsigned if an operand is
uint64_t, signed otherwise), much as C widens small types before
arithmetic. A value is wrapped to its type's width when it is stored in
a narrower variable or array element, or cast to a narrower type. Float
expressions are computed in float, or in double if an operand is a
double. A cast converts explicitly ((int32_t) x, (double) i).
Variables¶
-
Global variables, scalars and one- or two-dimensional arrays, live in DMEM. They are laid out from address 0 in declaration order, each aligned to 8 bytes, unless
__at(addr)places one at a fixed DMEM byte address (a multiple of 8):They may have constant initializers, which go into the data file. Without one, a global starts as zero on the standalone engine, whose DMEM starts as zero (in a system, see In a system).
-
Local variables and parameters are scalars, each kept in a register for the function's whole life. There is no stack, so there are no local arrays.
Statements and expressions¶
- Statements: declarations, assignments (
=,+=,-=,*=,/=,%=,&=,|=,^=,<<=,>>=),++and--,ifandelse,while,do...while,for,break,continue,return, and blocks. - Expressions: the usual arithmetic, bitwise, shift, comparison and
logical operators,
?:, casts, array indexing (A[i],M[i][j]),&A[i](an element's DMEM byte address, for the intrinsics), and calls. - Preprocessor: object-like
#define NAME value. Other preprocessor lines,#includeamong them, are ignored.
Functions and threads¶
- Functions may call other functions, but not recursively
(directly or through others). Each variable gets its own register,
different from those of every function on a call chain through it,
so a call has nothing to save: it is a
$calland a$return. __entrymarks a thread's entry point. Its parameters arrive in$r1,$r2, ... (set before the thread starts), and returning from it halts the thread. A program may have several entry points, one per kind of thread. Without any__entry,mainis the entry point.
Several threads may run the same entry function, each with its own
arguments. From c_programs/matmul_threads.c, a matrix product whose
rows are split among threads:
__entry void rows(int64_t first, int64_t count)
{
for (int64_t i = first; i < first + count; i++)
for (int64_t j = 0; j < N; j++) {
int64_t sum = 0;
for (int64_t k = 0; k < N; k++)
sum += A[i][k] * B[k][j];
C[i][j] = sum;
}
}
Every thread has its own registers, so each has its own i, j, k
and sum. They share the global arrays, so each must write only its own
part of them: the engine has no atomic operations (see Threads sharing
data).
Limits¶
The limits, by design:
- No recursion, no pointers (other than integer DMEM addresses), no structs, no local arrays.
- At most about 28 variables live along any call chain (the engine has 31 usable registers, and expressions need temporaries). The compiler says so when a program needs more.
- Comparisons test the sign of the difference, so two 64-bit operands must differ by less than 263.
- No standard library. There is nowhere to print to: a program leaves its results in DMEM.
The engine's own instructions¶
Intrinsics¶
The intrinsics reach the vector instructions. Each works on 64-bit
values holding packed vectors, as the engine's registers do. T names
the element type, as a string: "vi8", "vu16", "vf32", and so on
(vi, vu or vf, 4 to 32 bits).
| Intrinsic | Instruction | Does |
|---|---|---|
__ld64(addr), __st64(addr, v) |
$ld, $st ($i64) |
A 64-bit load or store at a DMEM byte address (&A[i]): 8 int8_t elements at once, for example. |
__dot(T, a, b) |
$dot |
The sum of the element products. |
__dot_acc(T, acc, a, b) |
$dot $accumulate |
acc + __dot(T, a, b). |
__vadd, __vsub, __vmul (T, a, b) |
$v + - * |
Element-wise. |
__vadd_rep, __vsub_rep, __vmul_rep (T, a, s) |
$v ... $replicate |
The same, with s's lowest element for every element of the second operand. |
__vreduce(OP, T, a) |
$vreduce |
All the elements combined by OP: "+", "*", "|", "&", "^". |
__fsqrt(x) |
$fsqrt |
The square root of a float or double. |
__slice(x, high, low) |
$slice |
Bits [high:low] of x (constant positions). |
__halt() |
$halt |
Stop the thread here. |
Inline mcode¶
__asm__("...") puts mcode into the program as it is, with %name
standing for a variable's register. One instruction becomes a bundle of
its own; a whole bundle, || ... ;, is kept as written. The compiler
never moves other instructions into, or across, inline mcode. From
c_programs/inline_asm.c:
int64_t v = 0x0102030405060708;
int64_t s = 0;
int64_t t = 0;
__asm__("$vreduce + %s ($vi8) %v"); // s = 36
__asm__("|| + %t ($i64) %s %s - %v ($i64) %v %v ;"); // t = 72, v = 0
The second bundle shows the engine's bundle semantics: both
instructions read s and v as they were before the bundle.
Checking a program on the host¶
compiler/apara/include/apara.h defines __entry, __at and every
intrinsic in plain C, so the same program also compiles with an ordinary
C compiler, to check its results on the host before running it on the
engine. A small harness calls the program and prints its results:
#include <stdio.h>
#include "apara.h"
#define main kernel_main
#include "dot_vector.c"
#undef main
int main(void)
{
kernel_main();
printf("result = %lld\n", (long long) result); // -56180, as on the engine
return 0;
}
apcc.py ignores the #include, and knows the intrinsics itself. On
the host, __ld64 and __st64 use the host's byte order. Element-wise
results and dot products come out the same as on the engine, but the
elements' positions within a 64-bit value differ, so __slice of a
loaded vector does not.
Code generation¶
- One register per variable, temporaries for expressions.
- Loops test at the bottom, so each iteration takes one branch, which the branch predictor learns (see Branch prediction).
- Bundles. With
-O1, a scheduler packs consecutive instructions into one bundle when none depends on another, and when the bundle stays legal (at most one control transfer, four memory operations and one divide, and no register written twice). Since every instruction in a bundle reads the registers as they were before the bundle, the result is the same as running the instructions one by one.-O0puts every instruction in its own bundle, for comparison: thec_programs/dot_scalaranddot_scalar_O0tests run the same program both ways.
The hardware resolves every dependence between bundles (see Hazards), so the compiler never needs to insert waits: a bundle that needs an earlier result waits in the pipeline until it is ready. The compiler's job, for speed, is to give the pipeline independent work, by unrolling loops by hand, or by splitting the work among threads.
Linear algebra on the engine¶
The engine's strength is packed vectors of small elements: one $dot
on $vi8 operands is 8 multiplications and 8 additions. A linear
algebra kernel gets that speed when its inner loop works on whole
64-bit words.
- Dot products and matrix products: load 8
int8_t(or 4int16_t) elements with__ld64, and accumulate with__dot_acc, as in the first program. For a matrix product, store the second matrix transposed, so that a column is contiguous in DMEM. - Element-wise operations (vector addition, scaling):
__vadd,__vmul_rep(a vector times a scalar), and__st64to store 8 results at once (c_programs/vadd_vector.c). - Floating point:
floatanddoublearithmetic compiles to the engine's float instructions (c_programs/saxpy.c: y = a·x + y). - More threads: give each thread its own rows or its own slice, as
matmul_threads.cdoes, and start every thread at the entry function, each with its own arguments.
The test folder validation/apara/c_programs/ has each of these, with
expected results computed independently. Running one is the quickest
way to start (see APARA Validation Suite).
For example, matmul_threads.c on three threads, on the timing model,
reports its cycles and how the pipeline spent them:
python3 compiler/apara/apcc.py matmul_threads.c
python3 compiler/apara/apara_as.py matmul_threads.mcode
printf '0 1 0x0\n0 2 0x2\n1 1 0x2\n1 2 0x2\n2 1 0x4\n2 2 0x2\n' > registers.txt # first row, row count
model/system_models/apara/sitar_model/executable/apara_sim_sitar -p 0x0 -p 0x0 -p 0x0 \
-i matmul_threads.obj -d matmul_threads.data -a registers.txt --stats
ran 4554 bundles in 7125 cycles
...
register-lock stalls 914 cycles
REGFILE token stalls 1670 cycles
rows is at IMEM address 0x0 here, so each -p 0x0 starts one more
thread there (apara_as.py --aligned shows every label's address).
Each thread gets two rows, through $r1 and $r2. The counters show
where the time goes: APARA Timing Model explains
them.
In a system¶
In sparc_apara, a SPARC program loads the engine program and its data,
starts its threads, and collects the results. The engine's DMEM appears
there as a window at 0x0100_0000, and an engine program must address
it that way (any other address goes out to the system's main memory).
So a C kernel for the system is compiled with --dmem-base 0x01000000.
compiler/apara/apara_obj_to_c.py does that, and turns the result into
a C header for the SPARC program:
static const uint32_t vector_stats_program[] = { ... }; // the IMEM words
#define VECTOR_STATS_PROGRAM_WORDS 160
#define VECTOR_STATS_MAIN 0x0 // each function's IMEM address
#define VECTOR_STATS_DMEM_DATA 0x0 // each global's DMEM byte offset
#define VECTOR_STATS_DMEM_SUM 0x100
...
static const uint32_t vector_stats_dmem_image[] = { ... }; // the initializers
#define VECTOR_STATS_DMEM_IMAGE_WORDS 0
kernel_dmem_image[] holds only the non-zero words of the initial
data. DMEM is zero at power-up, but keeps whatever an earlier program
left there, so a kernel that is run after another one should not count
on its uninitialized (or zero-initialized) globals being zero: the SPARC
program clears them, or the kernel sets them itself.
The SPARC program then uses the engine through the routines of
compiler/include/apara_access_routines.h:
#include "vector_stats.c.h"
apara_load_program(vector_stats_program, VECTOR_STATS_PROGRAM_WORDS);
apara_load_dmem_image(vector_stats_dmem_image, VECTOR_STATS_DMEM_IMAGE_WORDS);
apara_dma_load(0, VECTOR_STATS_DMEM_DATA, (uint32_t) data, sizeof(data)); // the input
apara_start_thread(0, VECTOR_STATS_MAIN);
apara_wait_thread(0);
sum = apara_dmem_read_dword(VECTOR_STATS_DMEM_SUM); // the result
The Engine in a System describes these steps in full. A
test in the validation suite names its C kernels in its .vprj, with an
MCODE = vector_stats.c line, and validation/build_hex.py writes the
header before compiling the test.