Skip to content

The APARA C Compiler

compiler/apara/apcc.py compiles a small subset of C to mcode, the engine's assembly language. It lets you write engine programs, such as linear algebra kernels, in a familiar language, and still reach the engine's vector instructions where they matter.

It is a teaching compiler, kept small enough to read: one Python file, standard library only. It does not optimize much, and it leaves out the parts of C that need a stack, since the engine has none. Its output is plain mcode, so you can read what it produced, and replace a part with hand-written mcode when speed matters (see inline mcode).

The tool chain, for a program prog.c:

prog.c --apcc.py--> prog.mcode --apara_as.py--> prog.obj --> simulator
               \--> prog.data  (DMEM's initial contents) --> simulator

A first program

A dot product of two 64-element int8_t vectors, from validation/apara/c_programs/dot_vector.c:

#define N 64
int8_t A[N] = {-43, -9, 13, 117, ...};
int8_t B[N] = {54, -114, -11, -113, ...};
int64_t result;

void main(void)
{
    int64_t sum = 0;
    for (int64_t i = 0; i < N; i += 8)
        sum = __dot_acc("vi8", sum, __ld64(&A[i]), __ld64(&B[i]));
    result = sum;
}

The global arrays live in DMEM. Each __ld64 loads 8 elements as one 64-bit value, and __dot_acc multiplies the 8 pairs and adds the products to sum, in one $dot instruction. Compile, assemble, and run it on the standalone engine:

python3 compiler/apara/apcc.py dot_vector.c        # writes dot_vector.mcode and dot_vector.data
python3 compiler/apara/apara_as.py dot_vector.mcode
model/system_models/apara/cpp_model/executable/apara_sim_cpp \
        -i dot_vector.obj -d dot_vector.data --dump-results -
! thread 0 halted, ERROR 0x2000000
0 pc 0x48
0 mem 0x10 0xffffffffffff248c
...

result is DMEM dword 0x10 (byte address 0x80), and holds -56180. The compiler lists where it put each global at the top of the mcode file:

// DMEM layout (byte offsets):
//   A                0x0000  int8_t[64]
//   B                0x0040  int8_t[64]
//   result           0x0080  int64_t
// Generated by apcc.py (-O1). Edit the C source, not this file.

main:
    || + $r1 ($i64) $r0 0  + $r2 ($i64) $r0 0 ;
    || - $r3 ($i64) $r2 64 ;
    || ? ($i64) $r3 >= $goto L3_done ;
L1_loop:
    ...
    || $ld ($i64) $r4 [$r5 + 0] ;
    || $dot $accumulate $r1 ($vi8) $r3 $r4 ;
L2_next:
    || + $r2 ($i64) $r2 8 ;
    || - $r3 ($i64) $r2 64 ;
    || ? ($i64) $r3 < $goto L1_loop ;
L3_done:
    || $st ($i64) [$r0 + 128] $r1  $halt ;

Comments in the mcode give the C line each part came from.


Running it

python3 compiler/apara/apcc.py prog.c [-o prog.mcode] [--data prog.data] \
        [--symbols prog.sym] [-O0] [--dmem-base addr]
Option Does
-o <file> The mcode file (default: the source's name with .mcode).
--data <file> DMEM's initial contents, a data file (default: .data), from the globals' initializers.
--symbols <file> Also write each global's DMEM byte offset, one name offset per line.
-O0 One instruction per bundle. The default, -O1, packs independent instructions into bundles (see Code generation).
--dmem-base <addr> The address of DMEM byte 0 as the program sees it: 0 (the default) for the standalone engine, 0x01000000 for the engine in sparc_apara, whose DMEM is a window in the system's address space (see In a system).

Errors stop the compilation, with the file, the line, and the reason:

apcc.py: error: prog.c:2: recursion (f -> f) is not supported
apcc.py: error: prog.c:7: x is not defined

python3 compiler/apara/test_apcc.py runs the compiler's own unit tests: small programs whose layout, output and error messages it checks, and whose mcode must assemble. The programs' results are checked by the validation/apara/c_programs/ suite.


The language

Types

C type Engine type
int8_t, int16_t, int32_t, int64_t (and char, short, int, long) $i8 .. $i64
uint8_t .. uint64_t (and unsigned char ..) $u8 .. $u64
float, double $f32, $f64
void (functions only)

Integer expressions are computed in 64 bits (unsigned if an operand is uint64_t, signed otherwise), much as C widens small types before arithmetic. A value is wrapped to its type's width when it is stored in a narrower variable or array element, or cast to a narrower type. Float expressions are computed in float, or in double if an operand is a double. A cast converts explicitly ((int32_t) x, (double) i).

Variables

  • Global variables, scalars and one- or two-dimensional arrays, live in DMEM. They are laid out from address 0 in declaration order, each aligned to 8 bytes, unless __at(addr) places one at a fixed DMEM byte address (a multiple of 8):

    int8_t A[64] __at(0x100);
    

    They may have constant initializers, which go into the data file. Without one, a global starts as zero on the standalone engine, whose DMEM starts as zero (in a system, see In a system).

  • Local variables and parameters are scalars, each kept in a register for the function's whole life. There is no stack, so there are no local arrays.

Statements and expressions

  • Statements: declarations, assignments (=, +=, -=, *=, /=, %=, &=, |=, ^=, <<=, >>=), ++ and --, if and else, while, do ... while, for, break, continue, return, and blocks.
  • Expressions: the usual arithmetic, bitwise, shift, comparison and logical operators, ?:, casts, array indexing (A[i], M[i][j]), &A[i] (an element's DMEM byte address, for the intrinsics), and calls.
  • Preprocessor: object-like #define NAME value. Other preprocessor lines, #include among them, are ignored.

Functions and threads

  • Functions may call other functions, but not recursively (directly or through others). Each variable gets its own register, different from those of every function on a call chain through it, so a call has nothing to save: it is a $call and a $return.
  • __entry marks a thread's entry point. Its parameters arrive in $r1, $r2, ... (set before the thread starts), and returning from it halts the thread. A program may have several entry points, one per kind of thread. Without any __entry, main is the entry point.

Several threads may run the same entry function, each with its own arguments. From c_programs/matmul_threads.c, a matrix product whose rows are split among threads:

__entry void rows(int64_t first, int64_t count)
{
    for (int64_t i = first; i < first + count; i++)
        for (int64_t j = 0; j < N; j++) {
            int64_t sum = 0;
            for (int64_t k = 0; k < N; k++)
                sum += A[i][k] * B[k][j];
            C[i][j] = sum;
        }
}

Every thread has its own registers, so each has its own i, j, k and sum. They share the global arrays, so each must write only its own part of them: the engine has no atomic operations (see Threads sharing data).

Limits

The limits, by design:

  • No recursion, no pointers (other than integer DMEM addresses), no structs, no local arrays.
  • At most about 28 variables live along any call chain (the engine has 31 usable registers, and expressions need temporaries). The compiler says so when a program needs more.
  • Comparisons test the sign of the difference, so two 64-bit operands must differ by less than 263.
  • No standard library. There is nowhere to print to: a program leaves its results in DMEM.

The engine's own instructions

Intrinsics

The intrinsics reach the vector instructions. Each works on 64-bit values holding packed vectors, as the engine's registers do. T names the element type, as a string: "vi8", "vu16", "vf32", and so on (vi, vu or vf, 4 to 32 bits).

Intrinsic Instruction Does
__ld64(addr), __st64(addr, v) $ld, $st ($i64) A 64-bit load or store at a DMEM byte address (&A[i]): 8 int8_t elements at once, for example.
__dot(T, a, b) $dot The sum of the element products.
__dot_acc(T, acc, a, b) $dot $accumulate acc + __dot(T, a, b).
__vadd, __vsub, __vmul (T, a, b) $v + - * Element-wise.
__vadd_rep, __vsub_rep, __vmul_rep (T, a, s) $v ... $replicate The same, with s's lowest element for every element of the second operand.
__vreduce(OP, T, a) $vreduce All the elements combined by OP: "+", "*", "|", "&", "^".
__fsqrt(x) $fsqrt The square root of a float or double.
__slice(x, high, low) $slice Bits [high:low] of x (constant positions).
__halt() $halt Stop the thread here.

Inline mcode

__asm__("...") puts mcode into the program as it is, with %name standing for a variable's register. One instruction becomes a bundle of its own; a whole bundle, || ... ;, is kept as written. The compiler never moves other instructions into, or across, inline mcode. From c_programs/inline_asm.c:

int64_t v = 0x0102030405060708;
int64_t s = 0;
int64_t t = 0;
__asm__("$vreduce + %s ($vi8) %v");                       // s = 36
__asm__("|| + %t ($i64) %s %s  - %v ($i64) %v %v ;");   // t = 72, v = 0

The second bundle shows the engine's bundle semantics: both instructions read s and v as they were before the bundle.

Checking a program on the host

compiler/apara/include/apara.h defines __entry, __at and every intrinsic in plain C, so the same program also compiles with an ordinary C compiler, to check its results on the host before running it on the engine. A small harness calls the program and prints its results:

#include <stdio.h>
#include "apara.h"
#define main kernel_main
#include "dot_vector.c"
#undef main

int main(void)
{
    kernel_main();
    printf("result = %lld\n", (long long) result);   // -56180, as on the engine
    return 0;
}
gcc -I compiler/apara/include host_check.c -o host_check && ./host_check

apcc.py ignores the #include, and knows the intrinsics itself. On the host, __ld64 and __st64 use the host's byte order. Element-wise results and dot products come out the same as on the engine, but the elements' positions within a 64-bit value differ, so __slice of a loaded vector does not.


Code generation

  • One register per variable, temporaries for expressions.
  • Loops test at the bottom, so each iteration takes one branch, which the branch predictor learns (see Branch prediction).
  • Bundles. With -O1, a scheduler packs consecutive instructions into one bundle when none depends on another, and when the bundle stays legal (at most one control transfer, four memory operations and one divide, and no register written twice). Since every instruction in a bundle reads the registers as they were before the bundle, the result is the same as running the instructions one by one. -O0 puts every instruction in its own bundle, for comparison: the c_programs/dot_scalar and dot_scalar_O0 tests run the same program both ways.

The hardware resolves every dependence between bundles (see Hazards), so the compiler never needs to insert waits: a bundle that needs an earlier result waits in the pipeline until it is ready. The compiler's job, for speed, is to give the pipeline independent work, by unrolling loops by hand, or by splitting the work among threads.


Linear algebra on the engine

The engine's strength is packed vectors of small elements: one $dot on $vi8 operands is 8 multiplications and 8 additions. A linear algebra kernel gets that speed when its inner loop works on whole 64-bit words.

  • Dot products and matrix products: load 8 int8_t (or 4 int16_t) elements with __ld64, and accumulate with __dot_acc, as in the first program. For a matrix product, store the second matrix transposed, so that a column is contiguous in DMEM.
  • Element-wise operations (vector addition, scaling): __vadd, __vmul_rep (a vector times a scalar), and __st64 to store 8 results at once (c_programs/vadd_vector.c).
  • Floating point: float and double arithmetic compiles to the engine's float instructions (c_programs/saxpy.c: y = a·x + y).
  • More threads: give each thread its own rows or its own slice, as matmul_threads.c does, and start every thread at the entry function, each with its own arguments.

The test folder validation/apara/c_programs/ has each of these, with expected results computed independently. Running one is the quickest way to start (see APARA Validation Suite). For example, matmul_threads.c on three threads, on the timing model, reports its cycles and how the pipeline spent them:

python3 compiler/apara/apcc.py matmul_threads.c
python3 compiler/apara/apara_as.py matmul_threads.mcode
printf '0 1 0x0\n0 2 0x2\n1 1 0x2\n1 2 0x2\n2 1 0x4\n2 2 0x2\n' > registers.txt   # first row, row count
model/system_models/apara/sitar_model/executable/apara_sim_sitar -p 0x0 -p 0x0 -p 0x0 \
        -i matmul_threads.obj -d matmul_threads.data -a registers.txt --stats
ran 4554 bundles in 7125 cycles
...
register-lock stalls     914 cycles
REGFILE token stalls     1670 cycles

rows is at IMEM address 0x0 here, so each -p 0x0 starts one more thread there (apara_as.py --aligned shows every label's address). Each thread gets two rows, through $r1 and $r2. The counters show where the time goes: APARA Timing Model explains them.


In a system

In sparc_apara, a SPARC program loads the engine program and its data, starts its threads, and collects the results. The engine's DMEM appears there as a window at 0x0100_0000, and an engine program must address it that way (any other address goes out to the system's main memory). So a C kernel for the system is compiled with --dmem-base 0x01000000.

compiler/apara/apara_obj_to_c.py does that, and turns the result into a C header for the SPARC program:

python3 compiler/apara/apara_obj_to_c.py vector_stats.c     # writes vector_stats.c.h
static const uint32_t vector_stats_program[] = { ... };   // the IMEM words
#define VECTOR_STATS_PROGRAM_WORDS 160
#define VECTOR_STATS_MAIN 0x0                               // each function's IMEM address
#define VECTOR_STATS_DMEM_DATA 0x0                          // each global's DMEM byte offset
#define VECTOR_STATS_DMEM_SUM 0x100
...
static const uint32_t vector_stats_dmem_image[] = { ... };  // the initializers
#define VECTOR_STATS_DMEM_IMAGE_WORDS 0

kernel_dmem_image[] holds only the non-zero words of the initial data. DMEM is zero at power-up, but keeps whatever an earlier program left there, so a kernel that is run after another one should not count on its uninitialized (or zero-initialized) globals being zero: the SPARC program clears them, or the kernel sets them itself.

The SPARC program then uses the engine through the routines of compiler/include/apara_access_routines.h:

#include "vector_stats.c.h"

apara_load_program(vector_stats_program, VECTOR_STATS_PROGRAM_WORDS);
apara_load_dmem_image(vector_stats_dmem_image, VECTOR_STATS_DMEM_IMAGE_WORDS);
apara_dma_load(0, VECTOR_STATS_DMEM_DATA, (uint32_t) data, sizeof(data));   // the input
apara_start_thread(0, VECTOR_STATS_MAIN);
apara_wait_thread(0);
sum = apara_dmem_read_dword(VECTOR_STATS_DMEM_SUM);                       // the result

The Engine in a System describes these steps in full. A test in the validation suite names its C kernels in its .vprj, with an MCODE = vector_stats.c line, and validation/build_hex.py writes the header before compiling the test.