Track Can Now Read Itself

The first version of Track could compile examples. That was useful, but it was not yet a serious test of the language.

A language becomes much harder to fake when its compiler starts depending on it.

Track v0.7.0 now contains the first piece of its compiler written in Track itself: a native lexer under compiler/src. It recognizes the language’s tokens, skips whitespace and comments, scans identifiers, strings, integers and operators, and runs through the same yard build path available to any other Track package.

This is not full self-hosting. The current Rust compiler still compiles that lexer. But it is the first point where Track has to be expressive and correct enough to describe part of its own implementation.

That distinction matters.

What the compiler actually looks like now

The current repository has three separate command-line programs:

• track, the single-file compiler and checker;
• yard, the package manager, build orchestrator and native test runner;
• track-lsp, the language server.

The bootstrap compiler is written in Rust. Its default native-code backend is Cranelift, not LLVM. LLVM 22 remains available behind an optional Cargo feature, but it is no longer the only path through the compiler.

That corrects an assumption from my first Track note. Compiler architecture changes when an experiment meets its actual constraints. The useful thing is to record the new boundary clearly rather than preserve an outdated diagram.

The language itself still follows the same central idea: deterministic resource management without a garbage collector, runtime, or lifetime annotation syntax. The checker models owned values through explicit states—Active, Borrowed, Locked, and Spent—and validates their transitions across moves, lexical lenses and control-flow merges.

Why the lexer came first

A lexer is small enough to port without first solving the entire compiler, but foundational enough to expose missing language and runtime features.

The native token definition already covers more than 50 variants:

union Token {
    Import, Use, Let, Mut, With, Fn, Return, If, Else,
    While, For, In, True, False, Struct, Enum, Union, Match,
    TyU8, TyI8, TyI32, TyU32, TyI64, TyU64, TyBool, TyVoid,
    Int(i64), StrLit(Str), Ident(Str),
    Eq, EqEq, Neq, Arrow, FatArrow, AmpAmp, PipePipe,
    Shl, Shr, Eof, Unknown,
}

The scanning code is deliberately ordinary. It walks the source by byte, branches on character classes and returns a token together with the next position:

fn lex_number(src: &Str, start: i64) -> (Token, i64) {
    let len = str_len(src);
    let mut p = start;
    let mut val: i64 = 0;
    let mut cont = true;

    while p < len && cont {
        let ch = str_char_at(src, p);
        if char_is_digit(ch) {
            let digit: i64 = ch - 48;
            val = val * 10 + digit;
            p = p + 1;
        } else {
            cont = false;
        }
    }

    return (Token::Int(val), p);
}

There is no iterator framework hiding the loop and no exception path hiding failure. The function makes its state, termination condition and result position visible. That style is intentional: compiler code is where Track’s preference for explicit control flow should either prove useful or become obviously unbearable.

Self-hosting is a pressure test, not a badge

Writing the lexer in Track exposed faults in the bootstrap implementation that simpler examples had not reached.

Variant constructors such as Token::Red were initially treated like linear values. Reusing one across branches could therefore produce an invalid Spent/Active merge. The checker had to learn that qualified variant constructors are copyable values.

The checker also declared str_len and str_eq, while the runtime failed to define their symbols. Small programs had avoided the missing link. A real string-heavy lexer did not.

An early return inside a match arm also generated an already-filled block during code generation. The current lexer works around that path by assigning to an output variable and returning once after the match. That is not glamorous compiler work, but it is exactly why a compiler should consume its own language early: it forces the implementation through combinations a feature demo will never cover.

The native lexer also reveals a current limitation honestly. Token is defined canonically in token.trk, but a matching definition is temporarily duplicated in lexer.trk so the present code generator can see the union inside that compilation unit. Removing that duplication is part of making the compiler’s module and type model real rather than cosmetic.

A native test path, not a demo binary

The lexer is tested through yard test. The runner discovers src/*_test.trk and tests/**/*.trk, builds each test as a temporary package, links its local modules and executes the result.

The current lexer suite checks lets, imports, function syntax, if/else, arithmetic, comparisons, logical operators, shifts, strings and comments. CI then applies several independent gates:

1. build the Rust toolchain;
2. run the Rust tests;
3. check every positive and negative Track example;
4. check, build and test the native compiler package through yard;
5. check every native compiler source through track;
6. verify the reported track and yard versions.

The latest workflow on the current main branch passes these gates. More importantly, the native component is no longer validated by visually inspecting token output. It participates in the repository’s regression boundary.

The backend decision: Cranelift here, C there

The next self-hosting step needs a backend, and this is where the bootstrap design becomes easy to overcomplicate.

The Rust bootstrap compiler will continue using Cranelift. Reimplementing Cranelift—or building a large native FFI surface for it—would turn self-hosting into a backend-binding project before Track has a complete native parser and checker.

Instead, the first self-hosted compiler will emit portable C and call the platform C compiler through Track’s explicit process API.

This is a smaller and more portable contract:

• the native compiler owns Track semantics;
• the C emitter owns a documented translation;
• the platform compiler owns machine-code generation;
• process execution stays explicit and observable.

It also separates two questions that are often mixed together: whether Track can implement its own compiler, and whether Track can implement a production machine-code backend. The first is required for self-hosting. The second can evolve independently.

What v0.8a is laying down

The current v0.8.0a work is intentionally foundations-first. Before beginning the parser, the repository defines four exit criteria:

• token parity between the native and bootstrap lexers;
• independently testable source spans, diagnostics, AST and collection modules;
• a checked-in C backend interface, including the exact external compiler command;
• deterministic module discovery and emitted-file ordering.

The order is important. If module order or generated filenames are unstable, a later bootstrap comparison becomes noisy. If source spans are bolted on after the parser, diagnostics inherit the wrong representation. If the token sets differ, every parser failure becomes ambiguous.

The immediate sequence is lexer parity, source spans and diagnostics, an AST and parser subset, then the C-emitter interface. The complete checker and code generator follow after those representations stop moving.

The actual self-hosting milestone

The roadmap calls v0.9.0 the bootstrap milestone. Its proposed verification has three compilation stages:

Rust track compiler  ──compiles──>  Track compiler, stage 1
Track stage 1        ──compiles──>  Track compiler, stage 2
Track stage 2        ──compiles──>  Track compiler, stage 3

The final gate compares the stage 2 and stage 3 binaries byte for byte.

That is stricter than “the compiler can compile its own source.” It asks whether a compiler produced by Track agrees exactly with the compiler that it produces next. Stable output requires deterministic module order, code emission and build metadata—the mundane details now being specified in v0.8a.

There is still a long distance between a native lexer and that result. The parser, linear checker, type inference, escape analysis and code generator remain Rust bootstrap components today. Generic structs are not fully monomorphized yet. The duplicated token definition needs to disappear. Lexer parity must be exhaustive, not approximate.

But the direction is now executable rather than aspirational.

Running the current milestone

Track requires a Rust toolchain to build the bootstrap compiler:

git clone https://github.com/dev-dami/track.git
cd track
cargo build --release

Then the self-hosted lexer can be checked, built and tested as a normal yard package:

cd compiler
../target/release/yard check
../target/release/yard build
../target/release/yard test

# Ask the native lexer to tokenize a small Track program
./target/trackc "let x = 42;"

The implementation and current design record are in the dev-dami/track repository. The most useful feedback right now is not syntax bikeshedding. It is a minimal program that makes ownership merging, module resolution, diagnostics or native code generation behave incorrectly.

That is what the self-hosting work is for: making the language large enough to find its own lies.