null theory.
← Back to notebook Build log, October 2026

Taking the transformer apart.

Implementing a small causal transformer on aaronnet, our in-house autograd engine, then running one careful ablation to see what each piece actually contributes.

In progress Neural architectures aaronnet

Starting question

What is learned inside a small causal transformer, and which architectural decisions make learning easier or harder? A useful answer should go beyond a diagram: it should identify specific operations, testable expectations, and measurable differences across controlled changes.

What we’re building

A small causal transformer, from scratch, on top of aaronnet , the lab’s in-house CPU autograd engine. Owning both the model and the engine keeps the stack small enough to read end to end, so every operation in the forward and backward pass is one we can inspect.

  • Token and position embeddings, and a stack of transformer blocks with masked scaled dot-product attention, residual connections, and layer normalization.
  • An autoregressive training loop with cross-entropy loss, written against aaronnet’s autograd.
  • Deterministic tests of the causal mask, tensor shapes, forward pass, and gradients.

The matrix M masks future positions for causal attention. The softmax converts the available attention scores into weights, and their weighted combination of the value vectors produces the output.

First experiment

One ablation: remove layer normalization while keeping everything else fixed. Train on a small, controlled synthetic task, across multiple seeds, with one fixed evaluation set and matched compute budgets for both conditions. Record final loss and accuracy per seed, plus a few selected internal activations.

The goal is not a benchmark number. It is to see whether a specific architectural piece changes the training trajectory in a way that reproduces across seeds, under matched conditions, on an evaluation set chosen before the runs.

Results

Runs in progress

When the runs are complete, this section will hold loss curves across seeds with and without layer norm, final evaluation accuracy per seed, and the exact training script and config.

Related work

  • Karpathy, nanoGPT.
  • Elhage et al. 2021, “A Mathematical Framework for Transformer Circuits.”
  • Olsson et al. 2022, “In-context Learning and Induction Heads.”
  • Power et al. 2022, “Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets.”
  • Nanda et al. 2023, “Progress measures for grokking via mechanistic interpretability.”

Our approach follows this line of work: reimplement at small scale, then change one component at a time and measure what moves.

Next note: aaronnet vs PyTorch