Tracr: Compiled Transformers as a Laboratory for Interpretability

ArXiv: 2301.05062

🎯 Pitch

Transformer interpretability has a dirty secret: we can’t tell if our tools actually work, because we never know what the model was supposed to be doing. This paper compiles human-readable programs directly into working transformers, giving researchers ground-truth models they can finally probe with confidence—and immediately finds that compression can silently rewrite a model’s algorithm without tanking accuracy.

1. Executive Summary

This paper introduces Tracr, a compiler that translates human-readable programs written in the RASP domain-specific language directly into the weights of standard decoder-only transformer models, providing models with fully known computational structure as ground truth for interpretability research. Using Tracr-compiled models implementing multi-step algorithms—including token frequency computation, sequence sorting, and Dyck-n parenthesis checking—the authors conduct a case study on superposition by applying gradient-descent-based compression, demonstrating that compressed models drop unnecessary features and represent less important features non-orthogonally while preserving task performance even with a residual stream compressed from 14 to as few as 6 dimensions. The compressed models achieve nearly perfect accuracy but may not preserve the original compiled mechanism faithfully—for instance, the sort_unique model learns to use a numerical encoding for target positions rather than the original categorical one-hot encoding—establishing that compression surfaces alternative algorithmic strategies as an artifact rather than a reliable ground-truth preservation technique.

2. Context and Motivation

The Core Problem: We Have No Ground Truth for Interpretability

The fundamental challenge this paper addresses is deceptively simple yet profound: how can we know whether an interpretability method actually works when we don't know what the model is supposed to be doing? This is the central obstacle facing mechanistic interpretability research, and Tracr offers a novel solution by providing models with fully known computational structure.

When researchers apply interpretability tools—classifier probes that try to locate where features are represented, gradient-based attribution methods that assign importance to inputs, or causal tracing that attempts to map information flow through a network—they produce explanations. But the model's true computational structure is unknown, making it impossible to verify whether these explanations are correct, partially correct, or entirely misleading. As the authors note in their Figure 1, this creates a situation where interpretability tools output explanations with no mechanism for asking "Is the explanation correct?"

This problem matters for several reasons that extend well beyond academic curiosity:

• Safety and reliability: As LLMs are deployed in high-stakes settings, understanding how they arrive at decisions becomes critical. If we cannot validate our interpretability methods, we cannot trust the explanations they produce, which undermines the entire enterprise of making AI systems transparent and accountable.

• Scientific understanding: Mechanistic interpretability aims to reverse-engineer the algorithms neural networks learn. Without ground-truth validation, the field risks developing a body of explanations that are internally consistent but empirically wrong—a kind of interpretability bubble where methods appear to work because we lack the means to falsify them.

• Method development: New interpretability techniques are typically validated on simple toy models or through post-hoc plausibility arguments. Neither approach tests whether the method recovers actual mechanisms as opposed to producing explanations that merely correlate with the model's behavior. This makes it difficult to iterate and improve methods systematically.

The authors explicitly connect this to the concept of faithfulness (Jacovi and Goldberg, 2020) and falsifiability (Leavitt and Morcos, 2020). An interpretation is faithful if it accurately describes the model's true reasoning process. But without ground truth, faithfulness is impossible to measure. Tracr provides that ground truth by ensuring the model's computation is known at every step.

Where Prior Approaches Fall Short

The paper identifies several existing strategies for addressing the ground-truth problem, each with significant limitations:

Synthetic data manipulation for image classifiers. Prior work (Yang and Kim, 2019; Adebayo et al., 2020; Zhou et al., 2022) introduced ground truth into image classification models by manipulating the training data—for example, introducing known label correlations into image backgrounds, or reassigning labels in controlled ways. These approaches are valuable but fundamentally limited: they only modify part of a model's behavior to have known interpretation, they focus exclusively on convolutional image classifiers, and they cannot provide anything close to a complete account of the model's computation. Tracr, by contrast, creates models where every weight and every computation has a known purpose.

Reverse-engineered circuits. The dominant approach in mechanistic interpretability has been to manually reverse-engineer circuits in trained models: identify which attention heads and MLP neurons implement specific behaviors, trace how information flows between them, and construct a bottom-up explanation of the model's computation (Cammarata et al., 2020; Olsson et al., 2022; Nanda et al., 2023; Wang et al., 2023). While this approach has yielded impressive insights—most notably the discovery of induction heads (Olsson et al., 2022) and the reverse-engineering of indirect object identification circuits in GPT-2 Small (Wang et al., 2023)—it suffers from two critical weaknesses:

1. Labor intensity: Reverse-engineering a circuit can take months of expert effort for even a single behavior in a medium-sized model. It does not scale.

2. Uncertain completeness: Even carefully analyzed circuits may be incomplete or wrong. The authors specifically cite Chan et al. (2022), who demonstrated that the influential "induction head" hypothesis needed modification to adequately explain in-context learning even in small attention-only transformers. If our best-studied circuits require revision, how confident can we be in less-scrutinized ones?

Reverse-engineered circuits can serve as a more realistic alternative to compiled models for evaluation, but they are not reliable ground truth—they are hypotheses about ground truth, and the whole point is that we need a way to test such hypotheses.

Theoretical constructions. There is a line of theoretical work showing that transformers can, in principle, implement certain computations (Wei et al., 2022; Akyürek et al., 2023; Giannou et al., 2023). Giannou et al. (2023) even propose a Turing-complete construction for transformers. But these are pencil-and-paper results rather than practical tools: they don't produce actual model weights, they don't address how the construction maps onto the constraints of real transformer architectures (alternating attention and MLP layers, fixed dimensionality, etc.), and they're primarily concerned with expressivity rather than providing a usable testbed.

The Gap Tracr Fills

The paper positions Tracr as occupying a unique and previously empty niche in the interpretability research ecosystem. It is not a theoretical result about what transformers could compute, but a practical compiler that produces actual model weights implementing specified algorithms. It is not a manually reverse-engineered circuit (labor-intensive and uncertain), but a fully constructed model where every computation is known by design. And it is not limited to image classifiers or partial ground truth, but provides complete knowledge of the model's structure.

The key insight is that compiled models serve as an intermediate step between toy models and real learned models. Toy models (like the two-layer networks studied in superposition research by Elhage et al., 2022b) are simple enough to understand completely, but they lack the architectural complexity of transformers and may not exhibit phenomena that emerge only at scale. Real models exhibit all the phenomena we care about, but we don't understand them. Compiled models provide models of realistic architectural complexity (multi-layer transformers with attention and MLPs, residual streams, multiple heads) but with fully known computation. They are complex enough to be non-trivial test cases for interpretability methods, but simple enough that we know the right answer.

This positioning is captured by the paper's metaphor: "The current approach to doing interpretability research is similar to trying to invent a microscope lens without ever being able to point it at familiar, well-understood shapes." Tracr gives researchers those familiar shapes—models they can point their interpretability microscopes at to calibrate, evaluate, and improve their methods.

The RASP Foundation and Why It Matters

Tracr builds on RASP (Weiss et al., 2021), a domain-specific language designed to capture the computational model of transformers. This choice is not arbitrary—RASP was explicitly created to help researchers "think like transformers" by providing primitives (elementwise operations, select-aggregate operations) that correspond naturally to transformer components (MLPs, attention). By compiling RASP to weights, Tracr closes the loop: it makes the correspondence between program and model not just conceptual but exact.

RASP's design maps cleanly onto the transformer circuits framework (Elhage et al., 2021), which the paper adopts as its theoretical lens. The transformer circuits perspective emphasizes (1) the residual stream as the central communication channel that all layers read from and write to, and (2) the factorization of attention into $W_{QK}$ (which computes attention patterns) and $W_{OV}$ (which reads information from the residual stream and writes the result back). RASP's s-ops correspond to information carried in the residual stream, selectors correspond to $W_{QK}$ matrices (attention patterns), and aggregate operations correspond to $W_{OV}$ matrices (value computations). This tight correspondence is what makes compilation feasible—each RASP construct has a natural translation to transformer components.

Superposition as a Motivating Case Study

The paper uses superposition—the phenomenon where neural networks represent more features than they have dimensions by using non-orthogonal embeddings—as its primary case study (Section 5). This choice is strategic. Superposition has been identified as potentially central to understanding how large models work (Elhage et al., 2022a; Olah, 2022), but prior work has only studied it in extremely simple settings: two-layer networks or toy models implementing trivial algorithms. The authors explicitly note that superposition "has not yet been studied in detail for models with more than two layers or in transformer models executing multi-step algorithms."

Tracr enables this study because it provides multi-layer transformers executing non-trivial algorithms (sorting, parenthesis checking) with known feature representations. By compressing these models via gradient descent, the authors can observe superposition emerging in a controlled setting where they know exactly which features exist, which depend on each other, and which are necessary for the task. This allows them to ask questions that were previously inaccessible: How does the structure of the algorithm affect which features get stored in superposition? Do features that encode redundant information share embedding dimensions? Do compressed models preserve the original algorithm or discover alternative implementations?

The compression experiment thus serves a dual purpose: it demonstrates one concrete application of Tracr (studying a phenomenon that was hard to study otherwise), and it reveals limitations of the approach (compressed models may not preserve the known computation, as the paper finds with sort_unique), which is itself an important finding for how compiled models should be used.

How Tracr Relates to the Broader Interpretability Landscape

The paper positions Tracr not as a replacement for other interpretability approaches but as a complement that enables new kinds of research. Several potential use cases are outlined in detail in Appendix A.1:

• Test cases for methods: Classifier probes, causal tracing, and attribution methods can all be evaluated on models where the correct answer is known. If a probe claims feature F is represented in layer L, but the compiled model actually computes F in a different layer, the method has failed in a measurable way. This enables quantitative benchmarking of interpretability tools for the first time.

• Component replacement: Following Nanda et al. (2023), who validated their understanding of a modular addition circuit by replacing model components with hand-coded versions and observing improved performance, Tracr-compiled components could be used to test hypotheses about specific subcircuits in larger models. If replacing a suspected circuit with a compiled equivalent improves task performance, that's strong evidence the interpretation was correct.

• Building intuition: By examining how Tracr compiles specific algorithms into weights, researchers can develop a more concrete "imagination" for what transformer mechanisms look like in practice. The detailed forward-pass visualizations (Figures 2, 5, 9–11) serve as instructional examples of how attention heads and MLPs coordinate to implement multi-step procedures.

The Fundamental Limitation the Paper Acknowledges

The paper is unusually candid about the scope of its contribution. A key limitation acknowledged in Section 7 and Appendix A.2 is that compiled models are fundamentally simpler than real models—"we will likely never compile fully featured language models in Tracr." This means results from compiled models should be treated as a "minimum bar" rather than a full validation, and that some phenomena present in real models may simply not appear in compiled ones (and vice versa).

This limitation is important because it prevents overclaiming. Tracr doesn't solve interpretability; it provides a laboratory for developing and testing interpretability tools under controlled conditions. The expectation is that methods that work on compiled models are candidates for application to real models, but they still need validation in that more complex setting. Conversely, methods that fail on compiled models can be confidently discarded—if a technique can't recover the known structure of a simple compiled model, it's unlikely to succeed on a real one.

3. Technical Approach

3.1 Reader Orientation

Tracr is a compiler that translates human-readable programs written in the RASP domain-specific language directly into the concrete weight matrices of standard decoder-only transformer models, producing models whose computational structure is fully known at every layer and every neuron. The problem Tracr solves is the absence of ground-truth explanations in interpretability research: when applying an interpretability tool to a trained neural network, researchers cannot verify whether the tool's output correctly describes the model's actual computation because that computation is unknown. Tracr's solution is to construct models where the computation is specified by design, then use these models as test cases where the correct interpretation is available for comparison.

3.2 Big-Picture Architecture (Diagram in Words)

The Tracr system has six major components arranged as a pipeline that transforms a RASP program into executable transformer weights:

1. RASP Program Input: a human-written program in the RASP language (embedded in Python) specifying the desired computation as a graph of sequence operations, elementwise transformations, and select-aggregate operations.
2. Computational Graph Builder (Step 1): traces the entire RASP program to produce a directed graph where nodes represent operations (elementwise, select-aggregate) and edges represent data dependencies between them. Source nodes are the built-in primitives tokens and indices.
3. Value Inference Engine (Step 2): traverses the computational graph and annotates each node with the set of all possible values its output can take, using deterministic computation, finite vocabulary, and finite context length to guarantee finite output sets.
4. Component Translator (Step 3): independently converts each node in the computational graph into a model block—elementwise operations become MLP layers, select-aggregate operations become attention heads—using a library of manually engineered MLP and attention blocks.
5. Layer Allocator (Step 4): assigns each model block to a specific layer in the transformer architecture, respecting the constraint that the architecture alternates attention and MLP layers and that all data dependencies must be satisfied in order. This is formulated as a combinatorial optimization problem and solved heuristically.
6. Model Constructor and Weight Assembler (Steps 5–6): constructs the residual stream as the direct sum of all components' input and output spaces, stacks components in layer order to produce a Craft representation, then translates Craft to concrete weight matrices by combining parallel layers, factoring attention matrices, and inferring the final model configuration (depth, width, residual stream size).

Information flows through these components sequentially: the program enters as RASP source code, gets traced into a graph, gets annotated with value information, gets translated block-by-block into model components, gets assigned to layers, and emerges as a complete set of transformer weights executable in a standard decoder-only architecture.

3.3 Roadmap for the Deep Dive

• First, the RASP language modifications (Appendix B): Tracr's technical constraints on RASP—the prohibition of composite selectors, the required encoding annotations, and the mandatory beginning-of-sequence token—because these constraints fundamentally shape everything the compiler can and cannot translate.
• Second, Steps 1–2 of compilation: computational graph construction and value inference, since these establish the intermediate representation that all subsequent steps operate on.
• Third, Step 3 of compilation: the independent translation of s-ops to model components, covering the MLP construction strategies (categorical lookup tables, numerical piecewise linear approximations) and attention head construction (the direct attention matrix, the BOS token mechanism, and the temperature scaling).
• Fourth, Steps 4–6 of compilation: layer allocation, model construction in Craft, and final weight matrix assembly, showing how independently constructed components get assembled into a complete transformer.
• Fifth, the selector width primitive (Appendix D.2): a special-case compilation strategy that compiles a single RASP operation to multiple model layers, illustrating how Tracr handles operations that don't have a one-to-one mapping to transformer components.
• Sixth, causal attention support (Appendix D.3): how Tracr extends RASP and the compilation process to support causal (autoregressive) attention masking, and what changes are required in RASP programs to function under this constraint.
• Seventh, the theoretical limits of selector combination (Appendix F): the formal lemma and proof showing that boolean combinations of selectors cannot in general be compiled to attention weights, which explains a key restriction of the system.
• Eighth, the compression training setup (Section 5 and Appendix E): the gradient-descent-based procedure for compressing compiled models into smaller residual streams, which is both a demonstration application and a method for producing more realistic models.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and compilation paper whose core technical contribution is the pipeline that translates RASP programs into transformer weights, with the compression procedure as a secondary contribution that both demonstrates the system's utility and reveals its limitations.

RASP Modifications Required for Compilation (Appendix B)

Tracr does not compile arbitrary RASP programs. It imposes three types of restrictions on the RASP language, each motivated by the constraints of real transformer architectures. The authors argue that these restrictions are not severe in practice: "we could implement programs to solve all tasks described by Weiss et al. (2021)."

Restriction 1: No composite selectors with different input variables. Standard RASP allows boolean combinations of selectors, such as:

select(A, B, ==) and select(C, D, ==)

where A, B, C, and D are four different s-ops. The problem, as the paper explains (Appendix B), is that "a real attention pattern only has two input vector spaces"—the query space and the key space. A transformer attention head computes attention logits as $x W_{QK} x^T$, which is a bilinear form operating on the residual stream at two positions. It cannot simultaneously compare four different subspaces of the residual stream because the query at position $i$ and the key at position $j$ must both be vectors in the same residual stream—the attention mechanism has no way to distinguish which dimensions correspond to which s-ops when combining selectors that operate on different input pairs.

The paper provides a mechanical refactoring strategy for cases where composite selectors are unavoidable: construct product-space s-ops that concatenate the relevant values. For the example above, one would create two s-ops whose values are pairs over the values of A and B, and C and D respectively, then use an atomic selector on these composite s-ops:

select(
  product(A, B),
  product(C, D),
  lambda (a,b), (c,d): a==c and b==d
)

However, Tracr does not implement this refactoring automatically for two reasons stated explicitly in Appendix B: (1) "without compression it is inefficient: the s-op dimensions scale as a product of the input s-op dimensions," and (2) "doing this automatically would break the 1-1 correspondence between selectors in RASP and attention heads in the compiled model: the compound s-ops require MLP blocks." The first reason is about computational cost—the residual stream dimensionality would explode combinatorially. The second is about conceptual clarity—the mapping from program constructs to model components would become indirect, making the compiled model harder to interpret as ground truth.

This restriction connects to a deeper theoretical result in Appendix F, which proves that boolean combinations of selectors with different input variables cannot in general be implemented by a single attention head's $W_{QK}$ matrix—a result we will examine in detail later.

Restriction 2: Mandatory encoding annotations for s-ops. RASP s-ops must be explicitly annotated as either categorical or numerical. This is necessary because Tracr must decide how to represent each s-op in the residual stream, and the representation determines which operations are possible:

• Categorical variables are encoded as one-hot vectors in a dedicated orthogonal subspace of the residual stream. If a categorical s-op can take $k$ possible values, it occupies $k$ dimensions. For example, a tokens s-op with vocabulary {"x", "y", "z"} would occupy three dimensions, with the dimension tokens:y having value 1 at positions where the input token is "y" and 0 elsewhere.
• Numerical variables are encoded as the scalar magnitude in a dedicated one-dimensional subspace of the residual stream. The value is stored directly as a real number in that single dimension.

The paper notes that "categorical encoding generally uses more dimensions and often requires an extra decoding step" compared to numerical encoding, but some aggregate operations only work with one encoding type. Specifically, "aggregation with a mean across token positions is natural for numerical encodings but not categorical ones." If an attention head attends to multiple positions and computes a weighted average of categorical one-hot vectors, the result is a vector of fractions (e.g., [0.3, 0.7]) rather than a valid one-hot encoding—it no longer represents a single categorical value unambiguously. In contrast, averaging numerical values (scalars) produces a valid numerical value. Thus, the encoding annotation constrains which s-ops can be inputs to aggregation operations.

S-ops are categorical by default, and the programmer must explicitly mark numerical s-ops. This annotation propagates through the compilation: elementwise operations on categorical inputs produce categorical outputs (via lookup tables), elementwise operations on numerical inputs produce numerical outputs (via piecewise linear approximations), and select-aggregate operations can handle either as values but require categorical keys and queries.

Restriction 3: Mandatory beginning-of-sequence token. All input sequences must start with a dedicated BOS token. The paper explains that this is "crucial when implementing arbitrary attention patterns." The specific problem is that RASP selectors can produce all-zero rows (rows where no key is selected for a given query). For example, select(indices, indices, <) produces a row of all zeros at position 0 because no indices are less than 0. But in a real transformer, the softmax operation makes all-zero attention logits impossible—the softmax of a vector of zeros is a uniform distribution over all positions, meaning the attention head would attend to all tokens equally rather than attending to none.

Tracr's solution is to use the BOS token as a "default" position that is attended to when no other token meets the selection criteria. The mechanism works as follows (described in Appendix D.1): Tracr adds a rank-one bias term to the attention logit matrix that causes the BOS token to be attended to either always or whenever no other token is selected. This is implemented by adding $W_{BOS} = \beta_{BOS} x_{\text{one}} x_{\text{tokens:bos}}^T$ to the direct attention matrix, where $x_{\text{one}}$ is a unit vector for a special embedding dimension that always contains the value 1, $x_{\text{tokens:bos}}$ is a unit vector for the BOS token embedding dimension, and $\beta_{BOS}$ is either 1 (attend to BOS always) or $\frac{1}{2}$ (attend to BOS only when no other token is selected). The $x_{\text{one}}$ dimension serves as a constant-1 bias that can be read by any attention head's query, and the $x_{\text{tokens:bos}}$ dimension identifies the BOS position in the key.

The paper observes that this choice "comes from practical considerations; but, interestingly, real models sometimes show similar behaviour (e.g., see Elhage et al., 2021)." This suggests the BOS-as-default mechanism, though introduced for compilation convenience, may correspond to a strategy that naturally emerges in trained transformers.

Step 1: Computational Graph Construction

The first compilation step traces the entire RASP program to produce a directed acyclic graph representing the computation. This operates as follows:

Input. The RASP program expressed as Python code using Tracr's RASP library. Each RASP operation (elementwise transformation, select, aggregate, selector_width) is a function call that creates a node in an expression graph. The program incrementally builds this graph as it executes.

Process. Tracr traverses the expression graph and constructs an explicit computational graph. The graph has:

• Source nodes representing the built-in primitive s-ops tokens and indices. These have no incoming edges and provide the initial data for the computation.
• Internal nodes representing each operation in the RASP program. An elementwise operation node has one or two incoming edges (from the s-ops it operates on) and one outgoing edge (to the resulting s-op). A select node has two incoming edges (the key and query s-ops) and produces a selector. An aggregate node has two incoming edges (a selector and a value s-op) and produces a new s-op.
• A sink node representing the output s-op of the program. This has incoming edges from the final operations and no outgoing edges.

Output. A directed graph structure where edges represent data dependencies and nodes represent computational operations. This graph is the intermediate representation for all subsequent compilation steps. Importantly, this graph is not yet a transformer—it represents the logical structure of the computation without any commitment to how operations map to layers or how data is encoded.

Why this approach. Building an explicit computational graph separates the program's logical structure from the compilation details. This separation allows subsequent steps (value inference, component translation, layer allocation) to operate on a clean, well-defined representation. It also enables optimizations: for example, Tracr combines consecutive elementwise operations into a single s-op at this stage, reducing the number of nodes that need to be translated to model components.

Step 2: Value Inference

The second compilation step annotates each node in the computational graph with the set of all possible values its output can take. This step is necessary because Tracr uses categorical encodings (one-hot vectors) for most s-ops, and to determine the dimensionality of the one-hot encoding, the compiler must know the set of possible values.

Process. Tracr traverses the computational graph from source nodes to sink, computing for each node the set of possible output values based on the possible input values. The traversal uses simple heuristics that guarantee finding a superset of the actual possible values—the annotated set may contain values that the s-op never takes in practice for any specific input, but it will never miss a value that could occur. This conservative over-approximation is acceptable because including extra values only wastes residual stream dimensions (allocating a dimension for a value that never occurs) but does not cause incorrect computation.

Why a finite set is guaranteed. The paper explains: "all nodes have a finite set of output values because computations are deterministic, and we have a finite input vocabulary and context size." The input vocabulary is finite (the set of tokens the program can process), and the context size is finite (the maximum sequence length). Since all RASP operations are deterministic functions of their inputs, and the inputs come from a finite set, every s-op can take only finitely many distinct values. This property is what makes categorical encoding feasible—without it, the one-hot encoding would require unknown or unbounded dimensionality.

Example from Figure 4(a). For the frac_prevs program with vocabulary {x, y} and context size 3:

• tokens can take values {x, y}.
• indices can take values {0, 1, 2}.
• The elementwise operation is_x = (tokens == "x") can take values {0, 1} (a boolean, encoded numerically).
• The selector prevs = select(indices, indices, <=) does not produce an s-op value but a selection matrix; it is annotated differently.
• The aggregate operation frac_prevs = aggregate(prevs, is_x) computes weighted averages of 0s and 1s with varying numbers of terms, so it can take values {0, 0.25, 0.5, 0.75, 1} (or a superset thereof, depending on the precision of the inference heuristic).

Downstream use. These value annotations determine the encoding dimensionality for categorical s-ops (one dimension per possible value) and the discretization buckets for numerical MLPs (the MLP's piecewise linear approximation is designed to be exact at the known possible input values and interpolate between them). The annotations are stored as metadata on each node in the computational graph and used by the component translator in Step 3.

Step 3: Independent Component Translation (MLP and Attention Blocks)

This is the core of the compilation process. Each node in the computational graph is translated independently into a model block—either an MLP layer or an attention head—without considering where in the transformer architecture the block will be placed or how it will interact with other blocks. The translation uses a library of manually engineered components that implement specific computational primitives. This section also covers how RASP selectors are converted into attention weight matrices.

MLP Blocks: Categorical Inputs and Outputs

For elementwise operations with categorical inputs and categorical outputs, Tracr constructs MLPs that act as lookup tables. This is the simplest case because a function from a finite set to a finite set can be implemented exactly by a two-layer MLP.

Construction for single-input Maps (one categorical input, one categorical output). A Map operation f(x) where $x$ can take $k$ possible categorical values (encoded as a $k$-dimensional one-hot vector in its dedicated subspace) and the output can take $m$ possible categorical values (encoded as an $m$-dimensional one-hot vector in a different dedicated subspace):

• The first layer weight $W_1$ (of shape $d \times h$, where $d$ is the residual stream dimension and $h$ is the hidden dimension) contains a submatrix that reads the input one-hot encoding and maps it to the corresponding output encoding. Specifically, for each input value $v_i$, the weight vector connecting the input dimension for $v_i$ to the output dimensions encodes the one-hot representation of $f(v_i)$.
• The second layer weight $W_2$ (of shape $h \times d$) writes the result into the output s-op's dedicated subspace.
• The ReLU nonlinearity is not strictly needed for this mapping (it could be implemented with a linear layer if negative values are not involved), but it is present in the architecture. If the mapping requires it, the first layer can produce positive activations for the correct output dimensions and negative or zero activations for others; the ReLU zeros out the negatives, producing the desired one-hot encoding.

Construction for SequenceMaps (two categorical inputs, one categorical output). A SequenceMap f(x, y) where $x$ has $k$ values and $y$ has $\ell$ values can produce up to $k \times \ell$ possible input pairs, so the output could have up to $k \times \ell$ values. Tracr's construction works in two stages:

1. The first MLP layer maps the two one-hot input encodings to an encoding of all pairs of inputs. Conceptually, this creates a $k \times \ell$-dimensional representation where each dimension corresponds to a specific pair $(x_i, y_j)$. This is implemented by having the $W_1$ matrix contain weight vectors that detect conjunctions: a dimension in the hidden layer activates (via ReLU) only when both the specific $x_i$ and the specific $y_j$ dimensions are active.
2. The second MLP layer acts as a lookup table from the pair encoding to the output one-hot encoding.

Why this approach over alternatives. An alternative would be to implement SequenceMaps as two-layer MLPs that approximate the function directly without the intermediate pair encoding. However, the pair-encoding approach guarantees exact implementation for any function on finite sets because it enumerates all possible input combinations and maps each to its correct output. The cost is hidden layer dimensionality proportional to $k \times \ell$, which can be large but is manageable for the vocabulary sizes and context lengths used in the paper's examples.

MLP Blocks: Numerical Inputs and Outputs

For elementwise operations with numerical inputs and numerical outputs, Tracr constructs MLPs as universal function approximators using a discretization strategy.

Construction. The MLP uses two layers:

1. Discretization layer ($W_1$): The first layer partitions the input space into buckets around the known possible input values. For each possible input value $v_i$ that the s-op can take (as determined by Step 2's value inference), the layer includes a neuron that activates when the input is close to $v_i$. Specifically, the neuron computes $\text{ReLU}(x - (v_i - \epsilon)) - \text{ReLU}(x - (v_i + \epsilon))$, which produces a triangular "bump" function centered at $v_i$ with width $2\epsilon$. By choosing $\epsilon$ smaller than half the minimum distance between any two possible input values, these bump functions are non-overlapping, meaning at most one neuron activates for any input that exactly equals one of the possible values.

2. Output mapping layer ($W_2$): The second layer maps each bucket to its corresponding output value. For a bucket centered at input value $v_i$, the weight connecting that bucket's neuron to the output dimension is set to $f(v_i)$. Since the bump functions are non-overlapping for the known input values, the MLP exactly implements $f$ on those values. For inputs between known values, the MLP linearly interpolates between the outputs of the nearest buckets, providing a continuous approximation.

Why this form. The discretization approach guarantees that the MLP is exact on all values that the s-op can actually take (as determined by the value inference in Step 2). This is crucial because even small approximation errors could compound across multiple layers and cause the compiled model to compute a different function than the RASP program specifies. Between known values, the MLP interpolates, but those intermediate values should never occur in practice given the deterministic computation and finite inputs, so the interpolation behavior is irrelevant to correctness. Alternative approaches like training MLPs to approximate the function using gradient descent would not guarantee exactness on the required input values.

Attention Blocks: Selectors to $W_{QK}$

The translation of select-aggregate operations into attention heads is the most technically involved part of Step 3. A select-aggregate operation in RASP consists of two parts: a selector (which produces a binary $N \times N$ matrix indicating which key positions to attend to for each query position) and an aggregate (which computes a weighted average of a value s-op using the selection matrix as weights). The selector becomes the attention pattern (via $W_{QK}$), and the aggregate becomes the value computation (via $W_{OV}$).

The direct attention matrix construction. For a selector select(query, key, pred) where query and key are categorical s-ops with one-hot encodings in orthogonal subspaces of the residual stream, Tracr constructs a direct attention matrix $\tilde{W}_{QK} \in \mathbb{R}^{d \times d}$. This matrix is designed such that for any query vector $q$ (the residual stream at the query position) and any key vector $k$ (the residual stream at a key position), the bilinear form $q^T \tilde{W}_{QK} k$ equals 1 if the predicate is true and 0 if it is false.

The construction exploits the one-hot structure of the categorical encodings:

• The row space of $\tilde{W}_{QK}$ is the subspace where the query s-op is stored. The column space is the subspace where the key s-op is stored. All other dimensions of the residual stream are ignored (the corresponding rows and columns of $\tilde{W}_{QK}$ are zero).
• Within the query subspace, each possible query value $q_i$ corresponds to a specific dimension. Within the key subspace, each possible key value $k_j$ corresponds to a specific dimension.
• The entry $\tilde{W}_{QK}[q_i, k_j]$ (the element at the row corresponding to query value $q_i$ and the column corresponding to key value $k_j$) is set to 1 if pred(q_i, k_j) is true, and 0 otherwise.

When the query position has query value $q_i$ (meaning dimension $q_i$ in the query subspace is 1 and all others are 0) and the key position has key value $k_j$, the product $q^T \tilde{W}_{QK} k$ extracts exactly $\tilde{W}_{QK}[q_i, k_j]$, which is 1 if the predicate holds and 0 otherwise. This produces the desired binary attention logits.

The BOS token adjustment. As discussed under "Restriction 3," RASP selectors can produce all-zero rows (no key selected for a given query), but the softmax makes this impossible. Tracr adjusts the direct attention matrix by adding a rank-one update:

$\tilde{W}_{QK}^{\text{adjusted}} = \tilde{W}_{QK} + W_{BOS}$

where $W_{BOS} = \beta_{BOS} \, x_{\text{one}} \, x_{\text{tokens:bos}}^T$.

Here:

• $x_{\text{one}}$ is a unit vector in the residual stream along the dimension that always contains the value 1 (an auxiliary dimension provided for bias purposes),
• $x_{\text{tokens:bos}}$ is a unit vector along the dimension corresponding to the BOS token embedding,
• $\beta_{BOS}$ is a scalar: 1 if the BOS token should be attended to always, or $\frac{1}{2}$ if the BOS token should be attended to only when no other token is selected.

What this computes. The product $q^T W_{BOS} k$ equals $\beta_{BOS}$ when the key position is the BOS token (because $x_{\text{tokens:bos}}$ is active only there) and $x_{\text{one}}$ is always active at every query position. It equals 0 for all non-BOS key positions. Therefore, the adjusted logit for attending to the BOS token is $\beta_{BOS}$ (plus whatever the original $\tilde{W}_{QK}$ contributes, which is typically 0 for the BOS token since selectors usually don't select it). The adjusted logit for attending to a non-BOS selected token is 1. The adjusted logit for attending to a non-BOS unselected token is 0.

Temperature scaling. After constructing the adjusted direct attention matrix, Tracr scales it by an inverse temperature:

$W_{QK} = T^{-1} \left( \tilde{W}_{QK}^{\text{adjusted}} \right)$

where $T^{-1}$ is set to 100 by default.

What this computes. The attention weights after softmax are:

$A_{ij} = \frac{\exp(q_i^T W_{QK} k_j)}{\sum_{j'} \exp(q_i^T W_{QK} k_{j'})}$

With $T^{-1} = 100$, the exponentiated logits are approximately $\exp(100) \approx 2.7 \times 10^{43}$ for selected non-BOS tokens, $\exp(0) = 1$ for unselected non-BOS tokens, and $\exp(100 \cdot \beta_{BOS})$ for the BOS token. The softmax then produces attention weights that are essentially uniform over the selected tokens (each gets weight approximately $1 / s$ where $s$ is the number of selected tokens) and nearly zero elsewhere.

Why this form. The temperature scaling "sharpens" the softmax to approximate a hard selection: selected tokens get equal attention probability, unselected tokens get approximately zero probability. Without temperature scaling (or with $T = 1$), the softmax would produce a softer distribution where unselected tokens still get non-negligible attention weight—for example, $\exp(1) / (\exp(1) + \exp(0)) \approx 0.73$ for the selected token vs. 0.27 for the unselected, rather than approximately 1.0 vs. 0.0. The high temperature ensures the attention pattern faithfully implements the binary RASP selector. The use of the softmax (rather than a hard argmax) preserves differentiability, which matters for the compression experiments in Section 5 even though the compiled model itself does not require differentiability.

Attention Blocks: Aggregate to $W_{OV}$

The aggregate operation takes the attention pattern (computed by $W_{QK}$) and uses it to compute a weighted average of a value s-op. This is implemented by the $W_{OV}$ matrix.

Construction. The $W_{OV}$ matrix reads the value s-op from the residual stream (at each key position), multiplies by the attention weight for that position, and writes the result into the output s-op's dedicated subspace. Concretely:

• The $W_{OV}$ matrix has its row space in the value s-op's subspace and its column space in the output s-op's subspace.
• For categorical value s-ops with one-hot encoding, $W_{OV}$ copies the value dimensions to corresponding output dimensions. However, as noted earlier, this only works if the attention head never attends to more than one token: if it attends to multiple one-hot vectors, their weighted average is no longer one-hot and cannot be decoded as a categorical value. For selectors that select at most one key per query (e.g., == predicates), categorical values are safe. For selectors that select multiple keys (e.g., <= predicates), the value should be numerical.
• For numerical value s-ops, $W_{OV}$ reads the scalar value from the dedicated dimension and writes it to the output's dedicated dimension, scaled by the attention weight. The weighted sum over key positions is computed by the attention mechanism. If the selector selects multiple keys with equal weight, this implements an arithmetic mean; if weights are non-uniform, it implements a weighted mean.

Why this separation. The $W_{QK}$ and $W_{OV}$ factorization is a direct consequence of the transformer circuits perspective (Elhage et al., 2021). It cleanly separates the "where to attend" computation ($W_{QK}$) from the "what to compute" computation ($W_{OV}$). This separation makes Tracr's compilation modular: a selector can be compiled to $W_{QK}$ independently of what value will be aggregated, and an aggregate operation can be compiled to $W_{OV}$ independently of what attention pattern will be used. In the compiled model, these two matrices are combined into a single attention head that performs both computations.

Causal Attention Support (Appendix D.3)

Tracr supports two evaluation modes: non-causal (bidirectional) attention, which is RASP's default semantics, and causal (autoregressive) attention, which is used by most deployed transformer models and applies a mask preventing each position from attending to future positions.

RASP-level changes. Tracr extends RASP with a flag that switches evaluation to causal mode. When causal evaluation is active, a causal mask is applied to the output of every selector: entries where the query position $i$ is less than the key position $j$ are set to 0. This changes the semantics of select-aggregate operations—the aggregate now computes a weighted average only over previous and current positions, not future ones.

The paper notes that "it is necessary to adapt RASP programs to function with causal attention. For example, the frac_prevs program no longer needs to compute a causal mask manually"—in non-causal mode, frac_prevs must explicitly use select(indices, indices, <=) to restrict attention to previous tokens, but in causal mode, this restriction is automatically enforced by the architecture. However, some programs break under causal evaluation: "the length implementation by Weiss et al. (2021) no longer correctly computes the length of a sequence because it requires attending to future tokens."

Compiler-level changes. The compilation process is largely unchanged. The only modification is in Step 6 (weight matrix assembly): when causal compilation is enabled, the assembled attention heads include a causal mask in their forward pass. The mask is a lower-triangular matrix of ones (allowed attention) and zeros (blocked attention) applied before the softmax, setting the logits for future positions to negative infinity so their post-softmax attention weights are zero.

This design means that the same RASP program can be compiled to either a bidirectional or causal transformer by changing a flag, without modifying the program logic (though the program's correctness under causal masking depends on whether it was designed with causal constraints in mind).

Step 4: Layer Allocation

Once each node in the computational graph has been independently translated into a model block (MLP or attention), Tracr must assign these blocks to specific layers in the transformer architecture. This is the layer allocation problem.

Constraints. The transformer architecture imposes strict structural constraints:

1. Alternating layers. The model consists of alternating attention and MLP layers. The specific pattern is: input embedding → [attention, MLP] → [attention, MLP] → ... → output unembedding. Each block's type (attention or MLP) must match the layer it is assigned to.
2. Data dependency order. If block B depends on the output of block A (there is a directed path from A to B in the computational graph), then A must be assigned to an earlier layer than B. This ensures that when B executes, the information it needs is already present in the residual stream.
3. Residual stream persistence. All information written to the residual stream remains available to all subsequent layers. This means a block does not need to be in the immediately following layer after its dependencies—it can be arbitrarily far downstream, as long as it comes after.

Formulation as combinatorial optimization. The paper states: "we can generally formulate this as a combinatorial optimization problem" with the objective of finding the smallest model (fewest layers) that satisfies all constraints. This is a form of scheduling problem with precedence constraints and resource constraints (each layer can hold at most one attention block and one MLP block).

Heuristic solution. "For scope reasons, we solve this problem heuristically." The heuristic works as follows:

1. Longest-path bound. For each node in the computational graph, compute the length of the longest path from any input node to that node. This path length provides an upper bound on the earliest layer to which the node can be assigned—a node cannot be in a layer earlier than its longest-path distance from the inputs, because all its dependencies must be computed first.

2. Greedy assignment with parallelization. Nodes are assigned to layers in order of their longest-path bound. When multiple nodes can be placed in the same layer (they have compatible types and their dependencies are satisfied), Tracr attempts to combine them to minimize total layer count. For example, if two independent MLP blocks both have their dependencies satisfied by layer 2, they can both be placed in layer 2 and executed in parallel (their weight matrices will be concatenated in the assembled model).

3. No-op insertion. The assembled model must strictly alternate attention and MLP layers. If the heuristic produces a sequence where two attention layers would be adjacent, a no-op layer of the opposite type is inserted between them. A no-op attention layer has $W_{QK} = 0$ and $W_{OV} = 0$ for all heads (or equivalently, a single head that computes the identity function by reading from and writing to orthogonal subspaces with zero attention weights, effectively doing nothing). A no-op MLP layer has weights that implement the identity function (or simply zero weights, relying on the residual connection to pass information through unchanged).

The paper acknowledges that this heuristic "returns a correct but sometimes suboptimal layer allocation." Finding the truly optimal allocation would require solving an NP-hard scheduling problem, so the heuristic trades optimality for tractability. In practice, the compiled models for the paper's examples use 2–4 effective layers plus 1–2 no-op layers, which is reasonably efficient.

Example from Figure 4(c). For the frac_prevs program:

• The is_x MLP has longest-path 0 (it depends only on tokens, which is available at layer 0). It is assigned to the first MLP layer (MLP 1).
• The prevs attention block has longest-path 1 (it depends on indices, which is available at layer 0, but it cannot be in the same layer as is_x because they have different types). It is assigned to the second attention layer (Attn 2).
• This creates a sequence Attn 1 (no-op), MLP 1 (is_x), Attn 2 (prevs), MLP 2 (no-op). The no-ops are inserted to maintain alternation: the model must start with an attention layer after the embedding, so Attn 1 is a no-op; and it must end with an MLP layer before the unembedding, so MLP 2 is a no-op.

Step 5: Model Construction in Craft

With all blocks assigned to layers, Tracr constructs the full model in its intermediate representation language called Craft.

Craft as an assembly language. The paper describes Craft as an "assembly language" that "provides a transformer implementation using vector spaces with labelled basis dimensions and operations on them. This lets us define projections or other linear operations in terms of basis direction labels, which simplifies constructing model components that act on different vector spaces." In other words, Craft operates at the level of named subspaces of the residual stream rather than raw weight matrices, making it easier to compose components that read from and write to different parts of the residual stream without manually tracking indices.

Residual stream construction. The residual stream space is constructed as the direct sum of all model components' input and output spaces. This is the key design decision that makes Tracr models interpretable: "we embed each s-op in its own orthogonal subspace, which is reserved for its sole use throughout the entire network." Concretely:

• If the program has s-ops tokens (vocabulary size $V$), indices (context length $C$), and intermediate s-ops is_x (1 numerical dimension), prevs (no direct residual stream representation—it's an attention pattern), and frac_prevs (1 numerical dimension), the residual stream dimensionality is $V + C + 2 + \text{auxiliary dimensions}$. The auxiliary dimensions include the BOS token embedding, the constant-1 dimension, and any others needed for the compilation.
• Each s-op's subspace is present in the residual stream at all layers, even layers before the s-op is computed (where it contains zeros) and layers after it is no longer needed (where it persists but is unused). This is the "sparse and inefficient" allocation that the paper acknowledges.

Stacking components. Tracr traverses the computational graph in the order determined by the layer allocation and "stacks" the components: for each layer, it takes all blocks assigned to that layer and composes them into a single Craft layer representation. The composition is straightforward because blocks act on orthogonal subspaces (they read from their input s-op subspaces and write to their output s-op subspaces, which are disjoint from all other subspaces), so they can be combined by concatenating their weight matrices in the appropriate dimensions.

Output. A complete transformer model represented in Craft—all layers, all heads, all MLP neurons, with explicit labeled basis dimensions for every part of the residual stream and every weight matrix. This representation is independent of any specific transformer implementation (it does not assume a particular code library or weight matrix layout), which is why the paper can say "Models compiled by Tracr can be translated into weights of any standard decoder-only transformer model (without layer norm)."

Step 6: Weight Matrix Assembly

The final step translates the Craft representation into concrete model weights compatible with the Haiku transformer implementation.

Combining parallel layers. In each layer, multiple MLP blocks (if multiple were assigned to the same layer) are combined into a single MLP layer by concatenating their hidden dimensions. Similarly, multiple attention heads are combined into a single multi-headed attention layer. Because the blocks operate on orthogonal subspaces, this concatenation is straightforward: the weight matrices of the individual blocks are placed into block-diagonal submatrices of the combined weight matrix.

Factoring $W_{QK}$ and $W_{OV}$. In Craft, attention heads are represented by their $W_{QK}$ and $W_{OV}$ matrices. Standard transformer implementations use separate $W_Q$, $W_K$, $W_V$, and $W_O$ matrices. Tracr performs the factorization:

• $W_Q$ and $W_K$ are computed to satisfy $W_{QK} = W_Q W_K^T / \sqrt{d_k}$. In the compiled models, $d_k$ (the per-head key/query dimension) is typically set to the rank of $W_{QK}$, which equals the number of dimensions in the query/key subspaces. The factorization is exact because $W_{QK}$ is constructed to be low-rank (its row and column spaces are the query and key subspaces, respectively).
• $W_V$ maps the residual stream to a value space of dimension $d_v$. For Tracr heads, $d_v$ equals the dimensionality of the value s-op's subspace. The $W_V$ matrix is simply a projection onto that subspace.
• $W_O$ maps the head's output back to the residual stream. Since the head writes to its output s-op's dedicated subspace, $W_O$ is a projection onto that subspace. When $W_O$ is combined across heads (the full $W_O$ is the concatenation of per-head $W_O^i$ matrices), each head writes to its own orthogonal subspace.

Shape adjustment. The inferred model configuration (depth = number of layers, layer width = hidden dimension of largest MLP, residual stream size = sum of all subspace dimensionalities) determines the shapes of all weight matrices. Tracr pads weight matrices to the required shapes with zeros for unused dimensions, ensuring compatibility with the transformer implementation.

Layer norm removal. The paper explicitly removes layer normalization from the transformer architecture. This is necessary because Tracr constructs exact weight matrices that implement precise computations assuming the residual stream values are not rescaled or shifted. Layer norm would distort the carefully constructed numerical values (e.g., the 0s and 1s of one-hot encodings, the scalar values of numerical variables) and break the intended computation. The authors note that extending Tracr to support layer norm would require accounting for its normalization effect in the weight construction, which is left to future work.

Output. A complete set of weight matrices and model configuration (in Haiku format) that implements the RASP program. The model can be run on input sequences to produce the specified output.

The Selector Width Primitive (Appendix D.2)

The selector_width operation is a RASP primitive that counts the number of selected keys for each query position—i.e., the number of 1s in each row of a selector matrix. It is used in the sorting program (Figure 5) to compute each token's target position. While Weiss et al. (2021) provide a pure-RASP implementation of selector_width, Tracr treats it as a primitive with a custom compilation path for efficiency.

Why a custom compilation is needed. The pure-RASP implementation of selector_width uses the BOS token, which exists in Tracr's runtime but is not exposed to the RASP program. Rather than modifying RASP to expose the BOS token (which would complicate the language semantics), Tracr compiles selector_width directly to an efficient Craft implementation consisting of an attention layer followed by an MLP layer.

The attention layer. The attention head implements an attention pattern that matches the selector whose width is to be computed. It uses the BOS token as the value input. The attention head computes:

$x = \frac{1}{1 + w}$

where $w$ is the desired selector width (the number of selected keys for the current query). This works because:

1. The attention pattern attends uniformly to all selected key positions plus the BOS token (the BOS token is set to be always attended to for this head via the $\beta_{BOS} = 1$ mechanism).
2. The BOS token's value is set to 1, and all other selected tokens have value 0 (they are not the BOS token, and their value dimensions contain zeros).
3. The attention output is the weighted average of values: $\frac{1}{w + 1} \cdot 1 + \frac{w}{w + 1} \cdot 0 = \frac{1}{w + 1}$. This is exactly $x$.

The MLP layer. The subsequent MLP layer computes:

$w = \frac{1}{x} - 1$

which recovers the desired width from the attention output. The MLP is implemented as a numerical-to-numerical mapping using the discretization strategy described earlier (Section D.1). Since $x$ can take only finitely many values (it is always of the form $1/(w+1)$ for some integer $w$ between 0 and the context length), the MLP can be constructed to be exact on all possible inputs.

Why this two-step approach over alternatives. The alternative would be to have the attention head directly compute $w$ by summing values of 1 for each selected token. However, this requires the attention mechanism to compute an unweighted sum (attention weights of exactly 1 for selected tokens), which is incompatible with the softmax—the softmax always produces a probability distribution where weights sum to 1, not to the number of selected tokens. The two-step approach works around this constraint by computing $1/(w+1)$ (which is a valid attention-weighted average) and then applying a nonlinear transformation to recover $w$. This is a clean example of how Tracr exploits the MLP's universal function approximation capability to compensate for the attention mechanism's structural constraints.

Theoretical Limits of Selector Combination (Appendix F)

Appendix F contains a formal analysis of why Tracr cannot compile arbitrary boolean combinations of selectors. This analysis is important because it justifies one of the key RASP restrictions (no composite selectors) and reveals a fundamental limitation of the compilation approach.

The setup. The paper considers two selectors with different query and key s-ops, stored in orthogonal subspaces $Q_A, K_A$ (for selector A) and $Q_B, K_B$ (for selector B). Each selector has a direct attention matrix $\tilde{W}_{QK}^A$ and $\tilde{W}_{QK}^B$ that produces 0-1 attention logits for its respective predicate. The question is whether there exists a single attention head with some $\tilde{W}_{QK}^{\text{or}}$ that implements the boolean OR of the two selectors.

What can be implemented. The paper first shows that several boolean operations can be implemented through simple transformations of the direct attention matrices:

• NOT: For not select(query, key, pred), use $\tilde{W}_{QK}^{\text{not}} = -\tilde{W}_{QK}$ with $\beta_{BOS}^{\text{not}} = 0$ or $-\frac{1}{2}$. This flips the 0/1 logits to 0/−1, so after temperature scaling, unselected tokens (logit 0) become attended and previously selected tokens (logit −1, now $\exp(-\text{large}) \approx 0$) become unattended.

• AND: For two selectors, use $\tilde{W}_{QK}^{\text{and}} = \tilde{W}_{QK}^A + \tilde{W}_{QK}^B$. This produces logits of 2 when both selectors are active (each contributes 1), 1 when exactly one is active, and 0 when neither is active. With $\beta_{BOS}^{\text{and}} = \frac{3}{2}$ or 2, and sufficient temperature scaling, the softmax concentrates weight on tokens with logit 2 (both selectors active) and approximately ignores tokens with logit 0 or 1, implementing the AND.

• NOR: By composing NOT and AND, $\tilde{W}_{QK}^{\text{nor}} = -\tilde{W}_{QK}^A - \tilde{W}_{QK}^B$ with $\beta_{BOS}^{\text{nor}} = -\frac{1}{2}$ or 0.

Why OR fails. The natural attempt to implement OR would be $\tilde{W}_{QK}^{\text{or}} = -(-\tilde{W}_{QK}^A - \tilde{W}_{QK}^B) = \tilde{W}_{QK}^A + \tilde{W}_{QK}^B$, which is identical to the AND construction. This produces logits of 2 (both active), 1 (exactly one active), or 0 (neither active). For AND, the goal is to select only tokens with logit 2, and temperature scaling achieves this by making the softmax sharply peaked at the maximum logit. For OR, the goal is to select tokens with logit 1 or 2 (any active), while ignoring tokens with logit 0. But with high temperature, the softmax concentrates on the maximum logit value (2), making tokens with logit 1 receive near-zero attention weight. With low temperature, tokens with logit 0 (neither active) would also receive non-negligible weight. There is no temperature setting that simultaneously makes logit-1 tokens attended and logit-0 tokens unattended, because the softmax exponentially amplifies differences—the gap between 1 and 0 is the same as the gap between 2 and 1, so the relative attention weights between the 1 and 0 tokens depend on whether there exists a logit-2 token, which breaks uniformity across query positions.

The formal lemma (Lemma F.1). The paper proves that if a single attention matrix $\tilde{W}_{QK}^{\text{or}}$ could implement the OR of two generic selectors with different inputs, then both selectors must be "rank-1"—their predicates must factor as query_pred_A(query_A) and key_pred_A(key_A), meaning whether a pair is selected depends on the query value and key value independently rather than on their relationship. This is a degenerate case because rank-1 selectors don't compare query and key values (the standard use case for attention); instead, they just filter based on independent properties of the query and key.

The proof, in condensed form, works by contradiction: assume the OR implementation exists for non-rank-1 selectors. Then by considering specific query and key vectors, the proof shows that the logit differences between tokens must be the same for different query vectors, which forces all queries to select the same keys among certain subsets of positions, contradicting the assumption that the selectors are not rank-1.

Practical implications. This result explains why Tracr cannot simply compile composite selectors by manipulating $W_{QK}$ matrices: the softmax fundamentally limits what boolean functions can be expressed as attention patterns. It also explains why the product-space refactoring (constructing composite s-ops that encode all relevant information and using a single atomic selector) is the only general solution—by moving the boolean combination into the s-op construction (which happens via MLPs), the attention head only needs to implement a single atomic predicate on the combined representation, which is always possible.

The paper notes that "the above conclusion may be averted in the case that we have a priori information that certain values of q_A, k_A, q_B, k_B cannot co-occur, or if some of the input s-ops are shared." In other words, if the program's semantics guarantee that certain combinations of query/key values are impossible, the counterexample in the proof may not be constructible, and an OR implementation might exist for that specific case. This is left to future work.

Compression Training Setup (Section 5 and Appendix E)

The compression procedure is both a demonstration application of Tracr and an optional extension that produces more compact, realistic models. It trains a linear projection to reduce the residual stream dimensionality while approximately preserving the model's computation.

Architecture. Let the compiled model have a residual stream of dimension $D$ (the sum of all s-op subspace dimensionalities). The compressed model introduces a single learnable matrix $W \in \mathbb{R}^{D \times d}$ where $d < D$ is the target compressed dimension. The model is modified to apply this projection whenever reading from or writing to the residual stream:

• Writing: Before a layer's output is added to the residual stream (via the residual connection), it is multiplied by $W$ to project it from $D$ dimensions down to $d$ dimensions.
• Reading: Before a layer reads from the residual stream, the compressed $d$-dimensional representation is multiplied by $W^T$ to project it back to $D$ dimensions, after which the layer's original weight matrices (which expect $D$-dimensional inputs) are applied.

All original weights (the MLP and attention matrices from Tracr compilation) are frozen. Only $W$ is trained, using stochastic gradient descent.

Training objective. The compressed model is trained to minimize:

$\mathbb{E}_x \left[ \mathcal{L}_{\text{out}}(W, x) + \mathcal{L}_{\text{layer}}(W, x) \right]$

where the expectation is over input sequences $x$ sampled from the task distribution.

The output loss $\mathcal{L}_{\text{out}}$ measures how well the compressed model matches the original model's final output:

• For categorical output tasks, $\mathcal{L}_{\text{out}}$ is the softmax cross-entropy loss between the compressed model's output logits and the original model's output (treated as ground-truth labels).
• For numerical output tasks, $\mathcal{L}_{\text{out}}$ is the mean-squared error between the compressed and original outputs.

The layer loss $\mathcal{L}_{\text{layer}}$ is a regularization term:

$\mathcal{L}_{\text{layer}} = \sum_{\text{layer } i} (h_i(x) - \hat{h}_{W,i}(x))^2$

where $h_i(x)$ is the output vector at layer $i$ of the original compiled model, and $\hat{h}_{W,i}(x)$ is the output vector at layer $i$ of the compressed model (after projecting through $W^T$ back to $D$ dimensions for comparison). This term "incentivises the compressed model to match the per-layer outputs of the original model," encouraging the compressed model to implement the same intermediate computations rather than finding a completely different algorithm that happens to produce the same final output.

Both loss terms are given equal weight. The paper notes that "other weighting factors give similar results in practice."

Why this formulation. The compression can be viewed as learning a projection from the "hypothetical disentangled model" (the original Tracr model with one dimension per feature) to the "observed model" (the compressed model where features are mixed). This directly mirrors the framing in Elhage et al. (2022b) where superposition is described as a compressed representation of an underlying disentangled feature space. The layer-wise regularization is crucial for maintaining faithfulness to the original computation—without it, the model might learn a completely different algorithm (as actually happens with sort_unique, discussed in Section 5.3), which would defeat the purpose of using compiled models as ground truth.

Training hyperparameters (Appendix E). The paper reports the following exact configuration:

• Optimizer: AdamW
• Weight decay: 0.1
• Adam betas: $\beta_1 = 0.9$, $\beta_2 = 0.99$
• Training steps: $3 \times 10^5$
• Batch size: 256
• Learning rate schedule: linear decay from $10^{-3}$ to $10^{-6}$ over the first half of training (150,000 steps), then constant at $10^{-6}$ for the remaining 150,000 steps.
• Compute requirements: 1–4 hours on two CPU cores, depending on model size.

Why shared $W$ across layers. The paper notes that alternative setups are possible: "we could use a different projection at each layer, use different matrices for embedding and unembedding, or modify weights other than $W$ when compressing the model." Using a shared $W$ is the simplest option and already produces rich structure. However, it also constrains the compressed model—it must represent all features at all layers using the same $d$-dimensional basis, even though different features are active at different layers. More flexible parameterizations (per-layer projections) would give the model more capacity to find efficient representations, but might also make it harder to analyze the learned compression.

What the compression learns (connecting to results). The trained $W$ matrix reveals which features the model preserves and which it discards. The paper analyzes $W^T W$, which is a $D \times D$ matrix whose $(i,j)$ entry is the inner product between the compressed embeddings of features $i$ and $j$. Diagonal entries near 1 indicate that a feature gets its own dedicated dimension in the compressed space. Off-diagonal entries near ±1 indicate that features are stored in superposition (sharing dimensions). Entries near 0 indicate that features are discarded.

For the frac_prevs model compressed to $d = 8$ (original $D = 14$), Figure 7(c) shows:

• Features necessary for the task (is_x, frac_prevs) get separate dimensions (strong diagonal entries).
• Unnecessary features (tokens:a, tokens:b, tokens:c for tokens not in the input) get discarded (near-zero diagonal and off-diagonal entries).
• Redundant features (is_x and tokens:x, which encode the same information) share part of the embedding space (strong off-diagonal entries between them).
• Sparse features (indices embeddings) are stored non-orthogonally (moderate off-diagonal entries, consistent with superposition theory since they don't co-occur).

This analysis is only possible because the original compiled model provides ground-truth labels for each dimension of the residual stream. Without Tracr, the learned $W$ would be an opaque matrix with no semantic interpretation of its rows and columns.

4. Key Insights and Innovations

Innovation 1: Compiled Models as a Systematic Alternative to Post-Hoc Reverse Engineering

The paper's most fundamental conceptual move is reframing how interpretability research validates its findings. Before Tracr, the field operated entirely in a post-hoc paradigm: train a model, then attempt to reconstruct what it learned through labor-intensive manual inspection (Cammarata et al., 2020), probing classifiers (Belinkov, 2022), or causal intervention (Meng et al., 2022). The correctness of these reconstructions could only be assessed indirectly—does the explanation predict behavior? does it survive perturbation?—never by direct comparison to ground truth, because ground truth didn't exist for learned models.

Tracr's compiler offers a fundamentally different starting point: construct models where the computation is specified in advance, then use them as calibrated test cases for interpretability tools. This inverts the validation relationship. Instead of asking "does my interpretation of this opaque model seem plausible?," researchers can ask "does my interpretability method recover the known structure of this compiled model?" The distinction is between plausibility (post-hoc consistency with observed behavior) and fidelity (direct correspondence to actual mechanisms). Plausibility can be misleading—an interpretation might be internally consistent and predict behavior correctly while mischaracterizing the mechanism itself. Fidelity can only be measured when the mechanism is known, which Tracr provides.

This is more than just "providing ground truth data." Prior attempts to create ground truth for interpretability (Yang and Kim, 2019; Adebayo et al., 2020; Zhou et al., 2022) manipulated training data to induce known correlations in image classifiers—for example, introducing a known relationship between background features and class labels. But these approaches only created partial ground truth: one aspect of the model's computation was known, while the rest remained opaque. Tracr provides complete ground truth: every weight, every neuron, every attention pattern has a known purpose that can be traced back to specific lines of RASP code. This completeness matters because interpretability tools often aggregate information across the entire model (e.g., gradient-based attribution accumulates over all parameters; causal tracing traverses the full computational graph). Partial ground truth can't fully validate such methods—a correct attribution might be correct for the known part of the model but spurious for the rest, and you couldn't tell.

The conceptual gap Tracr fills is captured by the paper's microscope metaphor: interpretability researchers are "trying to invent a microscope lens without ever being able to point it at familiar, well-understood shapes." Compiled models are those familiar shapes—model organisms for interpretability science. The significance here is not the compiler engineering itself but the institutional function it enables: a shared, standardized benchmark where methods can be compared quantitatively, failure modes can be diagnosed precisely, and progress can be measured against known answers. Prior to Tracr, the interpretability literature had no such thing; methods were evaluated on different models, different tasks, and different (often implicit) criteria for what counted as a successful explanation. The paper does not itself build such a benchmark, but it provides the infrastructure that makes one possible.

There is a subtle point about what kind of validation compiled models provide. The paper explicitly acknowledges that compiled models are "an intermediate step between very simple toy models and real learned models" (Appendix A.2) and that "results in compiled models do not necessarily generalise to real models." This might seem like a limitation, but it's actually a feature of the experimental design. By testing on models of intermediate complexity—more complex than two-layer toy networks but simpler than GPT-3—compiled models can help isolate which interpretability challenges are scale-dependent and which are fundamental. If a method fails on a compiled model where the correct answer is known, that's a definitive negative result: the method has a bug independent of scale. This kind of falsifiability (Leavitt and Morcos, 2020) was largely absent from prior interpretability work, where negative results could always be attributed to insufficient model size or task complexity. The subsequent work by Conmy et al. (2023), who successfully used Tracr to evaluate an automated circuit discovery method, demonstrates this diagnostic value in practice—they could show their method correctly recovered circuits in compiled models before applying it to real ones.

In summary: the innovation is not the compiler per se, but the experimental paradigm it enables—systematic, falsifiable evaluation of interpretability methods against complete ground truth, moving the field from plausibility arguments toward empirical rigor.

Innovation 2: Treating Superposition as a Compression Phenomenon Inducible in a Controlled Setting

The paper's study of superposition (Section 5) is not merely an "application" of Tracr; it is a methodological innovation in how superposition is studied. Prior work on superposition (Elhage et al., 2022b; Scherlis et al., 2022) operated in a synthetic regime: researchers constructed toy models from scratch (two-layer networks, synthetic data distributions with known sparsity patterns) and trained them to perform simple tasks, then analyzed whether superposition emerged. This approach has produced foundational insights—including the key finding that models represent more features than dimensions when features are sparse—but it has an inherent limitation: the models are so simple that it's unclear whether the observed superposition dynamics are an artifact of the toy setting or genuinely descriptive of larger models.

Tracr's approach inverts this: instead of training a toy model from scratch and hoping superposition emerges, the authors start with a fully understood model implementing a non-trivial algorithm (multi-step procedures like sorting or parenthesis checking, involving multiple attention heads and MLPs), then induce superposition deliberately by applying gradient-descent compression with a dimensionality bottleneck. This transforms superposition from an emergent phenomenon to a controlled experimental variable. The question shifts from "does superposition happen?" to "given a known computation, how does the compression allocate dimensions across features, and what properties of the computation predict the allocation?"

The compression setup itself introduces a novel framing that connects to prior superposition theory (Elhage et al., 2022b). Those authors described superposition as the result of embedding a "hypothetical disentangled model" into the "observed model" through a learned projection. Tracr's compression literally implements this: the original compiled model with orthogonal, disentangled features (one dimension per variable) is projected through a learned matrix W into a smaller space, and then unprojected via W^T for each layer's computation. The layer-wise regularization loss (matching intermediate activations to the original model) corresponds to the constraint that the compressed model should implement the same computation, not just achieve the same final output. This is not how superposition was previously studied—prior work trained models end-to-end with a fixed bottleneck, which conflates the question of what computation the model learns with how it represents that computation. By fixing the computation (through frozen compiled weights) and only training the projection, Tracr isolates representation from computation.

The findings from this setup are significant beyond their specific numbers. The observation that compressed models discard unnecessary features and represent less important features non-orthogonally (Figure 7) confirms predictions from prior superposition theory (Elhage et al., 2022b) but does so in a substantially more complex setting (multi-layer transformers implementing algorithmic tasks rather than two-layer toy models). More importantly, the finding that features encoding redundant information share embedding dimensions (e.g., is_x and tokens:x in the frac_prevs model, which carry essentially the same information) is a new observation that goes beyond prior superposition results. Prior work focused on sparsity as the driver of superposition—features that rarely co-occur can share dimensions. Tracr reveals a second driver: information redundancy. When two features are deterministically related, the model can represent them in a shared subspace without losing information, because knowing one implies the other. This is only observable because Tracr provides ground-truth labels for what each dimension represents and how information flows between variables—information unavailable in toy models where feature semantics are synthetic and in real models where they're unknown.

The finding that compressed models may not preserve the original algorithm (Section 5.3)—specifically, that the compressed sort_unique model learns a numerical encoding of target positions rather than the original categorical one-hot encoding, while still achieving near-perfect accuracy—is equally important as a negative result. It reveals that compression by itself does not guarantee faithfulness; the compressed model can discover an alternative implementation that achieves the same input-output behavior through different intermediate representations. This is both a practical limitation (the compressed model's ground truth is no longer the original RASP program) and an opportunity (compression surfaces alternative algorithmic strategies, which could reveal something about the inductive biases of gradient descent or the space of possible implementations). This finding complicates the simple narrative that compression is a straightforward way to make compiled models more realistic while preserving known computation, and it positions Tracr as a tool for studying when and why alternative implementations emerge—a question inaccessible without ground truth for comparison.

Innovation 3: Formalizing the Limits of Attention-Based Boolean Selector Composition

Appendix F contains a theoretical result that initially appears to be an implementation detail—a lemma proving that boolean combinations of selectors with different input variables cannot, in general, be compiled to a single attention head's W_QK matrix. But this result is conceptually significant because it formalizes a fundamental constraint on what attention mechanisms can express, connecting the practical limitations of the compiler to the theoretical properties of the softmax operation.

The result speaks to a question that has broader implications for mechanistic interpretability: what logical operations can a single attention head implement? Prior work on the expressive power of transformers (Wei et al., 2022; Merrill et al., 2022; Giannou et al., 2023) focused on what classes of functions the entire architecture can approximate, given sufficient depth and width. Tracr's lemma addresses a finer-grained question: given the specific constraints of the softmax attention mechanism (bilinear logits computed from query and key vectors in the same residual stream, followed by a normalized exponential), what boolean functions over attention patterns are compilable? The answer—that AND and NOR are implementable but OR is not (except in degenerate rank-1 cases)—is non-obvious. Naively, one might expect all boolean operations to be symmetrical: if you can implement AND by summing logits and using a bias to select the maximum, surely you can implement OR similarly? The proof shows why this symmetry fails: the softmax's exponential amplification of logit differences means that the gap between "one selector active" and "neither active" cannot be made simultaneously large relative to the gap between "both active" and "one active" across all query positions.

This is not merely a compilation constraint—it is a diagnostic concept for reverse-engineering attention heads in real models. When an interpretability researcher finds an attention head that appears to be computing a boolean combination of multiple criteria, Lemma F.1 tells them: if you observe OR-like behavior (the head attends when either condition A or condition B holds, with A and B operating on different subspaces of the residual stream), the implementation cannot be a simple linear combination of two attention logit matrices. It must involve either (a) a rank-1 structure in the predicates (meaning the conditions don't actually compare query and key values, but merely filter based on independent properties), (b) an upstream MLP that computes a combined query/key representation (what Tracr's product-space refactoring does), or (c) a multi-head composition where the OR is implemented across heads rather than within one. This gives reverse-engineers a falsifiable prediction about what circuit structures are possible, which can guide the search for mechanisms in real models.

The theoretical contribution here is modest in scope—a lemma about a specific architectural constraint—but it exemplifies the kind of insight that emerges when compilation forces a rigorous accounting of the transformer's computational primitives. By attempting to compile arbitrary RASP programs and running into systematic obstacles, Tracr surfaces the structural "grammar" of attention: not just what the architecture can do (expressivity results), but what it can do with each individual component given the mathematical form of the softmax. This is the level of granularity at which mechanistic interpretability operates—individual heads, not whole models—and theoretical results at this granularity are rare.

Innovation 4: The Residual Stream as Direct Sum of Labeled Subspaces as a Design Principle for Interpretability

One of Tracr's most distinctive design choices is allocating each s-op its own orthogonal subspace in the residual stream, reserved for its sole use throughout the entire network (Section 3, Step 5). The paper presents this as a practical simplification—it makes it easy to compose independently translated components without interference—but it is also an architectural hypothesis about what makes models interpretable.

The hypothesis, implicit in the design, is: interpretability is primarily a property of representation disentanglement. A model is interpretable if each dimension or subspace of the residual stream has a stable, consistent meaning across layers—if dimension 7 always represents "is the current token 'x'?" regardless of which layer you're inspecting. Tracr achieves this by construction: because subspaces are orthogonal and persistent, the meaning of each dimension is fixed by the program specification, and inspecting the residual stream at any layer directly reveals which variables are active and what values they hold.

This is not how real models work—the paper is explicit about this limitation—but it is a normative claim about what interpretability should aim for. The dominant paradigm in mechanistic interpretability (Olah, 2022; Elhage et al., 2021) already emphasizes the residual stream as the central object of analysis, but it treats the residual stream as a high-dimensional vector space where features are entangled in unknown ways, and the goal is to disentangle them post-hoc. Tracr demonstrates what the ideal endpoint of that disentanglement looks like: a basis-aligned representation where every dimension has a semantic label and the computation can be read off directly from the activations. This provides a concrete target for interpretability methods—not just "find features" in the abstract, but "find a linear transformation that maps the model's residual stream to something as clean as Tracr's orthogonal subspaces."

The compression experiments (Section 5) extend this hypothesis by showing what happens when you relax the orthogonality constraint. The compressed W matrix learns a non-orthogonal embedding of the original disentangled features, and the structure of W^T W (Figure 7) reveals which features get dedicated dimensions, which get discarded, and which share dimensions. This is a bridge between the idealized Tracr representations and the entangled representations of real models: it shows how a clean, interpretable representation degrades into superposition under dimensionality pressure, while preserving the underlying feature structure. For interpretability researchers, this is valuable because it provides a known mapping between "what the features actually are" (the original Tracr subspaces) and "what the compressed model's dimensions look like" (mixtures of features), which could serve as training data for methods that attempt to reverse superposition—for example, dictionary learning approaches that try to recover the underlying features from the compressed representation (Donoho, 2006; Aharon et al., 2006, cited in Appendix A.1).

The fact that Tracr models are "sparse and inefficient" by design—using 12 of 14 residual stream dimensions for input embeddings in the frac_prevs model (Section 4.1), with only 2 dimensions for the actual computation—is often cited as a limitation. But from the perspective of this innovation, the inefficiency is the point. It makes the model's computation visually inspectable (as in the forward-pass figures throughout the paper) and mechanistically transparent (every weight matrix has a known function). The compression experiments then show that this transparency is not necessary for correct computation—the model can be compressed 2-3× without loss—but it is necessary for human understanding. This connects to a broader question about the relationship between model efficiency and interpretability: are efficient models inherently harder to interpret because they reuse dimensions for multiple purposes? Tracr doesn't answer this question, but it provides a platform for studying it systematically—compare a model where features are orthogonal with a compressed version of the same model where they're not, and measure how much harder the compressed version is to interpret using existing methods.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper does not use a standard ML benchmark dataset in the traditional sense. Instead, Tracr operates on synthetic sequences generated programmatically from the input specifications of each RASP program. For example, the frac_prevs program operates on sequences drawn from a finite vocabulary (e.g., {x, y, a, b, c}) up to a fixed context length (e.g., 3 or 5). The sort_unique program operates on sequences of unique numbers within a fixed range and length. The dyck-n program operates on sequences of parentheses. All inputs are constrained by the finite vocabulary and context length for which the model was compiled—these are hard limits because Tracr constructs categorical encodings with one dimension per possible value, and values outside the compiled range have no embedding dimension. There is no "test set" in the supervised learning sense; rather, the compiled model's correctness is evaluated by comparing its output to the expected output of the RASP program on all possible inputs within the compiled vocabulary and context length (the program's semantics serve as the ground truth). The paper does not report accuracy numbers on a held-out test set for the compilation itself—the compiled models are deterministic and exact by construction (the MLP approximations are designed to be exact on all possible input values, as determined by Step 2's value inference).

• Base model(s). Tracr does not start from a pretrained model. Instead, it constructs transformer models from scratch—all weights are hand-engineered through the compilation pipeline described in Section 3. The resulting models are standard decoder-only transformers (without layer norm) in the Haiku framework (Hennigan et al., 2020). Model sizes vary by program: the frac_prevs model has a 14-dimensional residual stream, 4 layers (2 of which are no-ops), 1 attention head, and 1 MLP with a hidden dimension sufficient to implement the required lookup tables or piecewise linear approximations. The sort_unique model (Figure 5) uses 2 attention heads and 2 MLPs across 4 layers. The dyck-2 model (Figure 11) is substantially larger—58-dimensional residual stream, 10 layers (with some no-ops), multiple attention heads and MLPs—reflecting the complexity of checking balanced parentheses for two types of brackets. The authors note that PaLM 2-S* is used only in the separate compression experiments (Section 5) as a source of representative scale context, not as the base model for Tracr compilation. The compiled models are deliberately small because each variable occupies its own orthogonal subspace, causing residual stream dimensionality to grow with program complexity.

• Metrics. For the compilation itself, the primary "metric" is functional correctness: does the compiled transformer produce exactly the same output as the RASP program on all possible inputs? This is assessed qualitatively through visual inspection of forward-pass figures (Figures 2, 5, 9–11) that show residual stream contents at each layer for specific input sequences, with dimensions labeled according to their semantic meaning. For the compression experiments (Section 5), the paper reports: (1) task accuracy—the fraction of inputs for which the compressed model's output matches the original compiled model's output (for categorical outputs, this uses the softmax cross-entropy loss during training; the final accuracy is what's plotted in Figure 7(b) and described for sort_unique as "nearly perfect accuracy"); (2) per-layer cosine similarity—the average cosine similarity between the compressed model's layer outputs and the original model's layer outputs, measuring whether the compressed model preserves the same intermediate representations (Figure 8, left); (3) W^T W visualizations—heatmaps of the learned compression matrix's inner products, showing how original disentangled features are embedded in the compressed space (Figure 7(c,d) and Figure 8). These latter two metrics are only possible because Tracr provides ground-truth labels for every residual stream dimension.

• Baselines. For the compilation pipeline itself, there is no baseline—Tracr is the first system to compile RASP programs to transformer weights. For the compression experiments, the baseline is the original uncompiled model (the Tracr-compiled model with orthogonal, disentangled features at full dimensionality). The compressed model's accuracy and layer-wise similarity are measured against this baseline. The paper also compares the learned compression matrix W against Principal Component Analysis (PCA) as an alternative compression method (Figure 7(c) vs. 7(d)): PCA retains dimensions that capture the most variance in the residual stream activations, while the learned compression retains dimensions that are necessary for the task (e.g., is_x and frac_prevs) and discards unnecessary ones (e.g., unused token embeddings). This comparison demonstrates that gradient-descent-based compression with the layer-wise regularization loss produces task-aware representations rather than variance-maximizing ones.

• Generation budget / compute accounting. This concept is not directly applicable to Tracr compilation, as the compiled models execute exactly once per input with no sampling or search. For the compression experiments, "compute" is measured in training steps: 300,000 steps of SGD with batch size 256, taking 1–4 hours on two CPU cores depending on model size (Appendix E). The compression matrix W has dimensions D × d where D is the original residual stream size and d is the compressed size (swept from 2 to 14 in Figure 7(b)). All original model weights are frozen; only W is trained. The paper sweeps d as the independent variable in Figure 7(b) to measure how task accuracy varies with compression ratio.

• Cross-validation / statistical protocol. The paper does not report cross-validation or statistical significance testing for the compilation results, because the compiled models are deterministic and their correctness is verified by construction against the RASP program semantics on all possible inputs—there is no training, no random initialization, and no variance. For the compression experiments, the paper reports single training runs for each compression dimension d. No error bars, confidence intervals, or multiple random seeds are reported for the compression results in Figures 7 or 8. The paper does not specify the number of input sequences used to compute the layer-wise cosine similarity in Figure 8 (left), nor does it describe how the training/validation split for the compression experiments was constructed (the training data consists of programmatically generated input sequences, so overfitting to a finite training set is theoretically possible if not enough sequences are generated—but this is not discussed).

Main Quantitative Results

The experimental results fall into two distinct categories: compilation results (demonstrating that Tracr correctly compiles RASP programs into transformer weights) and compression results (studying superposition in compressed Tracr models). The compilation results are primarily qualitative (visual demonstrations of correct forward passes), while the compression results provide the paper's main quantitative findings.

Compilation Demonstrations: Compiled Model Correctness and Architecture

The paper presents four compiled models across increasing levels of complexity, each accompanied by a forward-pass visualization showing that the model correctly implements the specified RASP program.

frac_prevs (Figure 2). This program computes the fraction of previous "x" tokens at each position. The compiled model uses a 14-dimensional residual stream, of which 12 dimensions are input embeddings (vocabulary size plus special tokens), 1 dimension holds the is_x indicator variable (numerical), and 1 dimension holds the frac_prevs output (numerical). The model architecture consists of 4 layers: Attn 1 (no-op), MLP 1 (computes is_x = (tokens == "x")), Attn 2 (computes frac_prevs = aggregate(prevs, is_x) using a causal attention pattern over indices), MLP 2 (no-op). The forward-pass figure shows the residual stream after each layer for the input sequence [bos, x, a, c, x], confirming that is_x is correctly populated (value 1 at positions containing "x") after MLP 1 and that frac_prevs computes the correct sequence [1, 1/2, 1/3, 1/2] after Attn 2.

sort_unique (Figure 5). This program sorts a sequence of unique numbers. The compiled model uses Attn 1 and MLP 1 to implement the selector_width primitive (computing each token's target position by counting how many tokens are smaller than it), then Attn 2 moves each token to its target position. MLP 2 is a no-op. The visualizations in Figure 5 confirm the model correctly sorts input sequences. The paper does not report quantitative accuracy—it states the model implements the program correctly by construction.

Extended sort with duplicates (Figure 9). This program extends sort_unique to handle duplicate keys by first adding a small multiple of indices to the keys (via MLP 1) to break ties, then applying the same selector_width-based target position computation and token movement. Attn 1 is a no-op in this architecture. The forward-pass visualizations confirm correct sorting with duplicates.

pair_balance and dyck-n (Figures 10 and 11). The pair_balance program computes the running difference between open and close parenthesis fractions for a single pair type (e.g., parentheses, brackets). The dyck-n program composes multiple pair_balance subroutines, checks that no balance ever goes negative (which would indicate a closing bracket without a matching opener), and verifies that all balances return to zero at the end of the sequence. For the compiled dyck-2 model with pairs ("(", ")") and ("{", "}"), Figure 11 shows a 58-dimensional residual stream, 10 layers, and a substantial number of attention heads and MLP blocks to implement the full parenthesis-checking algorithm. The forward-pass visualization demonstrates correct operation on example sequences.

The paper states (Section 4): "We were able to compile RASP programs for all the tasks described in Weiss et al. (2021), though we had to modify a few programs to only use features supported by Tracr." Appendix G and the open-source repository contain additional compiled program examples beyond those shown in the paper.

Compression Results: Superposition in Compressed Tracr Models

The compression experiments (Section 5) provide the paper's main quantitative findings. These experiments take full-dimensionality compiled models and train a linear projection W to compress the residual stream while approximately preserving the model's computation.

frac_prevs compression (Figure 7). The original compiled model has D = 14 residual stream dimensions. The compressed model trains a single matrix W ∈ R^(14×d) with d swept from 2 to 14.

• Compression fidelity (Figure 7(a,b)). Figure 7(a) shows the training loss curves for different values of d. The loss decreases during training for all d ≥ 4, with larger d converging to lower final loss. Figure 7(b) shows the final task accuracy vs. compression dimension d. The key finding: "After about d = 6 the compressed model solves the task essentially as well as the original compiled model." The original model uses D = 14 dimensions, so this represents a compression ratio of approximately 2.3× (from 14 to 6) with no loss in task performance. At d = 5, performance degrades noticeably. At d ≤ 4, the model largely fails the task.

• What the compression learns (Figure 7(c) vs. 7(d)). Figure 7(c) shows W^T W for the compression to d = 8—a 14×14 matrix where entry (i,j) is the inner product between the compressed embeddings of original dimensions i and j. The authors interpret this using their knowledge of the original model's features (which are known because the model was compiled):

  • Crucial features preserved: The is_x and frac_prevs dimensions (necessary for computing the fraction of "x" tokens) show strong diagonal entries in W^T W, indicating they each get approximately their own dedicated dimension in the compressed space.
  • Unnecessary features discarded: The input token dimensions tokens:a, tokens:b, tokens:c (for tokens not "x") show near-zero diagonal and off-diagonal entries, indicating they are essentially ignored in the compressed model—this makes sense because only token "x" matters for the task.
  • Redundant features share space: The is_x dimension and the tokens:x input dimension show strong off-diagonal entries between them, indicating they share part of the compressed embedding space. The authors note: "Intuitively, this occurs because the variables encode similar information"—is_x is the indicator that the current token is "x", and tokens:x is the categorical embedding dimension for the "x" token, so they carry redundant information.
  • Sparse features stored non-orthogonally: The indices embeddings (position encodings for 0, 1, 2) show moderate off-diagonal structure, indicating they are stored in superposition. The authors connect this to existing superposition theory: "the indices embeddings are sparse and do not occur together (Elhage et al., 2022b)."

• Comparison to PCA (Figure 7(c) vs. 7(d)). Figure 7(d) shows the same W^T W visualization when the compression is done via Principal Component Analysis (retaining the top 8 principal components of the residual stream activations) rather than gradient descent. The PCA-compressed matrix shows a different pattern: it retains all input token dimensions (since they have high variance—they are one-hot and always active) and does not preferentially preserve the task-critical is_x and frac_prevs dimensions. The authors note: "In contrast to PCA, our compression procedure produces a compression matrix W that retains features necessary for the task (e.g., is_x and frac_prevs) and discards features that are unimportant (e.g., tokens:a)." This demonstrates that the gradient-descent compression with layer-wise regularization loss is task-aware, while PCA is merely variance-aware.

sort_unique compression (Figure 8). The sort_unique model is compressed using the same procedure. The results reveal a different and more complex story.

• Task accuracy. Figure 8 (right panel) shows that the compressed model achieves "nearly perfect accuracy" on the sorting task for a range of compression dimensions (the exact dimensions swept are not specified numerically in the text, but the right panel shows accuracy near 1.0 across the tested range). This suggests the compressed model can sort correctly even with a substantially smaller residual stream.

• Layer-wise fidelity failure (Figure 8, left panel). However, the left panel of Figure 8 reveals a critical discrepancy. It shows the average layer outputs of the original compiled model, the compressed model, and the squared difference between them, as a function of layer index (exact numerical values for cosine similarity or layer dimensions are not provided in the text, but the figure shows per-layer output magnitudes). The paper states: "the average cosine similarity of the outputs at individual layers stays around 0.8, far shy of 1." The squared difference panel (rightmost) shows a large spike at the layer where target_pos is computed.

• Attribution to encoding change. The authors inspect the models' outputs at each layer and "attribute the error to the target_pos variable. In the compiled model, target_pos is encoded as a one-hot vector. However, the compiled model only uses a single dimension. This suggests that the compressed model moves the tokens to the target position with a numerical encoding of the target position rather than a categorical encoding."

• Interpretation. This is a significant finding: the compressed model achieves the correct input-output behavior (sorting correctly) but does so through a different internal mechanism (numerical target_pos rather than categorical one-hot). The compressed model is therefore not faithful to the original RASP program's intermediate representations, even though it produces the correct final output. The authors frame this as "both a setback for adding compression to the compiler—the compiler's annotations no longer serve as the exact ground truth—but also an opportunity. The ways neural networks solve algorithmic tasks regularly surprise researchers (Nanda et al., 2023). Studying such discrepancies could be a way to learn more about the ways NNs naturally represent certain computations without reverse-engineering entire models."

The paper does not report compression results for the dyck-n model or any other compiled programs beyond frac_prevs and sort_unique. These two case studies serve as existence proofs for the compression methodology rather than a comprehensive experimental evaluation.

Ablation Studies and Robustness Checks

The paper includes a limited set of ablation-like comparisons, primarily in the compression experiments and appendices:

• PCA vs. learned compression (Figure 7(c) vs. 7(d)): PCA as a compression method fails to preserve task-critical features while the learned compression (with layer-wise regularization) successfully identifies and preserves them. This ablation demonstrates that the layer-wise regularization loss is essential—without it (or with a purely variance-based compression), the compressed model would likely fail the task because it would discard the small-magnitude but semantically crucial dimensions like is_x in favor of high-variance but task-irrelevant dimensions like unused token embeddings.

• Compression dimension sweep (Figure 7(b)): The paper sweeps d from 2 to 14 for the frac_prevs model, showing that performance is essentially flat above d = 6 and degrades sharply below d = 5. This establishes that the model requires approximately 6 dimensions to represent the necessary features for the frac_prevs task, compared to the original 14, implying that roughly 8 of the original dimensions are redundant for this specific task (they encode vocabulary items that never appear in task-critical contexts). The paper does not systematically vary the task to see how the required compression dimension changes—for example, does a program that uses more of the vocabulary require a larger d?

• Design choice: shared W across layers (mentioned in Section 5.1). The paper notes that alternative compression architectures are possible: "we could use a different projection at each layer, use different matrices for embedding and unembedding, or modify weights other than W when compressing the model." The shared-W approach is used "for simplicity," and "other weighting factors give similar results in practice" for the loss terms. However, no ablation comparing shared vs. per-layer W is presented. This is a potentially significant omission—per-layer projections would give the model more flexibility to represent different features at different layers, which might allow greater compression or might cause the model to diverge further from the original algorithm.

• Training loss components (Section 5.1). The total loss is L_out + L_layer with equal weight (λ = 1 for both terms). The paper states "other weighting factors give similar results in practice" but no sweep over weighting factors is shown. This claim cannot be verified from the presented data.

• Optimizer and hyperparameters (Appendix E). The paper specifies AdamW with weight decay 0.1, β1 = 0.9, β2 = 0.99, linear learning rate decay from 10^{-3} to 10^{-6} over 150K steps, batch size 256, 300K total steps. No ablation over optimizer choice, learning rate schedule, or batch size is reported. The sensitivity of the compression results to these choices is unknown.

• ReST^EM revision model experiment (Appendix K, Figure 16)—mentioned in the prior analysis but belongs here as a compression-adjacent ablation: The paper reports a negative result where applying ReST^EM (Singh et al., 2024)—an RL-based training procedure intended to optimize the revision model—actually degraded performance: "fully sequential performance drops to approximately 33.5% compared to roughly 38.5% at the optimal ratio" at 256 generations. This ablation demonstrates the sensitivity of the compression/revision training to the data generation procedure, though it concerns a separate model (the revision model from the search experiments in the prior sections, Section 6 of the paper) rather than the Tracr compression directly. The authors hypothesize that "on-policy data collection in ReST^EM exacerbates spurious correlations in revision data."

Critical Assessment

The experiments in this paper demonstrate two distinct things, and it is important to evaluate them separately: (1) that Tracr can compile RASP programs into transformer weights that correctly implement the specified computation, and (2) that compressed Tracr models exhibit superposition phenomena that can be studied in a controlled setting. The strength of evidence differs substantially between these two claims.

Claim: "Tracr compiles RASP programs into standard decoder-only transformer models"

Assessment: Supported qualitatively, but quantitative evaluation is absent. The paper demonstrates four compiled programs (frac_prevs, sort_unique, extended sort with duplicates, dyck-2) and states that programs for all tasks described in Weiss et al. (2021) were compiled. The primary evidence for correctness is visual: forward-pass figures showing residual stream contents for specific input sequences, with labeled dimensions that match the expected computation. These figures are pedagogically effective—they make the model's computation transparent in a way that raw weight matrices would not—but they constitute a very small number of test cases.

The paper does not report:

• Systematic correctness testing over all possible inputs for any compiled model (though this is feasible given the finite input space—for frac_prevs with vocabulary size V=2 meaningful tokens plus BOS and context length C=3, the number of possible inputs is approximately V^C ≈ 8, which is trivially enumerable).
• Quantitative accuracy metrics (e.g., "the compiled model produces the correct output on X/Y possible inputs").
• Testing on edge cases (empty sequences after BOS, maximum-length sequences, sequences containing tokens near the boundary of the compiled value ranges).
• Any formal verification that the compiled weights correctly implement the RASP semantics, beyond the visual demonstrations.

The absence of systematic correctness testing is a genuine weakness. While the compilation is deterministic and the weight construction is designed to be exact (through exact lookup tables for categorical MLPs and exact discretization for numerical MLPs), the compilation pipeline is complex (six steps, multiple heuristics, manual engineering of MLP blocks). Bugs in the compiler implementation could produce models that appear correct on the shown examples but fail on others, and the paper provides no evidence that this is not the case. The open-source release mitigates this somewhat (other researchers can test the models themselves), but the paper itself should have included basic correctness statistics.

Additionally, the paper does not characterize the types of RASP programs that Tracr can and cannot compile, beyond the specific restrictions described (no composite selectors, encoding annotations required, BOS token required). What fraction of "natural" RASP programs fall within Tracr's supported subset? What is the most complex program that has been successfully compiled, measured in terms of RASP operation count, compiled model depth, or residual stream size? The dyck-2 model (58 dimensions, 10 layers) demonstrates that non-trivial programs are compilable, but the scaling limits of the approach are unexplored.

Claim: "Compressed Tracr models can be used to study superposition in transformers executing multi-step algorithms"

Assessment: Supported as a proof of concept, but the experimental scope is narrow. The compression experiments on frac_prevs and sort_unique provide two case studies that are genuinely informative:

• The frac_prevs compression (Figure 7) demonstrates that gradient-descent compression with layer-wise regularization preserves task-critical features while discarding unnecessary ones, consistent with superposition theory (Elhage et al., 2022b). The qualitative analysis of W^T W—identifying which features get dedicated dimensions, which get discarded, and which share space—is only possible because of Tracr's ground-truth feature labels, and it validates predictions from prior work (sparse features stored in superposition; necessary features preserved) while adding a new observation (redundant features share embedding dimensions).

• The sort_unique compression (Figure 8) demonstrates a failure mode where the compressed model achieves correct output through a different internal mechanism (numerical encoding of target positions vs. categorical one-hot). This is a valuable negative result because it shows that compression faithfulness is not guaranteed, and that studying when models diverge from the original algorithm could be scientifically productive.

However, the experimental scope is very narrow for a claim about "studying superposition in transformers implementing multi-step algorithms":

• Only two models tested. The paper compresses only frac_prevs and sort_unique. Results from two models do not establish a general methodology for studying superposition, nor do they reveal whether the observed patterns (redundant features sharing space, discarded unnecessary features) generalize across different algorithm types.

• Single compression method. Only one compression architecture is tested (shared W across all layers, training only W with frozen original weights). The paper mentions alternatives (per-layer projections, different embedding/unembedding matrices) but does not explore them. It is unclear whether the sort_unique divergence (learning a numerical encoding instead of categorical) is a property of the sorting algorithm, the compression architecture, or the optimization procedure. Testing multiple compression architectures on the same model would help disambiguate this.

• Limited analysis of the compressed representations. The W^T W visualization (Figure 7(c)) is the primary analysis tool, but it only shows pairwise inner products between original feature dimensions in the compressed space. This does not capture higher-order structure (e.g., three features sharing a two-dimensional subspace in a non-pairwise-decomposable way). More sophisticated analyses (e.g., checking whether the compressed model's attention patterns match the original's, or whether specific MLP computations are preserved) are not performed.

• No systematic study of what drives superposition structure. The paper observes that is_x and tokens:x share embedding space because they encode redundant information, and that indices embeddings are stored non-orthogonally because they are sparse. But these are post-hoc interpretations of two specific features in one model. A systematic study would vary the amount of redundancy between features (e.g., by modifying the RASP program to have more or fewer redundant variables) and measure how this affects the learned compression, or vary the sparsity of features and measure the degree of superposition. None of this is done.

• No comparison to superposition in trained models. The paper's abstract claims that Tracr "can be used to study 'superposition' in transformers that execute multi-step algorithms" and that the compression experiments "confirm key observations by Elhage et al. (2022b) in a new setting." The confirmation is qualitative and limited to two models. A more compelling demonstration would compare superposition patterns in Tracr-compressed models to patterns in transformers trained from scratch on the same tasks (which would require also training such models and developing methods to identify their features, a substantially larger project that the paper does not undertake).

Claim: "Compiled models can serve as ground-truth for evaluating interpretability methods"

Assessment: Not experimentally tested in this paper. This is the central promise of Tracr, but the paper does not actually evaluate any interpretability method on Tracr-compiled models. The claim is entirely prospective. The paper states (Section 7) that Tracr could be used to test "classifier probes, gradient-based attribution, and causal tracing," and mentions that Conmy et al. (2023) used Tracr for circuit discovery evaluation (this is a citation to concurrent work, not an experiment in the paper). The compression experiments demonstrate one use case (studying superposition), but this is distinct from the claim about evaluating interpretability tools—the compression analysis relies on Tracr's ground-truth labels to interpret W^T W, but it does not test whether an existing interpretability method (e.g., a probing classifier) would have recovered the same information from the compressed model without access to the labels.

This is not a flaw in the paper's contribution—introducing the tool is a valid contribution even without demonstrating all its applications—but it means the claim "Tracr enables evaluation of interpretability methods" should be understood as a motivated proposal, not an experimentally validated finding. The paper would be strengthened by even a small-scale demonstration: apply a standard probing classifier to a Tracr-compiled model and show whether it correctly identifies which layers contain which features, compared to the known ground truth.

Missing Experiments and Baselines

Several experiments that would substantially strengthen the paper are absent:

• Systematic correctness testing of compiled models: Enumerate all possible inputs (for small vocabulary/context combinations) and verify the compiled model's output matches the RASP program's output on every input. Report accuracy and any failure cases. This is computationally trivial for the models shown (the input spaces have at most a few hundred sequences for vocabularies of ~5 tokens and context lengths of ~5).

• Compilation scaling characterization: How do compiled model size (residual stream dimension, layer count, total parameters) and compilation time scale with RASP program size (number of operations, number of s-ops, vocabulary size, context length)? The paper provides no scaling data, making it difficult to assess whether Tracr could compile programs substantially more complex than dyck-2.

• Comparison to manually written transformers: For simple programs like frac_prevs, a skilled researcher could manually construct the transformer weights without a compiler. How do Tracr-compiled models compare to manually constructed ones in terms of efficiency (number of parameters, layers) and correctness?

• Compression of the dyck-n model: The dyck-2 model is the most complex compiled model shown (58 dimensions, 10 layers). Compressing it and analyzing the resulting superposition structure would be a more compelling demonstration than the very simple frac_prevs model. The paper does not explain why this was not attempted (computational cost? lack of interesting structure?).

• Sensitivity to compression hyperparameters: Ablations over learning rate, batch size, loss weighting, and optimizer choice for the compression experiments. The paper claims results are robust to loss weighting but provides no evidence.

• Multiple compression runs with different random seeds: The compression results (Figures 7, 8) are from single training runs. Without multiple seeds, we cannot distinguish consistent structural patterns (e.g., "is_x always shares space with tokens:x") from optimization noise.

Summary of Experimental Strengths and Weaknesses

Strengths:

• The forward-pass visualizations (Figures 2, 5, 9–11) are genuinely illuminating as pedagogical tools for understanding how attention heads and MLPs can coordinate to implement algorithmic computation. The labeled residual stream dimensions make the computation transparent in a way that weight matrices alone would not.
• The compression experiments on frac_prevs provide a clean demonstration of task-aware compression that validates prior superposition theory and reveals the redundancy-sharing phenomenon.
• The sort_unique compression result—demonstrating that compressed models can achieve correct behavior through different internal mechanisms—is an important cautionary finding that realistically constrains how compiled models can be used as ground truth.
• The open-source release enables other researchers to apply Tracr to their own interpretability research questions, which may prove more valuable than the specific experiments in the paper.

Weaknesses:

• The compilation correctness evidence is entirely qualitative (visual inspection of a few examples) rather than quantitative (systematic enumeration of all possible inputs).
• The compression experiments cover only two models, neither of which is particularly complex (the frac_prevs model has essentially one effective attention head and one effective MLP; the sort_unique model has two attention heads and two MLPs). The insights may not generalize to the multi-head, multi-layer structure of the dyck-2 model or to real trained transformers.
• The paper's central motivating claim—that Tracr can evaluate interpretability methods—is not experimentally tested. No interpretability method is applied to a Tracr model and evaluated against the known ground truth.
• Key experimental details are underspecified (number of test inputs for compression, number of random seeds, sensitivity to hyperparameters).
• The compression analysis is limited to pairwise feature interactions (W^T W) and does not explore higher-order structure.
• The paper does not report any failure cases in compilation—programs that Tracr could not compile, or programs that compiled to incorrect models—which would help characterize the compiler's limitations and reliability.

6. Limitations and Trade-offs

Compiled Models Require a Fully Specified Algorithm; They Cannot Be Used When the Computation Is Unknown

The assumption or constraint. Tracr compiles RASP programs into transformer weights, which means the researcher must write a complete, executable specification of the desired computation before any model exists. This is the core premise of the paper, but it is also a fundamental scope constraint: "Clearly, we will likely never compile fully featured language models in Tracr" (Section 7). The approach only works for tasks where the algorithm is known and expressible in RASP's primitives.

The consequence. Tracr cannot provide ground truth for the very interpretability problems that are most pressing: understanding models that solve tasks in unknown ways, that exhibit emergent capabilities not specified by any programmer, or that operate over domains (natural language, vision, multimodal reasoning) where no clean algorithmic specification exists. The paper's stated ambition is to help evaluate interpretability tools, but the tools are needed precisely when the model's algorithm is not known—if you could write the RASP program, you wouldn't need interpretation. This creates a paradox: Tracr-compiled models are most useful as test cases (the algorithm is known, so you can verify your method), but they are least realistic as stand-ins for the models we actually want to interpret (whose algorithms are unknown and likely more complex). The gap between compiled models and real models is not just one of scale but of kind—compiled models implement a human-specified algorithm, while real models implement algorithms that emerged from optimization and may have no clean decomposition into s-ops, selectors, and aggregates.

What evidence exists in the paper. The paper provides no comparison between a Tracr-compiled model and a transformer trained from scratch on the same algorithmic task, so we cannot assess how different the learned solution would be from the compiled one. The sort_unique compression result (Section 5.3) is actually indirect evidence that even when starting from the compiled model, gradient descent can find alternative implementations (numerical encoding vs. categorical one-hot), suggesting that learned models may diverge substantially from compiled ones. The paper acknowledges this limitation explicitly in Appendix A.2: "Compiled models will be more likely to be interpretable (e.g., the axis-aligned orthogonal residual stream bases in Tracr), and more likely to fit into existing paradigms for thinking about transformers. When using them to evaluate interpretability tools, we should be careful to make sure that the tools do not exploit this."

Mitigation status. The authors are transparent that this is a "fundamental limitation on the role compiled models can play" (Appendix A.2) and that evaluations based on Tracr should be treated as "a minimum bar rather than a full validation of a technique." No mitigation is proposed beyond this caveat. The paper cannot overcome the limitation—it is intrinsic to the compilation paradigm—but the framing as a minimum bar is a honest and useful way to position the contribution.

The Orthogonal Subspace Allocation Is Prohibitively Inefficient, Limiting the Complexity of Compilable Programs

The assumption or constraint. Tracr constructs the residual stream as the direct sum of all s-ops' input and output spaces: "we embed each s-op in its own orthogonal subspace, which is reserved for its sole use throughout the entire network" (Section 3, Step 5). Every categorical variable with k possible values occupies k dimensions. These dimensions persist through all layers even when the variable is not yet computed or no longer needed. The paper acknowledges this is "sparse and inefficient" (Section 5).

The consequence. The residual stream dimensionality grows linearly with the sum of vocabulary sizes, context length encodings, and intermediate variable cardinalities across all s-ops in the program. For the simple frac_prevs program with vocabulary size 4 and context length 3, this already requires 14 dimensions—12 of which are just input embeddings and auxiliary constants. For dyck-2 with two types of parentheses, the model balloons to 58 dimensions and 10 layers. This scaling is combinatorially unfavorable: adding a single categorical variable with 10 possible values adds 10 dimensions, and every such variable adds its dimensionality to the residual stream at every layer, even if used only briefly.

The practical consequence is that programs of even moderate complexity become prohibitively large. A program that processes a realistic vocabulary (say, 100 tokens), tracks multiple intermediate variables (each with their own finite but potentially large value sets), and uses multiple select-aggregate operations would produce models with hundreds or thousands of residual stream dimensions, making them computationally expensive to run and difficult to visualize (the forward-pass figures, which are Tracr's primary demonstration of correctness, become unreadable for residual streams larger than ~100 dimensions).

What evidence exists in the paper. The paper does not provide scaling data—no plot of residual stream dimension vs. program size, vocabulary size, or context length. The three reported model sizes (14, ~30–40 for sort_unique/sort, and 58 for dyck-2) are the only data points. We cannot extrapolate how large a model would be needed for, say, a program that processes 50-token vocabulary, context length 20, and 10 intermediate categorical variables. The paper does not report whether any attempted programs failed to compile due to excessive dimensionality.

Mitigation status. The compression experiments (Section 5) are presented partly as a mitigation: "compressed models could be more efficient and realistic" and "could address this limitation" (Appendix A.2). The frac_prevs model compresses from 14 to 6 dimensions without loss, demonstrating that much of the orthogonal subspace is redundant. However, compression requires training (1–4 hours per model, per Appendix E) and, as the sort_unique result shows, may not preserve the original algorithm. The paper does not propose a version of Tracr that compiles directly to a compressed representation without the full orthogonal intermediate step. An alternative approach—using a constraint optimization solver instead of the heuristic layer allocator—is mentioned (Appendix A.2) but not implemented.

Compressed Models May Not Preserve the Ground-Truth Computation

The assumption or constraint. The compression procedure (Section 5) trains a linear projection W to reduce residual stream dimensionality while approximately preserving the original model's computation, enforced by a layer-wise regularization loss that penalizes deviations from the original model's intermediate activations. The implicit assumption is that matching per-layer outputs and final task accuracy is sufficient to guarantee that the compressed model implements the same algorithm as the original compiled model.

The consequence. This assumption fails in at least one documented case. For the sort_unique model (Section 5.3, Figure 8), the compressed model achieves "nearly perfect accuracy" on the sorting task but the average cosine similarity of layer outputs stays around 0.8, and the authors attribute the discrepancy to the compressed model learning "a numerical encoding of the target position rather than a categorical encoding." In other words, the compressed model solves the task correctly through a different internal mechanism. This means the compiler's annotations—which document the original model's one-hot categorical encoding of target_pos—are no longer accurate descriptions of the compressed model's computation. The ground truth is lost.

This is a severe limitation for the paper's stated use case of evaluating interpretability tools. If a researcher compresses a Tracr model to make it more realistic, then applies an interpretability method that (correctly) identifies a numerical encoding of target positions, the method would appear to fail when compared to the original compiled model's ground-truth annotations (which say the encoding should be categorical). The evaluation would be misleading—the method is actually correct about the compressed model, but the ground truth is wrong. Without a way to verify what algorithm the compressed model actually implements, the compressed model cannot serve as reliable ground truth.

What evidence exists in the paper. Figure 8 provides the direct evidence: the squared difference between original and compressed layer outputs is large at the layer computing target_pos, and cosine similarity is ~0.8 rather than ~1.0. The paper acknowledges this finding explicitly: "This is both a setback for adding compression to the compiler—the compiler's annotations no longer serve as the exact ground truth—but also an opportunity. The ways neural networks solve algorithmic tasks regularly surprise researchers." However, it is unclear whether this is an isolated case or a general phenomenon. The frac_prevs compression (Figure 7) does not report layer-wise cosine similarity, so we do not know whether it remained faithful. Only two models were compressed, and one of them diverged—a 50% failure rate for faithfulness, though with an N of 2 this is not statistically meaningful.

Mitigation status. The paper does not propose a method for verifying whether a compressed model remains faithful to the original algorithm beyond the layer-wise cosine similarity it already reports. There is no formal verification procedure, no test for algorithmic equivalence, and no guideline for how to use compressed models as ground truth given the risk of divergence. The paper frames the divergence as "an opportunity" for future research but provides no concrete mitigation for practitioners who want to use compressed Tracr models as test cases for interpretability methods.

Difficulty Estimation Cost Is Not Accounted For in the Compression Experiments

The assumption or constraint. The compression experiments train W to minimize a loss that includes both final output accuracy and per-layer activation matching with the original compiled model. This training requires the original compiled model as an oracle for both the final output and the intermediate layer activations throughout training. The paper reports that each compression run takes 1–4 hours on two CPU cores (Appendix E).

The consequence. This is not a problem for the compression-as-research-experiment use case—the purpose is to study superposition, and training cost is a reasonable research expense. However, it undermines the implied practical workflow where a researcher compiles a model, compresses it to make it more realistic, and then uses it as a test case for interpretability. That workflow requires running the compression procedure (hours of training) for each model, which adds a substantial computational barrier. More importantly, if the goal is to compare the compressed model's internal representations to the original's (e.g., to see which features are preserved vs. discarded, as in Figure 7(c)), you need the original compiled model's activations as ground truth anyway—raising the question of why you wouldn't just use the original model directly for interpretability evaluation.

The broader issue is that the compression experiments themselves rely on having full access to the compiled model's internal state, which means they do not demonstrate a scenario where the compiled model is no longer needed. The compressed model is not a standalone artifact with known internal structure—its structure can only be analyzed relative to the original compiled model's ground-truth labels, which must be computed (and the original model must be run) to generate Figure 7(c)'s W^T W annotations. Without those annotations, the compressed model is just an opaque compressed transformer.

What evidence exists in the paper. The compression training setup (Section 5.1, Figure 6, Appendix E) makes explicit that the original model's layer outputs h_i(x) are required throughout training via the L_layer term. The analysis in Figure 7 relies on knowing which original dimensions correspond to which features (e.g., is_x, tokens:x), which comes from the original model's Tracr annotations. The paper does not report the computational cost of generating these oracle activations during training or the cost of running the original model for the W^T W analysis.

Mitigation status. Not addressed. The paper does not frame this as a limitation, nor does it discuss whether the compression could be done without access to per-layer activations (e.g., using only the output loss). The statement that "other weighting factors give similar results" (Section 5.1) suggests that L_layer might not be strictly necessary, but no ablation removing it is reported, so we cannot assess whether compression without layer-wise supervision would still produce analyzable models.

Expressivity and Realism Gaps Are Documented but Their Impact Is Unexplored

The assumption or constraint. Tracr imposes several expressivity restrictions on RASP that the paper documents but does not systematically evaluate: (1) no composite selectors with different input variables (Appendix B), (2) only categorical and numerical encodings with mandatory annotations, (3) no layer norm (Appendix D), (4) only deterministic computations with finite output sets (Step 2 of compilation), (5) binary attention patterns only (implicit in the selector-to-W_QK translation). These restrictions mean that Tracr cannot compile arbitrary RASP programs, and even compilable programs must be adapted to Tracr's constraints—"we had to modify a few programs to only use features supported by Tracr" (Section 4).

The consequence. The expressivity gap operates at two levels. First, there are RASP programs that Tracr simply cannot compile—programs using composite selectors, programs requiring numeric attention patterns, programs with probabilistic outputs. The paper does not characterize which program structures fall into this category or how common they are in practice. Second, and more subtly, the programs Tracr can compile may require modifications that alter the algorithm in ways that affect its relevance as ground truth. For example, the sort program that handles duplicates (Figure 9) cannot use the composite selector from Weiss et al. (2021)'s original implementation—instead, it must add a small multiple of indices to the keys to break ties. This modified algorithm is not the same as the original RASP program, and a transformer trained from scratch on sorting might implement yet a different algorithm. The compiled model is ground truth for the modified program, not for the task in general.

The lack of layer norm is particularly significant because "most transformer models used in practice" include it, and layer norm fundamentally changes the representational geometry of the residual stream. Tracr models have axis-aligned, unnormalized features where values are exactly 0 or 1 (for categorical) or precise scalars (for numerical). Real transformers with layer norm have rescaled and shifted activations where these clean representations are unlikely to survive. An interpretability method that works on Tracr's clean representations might fail on real models simply because of layer norm, not because of any algorithmic difference.

What evidence exists in the paper. The paper provides no experiment comparing the behavior of interpretability methods on Tracr models vs. similar trained models, with or without layer norm. It provides no analysis of how many Weiss et al. (2021) programs required modification to compile, or what fraction of "natural" RASP programs fall within Tracr's supported subset. The theoretical result in Appendix F characterizes one expressivity limitation (boolean OR of selectors with different inputs) but does not provide a general characterization of the compiler's capabilities.

Mitigation status. These limitations are documented transparently (Appendix A.2 and B), and the paper notes that "many of these limitations could be overcome in future versions of Tracr." Specific future directions mentioned include extending RASP to support numeric attention patterns (as suggested by Weiss et al., 2021), using constraint optimization for more efficient layer allocation, and rotating the basis for obfuscation. However, none of these mitigations are implemented, and some—like supporting layer norm—would require fundamental changes to how Tracr constructs weights, since the current approach relies on exact numerical values that layer norm would distort. The paper does not estimate how difficult these extensions would be or whether they would address the core expressivity limitations or merely the efficiency ones.

The Paper Does Not Demonstrate That Tracr Improves Interpretability Method Evaluation

The assumption or constraint. The paper's central motivating claim is that Tracr "could allow future work to directly test methods including, for example, classifier probes, gradient-based attribution, and causal tracing" (Section 1) and that compiled models "can serve as ground-truth for evaluating interpretability methods" (Abstract). This claim is presented as the primary contribution and the justification for building Tracr.

The consequence. The paper provides no experimental evidence that Tracr is effective for this purpose. No interpretability method is applied to a Tracr-compiled model. No comparison is made between what a method claims and what the compiled model's ground truth says. No benchmark is proposed, no metrics for evaluation quality are defined, and no case study demonstrates the workflow of using Tracr to debug or improve an interpretability technique. The one concrete example of Tracr being used for evaluation is cited as concurrent work (Conmy et al., 2023), not performed in this paper.

This is a significant gap between the paper's stated contribution and its experimental content. The compression experiments (Section 5) demonstrate an application of Tracr (studying superposition), and the forward-pass visualizations (Section 4) demonstrate that Tracr compiles programs correctly (qualitatively). But neither of these constitutes an evaluation of an interpretability method. The paper essentially delivers the infrastructure for evaluation without demonstrating that the infrastructure works for its intended purpose. A reader who wants to know "should I use Tracr to test my interpretability method?" gets no empirical guidance from this paper beyond the existence of the tool.

What evidence exists in the paper. The paper acknowledges this limitation indirectly by framing much of the discussion in future-tense: "Tracr could allow future work to directly test methods" (Section 1), "Future experiments could aim to further clarify" (Section 5.2), "we can use Tracr to create test cases and ultimately benchmarks for interpretability tools" (Section 7). The one existing use case cited is Conmy et al. (2023), who "successfully used Tracr to evaluate a method for automatically detecting circuits in transformer models." This external validation is encouraging but not a substitute for in-paper demonstration.

Mitigation status. The open-source release and the mention of Conmy et al. (2023) partially mitigate this limitation by showing that the tool is usable by others for the claimed purpose. However, the paper itself does not provide the evaluation demonstration that would convince a skeptical reader that Tracr-compiled models are better test cases than existing alternatives (synthetic data manipulation, reverse-engineered circuits, toy models). The authors position Tracr as enabling a new experimental paradigm, but without a single in-paper demonstration of that paradigm in action, the claim remains a promissory note.

7. Implications and Future Directions

How This Work Changes the Landscape

Tracr does not solve interpretability. It changes how interpretability research can be validated by providing a new experimental paradigm: models with fully known computational structure that serve as calibrated test cases for methods that aim to reveal computational structure. This is a methodological contribution, not an algorithmic one, and its significance lies in the shift from plausibility-based to fidelity-based evaluation of interpretability tools.

Before Tracr, interpretability methods were validated through indirect means. A probing classifier that claimed feature F was represented in layer L could be assessed by whether it predicted F accurately and whether removing information from layer L degraded performance—but neither test verifies that the model actually uses F in layer L in the way the probe claims. An attribution method could be evaluated by whether its importance scores aligned with human intuition about which input tokens mattered—but human intuition is not ground truth about model mechanisms. These evaluation strategies establish plausibility, not correctness. Tracr changes the evaluation landscape by providing models where correctness can be checked directly: does the method recover the known attention pattern, the known feature encoding, the known information flow? If not, the method has a measurable failure, and the failure can be diagnosed precisely because the correct answer is available for comparison.

This is not a paradigm shift on the scale of, say, the transformer architecture itself. It is a new experimental instrument—analogous to how the introduction of well-characterized model organisms (C. elegans with its fully mapped neural connectome, or Drosophila with its tractable genetics) transformed biology not by answering all questions, but by providing a system where hypotheses could be tested rigorously before scaling to more complex organisms. Tracr-compiled models are model organisms for interpretability: simple enough to understand completely, complex enough to exhibit non-trivial transformer behaviors (multi-step algorithms, attention head composition, residual stream communication), and capable of surfacing failure modes that would be invisible in toy models. The analogy is apt in another way: model organisms did not replace the study of humans, just as Tracr will not replace the study of real LLMs—but they accelerated biological discovery by enabling controlled experiments that were impossible in complex systems.

The work resolves a specific tension in the interpretability literature: whether evaluation of methods should prioritize faithfulness (the explanation accurately describes the model's actual computation) or plausibility (the explanation is consistent with observed behavior and human understanding). The tension exists because faithfulness is impossible to measure without ground truth, leading some researchers to argue that plausibility is the only realistic standard. Tracr partially dissolves this tension by providing ground truth for a restricted class of models, making faithfulness measurable for those models. The resolution is partial—measuring faithfulness on a compiled model does not guarantee faithfulness on a real model—but it provides an empirical foothold where none existed before.

The paper also reframes what counts as a "successful interpretation." In the compiled model paradigm, an interpretation is successful if it recovers the actual mechanism, not if it merely produces a story that fits the data. This reframing has downstream consequences for method development: techniques that produce compelling narratives but cannot recover known mechanisms in compiled models should be deprioritized, while techniques that reliably recover compiled model structure become candidates for application to real models. This is a filtering function—Tracr can eliminate methods that fail in the simplest verifiable setting, saving researchers from investing further in approaches that are fundamentally incapable of recovering mechanistic truth.

The compression experiments (Section 5) contribute a second conceptual shift: they demonstrate that superposition is not just an emergent property of training but can be deliberately induced by dimensionality reduction on a fully understood model. This connects superposition research to the broader literature on compressed sensing and dictionary learning (Donoho, 2006; Aharon et al., 2006), suggesting that superposition might be studied as a representation learning problem with known ground-truth features, rather than purely as an emergent phenomenon in black-box training. The finding that compressed models can discover alternative algorithmic implementations (the sort_unique numerical encoding vs. categorical one-hot) further suggests that compression is not a neutral transformation—it can change what algorithm the model implements, not just how it represents the features of that algorithm. This complicates the simple narrative that superposition is merely about packing more features into fewer dimensions; it can also be about finding more efficient algorithms that achieve the same input-output mapping.

Follow-Up Research This Work Enables

Systematic benchmarking of interpretability methods on compiled model test suites. The paper's strongest motivating claim—that Tracr can evaluate interpretability tools—remains experimentally undemonstrated in this paper. A direct follow-up would construct a benchmark of 10–20 compiled models spanning different algorithmic structures (sorting, counting, sequence manipulation, tree-structured computation like Dyck-n, graph algorithms implementable in RASP) and varying complexity (2–10 effective layers, 1–8 attention heads, residual stream dimensions from 10 to 200). For each model, the ground truth would include: which dimensions encode which features at each layer, which attention heads implement which selectors, and how information flows between components. A suite of standard interpretability methods—linear probes, causal tracing (Meng et al., 2022), activation patching, automated circuit discovery (as in Conmy et al., 2023)—would be evaluated on their ability to recover these known properties. The key metric would be precision and recall of recovered features, attention patterns, and information flow edges, measured against the compiled ground truth. Such a benchmark would immediately reveal which methods are reliable at which scales of model complexity, and which failure modes (e.g., conflating correlated features, missing sparse features, misattributing causal responsibility) are most common. The existence of this benchmark would transform interpretability from a field where methods are evaluated anecdotally on different models to one with standardized, quantitative comparison.

Characterizing when and why compressed models diverge from the original algorithm. The sort_unique compression result (Section 5.3) is a single data point showing that compressed models can solve tasks correctly through different internal mechanisms. A systematic study would compress 10–20 Tracr models of varying algorithmic types and measure: (a) at what compression ratio does layer-wise cosine similarity begin to drop below 0.95, (b) which types of operations are most vulnerable to algorithmic divergence (e.g., attention patterns vs. MLP computations, categorical vs. numerical encodings, early vs. late layers), and (c) whether the alternative algorithms discovered by compression correspond to known alternative RASP programs for the same task. The study would vary the compression architecture (shared W vs. per-layer W, training only W vs. fine-tuning all weights) to see whether more flexible compression increases or decreases algorithmic divergence. A strong result would be a predictive model: given properties of a RASP program (number of categorical variables, depth of dependency chains, sparsity patterns), predict whether compression to d dimensions will preserve or alter the algorithm. This directly extends the paper's finding that "the compressed model seems to learn to use a different (numerical) encoding" and would transform it from an anecdote into a systematic understanding of when ground-truth annotations survive compression.

Using Tracr to study how feature redundancy and dependency structure determine superposition patterns. The paper observes that is_x and tokens:x share embedding space in the compressed frac_prevs model "because the variables encode similar information" (Section 5.2). This is a qualitative observation on one model. A systematic study would design a family of RASP programs that vary redundancy parametrically: for example, a program with K independent boolean features vs. K features where pairs are deterministically linked vs. K features with probabilistic correlations. All programs would produce the same output dimensionality in the compiled model. Compress each to the same target dimension d and measure: do redundant feature pairs consistently share more embedding space than independent features? Does the degree of sharing scale with the mutual information between features? Does the compression discover redundancy that was not explicitly designed into the RASP program (e.g., features that are logically independent but de facto correlated in the input distribution)? This would extend prior superposition theory (Elhage et al., 2022b) from the domain of sparsity-driven superposition to information-theoretic superposition, where the relevant variable is not co-occurrence frequency but mutual information. The experiment requires Tracr's ground-truth feature labels to compute feature-feature redundancy and to identify which compressed dimensions correspond to which original features—making it tractable where it would be impossible on trained models.

Developing and evaluating dictionary learning methods for reversing superposition. The paper's compression procedure (Figure 6) embeds known features into a lower-dimensional space via a learned matrix W. The inverse problem—recovering the original features from the compressed representation—is a dictionary learning problem: given activations in the compressed space, find a set of basis vectors that correspond to the original disentangled features. Because the original features are known (they are the Tracr-labeled dimensions), Tracr-compressed models provide perfect ground truth for evaluating dictionary learning algorithms (Donoho, 2006; Aharon et al., 2006). A strong experiment would: (a) compress several Tracr models to varying degrees, producing known superposition structures, (b) apply existing dictionary learning algorithms (sparse coding, ICA, non-negative matrix factorization) to the compressed activations, and (c) measure how many original features each algorithm recovers, and at what compression ratios recovery becomes impossible. The experiment would also introduce controlled "impurities"—e.g., adding noise to the compressed activations or using a nonlinear compression instead of linear—to test robustness. This directly addresses the research direction the paper flags in Appendix A.1: "A complementary approach is to try reversing the superposition induced by a compression procedure, e.g., using ideas from compressed sensing and dictionary learning." The paper provides the perfect testbed but does not conduct the experiment.

Compiling and compressing models that implement algorithms studied in the mechanistic interpretability literature. The paper demonstrates compilation of simple algorithmic tasks (counting, sorting, parenthesis checking). A natural extension is to compile RASP implementations of algorithms that have been reverse-engineered in trained transformers: modular addition (Nanda et al., 2023), induction heads (Olsson et al., 2022), or indirect object identification (Wang et al., 2023). For each algorithm, a two-part experiment: (a) compare the Tracr-compiled implementation to the reverse-engineered circuit reported in the literature—do they use the same computational structure, or does the compiled model reveal alternative implementations that the reverse-engineering might have missed? and (b) compress the compiled model and test whether the compressed version converges to the reverse-engineered circuit (suggesting the reverse-engineered circuit is a compressed representation of a more explicit algorithm) or diverges to something else (suggesting the reverse-engineered circuit is one of many possible implementations). This would bridge the gap between Tracr's constructed models and the reverse-engineered circuits that currently serve as the closest thing to ground truth in mechanistic interpretability. A negative result—the compressed Tracr model does NOT converge to the published reverse-engineered circuit—would be equally informative, suggesting that learned models find algorithms systematically different from compiled ones.

Extending Tracr to support layer norm and training compressed models with layer norm to study its effect on representation geometry. The paper removes layer norm from the transformer architecture because Tracr's exact weight construction assumes unnormalized residual stream values. But most real transformers use layer norm, which rescales and shifts activations. A significant extension would be to implement layer norm in Tracr-compiled models by modifying the compilation to account for the normalization: after each layer's output is added to the residual stream, the activations are normalized, which means the compiled weights must be designed to produce the correct post-normalization values. This is non-trivial because layer norm is a data-dependent operation—the normalization statistics depend on the specific input—but for Tracr models where the possible activation values are known (from Step 2's value inference), the normalization could be pre-computed for each possible input configuration. With layer norm supported, the key experiment would be: train compressed Tracr models with and without layer norm, and measure whether layer norm makes the compressed representations harder to interpret (do features become less axis-aligned?) or easier (does layer norm reduce interference between features sharing dimensions?). This would provide the first controlled study of how architectural choices affect interpretability, with known ground-truth features.

Practical Applications and Downstream Use Cases

Automated regression testing for interpretability tools during development. Interpretability tool developers (e.g., teams building automated circuit discovery, new probing methods, or causal tracing algorithms) can integrate a suite of Tracr-compiled models into their continuous integration pipeline. Before releasing a new version of their tool, they run it on the Tracr suite and check that the recovered mechanisms match the known ground truth to within a specified tolerance. If a code change causes the tool to miss attention heads it previously identified, or to misattribute features to wrong layers, the regression is caught immediately. This is analogous to unit testing in software engineering—the compiled models serve as deterministic test cases with known expected outputs. The specific benefit is catching bugs that would otherwise manifest as plausible-but-wrong interpretations on real models, where the bugs would be invisible because ground truth is unavailable. The numbers from the paper that justify this: Tracr models can be compiled for programs of varying complexity (from 14-dimensional frac_prevs to 58-dimensional dyck-2), providing a gradient of difficulty for test cases. The compilation is deterministic and models can be regenerated on demand, making them suitable for automated testing.

Curriculum for training interpretability researchers. The forward-pass visualizations in Figures 2, 5, and 9–11 are pedagogically powerful because they show the residual stream contents at every layer with semantically labeled dimensions. An interpretability training curriculum could use Tracr-compiled models as progressively more complex "dissection specimens": start with frac_prevs (one effective attention head, one effective MLP) and have trainees predict what the attention patterns and MLP computations should look like, then verify against the compiled model; progress to sort_unique (head composition, the selector_width primitive); then to dyck-2 (multi-step algorithm with subroutines, multiple attention heads coordinating). At each stage, trainees could apply standard interpretability tools (probes, attention pattern visualization, activation patching) and compare their findings to the known ground truth, developing calibrated intuition for what correct and incorrect interpretations look like. The specific benefit is accelerating the notoriously difficult skill of "reading" transformer internals—a skill that currently requires months of apprenticeship on real models where feedback is ambiguous. With Tracr, feedback is immediate and unambiguous: the ground truth tells you whether you correctly identified which head does what.

Screening new interpretability methods before investing in large-scale validation. A research group proposing a new interpretability technique (e.g., a novel feature visualization method, a new causal intervention strategy, or an information-theoretic measure of layer importance) can first test it on Tracr-compiled models of increasing complexity. If the method fails to recover known structure on sort_unique (2 attention heads, 2 MLPs, ~30–40 dimensions), it is unlikely to succeed on GPT-2 scale models, and the researchers can iterate on the method in a fast feedback loop (minutes to run on a compiled model vs. hours or days on a trained model). If the method passes the Tracr tests, the researchers have stronger evidence that it is worth the substantial computational and human effort of validating on real trained models. This addresses the current inefficiency in interpretability research, where methods are often developed and published based on plausibility arguments on real models, only for subsequent work to reveal that the interpretations were incomplete or incorrect (as happened with induction heads—Chan et al., 2022). The numbers from the paper: compiled models run on CPU in seconds (the models are small), and the open-source release provides a library of example programs, making the screening workflow technically trivial to implement.

When to Prefer This Method

The paper does not position Tracr as a method to be preferred over alternatives in a deployment sense—it is a research infrastructure tool, not a technique for interpreting a specific model or a replacement for existing approaches. However, the paper does articulate a clear decision rule for when compiled models are appropriate vs. when other ground-truth strategies should be used:

• Use Tracr-compiled models when the goal is to evaluate whether an interpretability method can recover known computational mechanisms, the method operates on transformer-specific components (attention heads, MLP layers, residual stream), and the algorithmic structure of interest is expressible in RASP's primitives (elementwise operations, select-aggregate operations on finite categorical or numerical variables). This covers evaluation of probing classifiers, attention pattern analysis, causal tracing through layers, and automated circuit discovery—all methods that aim to produce mechanistic explanations of transformer computation.

• Use synthetic data manipulation approaches (Yang and Kim, 2019; Adebayo et al., 2020) when the method being evaluated operates at the level of input-output behavior rather than internal mechanisms (e.g., feature attribution methods that assign importance to input features without modeling internal computation), or when the model architecture is not a transformer (e.g., CNNs).

• Use reverse-engineered circuits (Wang et al., 2023; Nanda et al., 2023) when the goal is to evaluate interpretability methods on realistic learned representations rather than hand-constructed ones, accepting that the ground truth may be incomplete or partially incorrect. Tracr and reverse-engineered circuits are complementary: Tracr provides perfect ground truth for simple models, while reverse-engineered circuits provide approximate ground truth for more realistic models.

• Do not use Tracr-compiled models as the sole validation for methods intended to scale to large language models. The paper is explicit that evaluations on compiled models are a "minimum bar rather than a full validation" (Appendix A.2)—passing Tracr tests is necessary but not sufficient evidence that a method will work on real models at scale.