Transformer Circuits Thread Notes

The Transformer Circuits thread (opens in a new tab) is the running publication venue for Anthropic’s interpretability team. Since 2021 it has introduced, paper by paper, most of the standard toolkit of mechanistic interpretability: the residual-stream view of transformers, induction heads, superposition, sparse autoencoders, and attribution graphs.

These are my notes on the thread, one post per paper (or pair of closely related papers). Each post works through the paper’s core mechanism: how the thing actually works, how the experiments bear on it, and where the explanation runs out. Their structure follows the material rather than a fixed review template. The originals are free, interactive, and unusually well produced; these posts are written to be read alongside them, not instead of them.

The notes

Notes on A Mathematical Framework for Transformer Circuits

Notes on the 2021 paper that reframes transformers as operations on a shared residual stream: QK/OV circuits, virtual weights, and path decompositions of attention-only models.

Notes on In-Context Learning and Induction Heads

Notes on induction heads, the training phase change in which they form, and the six-part argument connecting them to in-context learning.

Notes on Toy Models of Superposition

Notes on how sparse features fit into superposition, the geometry and interference that result, and a toy-model account of memorization and double descent.

Notes on Towards Monosemanticity and Scaling Monosemanticity

Notes on decomposing MLP activations into interpretable features with sparse autoencoders, from a one-layer transformer to Claude 3 Sonnet.

Notes on Sparse Crosscoders

Notes on sparse crosscoders: one shared feature dictionary across layers or across models, and what that enables for cross-layer analysis and model diffing.

Notes on Circuit Tracing and the Biology of a Large Language Model

Notes on cross-layer transcoders, local replacement models, and attribution graphs — and the case-study companion paper on Claude 3.5 Haiku.

Notes on Tracing Attention Computation Through Feature Interactions

Notes on QK attributions, attention-head loadings, and what feature interactions reveal about why a transformer attends to one token rather than another.

The order is chronological, and chronology here is also the dependency graph: each paper leans on the vocabulary of the ones before it. The framework paper defines circuits and the residual stream; induction heads are the first circuit found in the wild; toy models explain why neurons are the wrong unit of analysis; the monosemanticity papers build the dictionary-learning response; crosscoders generalize dictionaries across layers and models; attribution graphs assemble all of it into end-to-end explanations of model outputs; and QK attributions recover the attention decisions those graphs had frozen in place.

Where the thread came from

The thread is a continuation of two earlier bodies of work on vision models, both published at Distill. The one-line descriptions below are the papers’ own.

Distill (feature visualization era):

• 2017-11 Feature Visualization (opens in a new tab) — How neural networks build up their understanding of images.
• 2018-06 The Building Blocks of Interpretability (opens in a new tab) — Interpretability techniques are normally studied in isolation. We explore the powerful interfaces that arise when you combine them — and the rich structure of this combinatorial space.
• 2019-03 Activation Atlas (opens in a new tab) — By using feature inversion to visualize millions of activations from an image classification network, we create an explorable activation atlas of features the network has learned and what concepts it typically represents.

The Distill Circuits thread:

• 2020-03 Zoom In: An Introduction to Circuits (opens in a new tab) — By studying the connections between neurons, we can find meaningful algorithms in the weights of neural networks.
• 2020-06 Curve Detectors (opens in a new tab) — Every vision model we’ve explored in detail contains neurons which detect curves.
• 2020-12 Naturally Occurring Equivariance in Neural Networks (opens in a new tab) — Neural networks naturally learn many transformed copies of the same feature, connected by symmetric weights.
• 2021-01 Curve Circuits (opens in a new tab) — We reverse engineer a non-trivial learned algorithm from the weights of a neural network and use its core ideas to craft an artificial artificial neural network from scratch that reimplements it.
• 2021-02 Visualizing Weights (opens in a new tab) — We present techniques for visualizing, contextualizing, and understanding neural network weights.
• 2021-04 Branch Specialization (opens in a new tab) — When a neural network layer is divided into multiple branches, neurons self-organize into coherent groupings.
• 2021-04 Weight Banding (opens in a new tab) — Weights in the final layer of common visual models appear as horizontal bands.

Anthropic’s Transformer Circuits thread (the papers these notes cover):

• 2021-12 A Mathematical Framework for Transformer Circuits (opens in a new tab) — Our early mathematical framework for reverse engineering models, demonstrated by reverse engineering small toy models.
• 2022-03 In-Context Learning and Induction Heads (opens in a new tab) — An exploration of the hypothesis that induction heads are the primary mechanism behind in-context learning. We also report the existence of a previously unknown phase change in transformer language models.
• 2022-09 Toy Models of Superposition (opens in a new tab) — Neural networks often seem to pack many unrelated concepts into a single neuron — a puzzling phenomenon known as ‘polysemanticity’. We build toy models where the origins and dynamics of polysemanticity can be fully understood.
• 2023-01 Superposition, Memorization and Double Descent (opens in a new tab) — We extend our previous work on toy models to shed light on how models generalize beyond their training data.
• 2023-10 Towards Monosemanticity: Decomposing Language Models With Dictionary Learning (opens in a new tab) — Using a sparse autoencoder, we extract a large number of interpretable features from a one-layer transformer.
• 2024-05 Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet (opens in a new tab) — Using a sparse autoencoder, we extract a large number of interpretable features from Claude 3 Sonnet. Some appear to be safety-relevant.
• 2024-10 Sparse Crosscoders for Cross-Layer Features and Model Diffing (opens in a new tab) — A preliminary note on a way to get consistent features across layers, or across models.
• 2025-03 Circuit Tracing: Revealing Computational Graphs in Language Models (opens in a new tab) — We describe an approach to tracing the “step-by-step” computation involved when a model responds to a single prompt.
• 2025-03 On the Biology of a Large Language Model (opens in a new tab) — We investigate the internal mechanisms used by Claude 3.5 Haiku in a variety of contexts.
• 2025-07 Tracing Attention Computation Through Feature Interactions (opens in a new tab) — We describe and apply a method to explain attention patterns in terms of feature interactions, and integrate this information into attribution graphs.

The ongoing tail (major papers since, not yet covered by a post):

• 2025-10 When Models Manipulate Manifolds: The Geometry of a Counting Task (opens in a new tab) — We find geometric structure underlying the mechanisms of a fundamental language model behavior.
• 2025-10 Emergent Introspective Awareness in Large Language Models (opens in a new tab) — We find evidence that language models can introspect on their internal states.
• 2026-04 Emotion Concepts and their Function in a Large Language Model (opens in a new tab) — We find representations of emotion concepts in Claude Sonnet 4.5 and show that they causally influence its outputs.
• 2026-05 Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations (opens in a new tab) — We train Claude to translate its internal state into natural language.
• 2026-07 Verbalizable Representations Form a Global Workspace in Language Models (opens in a new tab) — We find that Claude maintains a small, privileged set of representations it can report on, control, and reason with, atop a much larger volume of automatic processing.

The thread also publishes shorter monthly “Circuits Updates”; these notes stick to the major papers.