Deep learning notes and projects covering neural networks, sequence models, language models, generative modeling, and interpretability.
Neural networks, mostly the language side of them. There's writing on search and semantic search, how language models work, a survey of text diffusion, and some interpretability work that looks at what a model has actually learned.
It overlaps a lot with the machine learning page. This is the part that's specifically about deep learning and language models.
Paper-by-paper notes on Anthropic's Transformer Circuits thread: core claims, methods in plain language, and what the evidence does and doesn't show.
Notes on the 2021 paper that reframes transformers as operations on a shared residual stream: QK/OV circuits, virtual weights, and path decompositions of attention-only models.
Notes on induction heads, the training phase change in which they form, and the six-part argument connecting them to in-context learning.
Notes on how sparse features fit into superposition, the geometry and interference that result, and a toy-model account of memorization and double descent.
Notes on decomposing MLP activations into interpretable features with sparse autoencoders, from a one-layer transformer to Claude 3 Sonnet.
Notes on sparse crosscoders: one shared feature dictionary across layers or across models, and what that enables for cross-layer analysis and model diffing.
Notes on cross-layer transcoders, local replacement models, and attribution graphs — and the case-study companion paper on Claude 3.5 Haiku.
Notes on QK attributions, attention-head loadings, and what feature interactions reveal about why a transformer attends to one token rather than another.
How moral fine-tuning on iterated prisons changes LLMs.
Writing a FlashAttention CUDA kernel from scratch, tiling the attention matrix to avoid materializing N×N memory, building a KV cache for token generation, and running GPT-2 with custom kernels end-to-end.
A written version of my Latent Space talk on HRM: how the architecture works, and what the ARC Prize analysis says actually drives its performance.
Some notes and observations from my time at the 2025 International Conference for Machine Learning (ICML)
A written version of my Latent Space Paper Club talk surveying diffusion models for language, from denoising autoencoders and DDPM through LLaDA, Block Diffusion, and d1.
What's an embedding vector, and how can we use neural networks to improve the relevance of search results?
From n-grams to ChatGPT, how language models work and how they can be used to solve real-world problems.
Essential terms and concepts of how search works. Queries, indexes, & relevance.
