Open Source Mechanistic Interpretability Tools
The MI ecosystem, from activation hooking to circuit tracing — one field, one table.
The mechanistic interpretability (MI) ecosystem has matured rapidly, evolving
from ad-hoc PyTorch scripts into a rich toolkit of specialized frameworks. Depending on whether you are looking to patch
activations, train Sparse Autoencoders (SAEs), or visualize attention circuits, the open-source landscape offers tailored
tools for each layer of the stack.
| Category | Tool | Description | Best For | Organization | Source |
|---|---|---|---|---|---|
| Core Interventions & Activation Hooking | TransformerLens | The standard framework in MI research, originally created by Neel Nanda. Re-implements popular transformers (e.g., Llama, Gemma, GPT-2) into a standardized format with explicit, named HookPoints throughout the architecture, plus built-in utilities for activation caching, logit lens readouts, and causal activation patching. |
Analyzing circuit behavior, prompt interventions, and reverse-engineering attention heads on standard transformer architectures. | TransformerLensOrg (community) · originated by Neel Nanda, ex‑Anthropic | GitHub |
| Core Interventions & Activation Hooking | NNsight (and NDIF) | A flexible framework that operates directly on native PyTorch models without requiring manual model re-implementation. Intercepts and modifies tensor streams contextually, and supports remote execution on frontier-scale models hosted on remote clusters. | Large-scale models (70B+ parameters), non-standard architectures, or experiments requiring distributed multi-GPU interventions. | Northeastern University · National Deep Inference Fabric (NDIF) | GitHub |
| Core Interventions & Activation Hooking | pyvene | A declarative intervention library that treats interventions as first-class primitives. Allows configuring complex intervention schemas — e.g., swapping internal representations across multiple forward runs simultaneously — using clean configuration dictionaries. | Causal mediation analysis, distributed alignment search (DAS), and formal causal abstraction experiments. | Stanford NLP Group | GitHub |
| Feature Disentanglement & Sparse Autoencoders (SAEs) | SAELens | The go-to toolkit for training, evaluating, and running Sparse Autoencoders on neural activations. Integrates seamlessly with TransformerLens and Hugging Face models to decompose dense activation vectors into monosemantic, human-interpretable features. | Dictionary learning, feature extraction, and steering model outputs via sparse concepts. | Decode Research (Joseph Bloom, Curt Tigges, et al.) | GitHub |
| Feature Disentanglement & Sparse Autoencoders (SAEs) | Gemma Scope | An open suite of JumpReLU sparse autoencoders trained on every layer and sub-component of the Gemma model family — thousands of pre-trained, open-weight SAEs, saving massive amounts of GPU compute. | — | Google DeepMind | Hugging Face |
| Feature Disentanglement & Sparse Autoencoders (SAEs) | SAE Bench | A comprehensive benchmark of eight diverse SAE evaluations (unsupervised metrics and downstream tasks), open-sourcing a suite of 200+ SAEs across eight architectures and training algorithms. | — | Independent researchers (Adam Karvonen et al.) · interactive interface hosted by Neuronpedia | GitHub |
| Circuit Tracing & Automation | Anthropic Circuit-Tracing Tools | Open-source tools (circuit-tracer) that map internal computation pathways, generating attribution graphs that visually reveal how concepts and features interact layer-by-layer to produce specific outputs. |
Tracing complex multi-step reasoning pathways or understanding safety-relevant behaviors like refusal. | Anthropic (Anthropic Fellows Program) | GitHub |
| Visual Exploration & Notebook UIs | Neuronpedia | An open-source web platform and interactive interface for exploring SAE features and attribution graphs. Search for abstract concepts (e.g., "code syntax," "deception," or specific entities) and inspect feature activations across dataset samples interactively. | — | Decode Research (Johnny Lin, Joseph Bloom) · supported by Anthropic, Open Philanthropy, and others | Site |
| Visual Exploration & Notebook UIs | CircuitsVis | A JavaScript/Python visualization library for inline Jupyter notebook analysis — renders interactive attention head patterns, vector projections, and activation patching heatmaps directly in the research environment. | — | TransformerLensOrg · created by Alan Cooney and Neel Nanda | GitHub |