Every word takes a vast amount of math.
Ask a chatbot something and it answers one word at a time. Each word comes out of a huge network of simple parts called neurons. Each neuron is just a number that goes up or down. Send the message and watch them while it writes.
Same model. Closer each step.
Ask anything.
Illustration. Neurons that respond to the same kind of thing are drawn as one patch so you can see the pattern; real models don’t lay them out this way. The model in the essay, Qwen3-32B, has more than a million neurons over dozens of layers.
A neuron can fire for many things in two very different ways.
Picture a map where similar things sit close together: fruit near other round things, dogs near other animals, cars near buses. A neuron lights up part of that map.
Generalization: the neuron covers one neighborhood of similar things, because what they share is useful.
Cramming: the neuron covers scattered spots with nothing in common. A model out of room stuffs unrelated things together because they rarely show up at once. Researchers call this superposition.
Same map. Two kinds of neuron.
Illustration · a map of things a model has seen, with similar things close togetherIn the essay: Cammarata tested this by squeezing one layer of an image model from 480 neurons down to 10 and retraining. The neurons didn't scramble. They got coarser: one “curves and loops” neuron instead of ten specific curve detectors, like seeing the same picture at lower resolution.
That difference is the whole ballgame. With generalization, what a neuron groups together tells you how the model thinks. With cramming, it tells you nothing, and you have to untangle the model before you can read it. For years, the field believed the second was common.
A finished neuron can look like cramming when it isn't.
Here's a real neuron from the essay. In the finished model it fires for flower fields, tennis balls, green apples, shaggy dogs and broccoli: scattered spots all over the map. Cramming, right?
Cammarata didn't stop at the finished model. Models learn gradually, and you can save copies along the way. So he watched the neuron learn. Next to it is a made-up neuron that really is cramming, so you can compare.
Same finished neurons. Two different histories.
The key idea · Illustration of a neuron in the essayAn epoch is one pass through the training data. Neuron A is InceptionV1's last-layer neuron 338, which Cammarata retrained from scratch. Circle sizes are approximate, read from the essay's description, and the links are my reading of it.
Neuron A grew one step at a time. It starts on flower fields, a few with a dog sitting in them. Then dogs. The flowers are yellow-green and round, and soon so are tennis balls and apples. Every new spot touches an old one. It's generalization that kept going, which Cammarata calls recursive generalization. He says every neuron he watched grow did this.
Or find the one thing they share.
There's a second way to tell them apart. A real shared pattern predicts new cases. Cramming doesn't.
The most famous “cramming” neuron fires for cats and cars. For years it was the textbook example. Here are sixteen pictures hiding four groups of four, and one of the groups is what this neuron actually detects. Poke at them the way a researcher would.
Sixteen pictures. Four hidden groups.
Based on a neuron in the essayNow the real question. Which group does the cats-and-cars neuron fire for?
In the essay: neuron 4e:55 in InceptionV1. Its inputs include V-shaped flames and fox snouts, and round things push it negative, including clocks and a snail. The other pictures here are mine.
Cats and cars are a trap. They fit “animals” and “vehicles” just fine, and that's how people described this neuron for years. Cammarata spent a few seconds asking what one thing it might detect. Things that come to a point.
So why did we miss it?
Mostly, researchers only had the last frame.
The flower-field neuron from section 3, at epochs 0.7, 1.5, 3, 6 and 12. For the original 2014 model, the earlier frames were lost. On a phone, swipe the strips sideways.
Without the earlier frames, a neuron that grew one step at a time looks exactly like one that crammed. The tools for reading finished neurons made it worse. The program that draws what a neuron wants kept drifting toward dogs, and text neurons got judged on short snippets that cut off the context they were tracking. Anything researchers couldn't explain got filed under cramming. The essay calls it “superposition of the gaps.”
Two things changed. Newer model designs, built only to train better, happen to settle into cleaner neurons. And the tools caught up: fixed picture drawing, saved checkpoints, and AI agents that test explanations one neuron at a time, at scale. In 2023, an automated study of an earlier model could explain about 2% of neurons. In a preliminary 2026 audit of Qwen3-32B, Cammarata's team reports about 90% have one main job.
Back to Boulder.
Remember the neuron we zoomed into while the chatbot wrote? Every word lit up thousands of neurons at once, but we followed just that one and recorded its number. Here's the record, lined up under the words.
Same game as the pictures. What one thing is it tracking?
One neuron's record. Three guesses.
Illustration of a neuron in the essayTest your guess on a different city.
Pick a guess. Then we'll test it on a trip somewhere else.
It's a Colorado neuron. It fires on the commas because language models tend to keep a running summary on punctuation. An automated tool once labeled it a “comma neuron,” the text version of “cats and cars.” Turn it up in an unrelated conversation and the model wants to go hiking:
In the essay: a neuron in Qwen3-32B that tracks Colorado, summarized on commas, confirmed by turning it up. Both answers here are mine, and the records are drawn to match the essay's description.
That's the shift. The black box from the top of the page isn't fully open. But one neuron at a time it's readable, and each neuron you solve makes its neighbors easier.
Some of this is shown. Some of it is a bet.
| Claim | Status | Evidence in the essay |
|---|---|---|
| The famous “cramming” neurons were generalization | Shown | The cat-and-car neuron detects pointed shapes; the original dog-and-car example was a shared pose. |
| Neurons grow one sensible step at a time | Preliminary | Seen wherever he watched neurons learn. |
| Most chatbot neurons have one main job | Preliminary | Qwen3-32B audits; details are coming in a technical paper. |
| Understanding whole models may be within reach soon | Speculative | He labels this highly speculative himself. |
Cammarata works at Goodfire, a company that does exactly this kind of work, so he has a stake in it. And cramming isn't ruled out. He argues it's rare, and researchers who build untangling tools will push back.
Here's the lesson I'm keeping. A snapshot can make a pattern look like noise. Before calling something a mess, look at how it got that way.