A Most Comfortable Dead End

Right now, without asking you, your body is making mistakes on purpose. In a lymph node somewhere, a white blood cell is copying the gene for an antibody and getting it wrong, introducing errors at roughly a million times the normal rate, producing thousands of garbled variants, nearly all of them useless. Nothing in there is deciding anything. The cell cannot know which virus you will meet next month, so it does not aim. It scatters strangeness in bulk, and whatever happens to fit gets kept. This is not a flaw in the immune system. It is how the immune system works. A body that produced only the one correct, average antibody would be defenseless against anything it had not already seen, which is to say against the future.

Error is not the opposite of intelligence. In any system that has to survive a world it cannot predict, error is where the intelligence comes from.

We are now building, at staggering expense, a machine that does the reverse.

A large language model is an averaging engine. It can recombine its material in ways that genuinely surprise you, but its center of gravity is the middle: the likeliest next word, the sentence most people would have written, the median thought in clean prose. That is the design, not a defect, and it is seductive because the middle works. It answers the email, clears the filter, passes for competent. Sounding strange is something you pay for in being misunderstood.

The usual story says this is being done to us. But nobody is taking anything. We hand it over with a small exhale of relief, the way you tap the suggested reply instead of finding your own words. The machine did not steal our strangeness. It offered to spare us the cost of having any, and the cost is real, and the offer is free.

For most of history people made things by copying: songs set to borrowed tunes, icons painted to a pattern. And the copying was never clean. That is the part that matters. A scribe’s hand slipped. A singer misremembered a verse and the mistake was better than the line. Consider the pea on your plate. For centuries the vegetable was “pease,” a mass noun like rice, with no singular at all. People heard the final sound as a plural, sliced it off, and invented a lone “pea” that had never existed. Your language is full of these fossilized errors. You used several this morning.

Variation came from everybody, thrown off by people reaching for something and missing by a little — ten thousand wrong antibodies to find the one that works. The people we call original did not invent novelty. They foraged it from drift that nobody meant to make.

So the loss does not fall on the few first. It falls on the ground they stood on. What the machine removes is the scatter: it copies without error, and asked for the average sentence it gives you the average sentence, immaculate, none of the grit a tired hand leaves behind. You cannot mutate your way out of a monoculture when there is nothing left to mutate from.

We have heard all this before, and it was wrong every time. The printing press was going to flatten the scribal hand; photography would kill painting; recorded sound, the photocopier, the sampler, the web. Each was going to drown us in sameness, and each instead split the culture open and handed tools to people who had never had any.

Why is this different? Because those were machines for reproduction, and this is something stranger: a machine that generates, always toward the middle, and then feeds on what it makes. The press spread what humans wrote, errors and oddities and all. Many presses, many hands, every one of them wrong in its own way. It never read its own output, regressed toward its own average, and trained on the result. This one does. Each pass writes the likeliest version back into the pool it learned from, and the pool gets flatter. Researchers call the far end of this model collapseShumailov, Ilia et al. “AI models collapse when trained on recursively generated data.” Nature vol. 631,8022 (2024): 755-759..

Collapse in the lab is conditional and probably fixable. It sets in when a system trains mostly on synthetic data with no fresh human input, and the people building these things know it: they curate, mix in real text, strain out the slop. A training set can be groomed. So grant them the win, and suppose the output stays as varied as the writing it learned from.

It still does not help, for two reasons that have nothing to do with the weights.

The first is that the slip requires the attempt. “Pease” became “pea” because somebody had to produce the word with their own mouth and got it wrong. Drift is a byproduct of reaching. A sentence that arrives finished was never reached for, and however strange it is, the strangeness is not yours.

The second is that what kills a monoculture is not sameness. It is shared origin. A field can hold a hundred varieties and still die as one field if every seed came from the same supplier, because whatever that supplier never bred for, nothing in the field has. One blind spot, running through everything.

So a million people each getting a word slightly wrong is not the same kind of event as a million people each pulling something unusual out of the same machine. The second can be endlessly various and still have one edge, one set of things that never come up. And what falls outside it does not become rare. It stops occurring to anyone.

There is more than one machine, of course, but the plurality is shallower than the branding suggests. The major models scrape the same web, get tuned against the same kind of human preference, and increasingly feed on each other. Several vendors, one breeding program.

One corpus, though, is never curated, and it is the only one that finally matters: the inside of your own head. Nobody filters the pre-averaged sentences back out of it. The collapse that counts was never going to happen in the weights, where someone is paid to prevent it. It happens in us, where no one is watching the training data.

The words we share are not just the container thought rides in. They are part of what thought is made of. Thin them and you shrink what is easy to think. Orwell imagined this done from above: a state trimming the dictionary so rebellion would have no words to form in. The real version is gentler and worse. There is no state, no plan, nobody cutting anything. We are doing the thinning ourselves, one convenient sentence at a time, because the handier word is right there and the truer one costs a pause. Orwell feared a boot. We got a shrug.

And it will not arrive as catastrophe. Catastrophes are loud, and nobody here is suffering. It arrives as plenty: every song a little like one you know, every story cut closer to a contour you have already approved. The flat future will not feel like a cage. It will feel like being known.

The body does not wait for a hero antibody to arrive intact. It keeps a vast, mostly useless reserve of strange variants on hand, generating new ones it will probably never need. And it does not order them from a supplier. Every node makes its own, independently and all at once, which is the property that matters and the one a single machine cannot have however well it is tuned. A culture works the same way or not at all: many people, generating locally, nearly all of it wasted.

Which makes the move a matter of upkeep rather than heroism, and therefore available to anyone. Reach for the word yourself, and keep the one that comes out slightly wrong for the slightly truer reason. Leave in the sentence the machine would have smoothed away. Not because you are the last original soul in a flattening world. Nobody is, and nobody has to be. Because variation was never made by the few.

The machine will hand you the smooth, likely sentence every time, instantly and for free, and that is the trouble and also the tell. Anything that survives an unpredictable world keeps something around to be wrong with. A culture is a reserve like any other, held in common, restocked only when people reach and miss. Stop restocking it and the loss will not show. We become fluent, agreeable, never at a loss for words.

And stagnant. And defenseless against anything we have not already seen. And with no words for it when it arrives.