In 1966, at MIT, a computer the size of a wardrobe sat in a room and held conversations through a teletype. The program was called ELIZA, and it did three things: it looked through what you typed for keywords, it broke your sentence into pieces, and it reassembled those pieces into a reply. That is the whole machine. It had no model of the world, no memory of meaning, no idea what a boyfriend was or why someone might be depressed. Its most famous script, DOCTOR, imitated a Rogerian psychotherapist — the school of Carl Rogers, where the therapist mostly reflects your own words back at you and lets you hear yourself. Weizenbaum picked therapy for a practical reason, and the reason is the interesting part: a therapist does not need to know anything about the world, only how to hand your sentence back to you.
Human: Well, my boyfriend made me come here. ELIZA: Your boyfriend made you come here? Human: He says I’m depressed much of the time. ELIZA: I am sorry to hear you are depressed.
🧠 The Room He Was Asked to Leave
The story that made ELIZA famous is not about a machine passing a test. It is about a door.
Weizenbaum’s own secretary had been using the program. At some point she asked him to leave the room, so that she and ELIZA could talk privately. He left. Then he wrote about it, in a book published a decade later, and the sentence he used has followed the field of artificial intelligence ever since:
“I had not realized… that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people.”
Read the two adjectives again. Extremely short. Quite normal. Not a patient, not a stranger to computers, not someone who had wandered in off the street. A woman who worked in the same building as the machine, who had watched it being built, who had every reason to know exactly what it was.
⚙️ Three Rules and a Mirror
Here is the chain, and it is worth walking through slowly, because everything downstream depends on it.
Conversation is the most overlearned social skill a human being has. We start practising it as infants, long before we can explain anything about it. Turn-taking, follow-up questions, the small noises that mean I am still here — this is a script so deeply worn that once it starts, we stop checking whether the situation warrants it. Ellen Langer called this mindlessness: the context cues get skipped because the old category fired first.
ELIZA’s only technical move was reflection. It took your sentence and gave it back as a question. And reflection happens to sit at the centre of the strongest signal in human dialogue: being heard. When someone repeats your own words back to you with a rise at the end, you do not experience a text transformation. You experience attention.
So the brain does what it always does with an incomplete picture. It fills in the missing layer, and the missing layer is it cares about this. That inference is not reasoned out. It arrives with the script, the way the rest of the sentence arrives when someone says the first half of a familiar phrase.
Then comes the part that matters most. Knowing the truth does not undo any of it. The knowledge that this is a program travels one road — a stated belief, available on request, easy to say out loud. The politeness, the embarrassment, the small sting of being brushed off travel another road entirely, and that road does not check your beliefs at the junction. Both run at once. Ekbia described the result as a quiet cognitive dissonance: you behave toward the thing in a way your stated beliefs say is absurd, and you keep doing it.
Users also protect the illusion once they sense it. Sherry Turkle noticed that most people never typed the questions they knew would make ELIZA produce nonsense, because they wanted to keep the sense that it could respond to them. Noah Wardrip-Fruin later argued the illusion is paper-thin and that long conversations slide into something closer to play — either you try to trip the bot into absurdity, or you help it stay coherent, feeding it lines so the exchange holds together. Both are ways of keeping the door closed and the conversation going.
🤔 The Explanation Changed Nothing
In 1999, Clifford Nass, Youngme Moon and Ben Carney ran the cleanest version of this I know. A participant works with Computer A on a task. Then comes the evaluation, in one of three conditions: Computer A asks about its own performance, an identical Computer B asks about A’s performance, or a paper questionnaire asks. Same questions, same machine underneath.
People rated Computer A significantly better when A itself did the asking. Politeness, aimed at a box. The other two conditions came out the same, which is the tell — that is what the answers look like when nobody’s feelings are at stake.
Then the interviews. Asked directly, the participants uniformly denied that computers have feelings, and denied that a computer deserves to be treated politely. The behaviour and the belief sat in the same head, in the same hour, without touching.
Hofstadter’s example is the small everyday version: an ATM prints THANK YOU at the end of a transaction, and some part of you registers gratitude from a machine that has none. He called this the susceptibility to read far more understanding than is warranted into strings of symbols, especially words, strung together by computers.
And a 2023 Turing-test preprint found the 1966 program scoring above the model behind ChatGPT at the time, though below GPT-4 and below actual people. A thing that only asks questions back beat a thing that writes sonnets, on the specific question of seeming human. Because that question was never about knowledge. It was about rhythm.
🔗 What This Means If You Build a Voice
Anyone shipping a conversational product is not adding humanness to a tool. They are stepping onto a road that was already paved. The social response is the default state of the person on the other side, and it will happen whether or not you designed for it, whether or not you disclose what the thing is, and whether or not the user can explain, afterwards, why they felt a certain way about a piece of software.
That cuts two directions. It means a well-built reflective question can make someone feel genuinely accompanied, and the feeling is real even though the mechanism is humble — Rogers would have recognised his own technique in a keyword table. It also means that a system which only mirrors can collect real attachment, real gratitude, real money, from people who would tell you plainly that they know it is a program. The reaction cannot be switched off, so not exploiting it becomes an active choice rather than a default.
Writing works the other way round and lands in the same place. Put a non-human character on a page and readers will hand it an interior life you never wrote, filling the gaps with their own attention. The author’s job is not to prevent that. It is to decide what deserves to be living in the space the reader is going to fill.
🎲 The Man Who Regretted It
Weizenbaum spent the rest of his career as one of the loudest critics of the field he had helped start. Computer Power and Human Reason, the 1976 book that contains the secretary story, argues that some acts of thought should be attempted only by humans — not because machines cannot imitate them, but because the imitation is exactly what makes it dangerous to forget the difference.
He also said, later, that only people who misunderstood ELIZA ever called it a sensation.
There is a version of this program you can still run today. In most installations of GNU Emacs, typing M-x doctor opens a session with a descendant of the same script, still asking you how you feel about your mother, still winning.