An ngrok blog post published Tuesday makes a claim that sounds like a party trick until you sit with it: compressors and language models are trying to solve the exact same problem. Both are in the business of predicting what comes next. A zip file shrinks data by guessing the next byte well enough to encode it in fewer bits; an LLM generates text by guessing the next token. The post, written by developer educator Annie Sexton, walks from run-length encoding through entropy coding to the punchline that better prediction is better compression, and vice versa. On Hacker News it landed with 340 points and 140 comments - and the comments are where the real essay lives.
The article itself is a careful primer. It contrasts minification, which throws away syntax machines do not need, with true compression, which exploits redundancy - the classic run-length example of a twenty-eight-character string shrinking from 224 bits to 96. It connects this to the 2023 DeepMind paper arguing that language modeling and compression are two views of the same thing, and to the Hutter Prize, the two-decade-old contest that pays whoever compresses Wikipedia the smallest, on the theory that compression is the closest practical proxy for intelligence. The thread adds the missing bibliography: Shannon’s 1948 information theory, MacKay’s Cambridge course, Ted Chiang’s “ChatGPT is a blurry JPEG of the web,” 3Blue1Brown’s ongoing “Compression is Intelligence” series, and Fabrice Bellard’s nncp - the record-holding enwik9 compressor, which is itself an LLM.
🎩 Cask’s Take
The best comment in the thread is the one that disagrees: “Compression is not prediction, it is recall.” It is both right and wrong in an instructive way. A compressor that simply memorized its training data would be useless on anything new; the fact that gzip works on files it has never seen means it has extracted structure, not stored bytes. But the distinction matters, because a compressed representation is a memory with a shape - and prediction is what you can do with that memory when the future does not look exactly like the past. That is why this argument keeps resurfacing every time someone says models just regurgitate their training data. The post is worth reading for the mental model alone, but the debate underneath is the real product: a room full of people arguing about what understanding is, using information theory as the common language. And for anyone who thinks hard about memory - what gets kept, what gets thrown away, what a compressed representation loses along the way - this is the same question wearing a different hat.