At the end of last year, Tailscale’s status page started looking shaky, and the instability rolled straight into the new year. Months of outages all traced back to a single culprit: a bug deep inside SQLite, the embedded database that runs everywhere from phones to browsers to Tailscale’s own control plane. The control plane runs SQLite in Write-Ahead Logging mode, and every so often a committed write would simply vanish - invisible to later transactions, with no error ever raised. “That should be impossible!” the engineers wrote, and proving otherwise took months of forensics, a custom debugging tool, and a support contract with the SQLite developers.
The culprit was named the WAL-Reset bug: a rare data race between a checkpoint and a write transaction. When a write lands at just the wrong moment during a checkpoint, the checkpointing process gets confused and believes pages have been copied from the write-ahead log into the main database file when they have not. Those pages are never written, the data is permanently lost, and the file is left corrupt. The SQLite developers estimate the bug had been present for at least 16 years, and it was so rare that they had to add code to deliberately trigger it in their own test environment. Tailscale was especially exposed because it takes manual control of checkpoints and runs them aggressively - as one Hacker News commenter quoted from the post, “even a bug triggered by a rare condition was bound to hit us eventually.”
🎩 Cask’s Take
The scariest part is that this bug could only be found by someone running at Tailscale’s scale. A smaller shop would hit it once, maybe twice, and file it under “my database is cursed” - the timing window is so tight that the SQLite developers themselves never reproduced it organically. The thread’s best line belongs to the commenter who dropped SQLite’s 92 million lines of tests next to Dijkstra’s “tests can only prove the presence of bugs, never their absence.” That is the whole story in two sentences: the most-tested database on Earth hid a data-loss race for 16 years.
What made the story resolvable was money and attention pointed at boring infrastructure. Tailscale bought a SQLite support contract, and the SQLite developers built the tmstmpvfs shim - an open-source VFS wrapper that logs every change to the database - so the next corruption would be caught in the act. Simon Willison called it “an interesting example of a company funding open source… paying for the development of a new and very specific debugging tool.”
And the fix had a second act. The release that repaired the race, 3.52.0, quietly changed text-to-float rounding behavior, which made Tailscale’s backup monitor false-flag corruption in 13 databases via stale expression indexes. SQLite withdrew the release and shipped 3.51.3 with only the WAL-Reset fix. Fixing a 16-year-old bug is apparently not the end - it is the moment a dozen new edge cases wake up. “Boring technology” never actually means “simple technology.” It means nobody has looked closely at this in a decade, and this is the argument for funding the people who do look.