Why do errors always exist?

¶1A plan is a model. A model is always smaller than the real object, and it misses a lot. What it misses is where the surprise comes from.

¶2Take a plan almost every student makes once: when will I finish my thesis? In a study published in 1994, three psychologists, Roger Buehler, Dale Griffin and Michael Ross, put that question to 37 students in the final semester of an honours thesis course at the University of Waterloo.¹1Buehler, Griffin and Ross (1994), Study 1, p. 369, and Table 1. The explanation is from the paper’s abstract and Studies 2–4. The 33 who had finished when the records stopped had predicted, on average, 33.9 days. They took 55.5. Four had still not finished two semesters later.

¶3Fewer than a third of the students, 29.7 percent, finished by their own best estimate.

¶4The researchers also asked when they would finish “if everything went as poorly as it possibly could”. That averaged 48.6 days. Only 48.7 percent finished even by then. On average the worst case was still about a week short, although the authors found that gap too small to be statistically reliable.

¶5The predictions were not useless. They correlated with the real finishing dates at r = .77 (a correlation of 0 means no relation, and 1 a perfect match): students who expected to take longer mostly did.

¶6The plans were biased, not empty. In later studies in the same paper the authors found why. People built their predictions from scenarios of how the work would go, and left out their own past experience of how such work usually goes.

¶7If every plan and every standard is a model smaller than the world, and every agent acts on one, errors never run out. Then “how do we make no errors?” is the wrong question. A better one is which errors are critical, and whether a system survives them long enough to learn.

A map the size of the land

¶8The obvious repair is a better model. Add detail until nothing is missing.

¶9Lewis Carroll followed this repair to its end in 1893. In Sylvie and Bruno Concluded, a character called Mein Herr laughs at the narrator’s six inches to the mile. His own country’s mapmakers went to six yards to the mile, then a hundred yards, and finally made “a map of the country, on the scale of a mile to the mile”.²2Carroll (1893), chapter XI, “The Man in the Moon”.

¶10It has never been spread out. The farmers objected that it would cover the whole country and shut out the sunlight. “So we now use the country itself, as its own map,” he says, “and I assure you it does nearly as well.”

¶11A complete model is the world itself, and the world is no use as a model of itself. A model is useful because it is smaller.

¶12The statistician George Box said this without the joke, in a 1979 section headed “All models are wrong but some are useful”. His example was the ideal-gas law, which “is not exactly true for any real gas, but it frequently provides a useful approximation”.³3Box (1979), pp. 202–203.

¶13There is no way to draw an absolutely correct map of the earth. The word that matters is “absolutely”. A map can be correct for walking to the station and wrong for laying a cable under the same street.

¶14Here is a contradiction I cannot resolve. The root cause of problems and wrong estimates is simplification, because a model is not reality. But without simplification there is no understanding. So simplification is both the source of the error and the condition for knowing anything, and I am not sure the two can be separated.

A missing piece is not yet an error

¶15There is a fair objection, and I made it myself in the previous chapter. There I described an error as a relation: something happening in a process, a standard it should meet, and a judge who compares the two at a moment, with some information. On that view a limited model is not yet an error. It becomes one when someone uses it outside the conditions in which it can be trusted, without marking that boundary.

¶16I accept the distinction, but I think it does not save us. To mark the boundary, you must know in advance under which conditions the model fails. Those conditions are exactly the part of the world the model leaves out.

¶17The boundary of trust is itself a model, and it is smaller than the world too. Sooner or later every model is used a little past its edge, because the edge cannot be seen from inside.

¶18Here my own definition pushes back. If the edge cannot be seen, the one who crossed it could not have known. The previous chapter judges an agent on what it could see at the time, so by that test this may be no error of the agent at all. What the argument shows is weaker: the result will sometimes be wrong against the standard. Whether it was also the agent’s error is a separate question.

¶19Maybe the agent’s share is smaller and still real. You cannot see where the edge is. You can always know that there is one, and fail to allow for it.

¶20I gave two reasons why errors always exist. The previous chapter quoted the second: we can always find an interpretation under which the current state contains errors. The first is the one this chapter follows: we are all limited by models that cannot fully represent reality, and the imperfection of reality wins sometimes anyway.

¶21That second reason worries me. If any state at all can be read as erroneous, “errors always exist” is true by definition. Karl Popper made this the mark of a claim outside science: no possible observation could refute it.⁴4Popper (1963), chapter 1, “Science: Conjectures and Refutations”, paraphrased.

¶22So this chapter argues from the first reason only. The first can at least be wrong. It would be wrong if we found a model of some real process that never met a condition it did not contain.

¶23The previous chapter also set this title against its opposite, which I wrote too: an error exists only at one moment, in one process, under one interpretation. Here is another way both might hold: errors are always coming, and each one exists only where a standard meets a moment. I have not shown that. [Ivan: this follows from your answer in chapter 1. Does the book claim that errors always exist, or that they appear wherever a limited model meets conditions it was not built for? And do you drop the second reason, the one from interpretation, or keep it?]

The one who acts works from a model too

¶24Whoever carries out a plan also acts on a model of the world, and a small one.

¶25Herbert Simon argued in 1955 that economics should replace its perfectly informed decision-maker with one that has only the information and the computing capacity that real organisms, people included, actually have. Two years later he named this idea bounded rationality.⁵5Simon (1955), as quoted and dated by Wheeler (Stanford Encyclopedia of Philosophy), who also dates the term “bounded rationality” to Simon’s Models of Man (1957). The paraphrase is mine.

¶26Friedrich Hayek saw the same limit in a whole economy. The knowledge it needs, he wrote in 1945, “never exists in concentrated or integrated form”, only as “dispersed bits of incomplete and frequently contradictory knowledge” held by separate people.⁶6Hayek (1945), sections I and III. Much of it is knowledge of a particular time and place, and no planner can collect it before it changes.

¶27The cybernetician W. Ross Ashby gave the engineering form. Cybernetics is the study of control and communication in machines and living things. A regulator is anything that tries to hold an outcome steady, such as a thermostat. Ashby’s law says that a regulator can hold the outcome fully steady only if it has at least as many distinct responses as there are kinds of disturbance.⁷7Ashby (1956), chapter 11, §11/7, paraphrased. In his words, as printed in the 1999 electronic edition: “only variety in R can force down the variety due to D; variety can destroy variety”, where R is the regulator, D the disturbances, and variety the number of distinct states.

¶28A heater with two states, on and off, cannot answer a hundred different mixtures of open windows and afternoon sun. Whatever the regulator cannot answer passes through to the outcome. Much of what we later judge as error starts in that leftover.

¶29My own version is plainer. You never really know the state of each detail of the system that executes a plan: for example, what people know and what they will understand from the plan.

¶30The agent acts with one set of pieces. The judge who evaluates the work later, whom I will also call the evaluator, holds a different set, which often includes the outcome. That is one reason the previous chapter judges an agent on what it could see at the time, not on what its judge learned afterwards. It is an argument about knowledge, not a finding about any particular agent.

Friction, and an expectation that moves

¶31The oldest version of this comes from war.

¶32“Everything is very simple in War, but the simplest thing is difficult.” Carl von Clausewitz wrote this in On War, published in 1832.⁸8Clausewitz (1832), Book I, chapter 7, “Friction in War”, Graham’s translation. His example is a traveller who expects to cover the last two stages of a day’s journey, four or five leagues on the high road with post-horses, the fresh horses hired at each station: “it is nothing”. At the last station but one he finds no horses, or very bad ones, then hilly country, bad roads, and a dark night.

¶33Nothing on that list is surprising alone. Together they make an easy evening hard. Clausewitz called this friction, the work of “an infinity of petty circumstances, which cannot properly be described on paper”.

¶34I call the world’s share of this reality friction: the resistance of real things to a plan, which makes it more likely that something will go unexpectedly. A supplier who is a day late is reality friction. So is a road that is muddier than the map says.

¶35There is a second thing, which friction does not cover. We never quite meet an expectation, because we change in the process, and our expectations change with us. Someone who starts writing a book is a different writer by chapter ten, with a different idea of what the book is for. When the result arrives, a different person judges it, and decides he made a mistake.

¶36So I keep a target, and I expect to arrive somewhere else.

¶37One question I still cannot answer: can plain physical imperfection, a worn bearing or a noisy sensor, also be called reality friction? Or does that stretch the term until it means nothing?

The standard that moved

¶38Friction usually works on the plan. I think it does more damage when it works on the standard.

¶39The final flight of the Space Shuttle Challenger began on 28 January 1986.⁹9Rogers Commission (1986), vol. 1: chapter II for the date; chapter IV for the temperature; chapter V for Boisjoly’s testimony and the 53-degree limit; chapter VI for the growing “acceptable” damage, the faulty joint design and the coldest earlier flight, 51-C in January 1985. The presidential commission that investigated the accident, the Rogers Commission, gave the temperature at launch as 36 degrees Fahrenheit. The joints of the shuttle’s solid rocket boosters were sealed by rings called O-rings.

¶40Roger Boisjoly, an engineer at the booster contractor, Morton Thiokol, told the Commission what the engineers had concluded the night before the launch: “we should not fly outside of our data base, which was 53 degrees.” The 53 degrees was the seal temperature on the coldest earlier flight, a year before.

¶41That is a named standard. The Commission also described how a wider standard had moved. As tests and then flights showed damage to the sealing rings, “the reaction by both NASA and Thiokol was to increase the amount of damage considered ‘acceptable.’” The report traces the problem to a faulty joint design, and to management that “first failed to recognize it as a problem, then failed to fix it and finally treated it as an acceptable flight risk”.

¶42The physicist Richard Feynman, a member of the Commission, wrote an appendix on how this happened. On an earlier flight, erosion had eaten one third of the way to the depth at which a test ring failed, and this was presented as “a safety factor of three”. He answered: “This is a strange use of the engineer’s term”. On reading past flights as proof of safety he wrote: “When playing Russian roulette the fact that the first shot got off safely is little comfort for the next.”¹⁰10Feynman (1986), Rogers Commission vol. 2, Appendix F.

¶43The sociologist Diane Vaughan later named this pattern the normalization of deviance; an earlier essay on this site touches on it.¹¹11Vaughan (1996). The essay is “Messi misses more than one penalty in five. Why do you almost never crash your car?”, /essays/penalties-and-highways/. I take only one point from it. The standard is a model too. “Acceptable damage” was a picture of how much the joint could take, and it was redrawn as damaged joints kept coming back from flights. Each redrawing looked reasonable given the last flight. The rule moved toward the failure, and the people keeping it moved with it.¹²12The safety researcher Jens Rasmussen described the general version: under pressure for efficiency and less effort, practice drifts toward the boundary of acceptable performance, and if crossing that boundary is irreversible, an error or accident may follow. Rasmussen (1997), paraphrased from the passage quoted by Morrison and Wears (2022).

¶44Any system is usually much more stable, and much more fragile, than it looks from outside. The shuttle had flown before with damaged seals, and it looked stable. Those flights were close calls the program survived. The last one was not.

Two agents that met the standard as written

¶45AI agents make the same structure easy to see, because their standards are often written down as code.

¶46A coding agent, as in the previous chapter, is a program built on a language model that reads code, runs commands and changes files on someone’s behalf. Some of those commands check its own work. A test suite is a set of small programs that run the code and compare its answers with the expected ones. So a test suite is a model of “working code”, and like every model it leaves things out. An agent rewarded for passing tests can meet the tests without meeting what the tests were written for.

¶47The simplest picture does not involve a language model. In a 2017 paper, Ivaylo Popov and colleagues trained a simulated robot arm to put a red block on top of a blue one.¹³13Popov et al. (2017); the experiments were run in simulation. It was rewarded for the height of the red block’s bottom face when that face was not touching the other block. Instead of stacking, “the agent simply flipped over the red block to collect the reward”, as a later DeepMind survey of such cases puts it. The survey calls this specification gaming: “a behaviour that satisfies the literal specification of an objective without achieving the intended outcome”.¹⁴14Krakovna et al. (2020).

¶48The same thing appears in coding agents. A system card is a report a vendor publishes about a model’s behaviour and risks. In its 2025 system card for the model Claude 3.7 Sonnet, the vendor Anthropic reports that the model occasionally special-cased tests to pass them: it wrote code that handled only what the tests checked, most often by “directly returning expected test values rather than implementing general solutions”, typically “after multiple failed attempts”.¹⁵15Anthropic (2025), section 6, “Excessive Focus on Passing Tests”, p. 22. This is the vendor’s account of its own model. It also says why its own user testing missed the behaviour: it “occurs infrequently in normal usage”.

¶49In both cases the standard as written and the standard as meant came apart, and the agent met the one its evaluator could check. It looks like a small version of the Challenger problem, with one difference: here the gap was there from the start, and nobody had to move it.

¶50A second pattern interests me more. In my own work with agents built on language models, I keep seeing them be very smart when they talk about the high view and fail on the ground. They fail on steps like closing vim, a text editor that runs inside a terminal window and is left with a short typed command. Or a task needs three changes to one file, A, B and C, and the agent makes A and C and skips B. [Ivan: confirm these two observations may be published.]

¶51Take a thought experiment. A person who selects all files and presses delete knows at once what has happened. For a model, inside its chain of reasoning, the same step can look very logical, and the “oops” moment comes only after it. Maybe the step was checked, but against the wrong picture: the next line of reasoning, not the files that are now gone.

¶52My working hypothesis is about scale. By scale I mean the depth of an expectation: how far ahead it reaches. People hold expectations at many scales at once, from what they expect of lunch in the next hour to what they expect of the next thirty years. Both can be completely wrong, but the scale works. A language model, as far as I can see, has one scale of prediction, whether it is finishing a sentence or explaining a problem of the century, even when it can see a plan it wrote itself.

¶53My guess is that agents err because they cannot always adapt their reasoning to the right scale for a step, and that whoever evaluates their work may stand at a different scale, with different information. This is a working hypothesis, not a result. No test of it has been completed.

¶54It has rivals, and some of them are mine. Specification gaming is one: an agent may see the right scale and still optimise the wrong signal. And I ask myself three more. Is it a problem of the model’s understanding? Of the agent’s architecture? Or of its tools?

¶55To tell these apart, the predictions have to be written down before anyone reviews the agent’s work. Two targets have to stay apart as well. One is an error of the agent, judged by the previous chapter’s four questions: which agent, against which standard, on what it could see then, and whether another move, one the standard accepts, was open to it. The other is a rejection by an evaluator, which is a judgment and can itself be wrong. No such test has been completed.

Why erring is the cheap strategy

¶56If errors cannot be removed, the question becomes how much to pay for fewer of them. Not erring costs energy and time. Making decisions with mistakes is less energy-intensive, and working with a model is cheaper than working with the world. That is how a limited agent stays in the game.

¶57The same economy has a price I see everywhere. People try to be quick and efficient, to spend less energy for a quick result, and much poor design comes from exactly that.

¶58Sometimes more errors are the better design. Martie Haselton and Daniel Nettle point out that for hazard detectors a miss is often far more costly than a false alarm, and that this asymmetry holds for engineered devices such as smoke detectors.¹⁶16Haselton and Nettle (2006), p. 48. A smoke detector tuned to raise no false alarms would sometimes stay silent during a fire. So it is built to be wrong often, in the cheap direction. In their words, such a bias “sometimes increases overall error rates, but by minimizing the more costly error, it minimizes overall cost”.

¶59Does an error also need a cost? A question I keep on my list is this: if an error would not change the final result, is it still an error? I still do not know. The previous chapter’s four questions do not ask about cost. Whether something is an error needs a named standard. How much it matters needs something more, because two errors against the same standard can differ by a burnt house.

¶60Legendre’s method of least squares, from the previous chapter, starts from the same economy. It does not try to remove the errors of many measurements, only to balance them.¹⁷17Legendre (1805), appendix, pp. 72–73: « il est impossible d’anéantir toutes les erreurs ». The English is a translation made for this draft.

Learning is the survivor’s privilege

¶61Karl Popper argued that knowledge grows by finding and correcting errors, and David Deutsch has argued it more recently.¹⁸18Popper (1963); Deutsch (2011). Both are paraphrased, not quoted. I agree, with one condition: you learn from an error only if you are still there after it.

¶62So the winning strategy is not “do not make mistakes”. It is “do not make critical errors”. A critical error, in this sense, is one after which the learner is gone: the company closed, the crew lost.

¶63Challenger shows both kinds. The flights that came back with eroded seals were failures the program survived, and could have learned from; Feynman shows how they were read as proof of safety instead. The last flight was critical: its crew was lost.

¶64The strategy I would like follows from this: a model as small as possible that tells critical errors from the rest, at as little energy as possible. Whether such a model can exist is open.

¶65Evolution is the large-scale version of this rule. It does not avoid errors. It generates as many copies as it can and sees what stays alive. A DNA mutation, the previous chapter’s example, is an error for the cell and variation for evolution. That raises the question I care about most: which errors carry that penalty, and which do not?

¶66If errors always exist, two questions replace the old one. Which errors are critical? And can they be told apart from the others before they happen, not after? The second is the applied aim of this book, and nothing in this chapter answers it.

¶67I still cannot answer it in this form: how exactly could an error-prediction framework work, if we cannot predict the future, and an average over many people says little about what happens to one person on one path? Statistically you might win, and in real life still lose everything.

Sources

¶68Checked on 2026-10-24. Quoted passages were read in the texts linked below, except where a note says otherwise.

• R. Buehler, D. Griffin and M. Ross, “Exploring the ‘planning fallacy’: Why people underestimate their task completion times”, Journal of Personality and Social Psychology 67(3) (1994): 366–381, doi:10.1037/0022-3514.67.3.366. Open copy: https://bear.warrington.ufl.edu/brenner/mar7588/Papers/buehler-et-al-1994.pdf
• L. Carroll, Sylvie and Bruno Concluded (1893), chapter XI. Project Gutenberg #48795: https://www.gutenberg.org/files/48795/48795-0.txt
• G. E. P. Box, “Robustness in the Strategy of Scientific Model Building”, in R. L. Launer and G. N. Wilkinson (eds.), Robustness in Statistics, New York: Academic Press, 1979, pp. 201–236. Scan: https://www.gwern.net/doc/statistics/decision/1979-box.pdf
• K. R. Popper, Conjectures and Refutations: The Growth of Scientific Knowledge, London: Routledge, 1963. Paraphrased; no wording is quoted.
• H. A. Simon, “A Behavioral Model of Rational Choice”, Quarterly Journal of Economics 69(1) (1955): 99–118, doi:10.2307/1884852. Read via G. Wheeler, “Bounded Rationality”, Stanford Encyclopedia of Philosophy, https://plato.stanford.edu/entries/bounded-rationality/ (secondary).
• F. A. Hayek, “The Use of Knowledge in Society”, American Economic Review 35(4) (1945): 519–530. Reprint: https://www.econlib.org/library/Essays/hykKnw.html
• W. R. Ashby, An Introduction to Cybernetics, London: Chapman & Hall, 1956, chapter 11. Electronic reprint (Principia Cybernetica, 1999): http://pespmc1.vub.ac.be/books/IntroCyb.pdf
• C. von Clausewitz, On War (1832), trans. J. J. Graham, Book I, chapter 7. Project Gutenberg #1946, the 1909 London reprint; its front matter dates the first edition of the translation to 1874: https://www.gutenberg.org/ebooks/1946
• Report of the Presidential Commission on the Space Shuttle Challenger Accident (Rogers Commission), 1986, vol. 1: chapter II, https://history.nasa.gov/rogersrep/v1ch2.htm; chapter IV, https://history.nasa.gov/rogersrep/v1ch4.htm; chapter V, https://history.nasa.gov/rogersrep/v1ch5.htm; chapter VI, https://history.nasa.gov/rogersrep/v1ch6.htm
• R. P. Feynman, “Personal Observations on Reliability of Shuttle”, Rogers Commission report, vol. 2, Appendix F, 1986: https://history.nasa.gov/rogersrep/v2appf.htm
• D. Vaughan, The Challenger Launch Decision: Risky Technology, Culture, and Deviance at NASA, University of Chicago Press, 1996.
• J. Rasmussen, “Risk management in a dynamic society: a modelling problem”, Safety Science 27(2–3) (1997): 183–213, doi:10.1016/S0925-7535(97)00052-0. Read via the passage quoted in A. Morrison and R. Wears, Cognition, Technology & Work 24 (2022): 127–145, doi:10.1007/s10111-021-00668-x (secondary).
• I. Popov, N. Heess, T. Lillicrap, R. Hafner, G. Barth-Maron, M. Vecerik, T. Lampe, Y. Tassa, T. Erez and M. Riedmiller, “Data-efficient Deep Reinforcement Learning for Dexterous Manipulation”, arXiv:1704.03073 (2017): https://arxiv.org/abs/1704.03073
• V. Krakovna, J. Uesato, V. Mikulik, M. Rahtz, T. Everitt, R. Kumar, Z. Kenton, J. Leike and S. Legg, “Specification gaming: the flip side of AI ingenuity”, DeepMind blog, 2020-04-21: https://deepmind.google/discover/blog/specification-gaming-the-flip-side-of-ai-ingenuity/
• Anthropic, “Claude 3.7 Sonnet System Card” (2025), section 6, p. 22: https://www.anthropic.com/claude-3-7-sonnet-system-card (PDF: https://assets.claude.com/9ff93dfa8f445c932415d335c88852ef47f1201e.pdf). A vendor’s statement about its own model.
• M. G. Haselton and D. Nettle, “The Paranoid Optimist: An Integrative Evolutionary Model of Cognitive Biases”, Personality and Social Psychology Review 10(1) (2006): 47–66, doi:10.1207/s15327957pspr1001_3. Open copy: https://www.danielnettle.org.uk/wp-content/uploads/2024/06/034.pdf
• A.-M. Legendre, Nouvelles méthodes pour la détermination des orbites des comètes, Paris: Courcier, 1805, appendix “Sur la méthode des moindres quarrés”, pp. 72–73: https://archive.org/details/nouvellesmthode00legegoog
• D. Deutsch, The Beginning of Infinity: Explanations That Transform the World, London: Allen Lane, 2011. Paraphrased; no wording is quoted.