A record that holds a calculation and a reading of that calculation holds two claims, and they can disagree. This paper studies one set of decision records in which they do. Each record states, in its author's words, how many minutes a change to a procedure would save once the rework it brings is taken off. Each also keeps a small calculation that a programme executed on the author's figures, with the input, the output, a hash of the input and a replay that ran it again and got the same answer. In about half of the records the two agree. In the rest the author's words state a saving and the executed calculation reports that the change would add work. The question is what such a record takes its calculation to have checked, and what it does with the disagreement.
1The question and the records
The records come from AEGIS, the safety and protection studio of the RISCON family, which publishes several thousand authored decision cases. Every one of them is a synthetic setting: AEGIS marks each case realistic synthetic and calls each record an authored question seed whose judgment mark is the author's record, not a factual basis from public records. The thirteen cases read here are every authored AEGIS case whose run executes the rule that counts work saved net of rework and whose situation or question speaks of rework. They concern a standard form for computer entry, a simpler maintenance intake form, an annual stock survey, a weekly report, electronic approval of documents, the receipt of supply requests, copies made for disclosure, de-identification of documents, a stock tally with readers, the layout of a maintenance shop, warehouse automation, the drying of sludge and the planning of dispatch. The offices, forms and minutes in them are the author's assumptions. Nothing in this paper is a finding about any real office, and nothing in it says what any office should do.
The research question is asked of the record set, not of the world. When a decision record carries both the arithmetic its author wrote about the work a change would save and a calculation that a programme executed and replayed on the same figures, what does the record take the calculation to have checked, and what does it do where the two disagree?
2The contribution
The paper makes three claims about these thirteen records. First, one figure in them is read in two ways. The executed rule adds the minutes of rework to every case; seven of the records, by the arithmetic their own words give, subtract the rework once from the whole. The reading decides the sign: where the author subtracts once, the run reports an increase in work, and where the author adds the rework to each case, the run and the author agree. Second, nothing in the seven records where the readings part sets the two results side by side. The run's own interpretation restates the author's saving, the replay reports a match, and in five of the seven the recommended option rests on the author's figure by name. Third, what the records ask for next is measurement of work and rework in the field, not a reading of what the figure stands for. Their conditions for reopening four of the cases repeat the rework figure as a threshold, in the same words that admit both readings.
The contribution to the literature is a reading of where one designed record set puts its check. The debates gathered below argue about what checking a calculation can establish and why forecasts of time and cost go wrong. This record set gives a small, fully visible instance in which both questions meet in one figure, and in which the check that was built in is the one the debates find weakest.
3The debate
The first strand concerns what it means to verify a program. Richard A. De Millo, Richard J. Lipton and Alan J. Perlis, Social Processes and Proofs of Theorems and Programs (1979), argue that formal verifications of programs will not play the role in computing that proofs play in mathematics, and that the complexity of specifying real programs makes formal verification difficult to justify and manage. James H. Fetzer, Program Verification: The Very Idea (1988), holds that the notion trades on an equivocation: algorithms, as logical structures, can be verified deductively, while programs, as causal models of those structures, cannot. Naomi Oreskes, Kristin Shrader-Frechette and Kenneth Belitz, Verification, Validation, and Confirmation of Numerical Models in the Earth Sciences (1994), carry the argument to models of natural systems, which they hold cannot be verified or validated, only confirmed in part, so that their primary value is heuristic. R. G. Sargent, Verification and Validation of Simulation Models (2013), takes the practical side and separates four things a modeller may check: the validity of the conceptual model, the verification of the model as built, its operational validity and the validity of its data. This study draws on the last of these distinctions most: a calculation can be faithfully built and faithfully run and still take its data to mean something other than its author does.
The second strand concerns reproducing a computation. Roger D. Peng, Reproducible Research in Computational Science (2011), proposes reproducibility as a minimum standard for judging claims when full independent replication is not possible. Thomas Herndon, Michael Ash and Robert Pollin, Does High Public Debt Consistently Stifle Economic Growth? A Critique of Reinhart and Rogoff (2013), show what a replication by other hands can find: in their reading of two published studies, coding errors, excluded data and inappropriate weighting had led to serious miscalculations. This study takes from their article only that example of what an outside replication found, and draws nothing from it about public debt. Raymond R. Panko, "What We Know About Spreadsheet Errors" (1998), reports that errors seem to occur in a few percent of the cells of spreadsheets, that the question for a large one is how many errors it holds rather than whether it holds any, and that assumption sections may matter far less than testing after development.
The third strand concerns forecasts of time and cost. Daniel Kahneman and Dan Lovallo, Timid Choices and Bold Forecasts: A Cognitive Perspective on Risk Taking (1993), trace overly optimistic forecasts to an inside view that anchors predictions on plans and scenarios and neglects the statistics of the past. Roger Buehler, Dale Griffin and Michael Ross, Exploring the "Planning Fallacy": Why People Underestimate Their Task Completion Times (1994), find that people predicting their own task completion times focus on plan-based scenarios, are too optimistic, and lose that bias when told to connect past experiences with their predictions. Winston Harrington, Richard D. Morgenstern and Peter Nelson, On the Accuracy of Regulatory Cost Estimates (2000), compare estimates made before regulations with assessments made after and trace the difference between their total-cost and unit-cost results to frequent errors in estimating the quantities, the effects of individual rules. Bent Flyvbjerg, Mette Skamris Holm and Søren Buhl, Underestimating Costs in Public Works Projects: Error or Lie? (2002), find that cost underestimation in their sample cannot be explained by error and is best explained by strategic misrepresentation.
These strands have usually been read apart: the first two ask whether a computation is right, the third whether a forecast is honest and well grounded. The records read here sit where they meet. Each forecast is a plan-based scenario of the kind the third strand studies, and each comes with a built-in check of the kind the first two strands weigh. Because the records are synthetic, nothing here can show whether the stated savings are optimistic or the runs mistaken; and this study imputes no intent, so the contrast Flyvbjerg and his co-authors draw between error and misrepresentation is used only to say which of the two the records leave room to discuss. What the records can show is how a designed decision record places its check, and what that check leaves unchecked.
4The plan of the paper
The chapter that follows opens with the data and the method: how the thirteen were chosen, what AEGIS says they are, which fields were read and which were set aside as the studio's frame. Its second section shows the one figure read two ways and what the record carries where the readings part. Its third section reads what the built-in check covers and what the records ask to be measured instead. An appendix gives the records read, word for word in this study's translation. The conclusion sets the findings beside the three strands of the debate, states the contribution again, and names the limits of a study of thirteen authored records.