When an automated system decides something about a person, the usual demand is that it explain itself. This paper proposes, from a set of invented records, that the demand may point at the wrong object. In the thirteen records it reads, what lets a human auditor stand behind a judgment is not an account of how the system reasoned, and not a fuller statement of its reasons, but a narrower set of things: the inputs the judgment used, the date they speak for, the condition under which it would change, and an outcome that someone else can test. Where these are open, a judgment with sealed reasoning can be audited. Where they are closed, a judgment that recites a rule, a total or a summary cannot, and a judgment published in full can still mislead.
The records and the question
ASI 2030 is a studio in the RISCON family. It publishes records of an imagined course of events from late 2026 to the end of 2030, in which AI systems, and in the later years a system the studio calls an ASI, make judgments for welfare offices, grid operators, platforms, courts and ministries. Every record is a synthetic scenario. Its systems, institutions, accords and figures are invented, and its developments toward artificial general or superintelligent systems are the studio's scenarios, not facts and not forecasts of this paper. What the records offer is a worked-out picture of how a decision might be framed and checked, and that framing is what this paper studies.
In each record a person in a role puts a question, a judgment is given with the grounds it rests on, and a human auditor gives one of the four opinions of financial auditing: unqualified, qualified, adverse, or a disclaimer, which means no opinion can be formed at all.
After a shared past, the studio's story forks into two futures. In the first, which the studio opens under the title The Judgment Audit Acts, audit laws come into force and judgment records become public. In the second, which opens under the title The Emergency Delegation, the records show the reasoning behind judgments sealed. The studio files some of its records under an axis it calls Judgment Transparency. Read across the fork, this axis asks the same thing twice, once in a world that publishes and once in a world that withholds: what does a person need to see in order to check a judgment?
This paper's question follows from that design. What does the record set treat as enough of a judgment's reasons for someone, an auditor or the person a judgment concerns, to check it? Is it an account of how the system worked, the reasons it gives, or something else?
The debate over explanation
The question sits inside a debate that has two strands, one about explaining models and one about justifying decisions, and the two do not always meet.
The first strand comes from machine learning. Zachary C. Lipton, The Mythos of Model Interpretability (2018), argues that the task of interpretation is underspecified: papers give diverse and sometimes discordant motivations for it, and he separates transparency to humans from post-hoc explanations as competing notions. Marco Tulio Ribeiro, Sameer Singh and Carlos Guestrin, "Why Should I Trust You?": Explaining the Predictions of Any Classifier (2016), is an instance of the post-hoc kind. It holds that understanding the reasons behind a prediction matters for deciding whether to trust and act on it, and proposes explaining any classifier's prediction by learning an interpretable model locally around it. Cynthia Rudin, Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead (2019), answers that trying to explain black boxes used for high-stakes decisions, rather than building models that are interpretable in the first place, is likely to perpetuate bad practices. Within this strand, then, the dispute is over whether one should explain an opaque model after the fact or refuse the opaque model.
Two further works widen the strand toward what an explanation is for. Tim Miller, Explanation in artificial intelligence: Insights from the social sciences (2019), argues that most work in explainable AI relies on researchers' intuition of what a good explanation is, while philosophy and psychology show that people bring cognitive biases and social expectations to explaining. Brent Mittelstadt, Chris Russell and Sandra Wachter, Explaining Explanations in AI (2019), observe that the simplified models built to approximate a complex system are useful for teaching professionals to predict its decisions and how it might break, but are a do-it-yourself kit for answering what-if questions rather than explanations as philosophy and sociology understand them.
The second strand comes from law and political philosophy, and its object is the decision rather than the model. Sandra Wachter, Brent Mittelstadt and Luciano Floridi, Why a Right to Explanation of Automated Decision-Making Does Not Exist in the General Data Protection Regulation (2017), answer, as their title states, the claim that the General Data Protection Regulation mandates a right to an explanation of automated decisions, a right they describe as one that would require explaining how opaque mechanisms reach decisions. Lilian Edwards and Michael Veale, Slave to the Algorithm? Why a 'right to an explanation' is probably not the remedy you are looking for (2017), argue that such a right is unlikely to be a complete remedy: the law is unclear about when it is triggered, and the explanations computer scientists have built may not meet the legal idea of meaningful information about the logic of processing. They see promise in explanations centred on the person asking and in systems that learn a model from outside rather than taking it apart. Margot E. Kaminski, The Right to Explanation, Explained (2018), argues that the quarrel over an individual right has obscured the wider accountability regime the same law establishes.
Two works ask what an explanation must do to count as a justification. Andrew D. Selbst and Solon Barocas, The Intuitive Appeal of Explainable Machines (2018), separate inscrutability, which asks for a description of a model's rules, from nonintuitiveness, which asks why the rules are what they are. Since law and technique address mostly the first, they argue, one must seek explanations of the process behind a model's development, not only of the model. Reuben Binns, Algorithmic Accountability and Public Reason (2017), holds that decision-makers owe the people their systems decide about justifications for the systems' outputs, and gives an account of that duty in terms of the democratic ideal of public reason.
Read together, the two strands share an assumption that this paper puts to the records. Both treat the thing to be disclosed as either the working of the model or the reasons for the decision. The disagreement is over which, and how much. The ASI 2030 records, because they imagine both a world that publishes judgment records and a world that seals their reasoning, let one ask whether either object is what a checker actually needs.
The argument and the contribution
The paper's answer is that, in this record set, neither is. In the future of published records, every file can be public and still not be checkable, because the reasons it states rest on a data window, an assumption or a frame that the statement does not test; and the record set's own figures for that future show publication running ahead of audit. Under sealing, the line the auditors draw does not run between judgments that give reasons and judgments that do not. It runs between judgments whose inputs an outsider can obtain and judgments whose inputs stay closed, whatever else they disclose.
The contribution is to name a third object of disclosure beside the model's working and the decision's reasons. This paper calls it the checkable judgment: the inputs and their date, the condition under which the judgment would change, and a result that someone else can recompute or watch fail. Nothing here shows that such an object would suffice anywhere outside the scenario. What the records show is that a carefully built scenario of audited AI, imagined in two opposite futures, locates the check there, and they show what this costs and whom it leaves out.
Chapter 1 sets out the records, the selection rule and the method, then reads the published future and the sealed future in turn. The conclusion returns to the debate, states the contribution and its limits, and suggests further research. An appendix gives the thirteen records word for word.