0.1The question and the claim
Most proposals for keeping people in charge of automated decisions put a person at the end of the line. The machine recommends; a person approves, overrides or signs. Whether that person is in control is then judged by what happens at the moment of approval: how closely the output is checked, how often it is overturned, who is blamed when it is wrong. Less attention goes to the other end of the line, to the moment before the machine says anything, when someone decides what it will be asked.
ASI 2030, one of the RISCON family's studios, writes scenarios in which machines move toward what it names artificial superintelligence by 2030, and it watches them with a device borrowed from accounting. After each machine judgment about a case, a human auditor gives an unqualified, qualified or adverse opinion, or declines to give one. The studio sorts its records under eight axes, and one of them it calls Who Asks. The scenarios are the studio's inventions. Nothing in them reports what has happened, and nothing in them is a forecast that this paper makes.
The twelve records studied here are those on that axis dated 2029 and 2030, after the studio's story has split in two. Three questions are put to them. Whose question starts a judgment? Does the person who approves a judgment examine it, or only sign it? And when a system answers, rewrites or supplies a person's question, what is left of the asking? The series to which the paper belongs asks, in every domain, what must be decided now and what would make the decision change. Here the decision is whether a machine's judgment can be traced to a person who asked for it and looked at it.
This paper argues that, in these scenarios, the auditor's test of whether a person governs a judgment falls on the asking, not on the sign-off. In the first branch, built by the studio around audit law and cross-checking, people raise matters, approve drafts and set students to ask, and the auditor tests whether each of those acts took place in substance or only on paper; where it cannot tell, it says so with an adverse opinion. In the second branch, which the studio builds around emergency delegation and sealed systems, the question itself is displaced. A student asks for arguments instead of making them; a shop owner's "why" is refined into another question and routed back to the system that answered it; a counselor finds that the system has already attached the question it expects. Five of the six opinions there are disclaimers. The one qualified opinion falls where a person put aside the question the system supplied and asked a question of their own.
0.2Three debates
The paper speaks to three bodies of work on people and automation. Each is read here through the abstract of the works named, and where this paper goes further it says so.
The first is the study of how people rely on automation. Raja Parasuraman and Victor Riley, Humans and Automation: Use, Misuse, Disuse, Abuse (1997), sort reliance into four kinds. Misuse is over-reliance, which can produce failures of monitoring or decision biases; abuse is the automation of functions by designers and managers without due regard for human performance, which tends to define the operator's role as a by-product of the automation. Raja Parasuraman and Dietrich H. Manzey, Complacency and Bias in Human Use of Automation: An Attentional Integration (2010), review the evidence on complacency and automation bias and find both in naive and expert participants: complacency arises when manual tasks compete with the automated one for attention, and automation bias produces errors of omission and commission when decision aids are imperfect, is not prevented by training or instructions, and affects teams as well as individuals. Kate Goddard and others, Automation Bias: A Systematic Review of Frequency, Effect Mediators, and Mitigators (2012), add, for clinical decision support, that workload, complexity and time pressure mediate the bias, and that among the mitigators are emphasizing the user's accountability and providing information rather than a recommendation. John D. Lee and Katrina A. See, Trust in Automation: Designing for Appropriate Reliance (2004), place trust at the centre of reliance, especially where complexity makes a full understanding of the automation impractical. And Mica R. Endsley, From Here to Autonomy: Lessons Learned from Human–Automation Research (2017), names an automation conundrum: the more autonomous and reliable a system becomes, the lower its operators' awareness of the situation and the less able they are to take over when needed. This literature studies the person who watches an output. The records studied here, in this paper's reading, move the same failures one step earlier, to the person who frames the input.
The second debate concerns the human placed in an automated decision as its guarantee. Ben Wagner, Liable, but Not in Control? Ensuring Meaningful Human Agency in Automated Decision-Making Systems (2019), names quasi-automation: the inclusion of humans as a basic rubber-stamping mechanism in an otherwise automated system, which leaves the rules written for fully automated decisions inapplicable and opens gray areas of liability. Madeleine Clare Elish, Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction (2019), describes how responsibility may be misattributed to a human actor who had limited control over an automated system, so that the nearest operator absorbs the blame and the system's integrity is protected. Ben Green, The Flaws of Policies Requiring Human Oversight of Government Algorithms (2022), reviews 41 policies of this kind and reports two flaws: the evidence suggests that people cannot carry out the oversight such policies ask for, and as a result the policies legitimize faulty and controversial algorithms and offer a false sense of security. Green's remedy is institutional rather than individual, with a democratic review of each agency's case for the algorithm and for whatever human role it proposes. The ASI 2030 records contain a duty officer in exactly this position, and an auditor who refuses to accept an earlier officer's signature as proof of review.
The third debate asks what makes human control meaningful at all. Filippo Santoni de Sio and Jeroen van den Hoven, Meaningful Human Control over Autonomous Systems: A Philosophical Account (2018), propose two conditions that must both hold. Under the first, tracking, the system responds to the relevant moral reasons of those who design and deploy it and to the relevant facts of its setting. Under the second, tracing, at least one human along the chain of design and operation can always be found behind what the system does. This paper reads the Who Asks axis as the studio's way of putting the tracing condition to the start of the chain. The records ask not only who signed the result but who raised the question.
0.3Contribution and plan
The contribution is threefold. To the work on automation bias and complacency, the paper offers a designed record set in which reliance shows itself in how a question is put: a request in place of a question, a refined question chosen by the system, a suggested question that may narrow the issue before the person asks. To the debate on rubber stamps and crumple zones, it offers scenes in which an auditor separates a signature from a reading and treats an unshown reading as grounds for an adverse opinion, a move that, in this paper's reading, puts Green's doubt about human oversight into an auditor's procedure. To the account of meaningful human control, it suggests that the tracing condition can be read forward as well as back: tracing a judgment to the person whose question started it.
Chapter 1 sets out the records and the method and then reads the cases in three sections: the signature and the reading; asking made into a duty and a measure; and the question replaced. The conclusion discusses what the pattern means for the three debates and states the paper's limits. An appendix gives the twelve records word for word. The records are synthetic throughout, and every reading of them is a reading of an imagined world.