Introduction

The question

A freelance designer in one of ASI 2030's imagined futures opens the record behind a monthly contribution score and asks a short question:

The question has two halves, and they belong to two different parties. Less work is the designer's doing. A changed formula is the scorer's. Before the designer can ask what to do next, the designer needs to know which half the drop belongs to, and that is something only the party holding the formula can settle. This paper argues that the record set it reads treats exactly this split as the test of an answer about a change in a score: of the five records that ask about such a change, the one with a clean opinion is the one whose answer divides the change between what the person did and what the scorer did. Whether the formula is public matters less than whether that split can be made, and in the future where formulas are sealed, two of the auditors' lines on what would change their opinion record that nobody can say what would raise or restore a score.

The records are synthetic. They come from ASI 2030, one of the RISCON family's studios, whose library imagines the years from late 2026 to the end of 2030: AI systems make judgments, and a human auditor gives each one an opinion borrowed from financial auditing, unqualified, qualified, adverse, or a disclaimer, which means the auditor declines to give an opinion because the material for one is missing. The story runs as one shared past and then forks into two futures. Eighteen of them, filed under the axes Measuring Human Worth and Work and Income across both futures, are read here. The studio does not name its two futures; this paper calls them the open and the sealed future, after how many of their records keep part of a judgment's grounds undisclosed (2 of 9 and 7 of 9; the method section gives the count). Every system, platform, score, institution and figure in them is invented, and their developments toward artificial general and superintelligent systems are the studio's scenarios, not forecasts made here. What can be studied is the shape the records give to a decision about someone who is scored, and the questions their auditors put before letting it stand.

The research question is therefore narrow: in these eighteen records, what does the record set treat as an answer a scored person can use? It could be the formula, from which the person works out the rest; the reasons, the factors that weighed most; or, as this paper finds, a decomposition that shows how much of a change came from the person and how much from the scorer, and says what would change it.

The debate: recourse, and the worker who is scored

Two bodies of work meet over this question. The first asks what a person refused by an automated decision must be able to do. Berk Ustun, Alexander Spangher and Yang Liu, Actionable Recourse in Linear Classification (2019), hold that someone refused a loan by a credit score should be able to change its inputs so as to guarantee approval; otherwise the refusal lasts for as long as the model is in use, and the person has no hand in a decision bearing on their livelihood. They call this recourse, the ability to change a model's decision by altering actionable input variables, income for example rather than age or marital status, and they find that recourse can be significantly affected by standard practices in model development, so that it has to be evaluated in practice. Amir-Hossein Karimi, Bernhard Schölkopf and Isabel Valera, Algorithmic Recourse: from Counterfactual Explanations to Interventions (2021), press the point further: a counterfactual explanation tells an individual where they need to get to, but not how to get there, and they propose moving from explanations to interventions. Suresh Venkatasubramanian and Mark Alfano, The philosophical basis of algorithmic recourse (2020), give the stakes in moral terms: recourse, the systematic process of reversing unfavorable decisions by algorithms and bureaucracies, underwrites two components of a good life, temporally extended agency and trust.

This line of work has a premise that its own critics have named. Solon Barocas, Andrew D. Selbst and Manish Raghavan, The hidden assumptions behind counterfactual explanations and principal reasons (2020), observe that explanations which highlight the features most relevant to a decision, and withhold others, rest on assumptions that are easy to overlook: that a recommended change maps clearly to an action in the world, that features can be made commensurate from the training data alone, that features bear only on the decision at hand, and that the model is stable over time. They also note a tension between making such explanations useful and keeping the model hidden. The designer's question puts the assumption of stability in doubt: this study reads it as asking whether the formula behind any advice on what to change had moved.

The second body of work looks at people who are scored at work. Min Kyung Lee, Daniel Kusbit, Evan Metsky and Laura Dabbish, Working with Machines: The Impact of Algorithmic and Data-Driven Management on Human Workers (2015), studied ridesharing drivers whose jobs are assigned, optimized and evaluated through algorithms and tracked data, and describe how the drivers used online forums to make sense together of the algorithm's features. Alex J. Wood, Mark Graham, Vili Lehdonvirta and Isis Hjorth, Good Gig, Bad Gig: Autonomy and Algorithmic Control in the Global Gig Economy (2018), found algorithmic control central to how online labour platforms operate, offering flexibility and autonomy and, through the same mechanisms, low pay, isolation, irregular hours and exhaustion. Ryan Calo and Alex Rosenblat, The Taking Economy: Uber, Information, and Power (2017), ground their critique in asymmetries of information and power: a firm that sits between the parties can monitor and nudge all of them, including people whose livelihood may depend on the platform, much of its activity is hidden from view, and preliminary evidence suggests such firms may already use that position to disadvantage participants. Ali Alkhatib and Michael Bernstein, Street-Level Algorithms: A Theory at the Margins of Practice (2019), add a point about timing: where a human official refines the decision criteria while reasoning through a new case, an algorithm at best refines them after the decision is made, which yields illogical decisions in new or extenuating circumstances. And Reuben Binns, Max Van Kleek, Michael Veale, Ulrik Lyngs, Jun Zhao and Nigel Shadbolt, 'It's Reducing a Human Being to a Percentage': Perceptions of Justice in Algorithmic Decisions (2018), found in experiments that people judge algorithmic decisions with concerns about arbitrariness, generalisation and dignity, and that there may be no single best way of explaining such decisions.

Read together, the two bodies of work leave a gap that the records make visible. Recourse work asks what the person must change, and holds the scorer still. Work on algorithmic management shows the scorer moving, hidden and powerful, but asks less what a single answer to a single worker should contain. The records stand between: every one of them is a single question about a single score, put to the system that holds it, and checked by an auditor.

The contribution, and the plan

The paper makes three contributions. First, it names a test that comes before recourse and that the record set applies wherever a person asks about a change in a score: whether the change can be split into the person's part and the scorer's part. This paper calls such a score separable. Second, it shows that publication and separability come apart in these records: a public formula ends in a disclaimer when part of its input is other people's ratings that stay closed, an unpublished formula still draws a remedy that strips out the scorer's share, and the other three clean opinions go to questions in which no change was asked about. Third, it shows how the sealed future records, in two of the auditors' own lines, that nobody knows what would raise or restore a score, and how two of its three adverse opinions rest on data that someone outside the scorer collected.

Chapter 1 sets out the records and the method, then reads the nine records of the future with open judgment records and the nine of the sealed future. The conclusion returns to the debate, states the contributions again with their limits, and names further research. An appendix gives the eighteen records word for word.