A question with an answer is made before it is answered. Someone decides what counts as the answer, where it is to be found, how small it must be, and what it must not be taken to mean. This short study reads that making in fifteen public records. They are history questions from Dionyas, the music studio of the RISCON family, on three international instruments that concern performers, phonograms and broadcasting: the Rome Convention, the Geneva Phonograms Convention and the WIPO Performances and Phonograms Treaty, five questions on each. They ask when, where, under which article, who is protected, who receives the deposits. Each has a reference answer that its state line says was written from named documents; the public list counts those answers but does not carry them, and this study neither gives nor checks any of them.
The question and the contribution
The research question is this: how does a public record design a history question so that it has an answer, where does it place that answer, and what does it keep the answer from being taken to mean? The question is about design, not about the treaties. Nothing in this paper says what any of the three instruments provides, who is bound by it, or whether it applies anywhere today, and nothing here is legal advice.
The contribution is a close description of three moves, measured against the rest of Dionyas's history list and read through two literatures that rarely meet. First, the fifteen place their answers in named documents. Nine of them name in the question the document the answer is to come from, the document a question names is always one that its list of needs names too, and three say "according to", against 5 of the 500 history questions in the whole list. Second, they cut each instrument into five narrow facts, a date, a city, an article, a group of protected parties, an identifier, and none asks why, how, or what followed. Third, a boundary line closes each record and fences its answer: it says what the question does, and it says what the question is not, a judgment on law as it stands today or, for the WIPO treaty, on the lawfulness of AI training or distribution. The paper also offers a small method for public question sets of this kind: measure the wording a set shares within itself, set it aside as frame, and treat as evidence only what is each record's own.
Is a reference answer found or made?
The first debate concerns ground truth, the reference answers against which a system's outputs are scored. One line of work builds such answers on documents, and its designs make the document the place where the answer stands. Rajpurkar, Zhang, Lopyrev and Liang, SQuAD: 100,000+ Questions for Machine Comprehension of Text (2016), present a reading comprehension dataset of questions posed by crowdworkers on Wikipedia articles, in which the answer to each question is a segment of text from the corresponding passage. Thorne, Vlachos, Christodoulopoulos and Mittal, FEVER: a Large-scale Dataset for Fact Extraction and VERification (2018), go a step further: their annotators classify claims as supported, refuted or lacking enough information, and for the first two classes record the sentence or sentences that form the evidence needed for the judgment. Accuracy, they report, is lower when a system must give the correct evidence with its label than when the evidence is ignored. Lin, Hilton and Evans, TruthfulQA: Measuring How Models Mimic Human Falsehoods (2022), design questions in categories that include law, crafted so that some humans would answer them falsely because of a false belief or misconception.
A second line of work asks where such answers come from. Jaton, We get the algorithms of our ground truths (2017), follows scientists designing an image-processing algorithm and shows that the problems taken as starting points are provisional results of collective and material processes; problematization leads to referential databases called ground truths, which both shape algorithms and evaluate them, and which may be inherited from earlier problematizations and passed to later ones. Jaton proposes reading algorithms as instructions that retrieve outputs designed and designated during a specific problematization. Kang, Ground truth tracings (2023), drawing on fieldwork with practitioners who build machine listening systems, argues that learnability bends according to the ground-truthing practices by which a prediction task is formalized, and traces the ontological translations made along the way. Raji, Bender, Paullada, Denton and Hanna, AI and the Everything in the Whole Wide World Benchmark (2021), explore the limits of benchmarks set up as general, broad measures of progress and the construct validity issues that framing raises. Paullada, Raji, Bender, Denton and Hanna, Data and its (dis)contents (2021), survey critical work on how datasets are designed and used and advocate documenting and analysing datasets, qualitatively as well as quantitatively, while they are made.
This study reads the two lines as a disagreement about where an answer stands. In the first, the answer stands in a document and the question points to it. In the second, the answer is designed together with the question, and the document is one of the materials of that design. Neither line has much to say about a question set that is published as a design, with its reference answers counted but held back, by a studio that writes about music rather than trains a system.
What does citing a document do?
The second debate concerns facts and the procedures that make them. Tuchman, Objectivity as Strategic Ritual (1972), finds that the newspapermen studied claim objectivity by citing procedures that exemplify the formal attributes of a news story, for instance by quoting other people instead of offering their own opinions. It suggests that objectivity may be seen as a strategic ritual protecting them from the risks of their trade, and asks whether other professions use the term in the same way. Uscinski and Butler, The Epistemology of Fact Checking (2013), argue that political fact checking employs objectionable practices, among them treating a statement that contains several facts as a single fact, and that these practices presuppose facts to be unambiguous and not subject to interpretation. Graves, Anatomy of a Fact Check (2016), answers critics of this kind with an ethnographic account of fact checkers at work, and argues that in moments of institutional unsettlement verification relies on factual coherence rather than straightforward correspondence.
A fourth work asks what the thing cited is. Buckland, What is a "document"? (1997), notes that the word ordinarily denotes a textual record, but that the effort to give access to more and more documents raised the question of what should count as one; the article recalls a functional view of the document and a view that equated the document with organized physical evidence, and observes that digital technology renews old confusions between medium, message and meaning.
On this study's reading, the debate turns on whether naming a document settles a fact or only moves the question to the document. Tuchman's ritual and Uscinski and Butler's critique suggest the second; Graves's account suggests that the moving is itself a practice with its own epistemology, worth describing on its own terms. Buckland adds that the document a question points to is not one kind of thing.
The shape of the paper
Chapter 1 sets out the data and the method, then reads the fifteen questions in two sections: how they place each answer in a named document, and how they cut an instrument into small dated facts and fence the answer with a line on what it is not. An appendix gives the records read, word for word in this study's translation. The conclusion returns to the two debates and states the contributions, the limits and the work that could follow.