Introduction

A question about a decision can be asked in the abstract, or it can be set in a scene. Ask in the abstract whether fans care when a show changes a song, and the answer will be an opinion about fans. Set the same worry in a scene, with someone who planned the show and a chorus the fans had learned by heart, and the answer has to deal with that chorus and that planner. Social research has used such scenes for decades and has argued about them for as long. Design has its own version, the persona, an invented user with a job and a name, and it has its own argument. This study reads thirteen records that sit where these two arguments meet.

The records come from Dionyas, the music studio of the RISCON family. Dionyas publishes inquiries, not answers: each record is a question about music designed before anyone answers it, with a line on what it covers, the material it would need, a boundary on what it may claim, and its state. Most of its 1,539 inquiries are general designs, historical questions and cases about real releases. Thirteen in its qualitative lane are of another sort. They are synthetic, and they say so in their own words:

The same records say where each came from: its source card was

Nothing more is said about the model or about what the checking consisted of. This study takes the records at their word on both counts. It draws no fact about the music business from them; it reads them as designs of questions.

The research question and the contribution

The question of this study is how a synthetic inquiry, a question set in an invented scene, makes its own premise open to being shown wrong, and what its account of its own making lets it claim.

The answer it gives has three parts. First, the scene in these records is compressed almost to nothing: a role of a few words and a question of about ten. It is less a vignette than the stem of one. Second, each record carries a single line on what it would need, and that line, unlike the parallel lines elsewhere in the list, is its own and is written as a test; seven of the thirteen say which result would weaken the premise of the question they belong to. Third, the tests name nobody who would run them, and their shape changes with the series of source cards from which the records came, more than with the subject the records ask about.

The contribution is threefold. It gives a measured description of a form of question that neither of the two literatures below has studied: the invented scene that writes its own test. It moves an old objection to personas, that they cannot be verified or falsified, from the invented figure to the question the figure carries, and shows that the question can answer the objection where the figure cannot. And it reads the records' brief account of their making against the call to document how a dataset was made, as a partial answer that says who registered a record but not how it was checked.

Vignettes: the concrete hypothetical

The first debate is about vignettes, short invented situations put to people in surveys and interviews. Cheryl S. Alexander and Henry Jay Becker, The Use of Vignettes in Survey Research (1978), made the case for them early: systematically elaborated descriptions of concrete situations, they argued, produce more valid and more reliable measures of opinion than the simpler abstract questions more typical of opinion surveys. Their design varied the characteristics of the situation across respondents while keeping the number of versions small. Janet Finch, The Vignette Technique in Survey Research (1987), discussed the technique's potential for eliciting data of a normative kind, beliefs about what people ought to do, and compared British and American ways of using it.

Two strands grew from there. One made the vignette an experiment. Christiane Atzmüller and Peter M. Steiner, Experimental Vignette Studies in Survey Research (2010), describe vignettes as short descriptions of situations or persons shown to respondents to elicit their judgments, with important characteristics varied systematically, and show that how the subsets of vignettes are chosen has strong consequences for what the answers can be taken to mean. Herman Aguinis and Kyle J. Bradley, Best Practice Recommendations for Designing and Implementing Experimental Vignette Methodology Studies (2014), present the method as a way out of a dilemma: experiments are strong on internal validity and weak on external validity, field studies the reverse, and carefully constructed, realistic scenarios can hold both, enhancing realism while letting the researcher control what varies. They name the need to increase realism among the method's gaps.

The other strand kept the vignette qualitative. Rhidian Hughes, Considering the Vignette Technique and its Application to a Study of Drug Injecting and HIV Risk and Safer Behaviour (1998), describes vignettes as stories generated from a range of sources, including earlier findings, and weighs their value against their difficulties. Rhidian Hughes and Meg Huby, The application of vignettes in social and nursing research (2002), review their potential and their limits for studying attitudes and beliefs on sensitive subjects. Christine Barter and Emma Renold, 'I wanna tell you a story': Exploring the application of vignettes in qualitative research with children and young people (2000), show how a vignette lets young people discuss sensitive topics while keeping a high level of control over the research.

Running under both strands is a doubt about the hypothetical itself. An answer to an invented scene may not be the answer a person would give to a real one. Jens Hainmueller, Dominik Hangartner and Teppei Yamamoto, Validating vignette and conjoint survey experiments against real-world behavior (2015), start from exactly this doubt: little evidence existed on whether preferences about hypothetical choices in a survey are driven by the same things as actual choices. Comparing survey experiments with the outcomes of closely corresponding referendums, they found the estimates matched remarkably well, and that seemingly subtle differences in survey design made significant differences in how well they matched. The debate, then, is not settled in favour of realism or against it. It turns on design: on how much of the scene is described, what is varied, and whether anything outside the scene is used to check it.

Personas: a figure that cannot be falsified

The second debate is about personas, invented users with a job, a goal and often a name, that design teams keep in view while they work. John Pruitt and Jonathan Grudin, Personas: Practice and Theory (2003), describe the technique as one of considerable potential: personas engage team members very effectively, provide a conduit for conveying a broad range of qualitative and quantitative data, and focus attention on aspects of design and use that other methods miss. They also outline why personas engage more than design based mainly on scenarios.

Christopher N. Chapman and Russell P. Milham, The Personas' New Clothes: Methodological and Practical Arguments against a Popular Method (2006), answer with a sharp objection. It is hard to know how many users, if any, a persona represents; personas cannot be adequately verified or falsified and so have no demonstrable validity; they are likely to lead to political conflicts and to weaken the ability to settle questions with data. Tara Matthews, Tejinder Judge and Steve Whittaker, How do designers and user experience professionals actually perceive and use personas? (2012), found that experienced practitioners used personas almost only to communicate, not to design, and called them abstract, impersonal, misleading and distracting. Personas, they conclude, cannot replace immersion in actual user data.

The two debates meet in the records read here. Each Dionyas record gives its scene a role, as a persona does, and asks a question about a situation, as a vignette does. The objection of Chapman and Milham falls on the role: nobody can check whether a cover director of a fictional album team stands for anyone. This study asks whether the question can be checked even where the role cannot.

Saying how a record was made

A third, shorter line of work bears on the records' account of their making. Timnit Gebru, Jamie Morgenstern and others, Datasheets for Datasets (2021), observe that machine learning has had no standard way of documenting datasets, and propose, by analogy with the datasheets that accompany electronic components, that every dataset come with a sheet documenting its motivation, composition, collection process and recommended uses. This study borrows the idea as a lens and says where it fits badly: a Dionyas inquiry is a record, not a dataset, and its account of its making is a single clause.

What follows

Chapter 1 gives the data and the method and then reads the thirteen records in three sections: how they compress a scene into a role and a question; how each line on what an inquiry would need becomes a test, and which tests name the result that would weaken the premise; and how the tests leave unnamed who would run them while their form follows the series of source cards. An appendix gives every record, word for word in this study's translation. The Conclusion sets out what the reading adds to the two debates, its limits, and the work that would carry it further.