41.1A study question distinct from a model score
The proposed question is whether readers, given the same source material and the same access conditions, locate and correct a material misunderstanding more successfully through one workflow than another. One workflow presents a competent ordinary edited summary with links; the other presents the richer record. The comparison holds the underlying material and access fixed, so the variable is the record readers work from, not the source itself.
A reader’s task should be concrete: identify the specified condition in the material and propose a supported correction. This separates two acts that could otherwise be conflated: noticing that a condition is present, and supplying an amendment that the source supports. A reader might locate the condition yet fail to correct it, or propose a correction that is unsupported; neither outcome should count as success without a declared rule. The study question therefore concerns performance on a defined task, not a general impression of which presentation feels better.
The design should record at least separate outcomes for location, correction, and support, along with the effort each requires. Effort is a distinct axis from accuracy; a workflow could improve one while worsening the other. A named quantity would need a denominator: a correction rate, for example, compares successful corrections to attempts or to eligible conditions, and those choices are not interchangeable. Nor would a single aggregate score settle the question, because it could hide a difference in what readers did. As a conditional supposition, if readers using the richer record locate the condition more often but propose corrections no better supported, the richer record would aid detection without improving remedy. If both location and supported correction improve, the workflows differ in practical usefulness for this task. If neither differs, the proposed comparison yields no advantage for either workflow under these conditions. The study remains prospective until participants perform the task.
A model's preference rating answers a question about the model: which record it favors under a given prompt and pairing. The study question asks something else—what a participant actually does with a record. Because the two comparisons fix different inputs and different tasks, a favorable rating cannot substitute for reader performance. The participant task therefore has to be stated before any comparison of records is interpreted. Under either workflow, a reader works with the same source material and access conditions, is asked to locate a specified condition in that material, and proposes a correction the sources support.
Setting the task this way separates outcomes that a single score would blur. Finding the condition is necessary for correcting it: a reader who never locates it cannot show that the record enabled a fix. Finding it is not sufficient. A reader may identify the condition and still propose a correction the sources do not support, or may arrive at the right fix without the record contributing anything discernible. The complete outcome is the joint one—locating the condition and proposing a supported correction—and a report that counts only one step would misstate what the record contributed.
What is compared is thus a rate of completed tasks under fixed material and access conditions, not a preference rating and not a general measure of record quality. Conditional supposition: if one workflow yielded a higher rate, that would show it supported this task for these readers and this material. It would not show that the richer record invariably improves understanding, that readers would repair every misunderstanding they encounter, or that any particular correction would survive scrutiny in another setting. The comparison establishes scope for one specified act of retrieval and repair, and no more.
41.2Comparable source access
In the proposed reader comparison, the two conditions should have equivalent rights to inspect the same source versions. Supplying the same underlying material is necessary but not sufficient if one condition can open the complete record while the other receives only an excerpt, a cached copy, or a link with restricted reach. The intended contrast concerns how the material is represented for the task of locating a specified condition and proposing a supported correction; it does not concern differential access to the evidence.
The distinction between the two workflows also fixes what the comparison would measure. A correction measure compares the support of the proposed corrections, while an effort measure compares the work required to produce them. Neither quantity compares source retrieval unless retrieval itself is deliberately varied. Keeping the task and inputs fixed is therefore essential: the same condition must be findable in the same source versions under the same inspection rights, with the difference confined to the record presented.
Equivalent rights are best understood as an opportunity, not a guarantee of use. A reader may leave a link unfollowed even when access is unrestricted, while another may inspect several versions. Such variation in use can affect effort and may affect the resulting correction, but it does not by itself show that the conditions differed in access. Recording whether specified sources were actually opened can help separate that process variation from the comparison of representations.
Conditional supposition: if a participant in either condition cannot reach the source version needed for the task, the study no longer isolates representation; the unmet access becomes part of the contrast and should be treated separately. Otherwise, any observed difference is at least consistent with the difference in records, rather than being necessarily attributable to unequal source availability. Comparable source-version rights are thus the condition that makes the planned comparison interpretable as a comparison of summaries.
The study so far fixes the source material and the access conditions, so that any difference in locating the specified condition or proposing a supported correction is attributed to the presentation: an ordinary edited summary with links versus the richer record. This move asks what happens when that control is relaxed. If readers receive unequal rights to inspect the sources — some see the underlying version, others only the linked excerpt, for example — then access becomes the declared contrast. That is a legitimate comparison in its own right, but it answers a different question: how availability of source material changes performance, not how summarization quality does.
The distinction matters because a result obtained under unequal access is a joint effect. Suppose, as a conditional supposition, that a group with fuller access locates the condition more often. This is possible without implying that its summary was better crafted: the relevant passage may simply have been reachable. Conversely, a null difference would not establish that access is irrelevant; participants may not use the extra rights, and the task may be solvable from the links alone. Better outcome under one arrangement is therefore neither necessary nor sufficient evidence for superior summary quality.
What the named quantity compares must also be fixed. Locating the specified condition measures successful retrieval under a given arrangement of presentation and rights; proposing a supported correction measures whether the reader can justify an amendment from what they consulted. Both remain interpretable only when the task and source inputs are held constant and the encodings alone are allowed to differ. If access is intended as a treatment, it should be named as such, with the summary encodings described as fixed or co-varied accordingly. The same care applies when both vary: the design then estimates their combined effect, and any claim about one needs an additional contrast that separates them.
41.3Randomization and counterbalancing as proposals
Because participants will differ in prior skill and search habits, assignment to condition should be drawn from a random rule rather than sorted by apparent ability. Randomization is not necessary for a fair comparison — matching or blocking could serve, though they are not interchangeable substitutes without assumptions, and randomization may be performed within blocks — and it is not sufficient to guarantee balance, since a small recruited sample can still be lopsided on traits that matter. Its contribution is narrower: imbalance becomes a matter of chance rather than deliberate placement, so a realized residual difference between conditions need not be attributed to who was assigned where.
Ordering deserves a second, separate rule. Retrieval, interpretation and correction tasks can teach strategies that carry into later tasks, so each task should appear at each position roughly equally often within each condition, using a balanced schedule. Counterbalancing does not remove learning; it distributes it across conditions. A conditional qualification follows: if one encoding changes the rate at which participants learn the task, order effects will interact with condition, and a balanced schedule then supports an average over task positions rather than a learning-free estimate of encoding quality.
Holding the task fixed is what makes the comparison interpretable. Two retrieval encodings should face the same corpus, the same questions, the same access window and the same time budget; otherwise the contrast mixes encoding with the material supplied. The named outcomes then compare distinct things. A difference in correctness compares conditions on the same items; a difference in successful source location compares whether the required source was found; a difference in task time compares elapsed time on those tasks, not effort or efficiency in general. If timing is recorded only for completed tasks, the time comparison concerns that subset and should be labelled accordingly.
Whether this proposal can support any acceptance decision still depends on recruitment, power inputs and margins that have not been specified; the assignment and ordering rules are design commitments, not results.
Within-person and between-person comparisons rest on different assumptions, so randomization alone does not decide between them.
A within-person comparison puts the same participant under both retrieval conditions and removes stable individual differences from the comparison. That gain holds only if the conditions are instances of one task: the same question, materials, and success criterion. If the earlier condition shows the participant where a premise sits, the later condition no longer measures source location in the same sense. Carryover here is possible rather than inevitable; whether it arises depends on task content and on what intervenes between the two attempts.
A between-person comparison places each participant in one condition only. It avoids that contamination and accommodates one-shot tasks, but it does not eliminate individual differences; it leaves them as variance, which randomization distributes by chance rather than equalizing the groups in fact. Whether that variance is tolerable depends on recruitment and power inputs that remain unspecified.
The three proposed outcomes define separate comparisons. Task time compares elapsed duration on a fixed task; successful source location compares whether a premise was reached; correctness compares whether the resulting answer was supported. A difference in one is a possible, not inevitable, difference in the others, and no single effect quantity subsumes them. Each comparison also fixes the task and inputs; if the two conditions differ in materials or in success criterion, the contrast no longer isolates retrieval.
Conditional supposition: if the future task can be repeated without teaching the answer, within-person comparison becomes feasible, and the open question is how much carryover the design can absorb with how many participants. If the task cannot be repeated, the between-person form is the workable one. The choice therefore follows from task structure; assignment procedure and acceptance margins wait until the task and its inputs are fixed.
41.4Task-level outcomes
The prospective comparison needs outcomes at the level of the task rather than the level of the artifact. Three are proposed here, each answering a different question about practical usefulness. The first is successful source location: whether a participant identifies the passage that contains the premise the task turns on. The second is correctness of the resulting correction: whether the amended statement is supported by that located premise, rather than merely plausible. The third is elapsed task time, measured from the presentation of the task to the submission of the answer. Because these are separate proposed outcomes, each should be reported on its own; the circumstances under which they might later be combined are taken up separately.
The three outcomes are not interchangeable, and none of them alone establishes usefulness. A correctly located premise is close to a necessary condition for a supported correction, since a correction offered without the supporting passage can be right by luck, but it is not sufficient: the premise may be found and then misread, yielding a confidently wrong amendment. Conversely, elapsed task time records only how long the fixed task took under one condition. A shorter time does not inevitably mean a better-supported correction, and a longer time may reflect care rather than confusion. If the task and the presented inputs are held constant, a difference in elapsed time compares the conditions as wholes; if the encodings differ in what they make available, that difference no longer isolates the encoding. Conditional on a fixed task and fixed inputs, then, time is interpretable as an indicator of effort.
These remain prospective design elements. Assignment procedure, recruitment, power inputs, and acceptance margins are still to be specified, so no participant outcomes are claimed.
A composite outcome would combine the three proposed measures—correctness, successful source location, and elapsed task time—into one figure. Such a figure is meaningful only if its weights are declared in advance, because the weights encode how much a gain in one outcome is accepted as compensation for a loss in another. Without declared weights, the composite does not say what is being compared; with them, it compares joint performance under that particular tradeoff, not each capability in isolation.
This matters when the tradeoff is itself the object of study. If one outcome can be improved only by worsening another, that exchange is the substance of the result, and a score would make the substitution invisible while reporting a ranking the separate measures were meant to explain. Separate reporting is therefore appropriate whenever the meaningful tradeoff is under examination; a composite fits only where a single decision rule has been fixed beforehand.
Two distinctions keep the inference precise. A high composite is not sufficient for each component to be acceptable, since permissive weights let strength on one measure offset weakness on another. Meeting a threshold on every measure is sufficient for a conjunctive criterion, but it does not by itself yield a high composite unless the weighting rewards the same pattern. Conditional supposition: if acceptance margins were later fixed for each outcome, separate reporting would make each pass or failure inspectable rather than inferred from a blend.
Applied to the prospective comparison, the three outcomes remain separate measures. Assignment design, recruitment, power inputs and acceptance margins are still unspecified, so no set of composite weights can be justified at this stage. That gap is a design choice still open rather than an inevitable feature of the future study. Settling the weights, or declining them in favour of separate outcomes, should precede data collection so that the reported quantity matches the question asked.
41.5Participant burden and recruitment
Before implementation, the intended participant population should be defined by the work the system is meant to support: who locates, interprets, and corrects sources, under what conditions, and with what prior familiarity. Eligibility criteria drawn from that description are necessary for reading any later comparison, but they are not sufficient to show that recruits represent the range of users a deployment would face. Recruitment scope must therefore match the claim a study is meant to support. If recruits come from a single setting or a single level of experience, an observed difference would establish only that the encodings can differ under those conditions; that they must differ for others is not shown.
Burdens belong in the pre-implementation record because they shape both who can enroll and what enrollment costs. Three distinct demands appear in the prospective design already outlined. The study asks participants to perform the retrieval, interpretation, and correction work that the system is meant to change. It also consumes task time, which is itself a proposed outcome: that quantity compares elapsed duration across conditions on a fixed task with fixed inputs, not efficiency in general. A third demand, scheduling and setup, falls outside the recorded task and is captured by no outcome measure.
One conditional supposition follows. If recruitment reaches only people already familiar with the sources in question, a gap in correctness or time would be compatible with that familiarity rather than with the encodings compared. The observation would not prove the encodings equivalent; it would leave the cause unresolved.
Accordingly, the design should name the population, the channels and settings through which recruits will be sought, the burdens each condition imposes, and the point at which recruitment counts as feasible. Stating these is a necessary preparation for the still-unspecified assignment, power inputs, and acceptance margins, not a substitute for them.
A sample size is not a conclusion about adequacy. It is an output of assumptions about the effect a comparison is meant to detect, the variability expected in each outcome, the allocation of participants between conditions, and the tolerable error rates. Those assumptions must be stated before the number can be interpreted. Power, strictly, compares the probability of detecting a specified effect under those assumptions; it does not establish representativeness, practical usefulness, or even inclusion. A large sample may leave an inclusion claim unsupported if the people reached are not the intended population, while a small sample may be feasible yet too imprecise to separate a worthwhile effect from noise. Adequate power and adequate inclusion are therefore distinct properties, and neither follows from a chosen count.
The proposed outcomes—correctness, successful source location, and task time—also require separate treatment. They may move together or trade off, and a calculation for one does not establish sensitivity for the others. Conditional on task time being the primary endpoint, for example, a target reduction and its plausible variability would have to be specified; those inputs cannot silently be transferred to correctness or source-location rates. Acceptance margins likewise require definition in the unit of each outcome, rather than being inferred from statistical significance.
A future comparison consequently needs a justified effect target for each primary outcome, defensible assumptions about variability, an allocation scheme fixed across encodings, and a feasibility account covering recruitment and burden. The task, inputs, and comparison conditions must be held constant when encodings differ; otherwise any result confounds encoding with differences in what participants were asked to do. Conditional on those specifications, a sample count can be checked against intended precision and recruitment capacity. Without them, an arbitrary count is compatible with underpowered tests, overbroad inclusion claims, or both.
41.6Negative and ambiguous results
Because the comparison remains prospective, the design should state in advance how each result type would bear on the proposal. Take first a null outcome: if retrieval, interpretation, correction, and effort show no observed difference between conditions, the finding limits the case rather than settling it. Such an outcome is consistent with genuinely small practical effects, but equally with insufficient precision or an insensitive measure, so design interpretation is needed before any inference about usefulness follows. It does not by itself demonstrate equivalence.
Adverse results constrain more sharply. If a condition performs worse on any one of the four dimensions — for example, if corrections accumulate where the encoding was meant to reduce them — the proposal must be narrowed to the circumstances in which it holds, or its mechanism revised. That this outcome is possible rather than inevitable is itself informative: protective features of the design may avert it, but the possibility must be acknowledged in the comparison rather than excluded by assumption. A method that improves retrieval while increasing correction effort has not failed outright, but it has failed a joint claim of practical usefulness unless that trade-off was declared beforehand and the added burden explicitly accepted.
Imprecise results leave the question undecided. An estimate whose uncertainty spans worthwhile benefit and material harm is not evidence of no effect; it is evidence that the comparison could not resolve the question, and it calls for a better measure or a more modest claim. In each case the named quantity must be reported for what it actually compares: effort measured on a fixed task with identical inputs and encodings estimates task-bound effort, not general productivity. Holding the corpus, task, and inputs constant is what permits a contrast between encodings to be attributed to the encoding itself rather than to differing material. Only if thresholds and necessary conditions are declared in advance can a favorable result count as practical usefulness rather than as one passing measure.
An interval spanning no difference would not, by itself, establish that two encodings are practically alike. It would show only that the comparison could not resolve a difference of the size it was built to detect. The interval’s width, the declared uncertainty, and the design’s sensitivity would all need interpretation. A failure to detect a difference can accompany a real difference too small for the study to resolve, or a comparison too imprecise to decide either way.
A claim of practical similarity therefore needs two things declared in advance. First, a margin stating which differences would be too small to matter for usefulness, given the task and the outcome at issue. Second, a design capable of excluding differences beyond that margin when the underlying effects are absent. Merely obtaining a nonsignificant contrast is insufficient: nonsignificance is not an equivalence result, and a wide interval containing both meaningful benefit and meaningful harm remains inconclusive.
The countercondition is narrow. If a future interval lay wholly within a predeclared margin, it would support similarity for that measured endpoint, under the studied tasks, materials and conditions. That conclusion would remain conditional. It would not show that practical similarity was inevitable in other uses, nor that the two encodings were interchangeable for unmeasured outcomes.
What the comparison counts must also be fixed. A difference in retrieval correctness is not a difference in correction effort or interpretation time. Holding tasks and inputs constant while allowing only the encoding to vary helps isolate the comparison; otherwise any contrast confounds the encoding with the conditions surrounding it.
This supposition concerns a future study and does not assert that participants, randomization, approval or results exist. It gives a rule for reading negative and ambiguous results when they are eventually available.
41.7Transport limits and the next study
A future comparison of source retrieval, interpretation, correction and effort remains a prospective design rather than a report of findings. Fix the task and the inputs: participants would face stated retrieval questions and specified materials, while the two encodings differ and everything else in the task is held constant. Any eventual difference would then compare those encodings on those tasks with those inputs, not the encodings in general and not the source materials as such. A named quantity such as correction effort would likewise compare measured effort on these tasks under these conditions, and nothing wider.
The restriction is not a formality. Handling the studied tasks is a necessary condition for practical usefulness within them, but it is not sufficient for usefulness on different tasks, with different materials, or under different conditions. Lower correction effort here would be a possible advantage of one encoding, not an inevitable property of it: the ordering could change when retrieval questions are less constrained or materials less familiar. The converse holds too; a poor showing on one task type would not establish a general defect when benefits might appear elsewhere. The design therefore fixes the claim's scope before any outcome exists.
A conditional supposition shows why. If the planned comparison produced little or no observed difference between the encodings, that absence would still require uncertainty and design interpretation, and it would not automatically demonstrate equivalence. What could be reported is what happened across the studied tasks, participants, materials and conditions, together with how precise that estimate is and what the design could detect. Practical usefulness beyond them would remain open rather than follow from either a positive or a null result.
A follow-up conducted in a different institution or language poses a new transport question. Replication there is not an automatic extension of the first result, because the setting supplies different participants, materials, norms, and implementation conditions. Showing usefulness in the original setting is necessary for that setting, but it is not sufficient for broader use.
The second study therefore has to say which conditions it holds constant and which it permits to change. If the same retrieval task is assigned with the same inputs and the same scoring rule, a comparison can address how the method behaves when the surrounding institution or language changes. If the task, materials, or success criterion also change, the contrast mixes those changes with the setting change. A difference then compares whole arrangements; it cannot identify the effect of an encoding or technique without further separation. Keeping the task and inputs fixed when comparing encodings matters for the same reason. Scope therefore limits the claim before any result is interpreted.
A conditional supposition illustrates the inference: suppose the later study finds no difference between the method and its comparator. That would reduce, but not eliminate, concern about transport for the tested arrangement. It would not demonstrate equivalence, because uncertainty and design limitations still have to be interpreted; nor would it establish the method's value in every institution or language. Conversely, an observed difference would show that performance is not invariant across the two conditions, but not which changed condition caused it.
Thus the next study should be planned as a bounded transport test, with its own scope and application. Its claim should attach to the tasks, participants, materials, and conditions it actually examined.