37.1Three distinct tasks
A system built around a single material need still faces three evaluation tasks, distinguished by the inputs they accept and the outputs on which they are judged. Calling all three "compression" conceals which comparison is actually being made.
TaskD is discovery. Its input is the source material; its output is a set of candidate questions. In the closing-time case already described, the eligible output is the questions about closing time themselves, not their answers. Success is measured by whether genuine, answerable questions surface. Recovering exact alternatives is neither required nor scored here. The presence of a question is necessary for any later stage but is not sufficient to show that it is worth pursuing.
TaskR is deliberative representation. Its input is the questions already discovered; its output is their exact alternatives together with the material conditions attached to them. Success is fidelity: a representation succeeds only if the alternatives it presents are those that are available and their conditions survive intact. An alternative may be reconstructed accurately yet prove peripheral to the question, and a central alternative may be dropped entirely. Fidelity is therefore necessary for sound deliberation without being sufficient for it.
TaskP is decision preparation. Given the committee's stated remit and a faithful set of alternatives, the output is a brief. Success is judged against that remit: does the brief supply what this committee needs in order to decide? Strong taskR work does not by itself guarantee an adequate brief, and a weak taskR record caps what any brief can responsibly offer.
When comparing two representations or encodings, then, the task and inputs must be held fixed — same questions, same alternatives, same remit. Suppose one encoding preserves conditions that another omits. That difference counts only because taskR or taskP demands fidelity; measured under taskD, the same pair may be indistinguishable.
The case separates three operations whose outputs differ in kind. TaskD asks which questions about closing time deserve attention; TaskR requires the exact alternatives and material conditions; TaskP produces a brief for a committee with a stated remit. Because these are different acceptance conditions, a result on one cannot stand in for a result on another.
This creates a limit on scoring. A system may identify a strong set of questions while recovering their alternatives imprecisely, or may recover alternatives exactly yet prepare a brief that does not answer the committee's remit. Those outcomes are possible, not inevitable: one system might handle all three well. But a benchmark must not combine the outputs into a single score unless it states the weighting and the comparison being made. A discovery score, for example, compares question outputs for fixed inputs and source materials; it says nothing about exact representation or decision preparation. If the inputs vary, an apparent gain may reflect easier inputs rather than a better encoding.
Necessary and sufficient conditions therefore have to be kept apart. Recovering the exact alternatives is necessary for faithful TaskR performance, but it is not sufficient for TaskP: adequacy of a committee brief is measured against the stated remit. Conversely, if that remit requires the alternatives and material conditions to be presented, then their inclusion becomes a condition of preparation; this is a conditional requirement, not a property of every brief. Success on TaskD is not sufficient for TaskP, and, as the tasks are defined, it is not required as an input to it. Application should report task-qualified results, with each score attached to its task, inputs, and output test.
37.2Eligible sources and access limits
Eligibility and access must be settled before a test asks any output to preserve information, because the two conditions do different work. A source version is eligible when it is admitted to the input set for the task; an arm has access when it can actually inspect that version. In the proposed comparison—direct source reading, an ordinary edited summary with source links, and the proposed richer record—each arm receives the same eligible source versions and the same access, unless access is itself the declared treatment. That stipulation is a prospective design, not a reported result.
Eligibility is necessary but not sufficient for a preservation claim. An item may be eligible and yet be absent from the version an arm sees, or present but beyond what the arm's output format can carry. The set of target distinctions must be fixed before scoring begins and must not be narrowed merely because a given arm's output format cannot record them; excluding such items would conceal the loss being tested. Only items that are eligible and accessible can count toward a preservation score, and the set of such items must also be fixed before scoring begins; settling it after outputs are seen would change what the score measures. Access restrictions should be reported separately rather than folded into the denominator. What the score compares accordingly depends on what is held fixed. If task, input versions, and access are constant, a preservation difference bears on how each arm carries information. If access varies as the declared treatment, the same difference bears on access and method jointly, and does not isolate the record's contribution. Conditionally, then: were all three arms to inspect one version and only one to record audience or time, the divergence would belong to recording choices rather than to access.
Following the documentation approach proposed by Bender and Friedman (Bender and Friedman, 2018), the present protocol records what its decision actually needs and marks missing context. Because it is a documentation proposal rather than a measured guarantee, the task-minimal eligibility rule adopted here remains an editorial choice. That rule names, for each item, the admitted version, the arms permitted to inspect it, and whether the output format can hold the distinction at issue.
Once eligible versions and access conditions are fixed for the comparison, the next step is to separate unavailability from infidelity. If a distinction is absent from every supplied source version, then no output reflecting those versions can be expected to preserve it. That is a missing input, not a preservation failure. Scoring the output more harshly cannot manufacture the distinction retrospectively, and it says nothing about which encoding handled available distinctions better. The record should instead mark, for each distinction at issue, whether it was present in the supplied input, present but in a version the arm could not access, or present and accessible yet altered or merged in the output. Only the third bears on preservation behavior.
Non-recovery becomes inevitable only where no supplied source carries the distinction at all; where an accessible source elsewhere does, absence from one version is not yet unavailability. In the proposed illustration the three arms receive the same eligible source versions and the same access unless access is varied as the declared treatment. Equal access is necessary for attributing differences to the encodings, but not sufficient: an arm may still leave untouched a distinction that was present and accessible. Availability thus supports preservation without guaranteeing it.
The second condition concerns what a score actually compares. Conditional supposition: if retention is measured only among distinctions that reached an arm's input, the quantity is a rate of retention given opportunity; if measured across all distinctions in the underlying sources, it also charges the arm for what eligibility and access excluded. The first question concerns encoding; the second mixes encoding with access. Naming the quantity accordingly prevents a missing-input outcome from being read as an encoding defect.
37.3The input unit
The input unit is the smallest collection over which a task's output is defined, and it differs across the three tasks in the closing-time illustration. For taskD, discovery is directed at a contributor's inquiry as a whole: the unit is a collection of related questions bound by a shared task, because the sought output is a set of questions rather than an answer to any one of them. For taskR, recovery attaches to one question at a time: the unit is a single question together with the alternatives and material conditions that belong to it, since the required output is meaningful only relative to that question. For taskP, the unit is a collection assembled under the committee's stated remit; the brief is a product of that collection, so no individual question within it defines the output.
This yields a qualification: the three tasks do not share a natural unit, and imposing one across them distorts at least two. Supposing the single question were fixed as the unit throughout, discovery would have no defined output, since there is as yet no question to take as input. Supposing the collection were fixed instead, recovery would report success at a level at which the required exactness is not specified. These are possible distortions, not inevitable ones; a task could be defined so that a larger or smaller unit fits, but then it is a different task, with different success conditions.
Matching the unit to the task is therefore necessary for interpreting any measure, but it is not sufficient: the criteria of success still have to be stated. In taskP, for instance, the length of a brief measures nothing until the collection covered by the remit is fixed. Nor does a good result on one unit license claims on another: success in discovery does not define recovery, and recovery does not by itself make a brief suitable for the committee. A measure taken at the wrong unit thus answers a question the task did not pose.
Changing the input unit changes the meaning of omission and duplication, so any comparison must hold that choice fixed. When the unit is a single question, a further question about the same topic lies outside the frame and its absence is not an omission. When the unit is a contributor's related questions, the same absence is a gap in the declared set. Duplication moves in parallel: repetitions that fall in different single-question units are not double counting within any one of them, while repetitions inside a shared-task collection are exactly what a duplication measure should catch. The unit therefore determines which errors are countable at all.
The distinction can be read through the three tasks already in view, taken as a prospective design. For taskD, the unit is whatever set of questions discovery is meant to find; for taskR, it is the question whose exact alternatives and material conditions must be recovered; for taskP, it is the collection addressed to the committee's stated remit. Success on one does not define success on the others, because each task admits a different omission relative to its unit. A system that finds every relevant question may still recover a truncated alternative; one that recovers conditions exactly may still assemble a collection that falls outside the remit.
A caution about comparisons follows. When two encodings are compared, the task and the input unit must be held constant and only the encoding allowed to vary; otherwise a coverage or compression figure compares different selections, not different representations of one selection. Even then, unit match is necessary for a fair comparison but not sufficient for task success. Nor does an unfixed unit inevitably produce a misleading result; it makes the result depend on a choice the figure does not record, so readers cannot tell which kinds of omission were counted.
37.4The expected output object
Evaluation requires a declared output object: the set of fields the system must return for a specified task, given a specified input unit. Once task and input are held fixed, the object has four components. The first is a task-specific summary, the content the reader will actually use. For TaskD that summary is the set of questions raised about closing time; for TaskR it is the exact alternatives and the material conditions distinguishing them; for TaskP it is the material the committee needs under its stated remit. The second component is the set of relations required by the task, such as which alternative answers which question, or which condition qualifies which claim. The third is uncertainty, attached to the particular item that carries it: whether a question is genuinely open, whether an alternative is reconstructed rather than stated. The fourth is retrieval pointers, linking each element to the region of the input unit from which it was drawn so that a reader can check the derivation.
These four components are necessary for the object to be usable, but they are not sufficient for success. A system may return all four fields and still misstate a relation, so field presence cannot substitute for item-level correctness. Nor does success transfer between tasks: an output satisfying the TaskD specification says nothing about adequacy for TaskR or TaskP, because the summaries and relations demanded differ.
This matters for any quantity reported alongside the outputs. A figure labelled completeness must say what it compares: coverage of the required relations for one task against one input unit, not coverage of everything the unit contains. Conditional on holding task and input unit fixed, two encodings of the output object may be compared directly; if either varies, the comparison no longer isolates the encodings.
The claim that a richer output is better needs qualification: adding fields is neither necessary nor sufficient for usefulness. A short object can meet a need, and a long one can bury the answer. The relevant criterion is whether each field answers a reader need fixed by the task. This applies to TaskD, TaskR, and TaskP without redefining their aims. Discovery of questions, recovery of exact alternatives and material conditions, and preparation of a committee brief with a stated remit are distinct, so success on one does not establish success on another.
The qualification matters when judging an encoding. Compare encodings while holding the task and inputs fixed; otherwise the comparison confounds presentation with scope. Suppose one representation carries retrieval pointers and another carries uncertainty notes. Those fields are comparable only if both address the same reader need for the same task. If TaskR is the task, a pointer that locates exact alternatives may support recovery, while a free-form caution may not. That is a conditional inference from the task definition, not an observed result. For TaskD, pointers could help a reader find further questions, but adding them does not by itself improve discovery; the question list must still be discoverable. For TaskP, a field qualifies only insofar as it serves the committee's remit.
Thus the expected output is not defined by field count or by the presence of relations, uncertainty, or retrieval pointers in the abstract. Each addition must be justified by the need it serves, and the absence of a field is not automatically a defect. A richer object is better when its fields are eligible for the stated task and reader, not merely when it contains more of them.
37.5The comparison baselines
Because the record's shape is the declared treatment, the comparison arms fix the eligible source versions and the reader's access to them; what differs is only the encoding offered for the task. The prospective design — an invented illustration rather than a reported study — proposes three arms: direct source reading, an ordinary edited summary carrying source links, and the proposed richer record.
These arms do not ask the same question. Direct reading establishes what unaided access achieves, so a comparison against it shows only that some assistance is better than none. That result is necessary if the system is to be justified, but it is not sufficient: an ordinary summary with links might yield the same gain, in which case the extra fields have not earned their place. The contrast that isolates them is the one against the summary-plus-links arm. If the richer record outperforms there, the added structure can be credited with the difference, provided the task and the eligible inputs are held fixed across arms.
The third arm sharpens the test. Ordinary annotated editing already combines an editor's compression with comments and pointers back to the sources, making it the strongest of the simpler alternatives. A richer record that merely matches it has not shown that its additional apparatus contributes anything. Yet falling short of annotated editing would not, on its own, prove the apparatus worthless: the fields may simply be mismatched to the particular task the reader faces.
Finally, the named quantity must be specified before the comparison means anything. Time to a correct answer compares efficiency conditional on accuracy; agreement with a reference answer compares output quality. Neither compares the faithfulness of a record to its sources. Extra fields could also slow a reader or introduce inconsistencies among them, so the advantage is possible rather than inevitable, and it depends on whether each field answers a reader need.
A comparison baseline is defined by what it is given as well as by what it is asked to produce. In the illustration already introduced, the richer record and its comparators should receive the same eligible source versions and the same access to them, unless differential access is the treatment being studied. This is a necessary condition for a fair comparison, but not a sufficient one: equal access does not by itself make any observed difference attributable to the record’s structure, since the task, the eligible inputs, and the preparation each arm performs may still vary.
The rule matters because withholding source access can manufacture a baseline that is weaker for an incidental reason. A comparator that cannot reach a relevant passage may lose because of access, not because of its summary format or its support for deliberation. Such a result would establish only that restricted access is inferior in the tested setting. It would not show that the proposed richer record is superior when both arms can inspect the same material.
The named quantities must therefore be compared under a fixed task and fixed inputs. “Performance” for an encoding is otherwise a comparison of different problems: one arm may be solving discovery, another representation for deliberation, and another preparation for a decision. Once those tasks and eligible inputs are held constant, differences among encodings become interpretable.
Even then, the outcome is possible rather than inevitable. Under access parity, the richer record may help when a decision requires tracing source support, while a simpler format may suffice when the task is only to obtain an overview. The defensible claim is conditional: given the same sources and the stated task, one encoding can support a particular step better. It is not that the richer record must prevail across all tasks or all input sets.
37.6Held-out material and leakage
A frozen design earns an evaluation only against material that played no part in producing it. Cases used to revise the method are development material: once a case has influenced a choice of prompt, procedure, or success criterion, it can support debugging and argument, but not a claim about performance on material the design has not met. This manuscript makes no such claim. The toy cases already used here illustrate reasoning and design choices; none is offered as a held-out validation result.
Separation therefore has to be specified rather than assumed. An untouched evaluation set needs its own definition before evaluation begins: which cases are eligible, what inputs the system receives, what outputs count as answers, and which version of the design is frozen. Without that specification, held out names an intention rather than a property of the evidence, and a later reader cannot tell whether any instance belonged to the design phase.
Holding out case text is necessary but not sufficient. Design choices can carry information from development cases forward even when those cases never reappear: a success criterion selected after inspecting particular failures, or an illustrative framing tuned to particular inputs, embeds what those cases taught. The evaluation then compares a design that has already absorbed the development material against new material, while the task and inputs are held fixed; what varies is the cases, not the problem statement. Leakage here is not inevitable simply because development and evaluation draw on the same domain, but the risk is systematic wherever selection and evaluation use the same episodes.
Where a case has been used in both roles, its evidential weight belongs to development. As a supposition rather than a reported result: if the eligible set were specified and fixed before the design was frozen, a reader could check for each case whether it influenced any choice.
The relevant question is not only which cases were used to construct the system, but which cases have influenced its revision. If a failed evaluation case is used to revise the prompt, the case has crossed into development material. Its later success is development evidence: it shows that the revision answers a difficulty already seen and acted upon, not that the design handles cases it has not yet met. The contamination follows the use, not the content. A single failure that motivates an edit is enough to disqualify that case from confirming the revised design.
Because the revision is conditioned on the failure, re-scoring the same case compares a tuned design with a target it was built to address. To attribute any difference to the revision, the task and eligible inputs should be held fixed; only the design varies. Even then the comparison is internal to development.
Not every failure crosses this line. Suppose a failed case is inspected but no change follows. If the inspection was used to choose among candidate designs, or to decide which design to keep, the case has still informed the choice and belongs to development. If it was recorded and set aside without affecting any design decision, it may be preserved for later evaluation — provided that record is documented and the case remains untouched thereafter. The distinction is therefore between cases that informed a decision and cases reserved from it, and it must be recorded at the time rather than reconstructed after results are known.
What the revised design needs is a fresh evaluation set: material specified in advance and not used to revise the method. The toy cases already used to illustrate the argument in this manuscript are not offered as held-out validation results, and the scope of any claim about the revised design extends only to whatever task and inputs such a set is drawn from.
37.7Claims a benchmark can and cannot support
The eventual comparison is bounded to one domain and three tasks, and its claims should be read at that level. The domain is the bounded consultation workflow used throughout the method: a case record as the unit of analysis, running from raw case material to a prepared decision item. Within that domain, the three stages are separate tasks rather than stages of a single score. Compression serves discovery: its eligible input is the case record as supplied, and its output is reduced material that a reviewer can inspect for candidate issues. Representation serves deliberation: it takes a fixed set of case material produced or selected at the discovery stage and outputs a structured account on which argument about the case can proceed. Preparation serves a decision: it takes the deliberated account and outputs a bounded recommendation or options set tied to a stated question. The eligible inputs and outputs are part of each task's definition; a run that substitutes one stage's output for another's input is a different task and cannot be scored within the same comparison.
What the eventual comparison therefore supports is a statement about relative performance on these defined tasks under these defined inputs. It does not, by itself, show that success at one stage is sufficient for the next. A faithful compression is necessary material for deliberation, yet it can be faithful while the deliberation step fails on the same case; conversely, a strong prepared output can issue from an inadequate representation, so quality at the decision stage does not certify the earlier stages. Each task accordingly needs its own measure, and the comparison must fix the task and the case material while allowing the encodings to differ; otherwise variation in encoding is confounded with variation in what the method actually does.
The cases already used to revise the method remain development material. Until an untouched evaluation set is specified, none of those toy cases counts as held-out validation.
A synthetic benchmark earns its keep by isolating a single mechanism: when one input feature changes and the relevant context is held fixed, a difference in output can be traced to that mechanism rather than to confounded variation. That is an existence claim, and it is enough to establish that the mechanism is possible under the specified construction. It does not license the stronger claims. Possibility in an engineered setting is not prevalence in natural consultations, where the feature may be rare, differently distributed, or entangled with factors the benchmark suppresses; and prevalence, even if known, would not by itself establish broad institutional benefit, which depends on how outputs are used, by whom, and toward what ends.
The distinction matters for how results are worded. A benchmark result supports the claim that the mechanism can produce the effect when the specified conditions hold. It cannot support the claims that the effect is common in practice or that adopting the method improves outcomes overall. Those require evidence about the distribution of natural cases and about consequences beyond the benchmarked task. Nor is mechanism exposure sufficient for a claim of inevitability: the effect may be possible yet fragile, disappearing when a feature the benchmark fixes is allowed to vary.
There is a further qualification within the benchmark itself. Cases used to revise the method are development material, whatever their form; the method has already seen what they test. An untouched evaluation set therefore still needs specification — its selection, size, and eligible inputs and outputs — before any of the manuscript's already-used toy cases can count as a held-out validation result. None is claimed as one here. Without that separation, even the narrow mechanism claim rests on material shaped by the method's construction.
Finally, a benchmark count must be read as a comparison: it compares performance across the encoded variants on the fixed task and inputs, not across domains, institutions, or the encoding choices themselves.