39.1Claim support
Support review asks a bounded question of each expressed claim: given the supplied evidence and a stated interpretation of it, does the claim follow? Min et al. (2023) (Min et al., 2023) evaluate support over smaller factual claims instead of assigning one binary label to a mixed long answer. The change of unit matters because an overall verdict cannot say which claims the evidence carries and which it merely permits.
The interpretation must travel with the judgment. If two readings of an ambiguous sentence are both live, a claim can follow under one and remain unsupported under the other. Support is then possible under a stated interpretation, not inevitable across readings, and a verdict that records only "supported" hides which reading did the work. Recording the interpretation also makes comparisons interpretable: hold the task, the claim set, and the supplied evidence fixed while the encoding changes from one verdict on a mixed answer to separate verdicts on smaller claims. Otherwise a difference in results may reflect different inputs rather than a difference in how support was represented. Where the evidence only makes a claim plausible without warranting it, the review records the case as unresolved rather than supported.
In the prospective rubric, this review is confined to expressed claims; concern coverage and dissent review answer their own questions. Being supported on the supplied evidence is sufficient for a claim to pass this check, though it is not necessary for the claim to hold, since evidence outside the set supplied may settle matters this review cannot reach. The gain from evaluating smaller claims is therefore diagnostic: a reviewer can see where support breaks rather than only that a mixed answer failed.
Support review answers a narrow question: does each evaluated claim follow from the supplied evidence under a stated interpretation? That judgment is internal to the materials handed to the editor. It relates claims, evidence, and interpretation; it does not, by itself, reach beyond them.
Two limits follow, and they should not be conflated. First, evidential support is necessary for a claim to count as faithful on this dimension, but it is not sufficient for external truth. Suppose the supplied evidence is itself false, distorted, or drawn from a source that misdescribes the matter at hand; in that case a claim may be perfectly supported and still fail to describe the world correctly. Conversely, an unsupported claim is not thereby false; it is merely not licensed by the evidence as given. Support compares a claim with its stated evidential basis, not a summary with the world.
Second, support establishes nothing about the completeness of the selected claims. A summary may offer a short run of claims, every one well supported, while leaving aside distinctions that the task makes relevant. The distinction here is definitional rather than empirical: in the prospective rubric, claim support concerns expressed claims, while concern coverage concerns task-eligible source distinctions, and dissent review concerns the reason and condition of disagreement. Passing one judgment therefore does not pass the others, and failure on one does not entail failure on the rest.
This matters when quantities are reported. A support rate compares the supported claims among those evaluated with the number evaluated; it does not measure truth, coverage, or representativeness. A summary can thus score well on support while remaining partial, and the gap is possible rather than inevitable — it appears precisely where task-eligible distinctions go unselected. The next question is therefore whether those distinctions survive in the summary at all.
39.2Distinctive concerns
A coverage judgment follows claim support rather than replacing it. Claim support asks whether the summary’s expressed claims are grounded in the source; coverage asks whether the source’s distinct concerns relevant to the task remain recoverable from the summary. A concern counts as distinct when losing it would erase a task-relevant difference rather than merely a different wording of the same point. Recoverability is therefore the operative test: an editor working from the summary should be able to say which concern is at issue and what condition or reason accompanies it. For the prospective rubric sketched here, these are separable checks: passing one does not establish the others.
The recoverability test needs a qualification. Recovering a concern is necessary for representation, but not sufficient: the summary must also keep the concern attached to its proper condition. A summary that lists both a recommendation and its stated reservation has covered two concerns only if the reservation is still recognizable as the condition qualifying the recommendation. Otherwise two labels survive while the relation between them is lost. Conversely, an omitted concern does not make a later judgment wrong of necessity; it makes the error possible whenever the task turns on that distinction. If the task does not, omission may be harmless.
When two summary encodings are compared, the task and source set must be held fixed. A prose summary and a tabular summary may recover different numbers of concerns, but that comparison concerns the encodings’ recoverability under the same inputs. It does not compare truth, completeness outside the task, or reader preference. As prospective design rather than an empirical finding, the rubric states a test to apply to a given case; it does not report how any summary has performed.
The coverage judgment presupposes a set of task-eligible distinctions; the open question is where that set comes from. The boundary is not given by nature. An editor must decide which source distinctions the task makes relevant. In the prospective rubric, claim support addresses expressed claims, concern coverage addresses task-eligible source distinctions, and dissent review addresses the reason and condition of disagreement. Each domain already presupposes an eligibility rule, and a reasonable reader who drew that rule differently would not thereby be confused.
Contesting the boundary does not make it arbitrary. The answer is an adjudication note: record the eligibility rule actually used, the grounds for it, and at least one plausible alternative drawn more narrowly or more widely. The note keeps the denominator visible. A coverage proportion over one eligible set is not the same measurement as the same arithmetic over another; the two may agree completely on which distinct concerns remain recoverable and still report different figures, because the compared populations differ. Fixing the task and the source inputs does not by itself fix the quantity being reported. A note of this kind is necessary for the boundary to be checkable; it does not make the adopted rule correct.
The note also supports qualification rather than reopening. When the boundary dispute is substantive, report the result under the adopted rule and state whether the alternative rule would change the conclusion, instead of merging the two populations silently. If the alternative changes nothing, the dispute is real but immaterial to this task; if it changes the finding, the finding is conditional on the adjudication and should be labeled as such.
A falsification condition follows: the note fails if, for the same task, inputs, and recorded rule, two applications to one source set yield different eligible sets with no stated reason. Then the boundary has not been fixed at all.
39.3Disagreement preserved
A dissent-preservation judgment does not ask whether a summary contains a sentence reporting that someone disagreed. It asks whether the summary preserves the reason for the disagreement and the condition under which it holds. The difference matters because a bare report of dissent records only that conflicting judgments exist. It leaves a later reader unable to tell what the dispute is about, which sources support which side, or what would have to change for the dispute to dissolve. In the prospective rubric already sketched, dissent review is therefore kept separate from claim support, which concerns the claims the summary expresses, and from concern coverage, which concerns task-eligible source distinctions. Each judgment fixes its own object.
The countercondition is easy to miss. A summary may mention dissent prominently and still fail preservation: it can name the parties, or even the topic, while dropping the reason that generates the conflict or the condition that limits it. Preservation is thus not guaranteed by the presence of dissent language; conversely, disagreement can survive in compressed form if the reason and its condition travel together. Whether either feature is sufficient for a usable account is another question, since a retained reason may still be weak, misattributed, or conditional on a scope the current task does not use.
This distinction also disciplines any count of preserved dissent. A tally of dissent mentions compares surface occurrences across two summaries of the same fixed inputs under the same task; it does not compare how much of the reason-and-condition structure each encoding retains. The encodings themselves may differ in wording and length, so an equal count can accompany unequal preservation. Suppose, as a conditional case, that one summary records a disagreement only as unresolved, while another records what is disputed and under which assumption. The count is equal; the preservation judgment should differ.
Whether a retained reason then deserves equal evidential weight is a separate step.
Once a dissent is preserved with its reason and with the condition under which it holds, a further question remains: what force does that reason carry in the summary? Retention and weight are separate judgments. Keeping a reason in the record does not make it equal in evidential weight to the claims it opposes, and it does not give the dissent decisive authority over the conclusion. The prospective rubric keeps the checks apart for exactly this reason: none of them, as stated, assigns weight.
This distinction blocks two opposite failures. Drop the dissent and the summary misrepresents how far agreement extends. Present the retained reason as carrying equal force when no assessment has been made, and the summary asserts more than the record supports. Preserving a reason is necessary for representing the disagreement that was actually present, but it is not sufficient for deciding anything: what the reason can do depends on whether its condition obtains and on whether the claim it addresses is among those the summary undertakes to report.
The scope of application follows. A preserved reason bears on a particular claim under a particular condition; where that condition is absent, the dissent is possible pressure on the conclusion, not inevitable defeat of it. Where the condition is met, the dissent is relevant to the weight of that claim, even though the summary need not adopt the dissenting position. If the editor cannot assess the weight on the available record, that judgment stays unresolved: the summary records the reason and its condition without adjudicating their force. Thus fidelity to disagreement does not require flattening the evidential landscape, and a conclusion may still be stated provided it neither suppresses the dissent nor silently upgrades it into an equal or overriding authority.
39.4Readability without simplification of stakes
Readability review asks whether intended readers can follow the argument with its material qualifications intact. This is a judgment about the relation among a text, a reader group, and a reading task: can these readers trace which claim follows from which reason, and can they see what each qualification limits? Readability so understood is not the same as brevity, and neither is the same as support. A summary may read smoothly while omitting the condition that makes a recommendation actionable, and a summary may be laborious while preserving every distinction that matters to the decision. The review therefore has to name the qualification it is tracking and ask whether the reader can still perceive its scope and force.
Retaining a reason does not by itself give that reason equal evidential weight, so readable presentation must also make differential weight followable. Usability criteria should be declared rather than asserting readability as a universal necessary condition for representation: readability with qualifications intact is necessary for a usable summary to represent its inputs, but it is not sufficient. A text can let readers follow every qualification and still misstate what a source supports; those judgments remain independent, and where support is unknown it should stay unresolved.
The pairing matters when two encodings are compared, with the task and inputs held fixed; otherwise a difference in apparent readability may reflect a changed deadline or a narrower audience rather than the wording itself. Conversely, shortening is not a necessary route to legibility, and an omitted deadline is not an inevitable casualty of compression — it is a loss whenever that deadline is material to the task at hand. A named readability quantity, for instance one scoring sentence length or clause depth, compares surface features across texts; it does not compare whether qualifications survived, so it cannot substitute for this judgment. Whether any particular wording lets a particular reader group follow the stakes remains an open empirical question for that pair.
Shorter wording can make an argument easier to follow while silently changing what the reader is being asked to do. Those are separate properties, and an evaluation should measure them separately.
Consider, as a hypothetical illustration rather than an observed case, a report that states a submission deadline. A summary that drops the date may be easier to read: fewer specifics, smoother sentence, faster uptake. Yet the omission changes the task. A reader following the summary may prepare a timely submission or an untimely one; the shorter version no longer settles which. Readability has improved while the instruction's scope has been altered. High readability is therefore not a sufficient condition for faithful representation, and low readability is not sufficient evidence that stakes have been preserved. Nor is readability necessary for every purpose: a dense specification may serve an inspection task well even when it strains ordinary reading.
The same separation applies to the direction of the inference. A summary that retains the deadline is still only possibly adequate: it must also preserve who must act, by what authority, and under what qualification. Retaining one omitted condition does not guarantee that all material conditions survive. Conversely, an omission is possible without being inevitable in effect; whether the reader's conduct changes depends on what the full task requires and what other cues remain available.
This calls for distinct judgments. Ask separately whether intended readers can follow the argument, whether each material qualification and distinction remains stated, and whether the resulting claim still matches the input at the level of support being asserted. A readability judgment answers only the first question. Support checking, as noted earlier, asks whether the claim is warranted by what was supplied; it cannot be substituted by a fluency rating.
Because the case here is a design illustration, the point is methodological: define the comparison before scoring. Fix the reader, the task, and the inputs; then vary the encoding. Otherwise a measured difference may reflect a changed assignment rather than a clearer presentation.
39.5Finding the source again
Retrieval merits its own judgment, distinct from the judgments of support and readability already defined. Fix the task this way: given a claim in a summary and the reference offered for it, the reader must reach the exact source version and the relevant passage within it. Only when that has been done can inspection of the claim begin. Finding the source again names the completion of this task, and nothing less. Identifying a work, or even a section of it, is not yet supplying the version and passage the reader needs.
This definition settles what the retrieval judgment records. Successful retrieval is a necessary condition for inspection: without the passage in hand, the reader cannot compare the claim against what its source says. It is not a sufficient condition for support. A located passage may fail to warrant the claim, or warrant it only in part, and those questions belong to the support judgment rather than to retrieval. Conversely, failure to retrieve does not entail that a claim is unsupported; the source might be reached by another route. The summary has nonetheless failed the retrieval task as posed, and that failure should be recorded as such rather than converted into a verdict on the claim.
The task and its inputs must stay fixed when two ways of recording sources are compared. A retrieval measure compares completions of one task against one set of available sources. If one encoding is credited with finding a document while another is credited with finding a passage, the two quantities answer different questions, and their comparison shows nothing about the encodings themselves. The encodings, and the effort each demands, may vary; what counts as success may not.
Finally, passage and version can come apart. Whether a passage located in one version suffices to inspect a claim drawn from another is a question of version alignment, and it remains unresolved whenever the relevant version is unknown.
A link check is a transport result; the reader's task is to land on the cited version and the passage bearing on the claim. These two can come apart. A check that records whether an address responds compares that address against a reachability criterion. It does not compare the summary's claim with any passage the reader can find, nor does it establish that the document behind the address is the version cited. Against the retrieval task, a working hyperlink is therefore at most a precondition of one route to the source, and not a sufficient condition for completing that route. It is not a necessary one either, since a reader might reach the passage by search or from a held copy.
The countercondition keeps this from hardening into a blanket rule. A link to a long document does not inevitably defeat retrieval: if the document is stable, the summary names a passage locator, and the document's structure lets the reader navigate to it, the same offering may serve. What varies is not whether the link works but whether the reader's task finishes inside the material offered. To compare encodings honestly, hold the task and the inputs fixed — the same claim, the same cited version, the same requirement to find the relevant passage — and vary only what the summary supplies: a bare address, an address with a passage locator, or a copy held at a stated version. Each may pass or fail retrieval, depending on document length, stability, and navigability; which encodings fail how often is not settled by this illustration.
Apply this as a judgment of its own, kept apart from claim support. Support asks whether the passage actually located warrants what the summary says. Retrieval asks whether the reader can reach that passage and version at all. The second does not establish the first: a reader who recovers the exact source may find it says something else, while a reader who cannot find it has no basis yet to check either way. An editor can thus record the link as live and the retrieval task as unfinished, and leave the support question unresolved until the passage is in hand.
39.6Recovering after a correction
A correction task supplies a changed premise and then inspects whether the change reaches the derived interpretations that depend on it. In the closing-time illustration, the supplied time moves from 18:00 to 19:00 and the relevant maintained summaries are inspected for interpretations that draw on that time. The task therefore observes propagation along a specified path. Since no human ratings or composite score have been collected, a run of this task supports judgments about whether the changed premise reached a dependent interpretation, and leaves support, distinctions, dissent, readability and retrieval unresolved.
Reading the result requires an expectation fixed in advance. An unchanged interpretation counts against recovery only where it was expected to change; where a derived claim rests on premises the correction leaves alone, no change is required, so an unchanged passage is consistent with both handled and missed propagation. The converse inference also needs care. Seeing the new time in one dependent interpretation establishes that propagation through that path is possible, and shows at most that path; it does not settle whether other dependents were updated. What the corrected-interpretation count compares is the number reached against the number expected to change, not the summary's fidelity in general. Passing the test on one dependent is thus sufficient to demonstrate that the route exists and insufficient to certify the maintained set.
To compare how two maintained summaries encode the same dependent interpretation, hold the task and the supplied inputs fixed, including the 19:00 premise; the encodings may differ in form while representing the same change, so a difference in wording alone is not a propagation failure. Conditional on the editor recording beforehand which interpretations were expected to change, that record supplies the benchmark for the test; without such a record the outcome of the correction remains open rather than settled.
The closing-time case fixes a scope question. Changing 18:00 to 19:00 touches one supplied premise; the maintained summaries hold both a record of that premise and conclusions drawn from it. A timestamp field belongs in scope because it records the premise, while the obsolete conclusion belongs in scope because its support runs through it. Treating the timestamp edit as the whole correction conflates two distinct objects: the record of an input and the interpretation built upon it.
A necessary condition for recovery of a particular derived claim is therefore that the revised premise reach it: the claim is re-derived, amended, or withdrawn. That condition is not sufficient for the summary as a whole. Other claims may depend on the old premise indirectly, while yet others rest on premises this correction did not touch and must be judged separately. Nor does the case license the stronger claim that stale conclusions inevitably survive; it shows only that a record update leaves them in place unless some further step revises them.
Scope can accordingly be described by support relations rather than by an arbitrary list of text spans: the correction's scope covers the claims whose support passes through the changed premise, together with the fields that record it. Application is then local but still discriminating. Each claim in scope must come out updated, withdrawn, or explicitly unresolved. In the case as designed no human ratings and no composite score were collected, so nothing supports calling one maintained summary better overall than another; the observation is confined to whether the changed premise reached the relevant derived interpretation.
A further comparison must hold the correction task and its supplied inputs fixed. Two encodings of a summary can be compared for this correction only under the same premises and the same target claim; otherwise a difference in outcomes may reflect different inputs rather than different representation.
39.7A predeclared rubric and adjudication record
The rubric should be fixed before any summary is scored, and before its competitors are inspected. Predeclaration is not sufficient for a sound evaluation, but it prevents a dimension from being widened or narrowed after its result is known. Three things must therefore be settled in advance: what each judgment compares, which inputs are held fixed, and what counts as no judgment at all.
The dimensions are judged independently. Claim support asks whether the claims a summary expresses are supported by the inputs; it does not ask whether the underlying source is itself correct. Concern coverage asks whether task-eligible distinctions in the sources survive into the summary; distinctions irrelevant to the task do not count against it, and distinctions the task requires cannot be treated as optional. Dissent review asks whether the reason for disagreement and the condition under which it holds are represented; recording that a minority exists, without its reason or condition, is not enough. Readability, retrieval and revision are separate again: retrieval concerns finding what is there, revision concerns whether a superseded conclusion is still carried forward, as the correction case already showed.
Two further declarations matter. First, the task and the input set stay fixed across encodings. If, hypothetically, one summary were judged against a larger source set than another, the resulting gap could be misread as a difference in representation. Second, a judgment may be unresolved. A missing passage, an ambiguous claim, or an unverified condition is recorded as unresolved rather than converted into a pass or a failure.
The adjudication record carries one entry per judgment: the dimension, the object compared, the inputs held fixed, the evidence cited, the decision or its unresolved status, and the reason. Entries are kept apart, so none is quietly upgraded to stand for another. It is the record, not the recollection of an overall impression, that makes a later dispute inspectable.
The rubric's dimensions have distinct scopes, and those scopes determine what a conflict between them can mean. Claim support bears on claims a summary expresses; coverage bears on task-eligible distinctions present in the sources; dissent review bears on the reason and condition under which sources disagree. Because each dimension answers a different question, a strong result on one does not supply the judgment another withholds. A summary may be highly readable while leaving a central claim unsupported, or may cover every eligible distinction while misstating the condition on which a dissent turns. These are conflicts, not averages to be dissolved.
The record should therefore preserve the conflicting judgments side by side, with the evidence each consulted and any judgment left unresolved. Reporting the conflict matters even when one dimension dominates a decision: unless priorities were declared in advance, an overall recommendation that silently trades unsupported claims for coverage or brevity is not reproducible.
Suppose a reviewer treated a high coverage score as evidence that claim support had been checked; the inference would fail, because coverage counts task-eligible distinctions, not verification of the claims expressed. Independent scoring makes such conflict possible in any case, not inevitable in every one.
Nor is passing every dimension sufficient for an unconditional recommendation, though failing a necessary one may block a restricted use. Predeclaration should say which dimensions are thresholds, which can compensate, and for which task. "Overall" then names a weighted or ranked comparison, not a single latent quality. The same discipline applies to any quantity summarizing the rubric: a count of satisfied dimensions compares dimension labels under a fixed task and fixed source set; change the task's eligible distinctions or the inputs, and the count no longer compares the same object. What remains is an explicit account of priorities: which conflicts were accepted, which were blocking, and what the recommendation covers.