21.1Reference overlap
An evidence path earns its value only where the evidence can be read and understood. The same demand applies to a metric: before using one to compare systems, a reader should know exactly what its comparison relates. ROUGE measures overlap between generated summaries and reference summaries using specified units (Lin, 2004). The units might be words or longer sequences, but in every case the comparison holds two texts against one another.
Take a stipulated source that states one fact: the gate is unlocked. One accurate paraphrase says that the gate is not locked. Another says that the lock is disengaged, where "disengaged" is stipulated to describe the same unlocked state. The two paraphrases express the same source fact, yet they use different wording for that stipulated state. Whichever paraphrase is nominated as the reference, the other will look less overlapping than a near-copy of the reference would. The wording can therefore affect overlap even when the stipulated claim is preserved; no particular ROUGE score is calculated here.
The inferential step is therefore modest but consequential. Overlap reports agreement with the wording of a chosen comparator, not agreement with the source from which the comparator was itself written. The reference is selected: it is one faithful rendering among several, and its omissions and phrasing become the shape of the measurement. A low overlap between two accurate summaries is not evidence that either misstates the source; a high overlap with the chosen reference does not establish that the summary is supported or that it covers the source's concerns. An overlap measure might correlate with a quality judgment in a validated setting; it is not a proof of that quality in this example.
Against the purpose of representing questions, this is the dimension that disappears inside the score: whether the question's concern survives, rather than whether its words reappear. Overlap remains the right tool when the task is genuinely to reproduce a fixed reference using stated units. Beyond that narrow purpose it can still be reported, but only as what it is — a defined comparison, not a proof of factuality, source entitlement, or representation.
With the chosen comparator treated as fixed, its limits show up where shared surface material and shared meaning come apart. Compare "The room is open" with "The room is not open." Nearly every word is shared, and the two sentences even overlap around the negated phrase; still, one asserts accessibility and the other asserts its opposite. The exact changed negation — "not" — carries the reversal, and overlap does not by itself register that reversal. No ROUGE, BERTScore or model probability is computed here, and none is needed: the case shows what the measure counts, not how much it would award.
Scope, then: reference overlap registers coincidence of words and short sequences with a reference formulation. It suits a purpose such as checking whether a summary reproduces specified wording. It is a weak instrument, by itself, for representing the questions a source answers, because two sentences may share most of their material while disagreeing about a single decisive operator, whereas two accurate paraphrases may share little wording and preserve the same assertions. What looks like agreement at the level of matched spans is therefore compatible with a change in what is asserted.
Application turns on the purpose being served. If the question is whether each asserted claim is supported, the eligible items are the claims a summary actually makes. If the question is whether the summary covers the source's concerns, the eligible items also include concerns the summary leaves out. A candidate may thus preserve every fact it states and still omit an entire entrance concern. Claim support and concern coverage are distinct checks over the same text. Reference overlap against a chosen account of what should have been said does not adjudicate between them; it reports coincidence with that account, which is useful evidence but not a verdict on meaning, support, or coverage.
21.2Contextual similarity
Overlap has already shown the problem in miniature: whatever comes from matching strings must inherit that blindness. The useful alternative is therefore not a more forgiving cutoff but a different comparison: rather than asking which words coincide, ask how corresponding content is represented in context. Zhang et al. describe BERTScore as comparing contextual token representations instead of relying only on exact matches (Zhang et al., 2020; first preprint 2019). Applied to two invented summaries of a fixed source, this kind of comparison can credit a paraphrase — an ordinary reformulation that reuses little of the source wording — which surface overlap would count as barely related. That is the local gain: semantic likeness becomes creditable even where vocabulary does not survive, so a faithful restatement is no longer penalised for sounding different.
The gain is real but narrow. Similarity between representations establishes likeness, not preservation of the proposition, and in particular not preservation of dissent or of everything a source implies. Reported performance on the tasks Zhang et al. evaluated does not extend to those properties, and nothing in the present comparison measures them. Nor does a similarity judgment decide which source concerns a summary has taken up: one candidate may preserve every fact it expresses while omitting an entire entrance concern, and it can look highly similar to its source nonetheless, because the omitted material has no counterpart to be compared. Support checking and concern coverage therefore have different eligible items, and neither is settled by closeness alone.
This keeps the method in its proper role. Contextual similarity widens what a comparison can recognise, especially paraphrase, without converting resemblance into a guarantee that the assertion, its limits, or its acknowledged worries came through intact. That qualification matters when the same scores are later read as evidence about question representation rather than about wording.
Contextual similarity extends overlap by comparing representations rather than exact strings, so paraphrases that surface measures miss can still receive credit. But this gain has a boundary that matters once the purpose is representing questions: similarity evaluates how close two formulations are, not who put a proposition forward. Consider the stipulated case of "The room is open" set against "The room is not open." The shared vocabulary is extensive, yet the assertion differs. The limit generalizes: even a representation-sensitive comparison that copes well with paraphrase cannot, from closeness of wording alone, certify that the assertion or its owner has been preserved.
A changed speaker makes the point without altering the words themselves. If a fluent summary reports that "one contributor" raised an entrance concern while another formulation attributes the same concern to "the group," the two descriptions may register as highly similar in content and style. Attribution has nonetheless shifted in a way that matters for representing questions: a question about who claimed something is answered differently by each. No computed score is needed to see the mismatch, and none is reported here. The comparison is an invented example designed to separate two dimensions that a single similarity value tends to fuse—the expressed proposition and its source.
The relevant limit is therefore one of scope rather than accuracy. Contextual similarity is well suited to detecting semantic preservation across rewordings, and that is genuine credit that overlap cannot give. It is not equipped to establish provenance. A preference or similarity judgment favoring the altered attribution must not be recorded as an observed correction of the error; at most it indicates which wording a rater or model preferred under that task. Where the representation is required to preserve who asserted a proposition, attribution needs to be checked as its own item, on its own eligible elements. Coverage of omitted concerns remains separate again: a candidate can preserve every proposition it expresses while dropping an entire entrance concern from the source.
21.3Factual consistency
Factual consistency asks whether an output claim is supported by its fixed source, under a stated interpretation. The unit of judgment is the claim as read, not the surface wording of the summary, and the qualification matters: without a declared interpretation, "supported" has no determinate object. Consider the stipulated pair "The room is open" and "The room is not open." Their shared words cannot by themselves guarantee that the assertion has been preserved. An evaluator must first extract the claims, decide what each asserts, and then ask whether the source warrants that assertion. Nothing in the proposed comparison reports a ROUGE score, a BERTScore, or a model probability, and no model leaderboard results.
Attribution is part of the claim, not a decorative detail. If a fluent sentence changes "one contributor" to "the group," the source and the summary disagree about who committed to what. A similarity or preference judgment over those two sentences is still a judgment about resemblance and acceptability; it must not be reported as an observed correction of the attribution error. Consistency checking registers the discrepancy as a support question, whether or not the wording stays close.
This narrowing also sets the limit of what consistency can deliver. The proposed evaluation compares two invented summaries against a fixed source, and it produces neither a measured score nor a ranking of systems. Consistency marks which expressed claims the source sustains; it does not determine whether the summary is good for the purpose of representing questions, nor does it aggregate its judgments into a single order. On binary judgments, the limitations are recognized: factuality reduced to supported or unsupported obscures comparisons across factuality metrics (Pagnoni, Balachandran and Tsvetkov, 2021). Whatever finer distinctions are needed, they must be stated in advance rather than invented after results are in. Coverage of source concerns remains a separate matter, to be taken up next.
Once the unit of support is a claim in the output, the scope of the consistency check becomes clear: it applies to what the summary asserts, not to what it leaves out. That asymmetry matters whenever selection is possible. A candidate can be wholly supported and still fail to represent the source adequately, because support and coverage have different eligible items.
Consider the stipulated case. A source includes, alongside other material, an entire entrance concern and one opposing reason bearing on it. A summary states every fact it expresses accurately but omits the entrance concern altogether. Each claim it makes is backed by the source, so consistency raises no objection to those claims. Yet the omitted concern has no claim through which consistency could test it. The omission is invisible at the level where support is assessed. Wholly supported and materially selective are therefore compatible descriptions of the same output.
The inference is not that consistency is defective. It is that consistency answers a narrower question than coverage: are the asserted claims supported, versus which source concerns have been taken up? The relevant limit follows. A finding of complete support licenses no conclusion about recall of source concerns, and a finding of full coverage licenses no conclusion that each expressed claim is correctly supported. Treating either verdict as the other would collapse dimensions that this evaluation is meant to keep apart. The earlier example of shared words between "The room is open" and "The room is not open" showed that surface agreement cannot establish support; here the pressure runs the other way, showing that support cannot establish completeness.
Whether an omitted opposing reason is acceptable depends on the purpose assigned to the summary and on what a human rater is asked to judge. That question requires task instructions of its own.
21.4Human judgments and task instructions
Once consistency and coverage are kept apart, the same discipline applies to human ratings. A rater given two invented summaries and a fixed source may be asked whether a summary is good, but that broad instruction leaves the criteria partly implicit. A more specific task identifies which defects the rating should address. If the instruction is to check claim support, the pair "The room is open" and "The room is not open" becomes the salient evidence, because shared wording does not preserve the assertion. If the instruction is to assess coverage of source concerns, a candidate that preserves the source's factual assertions but omits its question about the entrance is the one at issue. If the instruction is attribution, the sentence that changes "one contributor" into "the group" is what counts. The source itself does not choose among these instructions, and a rating assembled from an unspecified request leaves the choice tacit.
The evidence made available to the rater narrows the judgment in a second way. A rater who sees only the two candidates cannot verify the attribution shift; a rater who sees the source but is asked for overall quality is still left without a rule saying which defect should dominate. Consequently, a preference or similarity judgment, however carefully collected, must not be reported as an observed correction of the attribution error. It records the preference or similarity judgment elicited under the supplied instruction and evidence; detecting and correcting an attribution error is a further task. A wider empirical inference would require suitable observations and a justified design, not merely further invented cases.
This is not an argument against human evaluation; it locates it within the chapter's question about what a summary score hides. A single overall-quality score can collapse different dimensions, whereas a score for a specified dimension keeps the intended criterion visible. Naming that criterion still does not establish rating accuracy. That is why the proposal here compares two invented summaries against a fixed source rather than reporting a leaderboard or a new measured score. The relevant lesson from SummEval (Fabbri et al., 2021) is methodological: automated scores should be compared with the judgments they are meant to approximate. Its news-domain evidence leaves the present task of question preservation untested.
Suppose two invented summaries of the same fixed source are put before readers. One is smooth and pleasant to read; the other is less graceful but makes a disputed premise easy to locate. Asked to choose the better prose, readers may favor the first. Asked which output lets them find the contested claim, the same readers may favor the second. Neither instruction measures every property of the pair.
The divergence does not show that the judgments are unreliable. Each answer is responsive to the question posed. But it does show that a preference cannot be promoted into a general verdict. In the stipulated case, "The room is open" and "The room is not open" share most of their words, yet they do not preserve the same assertion. A pleasantness judgment therefore cannot be reported as evidence that the assertion survived. Equally, a successful premise-location judgment says almost nothing about readability.
Coverage makes the same point from another side. One candidate may preserve every fact it expresses while omitting an entire entrance concern raised by the source record; support and coverage then have different eligible items. And where a fluent sentence silently changes "one contributor" into "the group," a preference or similarity judgment must not be described as an observed correction of that attribution error. The rater was not asked to audit attribution, or lacked the evidence needed to do so.
The inferential step is that each instruction licenses only claims about the dimension it solicits. Agreement across instructions does not establish a single underlying quality, and disagreement does not rank the outputs. Adding further instructions may widen the account, but no finite set of judgments becomes comprehensive by accumulation. The relevant limit is practical as well as logical: if the purpose is fixed before the outputs are seen, the judgment answers that purpose; if the instruction is selected afterward, the choice itself can manufacture the preferred winner. This comparison therefore reports scope, not a leaderboard or a new measured score.
21.5Omission beyond precision
Consider a hypothetical summary compiled from participant submissions. Every factual claim it expresses is supported by the relevant source: dates, named actors and described events all match the record. Yet one entire concern raised by a contributor appears nowhere. Readers cannot infer from supported sentences alone that the collection’s important concerns have been covered. Source support and omission are therefore separate properties even before anyone checks the underlying claims against the wider world.
The first judgment concerns precision over expressed claims. Min and colleagues’ FActScore divides a generation into atomic facts and measures their support against a reliable knowledge source (Min et al., 2023). That permits a more informative diagnosis than a single verdict on a mixed answer. It does not make every error harmless: one unsupported claim may still be decisive for the reader’s task. The proportion and the consequence of an error remain different questions.
The second judgment is coverage: of the concerns that were submitted, how many are represented at all? This coverage question is outside the support-precision quantity just defined; it should be added as a separate evaluation requirement. The omission problem is not that the metric computes the wrong quantity, but that precision over expressed claims is blind to claims never expressed. A summary can therefore be maximally precise over a systematically truncated sample of its inputs.
What the case requires is an additional question-coverage judgment, held separately from factual support: for each submitted concern, does the summary give an adequate, intelligible account of it? This judgment needs a defined unit of concern and a threshold for adequate representation, both of which are choices for the evaluation design rather than properties of the summary's individual sentences. Coverage and precision then report distinct facts about the output. Neither substitutes for the other, and a system that excels at one while failing the other deserves a diagnosis rather than a single rank.
The precision-like proportion has a structural limit: its denominator contains only claims the output actually makes. Deleting an unsupported claim can therefore raise the proportion without improving the summary's coverage of the source, because nothing in that denominator records what has disappeared. The calculation is not wrong; it is answering a narrower question than it may appear to answer.
A different denominator is therefore needed, built from the concerns submitted for representation rather than from the output's assertions. For each concern, the evaluation asks whether the summary gives an adequate and intelligible account of it. Two design choices follow, and both belong to the evaluation rather than to the sentences themselves: the unit at which a concern is counted, and the threshold at which its representation counts as adequate. Coverage and claim support then have different eligible items and report different facts.
The invented candidates make the difference concrete. One summary may preserve every fact it expresses yet omit an entire entrance concern; by the support proportion it is impeccable, and by question coverage it is incomplete. Deleting material cannot repair that omission, and neither can adding well-supported but off-concern detail. The separate denominator is what makes the omission visible at all.
The same case warns against borrowing substitute judgments. "The room is open" and "The room is not open" share nearly all their words, so lexical overlap cannot by itself establish that an assertion has been preserved; no ROUGE, BERTScore or model probability is computed here. Likewise, a fluent sentence that changes "one contributor" to "the group" misattributes the claim, and a preference or similarity judgment in its favour must not be reported as an observed correction of that error.
The qualification is that two proportions still do not collapse into a ranking. A system can score highly on supported output claims while omitting what the questions were collected to represent, and a system can cover every concern while filling the account with unsupported material. What the pair supports is a diagnosis of which kind of failure occurred. Which comparisons are legitimate, and which materials and dimensions count as eligible, remain to be settled before results are examined.
21.6Predeclared comparisons
Against the objection that several metrics merely license choosing after seeing what flatters a system, the remedy is procedural: fix the comparison before results are available. For this proposal, the task is to assess how well two invented summaries represent a fixed source. The eligible materials are therefore the source and those two candidates only; no model leaderboard enters, and no measured score is reported. Within that frame, the declared dimensions are overlap, contextual similarity, consistency, and human evaluation, each specified for the question of representing the source rather than for a general notion of quality.
Predeclaration makes a disagreement diagnostic. Consider "The room is open" and "The room is not open." Shared words alone cannot guarantee preservation of the assertion, so a consistency check registers a difference that lexical overlap may miss. Which dimension registers or misses the contrast is now visible, whereas a single summary score would conceal that difference, because the contrast was named in advance. Similarly, the candidate that preserves every expressed fact while leaving out an entire entrance concern forces the declaration to say which items are eligible: claim support and source-concern coverage draw on different sets, so a single proportion cannot silently carry both. That is only a reminder of the support/coverage distinction already needed elsewhere; the present point is that the denominator must be declared here too.
A fluent sentence that changes "one contributor" to "the group" shows how the declared treatment of attribution applies. A preference or similarity judgment about that sentence cannot be reported as an observed correction of the attribution error, because no such observation exists in this proposal.
The gain from predeclaration is modest and real: it prevents the set of dimensions from being reassembled after the fact in whatever order favors a system. It does not itself rank systems, produce scores, or prove that any one dimension is adequate to the purpose of representing questions. Ranking and adequacy remain to be argued.
Once the comparison task, eligible materials, and dimensions are declared, applying them is a matter of matching each judgment to the item type that the declaration admits. The fixed-source comparison of "The room is open" against "The room is not open" makes the point concretely: shared words are not by themselves an eligible basis for treating the assertion as preserved, because the negation changes the claim. No ROUGE score, BERTScore, or model probability was computed here, and none would have settled the question on its own. The scope of the comparison therefore has to track the dimension being assessed rather than whatever measure happens to be at hand.
The second candidate draws the boundary in the other direction. It preserves all of the facts it expresses, yet it omits an entire entrance concern. Claim support and source-concern coverage have different eligible items, so a record can score well on one while remaining silent on the other. Narrowing the comparison to whichever dimension flatters the candidate would leave the purpose of representing questions untested.
The third candidate is fluent but changes "one contributor" to "the group," an attribution shift that the source does not license. A preference or similarity judgment in favor of the fluent version must not be reported as an observed correction of that error; correction is a different claim from acceptability, and it requires the relevant ground to be checked.
These applications may still expose a gap in the rubric. When they do, exploratory revision is permissible: the altered criteria should be recorded as exploratory, then tested on fresh material. The one qualification that matters is presentational. A revision discovered during the comparison cannot be credited retrospectively as preregistered success on the material that prompted it. Recording preserves the distinction between a repair and a finding, so later comparisons can show whether the revised rubric earns its place on cases that did not shape it.
21.7A scorecard with explicit unresolved tradeoffs
A scorecard serves the purpose of representing questions only if its fields remain separate. A proposed evaluation would record four: support, coverage, readability, and correction. Each field has different eligible items, and that difference is the point.
Support asks whether what the summary asserts is preserved against the fixed source. Compare "The room is open" with "The room is not open." The sentences share most of their words, yet the assertion reverses; shared vocabulary cannot by itself guarantee preservation. No ROUGE, BERTScore, or model probability is computed for this comparison.
Coverage asks which source concerns have been addressed. One candidate may preserve every fact it expresses and still omit an entire entrance concern. The eligible items under support are the claims the summary makes; the eligible items under coverage are the concerns the source raises. Merging the fields would let a fully supported but incomplete summary register as adequate.
Readability records how the text presents what it says. It matters to a reader encountering the question, but it supplies no evidence about the source.
Correction is the field most easily collapsed into another. A fluent sentence changes "one contributor" into "the group." A judge may find that version clearer or closer to what was intended, yet such a preference or similarity judgment is not an observed correction of the attribution error. The correction field must name the defect and record whether the summary alters it.
The scorecard thus yields four recorded judgments rather than one number. Its qualification is that any one field can be satisfied while another fails: the second candidate is supported yet thin on coverage, and the fluent rewrite is readable yet still wrong on attribution. The entries remain comparable within their fields, and the proposed evaluation reports no system ranking from them.
The scorecard becomes consequential when its fields do not point the same way. Consider the proposed comparison of two invented summaries against a fixed source. One candidate preserves all the facts it expresses but omits an entire entrance concern. Another covers that concern yet is the fluent sentence that changes "one contributor" to "the group." Support and coverage therefore have different eligible items: the first candidate makes no unsupported claim, while the second attends to a source concern that the first drops entirely.
Neither candidate's merits cancel the other's defect. The fluent misattribution cannot be repaired by declaring it a similarity or preference finding; only an observed correction could establish that the attribution error has been fixed, and none has been reported. Likewise, the omission is not a support failure, because nothing in the first candidate contradicts the source. The fields record different failures rather than combining into a common measure of quality.
The inferential limit is equally concrete. If two options trade off support, coverage, readability and correction in this way, the unresolved choice must remain visible until priorities are supplied. A preference for complete source-concern coverage would favor one candidate; a stricter rule against unsupported attribution would favor the other. Both are coherent evaluations, but they are answers to different questions about what the representation is for.
What cannot follow is a neutral grand total. Such a total would silently decide how much misattribution is worth trading for how much omitted concern, precisely the priority that has not been supplied. No model leaderboard or new measured score is at stake here, and no ROUGE, BERTScore or model probability has been computed. The scorecard's scope is therefore diagnostic: it identifies the unresolved tradeoff and the decision needed to resolve it. If purposes differ by task—readability, accuracy, agreeability—the priorities must be chosen for that task rather than borrowed from a general ranking.