15.1Error and plurality
Uncertainty about what a collection represents is not the only source of disagreement in annotation. This study draws a sharper distinction once it asks what the instructions actually settle. Some conflicts are mistakes against an explicit, unambiguous source; others arise because the source underdetermines the label. These should not be handled alike.
Take the invented notice as a fixed case: the notice states a closing time of 18:00, while a draft reports 20:00. Under that fixed source version, recording 20:00 as the notice's closing time is a checkable source mismatch. The correct response is correction, and there is nothing to preserve in the alternative. By contrast, suppose the guideline says only "mark the main concern" and leaves "main" open between the first-mentioned concern and the one most strongly emphasized. Two contributors applying different readings are not making recording errors. The instruction is genuinely ambiguous, so incompatible annotations can survive while that ambiguity remains unresolved.
The service-allocation dispute marks a different boundary again. Contributors disagree over whether a fixed discretionary budget should fund step-free access work or quiet study space. No prices, legal duties, feasible combination, vote, or winning priority are stipulated. That disagreement is a value conflict about priorities, not a failure to transcribe a source; it persists because the decision weighs aims against each other, not because a record is wrong. If the budget could in fact cover both projects, the practical question would change into an empirical one; with the stated facts, no such resolution is available.
The qualification is that defensibility is bounded by the actual ambiguity. An interpretation survives only where the fixed materials genuinely leave the question open; where they state a value explicitly, the mismatch remains an error.
Disagreement establishes only that positions differ; it does not say why. Nor does minority status settle the matter. A position can be held by few and still track an explicit source exactly, while a majority can converge on a transcription slip. Classification therefore turns on the relation between a claim and the source or instruction under which it was made, not on the distribution of support.
In the invented notice, closing is at 18:00 while a draft reports 20:00, and the example fixes the source version. The mismatch is checkable, and the correction does not depend on which side is more numerous. Whatever the headcount, the entry contradicting the fixed record is a recording error to be repaired; no interpretive interest attaches to preserving it.
The discretionary-budget dispute differs in kind. Contributors prefer step-free access work or quiet study space from one fixed budget, while no prices, legal duties, feasible combination, vote or winning priority is stipulated. There is thus no established fact against which either preference can be shown mistaken. Equally, the conflict alone does not yet demonstrate valuable or reasonable plurality: that would require the rival commitments to be genuinely live together, and the case does not say whether the budget reaches both. If the budget were sufficient for both, the apparent scarcity would be a false premise rather than a value conflict; if it is not, the disagreement concerns priority under constraint, where correction is the wrong instrument.
The annotation guideline supplies the middle case. Told to mark the main concern, contributors split between first-mentioned and most strongly emphasized. Since the instruction leaves "main" unresolved, both readings can be applied coherently, so their disagreement is not yet error and should survive as competing readings until the guideline is made precise. The qualification is scope: plurality is confined to the ambiguous term, and any marking that contradicts an explicit part of the instruction still fails.
The governing criterion is therefore the source relation—explicit record, unresolved instruction, or constrained priority—and each relation licenses a different response.
15.2A common instruction applied differently
The invented notice and the draft are not equally hard to settle. Given the fixed source version, the notice closes at 18:00 while the draft reports 20:00. That is a checkable mismatch, and correcting the draft resolves it without interpretive residue. Annotation work becomes different in structure when the instruction itself underdetermines what counts as a correct application.
Consider the guideline: "mark the main concern." It does not say whether "main" means first-mentioned or most strongly emphasized. Two annotators can therefore apply the same instruction faithfully yet classify the same item differently. One marks the concern that appears first; the other marks the concern carried by repeated or forceful language. Their records disagree, but neither has misread the guideline, because the guideline leaves the relevant criterion unstated.
This case separates three things that superficially look alike. A clear recording error, such as 20:00 where the fixed source gives 18:00, is identifiable against that source. An empirical dispute would turn on what the item or the case actually contains. A normative dispute concerns which concern deserves attention. The ambiguous instruction produces none of these by itself; it produces a coding split whose source is the guideline's own slack.
Variation in annotation can itself be the phenomenon requiring explanation (Plank, 2022), rather than an error to erase before analysis. The qualification matters: this does not make every disagreement valuable, and a minority coding is not thereby a contribution. Misinformation and inconsistent instructions can still be corrected, so any apparent plurality has to be traced to the rule actually applied rather than inferred from the mere existence of conflict.
Until the intended sense of "main" is fixed, the pair's divergence cannot be charged as error to either side, and the annotation record itself supplies no reason to prefer one concern over the other. That is the immediate limit of the example.
The instruction to mark the main concern does not determine what counts as main. Once the codebook specifies whether "main" means first-mentioned or most strongly emphasized, two annotators confronting the same text apply one rule rather than two. That clarification resolves the coding disagreement at the level of the guideline: under the revised codebook there is now a determinate answer. Nothing in the step decides whether first mention or strongest emphasis is the better way to identify a concern. The rule becomes precise while its purpose remains open to argument.
The distinction matters because disagreements have different shapes. An invented notice says closing is at 18:00 while a draft reports 20:00. Under the example's fixed source version, that is a checkable source mismatch: one statement conflicts with the record designated as authoritative, and correcting the record settles what was said. The guideline ambiguity is different. Both readings cohere with the instruction, so there is no transcription error to repair; the defect lies in underdetermination. A clearer codebook removes the ambiguity without vindicating either substantive preference about salience. One case is settled by consulting the record; the other by amending the rule.
Nor does clarification reach the separate dispute over the fixed discretionary budget. Contributors may prefer funding step-free access work or quiet study space; no prices, legal duties, feasible combination, vote or winning priority is stipulated. Even a perfectly specified annotation rule leaves that allocation open. The limit is scope: clarification removes a source of inconsistent coding, but it cannot merge factual correction, rule specification and value choice into one operation. Where the instruction underdetermines the label, precision is available; where priorities compete, the disagreement survives clarification and must be assessed on its own terms.
15.3Conflicting values with accurate facts
Once a guideline is clarified—for instance, by stating that "main concern" means first-mentioned—the coding split it produced can be settled by applying the stated rule, without recasting the earlier split as error. The service-allocation dispute differs in kind. Fix the facts as follows: a discretionary budget of fixed size is available, and two projects compete for it, step-free access work and quiet study space. One contributor gives priority to the first, another to the second. Nothing about the budget is in doubt between them. Contrast this with the invented notice, where one record gives closing as 18:00 and a draft gives 20:00: under the example's fixed source version that mismatch is checkable, and one recording is wrong. There is no comparable source check that decides the budget dispute, because the contributors are not contradicting each other about the budget at all.
The inferential step is therefore a classification. Since the factual record is agreed and the disagreement concerns which use of the budget takes precedence, this is a value conflict over priorities, not an annotation error and not an empirical dispute. Clarifying an instruction can remove ambiguity about how to code a concern, but it confers no priority on access or quiet; those projects are endorsed for different reasons. A disagreement of this kind survives factual accuracy because accuracy is already present on both sides. Keeping it apart from recording errors matters practically: re-checking sources will not repair it, and treating it as error risks discarding a priority rather than answering it.
The stipulation is deliberately thin, and its limits should be preserved. No prices, legal duties, feasible combination of projects, vote, or winning priority is specified. Accordingly, this study may say the two priorities conflict in the contributors' stated ordering, but not that the budget is proven incapable of serving both aims. If the budget were sufficient for each, the practical dispute would dissolve; if a legal duty required step-free access, that would add a premise absent here. What the agreed record establishes is the type of disagreement, leaving its ordering untouched.
Once the budget facts are fixed and the two preferred uses compete for the same fixed budget, supplying a fuller description of the situation does not settle which use should take priority. More detail about the room, its users, or the timetable may confirm that both proposals are conceivable, but none of it ranks step-free access work against quiet study space. A ranking requires a normative premise—for instance, that mobility barriers should be removed before amenities are added, or that conditions for concentrated study are the budget's first duty. That premise is an additional commitment, not a factual discovery awaiting better description.
The correction boundary therefore remains sharp. In the notice, the invented closing time of 18:00 and the draft's 20:00 are checkable against the example's fixed source version, so the mismatch is a recording error and should be corrected rather than preserved as a disagreement. By contrast, the guideline instruction to "mark the main concern" is genuinely underdetermined between first-mentioned and most strongly emphasized; contributors applying the two readings are not making recording mistakes. Their disagreement is reasonable interpretive plurality, and it persists until the guideline itself is clarified.
The budget dispute occupies a third position. Because no prices, legal duties, feasible combination, vote, or winning priority are stipulated, no contributor can be charged with misstating the case, and no vote has yet ranked the alternatives. Claims about which project would produce better consequences would be an empirical dispute requiring evidence not supplied here; they should not be smuggled in as facts already agreed. If a binding duty required step-free access, or if a feasible way to fund both projects existed, the conflict could dissolve—but neither condition is stipulated, so both remain conditional possibilities rather than premises of the case.
The scope rule is modest: correction answers to a fixed record, plurality to genuine ambiguity, and incompatible priorities to explicit normative premises. Accurate description keeps the dispute honest; it does not rank it.
15.4Dissent from a minority position
A minority position can be entered in two registers that are easy to collapse: the number of people who hold it, and the reason they give. The two entries should be kept apart, especially while a disagreement is still being classified.
Start with the checkable case. In the invented notice, one document says closing is at 18:00 and a draft reports 20:00. Under the fixed source version this is a source mismatch, and a contributor who insists on 20:00 is not making a contested judgment. Headcount is irrelevant to the repair: the record is corrected and the relied-on source version is named, whether one person or many noticed the defect. The stated reason—here, an appeal to the draft—is simply the wrong source, and its defect can be identified without appeal to how widely it is held.
Contrast the annotation guideline that says "mark the main concern" without specifying whether "main" means first-mentioned or most strongly emphasized. A contributor who reads it the second way may be outnumbered, but the reason has to be written down separately from that count. The same discipline applies to the discretionary budget: some contributors prefer step-free access work and others quiet study space, and with no prices, legal duties, feasible combination, vote or winning priority stipulated, the minority's reason is a genuine premise about priority rather than an arithmetic embarrassment.
The inferential step is modest but useful. This study's tally tells a reader how widely a view is shared; the stated reason tells a reader what a defender of the prevailing reading must answer. Merging the two encourages twin mistakes: a reason dismissed because only one person gives it, and a reason treated as refuted because most contributors disagree. Mill's normative claim (Mill, 1859; Project Gutenberg edition) that understanding opposing reasons matters for holding an opinion well supports keeping those answers visible.
The limit is that this separation is a recording discipline, not a resolution. It neither repairs the 20:00 error nor decides what "main" means or which priority the budget should serve.
Once a minority's stated reason has been entered separately from the count of people supporting it, the next question is what that entry licenses. Retention of the reason does not assign it equal evidential weight or a veto over all action. It fixes the reason's place in the record so that later readers can see what was contested and why; it does not, by itself, rank that reason against the alternatives or make every downstream choice wait upon the minority's assent.
The notice dispute shows the limit plainly. Against the fixed source version, the notice says closing is at 18:00 while the draft reports 20:00. The mismatch is checkable, so the 20:00 entry is a recording error rather than an interpretive rival. The record can note that the draft said 20:00 and that it was corrected to 18:00, preserving the history without keeping the wrong time available as a policy option. Correction here is exactly the response owed to misinformation or inconsistent transcription.
The allocation dispute is different. Contributors disagree whether a fixed discretionary budget should fund step-free access work or quiet study space. Each reason is worth preserving, because each states a commitment that informs the decision. Yet no prices, legal duties, feasible combination, vote, or winning priority is stipulated, so retention alone establishes neither that both projects can proceed nor that either outranks the other. Only if it were separately stipulated that the budget covers both, or that one project is legally required, would the question collapse into a checkable constraint rather than a contest of priorities.
Finally, the guideline telling annotators to mark the main concern leaves "main" ambiguous between first-mentioned and most strongly emphasized. Recording both readings preserves a genuine interpretive choice without pretending the ambiguity has been resolved. A working annotation may still proceed on a declared reading; the minority's retained reason constrains how that choice is justified, but does not halt the work.
15.5False balance and unsupported assertions
A summary that records disagreement must separate two questions: whether a dissenting claim exists, and how much evidential weight that claim deserves. These are easily conflated, since a faithful report of an interpretive dispute can look like an endorsement of every side. The distinction is practical. An annotation error—such as a label applied against the stated guideline—can be documented, corrected, and retired from the record without losing the history of what happened. An unresolved value conflict, by contrast, does not disappear through better inspection; it requires an account of the competing commitments and the reasons they remain in tension.
Plank argues that human annotation differences can reflect subjectivity or multiple plausible answers rather than mere noise (Plank, 2022). That argument motivates examining the reason for a disagreement. The further distinction used here is analytical: some differences can be resolved against a clear source and instruction, whereas a conflict about which legitimate aim to prioritize may remain after factual correction. Neither outcome should be inferred from the existence of disagreement alone.
Suppose, in a synthetic example, that an archive notice clearly gives a closing time of 18:00 while a draft incorrectly reports 20:00. That error can be corrected against the notice. Separately, two contributors may agree on the actual hours but disagree about whether scarce resources should fund step-free access work or quiet study space. The second dispute is not resolved by checking the closing time. A summary can report the verified time and preserve the competing priorities without pretending the erroneous time remains an equally supported alternative.
The useful distinction is between representing a position and endorsing its warrant. Source-faithfulness evaluation in the summarization setting studied can help identify content unfaithful to the input (Maynez et al., 2020), while a separate account records why a practical disagreement remains. Preserving dissent requires this discrimination; it does not require treating all claims as equally credible.
A corrected factual premise has a limited but real downstream effect. Under the example's fixed source version, the notice's 18:00 closing time is a checkable mismatch with the draft's 20:00, so the recorded time should be corrected rather than treated as one of two equally supported accounts. The inference runs only as far as dependence: any conclusion that turns on the draft's 20:00—for instance, conditional advice to arrive by 19:30—must be revisited once the time is corrected. Nothing follows about conclusions resting on other premises, and the correction does not by itself settle any dispute that was independent of the closing time.
The service-allocation dispute is therefore not resolved by the correction. Contributors differ over whether a fixed discretionary budget should fund step-free access work or quiet study space. Because no prices, legal duties, feasible combination, vote, or winning priority is stipulated, there is no factual premise here to repair and no basis on which one priority can be branded a recording error. The disagreement is a value conflict about how a constrained resource should be used. Recording both positions is not false balance so long as the record does not pretend that the competing priorities rest on the same evidence or that their acceptance has already been decided.
The annotation case sits between these poles. A guideline saying "mark the main concern" leaves open whether "main" means first-mentioned or most strongly emphasized. Disagreement produced by that ambiguity is reasonable interpretive plurality unless the guideline is revised to choose one reading; it is not a factual mismatch of the kind exemplified by 18:00 and 20:00. The practical limit is that clarity about category does not settle the content. Correcting the time leaves the budget conflict intact, and clarifying the guideline would still leave open which concern deserves priority.
15.6When adjudication is necessary
Whether adjudication is needed depends first on what the disagreement is actually about. The invented notice gives closing time as 18:00, while a draft reports 20:00. Under the example's fixed source version, this is a checkable source mismatch, not a clash of values or competing reasonable readings. Once the controlling version is identified, the recording error should be corrected; leaving both times in place as equal alternatives would confuse correction with tolerance.
The annotation dispute is different. The guideline says to mark the main concern without specifying whether main means first-mentioned or most strongly emphasized. Once that ambiguity is acknowledged, a contributor's label may be defensible without being uniquely required. The immediate question is how much discretion the guideline currently leaves open. Adjudication becomes necessary when the label must be settled for later comparison, or when a downstream rule depends on one recorded judgment. Otherwise, the ambiguity is better documented than artificially resolved.
The service-allocation case has a sharper limit. A fixed discretionary budget is contested between step-free access work and quiet study space, yet no prices, legal duties, feasible combination, vote, or winning priority are stipulated. On those facts, there is no basis for declaring one allocation correct and the other a mistake. That is a value conflict rather than a recording error.
Deadlines may nevertheless force action. If contributors must report one allocation before further inquiry is possible, the responsible move is to select an action while preserving a record of the unresolved objection. The record should note which priority was chosen, what information was missing, and what would count against the choice. Such a record does not by itself resolve the dispute; it keeps the reason for the decision visible once the time for choosing has passed.
A decision can therefore settle what happens next without settling every underlying disagreement. That distinction matters because the forms of dispute call for different responses. A notice stating 18:00 while a draft states 20:00 is a recording mismatch against the example's fixed source version. Correct the entry, and continue. No interpretive hearing is needed to decide which timestamp the source records.
The remaining cases resist that treatment. When contributors disagree over whether a fixed discretionary budget should fund step-free access work or quiet study space, they are not relying on different records. They are ranking goods under a budget constraint. Since no prices, legal duties, feasible combination, vote or winning priority are stipulated, the example does not identify a unique answer, and selecting one project need not represent the other priority as mistaken. At most, it determines where the money goes while the case for the unfunded option remains on record.
The annotation guideline says "mark the main concern" without specifying whether "main" means first-mentioned or most strongly emphasized. Here disagreement follows an underspecified instruction, so the useful correction is prospective: define the criterion before the next annotation round. That correction does not prove that either contributor applied a false proposition under the text as it stood.
The force of these distinctions is procedural rather than evidential. When a deadline or authority requires action, selecting a project or applying one stated convention fixes the next step. It does not convert the rival priority into misinformation, retroactively remove the guideline's ambiguity or establish consent. The qualification is modest but essential: an action's necessity is evidence for acting by a stated rule, not evidence that the disagreement has been resolved on its merits.
15.7Recording the unresolved remainder
Correction and adjudication narrow the record; they do not empty it. Once the fixed source version is checked, the disagreement between the notice's 18:00 closing time and the draft's 20:00 is a recording error in whichever document departs from that version. The correction is local: it restores the stated time. It does not turn every other disagreement into carelessness, and it settles nothing about annotation or allocation.
The annotation dispute shows the distinction sharply. Contributors disagree about which concern to mark because the guideline says to mark the main concern without specifying whether main means first-mentioned or most strongly emphasized. An adjudicator may adopt one reading so that work can continue, but the unresolved issue is not who misread the instruction. It is the specification gap: the guideline does not say which sense of main governs. Recording that gap states a question about the guideline, not a charge against contributor conduct.
The allocation case leaves a different remainder. After a decision directs the fixed discretionary budget to one project, the unresolved question is which of step-free access work and quiet study space should have priority within that budget. The stipulation supplies no prices, no legal duties, no feasible combination, no vote, and no winning priority, so the record cannot describe one choice as the uniquely correct reading of the constraints. If costs were later established and showed that only one project fit within the fixed budget, that would narrow the question; absent such figures, it remains open.
What survives, then, is stated at the right level. The time mismatch is corrected as a recording error; the open specification question concerns the meaning of main; the open allocation question concerns priority within a fixed budget. An adjudicated action leaves those latter questions intact, and naming them exactly is what keeps an unresolved entry from functioning as a verdict that the participants who lost were unreasonable.
Having identified what remains unresolved after correction and adjudication, the record must say what the surviving disagreement is about. An unresolved label earns its place only when it names a question that someone can later revisit. That requirement separates it from a holding pen for complaints nobody has examined. In the invented notice example, the fixed source version settles the matter: the notice reports closing at 18:00 while the draft reports 20:00, so the mismatch is a recording error and gets corrected. Attaching an unresolved label there would preserve a defect as though it were a judgment.
The service-allocation dispute is different. Contributors disagree about whether the fixed discretionary budget should fund step-free access work or quiet study space. The stipulations supply no prices, no legal duties, no feasible combination of the two, no vote, and no winning priority. What survives, then, is not a verdict but a question: which priority should govern the use of that fixed budget, given only the options actually in dispute. Recording that question keeps the competing proposals visible. Recording merely "allocation pending" would not: it would leave the step-free access proposal and the quiet study space proposal equally nameless, so a later reviewer could not see what had been asked.
The same discipline applies to the annotation guideline that says "mark the main concern" without specifying whether "main" means first-mentioned or most strongly emphasized. Disagreement produced by that ambiguity is not a source mismatch like 18:00 versus 20:00. The unresolved label should therefore name the specification question to be settled: which sense of "main" the guideline intends. Conditionally, if a later instruction fixed "main" as first-mentioned, the ambiguity would end and any remaining disputes should be reclassified rather than preserved.
The limit is that unresolved labels do no work where an explicit rule or a corrected record already decides the case, and they do not license deferring a judgment that the available stipulations actually settle. They preserve a live question; they do not grant every interpretation equal standing.