38.1An explicit synthetic boundary
A synthetic pair earns its place in an evaluation by making exactly one difference between its two members. Everything else — the situation described, the parties assumed present, the register of the statement, the source's standing — is held fixed, so that a change in the system's output can be attributed to the single varied property. In the invented illustration used here, the source statement is "The room is not open" and the candidate offered against it is "The room is open." The room and the plain declarative framing are the same; polarity is what moves. The pair is thus explicit about its own boundary: it declares which features were frozen and which one was released.
What it tests is therefore narrower than whether a system understands negation in general. It asks whether, under a stipulated interpretation on which "not open" and "open" are incompatible, the candidate preserves the polarity of the source while every other feature stays constant. That is the comparison the construction licenses: source against candidate, at fixed context and fixed task. A quantity computed over such pairs would compare precisely this. It would not compare two encodings of the same material unless the task and inputs were held fixed while only the encoding varied; nor would it say which of the controlled properties matters most, since a score over polarity pairs measures polarity preservation and nothing more.
Two conditions govern the inference. First, a mismatch is attributable to polarity only if the pair truly differs in polarity alone; if the candidate also shifts modality or source status, the attribution collapses. Second, the result speaks to the stipulated reading, so a system answering under some other sense of "open" has not failed the mechanism the pair was built to probe. Passing such a pair establishes the narrow capacity it names; a failure marks a polarity loss in this instance.
Once the negation pair is in view — the source states that the room is not open, while the candidate states that the room is open — the comparison fixes the relevant context and varies the polarity alone. What the pair establishes is narrow but real: under the stipulated interpretation, in which the candidate is expected to preserve the source's claim, the candidate's reversal counts as a defect. A single constructed failure is sufficient to show that this failure mode is possible; it is not necessary that any particular system fail this way, and it certainly does not make such a failure inevitable.
The scope of that inference needs stating precisely. The result is a claim about possibility under a stipulated interpretation, not about the frequency of the defect in real work. Because the pair is invented, it carries no prevalence information whatever: nothing here indicates how often polarity reversals arise in practice, in which tasks, or across what populations of sources. Treating the pair as evidence about such rates would confuse a controlled demonstration with a sampled measurement.
Nor does the comparison rest on mismatched terms. Holding the task and inputs fixed — the source, the surrounding context, and the expectation of fidelity — means that any difference in outcome is attributable to the changed negation. If the encoding of the candidate were allowed to vary together with its content, the comparison would lose that attribution, and the pair would no longer isolate one feature.
The application therefore takes a conditional form: if a system is evaluated on this pair under this interpretation, its treatment of the candidate shows whether the polarity defect appears for these items. That finding can motivate a broader test in which speaker, modality, urgency, source status or disagreement are each changed in turn, but that broader test is a separate design whose outcomes remain open.
38.2One changed negation
Consider a synthetic pair, offered here as a prospective design illustration rather than as evidence drawn from a corpus. The source statement is "The room is not open"; the candidate is "The room is open." The negative particle is the only altered element, and its scope is local: it denies the predicate "open" as applied to this room. The proposition that reverses is therefore open(room). The source withholds that proposition, the candidate asserts it, and under a bivalent reading the two cannot both hold. The pair isolates a single polarity change and nothing else.
The reversal should not be overstated. The candidate contradicts the source only with respect to openness; it does not positively license a replacement such as "The room is closed," since once more than two room-states are admitted, the source's denial leaves those alternatives undecided. What the pair contrasts, then, is denial against assertion of one predicate, not open against every state compatible with not-open. This matters for reading a system's response: an answer that marks the statements as different is not thereby locating which proposition moved, whereas an answer that names open(room) as the reversing item shows the change is registered at the level the construction actually specifies.
A difference in outputs on this pair is attributable to negation handling only under the stipulated conditions: the same task, the same inputs, and the same speaker, modality, urgency and source status, with the encodings differing solely in the presence of the negative particle. Detecting the change is sufficient to show sensitivity to this negation in this construction; it is not sufficient for general negation competence. Conversely, missing the change is possible evidence of insensitivity here, but does not make such a failure inevitable across other constructions. A pair-level score accordingly compares behavior under one reversed proposition, not accuracy across a population of statements.
In the synthetic case the source reads "The room is not open" and the candidate removes the negation to read "The room is open." The negative attaches to the simple predicate "open" within a clause containing no quantifier, modal, or focus-sensitive element, so its scope is local and its reading is unambiguous. On that reading the source denies the property of the room and the candidate asserts it: the reversed proposition is exactly that the room is open, and nothing else in the pair changes. This is an invented illustration of design intent, so it supplies no observation of how any system responds; it claims only that the contrast isolates one feature, and that claim holds while no other feature is altered.
Scope discipline is what earns the clean reversal. Negation does not generally flip an entire sentence. When a negative element is combined with quantification or modality, one wording can support more than one scope assignment. Supposing the source instead stated that the room is not open to every applicant, a wide-scope reading would deny that the room is open to all applicants, while a narrow-scope reading would say it is open to none; removing the negation would then identify no single reversed proposition until the intended scope is fixed. Lexical negatives complicate matters further, since a word such as "unhappy" need not encode the exact complement of "happy."
The design rule follows: record the scope as part of the case specification, and prefer local constructions whose scope a reader cannot dispute. Stating scope also separates what the source commits the study to from what it merely leaves open. The source entails only that the room lacks the property "open" at the relevant time and under the relevant standard; it does not entail any particular alternative state, so the reversal cannot be strengthened into a claim about how the room is occupied or used. With that bound, the pair supplies the intended contrast without importing further properties.
38.3One changed speaker
Begin from the source attesting only one speaker's request: Mira requests later hours. The candidate, which reads that all contributors request later hours, keeps the request, its object, and the plain indicative unchanged and replaces the individual subject with a universal group subject. Because no further endorsements are supplied, the candidate credits contributors whose positions the source does not report. As a prospective design illustration rather than a recorded exchange, the pair isolates one edit: the extension of the set credited with the request.
The comparison concerns attribution scope. The group quantifier does not express greater confidence in Mira's request or point to a second channel of evidence; it enlarges the class of speakers to whom the request is assigned. Treating the substitution as stylistic would miss what changed. What the case compares is the set of speakers warranted by the source against the set asserted by the candidate — not the number of sources, and not the strength of the endorsement.
Whether the candidate is false is a further question. If Mira were the only contributor, "all contributors" would happen to coincide with her, and the sentence would be true in that scenario. The defect is therefore possible rather than inevitable: absent roster information, the candidate can overshoot the evidence while matching the truth. The case should carry that conditional supposition rather than assume unstated contributors.
Two limits follow. Holding the rest of the candidate fixed helps isolate the speaker-scope edit, but it is not a universal necessary condition for causal inference about the failure. And a matcher keyed to the shared phrase "requests later hours" may pass the group version, since the predicate is exactly the residue left untouched; a semantic predicate-only matcher may miss the scope change, while an exact matcher need not. This pair is built to catch that miss and to separate an attribution-scope error from an error about the request itself.
In the case already introduced, the source records that Mira requests later hours, while the candidate reports that all contributors do. What has changed is not the request but the scope of its attribution: the candidate treats one member's position as the position of the group. Because no further endorsements are supplied, the available evidence covers Mira alone.
Attaching the original source URL does not repair this defect. The link marks where the report came from; it does not enlarge what that source attests. Consider a conditional supposition: if the candidate is left unchanged and the URL is appended, a reader can trace the report to a document mentioning one person's request and then find that the group-level claim outran it. Traceability is thus necessary for auditing the report but not sufficient for supporting the quantified claim. What would be sufficient is an endorsement from every contributor, or a record of a collective decision; without either, the generalization is unlicensed. Provenance and evidential scope answer different questions, and only the second is at issue here.
The distinction also keeps the counter-condition honest. Contributors may share Mira's preference: that is possible, and the silence of the supplied material does not show that they oppose later hours. What fails is the step from one request to group-wide support. The candidate is therefore unsupported at its stated scope, not established as false.
Application follows from that diagnosis. Either narrow the candidate to what the source attests — that Mira requests later hours — or widen the evidence to cover the group. If the general form must be kept, the extension should be marked as an inference rather than as a source-backed statement.
For a paired test, fix the task and the source, and vary only the speaker scope in the candidate: one individual versus the whole group, with the request itself unchanged. Any difference in acceptability then belongs to attribution scope, not to the content of the request.
38.4A modal qualification
The invented source states "Repairs may finish Friday" and the candidate renders this as "Repairs will finish Friday." The change is confined to the modal: the source permits the outcome, while the candidate asserts it. To name the comparison precisely, what is measured is the force of the attributed claim under fixed inputs — same speaker, same event, same date — not the truth of the repair schedule. The two encodings differ only in the strength assigned to the possibility.
The defect is therefore not that the candidate is shown false. "May" establishes that Friday completion is consistent with the source; it does not establish that completion occurs. A possibility claim supports reporting that possibility, but does not by itself support a categorical prediction. Replacing "may" with "will" strengthens the prediction; it does not turn the predicted outcome into an established fact.
This also exposes what a consistency check would miss. The asserted outcome is compatible with the source's possibility, so a check asking only whether the source and candidate could both be true would let the strengthening pass. What must be compared is whether the source licenses the candidate's force: "may" does not entail "will," and compatibility is too weak a relation to detect the change.
A conditional supposition can be stated: if the source had said the repairs will finish Friday, the candidate's modality would match. Without that source change, the categorical prediction remains unsupported, and "will" asserts more than the record bears.
The qualification needs care about what it denies. Upgrading "Repairs may finish Friday" to "Repairs will finish Friday" is not illegitimate in itself. A categorical prediction can be justified when additional evidence supports it — a confirmed schedule, a completed inspection, a binding commitment. The point is narrower: the synthetic packet supplies none of this, so within the packet the upgrade has no warrant to rest on. The stronger claim is possible to justify in principle, but not inevitable from the wording change alone, and not licensed here.
The distinction between the two readings matters because an evaluation can easily conflate two judgments: that a modal upgrade is a distortion in every setting, and that this particular upgrade exceeds its evidence. Only the second is supported. The candidate's "will" is a categorical prediction; the source's "may" reports possibility. Converting the possibility into a categorical prediction changes what the evaluation should record as the task's target. Nothing in the case establishes Friday as an outcome, so treating "will finish Friday" as the correct answer would import an outcome the inputs do not contain.
The practical consequence is that error measurement on such a case should be keyed to the evidential gap rather than to the mere presence of stronger language. A candidate that predicts completion categorically fails not because confidence is forbidden, but because the inputs that would license confidence are absent. Conversely, were such evidence supplied later — as a hypothetical extension of the packet, not as a fact of it — the same wording could become appropriate and the evaluation would need to change accordingly. Keeping the condition attached to the particular claim prevents the stronger, categorical generalization that modal upgrades are always errors. That generalization is a possible reading, but not an inevitable one.
38.5A rare conflicting concern
Synthetic D, an invented illustration for a prospective evaluation, holds the majority wording of a source collection fixed and alters exactly one item: one contributor's support now ends with the clause only if staffing is available. The claims, the other contributions, and their order are left as first given. Isolating a single opposing condition is necessary but not sufficient to attribute any later difference in an output to that clause rather than to a different case: generation randomness and scoring conditions require control or repetition.
The clause also changes what the contributor asserts. “Only if staffing is available” makes staffing a necessary condition of that support: absent staffing, the support does not hold. It does not make staffing sufficient for the contributor's conclusion, and still less does it make the condition sufficient to settle the collection. The added material is therefore a qualification on one item, not a warrant for an overall verdict. Treating the two as the same would confuse the form of an assertion with the weight it carries, and the weight is precisely what this case does not determine.
To compare representations of the collection, keep the task and the inputs fixed and vary only the encoding: one representation states the support, another states the support together with its necessary condition. A comparison that also changes the collection measures case difficulty instead of the encoding, and the two effects become inseparable. The relevant contrast is thus narrow by construction.
The resulting design supports a distinction worth carrying forward: whether a condition is preserved in a summary, and what priority that condition should receive, are separate questions. The first can be posed of this collection directly. The second requires an argument about weight, rarity, or decisive force that Synthetic D, as a prospective illustration, does not supply.
Synthetic D adds "only if staffing is available" to one contributor's support while leaving the majority wording fixed. The phrase is a necessary condition on that support: it does not withdraw the support, nor does it state a rival recommendation. If staffing is unavailable, the support does not apply; if staffing is available, the support stands exactly as before. The contrast is therefore narrow, and the majority's position is untouched.
The question this case raises is retention, not precedence. Does the condition remain visible in a summary or representation, attached to the support it qualifies? Rarity supplies no license to grant it automatic priority. A rare condition can be dropped without changing the answer, while a common one can decide it; rarity is neither sufficient nor necessary for influence. Treating the scarce item as decisive would answer the weighting question before showing that the condition survived at all.
Outcome claims need the same care. Failure to retain is possible, not inevitable; if it happens, a named measure of agreement with the majority wording could stay high while the qualification is lost, so such a measure would compare surface overlap between two versions of the same case rather than fidelity to the conditional support. Fix the task and the inputs when comparing encodings: same text, same added phrase, same demand to report each contributor's position, with only the representation varying. Any difference in retention can then be charged to the encoding.
Retention itself has a minimum criterion: the output marks that contributor's support as conditional and locates the condition there rather than in the majority. Even so, the phrase leaves open who assesses staffing or when; if a task required that determination, the invented case would be underspecified. Retention shows visibility; it does not decide the weight the condition deserves.
38.6A changed access condition
In this synthetic case, the summary and the cited source remain fixed; the only change is whether reader R may consult that source. Naming this precisely matters, because the manipulation alters a relation between reader and evidence, not the evidence itself. The summary's truth does not track R's permissions. What tracks them is the summary's inspectability for R: whether R can put the claim beside its source and judge the fit.
Two qualifications follow. First, access is necessary for direct source-checking, but not sufficient for successful verification; R must also locate the relevant passage and read it competently. Second, withdrawn access makes such checking unavailable to R; it does not make the summary false, misleading, or ungrounded. Those outcomes remain possible in particular cases, but they are not the inevitable consequence of a changed access condition. The contrast therefore isolates a capability condition rather than a verdict about the text.
Suppose, as a countercondition, that R's access is later restored while text and source stay exactly as they are. The summary would then be inspectable to R again without any edit, which is the inference the paired design supports: the property under test belongs to the reader-source relation, not to the wording.
An evaluation that counts checkable claims must therefore say which quantity it means: claims with sources that exist, or claims whose sources the specified reader can reach. These are different quantities, and comparing them without fixing the reader's relation to the evidence collapses the contrast. Comparisons between representations of the same case are likewise informative only when task and inputs are held still — same summary, same source, same reader role — so that access is the sole varying factor.
Even then, access changes the opportunity to inspect, not whether R actually consults the source. A permitted reader may accept the summary unexamined, so inspectability and uptake must be scored separately.
In this synthetic case, the summary text and the source it cites remain fixed; what changes is whether reader R may consult that source. The truth of the summary therefore stands or falls exactly as before. What has shifted is inspectability: whether R is in a position to compare the summary against its source. A test built on this pair should attribute any difference in R's conduct or judgment to that property alone, so far as the text and task are held fixed.
Scope matters. Inspectability is relational, attaching to a text, a cited source and a particular reader under a particular access rule. Changing R's permission changes R's inspectability without changing the text's semantic content, and without settling what other readers can do. It is therefore wrong to report the manipulation as a change in the source's status generally.
Access being permitted is necessary but not sufficient for inspection. R may lack time, competence or willingness, so a permitted-access condition leaves actual verification only possible, not inevitable. Conversely, when access is unavailable, comparison through that cited source is impossible for R; this closes one route to checking but does not entail that the summary is false, misleading or unusable. Supposing R's judgment shifts in the second condition, the pair would license attributing that shift to lost inspectability — not to altered truth — provided the text and task are held fixed. That is a conditional claim about attribution, not a prediction that any shift will occur.
Applied consistently, the case fixes the comparison: same text, same task, different access. Any quantity computed across the two conditions measures the effect of that access difference for R, not a difference in representation or content quality. The case thus yields an inspectability contrast to carry into broader testing.
38.7From adversarial examples to broader testing
The chapter's deliberately constructed contrasts have served their purpose: each isolates one property and lets this study inspect whether changing that property changes the judgment. They are development constructions, so their evidential reach stops at the cases themselves. The prospective step is to ask what a test on separately selected natural material would have to specify.
First, declare the population: the kind of naturally occurring record the evaluation concerns, with the range of speakers, sources and situations it is meant to include. Then fix the judgment task and the candidate inputs before comparing any two encodings of a property. If the encodings differ while the task and inputs are held constant, a difference in behavior is attributable to the encoding rather than to a different question being asked. Select cases against an inclusion rule stated in advance and without consulting system performance.
Natural material will rarely supply the clean one-property pairs the authored cases provide. Where a counterpart differing in exactly one dimension — speaker, negation, modality, urgency, source status, disagreement — happens to occur, pair it and inspect the joint result. Where none occurs, mark the gap rather than authoring a substitute and presenting it as found. That gives a further inference: pairing alone cannot be assumed sufficient for the declared population, because the dimensions may be exercised only sparsely and in combinations the test does not control. A coverage tally therefore counts how many dimensions and contrasts were sampled; it is not itself a rate of correct behavior over the population.
The same restraint applies to individual outcomes. If a naturally occurring pair breaks consistency, that establishes a possible failure mode; it does not show the failure is inevitable, and consistency elsewhere does not rule it out. What the proposed evaluation adds is a declared frame, an independent selection rule, and an honest record of the contrasts that natural material did and did not furnish.
The preceding contrast pairs were built to isolate selected properties one at a time. Their role in development should therefore be recorded explicitly: they are authored constructions that helped shape and test the procedure, not a sample from any stated population. Success on them establishes only that the system handled those particular controlled inputs. It does not by itself yield a representative accuracy estimate, because the denominator would consist of cases selected for the purpose of exposing specific failure modes. That estimate, if later attempted, must be reported for a separately selected natural-material set whose population and selection rule are declared.
The distinction matters when interpreting the proposed move to broader testing. A procedure that passes the authored contrasts has cleared a necessary check for some targeted behaviors, but passing is not sufficient evidence of general performance. Conversely, failure on a contrast does not prove that the behavior is inevitable in ordinary use; it shows a possible failure under the stated construction. The inference is therefore bounded by the construction and by the task definition.
If results are compared across encodings or representations, the task and the inputs must be held fixed, while the encodings are allowed to differ. Otherwise a change in score may reflect a change in the cases rather than in the representation. The same discipline applies to the case record: list which contrasts were consulted during development and keep them separate from the future evaluation material.
On this basis, the appropriate broader test is prospective. Its cases should be selected independently after development, with a declared target population and an explicit comparison of what the success quantity measures. The authored contrasts remain useful as a focused diagnostic, but they cannot stand in for the later population estimate.