24.1A judge with a stated task
The proposed workflow assigns three roles: a drafter, a model judge, and an accountable editor. The labels do the separations, but the labels alone validate nothing. A frontier model is not independent merely because it is stronger than the drafter, and hiring a judge does not certify the judgment. What can be certified, or at least inspected, is whether the judge was asked a question precise enough for its answer to be checked.
So define the task narrowly. Give the judge one specific claim and one accessible source passage, and ask whether that passage supports that claim. This is a claim–passage relation with a fixed object on each side. The editor can then reopen the passage and see what in it is supposed to carry the claim, rather than asking whether the output is generally good. Where the relevant source is short and coherent, direct reading may perform the same task more simply; the model judge earns its place through scale or repetition, not through abstraction.
There is historical reason to think model-based evaluation can track human judgments in some settings. G-Eval reports improved alignment with human judgments in its evaluated tasks, and raises a potential bias toward model-generated text (Liu et al., 2023). That is a report about the tasks then tested, not a verification of present models on this claim-support question, and a verdict about readability or quality would not establish that a factual claim is supported in any case.
The inferential step is deliberately modest: an inspectable support verdict licenses a conclusion about one claim against one passage, and nothing further. Whether supported premises warrant their combined conclusion is a separate check, and it must be posed as such rather than folded into the support label. Nor does inspecting verdicts yield an error rate: purposively selecting difficult claim–passage pairs can expose defects, but converting the defect proportion into a prevalence estimate would require a defined population and sampling design.
This narrowness is the price of auditability. Each verdict states what it decided, and the editor can contest it on the stated evidence.
Once a narrow verdict is in hand—say, whether one claim is supported by one accessible passage—its scope should remain visible rather than being collapsed into a larger judgment. A general quality score cannot silently stand for all of support, completeness, readability, and representational authority, because these properties can come apart. A draft may be fluent and well supported on one claim yet omit the population qualification that changes what the claim means.
The stipulated illustration makes the separation concrete: a drafter and a judge accept an invented statement that a cited trial includes only outpatients, although the accessible source says the trial includes both inpatients and outpatients. Nothing here measures how often such errors occur. But the failure is diagnostic: readability and apparent support were assessed, while the representational question—whether the text attributes the right population to the source—went unchecked. A single high score would have concealed which property was ever inspected.
So the workflow needs verdicts that name their object. One check asks whether one claim is supported by one passage; a separate inference check asks whether supported premises warrant their combined conclusion. Keeping these apart also creates observable error checks: when the editor rejects a draft, the record can show which check failed and whether the judge or the editor caught it. That record is more useful than agreement rates between a drafter and a judge, since two roles can share the same mistake; a stronger model is not independent merely because it occupies a different role.
The limit is cost. Human review is fallible and expensive, and evaluating the evaluator without end is not a workable design. Task-scoped verdicts bound that cost: each check has a defined object and a defined failure, so escalation is reserved for claims where support and representational authority both matter. Role separation contributes no validation by itself; it contributes traceable checks whose coverage can be inspected.
24.2Order and length effects
A verdict is delivered after an encounter with two texts, and that encounter has an order and a length. The judge task remains narrow — whether one specific claim is supported by one accessible source passage — but the order in which candidate answers appear and how much each answer says are not features of the claim. They are features of the presentation, and they can move a verdict for reasons unrelated to support.
The judge-evaluation work makes this a live concern rather than a theoretical one. Zheng et al. (2023) evaluate strong-model judging against human preferences and identify position and verbosity biases, and self-enhancement (Zheng et al., 2023). The reported agreement belongs to a preference comparison: it concerns which of two candidate answers human raters prefer, not an audit of either answer against the source. A model that agrees with raters about which answer is better can still accept an invented characterization of a cited study, if neither the rater nor the judge consults the passage that would expose it. That is a possible failure route, not an observed event in this manuscript.
Two consequences follow for the review workflow. First, order and verbosity are candidate confounds of the judge's verdict, so they belong among the observable error checks rather than among the unexamined conditions of evaluation. Second, calling a role "judge" does not establish that its verdict is insensitive to presentation; separation between drafter, judge, and accountable editor is organizational, not evidential.
The relevant limit is that discovering an order or length sensitivity establishes only that it is present in the tested comparison. It does not size the effect for this workflow's particular claims and passages, and it cannot certify that any later adjustment has removed it. Conversely, failure to detect sensitivity in a limited test would not prove complete presentation independence. That is why the next step is a design proposal, not a cleanup already done.
Motivating the check is only the first step; the workflow has to specify what is checked and what the result would license. In the proposed setup, the judge is asked whether one specific claim is supported by one accessible source passage, while a separate inference check asks whether supported premises warrant their combined conclusion. Order and verbosity bear on the first task: the position of a candidate answer and the amount of prose supporting it can shape a judgment that is nominally about the passage alone.
Two bounded measures follow. Balanced order means judging the same pair of drafts in both presentation orders, so that a preference tied to position can be distinguished from one tied to content. Comparable output lengths means presenting the candidate answers at matched length — by trimming the material the judge sees to a common cap, or holding drafter and judge-facing variants to the same limit — so that extra prose is not mistaken for better support. Both measures are proposals for how the workflow should be run; neither has been applied here, and no effect size is claimed.
The limit is that they make order and verbosity effects detectable and less confounded with quality, not that bias has been removed. A judge whose preference persists after these controls is not thereby shown to be unbiased, and a clean run on a few pairs is not a rate. Difficult paragraphs can be purposively selected to discover such failures; converting the share of failures into an estimate of how often the workflow misjudges would require a defined population and a sampling design. The accountable editor therefore has cheap, interpretable checks to run and a bounded reason to run them: balanced order and matched length narrow two known confounds, while what the judge reliably decides still has to be shown by the observable error checks rather than by role names.
24.3Shared assumptions and correlated errors
A stronger model judging a weaker drafter improves role separation, but role separation is not statistical independence. The two systems may differ in capability while still relying on the same background assumptions, the same retrieved context, or the same kinds of inferential shortcuts. When their errors are correlated, agreement between drafter and evaluator is weak evidence of correctness: both may endorse a claim for the same flawed reason. No correlation rate is estimated here; the point is structural possibility, and it should be tested rather than assumed away or assumed to dominate.
Zheng et al. (2023) evaluate strong-model judging against human preferences and identify several judgment biases (Zheng et al., 2023). That work motivates scrutiny of automated evaluation, but it does not show that every cross-model judgment fails, nor does agreement with human preference ratings establish independent fact checking. This chapter's inference is narrower: capability and preference alignment are distinct from epistemic independence, and neither can be treated as evidence that the systems’ errors are statistically independent. Observable checks can test particular shared premises; measuring error dependence would require a separate comparison.
Consider a hypothetical case. A drafter asserts that a cited trial enrolled only outpatients, and the judge accepts the claim because the surrounding passage reads coherently and the citation exists. Stipulate that the source explicitly includes both inpatients and outpatients. If the judge merely reads the draft, that contradiction can survive. A direct source check can expose it: extract each load-bearing premise from the draft; verify that the cited passage actually supports it; attempt an independent retrieval of the underlying evidence; and record which source sentence supports which claim, so that unsupported premises leave a visible gap. A deliberate adversarial pass that tries to construct counterexamples adds another angle, though it too inherits the reviewer's blind spots.
What matters is that these checks are observable and auditable, not that the evaluator is nominally superior. Stronger judges can still share a drafter's assumptions; evidence verification provides a concrete opportunity to expose a shared error that role labels alone would leave unexamined.
Agreement between drafter and judge becomes more informative only when it is tied to evidence that is independently accessible. In the stipulated example, both accept an invented claim that a cited trial includes only outpatients, while the source passage includes inpatients and outpatients. Calling one actor a judge, or selecting a stronger model, does not change the statistical standing of their agreement: role names and capability do not establish independence, since both may carry the same assumption into the same reading.
What makes the agreement probative is the checkable link between a particular claim and a particular source passage. A proposed judge task asks whether one specific claim is supported by one accessible passage. Because the passage is available to others, a later reader can confirm or overturn the verdict on grounds they can inspect. That also answers the cost of review: effort goes to links that can be shown to hold or fail, rather than to an unbounded series of evaluators checking evaluators.
Two limits keep the scope modest. Support and inference come apart: even when the premises of a combined conclusion are supported, whether they warrant the conclusion needs its own check, distinct from looking up a factual claim. Separately, purposively selecting difficult paragraphs can discover defects, but the resulting count is not a prevalence estimate; that would require a defined population and a sampling design, and a defect proportion drawn from hard cases must not be reported as manuscript accuracy.
The remaining qualification is that evidence-tied agreement is still not proof of correctness. Drafter and judge can share a misreading of the same passage. Yet a shared error anchored to an accessible source can at least be exposed and corrected, while agreement generated by role labels alone cannot. Independence has to be built into what reviewers can check, not inferred from who they are.
24.4Checking atomic claims
The proposed workflow assigns three roles — drafter, model judge, accountable editor — but the names themselves establish no independence. What can make the judge useful is narrowing what it is asked. Rather than submit a mixed long answer and receive one binary verdict, break the passage into small factual claims and ask, for each, whether one accessible source passage supports it. Min et al. describe a metric that evaluates the support of smaller factual claims instead of reducing a mixed long answer to one binary label (Min et al., 2023). Localizing the question is what makes the error localizable.
Return to the stipulated illustration from 24.3.1: the drafter and judge accept an invented claim that a cited trial includes only outpatients, while the stipulated source includes both inpatients and outpatients. If the judge receives the paragraph as a whole and returns a single verdict, the defect sits inside a judgment that bundles many statements, so the verdict need not identify the sentence that departed from the source. If instead the judge receives one question — does this passage support the claim that the trial includes only outpatients? — the same defect becomes one answerable item tied to one accessible passage, and the accountable editor knows which claim to reject and rewrite. The example is an illustration only: it is not medical evidence, and it is not a measured estimate of how often drafter and judge errors coincide.
Two limits belong with this gain. First, support checking tells you whether asserted claims are backed; it does not tell you whether important perspectives or questions have been omitted. Second, as noted in 24.1.1, purposive selection can reveal defects worth fixing without producing a manuscript accuracy figure.
With atomic items marked as supported or unsupported, the text has a record of which small claims failed and where. Whether the supported premises actually warrant the author's combined conclusion is a separate inference check.
Decomposition creates a second check, not merely a larger number of checks. Once a passage is divided into claims, the judge should first ask whether each claim is supported by the accessible source passage assigned to it. That question is local: it compares one specified proposition with one specified passage. The invented claim that the cited trial includes only outpatients illustrates a failure of this kind, since the stipulated source includes both inpatients and outpatients. Naming a drafter, a judge, and an accountable editor does not make that comparison independent or reliable.
The relation among claims then needs separate inspection. Supported premises do not guarantee that the author’s conclusion follows. A judge may correctly verify several claims about the trial and still reject, or wrongly accept, the inference that combines them. If the premises are supported, the next question is whether they warrant the particular conclusion presented, rather than whether every sentence can be traced to a source. This distinction keeps local factual repair from being mistaken for validation of the argument.
The inference check also has a practical boundary. Selecting difficult paragraphs purposively can discover defects, and those discoveries are useful for improving the workflow. But the resulting proportion of defective passages is not a prevalence estimate for the manuscript. Converting it into accuracy would require a defined population of paragraphs and a sampling design. With the present method, this study can identify examples of failure and reason about their structure; the sample cannot be treated as evidence about how often such failures occur.
This boundary does not require endless evaluation of evaluators. It requires an accountable editor to record which claim was checked against which passage, and separately whether the supported claims warrant the stated conclusion.
24.5Adjudicating disagreement
When the drafter and the model judge both accept the invented claim that the cited trial includes only outpatients, their agreement settles nothing: the stipulated source passage includes inpatients and outpatients as well. What is needed at that point is not a further round of checking but an adjudication entry. The entry fixes three things: the contested sentence, the particular source passage being appealed to, and the reason why that passage warrants the revision rather than the original wording. The reason must be revisable by anyone who can reopen the passage; "the judge preferred it" records a preference, not an adjudication.
The entry should also keep the two checks distinct. A sentence may be exactly supported by a source passage and still be a poor premise for the conclusion the author draws from a bundle of such sentences. Recording the source for a factual dispute does not certify that the combined conclusion follows. Collapsing those two questions into one line is how a draft becomes harder to repair after the fact, because the reviewer can no longer tell whether the objection is to the evidence or to the inference.
Nothing here follows from the role names. A drafter, a model judge and an accountable editor are three positions, not three independent pieces of evidence; a stronger model can reproduce the same misreading of the same passage. What the record contributes is observable error checks: a later reader can see which sentence was at stake and whether the cited source actually says what the revision assumes.
That is a finite answer to the cost objection. The workflow does not require evaluating every evaluator in turn. It requires, when a claim is genuinely contested, one bounded entry that an accountable editor can accept or reject, with a reason grounded in a source rather than in the standing of whoever offered it.
When the drafter and the model judge both accept the invented claim that the cited trial includes only outpatients, their agreement carries no evidential weight: the stipulated source includes inpatients and outpatients. Two favourable verdicts would turn repetition into apparent corroboration, yet neither answers the question the dispute turns on. Model votes measure how often a claim has been accepted in this workflow, not whether the accessible evidence supports it.
The available source settles the bounded question, and it should be used for that purpose. The judge's task is narrow: does one specific claim find support in one accessible source passage? On that question the passage is decisive, because it contradicts the claim about the trial population. This is the familiar support distinction, applied to review: what a source says, and what follows from a set of claims, are separate questions requiring separate checks.
Correcting the population description therefore disposes of the factual dispute but does not vindicate anything built on it. The separate inference check asks whether the now-supported premises warrant their combined conclusion. If that conclusion depends on the trial being outpatient-only, it fails for a different reason than the claim itself, and fixing the sentence does not repair the argument. Keeping the two failures apart prevents a source-backed correction from being mistaken for a validated result.
The rule has a limit. It applies where an accessible source speaks directly to the disputed fact. Where the disagreement concerns the reach of an inference, or where sources genuinely conflict, no single passage decides it, and the accountable editor must resolve it on stated reasons. Nor does making the judge stronger make it independent: role names and model capability create no separation that observable checking does not. Where no source bears on the question at all, counting verdicts can record consensus, but consensus is all it records.
24.6Sampling expensive review
Costly review is best spent by deciding first which question the check is meant to answer. A purposive inspection of risky passages is a discovery exercise: the editor points the judge at paragraphs where an invented or overstated claim would do the most damage, and the question is whether serious defects can be found at all. The example already given illustrates the kind of defect in view. A drafter and a judge accepted an invented claim that a cited trial included only outpatients, although the stipulated source includes both inpatients and outpatients. That is an illustration of failure, not medical evidence and not a measured error correlation. It nevertheless earns such a paragraph a place in a purposive check.
Within that check the model's assignment stays narrow. The judge asks whether one specific claim is supported by one accessible source passage; a separate inference check asks whether the supported premises warrant their combined conclusion. Keeping these apart prevents a source match from certifying a whole paragraph. It still does not make the judge independent. A drafter, a model judge, and an accountable editor are components of a proposed workflow, and no independent validation follows from role names alone; the editor must own the judgment or commission a check outside the workflow.
A sample intended to estimate a manuscript-wide error rate is a different instrument. It needs a defined population — claims, paragraphs, or citations, stated in advance — and a sampling design that fixes how units come to review, because the resulting proportion is meant to speak for the units left uninspected. A purposive sweep may define its population clearly, but selecting risky passages alone does not justify extrapolating the observed defect fraction to the whole manuscript. Under a fixed budget, the defensible allocation depends on the goal: targeted checks can serve defect discovery, while estimating prevalence requires a design supporting that inference. The budget also requires an explicit stopping policy; cost alone does not determine when further checking has too little value. What it does not deliver is a manuscript accuracy figure, and the proportion of defects found among hand-picked passages must not be reported as one.
A purposive sample earns its keep when the aim is discovery. Suppose the editor selects paragraphs containing empirical claims, disputed attributions, or passages where several premises support one conclusion. Finding that the accepted claim about the cited trial concealed its inclusion of inpatients would be a serious result, regardless of how the paragraph was selected. Defect discovery does not require representative sampling: one demonstrated error is enough to expose a gap in the drafter–judge exchange and to require correction before sign-off.
What the selection cannot support is a manuscript-wide error rate. If ten difficult paragraphs yield four defects, four in ten is not the proportion of defective claims in the manuscript. The paragraphs were chosen because they looked difficult, so their defect rate is shaped by that choice. A prevalence estimate would need a defined population of reviewable claims or passages, a sampling design giving each item a known opportunity for selection, and consistent criteria for whether the judge has failed. Without those elements, the purposive proportion estimates only the purposive set.
The distinction matters even when review is expensive. Cost makes a full census unattractive; it does not license treating a convenience sample as representative. If resources permit only a limited audit, the accountable editor should report what that audit was for: testing particular risks or supplying evidence about the whole. A conditional sampling design—say, drawing claims at random within predefined strata—could later support inference to those strata, but that is a different procedure from selecting paragraphs because they raise concern.
Role separation remains necessary throughout. The model judge may catch the invented characterization, and the editor may correct it, yet neither act establishes independence merely because the judge is stronger than the drafter. The sample checks the workflow’s observable failures; it does not certify the workflow itself.
24.7The boundary of a passing verdict
A passing verdict should travel with the exact claim it covers, the source version examined, the criteria applied, and the dimensions left unresolved. Reporting only "pass" turns a narrow finding into an implied certification of everything nearby, which the workflow cannot support.
Consider the stipulated review: a drafter and a model judge both accept a claim that a cited trial enrolled only outpatients, although the accessible source includes inpatients and outpatients. This illustrates that agreement between roles can still leave an error in place; it is not measured evidence about how often such agreement fails. The judge's task here was narrow—one specific claim checked against one accessible source passage. A separate inference check asks whether supported premises warrant their combined conclusion. These are different objects, and a verdict over one does not settle the other.
A defensible report therefore names its scope explicitly. For the support check: the claim as drafted, the passage version used, the criterion that the passage substantiates the claim as stated, and the unresolved question of whether the combined argument follows. The accountable editor records that scope rather than absorbing it into an overall approval. Where a claim and its conclusion were checked by separate passes, the record should say which pass passed and which was not attempted.
Role names do not create independence. The drafter, the model judge, and the editor are separate positions in the workflow, but a stronger judge is not thereby independent of assumptions the drafter introduced or of sources the judge cannot reach. Observable checks—distinct tasks with distinct recorded verdicts—make those limits visible without requiring that every reviewer be free of shared error.
One qualification matters here: purposive review can expose serious defects, but a defect rate found among deliberately difficult paragraphs is not a prevalence estimate for the manuscript. A passing verdict reports what was checked against what, under which criteria, and leaves the rest open.
A passing verdict attaches to the object actually examined: one specific claim checked against one accessible source passage, under the stated criteria and at the stated version. It does not travel forward automatically. Once the source changes, or the text is substantively edited, the verdict's authority over the affected material lapses even if its wording sounds general.
The useful discipline is to separate what remains valid from what must be reopened. A change confined to a passage that the judge never consulted leaves the original finding untouched; the same is true, conditionally, of an edit that alters only formatting and cannot change what the claim asserts or what the source says. By contrast, revising the claim, replacing the cited passage, or sharpening the conclusion so that the combined inference differs takes the affected question outside the verdict. The separate inference check matters here: premises that were each supported may still fail to warrant a conclusion that has since been rewritten.
What survives such a change is not certification but a record. The record preserves the object, version, criteria and unresolved dimensions already reported, and it says precisely which support relation was tested. That is enough to re-run only the damaged portion rather than restart evaluation of the whole document, which is the practical answer to the cost of review without pretending that review can be skipped. The accountable editor then decides whether the damaged portion needs re-checking before release. If a workflow instead stores only the bare word pass, every later edit forces either a full repeat or an unjustified confidence in an object that no longer exists.
So a passing verdict is best treated as scoped evidence with inputs that can expire, not as a durable property of a manuscript. Its boundary is set jointly by the claim, the version checked, and the criteria applied.