8.1Choosing a relevance variable
An encoding does not announce what it is for. In the executed construction E01, the four pairs of independent fair bits are equiprobable, and the compressor C=a keeps one bit while discarding the other. If the relevance target is a, then C preserves it completely: C and a have identical information, so recovery from the code alone is exact. If the relevance target is b, the same code supplies no information at all; the best correct binary guess succeeds with probability one half. The full original pair can likewise be reconstructed from C alone with probability only one half, since the discarded bit must be guessed. Nothing in C's statistics distinguishes these verdicts. Their difference comes from which variable was designated as relevant.
A relevance target, then, specifies the variable an encoding is meant to preserve. In a hypothetical inquiry, it is the person who poses the task that states this target: framing a question marks which variable must survive processing. Consider a stipulated archive first organized to answer opening-hour questions and later asked about entrance conditions. The person asking the opening-hour question makes opening hours the relevant variable for that task; the later person asking about entrance conditions makes a different variable relevant. No real organizer, mandate, or measured human information value is asserted here. The initial task choice is revisable, because a changed inquiry can require preservation that the earlier organization did not provide.
If both bit targets were stipulated to require exact recovery from the code alone, retaining both bits would meet that larger toy requirement, at the price of a code entropy of two bits rather than one. A reader who retains original access to the data occupies a different information position from one receiving only the code.
The limit of the example is correspondingly narrow: it is a two-bit mathematical construction, not a measurement of human meaning or political representation. Even so, it shows why “worth keeping” cannot be inferred from the code itself. It must be specified through a task, and that specification is where relevance gets chosen.
Once a task is stated, its relevance variable marks what the encoding is expected to preserve, and that is enough to evaluate an existing code. In the two-bit construction E01, take the independent fair bits \(X=(a,b)\) and the compressor \(C=a\). Against target \(a\), the code is sufficient: it preserves all one bit of target information. Against target \(b\), it supplies no information at all, so the best probability of a correct binary guess from the code alone remains one half. The same compression therefore succeeds or fails according to the target against which it is assessed. If both bits must be recoverable exactly from the code alone, retaining both meets that toy requirement at a larger code entropy, two bits rather than one. A reader with access to the original pair has a different information set from one given only \(C\), and the comparison must state which applies.
This makes target choice consequential before anyone settles every dispute about it. Suppose, stipulatively, that an archive was first organized to answer opening-hour questions and later receives a task about entrance conditions. The initial choice of target was useful for the first task but does not answer the second; it can be revisited when the task changes. No real organizer, mandate, or constituency is asserted by this example.
The inferential step is modest but important: usefulness under a stated task does not confer universal authorization. A librarian, investigator, or user may provisionally specify a relevance variable so that exploratory work can proceed without waiting for general agreement. That provisional target licenses conclusions only within the stated scope of application. If later stakeholders formulate a different task, the encoding must be assessed again, and one perfect preservation claim will not settle what their task requires. Values enter here because accepting a target amounts to deciding which question the representation is meant to answer; agreement on the target can remain open even while a provisional one does useful work.
8.2The information bottleneck proposal
The information-bottleneck proposal states a precise task for a compressed representation: reduce the description while retaining information about a specified relevance variable (Tishby, Pereira and Bialek, 2000 arXiv deposit). The proposal therefore begins once a target has been named. Its aim is intelligible without an agreed theory of meaning or a completed stakeholder process: compress, and keep what matters for the stated variable.
The two-bit construction shows what "keep" and "lose" mean in this setting. Take two independent fair bits, \(X=(a,b)\), and a compressor \(C=a\). If the specified target is \(a\), the code preserves it completely: \(I(C;a)=1\) bit, and \(a\) is recovered exactly from \(C\). If the specified target is \(b\), the same code preserves nothing: \(I(C;b)=0\) bits. A reader told only that \(C=a\), and asked to guess \(b\) correctly, succeeds with probability one half — no better than guessing without the code. The representation has not degraded; it has simply been organized around one target rather than the other.
This is the proposal's practical appeal and its discipline at once. A single compression can be entirely adequate for one relevance variable and worthless for another, so the label attached to the target is doing real work in the assessment. That work is distinct from the compression itself: an objective can be satisfied for its declared variable while leaving a different, needed variable unserved. Changing the target to \(b\) would expose the loss; the same code would not succeed under that new criterion.
One qualification belongs here. These are fair bits and exact entropies from a two-bit mathematical example, not measurements of human meaning or political representation. They establish what perfect preservation and total loss look like under a specified target. They do not decide which target deserves specification.
Choosing a target is a separate decision from optimizing a code for it. Once a relevance target has been named, the proposal can ask which compressed representation preserves the most information about that target while discarding the rest. That question is answerable relative to the target. The proposal does not tell a reader why one target rather than another should govern what is kept, and no amount of refinement inside the optimization supplies that reason.
The executed two-bit construction marks the division. Let X be the pair (a,b) of independent fair bits, and let C retain a. The code is sufficient for a and carries no information about b, so the best binary guess at b from C alone succeeds with probability one half. That contrast turns on the specification of relevance: the code's arithmetic is indifferent between the two targets. If instead both bits must be recovered exactly from the code alone, retaining both meets that requirement at a larger code entropy. A reader given access to the original pair has a different information set, so what the code alone must support is itself a question to settle.
Consider, as a stipulated case rather than a report, an archive first organized to answer questions about opening hours that later receives a request about entrance conditions. The earlier task choice is revisable; no organizer or mandate is implied. Either task can be assigned a target, and optimization can improve a code for it. What it cannot do is decide which questions the archive should be prepared to answer.
This leaves room for exploratory work without prior agreement. A provisional target can be adopted and revised as the questions sharpen. The point is not that consensus must precede every inquiry; it is that revising the target changes what counts as the right representation.
8.3A two-target counterexample
A synthetic mathematical example makes the point exact. Let X be a pair (a,b) of independent bits, each taking the values 0 or 1 with equal probability, so that the four states (0,0), (0,1), (1,0), and (1,1) are equiprobable. A compressor C retains a and discards b. Nothing here measures human meaning or political representation; it is a two-bit construction designed to separate two questions that a single notion of relevance tends to blur together.
With the standard entropy definitions, the original variable has entropy two bits, the compressed variable one bit, and one bit of uncertainty about X remains once C is known. In symbols, I(C; a) = 1 bit, since a is retained without error. The mutual information between C and b is zero, because C is built solely from a and a was chosen independently of b. The representation is therefore neither uniformly informative nor uniformly lossy: it preserves one bit about a and nothing about b, though both coordinates occupy the same original state.
The operational consequence is small and concrete. An observer holding C who must guess b has no better than a one-half chance of being right, the same performance an unweighted fair coin would give. The same observer can report a with certainty. Adequacy thus depends entirely on which variable is taken as the relevance target, and the two targets cannot be satisfied together by this compression.
This dependence is the counterexample to treating compression as self-interpreting. In the information bottleneck formulation, the objective is to preserve information about a specified relevance variable (Tishby, Pereira and Bialek, 2000 arXiv deposit); the mathematics does not supply that variable, and the optimization objective alone does not establish which target a public inquiry should adopt. A verdict about adequacy therefore needs a declared target. When alternative targets express competing priorities, choosing among them also requires a value judgment. The example neither quantifies political value nor ranks a and b. It shows only that a single representation can be perfect for one target and empty for another.
The counterexample has a practical edge once a target is allowed to change. Suppose, conditionally, that the code C = a was built to answer a question about a, and that a recipient later needs the independent bit b. At that point the compressed bit contains no information that improves prediction of the new target. Because a and b are independent, nothing inferable from C raises the chance of a correct binary guess of b above one half. The new question is not underspecified or ambiguously phrased. It is simply about information the code was never required to carry.
What would restore the option is the original information that the compression discarded: access to the pair X = (a, b), or equivalently retention of b alongside a. The full pair has entropy of two bits, against one bit for C. A reader given access to the original therefore occupies a different information position from a reader given the code alone: the first can recover b exactly, the second cannot improve on guessing. Retaining both bits meets the stronger requirement that either target be answerable exactly from the retained representation, but it does so at the larger code entropy. There is a cost, and the choice of what to pay it for is exactly where values enter: someone must say which questions count.
The scope of this point should be kept tight. This is a two-bit mathematical construction, not a measurement of human meaning or of political representation. It shows nothing about which targets a real institution should adopt, and it does not establish that every possible future question deserves retention in advance. Its claim is narrower and firmer: a representation perfect for a declared target can leave a later, independent target unanswerable from that representation alone, and no amount of reanalysis of the compressed bit recovers what was dropped.
8.4Unknown future questions
Suppose the recipient's need shifts from target a to target b. The code C = a remains exactly what it was, and for b it carries no information: I(C;b) = 0 bits, so the best correct binary guess succeeds with probability one half. What restores the lost option is access to the original pair X = (a, b), and specifically to the b coordinate that the reduction discarded. The compressed bit alone cannot do it.
That failure of the compressed bit to carry information about the new target matters whenever the relevance target is not yet fixed. Unknown future questions therefore give a reason to keep an accessible original alongside the task-specific reduction, rather than treating the reduction as the sole record. Stipulate an archive first organized to answer opening-hour questions that later receives a task about entrance conditions. Nothing here asserts a real organizer or mandate; the point is only that the initial task choice is revisable. Requiring agreement on a target before anything is retained would block precisely the exploratory work that later questions may reward. Preservation keeps the option of applying a criterion that the first reduction could not have anticipated.
The qualification is that this argument establishes an option, not its sufficiency. In E01, if both bit targets require exact recovery from the code alone, retaining both bits meets that toy requirement at a code entropy of 2 bits rather than 1. A reader given access to the original has a different information set from a reader given only C: the former can recover b, the latter cannot. Still, the construction is a two-bit mathematical example, not a measurement of human meaning or political representation. What it supports is therefore conditional: when future targets may differ from the one that shaped the reduction, an accessible original preserves choices that a single task-specific record necessarily forecloses. Whether that preservation is affordable and genuinely usable is a separate question, and it is the one that follows.
Preservation is therefore valuable as insurance, but it is not a guarantee. Keeping an original alongside a task-specific reduction consumes more resources than keeping the reduction alone, and its usefulness depends on access: who can reach the retained material, under what conditions, and with what ability to compare it against the reduced form. These are practical limits, not deductions from information theory.
The two-bit construction makes the scope precise. With four equiprobable pairs and the code \(C=a\), the code retains one bit of the original two. For target \(a\), that is sufficient. For target \(b\), it supplies no information: the best correct binary guess remains at one half. Keeping \(X=(a,b)\) instead preserves both bits, at an entropy of two bits rather than one. A reader with access to the original therefore has a different information set from a reader given only the code.
Yet retaining \(X\) still cannot guarantee that every future interpretation will be recoverable. The guarantee is relative to a specified target and to the distinctions retained. If a later question turns on a feature that the archive never organized or that its access rules prevent being consulted, the original does not resolve the matter by mere availability. In the stipulated archive case, the collection was first organized around opening-hour questions and later receives a task about entrance conditions. The original task choice is revisable, and the case asserts no real organizer or mandate.
So the scope of preservation is conditional: retain originals when the cost and access burden are justified, while accepting that target selection remains necessary. The two-bit example is a mathematical illustration, not a measurement of human meaning or political representation.
8.5Several targets and visible tradeoffs
Retention answers a question only after someone has specified which distinctions matter. That specification can be revised, but until it exists, "preserved" has no determinate content. A single code can be perfectly adequate for one target and useless for another.
Take the executed two-bit construction E01. The four pairs X=(a,b), with a and b independent fair bits, are equiprobable, and the code is C=a. The code carries 1 bit while the pair carries 2, leaving 1 bit of X unknown given C. Relative to target a, the code is sufficient: I(C;a)=1 bit, so a is recovered exactly. Relative to target b, the code carries no information at all: I(C;b)=0 bits. Anyone required to guess b from the code alone is right with probability one half, no better than a fair coin. Nothing about the compression malfunctioned; the retained distinction matched one target and discarded the other.
The conflict is thus visible without inventing a welfare score. Both targets are specified in advance, and the same 1-bit code ranks differently under them. Choosing a target is already choosing which distinctions to keep. As a stipulated example, an archive organized first to answer opening-hour questions and later given a task about entrance conditions has made such a choice, and that choice is revisable; no real organizer, mandate, or outcome is asserted.
One limit governs this result. E01 is a two-bit mathematical construction, not a measurement of human meaning or political representation. Its lesson is structural: distinct targets may require different retained distinctions from the same source. It supplies no values that decide which target should govern, and it settles nothing about who is entitled to make that specification. That is the live question left open by showing the tradeoff rather than pricing it.
In E01 the remedy is easy to state. If both targets must be recovered exactly from the code alone, the compressor can retain X itself: both bits are then preserved, and the code entropy rises from one bit to two. The extra storage is the visible price of refusing to choose between a and b at the representation stage. Nothing in the construction decides which target deserves preference; that is supplied by whoever specifies the question being asked.
The remedy has a precise scope. It works here because the four pairs are discrete and equiprobable, and because the extra distinction needed for b can simply be carried along. Conditionally, where the relevant distinctions are recordable and the cost of carrying them is accepted, a larger code can preserve several targets at once; where a distinction is unavailable to the representation, or its cost falls to a party with no reason to accept it, more storage does not settle the matter. Nor is size the only variable: a reader given access to the original X has a different information set from a reader who receives C alone, so the figures in E01 describe recovery from C alone.
The stipulated archive keeps the point practical without pretending to measure human meaning. Organized first to answer opening-hour questions, it later receives a task about entrance conditions; its initial task choice is revisable. Demanding agreement on a final target before any work begins could therefore block the discovery of useful distinctions. Yet even a two-bit case shows that every retained distinction quietly answers the question of which information is worth keeping, and that enlarging the representation is one available answer rather than a general escape from tradeoffs.
8.6Exploration before fixed objectives
Before an objective is fixed, a summary need not be judged against a target that no one has yet named. Its provisional work is more modest: to make candidate targets visible. The two-bit construction shows why the order matters. With C = a, the code supports a fully, by one bit of mutual information. It supports b not at all, by zero bits; a reader guessing b from the code alone is right with probability one half. The same representation is sufficient for one question and uninformative for another, so the code cannot choose between them. Target selection is where value enters, and the arithmetic does not tell a reader which target ought to matter; a public, an institution, or a policy must supply that judgment.
Suppose an archive was first organized to answer opening-hour questions and later receives a task about entrance conditions. Nothing here asserts a real organizer or mandate; the stipulated case is about the order of inquiry. The first task choice was revisable, and the later request brings a second candidate target into view. A summary assembled under the first question can help make that second question articulate, even though no one specified it in advance. The encounter supplies a candidate worth naming, which is a different contribution from preserving a target already chosen.
Requiring settled agreement before any summarization would therefore have a cost: the entrance-condition question would have to be recognized before there was anything organized around opening hours. Exploration can come first because a compact view can expose targets before an objective is fixed. Someone must still select among the exposed candidates and decide what counts as worth keeping; that judgment is not produced by information theory. Its limit is the narrow scope of this two-bit construction.
Once exploratory summaries are judged as tools for discovering questions, the later arrival of a target does not turn them into failures. In the two-bit construction E01, the code C=a preserves a perfectly: I(C;a)=1 bit. It supplies no information about b: I(C;b)=0 bits, so the best binary guess of b from C alone succeeds with probability one half. If b is subsequently selected as the target, that shortfall is genuine relative to b, but it was not a shortfall against any criterion then operative. The record says that the summary was sufficient for a, uninformative for b, and that b became the question only later.
The scope of this correction is narrow. E01 is a two-bit mathematical construction, not a measurement of human meaning or political representation; no measured human information values enter the argument. The archive case is stipulated for the same reason: suppose an archive is first organized to answer opening-hour questions and later receives a task about entrance conditions. Its initial task choice is revisable, with no real organizer or mandate asserted. Calling the earlier organization a failure would import a criterion it never had. Judging it instead as a device that surfaced questions keeps the record faithful to the order in which targets arose.
What this does not establish is that summaries are target-neutral. If entrance conditions had been fixed from the start, the verdict would run the other way: the code must then be judged against that target from the outset. Nor is preservation costless in the toy case, where requiring exact recovery of both bits from the code alone is met only by retaining both, at a code entropy of two bits rather than one; a reader given original access to the pair has a different information set. The qualification that matters is temporal: a target fixed after the summary licenses discovery, not retrospective failure marking. Changing the target itself must therefore be recorded rather than silently assumed.
8.7Revising the target openly
A revised target should be recorded as an addition alongside the target it supersedes, together with the reason for the change. Three entries belong in the record: what was previously counted as worth preserving, what is now wanted, and why the change was made. Without the middle entry the revision silently replaces one assignment with another; without the third it cannot be assessed as a considered choice rather than a convenient afterthought.
The two-bit construction shows why the reason matters. With four equiprobable pairs X=(a,b) and compressor C=a, the code uses one bit and preserves a perfectly: I(C;a)=1 bit. It preserves b not at all: I(C;b)=0 bits, so a reader relying only on the code guesses b correctly with probability one half. Nothing in the mathematics selects a as the relevant target. That selection is an evaluative commitment, and it can change when the questions change.
The stipulated archive supplies the pattern without asserting any real organizer. It was first arranged to answer opening-hour questions and later receives a task about entrance conditions. The proper record does not say that the opening-hour arrangement was mistaken. It says that a new task has been adopted, for the stated reason that entrance conditions now need answering, and it leaves the earlier arrangement visible as what it was.
The limit is equally important: revising a target is not permission to rewrite the original one retroactively, nor to present performance under the new target as confirmation of success under the old. The change is a new declaration, and it carries its own demand on the representation. If both bit targets had to be recovered exactly from the code alone, retaining both bits would meet that toy requirement at a larger code entropy—two bits rather than one—while a reader with original access would face a different information set. Either way, the two-bit case remains a mathematical example, not a measurement of human meaning or political representation. Which scores may be compared across the two assignments is a further question.
Recording the earlier target, the revised one, and the reason for the change is only the bookkeeping half of revision. The other half concerns the numbers already collected. In E01 the four pairs X = (a,b) are equiprobable and the compressor C = a emits one bit. The figure H_C = 1 bit is the same whether the declared relevance target is a or b. Under target a the unchanged figure sits on a success: I(C;a) = 1 bit, so a is recovered exactly from the code. Under target b the same figure sits on no help at all: I(C;b) = 0 bits, and with no other information a binary guess of b is correct with probability one half. The score has not moved; the claim of success has.
That is why a comparison must travel with the task it answers. Two runs presented as retaining one bit with no loss are not comparable if one scored a and the other scored b: identical numbers certify sufficiency in the first case and nothing in the second. Each score must stay paired with the target it measured. If both targets instead required exact recovery from the code alone, C = a would fail that requirement; retaining both bits would meet it at larger code entropy, with H_X = 2 bits.
The stipulated archive shows the scope of the point. It was organized to answer opening-hour questions and later receives a task about entrance conditions. No real organizer or mandate is asserted here; the point is only that a score computed for the first task may not answer the second. When the archive's target is revised, the earlier score should remain attached to the earlier question, and any re-scoring should be recorded as answering the new one.
None of this requires agreement on a target before useful exploratory work can begin. Provisional organization may proceed without a settled target; the restriction falls on comparison and on claims of success, which need the target each score answered to remain visible. The construction itself stops short of such cases: it is a two-bit mathematical example, not a measurement of human meaning or political representation.