Chapter 1. Who Decides, by When, and What Reopens It

The texts answer the paper's three questions unevenly. They say exactly who spends and who reports, but not who decides that a result is progress. They set dates for programmes and reports, not for results. And they write down many conditions for looking again at a result, while seldom saying whose duty it is to look. This chapter first says how the texts were chosen and read, then takes the questions in turn.

1.1Data and method

The paper reads ten texts, chosen by one rule: every text saved for this study on cancer research and its evaluation whose licence allows it to be quoted, that is, every work of the United States government (which is not subject to copyright) and every article published under the Creative Commons Attribution licence. The list, with the licence and the evidence for it, is kept beside the texts in a manifest.

Four texts are government works. One is two sections of the 21st Century Cures Act of 2016: section 1001, which set up an account to fund innovation projects at the National Institutes of Health, cancer research among them, and which carries the name of the Beau Biden Cancer Moonshot; and section 2011, which encourages a Precision Medicine Initiative. Two are pages of the National Cancer Institute, on the Cancer Moonshot and on the National Cancer Plan. One is a page of the FDA's Oncology Center of Excellence on Project Optimus, an initiative on how doses of cancer drugs are chosen. The pages were read as they stood when they were retrieved, as text; the paper says nothing about earlier or later versions.

Six texts are articles under CC BY 4.0, each read in full from PubMed Central: those of Kemp and Prasad, Haslam and Prasad, Hackshaw and others, Wong and others, Sammut and others, and Rojas and others, introduced above. One of them, Hackshaw and others, states that three of its authors were employed by the study's funder; the paper reports the article's findings as the article states them and adds nothing to them. Other works are cited only from their abstracts, as the bibliography lists them, and nothing is said of them beyond what an abstract says.

Each text was read for three things: the person or body it names as deciding whether a result is acted on; any date or period it sets; and any condition it writes down under which a result or a decision about it should be looked at again. Every block quote is a passage of a saved text, word for word, and its source line names the file. Every number in the prose stands in a saved text and is listed, with its file and place, in a numbers file. Goals, forecasts and hopes are given as their authors' statements. Findings about drugs, tests and models are reported as the articles report them; the paper makes no claim about any product's value to patients.

The texts are not a sample of anything. They are documents through which one country's legislature, its cancer institute and its drug regulator have spoken about cancer research in the last decade, with six scholarly voices beside them. What they show is how the question is framed in those documents, not how any result was in fact weighed.

1.2Who decides

The statute gives the money to one office and the oversight to another, and it judges progress by a plan. Section 1001 of the Cures Act mentions the FDA only once, in a project on regenerative medicine, and it does not speak of approving anything. It sets up an account in the Treasury and directs the Director of the National Institutes of Health to spend what is appropriated from it on named projects. One of them is cancer research, described in a sentence that the printed law breaks across two pages, so that it is quoted here in two pieces:

Three things in this sentence matter for the paper's question. The list of kinds of research, vaccines, diagnostic tests, immunotherapy, combination therapies, reads today like a list of the articles this paper read; the statute named the bets before the results arrived. The phrase "inherently higher risk" tells the spender that some projects are expected to fail, so a failed project is not in itself a breach. And the ceiling, $1,800,000,000, is the only measure of the project the sentence gives. The statute says how much may be spent, not what the spending must produce.

The law does say how the spending will be judged. Before spending, the Director must seek advice:

After spending, the Director reports each year to four committees of Congress, and the report must say, among other things:

This is the statute's only test of progress, and it is a test of direction rather than of outcome. A project passes if it advances priorities that the National Institutes of Health set for itself in a plan. Nothing in the section asks whether any patient lived longer. The decider of progress, in the law's sense, is the Director, advised by the Advisory Committee to the Director and answerable to committees that receive reports. The section on precision medicine follows the same pattern. It lists what the initiative may include and closes the list with discretion:

The programme pages describe how the Moonshot's goals were set inside that frame. The National Cancer Institute did not adopt a panel's recommendations as law; it used them as a guide and chose its own goals:

The goals named are to accelerate discovery, increase collaboration and expand data sharing. They are goals about the research enterprise, and the page's list of accomplishments matches them: networks built, a data commons constructed, an atlas of tumours, a study of new screening tests begun. This study reads the page as answering the question "what counts as progress?" with what the programme set out to build, which is a fair answer for a programme and a different answer from the one a patient or a regulator would give.

For the decision that a drug's result is good enough to reach patients, the texts point elsewhere, to the FDA, and here the saved government texts are almost silent. The one FDA page in the set concerns doses, not approvals. It describes a habit the agency wants to change:

The sentence does not say that the FDA chooses doses; it says that the paradigm it describes leaves doses poorly understood before the trials meant to support approval. The remedy the page lists is persuasion and early contact: communicating expectations through guidance and meetings, and encouraging developers to meet the review divisions well before those trials. The regulator, on its own page, appears as a body that shapes evidence before it is gathered as much as one that judges it afterwards.

What the regulator does at the moment of approval is described in the saved texts only by the articles, and Kemp and Prasad describe it as a choice between two routes:

The phrase in inner quotation marks is the hinge. "Reasonably likely to predict" is a standard of judgment, not of measurement. It asks a person to judge that a surrogate will, more likely than not in some loose sense, foretell what patients care about. Kemp and Prasad's point is that this judgment is often made without the studies that would support it: of the drugs on the faster route that they counted, from a study they cite, most had no trial-level validation study of the surrogate. Whatever one makes of their argument, it shows where the decision sits. It sits with reviewers applying a phrase whose meaning they must fix case by case.

A second phrase widens the room further. The faster route is meant for diseases with few good options, and Kemp and Prasad find that this condition, too, is a matter of judgment:

Two judgments, then, open the door: that a need is unmet, and that a measure is reasonably likely to predict benefit. The texts this paper read do not say who defines either one, beyond the agency itself. Kemp and Prasad report a study which found the first phrase applied in some cases to cancers with many recommended treatments and high survival; they read this as overuse. This study reads it more narrowly, as evidence that the phrase does not, by itself, bind the person who uses it.

The remaining articles address readers they do not name with any precision: policy makers, journalists and physicians in one, developers, investors, policymakers and patients in another. The texts as a whole thus describe a chain in which money is decided by a named official, progress in the statute's sense is judged against the official's own plan, results reach patients through a regulator whose standards rest on phrases it interprets, and the evidence about all of these is produced by scholars who can advise everyone and bind no one.

1.3By when

Every date in the government texts falls on a programme or a report. None falls on a result. The statute counts in fiscal years. It transfers named sums to the account for each of the fiscal years from 2017 to 2026, it sets a deadline for the Director's first plan, and it gives itself an end:

The sunset is the clearest date in the set, and it had already passed when this paper was written. The paper read only the enacted text and says nothing about later appropriations or later law. What the sunset shows, within the text, is the shape of the statute's time. Money flows for ten fiscal years, a plan is due within 180 days, reports are due each year by October 1, and then, by its terms, the section expires. The cancer research line itself runs out sooner: its schedule lists sums for the fiscal years 2017 to 2023 and none after. Within that schedule, a project can be judged only by what it shows inside the years it is funded, and the statute's own test, whether a project advances the plan's priorities, can be answered within a year. The statute does not ask anything that would take longer to know.

The programme's own goal was faster still. The Moonshot set out to make a decade's worth of progress in five years, a goal stated by the Institute in its words and measured, on its page, in projects and programmes supported rather than in outcomes. The page reports what the Institute counts as done:

The sentence says the projects have helped to achieve the goals the Institute set; it does not say that a decade's progress was made in five years, or what that progress would look like. The goal's deadline is not paired in the text with any measure by which it could be found met or missed.

The articles set no deadlines, but they say how long the evidence takes, and their answers are much longer than the programme's clocks. Wong and others estimate how long trials last:

If the figure is taken as the authors give it, a median cancer programme spends longer in the clinic than the Moonshot's five years and longer than the ten fiscal years of the statute's account. A programme funded in its first year would not, at the median, see its drug through the three phases of trials before the funding schedule ended. This study claims nothing about any particular project; it notes only that the texts' clocks and the evidence's clock run at different speeds, and that no text reconciles them.

The articles on new methods say the same of their own evidence. Hackshaw and others, writing about a blood test meant to find many cancers, place their modelling in the gap before the decisive trials:

The sentence is a small decision record of its own: it names what would settle the question, randomised trials, says when, in several years, and calls its own result preliminary until then. It does not say what anyone should do in the meantime, and neither does this paper. Kemp and Prasad show why the waiting matters for surrogates in particular. A surrogate is used to save time, but proving that it can be trusted also takes time:

Read beside the statute, the sentence describes a trap of timing. A programme with a fixed horizon wants results it can see within the horizon. Surrogates offer them. But the validation that would make a surrogate trustworthy takes years that the horizon may not contain. Wong and others add a caution about their own most recent numbers that shows the same problem from the side of measurement:

A recent result looks better than it is because the failures have not yet had time to be recorded. The authors give the reason: too little time had passed for their method to conclude that a trial had failed. This study reads the caution as general. Any result judged close to its own date carries the same bias, because the evidence against it arrives later than the evidence for it.

1.4What would reopen the decision

The texts write down many conditions under which a result, or a decision about it, should be looked at again. They are specific and observable. What the texts seldom do is bind anyone to act on them.

The statute's conditions are reports. Each year the Director must report the money spent and whether projects advance the plan, and committees may ask for further updates and reports. These are occasions for review, and they are real: a committee that reads a report can act. But the condition the reports test, alignment with a plan, is not one that a cancer result could fail by turning out not to help patients. The statute writes no condition of that kind.

The articles write many. Kemp and Prasad set out the condition that, in their view, should accompany any decision made on a surrogate, and they state it twice, in their abstract and their conclusions:

The condition is notable for its form. It does not ask for the answer before the decision; it asks that the means of getting the answer be in place before the decision. A study measuring survival must already have its patients. If it does, the decision can be revisited when it reports. If it does not, nothing is waiting to reopen the decision. The same article reports how often the reopening fails in practice:

The two sentences describe two different failures. In the first, the evidence that would reopen the decision is never gathered. In the second, it is gathered and the decision stands anyway. Kemp and Prasad give, from a study they cite, a count for the second: of 36 cancer drugs approved on a surrogate between 2004 and 2008, 18 did not improve survival in later trials, and only one of the drugs without clear benefit in survival or quality of life had had its authorisation removed at the time they wrote. These are their figures and their reading. What they show for this paper's question is that a condition for revisiting can be written, met and still not acted on, because the decision to act on it belongs to someone who is not bound by it.

Haslam and Prasad meet the same fact from the other side, as a choice of method. They included drugs approved on the faster route in their count of eligible patients, and they explain why:

The authors treat the persistence of approvals as a fact of the world to be modelled, not a failure to be fixed. This study reads the sentence as the authors' observation that reopening an approval by withdrawal is uncommon.

The articles on new methods write their reopening conditions into their own limits. Hackshaw and others name the outcome they did not measure:

Whether finding more cancers counts as progress depends, on the authors' account, on whether finding them earlier means fewer deaths, and that is exactly the number the study could not give. A related validation study, E. A. Klein and others, Clinical validation of a targeted methylation-based multi-cancer early detection test using an independent validation set (2021), by its abstract reports how often such a test was right in a separate set of people and concludes that it is feasible as a complement to existing screening. This study reads that conclusion as one about detection, the step before the number Hackshaw and others could not give. The authors draw the consequence themselves, in a sentence that reads as a condition on any decision to act:

The sentence is the authors' statement, made in an article of which, it says, three authors are employed by the funding source, though it calls the study a collaborative academic study, and it limits the authors' own result. The programme texts show that a public body took up the same open question. Among the Moonshot's accomplishments the Institute lists a study designed to answer it:

Here, for once, a reopening condition and a body charged with testing it meet in the texts. The question the article left open, whether earlier detection reduces deaths, is the question the Institute's study is designed to approach. Even so, the sentence is careful about its reach: the study will assess feasibility, of using such tests in larger trials, which would then assess benefit and harm. The decision whether the tests are progress is put off by two steps, and the texts give no date for either.

The two articles that report new methods on patients state their own conditions most plainly. Sammut and others tested their model on patients from other cohorts, and they report where it weakened:

The sentence names, in the language of the method, the condition under which a model's result should be looked at again: a population that differs from the one the model learned from. The authors report that their integrated model predicted response to treatment in the external cohort of 75 patients, and they say that such models may enable molecular and digital pathology to inform therapy choice in future clinical trials; that is their claim, in their words, and the paper takes no view on it. The reopening condition they write is the more general finding: a model is as good as the match between the patients it learned from and the patients it meets.

Rojas and others, reporting a first trial of a vaccine made for each patient, are the most explicit of all about what their result does not yet show:

The authors call their observations preliminary, and the trial's main measure, by its own description, was safety; it treated 16 patients. What they ask for is more study, and they report that a larger randomised trial was about to begin. The decision record their article implies has a clear shape: the result is acted on by doing the next trial, and the next trial is the condition for revisiting. That shape is the one Kemp and Prasad recommend for surrogates, arrived at by authors writing about a different kind of result.

Across the articles, then, the conditions for reopening fall into three kinds. There are unfinished studies: confirmatory trials, randomised trials of screening, larger trials of a vaccine. There are untested populations: the people a model did not learn from, the people a screening study did not include, the patients a small trial did not enrol. And there are unmeasured outcomes: deaths prevented, survival gained, harms avoided. Each kind is something a reader of the texts could observe. None of the texts, except in the single case of the Institute's screening study, names a body that must observe it by a given date.