A study can preregister twenty outcomes and still produce a misleading headline if it celebrates whichever outcome happens to cross a conventional significance threshold. Planning the measurements is only part of planning the inference. Researchers also need to define which claims belong together and what error rate the decision procedure will control.
TL;DR: Specify the hypothesis family, primary claims, time points, contrasts, and multiplicity procedure before examining results. Familywise error control and false discovery rate control serve different purposes. Report effect sizes and uncertainty for the full planned family, distinguish exploratory discoveries from confirmation, and document amendments with the information available when they were made.
File zero: what would count as a successful study?
Consider a fictional education study evaluating a new training program. Its questionnaire contains twenty outcome scales covering knowledge, confidence, persistence, collaboration, and satisfaction. The investigators initially write that the program will be considered effective if any outcome has a two-sided p-value below 0.05. They plan to measure each scale at the end of training and again three months later.
That success rule creates many opportunities for a favorable result. Even before considering subgroups or alternative analyses, twenty scales at two time points can produce forty distinct tests. The investigators have not yet said whether the headline concerns a particular learning outcome, a broad pattern across domains, or evidence that at least one measured quantity changed.
The distinction is scientific rather than clerical. A program that increases satisfaction but leaves knowledge unchanged may be useful, but it does not establish the same claim as improved learning. A family of tests should be connected to the claims that readers might accept and the decisions that follow from them. Merely grouping columns by their position in a spreadsheet does not provide that rationale.
The revised protocol therefore starts with a sentence describing the intended confirmatory claim. In this example, the main claim concerns knowledge retained at three months. A small set of additional claims concerns confidence and collaboration, while the remaining outcomes support exploration and descriptive interpretation. This prioritization is invented for the case; another study could legitimately choose a different scientific structure.
The FDA's guidance on multiple endpoints addresses the problem in a regulated clinical context, including how multiple opportunities for conclusions can increase erroneous claims. The general lesson about specifying objectives and endpoint structure is useful here, although an education study does not inherit clinical regulatory requirements simply by citing that document. FDA multiple-endpoint guidance
The protocol now contains a map from each planned claim to its outcome, time point, contrast, analysis population, and test. This map allows a reviewer to see which choices could generate the same headline. It also makes clear which measurements are valuable without being independent routes to declaring the program successful.
File one: the twenty-test calculation
Suppose, solely for illustration, that twenty null hypotheses are all true, each test has a five percent false positive probability, and the tests are independent. The probability of no false positive is 0.95 raised to the twentieth power, approximately 0.358. The probability of at least one is therefore about 0.642, or sixty-four percent.
This is not a claim that every real twenty-outcome study has a sixty-four percent error rate. Outcomes are often correlated, tests may have different properties, and some null hypotheses may be false. The calculation isolates the consequence of repeated opportunities under a simple set of assumptions. It explains why a five percent threshold for each test does not imply a five percent chance of any false claim across the family.
The expected number of false positives in that all-null example is one: twenty times 0.05. This expectation does not require independence when each individual false positive probability is five percent. An expected count of one and a probability of at least one near sixty-four percent describe different features of the same hypothetical experiment.
A reader should not interpret a p-value of 0.04 as a ninety-six percent probability that the program works. A p-value concerns the compatibility of the observed test statistic with a specified null model and testing procedure. It does not by itself supply the probability that a hypothesis is true, the size of an effect, or the chance that a selected headline will replicate.
Selection makes the reporting problem worse. If the investigators show only the smallest p-value, readers lose the denominator of opportunities that produced it. Publishing a table of all twenty results helps, but interpreting only the favorable row can still mislead. The planned family needs to remain visible in both analysis and narrative.
File two: familywise protection is a decision rule
A simple Bonferroni procedure tests each of twenty hypotheses at 0.05 divided by twenty, or 0.0025, to control the probability of one or more false rejections in the family at no more than five percent, provided the individual tests are valid. The basic guarantee does not require the tests to be independent.
The procedure can be conservative, particularly when many tests are strongly related. That does not make multiplicity disappear. It creates a design question about whether a more suitable valid procedure can use the structure of the hypotheses and their dependence. The choice should be justified before the observed results reveal which procedure yields the most rejections.
Holm's sequential procedure offers another way to control familywise error. It orders p-values from smallest to largest and compares them with progressively less stringent thresholds, stopping when a comparison fails. The original paper develops this sequentially rejective approach. Holm's original paper
Use a five-test example with ordered p-values of 0.001, 0.012, 0.018, 0.040, and 0.200. At a familywise level of 0.05, the first threshold is 0.010, so the first test passes. The second threshold is 0.0125, so the second also passes. The third threshold is approximately 0.0167, so the third fails and the procedure stops.
| Ordered test | P-value | Holm threshold at that step | Result in this example |
|---|---|---|---|
| First | 0.001 | 0.0100 | Reject |
| Second | 0.012 | 0.0125 | Reject |
| Third | 0.018 | About 0.0167 | Stop |
| Fourth | 0.040 | Not reached | Do not reject |
| Fifth | 0.200 | Not reached | Do not reject |
Later steps cannot be resumed after the procedure has stopped, even if some later comparison would otherwise look favorable. More generally, the stepwise rule is a complete algorithm, not a collection of thresholds from which the analyst can choose independently. Here the first two hypotheses are rejected, while the remaining three are not.
For clarity, the report can provide adjusted p-values alongside raw p-values and effect estimates, naming the family and method. It should not call an adjusted p-value the probability that a result is false. Adjustment changes the testing procedure's error control; it does not turn a frequentist test into a posterior probability for an individual claim.
File three: discovery screening asks for a different guarantee
The research team also wants to explore a large set of potential mechanisms for a future study. In that setting, it may be willing to tolerate some false discoveries among a useful collection of leads. False discovery rate control addresses the expected proportion of false rejections among all rejections, conventionally treating that proportion as zero when there are no rejections.
This is different from controlling the probability of even one false rejection. The distinction matters for the decision. A confirmatory claim that triggers adoption may require stronger familywise protection, while a screening exercise followed by independent validation may use a different error target. Neither criterion is inherently superior without a description of the consequences.
Benjamini and Hochberg introduced a widely used procedure for controlling the false discovery rate under independence, with later work studying additional dependence settings. A researcher should verify that the chosen procedure's conditions fit the analysis rather than assuming that every correlated collection of p-values automatically qualifies. Benjamini–Hochberg original paper
For the same five ordered p-values, consider a target false discovery rate of 0.05. The step-up thresholds are 0.010, 0.020, 0.030, 0.040, and 0.050. The largest rank satisfying its comparison is the fourth, whose p-value equals 0.040. The procedure rejects the first four hypotheses in this illustrative calculation.
That differs from Holm's two rejections because the procedures control different error criteria. It is not evidence that one discovered the truth and the other missed it. Choosing between them after seeing this contrast would undermine the purpose of having a prespecified claim and error target.
An FDR target of five percent does not guarantee that exactly five percent of the discoveries in this one study are false. The realized false discovery proportion can vary, and the true status of hypotheses is generally unknown. The guarantee is an expectation under the method's assumptions across the relevant repeated-study framework.
The exploratory report therefore labels the selected results as leads for further investigation. It preserves the full candidate set and the selection rule, so the next study knows what was screened. Independent confirmation should not quietly reuse the same observations that made a lead attractive in the first place.
File four: the family follows the claim across time and subgroups
The original protocol's two follow-up times create a temptation. If immediate knowledge improves but retained knowledge does not, the investigators might headline the immediate result. If only the later result improves, they might describe durable learning. Both are potentially interesting, but the confirmatory success rule cannot be rewritten to fit whichever time point looks favorable.
A family definition should therefore specify whether time points represent separate claims, a single longitudinal contrast, or a hierarchy with a primary time point. The statistical model and multiplicity strategy must match that choice. Collecting repeated measurements does not automatically make every comparison part of one test merely because they are analyzed in the same software command.
Subgroups add another layer. The team might examine effects by prior experience, age band, training location, and baseline score. Evidence that an effect is statistically significant in one subgroup but not another does not by itself establish that subgroup effects differ. A direct interaction or other appropriate contrast addresses the difference between them.
The subgroup definitions also need to precede the results when they support confirmatory claims. Searching many cutoffs for baseline score creates more opportunities than a single predeclared split. The report should not present the selected cutoff as a natural boundary if it was chosen because it produced the strongest contrast.
Different analysis populations can generate multiplicity too. Complete cases, all randomized participants under a specified missing-data method, highly engaged participants, and participants who completed every lesson do not necessarily estimate the same quantity. Switching populations until a favorable result appears is not a harmless robustness check.
A genuine sensitivity analysis asks whether the main conclusion changes under plausible alternative assumptions. Its purpose is to examine fragility, not to provide additional tickets in a lottery for significance. The report should explain why each sensitivity was run and preserve results that weaken the preferred conclusion as well as those that strengthen it.
File five: a hierarchy can encode scientific priorities
The revised education protocol could use a fixed sequence: test retained knowledge first, then confidence, then collaboration, continuing only when the previous hypothesis is rejected under the specified procedure. Such a hierarchy can support familywise error control under appropriate conditions, but its order must represent genuine scientific priorities rather than an ordering chosen after results are known.
If the first test fails, later outcomes can still be estimated and discussed. The hierarchy limits confirmatory rejection under that procedure; it does not erase the data or prohibit exploration. The distinction should be visible in the report so readers understand why a small raw p-value later in the sequence is not being presented as a protected confirmatory finding.
A hierarchy also has a scientific cost. A potentially useful effect on collaboration might receive no confirmatory declaration because knowledge did not improve. If that would be unacceptable for the intended decision, the investigators should choose a different valid design before data collection. Statistical efficiency cannot substitute for agreement about what the study is trying to establish.
A composite outcome is another possible design choice, but it changes the question. Combining knowledge, confidence, and satisfaction into one score may reduce the number of tests while making the effect harder to interpret. The weights, scale construction, and meaning of improvement need substantive justification. A composite should not be created after the individual results are inspected merely to obtain a favorable aggregate.
The protocol should also consider power under its actual multiplicity strategy. Planning sample size for a single unadjusted test and later imposing a much stricter familywise threshold may leave important claims poorly supported. Conversely, measuring fewer, better justified primary outcomes can concentrate resources on the questions that matter most.
File six: an amendment records what was known when
Halfway through recruitment, the investigators discover that one collaboration scale has a scoring error in the published questionnaire instructions. They need to correct the analysis plan. Preregistration should make that change transparent, not force them to knowingly use a defective measure.
The amendment records the original plan, the problem, the proposed correction, the date, and what data or results the decision-makers had seen. A correction made without access to comparative outcomes has a different evidential context from a change made after discovering that the original analysis was unfavorable. Both can be reported honestly, but they should not be described identically.
The Center for Open Science recommends preserving the distinction between planned and data-driven analyses and documenting changes. Its guidance explicitly allows appropriate amendments while warning against rewriting a registration after results are known to make an analysis appear confirmatory. Preregistration guidance
The team keeps the original registration accessible and links the amendment rather than silently replacing the old document. Where meaningful, it reports the originally planned analysis alongside the corrected one and explains why the correction is preferable. If the original analysis is genuinely invalid, the report says so instead of treating two incomparable numbers as equal alternatives.
An amendment is not a device for shrinking the family after disappointing outcomes appear. Removing eighteen unsuccessful tests and declaring that only the remaining two ever mattered would conceal the actual opportunity for selection. The relevant record is the sequence of decisions and information, not merely the final polished protocol.
The same principle applies to unplanned discoveries. An unexpected association can be scientifically valuable. Labeling it exploratory preserves its value while making clear that the current data helped generate the hypothesis. The next step can be an independent confirmatory study with a precisely defined outcome and analysis.
File seven: report estimates before celebrating thresholds
A complete results table shows each planned outcome's scale, estimated contrast, uncertainty interval, raw p-value where used, and multiplicity-adjusted result under the specified procedure. It identifies missing observations and deviations from the analysis plan. Readers can then assess magnitude and consistency rather than receiving a list of significant and nonsignificant labels.
Confidence intervals need a clear description. Ordinary pointwise intervals do not automatically provide simultaneous coverage for an entire family. If the report uses simultaneous intervals, it should name the method and the family they cover. If it uses pointwise intervals for descriptive precision, it should not imply that all intervals jointly have the same nominal coverage.
The effect size also needs a meaningful reference. A very small improvement can cross a stringent threshold in a large study while having little practical value. A potentially important effect can remain uncertain in a small study. The scientific conclusion should connect the estimate and uncertainty to the adoption decision, not substitute a binary testing result for that reasoning.
The headline must reflect the protected claim. If retained knowledge did not meet the confirmatory criterion but an exploratory satisfaction score improved, the report should describe that pattern directly. It should not announce that the program improves learning and leave the distinction to a supplementary table that few readers will see.
Replication plans should preserve the original discovery's selection history. A follow-up study testing one selected outcome is not automatically equivalent to a study that had always targeted that outcome. The new data can provide independent evidence, while the earlier evidence remains part of a selection process that should be understood when combining results.
An unplanned interim look creates another opportunity
The research director asks to inspect the knowledge result after every fifty participants and stop recruitment as soon as the p-value falls below 0.05. That request changes the testing procedure. A fixed-sample p-value calculation does not automatically preserve its advertised error rate when the stopping rule repeatedly searches for a favorable crossing.
The solution is not to prohibit every interim analysis. A study can plan sequential monitoring with an appropriate method, or use another framework whose validity covers the intended observation and stopping process. The important requirement is to include those opportunities in the design. A multiplicity adjustment across outcomes alone does not necessarily address repeated looks over time.
The amendment file should distinguish operational monitoring from outcome-driven stopping. Checking recruitment counts, data completeness, or instrument failures may be necessary to run the study. Inspecting treatment contrasts to decide when to stop is a different use of information. Access controls and reporting responsibilities can help preserve that distinction, but the statistical plan still needs to match what actually happened.
Valid p-values come before adjusted p-values
No multiple-testing procedure can reliably repair individual tests whose assumptions are badly wrong. If the analysis ignores the unit of randomization, uses an invalid variance estimate, or selects a favorable model after inspecting outcomes, multiplying the resulting p-values by twenty does not restore the promised familywise guarantee. Adjustment operates on inputs that must themselves satisfy the relevant validity conditions.
The education study therefore checks assignment, outcome coding, missingness assumptions, and model specification before applying its chosen procedure. A small p-value caused by duplicated records or a scoring mistake is not a discovery that merely needs stronger correction. It is a data or analysis problem requiring repair and transparent documentation.
This also explains why counting visible table rows may underestimate the effective search. Suppose an analyst tries several transformations, alternative exclusion rules, and different covariate sets, then reports only the best result for each outcome. The twenty displayed p-values have already been selected from a larger analysis process. Treating them as twenty untouched prespecified tests misrepresents how they were obtained.
A useful audit trail records the analyses run and their purpose. It does not require publishing every temporary debugging command as a scientific result. It does require preserving consequential choices that could change the claim, especially when those choices were informed by the observed effect. Transparency makes it possible to separate legitimate model repair from selective inference.
The final protocol has fewer ambiguities than measurements
The study can still collect twenty outcomes. Rich measurement is useful for understanding mechanisms, unintended consequences, and future research directions. The improvement is that the protocol no longer gives every favorable fluctuation equal authority to establish the main claim.
Its confirmatory family is tied to specific decisions, its exploratory analyses have an appropriate discovery framework, and its amendments preserve the history of what researchers knew. The report presents all planned outcomes with their practical meaning and uncertainty. Readers can distinguish a supported primary conclusion from a promising lead without having to infer the difference from selective emphasis.
Multiplicity is therefore handled most effectively before the p-values exist. The essential work is deciding which claims the study intends to protect, how much error those decisions can tolerate, and what evidence would change the conclusion. A correction formula becomes useful once that scientific structure is clear.