A Cutoff Is Not a Causal Argument: Put Your RD Study on Trial

TL;DR

Regression discontinuity can identify a local causal effect when treatment changes at a known score cutoff and the relevant counterfactual outcomes behave continuously there. The cutoff alone supplies neither assumption nor proof. Investigate how scores were assigned, what else changed at the threshold, whether treatment was actually received, and how outcomes were measured. A credible study can still answer only a narrow question about people near that particular boundary. This article puts a constructed grant evaluation through a design hearing before interpreting its result.

Case file: a grant at seventy points

A regional training program assigns applicants a continuously measured assessment score. Applicants scoring at least seventy become eligible for a grant that covers course materials. An evaluator wants to know whether eligibility increases course completion within six months. The proposed report compares completion just above and below seventy and attributes any jump to the grant.

The program and all numerical examples in this article are fictional. They are used to distinguish assumptions from observations, not to report an empirical effect. The hearing is an editorial device: each objection exposes evidence the evaluator would need before accepting the causal interpretation. The underlying methods come from the primary research and author-maintained resources linked below.

The starting design has three named objects: the assessment score, the eligibility threshold, and the intervention. Cattaneo, Idrobo, and Titiunik's Foundations explains the canonical sharp regression-discontinuity design and its continuity-based interpretation. In a sharp design, treatment follows the threshold rule exactly. Actual grant programs may not satisfy that condition, so eligibility and receipt must initially be recorded separately.

The initial estimand is the effect of eligibility at the seventy-point boundary. It is not the average effect for every applicant, an effect of raising scores, or the effect of replacing the whole training program. Those distinctions should appear in the research question before anyone selects a regression specification.

The evaluator should assemble the original rule, the score-generation protocol, dated applicant records, payment records, and an outcome definition. A spreadsheet containing only final scores and completion flags is not enough to reconstruct the assignment mechanism. The causal argument depends on how the threshold became consequential in the real institution.

The numerical claim submitted to the hearing

Suppose an illustrative outcome schedule without a grant rises by one percentage point for each additional assessment point near seventy. Its completion rate is fifty-eight percent at sixty-eight and sixty percent at seventy. Suppose receiving the grant adds six percentage points locally. At seventy-two, the treated completion rate would then be sixty-eight percent.

Comparing the two observed positions, sixty-eight and seventy-two, yields a ten-percentage-point difference. Six points come from the constructed treatment effect; four come from the underlying score trend. The desired discontinuity compares the two sides at the same boundary, where the hypothetical rates are sixty and sixty-six percent. It is not simply the difference between two nearby group means.

These stipulated schedules explain the estimand. They do not show that a real dataset follows either line, and they do not supply a sampling uncertainty estimate. The hearing now asks whether the real assignment process would justify interpreting such an estimated boundary jump causally.

Objection one: applicants can choose which side they occupy

The first witness is the assessment administrator. Are scores generated before applicants learn the threshold? Can applicants retake the assessment? Can staff adjust a score after seeing whether it qualifies? Are appeals handled differently for applicants near seventy? Each answer concerns the comparability of people immediately on opposite sides.

Imagine applicants with strong personal connections can obtain a manual adjustment from sixty-nine to seventy. They may also have better employment networks, more support, or more ability to complete a course. A jump in outcomes could reflect a change in the composition of applicants, rather than the grant's effect. A small numerical score difference does not remove that institutional selection.

McCrary's manipulation-test paper develops a density-based diagnostic motivated by sorting around a threshold. For this case, an unexpected accumulation immediately above seventy would prompt investigation. It would not identify which applicants moved or establish how large the resulting outcome bias is.

Conversely, an inconclusive density test is not proof that applicants could not sort. Limited data, rounding, or offsetting movements can make a diagnostic uninformative. The causal argument should combine score-process evidence with empirical checks, rather than turning a nonsignificant test into a certificate of random assignment.

Request the first recorded score, the final eligibility score, timestamps, retake history, and appeal decisions. If the final score is the actual assignment variable, analyzing the first score as though it determined eligibility changes the design. The first score may still reveal the adjustment process, but it cannot silently substitute for the operational rule.

There is a useful distinction between imperfect control and exact control. Applicants can prepare for an assessment without necessarily controlling whether they land just above the threshold. The relevant institutional question is whether the process permits precise sorting that undermines comparability at the boundary. A general statement that motivation affects scores does not, by itself, settle that question.

The hearing's provisional ruling is therefore evidence-dependent. A documented scoring procedure with limited discretion supports further analysis. A discretionary override targeted at the threshold requires a revised identification argument. No choice of bandwidth repairs an assignment process the study has misdescribed.

Objection two: the threshold changes more than a grant

The program director reveals that eligible applicants also receive priority tutoring appointments. The analyst now faces a different treatment. If both benefits start at seventy, an outcome jump can describe the effect of the eligibility package under suitable assumptions. It does not isolate the effect of money for materials.

A second policy could make the problem subtler. Suppose an employer uses the same score to decide whom to interview, independently of the training program. Crossing seventy may then alter job prospects and the incentive to finish the course. Labeling the observed difference a grant effect would omit another mechanism that changes at exactly the same boundary.

Lee and Lemieux's review of regression-discontinuity designs discusses identification, implementation, and the limits of interpreting these designs. Applied here, the key continuity question concerns the outcomes applicants would have had under each treatment state. The observed outcome can jump because the treatment changes; the counterfactual relationship cannot contain an unrelated jump that the argument ignores.

Ask for a threshold map, not just a treatment column. It should list every administrative, financial, informational, and access change triggered by seventy. Include communications: a congratulatory eligibility message can alter expectations even before any money arrives. Whether that message belongs inside the treatment definition depends on the research question.

Participants can also share materials or tutoring advice with applicants below the threshold. For this program, the evaluator should investigate those interactions before interpreting eligibility as an isolated individual intervention. The comparison may reflect access under a shared training environment, which differs from delivering the package in a setting where participants cannot influence one another.

This investigation can improve the study without discrediting it. The evaluator may revise the question to “What is the effect of becoming eligible for this combined support package near seventy?” That is a legitimate, narrower description of the intervention the institution actually delivers. It is often more useful to decision-makers than an apparently precise estimate attached to a nonexistent isolated grant.

If the funder specifically needs the effect of money apart from tutoring, the existing cutoff may not identify it. The next design could randomize an additional benefit among otherwise comparable eligible applicants, or exploit a separate policy change with an appropriate argument. The response is to obtain variation that separates mechanisms, rather than to add control variables mechanically until the preferred label seems plausible.

Objection three: the graph manufactures the jump

The evaluator presents a smooth curve fitted across the entire score distribution. It bends sharply near seventy and displays an appealing discontinuity. The statistician asks whether the reported effect comes from observations close to the threshold or from assumptions about the shape of the relationship far away.

Gelman and Imbens' paper on high-order polynomials explains problems with global high-order polynomial approaches in regression-discontinuity analysis. Its recommendation favors local low-order approaches over an increasingly elaborate global curve. For the grant study, the graph should expose the local data rather than use mathematical flexibility to create visual confidence.

Bandwidth is the neighborhood around the cutoff used for local estimation. A narrower neighborhood compares more similar scores but includes fewer observations. A wider neighborhood supplies more data while requiring an adequate approximation over a larger region. These consequences do not make the narrowest or widest choice automatically correct.

Consider the hypothetical ten-point raw difference submitted earlier. Averaging everyone from sixty to sixty-nine against everyone from seventy to seventy-nine would mix a considerable score gradient into the comparison. A local model attempts to account for the score relationship while approaching the threshold from each side. It should not be described as simply comparing “almost identical” people in two broad bins.

The author-maintained rdrobust repository documents local-polynomial estimation, data-driven bandwidth selection, and robust bias-corrected inference. Such tools help implement an accepted statistical procedure. They do not decide whether the score was manipulated, whether tutoring also starts at seventy, or whether the target outcome is measured consistently.

Prespecify a primary analysis and a limited set of substantively defensible sensitivity checks. Show the effective observations on each side and the bandwidths used. Do not try dozens of windows and present only the one producing a favorable p-value. The uncertainty statement must account for the estimation method, rather than treating a chosen smooth curve as the known outcome relationship.

A binned plot is an exhibit, not the result itself. Bin widths affect what the eye sees, while estimation choices affect the numerical discontinuity and its uncertainty. Present both transparently. If a conclusion depends on one visually convenient binning scheme, the hearing needs more than a striking before-and-after picture.

A score recorded as an integer needs a different defense

Suppose the archive contains only whole-number scores. There are no observations at 69.99 or 70.01, however many applicants the program enrolls. The usual image of getting arbitrarily close to the boundary no longer describes the observed support. More applicants at the same few values do not fill the gaps between those values.

The evaluator should investigate whether underlying unrounded scores exist and actually governed eligibility. If only discrete scores determined treatment, the method and identifying assumptions need to address that structure. Cattaneo and colleagues' Extensions treats discrete-score designs as an extension requiring explicit attention, rather than a trivial formatting issue.

Deleting all applicants with a score of seventy is not an automatic repair. It removes information exactly where the rule changes and introduces extrapolation across a wider gap. Any such exclusion needs a reason tied to the assignment process and an account of what parameter remains identifiable. A tidy plot after removing inconvenient observations does not supply that account.

Objection four: eligibility and receipt are being confused

Payment records show that some eligible applicants decline the grant, while a small number below the cutoff receive emergency assistance. The original sharp-treatment description is now wrong. The threshold can still change the probability of receiving assistance, but the outcome discontinuity first describes the effect of eligibility, not a simple treated-versus-untreated contrast.

Suppose grant receipt jumps from twenty percent just below seventy to eighty percent just above it, a sixty-percentage-point first stage. Suppose the completion outcome jumps by six percentage points. Dividing 0.06 by 0.60 gives 0.10. Under the additional assumptions for a fuzzy design, that ratio can identify a local effect for applicants whose receipt changes because of eligibility.

The assumptions matter as much as the arithmetic. The Extensions monograph discusses fuzzy designs, including the role of exclusion and monotonicity conditions. Eligibility must not affect the outcome through a separate path omitted from the treatment definition, and the relevant monotonicity argument must rule out a group whose receipt moves in the opposite direction because of eligibility.

Return to the priority tutoring discovered earlier. If eligibility changes tutoring independently of grant receipt, the ratio cannot simply be called the effect of receiving money. The numerator includes a pathway that the denominator does not represent. Dividing two well-estimated jumps does not remove this conceptual mismatch.

The ten-percentage-point ratio is also not the effect for all grant recipients. Some would receive support regardless of the cutoff; others would not receive it on either side. The local complier interpretation concerns the applicants whose receipt responds to eligibility near this boundary. It should not be exported to every participant who happens to receive a grant.

Report the first-stage estimate and its uncertainty beside the outcome discontinuity. If the threshold barely changes receipt, the ratio can become unstable. A weak administrative rule is not strengthened by describing the eligibility cutoff with confidence. The evaluator may conclude that the policy's effect on eligibility is estimable while the effect of actual receipt remains too uncertain for the intended claim.

Objection five: completion is easier to observe on one side

The data manager explains that grant recipients must submit a completion certificate to close their payment file. Applicants without grants have no equivalent reporting requirement. An apparent jump in recorded completion could therefore reflect a jump in documentation, even if actual learning behavior were unchanged.

The outcome definition must use a comparable source across the boundary. Course-provider administrative records may supply one option if they cover both groups under the same rules. A follow-up survey may supply another, but its response process needs examination. The hearing should establish what a recorded zero means before treating it as failure to complete.

For this constructed program, distinguish “no completion record located” from “confirmed noncompletion” in the data investigation. The final analysis may require a consistent operational definition, but collapsing those states early hides a measurement problem. The topic here is comparability at the cutoff; statistical sophistication cannot compensate for a reporting obligation that changes with treatment.

Predetermined outcomes can be informative exhibits. If the same kind of certification gap appears in records from before eligibility was assigned, the analyst should investigate the administrative system. A jump in a pre-treatment characteristic or outcome can challenge the proposed design, though a smooth pattern in a few observed variables cannot establish continuity for every unobserved counterfactual.

The research record should preserve the data lineage: which organization supplied the outcome, when it was measured, how people were linked, and which linkage failures occurred on either side. These details let reviewers distinguish an effect of the support package from an effect of the institution's record-keeping rules.

Objection six: a local answer is being sold as a national policy

Suppose the hearing accepts the assignment mechanism, the measurement process, and a precisely estimated positive eligibility effect near seventy. The policy team asks whether giving the same package to everyone scoring above forty would produce the same benefit. The study does not answer that question merely because its estimate is credible.

Applicants near forty may face different barriers, need different amounts of support, or respond differently to tutoring. Expanding the program can also change provider capacity, waiting times, and interactions among participants. The effect of a marginal eligibility change at seventy need not equal the effect of a broad expansion under a different delivery system.

Describe the target population using substantive characteristics as well as scores. What does seventy represent in the assessment? Which applicants cluster near it? How large is the grant relative to their material costs? These descriptions help decision-makers judge whether a new population resembles the one for which the local evidence is relevant, without pretending resemblance proves transportability.

Cost-effectiveness introduces another distinction. A completion-rate effect is not itself a cost-per-additional-completion estimate for an expanded program. The latter needs the relevant incremental costs, the number of people whose treatment changes, and assumptions about scale. The original cutoff evidence can inform that calculation while leaving several inputs unresolved.

The evaluator should therefore offer a bounded next decision. Maintain the current eligibility rule while investigating another margin, pilot an expansion with a design suited to that population, or gather evidence about capacity before national rollout. A narrow causal result can support useful action precisely because it states what remains unknown.

Objection seven: the decision requires more precision than the study has

The funder says an increase of at least five percentage points would justify continuing the package at its current cost. The evaluator reports an illustrative estimate of six percentage points. Without an uncertainty statement, those two numbers appear to settle the decision. They do not show how clearly the data distinguish the relevant possibilities.

Imagine the analysis instead reports an interval from negative two to positive fourteen percentage points. These endpoints are hypothetical and are not calculated from the earlier outcome schedule. They describe a possible reporting problem: an estimate can be centered above the policy threshold while remaining compatible with effects well below and well above it. The decision-maker needs to see that ambiguity.

Conversely, a narrowly estimated increase of two percentage points might be statistically distinguishable from zero yet fall short of the funder's stated requirement. Statistical detection and the policy's minimum useful effect are different comparisons. State the substantive threshold before viewing results, and explain whether it concerns percentage points, a relative change, or a different outcome altogether.

In the constructed sharp case, moving from sixty to sixty-six percent is an increase of six percentage points and a ten percent relative increase. Calling it a sixty-six percent improvement would confuse the outcome level with its change. In the fuzzy case, the ten-percentage-point receipt-effect illustration is yet another quantity. The memo must keep these units and targets separate.

If the sample cannot support the required precision, the study can still reveal an implementation problem or provide provisional evidence. The honest next step is to assess whether more cohorts, a longer collection period, or a different design could resolve the decision. A positive point estimate should not be inflated into certainty merely because the program needs an answer this funding cycle.

Keep a record of what changed during the hearing

A useful design record contains more than the final accepted specification. In this case it should show that the initial grant-only question became a package-eligibility question after tutoring was discovered. It should also show whether integer scores replaced the assumed continuous scores, or whether payment records changed a sharp design into a fuzzy one.

These revisions are substantive findings about the institution. Preserve the original question and the reason for each change so another analyst can reconstruct why the accepted claim has its particular scope. That record prevents later presentations from drifting back toward the simpler, stronger claim that first attracted attention.

Separate changes motivated by administrative evidence from changes motivated by an inconvenient estimate. Discovering that the outcome source covers only recipients is a design defect that needs attention regardless of the estimated effect. Choosing a new bandwidth solely because it raises the point estimate has a different justification and should not be disguised as the same kind of correction.

The final file should allow an independent reader to regenerate the population, apply the eligibility rule, trace receipt, and reproduce the outcome construction before fitting any model. That order puts the causal argument ahead of the statistical output. It also makes disagreement productive: a reviewer can identify a particular rule or assumption instead of accepting or rejecting the study as a whole.

The decision memo: accept a claim with a perimeter

The hearing should end with a statement of the accepted intervention, outcome, population, and identifying assumptions. Attach the score-process evidence, the threshold map, the primary estimation choices, and the most consequential unresolved objection. A favorable estimate belongs inside that perimeter.

For the fictional grant, the most defensible conclusion might concern eligibility for a combined support package among applicants near seventy. It would become a receipt effect only with an additional valid fuzzy-design argument. Neither result would establish the benefit of changing the threshold to forty. That perimeter is the causal finding's useful shape, not a weakness to hide in a footnote.