A well-run experiment can establish an effect for its participants without establishing the same average effect for the people who will receive the program next. Transporting evidence requires a target population, a description of relevant differences, and assumptions connecting the two settings. A larger original trial does not remove that requirement.
TL;DR: Define who the new decision concerns, keep the intervention and outcome comparable, and identify effect modifiers that differ between the study and target populations. Standardization or weighting can adjust observed differences under suitable assumptions, but neither creates evidence for absent groups or repairs an unmeasured change in how the program works. This dossier uses two fictional cities and explicitly hypothetical effects.
Dossier cover: one program, two cities
City Harbor ran a randomized trial of a tutoring program for adult learners. The outcome was a standardized assessment score measured three months after enrollment. City Ridge now wants to adopt the program. Harbor's trial is internally credible under its design and follow-up assumptions, but Ridge's learners have a different distribution of prior preparation and face different constraints on attendance.
The policy team initially asks whether the Harbor study is generalizable. That question is too broad to answer usefully. Generalizable to whom, under which delivery model, against which alternative, and over what period? The dossier replaces it with a specific target: eligible adults entering Ridge's municipal learning centers during the next enrollment cycle.
The intervention is also specified. It includes the tutoring curriculum, contact hours, tutor training, access arrangements, and the support that helps participants attend. The comparator is Ridge's existing program, not an undefined absence of tutoring. If Harbor compared its intervention with a different alternative, the desired contrast may already differ before any population adjustment begins.
The outcome must retain meaning across cities. A score collected through a supervised assessment in Harbor and an optional online quiz in Ridge would not automatically measure the same construct with the same observation process. The team should harmonize measurement or explicitly model the difference rather than treating identical column names as proof of comparability.
Transportability research formalizes the conditions under which experimental information from one setting can inform another. Bareinboim and Pearl's data-fusion framework emphasizes that assumptions about how environments differ are part of the identification problem. The source study's credibility and the target study's relevance must both be examined. Causal inference and data fusion
The dossier therefore has two distinct evidentiary questions. First, what does the Harbor trial support about the intervention among its participants? Second, what additional information and assumptions allow that evidence to inform Ridge's decision? Keeping the questions separate prevents enthusiasm for a strong experiment from becoming an unsupported claim about every future implementation.
Exhibit one: a different population mix changes the average
Suppose the program's average effect is ten score points among learners with less prior preparation and two points among learners with more preparation. Assume for this worked example that these subgroup effects are known without uncertainty and would be the same in both cities under the specified intervention. Those are simplifying assumptions, not empirical findings.
Harbor's trial population is twenty percent less prepared and eighty percent more prepared. Its average effect is therefore 0.20 times ten plus 0.80 times two, or 3.6 points. Ridge's target population is seventy percent less prepared and thirty percent more prepared. Applying the same subgroup effects to Ridge gives 0.70 times ten plus 0.30 times two, or 7.6 points.
| Preparation group | Assumed effect | Harbor share | Ridge share |
|---|---|---|---|
| Less prior preparation | 10 points | 20 percent | 70 percent |
| More prior preparation | 2 points | 80 percent | 30 percent |
| Population average | Depends on mix | 3.6 points | 7.6 points |
The difference between 3.6 and 7.6 points does not mean the randomized trial was biased for its own population. It means the average is a property of both subgroup effects and the distribution of people being averaged. An internally valid estimate can be inappropriate for a different target average without any flaw in randomization.
This calculation is standardization in its simplest form: estimate effects within relevant groups and average them using the target population's distribution. In practice, subgroup effects are estimated with uncertainty, variables may be continuous, and models may be required. The simple table makes the assumptions visible before those complications arrive.
The key assumption is not merely that preparation is measured in both datasets. It is that, after accounting for the relevant variables, the effect being transported is sufficiently comparable between trial participants and the target population. Unmeasured differences in support, motivation, or delivery could break that connection even if preparation distributions are matched perfectly.
Methods for extending randomized trial inferences describe conditional exchangeability and positivity requirements alongside the trial's own treatment-assignment assumptions. These conditions explain what a population adjustment can and cannot establish. Study designs for extending trial inferences
Exhibit two: weighting reproduces the same arithmetic
The same two-group example can be expressed with weights. To make Harbor's preparation mix resemble Ridge's, less-prepared trial participants receive a relative weight of 0.70 divided by 0.20, or 3.5. More-prepared participants receive 0.30 divided by 0.80, or 0.375. The weighted shares become seventy and thirty percent.
If Harbor has one thousand participants, two hundred less-prepared learners contribute a total weight of seven hundred, while eight hundred more-prepared learners contribute three hundred. The total weight remains one thousand in this simplified construction. Applying the subgroup effects again produces the same 7.6-point target average.
These numbers illustrate a density-ratio weighting idea, not a universal recipe for every sampling design. Actual participation weights depend on how trial and target data were sampled and which population is being targeted. A probability of trial participation estimated from a convenience sample can have a different interpretation from one estimated in a known population frame.
The weights also reveal an information cost. Using the common descriptive effective-sample-size formula, the square of the sum of weights divided by the sum of squared weights, this example yields approximately 390.2. The denominator is two hundred times 3.5 squared plus eight hundred times 0.375 squared, or 2,562.5. One million divided by that number gives the result.
This is a diagnostic for weight concentration, not a substitute for the analysis's variance calculation. It shows that a nominal sample of one thousand does not retain the same information when a small subgroup carries much more influence. A target estimate can be relevant yet imprecise because the source study contains little evidence about the people most common in the target.
The team should inspect the weight distribution, the support of important covariates, and the influence of individual observations. A mean weight near one does not establish healthy overlap. A few very large weights can coexist with that mean and dominate the result. Diagnostics should examine the distribution and the substantive identity of the highly weighted participants.
Exhibit three: an empty subgroup cannot be repaired by a large weight
Now suppose Harbor excluded learners who required evening sessions, while half of Ridge's eligible population can attend only in the evening. If attendance constraints are relevant to how the program affects outcomes, the trial contains no direct support for a major part of the target population. There is no finite reweighting of observed daytime learners that creates evening-only participants.
This is a positivity or support problem relative to the variables needed for the transport assumption. A model can still output a prediction for the absent group, but that prediction relies on extrapolation. The report should distinguish model-based extrapolation from estimates supported by comparable trial participants.
The policy team has several options. It can narrow the target claim to learners whose circumstances overlap with the trial, gather new evidence in the missing group, or make an explicitly assumption-dependent extrapolation and assess its sensitivity. What it should not do is quietly remove the unsupported group from the analysis while continuing to describe the result as applying to all eligible Ridge learners.
Restricting the target is sometimes the most honest immediate decision, but it changes who benefits from the evidence. A study that supports daytime learners may be useful for a partial rollout while leaving an important equity and service question unresolved. The dossier should show both the supported population and the excluded share so that the limitation remains visible to decision-makers.
Near-empty groups create a softer version of the same problem. If Harbor includes only a handful of learners with a particular combination of preparation, language, and attendance constraints, extreme weights may result. Trimming or stabilizing weights can improve numerical behavior, but it may change the target or introduce a bias-variance tradeoff that needs explanation.
There is no universal maximum acceptable weight that solves this scientific question. The team needs to understand why the weight is large, whether the corresponding participants are representative of the target subgroup, and how much the conclusion depends on them. A convenient numerical threshold should not replace a discussion of who is missing from the evidence.
A population difference matters through the causal question
Not every difference between Harbor and Ridge requires adjustment. A variable can differ substantially between cities without modifying the intervention effect on the chosen scale. Conversely, a modest difference in an important effect modifier can matter greatly. The variable-selection process should use substantive knowledge and the estimand rather than only a table of statistically significant baseline differences.
In the worked example, preparation matters because the assumed effect differs between preparation groups. If the effect were exactly five points in both groups, changing their proportions would not change the average effect on that additive score scale. The demographic difference would still exist, but it would not explain a change in the average effect under that assumption.
Effect modification depends on scale. For a binary outcome, a constant risk ratio can produce different risk differences when baseline risks differ. Suppose an intervention halves a risk from twenty percent to ten percent in one population and from four percent to two percent in another. The risk ratio is 0.5 in both, while the absolute reductions are ten and two percentage points.
Transporting a relative effect therefore does not automatically transport an absolute benefit. A decision about expected numbers of successful learners, prevented failures, or avoided costs may require target baseline risks as well as an effect measure. The analysis should state which quantity is assumed stable and what additional information is needed to calculate the decision-relevant quantity.
Dahabreh and colleagues examine transport assumptions for relative effect measures specifically, illustrating that the chosen scale is part of the identification strategy. It is not merely a reporting preference applied after an estimate has been obtained. Transporting relative effect measures
The dossier records a causal rationale for each proposed modifier. This does not make unmeasured modifiers disappear, but it makes the assumptions inspectable. A long list of available variables is not necessarily better than a carefully justified set, especially when indiscriminate adjustment creates sparse combinations and unstable estimation.
The intervention may change when it crosses the city boundary
Harbor's trial used experienced tutors, small groups, travel support, and regular reminders. Ridge proposes larger classes, newly hired tutors, and no travel support. Calling both packages the same tutoring program may conceal changes in the treatment itself. Population weighting cannot compensate for a different intervention unless the analysis includes a defensible model of those changes.
Implementation differences can also alter adherence. Learners who would attend with travel support may miss sessions without it. An effect of assignment to Harbor's package incorporates the participation behavior that package produced. Transporting that assignment effect to a setting with different engagement requires more than matching baseline characteristics.
Research on transporting treatment-assignment effects under nonadherence highlights how trial engagement and selection related to adherence can limit simple transport analyses. The relevant question is whether the target implementation preserves the pathways through which assignment affects actual participation and outcomes. Transportability under nonadherence
The policy team therefore creates an implementation comparison alongside the population comparison. It records what is held constant, what changes, and which changes could affect outcomes. This is not a demand to reproduce every incidental detail of the trial. It is a way to identify the components that define the intervention being evaluated.
The comparator deserves the same attention. If Ridge's existing services are stronger than Harbor's control condition, the incremental effect of the new program may be smaller even when the program itself works identically. Evidence about program versus minimal support does not automatically establish program versus an effective established service.
A local pilot can answer focused implementation questions without pretending to be a definitive effectiveness trial. It might measure attendance, delivery fidelity, assessment completion, and resource use. Those observations can test parts of the transport argument and identify where a larger local evaluation is needed. They should not be used to claim a precise outcome effect if the pilot was not designed to estimate one.
The target dataset needs its own quality assessment
Ridge's population profile comes from an enrollment database. That database may omit people who are eligible but have not yet applied, use outdated preparation measures, or contain duplicate records. Reweighting a trial to a flawed target distribution produces a precise answer to the wrong population description.
The team must therefore define whether its target is all eligible residents, applicants, enrolled learners, or people likely to attend under the new offer. Those groups can differ. A distribution measured among current enrollees may not describe people attracted by a new program, especially if the intervention changes access or demand.
Sampling weights in the target survey also matter. If Ridge deliberately oversamples underserved neighborhoods, the raw survey proportions are not automatically population proportions. The transport analysis should account for the target sampling design in addition to the trial's design and participation process. These are separate layers of weighting with different purposes.
Measurement harmonization needs more than recoding categories. A preparation test administered in a supervised setting may not be equivalent to self-reported preparation in an application form. Combining them as one covariate can leave residual differences while creating the appearance of adjustment. The dossier should document definitions, timing, missingness, and evidence of comparability.
Missing target covariates can be consequential even when outcomes are not required from the target sample for a particular identification strategy. If the variables needed to connect populations are missing selectively, the transport assumptions may not be supported by the observed data. Any imputation or modeling approach needs its own assumptions and sensitivity assessment.
The uncertainty file includes more than sampling error
The 7.6-point transported effect in the opening example was calculated as though subgroup effects and target proportions were known exactly. In a real analysis, both may be estimated. A valid uncertainty calculation should reflect the estimation strategy, including relevant sampling and model fitting stages, rather than attaching the original trial's standard error to a new weighted point estimate.
Model misspecification is another concern. An outcome model that misses an important interaction can misrepresent target effects even when the target covariate distribution is measured well. A participation model can generate poor weights if it fails to represent selection adequately. Flexible methods can help, but their performance and overlap requirements still need assessment.
Some estimators combine outcome and participation models to gain robustness under specified conditions. The phrase doubly robust should not be translated into immune to all errors. Identification assumptions, support, measurement quality, and the required model conditions remain relevant. Methods for transporting multiple trials describe such estimators together with their assumptions. Robust transport from multiple trials
The hardest uncertainty may be an unmeasured difference between cities. A sensitivity analysis can ask how large an unaccounted effect modification would need to be to change the adoption decision. This does not verify the transport assumption, but it makes the decision's dependence on that assumption more concrete.
For example, suppose the target average is estimated at 7.6 points and adoption requires an expected benefit above five points after considering costs. An unmodeled implementation penalty of three points would move the estimate to 4.6. That arithmetic is not evidence that the penalty exists. It identifies a magnitude of uncertainty that matters and directs further investigation toward delivery conditions capable of producing it.
The dossier should distinguish statistical uncertainty intervals from such assumption-based scenarios. Combining them without explanation can make a judgmental range look like a formally estimated confidence interval. Both can be useful when their meanings remain clear.
Scaling changes the environment people share
The original trial enrolled a small fraction of Harbor's eligible learners. A citywide rollout in Ridge could change tutor availability, class composition, or access to other services. Those changes may affect outcomes even for learners whose baseline characteristics resemble trial participants. The effect of offering a program to a few people need not equal the effect of offering it to nearly everyone.
Suppose the trial's tutors each taught two small groups, while the rollout requires the same tutors to supervise six groups and train new staff. The intervention's written curriculum remains unchanged, but its delivery conditions do not. A population adjustment based on learner preparation would miss that capacity constraint entirely.
There can also be interactions among participants. Learners may share materials, motivate one another, or compete for limited placements after training. If one person's outcome depends on how many others receive the program, a simple individual treatment effect may not describe the rollout policy. The relevant intervention may need to specify coverage or allocation at a center or city level.
The dossier should not assume that every such interaction is large. It should identify plausible pathways, inspect the scale of implementation, and state where the analysis relies on their absence or stability. A narrowly targeted local evaluation can then measure the most consequential operational differences rather than repeating the entire original study without learning why transfer might fail.
Cost transport is a separate calculation
Even if the target effect were exactly 7.6 points, Harbor's cost per learner might not apply in Ridge. Tutor wages, travel distances, venue availability, and the proportion requiring additional support can differ. The adoption decision therefore combines a transported outcome estimate with a locally grounded resource model.
For a simple illustration, suppose the incremental cost is three hundred currency units per learner in Harbor and five hundred in Ridge. Using the hypothetical average effects, cost per score point is about 83.3 in Harbor and 65.8 in Ridge. The higher-cost city can still have a lower cost per point because its expected benefit is larger. These ratios are illustrative, not a complete economic evaluation or a claim that score points have a fixed monetary value.
The calculation reinforces the need to keep quantities aligned. Using Harbor's cost with Ridge's benefit would describe a hybrid program that may exist in neither city. Local decisions require the intervention's expected effects and resource requirements under the same target implementation.
The local decision is a claim with a boundary
Ridge may decide to adopt the program for the population with adequate overlap while evaluating evening delivery separately. It may preserve travel support because the transport argument depends on comparable attendance. It may commission a new trial if the target population and implementation differ too much for existing evidence to support a confident decision.
The final recommendation should state the target population, intervention version, comparator, outcome, horizon, and assumptions that connect the evidence. It should present the unadjusted trial estimate beside the transported estimate and explain the reasons for any difference. That comparison makes the adjustment understandable rather than turning it into an opaque statistical upgrade.
The strongest conclusion is not that a study works everywhere. It is that a specified body of evidence supports a specified decision for a specified population, with remaining gaps identified. Transportability analysis earns its value by making those boundaries explicit and showing what new information would most improve the decision.