Two instruments can rank every sample in exactly the same order and still give values that cannot be used interchangeably. Correlation describes association. A replacement decision requires evidence about the size, pattern, and practical consequences of differences between measurements.
TL;DR: Define acceptable disagreement before collecting comparison data. Examine paired differences across the operating range, distinguish average bias from individual variability, and assess repeatability, operator effects, and drift. Limits of agreement describe the distribution of differences under stated assumptions; they do not decide what is acceptable. This benchbook uses hypothetical industrial measurements, not clinical thresholds.
Bench card one: a perfect result that fails immediately
A laboratory is considering a portable instrument for measuring the thickness of manufactured sheets. The existing instrument reports ten, twenty, thirty, forty, and fifty units for five demonstration samples. The portable device reports fifteen, twenty-five, thirty-five, forty-five, and fifty-five. The measurements have a Pearson correlation of one because the second sequence is exactly the first sequence plus five.
The laboratory's production decision requires differences no larger than two units in this illustrative example. Every sample fails that requirement. The perfect correlation is correct, but it answers a question the purchasing committee did not ask. It says that the readings move together along a straight line, not that substituting one reading for the other preserves decisions.
A second demonstration produces portable readings of twenty, forty, sixty, eighty, and one hundred. Correlation is still one, but the new device now doubles every value. A constant correction would not repair this scale difference. The two demonstrations show why a single association coefficient cannot distinguish a useful replacement from a consistently displaced or rescaled instrument.
Bland and Altman's original method-comparison paper addresses this mismatch between correlation and agreement. Its central practical contribution is to make differences between paired measurements visible and interpretable. The article below builds original examples around that distinction rather than using a correlation threshold as an acceptance rule. Original method-comparison paper
The laboratory keeps the correlation calculation in its exploratory notes because it can still describe the data. It removes the value from the purchase acceptance criterion. This is an important editorial decision: a statistic should not occupy the most prominent position merely because it looks impressive or has a familiar scale from zero to one.
Bench card two: write the tolerance before opening the data
The production engineer defines what the thickness reading is used for. Some measurements guide coarse sorting, while others determine whether a sheet is close to a rejection boundary. A difference that is harmless for the first use can change the second decision. The study therefore needs an intended use, an operating range, and a definition of acceptable disagreement.
For the main comparison, assume the laboratory specifies that individual differences should ordinarily remain between minus two and plus two units under the intended procedure. This is a fictional engineering requirement, not a universal statistical recommendation. The laboratory also states how it will account for uncertainty in estimating agreement and what evidence would trigger further investigation.
The requirement cannot be chosen by looking at the observed spread and then drawing a tolerance band just outside it. That would turn the acceptance test into a description of whatever happened. If the committee later revises the tolerance because the intended use changes, the revision needs a substantive explanation independent of whether it makes the device pass.
Asymmetric consequences may justify asymmetric limits. Suppose overestimating thickness wastes material while underestimating it can allow unsuitable sheets into an assembly. The acceptable difference in one direction might then be tighter. The agreement analysis can estimate directional behavior, but deciding the acceptable tradeoff requires knowledge of the process and its consequences.
The laboratory also specifies the actual measurement procedure. Will an operator take one reading or average three? Will the sheet be cleaned, clamped, or measured at several points? Will the portable device be used on the factory floor or only in a controlled room? A replacement claim applies to that procedure, not to an instrument name divorced from how people use it.
These details prevent a common procurement misunderstanding. A study of carefully averaged measurements by an expert may support a specialized laboratory procedure while providing little evidence about a single rapid reading by a new operator. Both procedures could use the same hardware and still have different measurement properties.
The first paired dataset separates bias from scatter
The laboratory measures a larger set of representative sheets with both methods. For each sheet it subtracts the existing instrument's reading from the portable device's reading. Suppose the mean difference is 0.5 units and the standard deviation of differences is 1.5 units. These are invented summary statistics for illustrating interpretation.
If independent paired differences are reasonably described by a normal distribution with stable mean and spread across the range, conventional estimated limits of agreement are the mean difference plus or minus 1.96 standard deviations. Here they are approximately minus 2.44 and plus 3.44 units. The average bias is small, but the spread extends beyond the hypothetical tolerance band.
| Quantity | Illustrative value | Interpretation |
|---|---|---|
| Mean portable-minus-existing difference | 0.5 units | Average directional difference |
| Standard deviation of differences | 1.5 units | Variation among paired differences |
| Estimated lower agreement limit | Minus 2.44 units | Lower boundary of the modeled central spread |
| Estimated upper agreement limit | Plus 3.44 units | Upper boundary of the modeled central spread |
| Prespecified acceptable band | Minus 2 to plus 2 units | Engineering requirement |
The limits are not confidence limits for the mean difference. A confidence interval for average bias can become narrow in a large study while individual differences remain too wide for substitution. Increasing sample size improves knowledge of the distribution; it does not automatically make the instruments agree more closely on the next sheet.
Nor are estimated limits a promise that exactly ninety-five percent of every future batch will fall inside them. Their interpretation depends on the distributional and sampling assumptions, and the limits themselves are estimated with uncertainty. A small or unrepresentative sample can leave substantial uncertainty about their location and about performance in future conditions.
The committee therefore asks for confidence intervals around the agreement limits and an explanation of the method used to calculate them. It wants to know whether the evidence supports the acceptance rule with adequate precision, not merely whether the estimated lines happen to fall near a target. The original authors' full paper discusses the relation between differences, their distribution, and repeatability. Author-hosted full text
A test of whether the mean difference equals zero would answer a narrower question. Failure to reject zero bias does not prove interchangeability. A device can have differences evenly distributed around zero and still produce errors large enough to disrupt production. Conversely, a small consistent offset might be manageable through a validated calibration procedure even if a large study detects it clearly.
A plot exposes what the average conceals
The next page plots each paired difference against the average of the two readings. This allows the team to examine whether disagreement changes with measurement magnitude. It also preserves individual observations that disappear inside a mean and standard deviation. The plot is a diagnostic view, not an automatic certification graphic.
In one possible pattern, differences remain centered near 0.5 units but become more variable for thicker sheets. A single pair of horizontal agreement limits would then hide useful structure. The committee needs to consider a model with magnitude-dependent variability or a suitable transformation, and it should report performance across the range in terms meaningful to users.
In another pattern, the mean difference grows steadily with thickness. That suggests a scale-dependent discrepancy rather than a constant offset. Subtracting 0.5 units from every reading would not solve it. A calibration model might help, but the model must be justified and validated on new samples across the intended range.
A third pattern contains two distinct clouds corresponding to two operators. Pooling them could produce a wide overall spread while missing a correctable procedural difference. Alternatively, the device may be sensitive to how it is positioned, making operator variation part of its real deployment performance. The diagnostic question is whether the pattern reflects a removable study artifact or a condition the replacement must handle.
A difference-versus-average plot also needs cautious interpretation when the two methods have substantially different precision. Mathematical relationships between the axes can contribute to apparent trends. The laboratory should investigate the measurement model and replicate data rather than treating every visible slope as definitive evidence of proportional bias. A plot generates questions that the design must be able to answer.
Outliers receive individual review, but the laboratory does not delete them merely because they make agreement look worse. A transcription error, damaged sample, documented procedural violation, or genuine device failure has a different meaning. The report records the reason for any exclusion and shows how conclusions change when questionable observations are retained.
Ratio errors require a different language
Suppose the portable device's disagreement is more naturally proportional than additive. An error of one unit at a reading of ten and ten units at a reading of one hundred both represent ten percent. In such a setting, absolute differences can increase with magnitude even when relative behavior is stable.
For strictly positive measurements, a logarithmic analysis may be appropriate if it produces a suitable model of the differences. The difference between log readings corresponds to the log of their ratio. Transforming the results back then describes multiplicative agreement rather than adding and subtracting a fixed number of units. The choice needs diagnostic support and an interpretation tied to the measurement task.
For illustration, suppose the mean log ratio is zero and the standard deviation is 0.05 using natural logarithms. Approximate limits on the log scale are minus 0.098 and plus 0.098. Exponentiating gives ratios of about 0.907 and 1.103. Under the model, the central spread is roughly from nine percent below to ten percent above the comparison reading.
The asymmetry after transforming back is expected; equal distances on the log scale correspond to reciprocal ratios. Reporting those limits as exactly plus or minus 9.8 percent would lose that distinction. More importantly, the committee must decide whether such relative disagreement is acceptable at both low and high values for the actual production decision.
Logarithms do not solve every problem. Zero or negative readings require careful consideration, and adding an arbitrary constant changes the meaning of ratios. A transformation chosen solely to make a plot look tidy may obscure the quantity users care about. The study should preserve the original scale and explain why the transformed model is useful.
If the requirement combines an absolute tolerance at low readings with a relative tolerance at high readings, the acceptance analysis should reflect that hybrid rule explicitly. Forcing all observations into a single summary can hide failure near a critical boundary. The appropriate reporting unit is the one that supports the intended decision.
Repeated readings are useful, but they are not extra independent sheets
The laboratory now takes three readings with each instrument on every sheet. This can help distinguish short-term repeatability from differences between methods. It also creates dependence: readings from the same sheet share the same underlying material and often the same preparation conditions. Treating every pair as an independent sample exaggerates the amount of independent information.
The analysis depends on whether the underlying quantity is stable during measurement. For a stable sheet, repeated readings can characterize measurement variation around the same thickness. For a changing process, measurements taken at different times may differ because the quantity itself changed. Pairing and timing then become part of the scientific model rather than simple spreadsheet bookkeeping.
Bland and Altman developed methods for agreement studies with multiple observations per individual, including settings with changing and unchanging underlying quantities. Their work is a reminder that the basic independent-pairs calculation is not a universal formula for repeated data. Repeated-observation agreement methods
Averaging replicates also changes the procedure being evaluated. If the study averages three readings from the portable device but deployment uses only one, the reported agreement may be too optimistic for practice. If deployment genuinely uses the same three-reading average, the study should evaluate that average while retaining enough detail to understand its repeatability and failure modes.
The order of measurements can matter. If handling, temperature, drying, or fatigue changes the sample or operator, always using the existing instrument first may confound method with order. Randomizing or balancing order where appropriate helps separate these effects. The design should also minimize carryover, such as an operator remembering the first reading and adjusting the second procedure toward it.
Blinding operators to the other method's result can reduce that influence when feasible. Automated data capture can reduce transcription errors. Neither replaces a clear protocol for identifying which readings belong to the same sample and time point. A perfectly executed statistical analysis cannot repair pairs that refer to different physical locations on an uneven sheet.
The existing instrument is a comparator, not automatically truth
Agreement between two methods does not establish that either one measures the intended quantity accurately. They may share a calibration error or respond similarly to an environmental influence. A replacement study answers whether measurements are sufficiently consistent for a specified use; establishing accuracy relative to a reference requires additional evidence.
The committee therefore distinguishes the existing production method from any reference standard used for calibration. It documents the reference's uncertainty, the range over which it applies, and how the measurement chain is maintained. NIST describes calibration as relating an instrument's response to a reference base through a defined procedure, with the aim of reducing or eliminating bias relative to that base. NIST calibration guidance
A calibration correction should be estimated on appropriate data and evaluated independently. Fitting a line to the comparison sample and then reporting excellent agreement on those same corrected readings can overstate future performance. The correction has used information from the data being judged. New samples, new days, or a separate validation phase are needed to test whether the relationship persists.
The validation range must match the claim. A model fitted between ten and fifty units should not be assumed reliable at two or two hundred. Extrapolation can fail even if the fitted relationship is smooth and the training residuals are small. The report should state where the evidence ends rather than treating software's ability to produce a number as proof that the number is supported.
Calibration also does not erase random variation. If the device has a stable constant bias, correcting that bias can move the center of the difference distribution. It will not necessarily narrow the spread. The committee should compare both before and after correction so that improved centering is not mistaken for improved precision.
Acceptance lasts only as long as the measurement process stays comparable
A successful comparison conducted on one day establishes a limited starting point. Production instruments experience maintenance, software updates, changes in consumables, and environmental variation. The replacement decision should include a way to detect when the measurement process has moved beyond the conditions that supported acceptance.
NIST distinguishes repeatability under the same conditions from reproducibility under specified changed conditions. That language helps the laboratory decide what it has actually tested. Repeating a reading many times in one room with one operator does not characterize performance across operators, locations, and months. NIST measurement terminology
The laboratory builds a modest monitoring program using appropriate check materials and scheduled comparisons. It records device identity, software version, operator, environmental conditions when relevant, and maintenance events. These records make a future shift interpretable. Without them, a disagreement discovered months later may be impossible to trace to drift, a changed procedure, or a different sample population.
The monitoring thresholds need their own rationale. A process control signal indicates that the measurement behavior may have changed; it is not automatically the same as a product rejection threshold. Confusing those roles can either trigger unnecessary production stops or allow instrument drift to continue until unsuitable material has already been accepted.
A software update deserves particular attention when it changes the transformation from sensor output to reported value. The hardware may be unchanged while the measurement method has changed materially. The laboratory should assess the update against the accepted procedure and decide what verification is needed before treating historical and new readings as interchangeable.
A decision-boundary audit asks a separate question
Suppose the production rule sends sheets below twenty units for additional inspection. A portable reading of 20.4 and an existing reading of 19.6 differ by only 0.8 units, within the earlier illustrative tolerance, yet they trigger different actions. This does not automatically invalidate the tolerance. It shows that numerical agreement and decision agreement should be connected explicitly when readings are used near a threshold.
The study can oversample sheets near that boundary to understand the practical consequence, while clearly distinguishing this targeted sample from the production distribution. If half the study consists of borderline sheets but only a small fraction of production does, the observed percentage of discordant decisions cannot be used directly as the expected factory-wide discordance rate without appropriate accounting for that sampling design.
A separate illustrative audit contains two hundred sheets: one hundred sixty receive the same action under both procedures, twenty-five are flagged only by the portable method, and fifteen only by the existing method. The overall action agreement is eighty percent. The direction of the forty disagreements matters because unnecessary inspection and missed inspection have different costs. The table should preserve both directions rather than reporting only the total agreement percentage.
Those categories still do not identify which method made the correct decision. The existing method is the comparator, and its own uncertainty remains relevant. A reference procedure or further adjudication may be needed to investigate the consequences. Calling every disagreement a portable-device error would assume the very superiority the study has not established.
The laboratory can also design a practical indeterminate zone around the boundary, routing those sheets to a more precise procedure. Such a policy should be evaluated as a complete workflow: how many sheets require additional measurement, what delays occur, and what final decisions result. A device with limited interchangeability may still be useful when its uncertainty is handled openly in the operational process.
The purchase decision becomes a bounded statement
The committee can now make a precise decision. It may approve the portable device for coarse sorting across a specified range while requiring the existing method near critical boundaries. It may approve a calibrated three-reading procedure but reject single-reading use. It may defer replacement because uncertainty around the agreement limits remains too large. These are useful outcomes even when the answer is not a universal yes.
The final report presents the intended use, sample selection, pairing procedure, acceptable differences, observed patterns, uncertainty, and any calibration or averaging rule. It describes exclusions and missing measurements, including failures to return a reading. A device that agrees well only on the subset of samples it successfully measures needs that success rate reported alongside agreement.
The most persuasive result is not a correlation printed to three decimal places. It is a supported statement about how much two procedures can differ, under which conditions, and whether that difference preserves the decisions people will make. That is the evidence required to replace a measurement method without quietly changing the meaning of the number.