TL;DR: Evaluate an object detector with the same unit the inspection line releases. Box precision and recall explain localization errors. Part-level review and escape rates explain how much work the line can sustain and which defects can reach customers. Choose the rule using separated production lots, verified defect families, and measured handling time at the intended station.
The transparent lid is hard to see against the white belt. Under a side lamp, a narrow fracture near its closing tab becomes visible as the lid turns. The detector draws one box around the tab and another around a reflection. Six photographs of this turn can produce twelve candidate boxes and one physical lid that someone will either hold or release.
Juniper Packaging is a fictional producer preparing an inspection trial for that station. Its quality team is interested in cracks near the tab, scratches on the top face, and malformed edges around the seal. These are illustrative product requirements. All station rates and defect counts below are invented to show the accounting that connects a detector trial to a release decision.
A data scientist can compute box metrics with the familiar practices formalized in COCO object detection. COCO evaluates overlap between detected and annotated objects across multiple intersection-over-union thresholds, with summaries for object area ranges and limits on detections. Juniper keeps that kind of diagnostic because it points toward localization, class confusion, and resolution problems. It adds a second worksheet for the decision the production team actually makes. COCO detection evaluation, COCO evaluation implementation.
The first sheet belongs to the camera
The trial specifies when an image begins, which lid it depicts, and how it can be attached to the lid leaving the station. The conveyor controller assigns a run-local part identifier. Each camera event records that identifier, capture time, view, model version, and whether the image met the station's exposure and framing checks. The production event then records what happened to the part.
This matters even with perfect box coordinates. A new tray entering before the previous tray has completed a turn can trigger two closely spaced captures. If the capture associates a good image with the wrong part, the box evaluator running on individual images may look excellent. The release system has used evidence from a different physical object. Deferred or reordered results need explicit ownership before anyone trusts the final move.
Annotation starts with an image policy. An image reviewer marks visible defects and records regions where judgment is impossible. A surface hidden behind a handling jaw remains unobserved in that view. The trial keeps that state distinct from a visible clean surface. A collection heavy in blank annotations can conceal where an inspector could never have seen the intended condition.
The station engineer chooses the trial's lighting and exposure conditions before comparing models. Models trained on side-lit defects can look dramatically different when reflections run parallel to the camera. The idea is to measure settings associated with foreseeable production, including clean lamps, aging lamps, and normal variation in lid material. Trial photographs also preserve lot identifiers so the evaluator can later separate repeats.
Image capture is a design variable before it is an algorithm question. If a tiny fracture occupies two pixels in a view, changing the lighting geometry or camera placement may add information more cheaply than fitting a larger detector. Juniper keeps paired captures when testing such a change. The paired data let an engineer ask what happened to the same physical mark when the view changed.
Match one object and then ask who owns the miss
Say the box from an annotator covers a small fracture and the model places a slightly larger box over the same area. Suppose their intersection is 72 square pixels and their union is 96. Their intersection over union is 0.75. Whether they match in a particular diagnostic depends on the overlap threshold, the class policy, and which predictions have already been matched to available annotations.
Consider a hands-on review of twelve cracks across fourteen boxes at a chosen confidence cutoff. Nine predictions correspond to nine annotated cracks under the review rules. Two boxes duplicate cracks that already have an assigned prediction, one points at a reflection, and two occupy clips with clean material. Box precision is nine of fourteen, about 64 percent. Box recall is nine of twelve, 75 percent. Three annotations have no accepted corresponding box.
One of those three marks sits on a lid with another correctly noticed crack. That lid is already held. Another mark is a crack seen in a second view where the model matches it correctly. If either camera can hold a lid, the station has a reason to review that item as well. A third mark belongs to a different lid with no other evidence. Only an item-level reconstruction determines how many physical pieces escaped.
The duplicate boxes still matter. Operators who inspect each region individually may spend twice as long on the same surface. A system that highlights only one region per lid may avoid that repetition but suppress a second independent clue that the operator would have used. If a team changes region merging, it should measure review time and review accuracy on parts with multiple defects. Duplicate suppression is a decision with a human consequence downstream.
For a small crack, one shifted edge can change the overlap score sharply even while the operator sees the correct area. For a broad abrasion, a loose box may satisfy a low overlap threshold while hiding the portion that controls the disposition. Juniper labels two separate uses of coordinates. A highlighted region helps someone locate evidence. An estimated dimension can inform a tolerance decision only after dimensional accuracy has been validated independently.
The worksheet therefore records the consequence as well as the category of an error. A weak localization that still leads the operator to the right mark has one consequence. A perfectly placed region with the wrong severity tag has another. A confident region on the right kind of defect but on the wrong lid is a traceability failure. Giving each error a specific route to repair keeps engineering time focused on the station instead of the score alone.
At the second sheet the lid gets a disposition
Juniper uses three state changes at the station. A lid can leave automatically, transfer to an operator, or reach an unresolved state because its evidence is incomplete. A separate verification crew samples all three routes. It examines the physical lids under a reference fixture and records whether each meets the same product requirements that a normal disposition needs.
The disposition tally has denominators that cannot be interchanged. Defective parts passed automatically over all verified defective parts is a miss rate for the automatic rule in the evaluated set. Parts held over all parts is a workload measure. Good parts held over all verified good parts describes avoidable interruptions. Defective parts that reached release after all review steps constitute a different endpoint again. The station can improve one rate while worsening another.
Start with an invented shift of ten thousand lids whose final status is available for the exercise. Two hundred are defective according to the reference fixture. A high cutoff directs 120 lids to review, with 96 defective and 24 good. Of the 200 defective lids, 104 enter the automatic route. Of the 9,800 conforming lids, only 24 are held. The operator's queue looks efficient and the product outcome is poor.
A broad cutoff directs 420 lids to review, with 160 defective and 260 good. Forty defective lids enter the automatic route. The system reduced automatic misses by 64 and created 300 extra reviews. The incremental review group therefore contains 64 defective lids out of 300, roughly 21 percent. Aggregate precision in the first 120 holds answers a different economic question from the yield of widening the band.
There is no guarantee that all 160 held defects will be found by the operator. If the operator correctly intercepts 150 of the 160, ten further bad items are released, assuming a release follows any missed decision. The broad rule then has 50 fictional escapes across both routes. A sheet that reports forty has omitted the operator's part of the chain. Visibility of the extra ten is what makes changes to the human interface assessable.
Now imagine a third setting routes 260 lids to review. It holds 140 defective and 120 good lids. If the operator catches 136 of the 140 defects, then the total escapes are sixty automatic plus four after review, or 64. The broad setting reduced escapes in the worksheet further. Before giving it the release switch, Juniper checks whether the review service that caught 150 defects can keep working at the rate it would actually receive.
The service window has edges
On the reference shift the medium setting generates 260 cases. At a mean 48 seconds of handling and an additional mean 12 seconds of movement and logging, it needs 260 minutes. The broad setting needs 420 minutes at the same mean. A station that can provide 300 review minutes cannot independently finish 420 minutes' work as it arrives. Its held queue persists into a later staffed window or another path absorbs the excess.
Arrival and handling times also vary. If a surface finish produces more reflections for twenty minutes, even the medium setting may overwhelm a nearby buffer. Operators see the oldest work wait while new lids keep arriving. Juniper measures service-time multiples for individual cases and queue age in addition to mean labor. A second camera takes milliseconds to produce a picture and may take the operator longer to interpret when two candidate images disagree.
The team tests handling in a realistic order. A demonstration workstation can serve suspicious pieces grouped neatly by category. On an active line, a reviewer switches between edge and surface work and may divide time with an unrelated task. Evaluation needs the intended interface, available lamp, and movement path. The operator's measured attention belongs in the cost model only when the evaluation makes those conditions recognizable.
Consider a lot that yields twenty-six holds from 400 lids in ten minutes. If each hold takes a minute and one reviewer started with an empty queue, ten can be finished in that interval and roughly sixteen will wait. A planning system that knows the average 260 cases per day but lacks twenty-minute slices may size the physical buffer too small. The held inventory is an engineering state that persists across the end of a spreadsheet row.
Juniper sets a policy for missing views and missed service windows before weighing cutoffs. A queued lid stays identified and visibly held until a recorded disposition arrives or the production line uses the approved manual inspection route. The team tests a stop-and-restart drill with tagged trial parts. It reconciles what the controller remembered, what the tray actually contains, and what landed in the work queue. Lost queue work can produce defects even when the detector is performing to its offline specification.
A useful threshold comparison therefore requires conditional claims. The broad setting yielded fifty escapes in this simulated population under a stated human review outcome. Another arrival schedule, service rate, or stop policy changes that result. This is a reason to build a capacity margin and measure it during shadow operation. The cap is an operational contract that engineering and quality owners can inspect together.
The crack that matters arrives in a less common finish
A pooled count can hide which parts generate a workload. Suppose the 200 defects include 150 scratches, forty edge defects, and ten tab fractures. If the chosen setting catches 135 scratches, 35 edge defects, and four fractures before review error, its aggregate detection is 174 of 200, 87 percent. The important fracture slice has four of ten detected. An operational decision tied to the consequence of a failed closing tab uses that slice explicitly.
A different sample may include more rare conditions than a normal shift so the team can examine their mechanisms. The defect-family counts then help estimate behavior conditional on those families being present. A workload forecast requires representative arrival frequencies or explicit weighting. Multiplying an enriched set's hold percentage directly by daily volume makes the queue estimate depend on how an evaluator filled the sample, instead of the mix on the line.
Collection separates three jobs. The development set helps the engineer choose views, decision rules, and examples. A reserved production-lot set measures the chosen rule. A targeted specimen drawer exercises scarce but consequential combinations such as a small crack on a translucent finish with a worn lamp. Combining the latter two into one score removes the information about why each set exists.
Lot separation improves the independence of the comparison. Closely related parts can share material, supplier batch, equipment settings, and a characteristic flaw. Train and review data split randomly across consecutive frames can contain near-duplicates of the same lid. Splitting by known source of shared behavior makes the test represent its intended transfer. A later shift captured with another camera is a different test still, and naming that change makes the result teachable.
The shift log includes intervals or underlying counts rather than rounding a ten-fracture test to a universal rate. Four observed successes contain considerable uncertainty. Juniper can use repeated lots and targeted sampling to improve information while retaining the held-out design. If fracture evidence remains sparse, quality can specify a continuing manual verification route for the relevant class. The unsupported fraction stays visible in release notes instead of disappearing into an aggregate green cell.
The quality audit opens one carton in ten
The real station will rarely provide independent inspection for every part. Juniper can sample released lids while preserving an interpretable result. Its sampling rule selects identifiable units before the additional inspection result is known. Every selected lid gets an outcome or a documented loss. The review crew records whether it sampled the automatic stream, operator releases, or held inventory. Those strata come with different selection probabilities and widely different expected rates.
Suppose the station released 9,500 lids automatically and the audit examines a simple random sample of 950. It finds three defects. Under that design, multiplying each detected defect by ten produces an estimated thirty bad releases in that stream. A separate sample of 100 of 400 operator releases finds one. Its expansion contributes four estimated bad releases. The point estimate for the two streams combined is thirty-four across 9,900 releases. A defensible uncertainty estimate should reflect both sampling designs and the small observed counts.
Auditing one lid out of every ten by position sounds convenient. It can align accidentally with a machine cycle that places a particular mold cavity at regular positions. A rotating or randomized starting point can improve coverage, and the evaluation should check whether the interval repeats in sync with the process. Juniper's fixtures have traceable cavity and material identifiers. The audit can tell whether patterns by position are evidence of a particular mechanism or an artifact of sample selection.
Historical quality claims need the same denominator distinction. If operators previously recorded only items they pulled aside, returns or late independent audits might be the only information about unseen escapes. Comparing the new independently verified escape estimate with last month's recorded defects would give a false impression of worsening quality. The team can run an overlapping audit around the old procedure, or state that historical records can support queue comparisons while escape measurement begins with the new design.
The quality crew also estimates residual ambiguity in the reference result. Two inspectors may disagree on a fine scratch under different lighting. A second inspection of a sample of both straightforward and difficult lids lets the team quantify that disagreement. It checks whether uncertainty clusters near the physical requirement or around a particular view. A benchmark with a single adjudicated answer can report the operational match rate alongside the number of unresolved cases behind it.
Count material and attention on their own terms
The purchasing decision can change if physically holding a lid consumes packaging space or interrupts dispatch. Suppose two policies differ by 160 holds and the warehouse can reuse 140 lids after review. The carrying workload includes all 160 visits to the hold area, while the material consequence depends on final rejection and scrap rules. Charging material loss against the entire hold count would penalize policies for items they return intact. Counting only rejected material would erase the meaningful cost of the temporary diversion.
Actual staffing and inventory constraints set the scale of these costs. Juniper can assign local estimates to extra review minutes and rehandling moves. It can keep contractual quality tolerances as constraints rather than choose a convenient currency conversion for a defect it has little basis to price. Given two settings that satisfy evidence and quality requirements, the decision may turn on inventory congestion and ability to handle a bad lot. A cheap threshold that frequently puts dispatch at risk has a cost outside the camera team.
A paired trial offers a way to diagnose any difference. Apply both candidate thresholds to the same saved views without changing the underlying physical releases. Count each lid where the decisions disagree and verify the relevant sample. The threshold team then knows which incremental reviews carry incremental defects and which depend on a stale or missing view. Replay still leaves open how a reviewer would perform with a larger live queue. Recording that distinction keeps the trial's benefit tied to the evidence actually collected.
The worksheet closes at the diverter
The trial ends with a physical reconciliation. For each checked lid the team can trace the input views, the matched defect evidence, route reason, destination of the handling equipment, operator action, and final verified disposition. A model score might be useful evidence in that chain. A closing count asks whether any lid that should have been on the hold tray instead moved into the finished carton.
Juniper runs a counterbalanced comparison after choosing the trial protocol. During shadow measurement a current inspection method remains responsible for the product, while candidate decisions and timing are recorded. During a permitted supervised stage, the lead checks whether the trial rule creates the forecasted holds and whether a sample of automatic releases looks consistent with the independent fixture. Lag, repeat images and human resolution are measured under the same accounting.
The maintenance schedule preserves comparability. Record the camera, crop, light and specimen criteria with any later model change. The term small in a benchmark area range refers to image annotation geometry under that evaluation, while Juniper's fracture requirement refers to a physical feature under specified sight lines. A rescaled crop can change the former without changing the latter. Both facts help diagnose whether a better score reflects new evidence or a different representation.
Several releases of an inspection rule can coexist in records. A retained example needs to link to the rule the station actually applied rather than the currently installed rule. If a customer reports a defect a week after rollout, the examiner can ask whether the lid passed automatically, whether a reviewer accepted it, and whether the exact view was present. That evidence determines which controls to revisit without assigning every failure to the model.
A quality owner can authorize a constrained trial when the relevant defect families have an adequate measurement plan, automatic and human escapes are separated, and the real queue can stay within its physical and staffing limits. Keep the benchmark sheet nearby for localization and score sensitivity. File the final trial record under the lid's destination. That is the unit a shipment uses.