TL;DR: Active learning spends labels on examples selected to improve a model, but uncertainty alone is an incomplete purchasing rule. Allocate annotation time across informative cases, diverse coverage, representative monitoring, and label-quality work. Keep an independent evaluation set, measure gains against a random-sampling baseline, and stop when another batch costs more than the decision improvement it delivers.
A fictional manufacturer called Cobalt has 80,000 unlabeled equipment photographs and 120 staff-hours available for a labeling cycle. Its model classifies visible surface conditions for a human review queue. The team cannot label everything, and the least confident images look like an obvious place to start. The purchasing ledger below shows why that intuition needs a broader plan.
The methodological starting point is Burr Settles' Active Learning Literature Survey. Later work on core-set selection and BADGE illustrates different approaches to batch diversity and uncertainty. Cobalt's budget, images, and policies are hypothetical. They are used to reason about annotation decisions, not to claim that any algorithm achieved a measured improvement at a real factory.
Purchase order: Buy information, not a row count
The project manager initially requests 1,000 labels. The annotation lead asks how long each case takes and what counts as a finished label. A clear photograph with one visible condition may take minutes. A difficult image may require checking a second view, consulting a specialist, or deciding that the available evidence is insufficient.
The budget is therefore expressed in staff-hours and accepted outcomes. A labeled image includes the classification, any permitted uncertainty flag, and enough provenance to understand how the decision was made. An unresolved image can still provide useful information about the labeling policy, but it is not automatically a usable training example.
Cobalt has an existing independently collected evaluation set. Maintaining that set is a separate responsibility and budget. This cycle's 120 hours buy new training labels and quality work. The distinction prevents a proposed acquisition strategy from consuming the very data needed to evaluate whether it helped.
The intended improvement is also explicit: reduce avoidable human review while preserving detection of the conditions the business considers important. Overall accuracy is one diagnostic, but it is not the entire purchasing objective. A batch that improves a common easy class while missing rare consequential conditions may fail the operational goal.
The first basket contains only uncertain images
The model ranks photographs by uncertainty, and the top candidates contain many blurry close-ups of the same machine surface. Some are consecutive frames from one inspection. Others show reflections that make the target condition difficult for a human to determine. The acquisition score has found model confusion, but it has not established that every image is worth the same annotation cost.
Uncertainty sampling is a family of strategies rather than a guarantee of useful information. The selected score can emphasize low maximum probability, small margins, or entropy, and its behavior depends on the model. Settles' survey describes these approaches and their practical limitations. Cobalt uses that literature to frame candidate strategies rather than assuming the most uncertain batch is necessarily optimal.
The annotation lead reviews a small pilot basket before purchasing the whole batch. She records labeling time, ambiguity, duplicate content, and guideline questions. That pilot can reveal that the acquisition rule is selecting a region where the current label definition is inadequate or the image quality is too poor to support a reliable answer.
The team keeps those findings. Rejecting an unlabelable image is not wasted work if it exposes a data-collection problem. It becomes wasteful when the same unresolved pattern is sent to annotators repeatedly because the acquisition pipeline never records why earlier cases could not be labeled.
The duplicate discount that the model did not offer
Suppose 40 of the first 100 selected photographs show nearly the same defect from one burst capture. Labeling all 40 may provide less new information than labeling a broader set, especially if the eventual model sees highly redundant training examples. The exact effect is empirical, but the acquisition list should at least make the redundancy visible.
Cobalt groups images by inspection session and uses similarity signals to identify candidate duplicates. It does not automatically treat every visually similar image as interchangeable. A slight angle change may reveal a feature that matters. The grouping supports human review and batch construction rather than silently deleting evidence.
The core-set approach motivates selecting examples that cover a representation space in a batch setting. That idea suggests a practical question for Cobalt: does the proposed batch span the kinds of cases the model will encounter? The answer depends on the representation and task; geometric coverage is not identical to coverage of business-relevant conditions.
The purchasing ledger therefore records both acquisition score and group membership. Reviewers can see when a batch spends many labels on one session or one visual cluster. A diversity constraint can be tested against alternatives, with the random baseline retained to show whether the additional machinery actually improves learning efficiency.
The approved ledger for one cycle
Cobalt allocates 100 randomly sampled images at an estimated six minutes each, 250 uncertainty-selected images at eight minutes each, 150 diversity-selected images at six minutes each, and 100 targeted coverage images at ten minutes each. That is 600 new images and 4,500 minutes, or 75 hours, of initial annotation.
It reserves 15 hours for independent second labels on a subset and ten hours for adjudication and guideline repair. Those activities bring planned work to 100 hours. The remaining 20 hours are a buffer for unexpected difficulty, retraining analysis, and resolving cases that the pilot underestimated. The figures are an illustrative budget, not a recommended universal allocation.
The random basket serves two purposes. It exposes ordinary cases the model's uncertainty rule may overlook, and it supports comparison with a less engineered acquisition policy. It is still part of the training acquisition process; it does not replace the independent held-out evaluation set.
The targeted basket addresses named coverage gaps, such as an equipment family newly entering the supported scope. Its inclusion is justified by the deployment population and business need. The team does not invent rare categories merely to make the batch look diverse. Every targeted slice has an owner who can explain why the model needs evidence there.
A label can be expensive because the question is wrong
The first difficult batch contains photographs that show both discoloration and a shallow scratch. The current instructions require exactly one condition. Annotators disagree because the policy does not explain which feature takes precedence. Treating this disagreement as worker error would produce inconsistent labels and an unfair assessment of annotation quality.
Cobalt pauses that subset and revises the labeling decision. It may permit multiple labels, define a primary-condition rule, or split the task into separate attributes. Each choice changes the model's target. The team records the version of the guideline used for each label so older examples can be reviewed when the target definition changes.
The acquisition strategy cannot solve this policy problem. A model can be uncertain precisely where the human task is underspecified. Buying more labels under the same ambiguous instruction may simply produce a larger contradictory dataset. The right expenditure is a short expert session that resolves the decision rule.
The revised guideline includes contrasting examples and a route for insufficient evidence. Annotators are allowed to say that a photograph cannot support the requested distinction. That response helps the imaging team improve capture conditions instead of encouraging confident guesses that make the dataset appear more complete than it is.
The quality reserve buys independent evidence
Cobalt's second-label budget is applied to a planned mixture of ordinary, difficult, and newly defined cases. The second annotator does not see the first label before deciding. This makes disagreement informative. A review that merely asks someone to approve a visible answer can miss the very ambiguity the quality process is intended to detect.
Disagreement is categorized before it is resolved. Some cases reflect accidental mistakes; others reveal poor images, unclear definitions, or a genuinely borderline condition. The adjudicator records the reason and the final treatment. The result can be a corrected label, a revised guideline, or exclusion from a narrowly defined training task.
The team avoids using agreement alone as a quality score. Two annotators can agree on the same misunderstanding, and a difficult but valid task can produce disagreement that deserves investigation. Expert review of selected cases provides context. The quality program asks whether labels support the intended model decision, not merely whether workers match one another.
The purchasing ledger includes these costs because label quality is part of the acquisition strategy. An algorithm that selects especially ambiguous cases may require more adjudication time. Comparing methods only by the number of labels purchased would hide that difference and could favor a strategy that is expensive in the resource the team actually lacks.
The acquisition score is not a performance estimate
The actively selected batch is intentionally unrepresentative of ordinary incoming photographs. It contains difficult, uncertain, or targeted cases. Accuracy measured on that batch cannot be read as the model's expected production accuracy without a suitable evaluation design. The selection mechanism changed the distribution of examples under review.
Cobalt evaluates each retrained candidate on its independent held-out set and on separately defined operational slices. It uses the acquired labels for training and diagnostics, with their provenance retained. The team does not mix the hardest selected images into a dashboard denominator and then announce that production quality suddenly collapsed.
The reverse mistake is also possible. If the acquisition rule avoids unfamiliar regions where the model is confidently wrong, performance on the selected batch may look reassuring while deployment failures persist elsewhere. Representative monitoring is necessary precisely because the model's own uncertainty is not a complete map of its errors.
The random basket can help reveal that blind spot, but it remains too small to prove every rare failure absent. Cobalt combines quantitative evaluation with targeted error investigation and scope decisions. A low observed error rate in a small sample is evidence with uncertainty, not a certificate that all unobserved cases are safe to automate.
Batch size changes the economics of learning
A team that acquires all 600 images at once makes every selection using the same initial model. A team that acquires six batches of 100 can retrain and choose later images using updated information. The latter may adapt better, but it also pays repeated training, analysis, and coordination costs.
Cobalt compares feasible batch schedules rather than assuming the smallest batch is always best. Annotators need a stable queue, model training takes time, and specialists may only be available on certain days. A theoretically appealing acquisition cycle can fail operationally if it leaves workers idle or delays delivery beyond the project's useful window.
BADGE is an example of a method designed to combine uncertainty and diversity within a batch through gradient-based representations. Its research results motivate an option to test; they do not establish the best schedule for Cobalt's images, model, and annotation process.
The team chooses three rounds of 200 new images for this illustrative cycle, with a review between rounds. It reserves the right to stop or redirect the final round if the first two reveal a labeling-policy problem. The ledger records what changed between rounds so later analysis can distinguish an acquisition improvement from a simultaneous change in model architecture or training procedure.
A fair comparison starts with the same starting point
To compare acquisition strategies, Cobalt holds the initial labeled pool, model setup, evaluation data, and available annotation budget constant as far as practical. If one method receives a better initial model or more expert cleanup, the comparison no longer isolates the acquisition decision.
Random seeds and repeated runs matter because a small initial pool can make learning curves variable. The team reports the spread of outcomes rather than selecting the most favorable run. It also compares at matched cost points, including annotation time and quality work, instead of only at matched image counts.
A hypothetical result might show one strategy reaching the target after 400 labels but requiring twice as much expert time per label. Whether that is an improvement depends on the actual bottleneck. If specialists are scarce, the apparently label-efficient method may be the more expensive choice. If unlabeled data is costly to transfer or inspect, another cost model may apply.
Cobalt's release decision uses a learning curve with staff-hours on one axis and the chosen operational metric on the other. A separate chart shows labels acquired so researchers can still compare the algorithmic behavior. The two views answer different questions and should not be collapsed into a single flattering headline.
Confident errors need a route into the next basket
The deployed model occasionally assigns a high score to the wrong condition. Pure uncertainty sampling may never prioritize those photographs. Cobalt creates several independent routes for such cases to enter review: representative sampling, staff corrections, newly supported equipment slices, and structured analysis of downstream failures.
Staff corrections are useful but selective. People are more likely to report an obvious mistake than a subtle one, and some teams may use the feedback control more often than others. The acquisition ledger records the source of each candidate so the training team can distinguish routine sampling from complaint-driven examples.
The team also investigates whether preprocessing contributes to confident errors. A crop may remove the relevant defect; a color transformation may alter the cue; a metadata shortcut may correlate with labels in training. Buying labels for those images can help diagnose the issue, but the final repair may belong in data collection or preprocessing rather than in the classifier itself.
The practical question is broader than “which image has the highest entropy?” It is “which evidence would most improve the next supported decision?” That can include labeling an example, inspecting a pipeline stage, revising a target definition, or narrowing the deployment scope until suitable evidence exists.
The stopping rule protects the remaining budget
Active learning can continue indefinitely if every round finds more uncertain examples. Uncertainty alone is not a reason to keep spending. Cobalt defines stopping conditions before the cycle: sufficient operational performance with acceptable uncertainty, diminishing improvement per staff-hour, exhaustion of labelable cases in the target pool, or evidence that another intervention is needed.
The team does not stop solely because a point estimate crosses a target once. It examines evaluation uncertainty and whether the improvement holds across the important slices. Conversely, it does not demand that every selected difficult image become easy before releasing a bounded workflow. Some ambiguity belongs to the task and should remain with human reviewers.
A useful budget question is what the next 20 hours could buy elsewhere. Better lighting at capture, a revised intake instruction, or a clearer label taxonomy may improve the system more than another batch of nearly identical uncertain photographs. The acquisition program should be compared with those alternatives, not only with the decision to do nothing.
When the cycle stops, unused budget is recorded rather than spent to satisfy an arbitrary label quota. The report explains the stopping reason and the evidence supporting it. That makes the next cycle easier to plan and prevents annotation from becoming a perpetual activity disconnected from model or business outcomes.
The handoff package for the next model owner
Cobalt's final dataset includes image identity, acquisition round, selection source, relevant grouping, guideline version, annotation provenance, and adjudication outcome. It also records exclusions and their reasons. An image that was unlabelable under one task may become useful under a later task, so its history should not be erased.
The model owner receives the evaluation protocol and the cost ledger alongside the labels. Without those records, a future team could mistake an actively selected pool for a representative sample or compare a new acquisition method against an undocumented mixture of expert interventions. Provenance makes the accumulated evidence interpretable.
The handoff also identifies remaining coverage gaps. A batch can be successful while leaving a newly introduced equipment family unsupported. The release notes should name that limitation rather than letting a broad model label imply universal capability. The next acquisition cycle can then begin with a concrete question instead of repeating the same discovery work.
The most useful artifact is a decision log explaining why each major basket was purchased. It connects selection to the failure or uncertainty the team intended to resolve. That record is more durable than a list of acquisition scores whose meaning depends on an old checkpoint and an undocumented preprocessing pipeline.
The unlabeled pool is not a neutral warehouse
The 80,000 available photographs come from a particular collection process. If one factory uploads every inspection and another uploads only failures, the pool's proportions do not describe the intended deployment population. An acquisition algorithm can optimize selection within that pool while inheriting its blind spots. Cobalt therefore inventories where the images came from before interpreting any coverage claim.
The inventory records collection periods, equipment families, capture devices, and inclusion rules where that information is available. Missing provenance is itself a limitation. The team may need a small new collection from an underrepresented site rather than a more sophisticated selection algorithm over the existing warehouse. No acquisition score can select an example that the pool does not contain.
Pool refreshes are versioned between rounds. If 10,000 new photographs arrive after the first round, the comparison report identifies the changed candidate population. Otherwise a later strategy might appear better simply because it gained access to more relevant images. The experiment should distinguish improvements in selection from improvements in the available raw material.
Protect the holdout from becoming the shopping list
After evaluating a candidate, an engineer wants to label near-duplicates of every failed holdout example. That may be useful for diagnosis, but repeated adaptation to the same evaluation set can make its score increasingly optimistic as a measure of future performance. The holdout is no longer an untouched source of evidence if it guides many rounds of targeted design.
Cobalt distinguishes a development evaluation set used for iteration from a final assessment reserved for a release decision. It also checks grouping across splits so images from one inspection session do not appear on both sides merely because their filenames differ. A near-duplicate leakage path can make an acquisition strategy look effective without demonstrating broader learning.
When the final assessment reveals a new failure pattern, the team can investigate it and plan another cycle. It then needs fresh or appropriately reserved evidence for the next final decision. The point is not to forbid learning from errors. It is to keep track of which evidence has already influenced the system and which evidence can still provide an independent check.
What the purchasing committee approves
Cobalt approves a mixed acquisition plan with independent evaluation, a quality reserve, and a cost-based stopping rule. It treats uncertainty and diversity as candidate tools, not as substitutes for a well-defined labeling task. The plan can adapt between rounds, but every change is recorded and evaluated against the same operational objective.
The first purchase is deliberately smaller than the full budget. Its purpose is to learn about annotation difficulty and guideline quality as well as model uncertainty. Later purchases use that evidence. This makes the program responsive to the actual task rather than committed to a large queue selected before anyone discovered its problems.
A successful cycle leaves the model more useful and the evidence easier to interpret. It may also reveal that the task needs better images, a different target, or a narrower scope. Those are valuable outcomes when they prevent the team from buying thousands of labels that cannot support a reliable decision.
Active learning works best as disciplined information purchasing. The team chooses what it needs to learn, pays attention to the cost of obtaining trustworthy answers, and verifies that the answers improve the decision it set out to make. The number of new labels is a receipt; the quality of the resulting evidence is the product.