TL;DR: New items need exposure before a recommender can learn whether people value them, but exposure is itself a decision that changes the data. Use a bounded exploration policy over eligible content, record the actual probability of each displayed choice, and evaluate user outcomes beyond clicks. A cold-start item with few impressions is uncertain; it is not necessarily unpopular.
A fictional reading service called Lattice publishes fifty new articles each day. Its ranking model favors articles with a strong click history. New articles have no such history, so they rarely appear, which prevents them from accumulating evidence. The editorial team calls this unfair. The engineering team calls it cold start. Both are describing a feedback loop in which yesterday's exposure helps determine tomorrow's apparent quality.
This article designs an exploration experiment for that feed. The primary research includes Li and colleagues' contextual-bandit recommendation work, their offline replay evaluation, Schnabel and colleagues' selection-bias analysis, and doubly robust policy evaluation. The Lattice examples are original hypothetical calculations, not reported results from those papers or from a deployed service.
Experiment charter: What deserves a fair trial?
Lattice's first decision is the eligible content pool. Exploration does not mean showing arbitrary material to users. Articles must pass the service's ordinary editorial requirements and match basic language and availability constraints. The experiment chooses among eligible options; it does not replace the process that determines whether an item belongs in the product at all.
The team also defines what “fair trial” means operationally. It might mean a minimum opportunity for new articles to be observed, a controlled amount of uncertainty reduction, or a policy that avoids permanently excluding items with little history. These goals are related but not identical. A fixed number of impressions per article is simple to explain but may ignore audience relevance.
Lattice chooses a limited exploration slot within the feed and evaluates the experience of the whole session. It does not require every new article to receive equal exposure to every user. The policy can use context while ensuring that supported candidates have a documented opportunity to be selected.
The charter states the intended benefit: discover useful new material sooner without a material decline in reader experience. It also names stopping conditions and guardrails. A learning system needs permission to gather evidence, but that permission should be bounded by the product's actual responsibilities to readers.
The first misleading leaderboard
Article A receives 1,000 impressions and 80 clicks. Article B receives 20 impressions and three clicks. Their observed click rates are eight percent and fifteen percent. Ranking B above A solely on those point estimates would ignore the much greater uncertainty in B's estimate.
The comparison is also affected by who saw each article and where it appeared. A may have been shown broadly in a lower position, while B was shown to a small group already interested in its topic. The observed rates combine item appeal, audience selection, placement, and timing. They are not isolated measurements of intrinsic article quality.
Lattice's dashboard therefore displays impressions, clicks, placement, audience context, and uncertainty together. It avoids labeling an item a failure after a handful of exposures. It also avoids declaring a winner because three enthusiastic readers produced an impressive percentage on a tiny denominator.
The team uses the leaderboard as a diagnostic rather than as an automatic promotion rule. The experiment's evaluation needs to account for the policy that generated the observations. Without that context, the recommender can reinforce its own prior choices and mistake the resulting exposure pattern for independent evidence.
A ten-percent exploration branch does not give every item ten percent
Suppose Lattice has five eligible candidates for one slot. Ninety percent of the time, it selects the current greedy choice. Ten percent of the time, it chooses uniformly among all five. The greedy item therefore has total selection probability 92 percent: its 90 percent exploitation probability plus two percent from exploration. Each other item has probability two percent.
The logger must record those total probabilities. Recording “ten percent” for every exploratory display would be wrong because ten percent is the branch probability, not the probability of the particular selected item. Recording only which branch ran is insufficient when later evaluation needs the action probability conditional on the actual candidate set and context.
If the eligible set changes from five items to four, the exploration probabilities change. If some candidates receive weighted exploration, they change again. Lattice computes and logs the probability from the executed policy, including all filters and fallback behavior that affected selection.
This detail is easy to miss because the user sees only one article card. The evaluation system needs the decision process behind that card. A recommender that randomizes without preserving its probabilities may sacrifice short-term relevance while failing to collect the evidence that justified the experiment.
The candidate set is part of the observation
An impression log containing user, article, and click is incomplete for many policy-evaluation questions. Lattice also records which candidates were eligible at the decision point, the relevant context features or their versioned representation, the policy version, and the probability of the selected action.
The candidate set matters because an alternative policy cannot choose an article that was unavailable at that moment. A later catalog snapshot may include items published after the impression or exclude items that have since expired. Reconstructing historical eligibility from the current database can therefore create impossible counterfactual choices.
The team balances useful evidence with data minimization and retention requirements. It need not preserve every raw user attribute indefinitely. It does need a documented representation sufficient for the evaluation it plans to conduct. If a feature cannot be retained, the evaluation claim must reflect that limitation rather than assuming perfect historical reconstruction.
The experiment includes logging validation before any performance comparison. Counts of decisions, delivered cards, visible impressions, clicks, and delayed outcomes are reconciled. A model improvement cannot be trusted if the underlying records systematically omit slow clients or failed renders.
An unshown article has no observed click outcome
For a given request, Lattice observes the outcome of the article it displayed. It does not observe what the same reader would have done with every other eligible article at that moment. Treating unshown items as non-clicks would turn missing counterfactual outcomes into negative labels.
The contextual-bandit formulation makes this partial-feedback structure explicit. Li and colleagues study selection based on context with feedback for the chosen action. Their research provides a framework for learning under that limitation; it does not remove the need to define the reward and ensure that the logged policy supports the intended evaluation.
Lattice distinguishes a visible impression with no click from an item that was never displayed. It also distinguishes a card delivered to the client from one that actually became visible. A user who leaves before scrolling to the slot has not necessarily rejected its article.
The product's training data preserves these states. This prevents the model from learning that low-position content is intrinsically bad simply because many users never reached it. Exposure is a cause of observed interaction, so the pipeline must retain enough information to avoid confusing the two.
A small replay experiment
Offline replay can evaluate a candidate policy using suitably randomized historical data by retaining events where the candidate's choice matches the logged action. Li and colleagues analyze this approach under its stated setup. Lattice uses it as one diagnostic when the data and policy satisfy the required conditions.
The practical cost is that many logged events may not match the candidate policy's action. With a large action space or a very different policy, the retained sample can become small. The team reports how much data remains and which contexts are represented rather than presenting a replay estimate as though it used every impression equally.
A deterministic historical policy that never showed an alternative cannot provide direct evidence about that alternative's outcomes. No replay filter can recover observations that were never collected. This is why deliberate exploration can be valuable: it creates support for evaluating choices beyond the incumbent's favorites.
Lattice does not use replay as a universal replacement for an online experiment. It uses the result to reject obviously weak candidates and identify uncertainty before a bounded rollout. The final decision still considers logging quality, population changes, and the ways a new policy may alter user behavior over time.
The inverse-probability worksheet
Inverse-probability methods reweight observed outcomes according to how likely the logging policy was to select the action. The idea addresses the mismatch between the historical exposure policy and a target policy, under assumptions including adequate support and correct probabilities. Schnabel and colleagues discuss selection bias in recommendation, while the policy-evaluation literature develops related estimators.
A tiny illustrative worksheet has four logged events. A deterministic target policy matches the logged action on two events. Each matched action had logging probability one half. One matched event has reward one and the other reward zero. The weighted reward sum is two, and dividing by four events gives an estimate of 0.5. This is arithmetic demonstrating the weighting rule, not a meaningful performance estimate from four observations.
Small action probabilities create large weights and potentially high variance. If an item was shown only rarely, a single observed reward can have substantial influence. Lattice examines weight distributions, effective information, and sensitivity rather than assuming that unbiasedness in an idealized setting guarantees a stable estimate in its finite dataset.
Clipping weights or using other stabilizing choices can introduce a bias-variance tradeoff. Those choices belong in the methodology record. An analyst should not quietly tune them until the preferred policy looks best and then report the resulting number as an objective verdict.
A reward model can help and still be wrong
Doubly robust policy evaluation combines outcome modeling with propensity-based correction under its assumptions. Dudík and colleagues develop this approach for partially observed rewards. The method can improve estimation, but the name should not be interpreted as immunity to every logging defect, unsupported action, or population shift.
Lattice keeps the reward model's development separate from the final policy comparison where the chosen evaluation procedure requires it. It checks predictions on appropriate held-out data and documents which features were available at decision time. Features created after the user clicked would leak outcome information into a model intended to predict that click.
The team also inspects disagreement among estimators. If a direct reward-model estimate, a weighted estimate, and replay tell very different stories, that is a reason to investigate support, model fit, and data quality. Averaging the numbers without understanding the discrepancy would hide the useful warning.
Offline evaluation is most valuable when it narrows uncertainty honestly. It can show that the data is insufficient to distinguish two policies. That result should guide a better experiment rather than pressure analysts to manufacture a decisive ranking from a weak evidence base.
Fifty new articles divide a limited exploration budget
Lattice serves 10,000 eligible requests in a day and allocates ten percent to exploratory selection. That creates about 1,000 exploratory choices. If those choices were distributed evenly across fifty new articles, each would receive about twenty exploratory exposures on average. Actual contextual eligibility can make the distribution uneven.
At an illustrative five-percent click probability, twenty exposures yield one expected click. Observing zero, one, or two clicks would not provide a stable ranking of article quality. The arithmetic makes clear why a large daily publication volume can overwhelm a modest exploration budget.
The team can respond by extending the learning window, reducing the eligible pool, using informative content features, or allocating exploration adaptively. Each choice has consequences. A longer window may outlast a time-sensitive article's relevance. A smaller pool can favor editorially selected items. Adaptive allocation needs careful logging and evaluation.
The budget should therefore be planned in terms of the evidence needed for decisions, not just as a percentage that sounds cautious. Ten percent of a small traffic stream may be too little to learn much, while ten percent of a large stream may be more disruption than necessary.
Content features provide a starting point, not a verdict
A new article can inherit information from its topic, format, language, author history, or text representation. These features help the model make an initial estimate before item-specific interactions accumulate. They can reduce cold-start uncertainty without pretending that the new item has already been tested.
Lattice checks whether such features create systematic disadvantages. A new author may lack history, and a new topic may be poorly represented in past traffic. Falling back to an average can be reasonable, but treating missing history as evidence of poor quality would reinforce the same exclusion loop the experiment is meant to address.
The team distinguishes exploration for new items from exploration for new users. A familiar article can be uncertain for a reader with no preference history, while a new article can be relevant to a well-understood audience. The policy's context should reflect which uncertainty it is trying to reduce.
The interface can also ask users for preferences directly where that is useful and proportionate. A few explicit topic choices may provide better initial evidence than inferring everything from sparse clicks. Those choices should remain editable because reader interests are not fixed identities.
A click can be the wrong success signal
Lattice's initial reward is a click, but the editorial team notices that sensational titles can attract clicks followed by immediate abandonment. Optimizing clicks alone may improve the metric while reducing satisfaction. The experiment therefore defines additional outcomes such as meaningful reading, saves, hides, and explicit feedback, with clear measurement rules.
Each outcome has limitations. Time on page can reflect confusion or an idle tab. A save may indicate future interest without actual reading. A hide can express dislike of a topic, an author, or repetition. The team should not pretend that a weighted combination of noisy signals becomes a direct measurement of human value merely because it is called an engagement score.
The reward design starts from the product's purpose: help readers find useful material. It uses several signals and periodic user research to check whether the optimization remains aligned with that purpose. The policy experiment includes guardrails that can stop a rollout even when the primary click metric rises.
Delayed outcomes require a defined observation window. Comparing a new policy's same-day clicks with an old policy's fully matured seven-day saves would be unfair. The analysis records when outcomes become available and avoids treating incomplete follow-up as a true zero.
The feed is a sequence, not an isolated slot
One article can affect whether a reader continues scrolling, returns tomorrow, or trusts the feed. A slot-level reward captures only part of that experience. If exploration repeatedly interrupts a coherent reading session with unrelated material, the policy may learn about individual items while degrading the product as a whole.
Lattice limits exploration frequency within a session and accounts for recent exposures. Repeating the same new article can generate more impressions without providing independent evidence about its appeal. The user may also click because of repetition rather than because the item would have been attractive on first exposure.
The experiment's unit is chosen to match the behavior under study. Randomizing requests can mix policies within a user session; randomizing users can better preserve a consistent experience but requires enough users and attention to repeated outcomes. The analysis should account for the chosen unit instead of treating every click as an independent person.
The team also examines catalog diversity and repetition as user-facing outcomes. These are not automatic substitutes for relevance, but they can reveal a feed that has become narrow or exhausting. A policy should be judged on the experience it creates, not only on the immediate reward attached to each isolated card.
The online trial has a decision schedule
After offline checks, Lattice runs a bounded randomized trial with a documented primary outcome, guardrails, and analysis plan. It specifies the population, duration, and conditions for stopping or extending the experiment. The team avoids repeatedly checking ordinary significance thresholds and ending the trial at the first favorable result without an appropriate sequential method.
The rollout begins with a limited population so logging and operational behavior can be verified. A bug in candidate eligibility or propensity recording can invalidate the learning objective even if users do not report an obvious defect. The team confirms that the executed policy matches the policy described in the experiment charter.
Results are examined across meaningful user groups and content slices, with uncertainty and multiplicity considered. A broad average improvement can coexist with a serious decline for a smaller audience. The response may be a scoped policy, further evidence collection, or rejection of the candidate rather than a universal launch.
The decision record states what changed, what was measured, and which uncertainties remain. A successful experiment is not merely one with a positive metric. It is one that produces enough reliable evidence to choose the next product behavior responsibly.
Exploration leaves a maintenance obligation
The article catalog, audience, and ranking model change over time. An exploration policy that worked during one trial may collect poor evidence later if eligibility rules or logging semantics drift. Lattice versions those components and monitors whether selection probabilities remain valid after changes.
The team also keeps a small continuing source of representative evidence where appropriate. Without it, a successful policy can gradually become self-confirming: it shows what it already believes is best and receives feedback only about those choices. The size and form of ongoing exploration should follow the product's learning needs and user costs.
When a new policy is introduced, historical data is assessed for support before reuse. A policy that chooses previously unexposed topics cannot be evaluated confidently from a log dominated by the old catalog. More data is useful only when it contains relevant evidence about the decisions now being considered.
The maintenance owner can explain why each randomization remains in the product. Exploration should not survive indefinitely as an unexplained percentage in a configuration file. It exists to answer a question, and the question should be revisited as the system and its audience evolve.
The feed earns the right to learn
Lattice's approved policy gives eligible new articles a bounded opportunity to be observed, records the probabilities behind those choices, and evaluates both immediate interaction and the wider reading experience. It treats sparse evidence as uncertainty rather than as proof that an item lacks value.
The final review also checks failure paths. If the exploration service times out and the application falls back to a default article, the logged action and probability must describe that fallback rather than the choice originally requested. A decision record written before the display is finalized can otherwise claim that the reader saw one item when another actually appeared. The experiment reconciles selection, delivery, and visibility identifiers to catch this mismatch.
The same rule applies when an article becomes unavailable between selection and rendering. The replacement is another decision with its own eligibility and probability. Treating it as a harmless implementation detail can corrupt the evidence used by every later policy comparison. Reliable experimentation requires the data to describe what the product did, including its ordinary recovery behavior, rather than only what the ranking model intended.
The team can now distinguish an article that received a poor trial from one that received almost no trial at all. It can also recognize when its logs cannot support a proposed comparison. Those distinctions make editorial and engineering decisions more honest, even when they do not produce a simple leaderboard.
Cold start is partly a modeling problem and partly a data-collection policy. The recommender creates the evidence it later learns from. Designing that process deliberately helps prevent the system from confusing its own exposure choices with independent judgments from readers.
A useful exploration program gives the product a controlled way to discover what it does not yet know. It earns that opportunity by preserving reader experience, documenting the decision process, and turning observed behavior into evidence with the limitations still attached.