A blockchain indexer is not correct merely because it can apply new events. If it serves data before finality, it must also reverse effects from blocks that leave the canonical chain and replay their replacements. A saved block number is insufficient evidence that the database represents the chain it claims to represent.
TL;DR: Store block hashes and parent links, identify event occurrences by their block context, commit projections with their checkpoint, and make rollback an explicit operation. Treat subscriptions as notifications rather than a complete durable history. The replay below uses a fictional application ledger to show how balances, aggregates, caches, and external actions need different recovery treatment.
Replay starts at a known ancestor
The fictional application, HarborLedger, indexes deposits and transfers emitted by a particular contract. Its database is consistent through block one hundred, identified by hash H100. Alice has one hundred units, Bob has zero, and Carol has zero in the application's derived ledger. These are illustrative application balances, not a claim that every token's balance can be reconstructed from a generic transfer event alone.
The next observed block is A101, whose parent is H100. It contains a transfer of ten units from Alice to Bob. HarborLedger applies the event and records balances of ninety, ten, and zero. It stores the block's number, hash, parent hash, and the event's occurrence identity.
Block A102 follows with parent A101 and contains a transfer of four units from Bob to Carol. The derived balances become ninety, six, and four. The database checkpoint says that the canonical projection has been applied through hash A102 at height one hundred two. Both the height and hash matter.
Now the provider reports a different branch: B101 descends from H100, B102 descends from B101, and B103 descends from B102. The application cannot append B103 to its A102 state merely because one hundred three is the next number. The parent relationship does not match the database's current head.
Ethereum's JSON-RPC data includes block identity and parent information needed to track that relationship. The application must use those fields to establish continuity instead of treating block height as a unique permanent identifier. Ethereum JSON-RPC documentation
HarborLedger pauses ordinary forward projection and finds the common ancestor H100. The task is now two-sided: remove the effects of A102 and A101, then apply B101, B102, and B103 in order. The source chain has changed its accepted history, so the derived view must change with it.
Ledger exhibit: reverse in the opposite order
Assume B101 transfers seven units from Alice to Carol, B102 transfers two units from Carol to Bob, and B103 contains no relevant event. Starting from H100, the correct new balances are Alice ninety-three, Bob two, and Carol five. The empty block still matters because it extends the canonical ancestry and checkpoint.
| Step | Alice | Bob | Carol | Applied head |
|---|---|---|---|---|
| Common ancestor | 100 | 0 | 0 | H100 |
| Apply A101 | 90 | 10 | 0 | A101 |
| Apply A102 | 90 | 6 | 4 | A102 |
| Undo A102 | 90 | 10 | 0 | A101 |
| Undo A101 | 100 | 0 | 0 | H100 |
| Apply B101 | 93 | 0 | 7 | B101 |
| Apply B102 | 93 | 2 | 5 | B102 |
| Apply empty B103 | 93 | 2 | 5 | B103 |
Reversal proceeds from the latest applied effect toward the ancestor. In this simple additive ledger, reversing the numerical deltas restores the earlier balances. More complex projections may require previous values, a reconstruction from retained events, or another explicit inverse operation. The important design choice is to know how each projection can be restored before relying on it in production.
The total remains one hundred throughout this example because every event is a transfer inside the three-account system. That invariant helps detect some mistakes, but it does not prove correctness. A wrong transfer from Alice to Bob can preserve the total while assigning funds to the wrong account. The test must verify individual state as well as aggregate conservation.
The orphaned A-branch records remain in a raw observation history with a noncanonical status. They are excluded from the current projection but retained for investigation. Deleting them immediately would make it harder to explain why a user saw a balance change and how the application recovered.
The new canonical branch is not declared complete until its relevant events and block records have been processed through the selected head. A partial rollback followed by an interrupted replay should not be exposed as a normal up-to-date balance view. The application needs a visible recovery state or a consistent prior snapshot for readers.
Event identity includes the occurrence, not only the transaction
A transaction can appear in an orphaned block and later be included in another block. A subscription can therefore report information related to the same transaction more than once. Deduplicating solely by transaction hash can suppress a valid event on the replacement branch, while ignoring identity entirely can apply the same occurrence twice.
HarborLedger identifies a raw log occurrence using its chain, block hash, and log index, with transaction identity retained as an additional field. This distinguishes where the event occurred from the transaction that produced it. The canonical projection includes only occurrences belonging to the selected canonical branch.
Within a block, event order must be deterministic. The application preserves transaction and log order supplied by the chain data rather than relying on the arrival order of parallel network responses. A projection that applies balance-dependent business logic can produce different results if events are shuffled even when it receives the same set of logs.
Geth's subscription documentation explicitly describes removed logs after reorganizations and repeated emissions associated with transactions on a new chain. It also notes that subscriptions concern current events and are tied to the connection. These semantics support using notifications to trigger reconciliation, not assuming a permanent exactly-once delivery stream. Geth real-time events
The application's idempotency rule is therefore state-aware. Re-receiving an already stored occurrence does not apply it again. Receiving the same transaction in a different block does not automatically mean the new occurrence should be discarded. A branch transition determines which occurrence contributes to the current projection.
A removed notification is useful evidence, but HarborLedger does not depend on receiving every such notification. The connection may fail during the reorganization. On reconnect, the application compares stored ancestry with the provider's chain and reconstructs the necessary transition from durable block data.
An empty log response needs a block identity
Suppose HarborLedger asks for logs at height one hundred two and receives an empty list. During the request, the canonical block at that height changes. Without a block identity attached to the query, the empty response alone cannot establish which version of the height was inspected.
EIP-234 adds a block-hash option for log filters so a client can request logs for a particular block rather than only a numerical range. That distinction is especially useful in the empty-result case because there is no returned log carrying a block hash to inspect. EIP-234
The application first establishes the block it intends to process, retrieves logs for that block through a supported hash-specific path, and verifies the returned context. If the provider cannot serve the requested block, the application treats that as a retrieval failure or a reason to reconcile ancestry, not as proof that the block contained no relevant events.
State reads used during projection need similar care. If a handler reads a contract value at latest while applying an older event, it can combine data from different chain states. The resulting projection may be impossible to reproduce even without a reorganization because latest changes between calls.
EIP-1898 specifies block-hash parameters for several state-query methods and an option concerning canonicality. Where supported, this lets an integration bind related reads to a specific block context. Provider support and historical-state availability still need to be verified. EIP-1898
For HarborLedger, the preferred projection derives state from the event payload when that is sufficient under the contract's semantics. When additional state is required, the handler records the exact query context and response. A replay should not silently substitute today's contract state for the historical state that informed the original calculation.
Commit the projection and its checkpoint together
A crash can occur after updating balances but before saving the processed head. On restart, a naive worker replays the same block and applies the balances twice. The reverse ordering is also dangerous: saving the head first can cause the worker to skip effects that were never committed.
HarborLedger groups each projection update with its event-application records and checkpoint change in a database transaction. Either the complete unit is committed or none of it is. PostgreSQL's transaction model provides the all-or-nothing behavior needed for this local consistency boundary. PostgreSQL transaction documentation
This does not make the database transaction atomic with the blockchain itself. The application still needs to reconcile the committed block hash with the chain later. The database transaction prevents a local partial update; ancestry checks handle changes in the external source of truth. These are complementary mechanisms with different responsibilities.
Rollback uses the same discipline. The inverse effects, canonical-status changes, and checkpoint movement should form a consistent transaction boundary. If the application reverses a large range in batches, it needs an explicit recovery marker and a way to prevent readers from mistaking an intermediate state for a complete canonical projection.
Concurrent workers require ownership rules. Two workers must not independently apply or undo the same chain segment against one mutable projection. A lock, serialized coordinator, or other carefully designed mechanism can enforce that boundary. The choice depends on the database and architecture, but the invariant is that the projection has one coherent ancestry transition at a time.
Parallel fetching can still improve throughput. Workers may retrieve and decode blocks ahead of the applied head, while a coordinator commits them in verified parent order. Separating acquisition from application allows performance without treating network completion order as chain order.
Some projections are not reversed by subtracting a number
The balance example uses additive changes, but applications also build rankings, maximum values, ownership histories, and distinct counts. These structures need different rollback strategies. If an orphaned event established the highest bid, subtracting its amount does not reveal the previous highest bid.
HarborLedger's auction projection stores the underlying bids and derives the highest canonical bid from eligible records. When a bid's block becomes noncanonical, the projection can recompute the winner from the remaining history. A separate previous-value journal could also work if its ordering and recovery semantics are correct.
A distinct-user count offers another example. Removing one event from Alice should not reduce the count if Alice still has another canonical event in the measured period. A simple increment on every new user and decrement on every removed event will fail. The projection needs membership counts or a recomputation that respects the definition of distinctness.
Time-windowed aggregates add an additional axis. A block reorganization can alter event timestamps and membership in daily summaries. The application must identify which windows are affected rather than updating only the latest total. A corrected all-time balance does not guarantee that historical charts are now consistent.
The projection catalog therefore lists, for each derived table, its source events, ordering assumptions, reversal method, and recomputation scope. This is a practical design artifact: it reveals projections that cannot currently be recovered before an incident forces the team to improvise.
Raw event retention makes rebuilds possible, but only if decoding and transformation versions are also recorded. A new decoder may interpret historical data differently. The application should distinguish a chain reorganization from a software migration and have a controlled way to rebuild projections under a new version.
Notifications and external actions cannot always be undone
A database balance can be reversed. An email already delivered cannot be unsent, and a physical shipment cannot be made never to have happened. An indexer that triggers external actions needs a policy for when chain evidence is sufficient to authorize those actions.
HarborLedger separates observed, provisionally applied, and finalized evidence in its product model. A provisional notification may say that a transaction was observed and later update its status. An irreversible fulfillment action waits for the evidence threshold defined for that workflow. The threshold should reflect the chain's actual finality model and the consequence of error.
Ethereum's proof-of-stake finality mechanism is not simply a fixed count of later execution blocks. Its consensus documentation distinguishes fork-choice behavior from checkpoint finality. An application should avoid presenting an arbitrary confirmation count as an unconditional finality guarantee. Ethereum Gasper documentation
The indexer can use provider-supported safe or finalized block references where appropriate, while verifying that its environment and provider expose the expected semantics. It should also report when finality information is unavailable or stalled rather than advancing a label based only on elapsed wall-clock time.
For downstream jobs, HarborLedger uses durable intent records tied to the evidence version that authorized them. Before execution, the worker verifies that the intent remains eligible. This reduces the chance that a queued action based on an orphaned event runs after the database has already rolled back that event.
The design still needs a response for an action that completed before the source changed. Compensation may be possible, such as reversing an internal credit or sending a correction. Where it is not, the system should preserve an audit trail and escalate the discrepancy. Reversibility of the index is not a promise that every consequence outside it is reversible.
Recovery rehearsals begin with a saved branch graph
The test fixture contains H100, A101, A102, B101, B102, and B103 with their parent links and events. The test first applies the A branch, then presents the B branch and checks the complete rollback and replay. It verifies account balances, canonical event membership, checkpoint identity, and affected aggregates.
Another test disconnects the subscription before the removed notifications arrive. On reconnect, the worker must discover the branch mismatch through ancestry reconciliation. A test that passes only when every notification is delivered in order has not established recovery from a realistic connection interruption.
Crash tests stop the worker before and after each local transaction boundary. Restart should either observe a fully committed block application or safely retry an unapplied block. Replaying the same fetched data twice should leave the projection unchanged after the first complete application.
The fixture also includes an empty block, a transaction re-included on a replacement branch, multiple logs from one transaction, and a reorganization deeper than the ordinary overlap scan. Each case exercises a different assumption. A happy-path test with one transfer per block cannot establish those properties.
If the common ancestor lies outside retained reversible history, the application should enter a defined rebuild path from a trusted checkpoint or snapshot. It should not guess an inverse from incomplete data. The retention policy and recovery objective must agree about how much history is available for automatic repair.
Provider disagreement is a reconciliation problem of its own
Suppose the primary provider reports B103 while a fallback provider still reports A102. The indexer should not alternate its projection between them on every request. It needs a documented source-selection and consistency policy, including how it detects a lagging provider, verifies chain identity, and pauses when the evidence is contradictory.
A fallback endpoint is not interchangeable merely because it answers the same RPC method. It may serve a different network, lag behind the head, lack historical state, or enforce different query limits. HarborLedger verifies the expected chain and checks the block context of responses before combining them with an existing recovery operation.
The application also distinguishes an unavailable response from an empty result. If a provider rejects a log range because it is too large, the worker should split or retry according to its retrieval policy. It should not record the range as processed with no events. A successful checkpoint is evidence that the requested data was obtained and applied, not simply that an HTTP request completed.
Backfill and live ingestion can overlap safely when they share occurrence identity and a serialized application boundary. The backfill worker may fetch a block already observed through the subscription, but the stored occurrence prevents duplicate application. The coordinator still verifies ancestry because two observations at the same numerical height may refer to different branches.
A useful operational metric is the gap between the provider's selected head and the database's applied head, accompanied by both hashes and the recovery status. Height lag alone can look healthy while the database remains on an obsolete branch at the same height. The identity of the head is part of the health signal.
A checkpoint snapshot needs a reproducible boundary
A snapshot can reduce the cost of rebuilding an old projection, but it must correspond to a known block hash and transformation version. A file labeled snapshot at block one million is incomplete if the underlying extraction mixed rows from different database moments. The snapshot creation process should use a consistent database view and record the boundary it represents.
The restoration procedure verifies that the snapshot's chain identity and ancestor relationship match the history being replayed. It then applies later canonical blocks using the same deterministic rules. If the decoder version has changed, the operator decides whether the snapshot remains compatible or whether an earlier rebuild is required.
The retained raw event archive and the snapshot have different roles. The snapshot accelerates restoration of derived state; the raw archive preserves evidence needed for auditing or rebuilding under a changed transformation. Deleting all historical events because a snapshot exists can make future corrections impossible when the snapshot itself contains a projection bug.
A recovery exercise should measure how long restoration takes and whether serving behavior remains coherent during the rebuild. If the application promises a short recovery interval but the only tested rebuild takes several days, the architecture has an unresolved operational gap. That gap should influence retention, checkpoint frequency, and the product's availability expectations.
The final verification compares a restored projection with an independently replayed reference on a bounded fixture or sample. Matching row counts are insufficient; the comparison should include balances, ownership, aggregates, and the exact applied head. This establishes that the recovery path reconstructs the same meaning, not merely a database of similar size.
The served balance includes its evidence boundary
HarborLedger's API can return the applied block hash and number with a balance response, plus a clear status indicating whether the result is provisional or based on the chosen finality boundary. Clients then have enough context to understand why a recently observed balance may change.
Caches use the projection version or block context so an orphaned balance is not served indefinitely after the database has recovered. Search indexes and analytics consumers receive correction or rebuild signals appropriate to their design. Recovery is complete only when the user-facing derived views agree with the repaired canonical state.
The essential invariant is simple: the application can explain which chain history produced each served projection and can replace that projection when the accepted history changes. An indexer that can only add events has implemented ingestion. An indexer that can reliably unlearn and replay has implemented a recoverable view of the chain.