PublishedEngineering
Designing Incremental Blockchain Pipelines
A blockchain indexer is an incremental pipeline over an append-only log that sometimes rewrites its tail. The design has to assume both.
Context
Batch ETL assumes a closed range. A chain does not close. Head follows new blocks; backfill closes historical ranges; reorgs reopen them.
Method
Two workers share one write path. The backfiller claims exclusive ranges. The head follower reads with a two-block lag. Both write rows keyed by (block_number, block_hash).
Findings
Sharing the write path matters more than sharing the fetch path. Divergent writers produce divergent schemas, and the canonical layer cannot repair that.
Implications for the data model
Every table that is derived from a block range must be rebuildable from that range. If it cannot, it does not belong in the pipeline.