DraftEngineering
Building Reliable Backfills
A backfill is a range job with checkpoints, not a one-shot script. Draft notes while the remaining historical ranges close.
Context
Genesis coverage is what makes the datasets usable for research. A hole at block 12 is a hole in every derived table.
Method
Ranges of 200 blocks, checkpoint every 1,000, adaptive RPC batching. Failed ranges go back on a queue rather than stalling the worker.
Findings
Draft. RPC version skew is the main source of retry storms so far.
Implications for the data model
The lineage table records node version per range. A later decoder change can rebuild from raw without re-fetching the chain.