#0032026-04-08
Designing the Raw Data Layer
What I builtA raw layer of four tables — arc.blocks, arc.transactions, arc.logs, arc.traces — written straight from node RPC with no interpretation.
Why it mattersEverything above depends on it. If raw is lossy or reorders fields, every decoder inherits the bug.
ArchitectureNode → fetcher → batch writer → ClickHouse MergeTree, partitioned by block range, ordered by (block_number, tx_index).
ImplementationBatches of 200 blocks; each batch is a single insert keyed by block hash so retries are safe.
Problems encounteredTraces are 30× the size of logs. First schema stored them as JSON strings and queries were unusable.
What I learnedFlatten early. Typed columns beat JSON blobs for anything that gets queried more than once.
What's nextToken transfers as the first derived table (#004).