SSSamarth Sinha

Note

Point-in-Time Correct Data

Make every past answer reproducible: record when data became known, query as of that moment, and stop the pipeline when data can't be trusted.

Oct 2026 · 1 min read

  • DuckDB or BigQuery
  • Object storage
  • Data contracts
  • Hashing
Point-in-time data flow with a fail-closed control plane Upstream sources to Raw zone. Raw zone to Curated zone. Curated zone to Point-in-time store. Point-in-time store to As-of queries. Raw zone to Dataset registry (hash). Dataset registry to Control plane. Control plane to Curated zone (block). Point-in-time store to Independent rebuilds (rebuild). hash block rebuild Upstream sources files, APIs, databases Raw zone immutable arrival time kept Curated zone contracts, drift versioned Point-in-time store business × knowledge time As-of queries what was known then Dataset registry hashes, freshness bounds Control plane fails closed Independent rebuilds byte-for-byte compare
IllustrativePoint-in-time data flow with a fail-closed control planeScroll →

Context

Any analysis that looks back in time (a backtest of a policy, an audit, a model trained on history) assumes the data describes what was known at that moment.

The problem

Late corrections, restatements and silent upstream changes leak later knowledge into historical queries. A result computed last month can't be reproduced today, and nobody notices until a number is challenged.

  • Upstream sources overwrite history instead of appending to it.
  • Schema drift is fixed in place, so old reads change meaning.
  • Pipelines keep running on stale or partial data because nothing tells them to stop.

How it works

  1. Land raw data immutably and keep two clocks: the business time a record describes and the knowledge time it arrived.
  2. Curate through data contracts (schema, ranges, keys). Drift is versioned, never patched in place.
  3. Serve as-of reads: every query states a knowledge time, and joins take the latest record known at that time.
  4. Verify daily: hash every dataset against a registry and fail closed on a mismatch, a gap or a freshness breach.
  5. Prove determinism: rebuild the canonical tables independently and compare them byte for byte with production.

Trade-offs

  • Storage grows, because history is kept rather than overwritten.
  • Queries get harder: as-of joins need care and good indexing.
  • Fail-closed hurts: a strict control plane stops pipelines, which is the point, but it needs an on-call habit.

When to use it

Use it when a decision will be audited or a result must be reproduced. It's overkill for dashboards that only show the present.