SSSamarth Sinha

Note

Delta Lake, Apache Iceberg and the open lakehouse

Open table formats let several engines share one copy of data. What they give you, and the decisions that matter more than the format.

Nov 2026 · 1 min read

  • Lakehouse
  • Delta Lake
  • Iceberg
One open copy of data, many engines Writers to Table format. Table format to Table metadata. Table format to Data files. Table metadata to Catalog. Catalog to SQL warehouse. Catalog to Spark jobs. Catalog to AI and ML. Writers batch and streaming Table format Delta Lake or Iceberg Table metadata snapshots and schema Data files Parquet, object storage Catalog names and access SQL warehouse BI and analytics Spark jobs data engineering AI and ML features and training
IllustrativeOne open copy of data, many enginesScroll →

What a table format adds

Parquet files alone are just files. A table format such as Delta Lake or Apache Iceberg adds a transaction log or metadata tree on top: ACID writes, schema evolution, time travel to earlier snapshots, and the ability for several engines to read and write the same tables safely.

Delta Lake and Iceberg, briefly

  • Delta Lake grew up with Spark and Databricks; its transaction log sits next to the data.
  • Apache Iceberg was designed for engine independence, with a metadata tree and hidden partitioning, and is widely supported across engines and clouds.
  • The gap has narrowed: both support time travel, schema evolution and concurrent writers, and there are ways to expose one format's metadata to readers of the other.

Decisions that matter more than the format

  1. The catalog. Which catalog is the source of truth for table names and permissions? Two catalogs means two answers.
  2. Who writes. Many readers are easy; several writers to one table need care with concurrency and compaction.
  3. Table maintenance. Compaction, snapshot expiry and clean-up of old files are not optional at scale.
  4. Partitioning. Partition on how data is filtered; Iceberg's hidden partitioning helps prevent user mistakes.

When it pays off

When more than one engine needs the same data (a SQL warehouse, Spark pipelines and ML training), one open copy beats three synchronised ones.