Apache Iceberg: tables on object storage

What you will be able to do

• Explain why a directory of Parquet files is not a table and what Hive-style tables got wrong on S3
• Walk the Iceberg metadata tree from catalog to data file and find each level in the lab bucket
• Describe how a commit stays atomic without locks or rename
• Use snapshots, time travel, and schema evolution across Spark and Trino

Why this matters for data work

In the last lesson you saw that object storage has no rename, no append, no locks, and a slow LIST. Yet Spark wrote bronze.orders, Trino read it a second later, and both agreed on 300 rows. Something between the engines and the bucket is doing the bookkeeping. That something is Apache Iceberg, an open table format: a specification for the metadata files that turn a pile of Parquet objects into a table with a schema, a history, and atomic commits.

Iceberg is what lets you swap engines — Spark today, Trino, Flink, DuckDB, or a warehouse tomorrow — without moving data. Snowflake, Databricks, BigQuery, AWS, and Google all read and write it. Once you can draw its metadata tree from memory, most lakehouse features stop being magic.

Concepts

File format, table format, catalog, engine

Four different words that a vendor slide often blurs:

Layer	Question it answers	In this lab
File format	How are rows encoded in one file?	Parquet
Table format	Which files make up the table right now, with what schema and partitioning?	Iceberg
Catalog	Where is the current metadata for table bronze.orders?	RustFS Iceberg REST at /iceberg
Engine	Who runs the SQL?	Spark (writes), Trino (reads)

Iceberg is not a service and not a process. It is a spec plus libraries the engines load (the iceberg-spark-runtime jar in the Spark image; the iceberg connector in Trino). Nothing in docker compose ps is called Iceberg.

The problem: Hive tables on object storage

Before Iceberg, a “table” on a lake was a directory. The Hive metastore remembered the directory path and a schema; the files were whatever LIST returned under that path, and partitions were sub-directories named dt=2026-09-18/. That worked on HDFS. On S3 it breaks in five ways:

1. Finding files is slow. A query planner must LIST every partition directory, thousands of paginated calls, before reading a byte.
2. No atomic commit. A job writing 200 files takes minutes. Readers that list mid-way see 120 of them, and a crashed job leaves garbage that looks like data.
3. Partitioning is exposed. Users must know the table is partitioned by dt and write WHERE dt = …; filtering on order_ts scans everything. Changing the partition layout means rewriting the table.
4. Schema changes are by position or name. Rename a column and old files disappear from the query; drop and re-add one and you read the wrong data.
5. No history. There is no “what did this table look like an hour ago,” so no safe rollback.

Iceberg’s move: stop trusting the directory. Write the complete file list, schema, and partition spec into metadata files, and make the only mutable thing one pointer in the catalog.

The metadata tree

  catalog  (RustFS Iceberg REST)
     |  "smoke.events" -> current metadata file
     v
  metadata.json          table root: schemas, partition specs, snapshots,
     |                   current-snapshot-id, properties
     v
  manifest list          one per snapshot   snap-<id>-….avro
     |                   lists manifests + partition ranges per manifest
     v
  manifest               ….-m0.avro
     |                   lists data files + per-column min/max, null counts,
     |                   row counts, partition values
     v
  data files             ….parquet   (and delete files in v2 tables)

Every level is an immutable object. A snapshot is the table at one commit: a manifest list plus the set of manifests it points at. A new commit writes new data files, a new manifest for them, a new manifest list that reuses the unchanged manifests, and a new metadata.json that carries the whole snapshot history. Then it swaps the catalog pointer.

Where this lives in your lab, from the objects you listed last lesson:

Level	Lab object
Catalog entry	Inside RustFS; served at http://127.0.0.1:19100/iceberg/v1/…
metadata.json	s3://lakehouse/.rustfs-table/warehouses/default/namespaces/smoke/tables/events/metadata/00002-….metadata.json — a hidden prefix the catalog owns; a plain LIST does not show it
Manifest list	s3://lakehouse/tables/<uuid>/metadata/snap-3597839601582952851-1-….avro
Manifest	s3://lakehouse/tables/<uuid>/metadata/80c8b78a-….avro (-m0)
Data files	s3://lakehouse/tables/<uuid>/data/00000-0-….parquet

On plain S3 with a Hive or Glue catalog, metadata.json sits next to the manifests under metadata/ with the version in its name (v3.metadata.json). RustFS keeps it in its own space, the way AWS S3 Tables does. Same tree, different parking spot.

How a commit stays atomic

A writer never touches an existing object. It:

1. Writes new data files (PUT, immutable).
2. Writes a new manifest listing them, and a new manifest list.
3. Writes a new metadata.json whose current-snapshot-id is the new snapshot.
4. Asks the catalog: “if the current metadata is still 00002-…, make it 00003-….”

Step 4 is a compare-and-swap in the catalog — the one place with a real atomic operation. If another writer committed first, the swap fails, the writer re-reads the table and retries on top of the newer snapshot (optimistic concurrency). Readers always start from a pointer, so they see a whole snapshot or the previous whole snapshot, never a mix. No locks, no rename, no LIST.

This is also why the lab warns you off CREATE TABLE AS SELECT: Spark implements it with a REST stage-create handshake that RustFS 1.0 does not offer. CREATE TABLE then INSERT uses the plain commit path above.

Features that fall out of the tree

Snapshots and time travel. metadata.json keeps every snapshot until you expire them. SELECT … FOR VERSION AS OF <snapshot-id> (Trino) or VERSION AS OF (Spark) reads the old manifest list. Rollback is a pointer move.

Schema evolution by column id. Every column has an integer id in the schema, and Parquet files store those ids. Renaming a column changes a name in metadata only; adding a column gives old files NULL for it; dropping one hides it. No file is rewritten, and no “wrong column” accidents when types match by position.

Hidden partitioning. A partition spec is a transform on a column — day(order_ts), bucket(16, customer_id). Writers apply it; readers filter on order_ts and Iceberg prunes. Users never learn a synthetic dt column, and a partition evolution changes the spec for future files without rewriting old ones.

Planning from statistics. Each manifest entry carries min/max per column, null counts, and row counts. WHERE order_ts > … skips whole files without opening them, and skips whole manifests using the partition ranges in the manifest list. The planner reads a few kilobytes of Avro, not the bucket listing.

Row-level changes. Iceberg v2 supports UPDATE, DELETE, and MERGE. Copy-on-write rewrites the affected data files. Merge-on-read writes small delete files (position or equality deletes) that readers apply on the fly — faster writes, slower reads until compaction.

Maintenance. Every commit adds files; nothing removes them. Real tables schedule expire_snapshots (drops old history and the files only it referenced), rewrite_data_files (compacts small files), and remove_orphan_files (cleans up crashed writes). Skip these and the bucket grows forever.

Many engines, one truth. Because the truth is in files plus a catalog, any engine with an Iceberg library can read or write — Spark, Trino, Flink, Hive, DuckDB, pyiceberg, Snowflake, BigQuery, Athena. The catalog protocol matters here: REST is the vendor-neutral one; Hive Metastore, AWS Glue, Nessie, JDBC, and Hadoop (directory-based, no real atomic swap on S3) are the others.

Iceberg beside Delta Lake and Hudi

	Iceberg	Delta Lake	Hudi
Origin	Netflix, 2017	Databricks, 2019	Uber, 2016
Commit log	Snapshot tree; catalog pointer	_delta_log/ JSON + checkpoints in the table directory	Timeline in .hoodie/
Partition evolution	Yes	No (rewrite)	Limited
Broadest engine support	Yes	Spark first, others growing	Spark/Flink first

All three solve the same Hive problem. Iceberg is the one most vendors now converge on, which is why this lab and the modules after it use it.

Practice on your machine

Lab up from Getting Started, with smoke.events created by smoke.sql. Trino’s metadata tables ("<table>$snapshots", $files, $manifests, $history, $metadata_log_entries) expose the tree without opening any Avro by hand.

Define a short alias for the Trino CLI in this terminal (Trino prints a harmless jline warning about the terminal each time):

cd ~/dcubes/lakehouse-lab
trino() { docker compose exec trino trino --server localhost:8080 --catalog iceberg --schema smoke --output-format ALIGNED --execute "$1"; }

1. One snapshot after smoke.sql

trino 'SELECT committed_at, snapshot_id, parent_id, operation FROM "events$snapshots"'

        committed_at         |     snapshot_id     | parent_id | operation
-----------------------------+---------------------+-----------+-----------
 2026-09-18 01:32:44.621 UTC | 3597839601582952851 |      NULL | append
(1 row)

Ids will differ on your machine. parent_id is NULL because this is the first commit. Now the data files it references:

trino 'SELECT file_path, record_count, file_size_in_bytes FROM "events$files"'

Two Parquet objects under s3://lakehouse/tables/<uuid>/data/, one row each — Spark used two tasks for the two-row INSERT.

2. Append from Spark, watch a second snapshot appear

Copy the snapshot_id from step 1; you need it for time travel. Then add a row with Spark:

docker compose exec spark /opt/spark/bin/spark-sql -e "INSERT INTO lakehouse.smoke.events VALUES (3L, 'gamma')"

trino 'SELECT snapshot_id, parent_id, operation FROM "events$snapshots" ORDER BY committed_at'

     snapshot_id     |      parent_id      | operation
---------------------+---------------------+-----------
 3597839601582952851 |                NULL | append
 7480765389787554687 | 3597839601582952851 | append
(2 rows)

The new snapshot names its parent. List the files again: three Parquet objects, and the first two have the same paths as before — the append added a file, it did not touch the existing ones.

3. Time travel

Read the table as it was at the first snapshot (paste your own id):

trino 'SELECT * FROM events FOR VERSION AS OF 3597839601582952851 ORDER BY id'

 id | payload
----+---------
  1 | alpha
  2 | beta
(2 rows)

Without the clause you get three rows. Nothing was restored or copied — Trino followed the old manifest list.

4. Evolve the schema in Spark, read it in Trino

Add a column and a row that uses it. Spark commits once for the ALTER and once for the INSERT:

docker compose exec spark /opt/spark/bin/spark-sql -e "ALTER TABLE lakehouse.smoke.events ADD COLUMN source STRING; INSERT INTO lakehouse.smoke.events VALUES (4L, 'delta', 'lesson');"

Trino never restarted and has no cache to clear:

trino 'SELECT * FROM events ORDER BY id'

 id | payload | source
----+---------+--------
  1 | alpha   | NULL
  2 | beta    | NULL
  3 | gamma   | NULL
  4 | delta   | lesson
(4 rows)

Old files were not rewritten; they simply lack column id 3, so it reads as NULL. Check the snapshot chain — you should now see three appends in a line (the ALTER changes the schema in metadata.json but creates no snapshot).

5. Find the root of the tree

trino 'SELECT file, latest_snapshot_id, latest_schema_id FROM "events$metadata_log_entries"'

Each row is one metadata.json version under s3://lakehouse/.rustfs-table/…/smoke/tables/events/metadata/: the CREATE, each INSERT, and the ALTER — four rows. The last row is what the catalog points at right now, and its latest_schema_id is 1 where the earlier rows say 0.

And the manifests one level down:

trino 'SELECT path, added_data_files_count, added_rows_count FROM "events$manifests"'

6. Reset

smoke.sql drops and recreates the table so the next lesson starts clean:

docker compose exec spark /opt/spark/bin/spark-sql -f /opt/spark/work-dir/jobs/smoke.sql

Note what DROP TABLE did not do: list the bucket with the curl from the last lesson and the old tables/<uuid>/ objects are still there. The catalog forgot the pointer; the objects need a maintenance job, or a DROP TABLE … PURGE on stores that support it, to actually go away.

What you should see

• One snapshot with a NULL parent after smoke.sql; two after the Spark INSERT, chained by parent_id.
• $files grows from two objects to three; existing paths unchanged.
• FOR VERSION AS OF returns two rows while the live table returns three.
• After ALTER TABLE, Trino shows the new column immediately, NULL for old rows, no rewrite.
• $metadata_log_entries lists one metadata.json per metadata change, all under the hidden .rustfs-table/ prefix.

Common mistakes

• Calling Iceberg a database or an engine. It is a spec and a library. If Spark and Trino disagreed about a table, the bug is in one engine’s Iceberg integration or the catalog, not in “Iceberg the service.”
• Pointing an engine at the bucket without a catalog. The bucket has no idea which metadata.json is current. Every engine in the lab is configured with the REST URI and the S3 endpoint; both are required.
• Expecting DROP TABLE or DELETE to free space. Objects stay until snapshots expire and orphans are removed. Budget for maintenance from day one.
• Writing thousands of tiny files. Every commit adds a manifest; every small file adds planning work. Batch writes, then compact.
• Using the Hadoop catalog on S3 “because it needs no server.” It commits by writing version-hint.text, which is not atomic on object storage. Fine on a laptop file system; not fine on a lake.
• Assuming Trino can write here. This lab sets iceberg.security=read_only on purpose. INSERT from Trino fails; that is the lab, not Iceberg.

Next

Warehouses and modeling is the next node on the roadmap and will build tables on this lab. Until then, re-run the Airflow-orchestrated smoke from Getting Started and read $snapshots on bronze.orders after each run.