Your lakehouse practice environment

What you will be able to do

• Clone the lakehouse lab from GitHub into ~/dcubes/lakehouse-lab
• Start RustFS, Spark, and Trino with Docker Compose
• Run the smoke write in Spark and read the same table in Trino

Why this matters for data work

Warehouses, Spark jobs, and “the lake” are not three unrelated tools. A lakehouse is object storage, a table format that engines share, a writer, a reader, and something that runs the job on a schedule. You will meet those pieces as named containers on your laptop before later modules explain modeling and distributed compute.

This site never starts the stack. Keep Airflow Getting Started in ~/dcubes/airflow-lab as it is — this is a new folder.

Concepts

Object storage — files addressed as buckets and keys, over an S3 API. Here that is RustFS, not AWS.

Iceberg catalog — the list of tables and where their metadata lives. RustFS 1.0 serves that as Iceberg REST at /iceberg on the same port as S3. The warehouse name is the bucket: lakehouse.

Spark writes, Trino reads — Spark creates Iceberg tables and appends rows. Trino is configured read-only. Do not INSERT from Trino in this lab.

Create, then append — RustFS 1.0.0 does not accept Iceberg REST stage-create. Spark CREATE TABLE AS SELECT and writeTo().create() fail. The smoke job creates an empty table, then appends.

Airflow is optional — profile orchestrate adds a second Airflow (UI on 18082) that starts a one-shot Spark container. The Airflow course stays on 18080.

Lab folder

The Compose files live in a separate GitHub repo, not in this website. Clone it (Git from the Git module):

mkdir -p ~/dcubes
git clone https://github.com/amhhaggag/dcubes-lakehouse.git ~/dcubes/lakehouse-lab
cd ~/dcubes/lakehouse-lab
ls

Repo: github.com/amhhaggag/dcubes-lakehouse

You should see compose.yaml, spark/, trino/, bootstrap/, airflow/, and samples/. Do not run Compose from the website repo — that file only serves the learning site.

To pick up lab fixes later:

cd ~/dcubes/lakehouse-lab
git pull

Env file

cd ~/dcubes/lakehouse-lab
cp .env.example .env

cd ~/dcubes/lakehouse-lab
sed -i "s/^AIRFLOW_UID=.*/AIRFLOW_UID=$(id -u)/" .env
sed -i "s/^DOCKER_GID=.*/DOCKER_GID=$(stat -c '%g' /var/run/docker.sock)/" .env

cd ~/dcubes/lakehouse-lab
printf '\nAIRFLOW_UID=50000\nDOCKER_GID=0\n' >> .env

Lab-only login for RustFS and Spark/Trino catalog access:

access key: dcubesadmin
secret key: dcubes-lab-not-for-prod

Never reuse these outside ~/dcubes/lakehouse-lab.

Ports (loopback only)

Start storage, Spark, and Trino

First up builds a Spark image (Iceberg jars) and pulls RustFS and Trino. Wait for it.

cd ~/dcubes/lakehouse-lab
docker compose up -d --build
docker compose ps

Wait until rustfs is healthy and trino is healthy (Trino can take a minute). bootstrap should have exited 0 after creating bucket lakehouse and putting samples/orders.csv at s3://lakehouse/landing/orders.csv.

Smoke: Spark writes, Trino reads

Two-row table smoke.events:

cd ~/dcubes/lakehouse-lab
docker compose exec spark /opt/spark/bin/spark-sql -f /opt/spark/work-dir/jobs/smoke.sql

What you should see: Spark prints 2 as row_count. First run is slow (JVM). Later runs are faster.

Retail CSV as bronze.orders (300 rows):

cd ~/dcubes/lakehouse-lab
docker compose exec spark /opt/spark/bin/spark-submit /opt/spark/work-dir/jobs/bronze_orders.py

Look for a line:

bronze.orders row_count=300

Read the same tables from Trino:

cd ~/dcubes/lakehouse-lab
docker compose exec trino trino --server localhost:8080 --catalog iceberg --output-format csv \
  --execute 'SELECT COUNT(*) FROM smoke.events'
docker compose exec trino trino --server localhost:8080 --catalog iceberg --output-format csv \
  --execute 'SELECT COUNT(*) FROM bronze.orders'

What you should see: "2" then "300". Open Trino in a browser at http://127.0.0.1:18081 if you prefer clicking.

RustFS console: http://127.0.0.1:19101 with the lab access key and secret. You should see bucket lakehouse.

Optional: Airflow orchestrates the same path

Only if RAM still feels comfortable. This does not replace ~/dcubes/airflow-lab.

cd ~/dcubes/lakehouse-lab
docker compose --profile orchestrate up -d --build

Wait until airflow-apiserver is healthy, then open:

http://127.0.0.1:18082

Log in with airflow / airflow. Unpause lakehouse_smoke, trigger it. The first task starts a one-shot Spark container on the Compose network; the second counts bronze.orders in Trino. A green run is enough — DAG design is still the Airflow module.

Stop and start tomorrow

Stop (keeps the RustFS volume):

cd ~/dcubes/lakehouse-lab
docker compose --profile orchestrate down

Start the default stack again:

cd ~/dcubes/lakehouse-lab
docker compose up -d

bootstrap is a one-shot: Compose will run it again if you up --build after a full volume wipe. If the bucket already exists, bootstrap still exits 0.

Common mistakes

• CREATE TABLE AS SELECT or writeTo().create() — RustFS rejects stage-create. Use CREATE TABLE then INSERT / append.
• Port 19100 already in use — another S3-compatible store on the host. Change the left side of the RustFS ports in compose.yaml.
• Trino starting: true — wait; the UI is up before the coordinator is ready.
• Running Compose in the website repo — the site’s compose file is nginx on 127.0.0.1:8088. This lab is only ~/dcubes/lakehouse-lab.
• Airflow on 18080 vs 18082 — 18080 is the Airflow course. 18082 is this lakehouse DAG runner.

Next

You now have a place to land files, commit Iceberg tables, and query them — still guidance only, still on your machine. The next two lessons explain what you just built: Object storage, file systems, and block storage, then Apache Iceberg. Warehouses, Spark, and cloud modules will reuse this lab and are still coming soon on the roadmap.