Why Docker for Data Engineering

What you will be able to do

• Explain why DE tools ship as images even if you develop on a laptop
• Name the weekly Docker skills this module will teach
• See where Docker sits on the DCubes roadmap

Why this matters for DE and AI

“It works on my machine” is how warehouse jobs fail. Your laptop has Python 3.11 and a local Postgres. The scheduler has neither. A container image is a frozen Linux filesystem plus a default command: the same Postgres, the same library versions, on your laptop and on a server.

You already met the idea in Linux: Why Linux listed Docker as a small Linux world. This module is that world, on purpose.

Where containers show up

• Local databases — postgres or clickhouse on your laptop without a native installer that fights the next project.
• The same stack as production — a Compose file with a warehouse mock, a queue, and a job. New teammates run docker compose up.
• Spark / Airflow / notebooks — many teams start from a published image rather than “install Java, then pip, then hope.”
• CI and schedulers — a job is often “run this image with these env vars and this volume.”
• You, later, on a VM — Linux skills still apply inside the container (ls, tail, df). Docker skills apply around it (ps, logs, exec).

A container is not a virtual machine. It shares the host kernel. It is a process with its own files, network namespace, and (usually) a writable layer that you should not treat as a database.

Skills you will use weekly

Skill	What it is for
Image vs container	The recipe vs one running copy
docker run / ps / logs	Start something and see why it died
docker exec	Get a shell inside the running job
Volumes and bind mounts	CSV in, warehouse files out — data must outlive the container
Ports	localhost:5432 is a publish, not magic
Environment variables	Passwords and DATABASE_URL stay out of the image
Compose	Several services, one file, one up / down
Cleanup	Images fill disks. df still matters

This module teaches those, in order, for someone who can already open a prompt.

What we will not do yet

• We will not deploy to Kubernetes or a cloud registry.
• We will not build Spark or Airflow from scratch.
• We will not run Docker on the DCubes website, or run your containers on our host.
• We will not use curl … | sudo bash to install anything.

Python, SQL, and Git come after Docker on the roadmap. They will assume you can start a container and read its logs.

Map of this module

1. Getting Started — you already did this (engine + ~/dcubes/docker-lab).
2. Images, containers, and Compose as words.
3. Run a container.
4. Lifecycle: start, stop, logs, remove.
5. Look inside with exec and inspect.
6. Images, tags, and registries.
7. Write a Dockerfile and build.
8. Persist data with volumes.
9. Publish ports.
10. Pass environment variables.
11. Compose: one file, one service, then up.
12. Compose a small data stack.
13. Debug and clean up disk.
14. Demo: First Compose shift — Postgres plus a job that loads a CSV.

Practice on your machine

Confirm Docker is still awake and the lab folder exists:

docker version
ls ~/dcubes/docker-lab

You should see Client and Server from docker version. The folder may be empty. If Cannot connect to the Docker daemon, start Docker Desktop (or the engine) before the next lesson.

No other commands in this lesson.

Common mistakes

• Skipping Linux. Paths, cd, and ls are how you find bind-mounted files. Go back if those are shaky.
• Leaving Desktop stopped. The client without an engine cannot run lessons 2–13.
• Waiting for an in-browser terminal. There isn’t one. That is the product.

Next

Images, containers, and Compose — the four words this module will keep using.