Why Linux for Data Engineering and AI

What you will be able to do

• Explain why DE and AI jobs run on Linux even if you develop on Windows or a Mac
• Name the weekly terminal skills this module will teach
• See where Linux sits on the DCubes roadmap

Why this matters for DE and AI

You can write Python on a Windows laptop. Training a model for twelve hours, running Spark on a terabyte, or debugging Airflow at 2 a.m. almost never happens on that laptop’s desktop. It happens on a Linux server you reach through a terminal.

If the terminal is a mystery, every later tool looks like magic — until it breaks.

Where the work runs

A short tour of machines you will meet:

• Your laptop, after Getting Started: a Linux-like prompt (WSL, macOS, or Linux). This is for learning and small scripts.
• A cloud VM: a rented Linux computer in AWS, GCP, or Azure. You SSH into it. Data jobs and notebooks often live here.
• A Spark worker or Kubernetes pod: still Linux. You may never log in, but the logs, disks, and permissions are the same ideas.
• A GPU box: Linux, plus extra commands later (nvidia-smi). Disk fills up with model caches. Memory runs out. You will debug that with skills from this module.
• Docker containers: a small Linux filesystem with your app inside. Same commands, smaller world.

“I have Python on Windows” is a good start. It is not the production environment.

Skills you will use weekly

Not sysadmin trivia. These show up in real DE and AI work:

Skill	What it is for
Moving around files	Data lands in directories. You must find it.
Reading logs (tail, grep)	Jobs fail. The answer is in a log file.
Permissions	Spark cannot write /data. Docker cannot read a mount.
Pipes and redirects	Glue small commands. The same idea as a pipeline.
Disk and memory	“No space left on device.” Out-of-memory during training.
Environment variables	API keys, PATH, later CUDA_VISIBLE_DEVICES.
SSH	You will log into machines that are not in front of you.

This module teaches those, in order, for someone who has never used a terminal.

What we will not do yet

• We will not install Spark, Airflow, or PyTorch here.
• We will not write long bash programs. A few-line script is enough.
• We will not dual-boot or buy a server.
• We will not run anything on the DCubes website.

Docker and Compose are next on the roadmap, then Python, SQL, and Git. They assume you can open a prompt and not panic.

Map of this module

1. Getting Started — you already did this (a prompt and ~/dcubes/linux-lab).
2. The terminal itself.
3. Files and folders — treat the filesystem like a tiny data lake.
4. Creating and organizing files.
5. Reading files and logs.
6. Permissions.
7. Finding files and searching text.
8. Pipes and redirects (mini pipelines).
9. Peeking at CSVs from the shell.
10. Environment variables.
11. Disk, memory, processes.
12. Installing software safely.
13. SSH as a mental model.
14. Demo: First data shift — one local lab that uses the pieces together.

Practice on your machine

Confirm the lab folder is still there:

ls ~/dcubes/linux-lab

You should see an empty listing, or the names of folders if you already peeked ahead. If you get No such file or directory, go back to Getting Started and recreate it.

No other commands in this lesson.

Common mistakes

• Skipping setup because “I have a GUI.” Later lessons assume a prompt in ~/dcubes/linux-lab.
• Waiting for an in-browser terminal. There isn’t one. That is the product.

Next

Meet the terminal — commands, flags, and how to cancel something that is taking too long.