First data shift

Do this after lessons 1–9. Lessons 10–13 help but are optional here.

You are pretending to be on call for a tiny extract job. The website still runs nothing. You run every command in ~/dcubes/linux-lab.

What you need

• A working terminal from Getting Started
• Sample files (right-click, save, then copy into the lab):
  • movies.csv
  • job.log

Copy them into place (adjust the Downloads path for your OS):

mkdir -p ~/dcubes/linux-lab/{data,logs,scripts}
cp ~/Downloads/movies.csv ~/dcubes/linux-lab/data/
cp ~/Downloads/job.log ~/dcubes/linux-lab/logs/

WSL users: Downloads is often /mnt/c/Users/YOURWINDOWSNAME/Downloads/.

cd ~/dcubes/linux-lab
ls data logs

You should see movies.csv and job.log.

Shift checklist

Work in order. Write answers in a file as you go:

touch logs/shift-notes.txt

1. Confirm the workspace

pwd
ls -la

You should be in linux-lab with data, logs, and scripts.

2. How many movie rows?

head -n 5 data/movies.csv
wc -l data/movies.csv

Header plus 20 movies is 21 lines. Number of movies = 20.

Append your answer:

echo "movie rows: 20" >> logs/shift-notes.txt

3. Which genres appear, and how often?

cut -d, -f3 data/movies.csv | tail -n +2 | sort | uniq -c | sort -nr

You should see Drama and Sci-Fi at the top of the sample. Save that report:

cut -d, -f3 data/movies.csv | tail -n +2 | sort | uniq -c | sort -nr > logs/genres.txt
cat logs/genres.txt

4. What went wrong in the job?

grep -n ERROR logs/job.log

Two lines about Permission denied writing /data/movies/out.parquet. That is the same class of failure as Permission denied.

Save the errors:

grep ERROR logs/job.log > logs/errors.txt
cat logs/errors.txt

5. Did the job claim to finish?

tail -n 3 logs/job.log

You should see a fallback write, then Job finished with status FAILED. Failed jobs can still write partial files. Always read the last lines.

6. How much space is the lab using?

du -sh ~/dcubes/linux-lab
df -h .

The lab should be tiny. df is the habit, not the number.

7. Optional: a tiny executable note

cat > scripts/shift-summary.sh << 'EOF'
#!/bin/bash
echo "lab: $(pwd)"
echo "movies lines: $(wc -l < data/movies.csv)"
echo "error lines: $(grep -c ERROR logs/job.log)"
EOF
chmod +x scripts/shift-summary.sh
./scripts/shift-summary.sh

What you should see: your path, 21 (or your line count), and 2 error lines.

You passed if

• data/movies.csv and logs/job.log exist in the lab folder
• You counted 20 movies (21 lines minus header)
• logs/errors.txt contains two Permission denied lines
• logs/genres.txt has counts per genre
• You ran du or df at least once

If something does not match, check pwd, then ls data and ls logs. Paths are the usual bug.

After this module

You can open a prompt, move around, inspect files, search logs, and compose a one-line pipeline. That is the weekly Linux skill for Data Engineering and AI.

Next on the roadmap: Docker and Compose, then Python, SQL, and Git — still guidance only, still on your machine.