Read files like a data engineer

What you will be able to do

• Page through a file with less and leave with q
• Read the start and end of a file without opening the whole thing
• Count lines with wc -l, the same move you will use on CSVs

Why this matters for DE and AI

Datasets and logs are often too big to open in an editor. You peek: first lines (schema), last lines (did the job finish?), a running tail (is it still alive?). Training runs and Spark jobs are the same habit: do not load the whole file.

Concepts

Command	Use
cat FILE	Print the whole file. Fine for tiny files.
less FILE	Page through. Arrow keys scroll. q quits.
head FILE	First 10 lines
head -n 5 FILE	First 5 lines
tail FILE	Last 10 lines
tail -n 20 FILE	Last 20 lines
tail -f FILE	Follow as new lines appear. Ctrl+C stops.
wc -l FILE	Count lines (for a CSV, roughly row count plus header)

cat is named for “concatenate.” People also use it to dump a file to the screen. If the dump is huge, use less or head.

tail -f is how you watch a job: open the log, wait, see new ERROR or INFO lines appear.

Practice on your machine

cd ~/dcubes/linux-lab
mkdir -p logs

Create a small fake job log. Type the whole block (the last line is EOF by itself):

cat > logs/job.log << 'EOF'
2026-03-01 09:00:01 INFO  Starting extract job movies_daily
2026-03-01 09:00:02 INFO  Reading source s3://example/movies.csv
2026-03-01 09:00:03 INFO  Row count so far 10
2026-03-01 09:00:04 WARN  Slow response from source
2026-03-01 09:00:05 INFO  Row count so far 20
2026-03-01 09:00:06 ERROR Failed to write /data/movies/out.parquet: Permission denied
2026-03-01 09:00:07 INFO  Retry 1
2026-03-01 09:00:08 ERROR Failed to write /data/movies/out.parquet: Permission denied
2026-03-01 09:00:09 INFO  Giving up
2026-03-01 09:00:10 INFO  Job finished with status FAILED
EOF

That cat > file << 'EOF' pattern is a here-document. Everything until a line that is exactly EOF is written to the file. You will see it in real scripts.

Print the whole log:

cat logs/job.log

Page through it:

less logs/job.log

Press q to quit. If you forget q you will feel stuck — now you know.

head -n 3 logs/job.log
tail -n 3 logs/job.log
wc -l logs/job.log

What you should see: first three INFO lines, last three lines including FAILED, and 10 logs/job.log.

Follow the file (it will not change until you add a line):

tail -f logs/job.log

Open a second terminal window, run:

echo "2026-03-01 09:00:11 INFO  Operator cancelled" >> ~/dcubes/linux-lab/logs/job.log

>> appends. The first window should show the new line. Press Ctrl+C in the tail -f window.

If you only have one window, skip the follow exercise and append with >> then tail as usual.

Common mistakes

• Stuck in less. Press q. Not Ctrl+C (that sometimes works, q always should).
• cat on a huge file. Your terminal will flood. Ctrl+C, then head.
• > instead of >>. One arrow overwrites. Two arrows append. Overwriting a log you meant to keep is a rite of passage — use >> for logs.

Next

Permission denied — why that ERROR line about /data is so common.