Pipes and redirects

What you will be able to do

• Redirect command output into a file with > and >>
• Pipe one command into another with |
• Explain stdout vs stderr at a beginner level

Why this matters for DE and AI

A data pipeline is “the output of this step is the input of the next.” Unix did that with text decades ago. | (a pipe) connects programs. > saves output to a file. When you later write Airflow or Spark, the picture is the same: extract → transform → load. This lesson is that picture at one-line scale.

Concepts

Every process has three streams:

Stream	Number	Default
stdin (input)	0	Your keyboard
stdout (output)	1	Your terminal
stderr (errors)	2	Your terminal

Redirects

• command > file — write stdout to file (overwrite)
• command >> file — append stdout
• command 2> file — write stderr to file
• command < file — read stdin from file (less common in this module)

Pipes

command1 | command2

stdout of command1 becomes stdin of command2. You can chain more.

Example you will use on data files:

grep ERROR logs/job.log | wc -l

“How many ERROR lines?” — a filter, then a count.

Practice on your machine

cd ~/dcubes/linux-lab
echo "first line" > data/notes.txt
echo "second line" >> data/notes.txt
cat data/notes.txt

What you should see: two lines. If you had used > the second time, you would only have second line.

Save ERROR lines to their own file:

grep ERROR logs/job.log > logs/errors.txt
cat logs/errors.txt

Count them without opening the file:

grep ERROR logs/job.log | wc -l

What you should see: 2 (or however many ERROR lines you have).

List lab files and keep only names containing log:

ls | grep log

Send a command’s error somewhere else so it does not clutter the screen. This tries to list a folder that does not exist:

ls data/missing 2> logs/ls-error.txt
cat logs/ls-error.txt

What you should see: an error message in logs/ls-error.txt, and little or nothing on the screen from ls.

Common mistakes

• > when you meant >>. Overwrites. Check with cat after you write.
• Piping to grep with no pattern. grep will wait. Always give a pattern.
• Thinking stderr goes through |. By default pipes pass stdout only. Errors still print. That is why 2> exists.

Next

Peek at data from the shell — a real (tiny) CSV, cut, sort, and uniq.