Streams, Redirection, and Pipes#

Concepts#

The Three Standard Streams#

Every program in Linux has three data channels connected by default:

Stream	Name	File Descriptor	Default Destination	Purpose
stdin	Standard Input	`0`	Keyboard	Where a program reads its input
stdout	Standard Output	`1`	Terminal screen	Where a program writes its normal output
stderr	Standard Error	`2`	Terminal screen	Where a program writes error messages

A file descriptor is a number the kernel uses to track open streams. You will use these numbers (0, 1, 2) in redirection.

By default, both stdout and stderr appear on your screen, which makes them look the same. But they are separate channels — and you can redirect them independently.

               ┌─────────┐
  stdin (0) ──→│         │──→ stdout (1) ──→ screen
  (keyboard)   │ program │
               │         │──→ stderr (2) ──→ screen
               └─────────┘

Output Redirection#

Redirect stdout to a file#

# > creates the file (or overwrites it if it exists)
echo "hello" > output.txt
ls /etc > filelist.txt

# >> appends to the file (does not overwrite)
echo "another line" >> output.txt
date >> output.txt

Redirect stderr to a file#

# 2> redirects only error messages
ls /nonexistent 2> errors.txt
# The error goes to errors.txt, not the screen

# 2>> appends errors to a file
ls /also_missing 2>> errors.txt

Redirect both stdout and stderr#

# &> redirects both stdout and stderr to the same file
command &> all_output.txt

# Equivalent older syntax:
command > all_output.txt 2>&1
# "Send stdout to file, then redirect stderr (2) to wherever stdout (&1) is going"

# Append both:
command >> all_output.txt 2>&1

Redirect stderr to stdout (merge them)#

command 2>&1
# Now stderr goes to the same place as stdout
# Useful when piping — pipes only pass stdout by default

Discard output#

# Discard stdout (silent operation)
command > /dev/null

# Discard stderr (suppress errors)
command 2> /dev/null

# Discard everything
command &> /dev/null

Input Redirection#

# < reads input from a file instead of the keyboard
sort < unsorted.txt
wc -l < /etc/passwd

# << is a "here document" — inline multi-line input
cat << EOF
This is line one.
This is line two.
Variable: $HOME
EOF

# <<< is a "here string" — single-line input
grep "hello" <<< "hello world"

Pipes#

A pipe (|) connects the stdout of one command to the stdin of another. This is one of the most powerful concepts in Linux.

command1 | command2 | command3

stdout of command1 flows into stdin of command2, whose stdout flows into stdin of command3.

# List files and count them
ls /etc | wc -l

# Find lines containing "bash" in /etc/passwd
cat /etc/passwd | grep bash

# Sort a file and show unique lines
sort data.txt | uniq

# Chain multiple steps
cat /var/log/syslog | grep "error" | sort | uniq -c | sort -rn | head -10
# Read syslog → keep lines with "error" → sort → count unique lines → sort by count → top 10

Pipes Only Carry stdout#

By default, stderr is NOT piped. It still goes to the screen:

# stderr from command1 is NOT passed to command2
ls /nonexistent 2>&1 | grep "No such"
# 2>&1 merges stderr into stdout BEFORE the pipe, so grep can see the error

`tee` — Write to a File AND the Screen#

tee reads from stdin and writes to both stdout (the screen) and one or more files:

# Save output to a file while also displaying it
ls /etc | tee filelist.txt

# Append to a file
ls /etc | tee -a filelist.txt

# Write to multiple files
ls /etc | tee file1.txt file2.txt

# Useful with sudo (common pattern):
echo "new line" | sudo tee -a /etc/somefile
# (sudo echo ... > /etc/somefile does NOT work because > is processed by your shell, not sudo)

Command Substitution#

Capture a command’s output and use it as part of another command:

# $(command) — modern syntax
echo "Today is $(date)"
echo "There are $(ls /etc | wc -l) files in /etc"
echo "Kernel: $(uname -r)"

# `command` — older syntax (same effect, but harder to nest)
echo "Today is `date`"

Process Substitution#

Feed a command’s output as if it were a file:

# <(command) — treat output as a file
diff <(ls /etc) <(ls /usr)
# Compares the file listings of /etc and /usr

# Useful when a command expects a filename but you want to give it a stream

Lab#

Exercise 1: Output Redirection#

mkdir -p ~/lab/io
cd ~/lab/io

# Redirect stdout to a file
echo "First line" > output.txt
cat output.txt

# Overwrite the file
echo "Second line" > output.txt
cat output.txt
# Only shows "Second line" — first line was overwritten

# Append
echo "Third line" >> output.txt
cat output.txt
# Shows both "Second line" and "Third line"

# Redirect a command's output
ls -la /etc > etc_listing.txt
head -5 etc_listing.txt

Exercise 2: Error Redirection#

cd ~/lab/io

# Generate an error
ls /nonexistent_directory
# You see the error on screen

# Redirect the error to a file
ls /nonexistent_directory 2> errors.txt
# Screen is quiet — error went to the file
cat errors.txt

# Redirect stdout to one file, stderr to another
ls /etc /nonexistent > success.txt 2> fail.txt
cat success.txt    # files from /etc
cat fail.txt       # error about /nonexistent

# Discard errors
ls /etc /nonexistent 2> /dev/null
# Only stdout appears — errors are silenced

Exercise 3: Pipes#

# Count files in /etc
ls /etc | wc -l

# Find your user in /etc/passwd
cat /etc/passwd | grep $(whoami)

# List the 5 largest files in /var/log
sudo du -sh /var/log/* 2>/dev/null | sort -rh | head -5

# Show only usernames from /etc/passwd
cat /etc/passwd | cut -d: -f1 | sort

# Chain: find all installed packages with "lib" in the name
dpkg -l | grep "^ii" | awk '{print $2}' | grep lib | head -10

Exercise 4: tee#

cd ~/lab/io

# Display and save at the same time
ls /etc | tee etc_files.txt
# Output appears on screen AND is saved to the file

# Verify the file
wc -l etc_files.txt

# Append mode
date | tee -a etc_files.txt
tail -1 etc_files.txt

# Using tee with sudo (a very common pattern)
echo "# This is a comment" | sudo tee -a /tmp/testfile.txt
cat /tmp/testfile.txt
rm -f /tmp/testfile.txt

Exercise 5: Input Redirection#

cd ~/lab/io

# Create a file with some numbers
echo -e "3\n1\n4\n1\n5\n9\n2\n6" > numbers.txt

# Use input redirection
sort < numbers.txt
wc -l < numbers.txt

# Here document
cat << END
Hello, $(whoami)!
Today is $(date).
Your home directory is $HOME.
END

# Here string
grep "world" <<< "hello world, hello universe"

Exercise 6: Command Substitution#

# Use command output inside another command
echo "Running kernel: $(uname -r)"
echo "Hostname: $(hostname)"
echo "Number of users: $(wc -l < /etc/passwd)"

# Create a timestamped backup filename
BACKUP="backup_$(date +%Y%m%d_%H%M%S).tar"
echo "Backup file would be: $BACKUP"

# Use in a practical context
echo "Disk usage: $(df -h / | tail -1 | awk '{print $5}') of root filesystem used"

Exercise 7: Combine Everything#

# A realistic pipeline: find the 5 most common shells used on the system
cat /etc/passwd | cut -d: -f7 | sort | uniq -c | sort -rn | head -5

# Save the result and display it
cat /etc/passwd | cut -d: -f7 | sort | uniq -c | sort -rn | head -5 | tee ~/lab/io/shell_stats.txt

# Clean up
cd ~
rm -rf ~/lab/io

Review#

1. What are the three standard streams and their file descriptor numbers?

stdin (0) — standard input, default is the keyboard
stdout (1) — standard output, default is the screen
stderr (2) — standard error, default is the screen

2. What is the difference between `>` and `>>`?

> creates the file or overwrites it if it exists. >> appends to the file without erasing existing content.

3. How do you redirect stderr to a file?

command 2> file.txt. Use 2>> to append. Use 2>&1 to merge stderr into stdout.

4. What does `command &> /dev/null` do?

It discards both stdout and stderr. All output from the command is silenced — nothing appears on screen and nothing is saved.

5. What does a pipe (`|`) do?

It connects the stdout of the left command to the stdin of the right command. This lets you chain commands together, with each command processing the output of the previous one.

6. Why is `echo "text" | sudo tee -a /etc/file` used instead of `sudo echo "text" >> /etc/file`?

Because >> is processed by your current shell (not by sudo). Your shell does not have permission to write to /etc/file. With tee, the sudo applies to the tee program, which is what actually writes to the file.

7. What is command substitution and what syntax does it use?

Command substitution captures a command’s output and inserts it into another command. Syntax: $(command). Example: echo "Today is $(date)". The older syntax uses backticks: `command`.

Previous: vim | Next: grep

Streams, Redirection, and Pipes