Input and output redirection is about taking the useful information from files and programs and doing what you want with them.

It will also involve some of the basics of file descriptors (FDs).

I want to save the output of a program

Suppose you and your friend have large music collections and want to share. You want to be able to tell your friend what you have. but your music folder isn't sorted by genre. Instead you just have a huge pile of albums. It would take ages to read them all and see which ones he wants.

Instead, you can make a list of all your albums and send him that.

ls Music/ > my-music-list

The above command runs ls Music/ and writes the output (a list of all your files) to the file my-music-list.

The > character means 'take the standard output of the program to my left and write it to the file to the right'

But I have more music in a different folder!

You have two options here, the boring way and the fun way.

The boring way!

Okay, ls is able to show the contents of multiple directories at once

ls Music/ other-music/ > all-my-music

The fun way!

Note: levels of fun may vary.

You can append to a file using '>>'.

ls Music/ > my-music-list
ls other-music/ >> my-music-list

I want all my music from the fake band Fiddlesticks

There's a handy-dandy program called grep, which shows every line that matches a certain pattern. There's a boring and an interesting way to use grep, as well.

I am a boring person

Okay, grep lets you specify a file to search in.

grep "Fiddlesticks" my-music-list

This will return every line in the 'my-music-list' file that contains the text 'Fiddlesticks'.

I am an interesting person

grep also reads from standard input, so this works just as well

grep "Fiddlesticks" < my-music-list

Here, the '<' character means 'Take everything from the file to my right and put it in the program to my left'

What's this 'standard input' and 'standard output'?

Standard input and standard output are simply streams that every running program has.

If you don't do any special input/output redirection, standard input is everything you type into the terminal with your keyboard (e.g. 'Y' on a yes/no prompt), and standard output is everything (except errors) that comes out of a program, which is typically what it displays in your terminal.

There is one other special stream, called standard error. This is where error messages from a program get written, separate from any ordinary output.

To see the difference, try running:

ls file-that-doesnt-exist > output

The terminal replies with ls: cannot access file-that-doesnt-exist: No such file or directory, but the file 'output' is empty.

I have too many songs by Fiddlesticks! I want something specific

The solution to your problem is pipelines. Pipelines connect the standard output of one program to the standard input of another.

We can use this to find the album 'THINGS' by fiddlesticks:

grep "Fiddlesticks" my-music-list | grep "THINGS"

This will show every line that contains the text 'Fiddlesticks' and 'THINGS'

I want to run this long, boring program overnight

Most virtual terminals only scroll back so far. If you come back to find that it spewed a huge log detailing a failure, it can be quite annoying for the exact information on how it failed to have been 1000 lines before the end of the output, where it was thrown away.

You want every line of output to be written to a file, so you can check it later. This can be done by:

run-boring-program &> log-file.txt

In this command, &> means 'write standard output and standard error of the program on the left to the file on the right'

If you want to see what the output is at the same time, you can use tee. tee is a handy little program that splits output (e.g. to standard output and to a file), resembling the letter 'T' and referencing the concept of a T-piece in a pipeline.

To write all output to a file, and also display it on the console, you would run:

run-boring-program 2>&1 | tee log-file.txt

The magic rune introduced here is 2>&1, which involves the operator >&, which means 'take the file descriptor to the left, and write it to the file descriptor to the right'.

A 'file descriptor' (FD) is a simple representation for a file (or stream). The three streams every running program has (standard input, standard output and standard error) take the file descriptors 0, 1 and 2. If the program opens any more files, they would be assigned other numbers. Since the input, output and error streams are defined to be FDs 0, 1 and 2 in POSIX, we can rely on them having these numbers no matter the platform.

Where can I learn more?

This information came partly from fiddling and experience, and partly from BASH Programming at tldp.org