Input and output redirection is about taking the useful information from files and programs and doing what you want with them.
It will also involve some of the basics of file descriptors (FDs).
I want to save the output of a program
Suppose you and your friend have large music collections and want to share. You want to be able to tell your friend what you have. but your music folder isn't sorted by genre. Instead you just have a huge pile of albums. It would take ages to read them all and see which ones he wants.
Instead, you can make a list of all your albums and send him that.
ls Music/ > my-music-list
The above command runs
ls Music/ and writes the output (a list of
all your files) to the file
> character means 'take the standard output of the program to my
left and write it to the file to the right'
But I have more music in a different folder!
You have two options here, the boring way and the fun way.
The boring way!
ls is able to show the contents of multiple directories at once
ls Music/ other-music/ > all-my-music
The fun way!
Note: levels of fun may vary.
You can append to a file using '>>'.
ls Music/ > my-music-list ls other-music/ >> my-music-list
I want all my music from the fake band Fiddlesticks
There's a handy-dandy program called grep, which shows every line that matches a certain pattern. There's a boring and an interesting way to use grep, as well.
I am a boring person
Okay, grep lets you specify a file to search in.
grep "Fiddlesticks" my-music-list
This will return every line in the 'my-music-list' file that contains the text 'Fiddlesticks'.
I am an interesting person
grep also reads from standard input, so this works just as well
grep "Fiddlesticks" < my-music-list
Here, the '<' character means 'Take everything from the file to my right and put it in the program to my left'
What's this 'standard input' and 'standard output'?
Standard input and standard output are simply streams that every running program has.
If you don't do any special input/output redirection, standard input is everything you type into the terminal with your keyboard (e.g. 'Y' on a yes/no prompt), and standard output is everything (except errors) that comes out of a program, which is typically what it displays in your terminal.
There is one other special stream, called standard error. This is where error messages from a program get written, separate from any ordinary output.
To see the difference, try running:
ls file-that-doesnt-exist > output
The terminal replies with
ls: cannot access file-that-doesnt-exist:
No such file or directory, but the file 'output' is empty.
I have too many songs by Fiddlesticks! I want something specific
The solution to your problem is pipelines. Pipelines connect the standard output of one program to the standard input of another.
We can use this to find the album 'THINGS' by fiddlesticks:
grep "Fiddlesticks" my-music-list | grep "THINGS"
This will show every line that contains the text 'Fiddlesticks' and 'THINGS'
I want to run this long, boring program overnight
Most virtual terminals only scroll back so far. If you come back to find that it spewed a huge log detailing a failure, it can be quite annoying for the exact information on how it failed to have been 1000 lines before the end of the output, where it was thrown away.
You want every line of output to be written to a file, so you can check it later. This can be done by:
run-boring-program &> log-file.txt
In this command,
&> means 'write standard output and standard error of the
program on the left to the file on the right'
If you want to see what the output is at the same time, you can use
tee is a handy little program
that splits output (e.g. to standard output and to a file), resembling
the letter 'T' and referencing the concept of a T-piece in a pipeline.
To write all output to a file, and also display it on the console, you would run:
run-boring-program 2>&1 | tee log-file.txt
The magic rune introduced here is
2>&1, which involves the operator
which means 'take the file descriptor to the left, and write it to the file
descriptor to the right'.
A 'file descriptor' (FD) is a simple representation for a file (or stream). The three streams every running program has (standard input, standard output and standard error) take the file descriptors 0, 1 and 2. If the program opens any more files, they would be assigned other numbers. Since the input, output and error streams are defined to be FDs 0, 1 and 2 in POSIX, we can rely on them having these numbers no matter the platform.
Where can I learn more?
This information came partly from fiddling and experience, and partly from BASH Programming at tldp.org