←	Nov 2024	→
S	M	T	W	T	F	S
					1	2
3	4	5	6	7	8	9
10	11	12	13	14	15	16
17	18	19	20	21	22	23
24	25	26	27	28	29	30

Informational shell utilities

Primitives

echo(1) will print its command line out. Despite its simple behaviour, it is remarkably useful, since it's a way to get your shell to put its variables to its standard output.

$ VAR=foo
$ echo "$VAR"
foo

Given its frequent use and simplicity, the version of echo(1) you use will actually be the version built into your shell.

It can be used to write strings to a file, when used with output redirection.

$ echo foo >bar

cat(1) can be used to retrieve the contents of a file.

$ cat bar
foo

You can put the contents of a file into a variable using subshell expansion.

$ baz=`cat bar`
$ echo $baz
foo
$ qux=$(cat bar)
$ echo $qux
foo

So with echo(1), cat(1), output redirection and subshell expansion, you can move data between variables, command line arguments and files.

cat(1) has one last trick up its sleeve; it can be used to concatenate files, hence its name.

$ echo foo >1
$ echo bar >2
$ cat 1 2
foo
bar

File trimming

head(1) and tail(1), without further arguments, read the first 10 or last 10 lines of a file and write them to standard output.

$ seq 20 >numbers # write the numbers 1 to 20 to the file called numbers
$ head numbers
1
2
3
4
5
6
7
8
9
10
$ tail numbers
11
12
13
14
15
16
17
18
19
20

The number of lines can be tuned with the -n flag. So head -n3 is the first 3 lines, and tail -n3 is the last 3 lines.

$ head -n3 numbers
1
2
3
$ tail -n3 numbers
18
19
20

The number given to the -n option of head(1) can be prefixed with a - to print all but the last n lines, and tail(1) can be prefixed with a + to print all but the first n lines.

$ seq 5 >numbers
$ head -n -1
1
2
3
4
$ tail -n +1
2
3
4
5

The option -c can be used to perform a similar operation, but on the number of bytes rather than the number of lines.

tail(1) has a few more tricks. One of tail's key uses is to display the last lines of a log, which are generally more important to know what just happened to the program that generated it.

However, it's also useful to have the output from the log be printed to an open terminal, to see what it's doing, without having to reconfigure it to log differently, so tail(1) also has the -f flag, which stands for follow. When -f is specified, it will print the last 10 lines, but also print new lines as they are added.

Pagers

less(1) and more(1) are pagers. Pagers are responsible for allowing you to more easily read the output of a program that generates a lot, by keeping it all on one page and letting you say when you've finish reading it and continue to the next page.

less(1) is like more(1), but better. If you're interested, someone also wrote a pager called most(1).

Please note that cat "$file" | less is rarely the right thing to do. Unless $file is special in some way this is equivalent to less <"$file", and it's clearer, in almost all cases, to simply run less "$file".

Environment processing

Both env(1) and printenv(1) can be used to list all the variables. Indeed they behave the same if they are called with no arguments.

However, if you are writing a shell script and need to programmatically get a list of all the environment variables, and maybe their values, then you need the GNU coreutils versions of these programs, and then you can use use the -0 switch to get them to NUL terminate output, rather than newline terminate, allowing you to process them with read -d ''.

After this, env(1) and printenv(1) differ, since printenv(1) will list variables you list on the command line

$ export FOO=bar
$ export BAZ=qux
$ printenv FOO BAZ
bar
qux

However, this is of limited use, since your shell will suport variable interpolation natively.

$ echo $FOO $BAZ
bar qux

env(1) will set variables then run a command.

$ env BADGER=stoat printenv BADGER
stoat

env(1) can be used to run a program in a reduced environment with the -i flag.

$ env -i BADGER=stoat printenv
BADGER=stoat

Apart from all this, env(1) is used on the she-bang line of scripts, to allow looking up the interpreter in $PATH, since various programs may be installed in different places on different systems, but env(1) is usually installed at /usr/bin/env.

Directory contents information

`ls`

ls(1) can be used to list the contents of a directory.

$ ls /
bin    dev   initrd.img      lib64  opt   run   sys  var
boot   etc   initrd.img.old  media  proc  sbin  tmp  vmlinuz
cdrom  home  lib             mnt    root  srv   usr  vmlinuz.old

If it can determine that it is writing to a terminal, it will tabulate the results like above, otherwise it is one entry per line.

$ ls / | cat
bin
boot
cdrom
dev
etc
home
initrd.img
initrd.img.old
lib
lib64
media
mnt
opt
proc
root
run
sbin
srv
sys
tmp
usr
var
vmlinuz
vmlinuz.old

It's common to see shell scripts use ls(1) to perform an action on every entry in a directory.

$ for dir in `ls /`; do
    echo $dir
done

However, this would have problems if any of the directory names had spaces in them, and shell globbing is shorter, and handles spaces, just not any files or directories with names that start with ..

$ for dir in /*; do
    echo $dir
done

You can use ls -a to list files that start with ., though ls -A is generally more useful since it excludes . and ...

To do this safely in scripts is more complicated.

A portable way to do this is:

$ command='for arg; do echo "$arg"; done'
$ find / -mindepth 1 -maxdepth 1 -exec sh -c "$command" - {} +
/home
/etc
/media
/var
/bin
/boot
/dev
...

A less portable way that uses fewer subprocesses is:

$ find / -mindepth 1 -maxdepth 1 -print0 |
while read -d '' arg
do
    echo "$arg"
done
/home
/etc
/media
/var
/bin
/boot
/dev
/lib
/mnt
/opt
...

If your loop needs to change variables outside the body of the loop, you need to do this instead.

$ count=0
$ while read -d '' arg
do
    count=$(( "$count" + 1 ))
done < <(find / -mindepth 1 -maxdepth 1 -print0)
$ echo $count
24

`pwd`

pwd(1) will print the path to the current directory to standard output.

$ pwd
/home/richardmaw

This is not generally useful when using a shell interactively, since your shell's prompt will usually have the current directory in it:

$ PS1='\w\$ '
~$ cd /
/$ pwd
/

However, pwd can be of use in scripting if you need to change directory and wish to return to the current directory later in your shell script.

#!/bin/sh
HERE="$(pwd)"
# Do work including changing directory
# ...
cd "$HERE"
# Do more work back in the original directory
# ...

`readlink`

readlink(1) is a thin wrapper over the readlink(2) system call in its default mode of operation.

$ readlink /vmlinuz
boot/vmlinuz-3.11.0-17-generic

It is generally more common to discover this by running ls -l, since it's fewer characters to type, though readlink(1) is more useful in scripts.

$ ls -l /vmlinuz
lrwxrwxrwx 1 root root 30 Mar  3 08:07 /vmlinuz -> boot/vmlinuz-3.11.0-17-generic

Another useful mode of readlink(1) is readlink -f, which is "canonicalize" mode.

$ readlink -f /vmlinuz
/boot/vmlinuz-3.11.0-17-generic
$ cd ~
$ readlink -f ../../vmlinuz
/boot/vmlinuz-3.11.0-17-generic

Another use for readlink -f is to get an absolute path to a shell script, to access packaged resources.

$ install -m755 /dev/stdin /tmp/thisdir-test.sh <<'EOF'
#!/bin/sh
THISDIR="$(readlink -f "$(dirname "$0")")"
echo "$THISDIR"
EOF
$ /tmp/thisdir-test.sh
/tmp
$ mv /tmp/thisdir-test.sh ~
$ ~/thisdir-test.sh 
/home/richardmaw

`stat`

stat(1) uses the stat(2) system call to report information about files.

It defaults to showing a wide selection of information.

$ stat thisdir-test.sh 
  File: ‘thisdir-test.sh’
  Size: 69          Blocks: 8          IO Block: 4096   regular file
Device: 1ah/26d Inode: 237403      Links: 1
Access: (0755/-rwxr-xr-x)  Uid: ( 1000/richardmaw)   Gid: ( 1000/richardmaw)
Access: 2014-03-30 20:50:10.266400076 +0100
Modify: 2014-03-30 20:49:01.786402099 +0100
Change: 2014-03-30 20:49:42.522400896 +0100
 Birth: -

stat(1) can be told to output specific information with its -c argument, which takes a format string. %n is the file name, %s is the file size.

$ stat -c '%n: %s' thisdir-test.sh 
thisdir-test.sh: 69

User status

whoami(1) prints the user name of the current user.

$ whoami
richardmaw

who(1) and users(1) list who is logged in, in slightly different formats.

$ users
richardmaw richardmaw richardmaw richardmaw
$ who
richardmaw tty1         2014-03-30 15:49
richardmaw tty7         2014-03-30 18:06 (:0)
richardmaw pts/1        2014-03-30 18:13 (:0)
richardmaw pts/2        2014-03-30 20:10 (:0)

id(1) displays more information about the current user, including the primary group, supplementary groups, and all the numeric ids for them, all comma separated

$ id | while read -d ',' FOO; do echo $FOO; done                                                                                                   ~
uid=1000(richardmaw) gid=1000(richardmaw) groups=1000(richardmaw)
4(adm)
20(dialout)
24(cdrom)
25(floppy)
27(sudo)
29(audio)
30(dip)
44(video)
46(plugdev)
104(scanner)
113(netdev)
114(bluetooth)
120(fuse)

id(1) can be given various options to limit its output, such as -Z to also print the SELinux security context.

date

date(1) prints the current date and time to standard output.

$ date
Sun Mar 30 21:36:16 BST 2014

It also takes a format string.

$ date +%F
2014-03-30

It can be told to format a date which isn't now, with the -d option.

$ date -d yesterday
Sat Mar 29 20:37:58 GMT 2014

wc

wc(1) prints line, word and character counts. By default it prints all 3 and the file name.

$ wc thisdir-test.sh 
 3  7 69 thisdir-test.sh

To print only one of those, specify -l, -w and -c respectively.

$ wc -l thisdir-test.sh 
3 thisdir-test.sh

To prevent wc(1) printing the file name, the file has to be provided as the standard input.

$ wc -l <thisdir-test.sh 
3

A common solution to finding out how many files in a directory is ls $dir | wc -l, though this will give incorrect results if the file names have new line characters in them.

The most concise, correct way to do this is:

$ find / -maxdepth 1 -mindepth 1 -print0 | grep -cz .
24