What is a job.

Abstraction. A pipeline is made of many processes, they are all in the same job.

I'm going to provide a brief overview, for more technical details see the "JOB CONTROL" section of the bash(1) manpage.

Starting a job in the background.

Similarly to how ending a command with && and || changes the logic flow for the next command, ending a command with a single & causes the command to be run "in the background" i.e. while the command is running, you can still enter other commands.

Identifying a running job

Immediately after running a background job, the $! variable contains the process ID of the last command to be run. This is different from the job ID, but it is still rather useful, provided you are only running one command at a time.

Additionally, bash will assign each job an ID, starting with the % symbol, so the first background job is %1. % %% and %+ all mean the current job (i.e. most recent), and %- refers to the previous job (second most recent).

The jobs command will list all active jobs, listing their job ID, the command they are running, and markers to say which is the current and which is the previous job.

Suspending a job.

Pressing ^Z (holding Control and pressing Z) is the usual shortcut key. This will return you to your shell prompt, and the job you were running is suspended (i.e. processing stops, it will not print any more output etc.)

Moving a job to the background.

The bg command can be used to resume an existing job, while leaving you access to the command line. Without any further options, it will resume the latest command to be run. bg can also be given a job spec as described above. bg, bg %, bg %% and bg %1 are all equivalent, as are bg - and bg %-.

Moving a job to the foreground.

fg will move a job to the foreground, so it would be identical to as if it had been run without the & specifier or suspended with ^Z.

fg has all the short-cuts that bg does, plus the job specifiers themselves are synonymous to passing the job specifier to fg, so % is equivalent to fg %.

The background specifier & can also be used with the fg command, which makes it equivalent to bg, so fg %1 & is equivalent to bg %1.

The logical consequence of this is that % & is equivalent to bg %+.

Waiting for a job to finish.

The wait command can be used to synchronise execution by waiting until an existing command finishes before returning to the command-line, or starting another job.

This differs from using the fg command, as wait may also be given a process ID, as retrieved from $!.

This is useful for when you have a long-running command, but you forgot you need to run another command afterwards. To handle this, you can suspend with ^Z, then run wait %+ && some_other_command.

Terminating a job.

There are a few ways to do this. You can foreground a job with fg then kill it with ^C. Alternatively, you could kill the job with the kill command.

This is not the same as the kill(1) program. Unless you invoke /bin/kill or "$(which kill)", you will get the kill built-in.

The kill built-in has the advantage of being able to be given a job spec, such as %1, instead of just a process ID.

Severing all ties with a job

Usually when a shell exits, it ensures all its child processes are terminated. This is a useful default behaviour, as it stops there being untidy processes being left around.

However, this is not always wanted, as you may care more about a job completing than being around to observe it.

Traditionally, this is solved by using the nohup(1) command, or running the process in a detached [screen(1)][] session. However, both these approaches require you to know in advance that this job is going to take longer than you have your terminal open.

To solve this, bash has a disown built-in command, which can remove running commands from job-control, and optionally prevent them being terminated by the shell.

$ some command </dev/null >~/nohup.out &
$ disown -h %+

Is functionally equivalent to running nohup some command. Running disown without -h won't prevent the process being killed when your shell exits.

Your shell will send the SIGHUP signal to its child processes when it exits, hence the naming of the nohup(1) command, and the -h option to disown.

Further reading

lwn is doing a series about process grouping. Later articles go into details about more modern Linux features, but the first article is good for a history of process management, and some technical discussions of how Job control works.