Previously we lamented that we had to read the cmdline for every process to work out whether the program we want to run is already running, and that we'd like to be able to look up a name to see if it's running.

The traditional way to do this is to write the process identifier (or PID) to a file called named after your program, such as /var/run/$PROGNAME.pid.

This is an obvious solution, that you will probably have seen before, and before systemd and upstart became popular, was the way you handled running services in Linux.

How people do pid files wrong

On the surface just writing the PID looks like a good idea, since you can use the presence of the file to tell that the process is running or read the contents of the file to see which PID it's using so you can kill(2) the process.

This however is less reliable than parsing files in procfs, since if you're reading from /proc you know it's always up to date since it's provided by the kernel's process table.

There are no such guarantees from the PID file. Your process could terminate abnormally and not clean it up.

Be extra cautious with PID files not stored in /run or /var/run, since there is no guarantee that they weren't left over from a previous boot.

Reliability can be improved by checking whether the PID from the file is in use, by running kill(pid, 0) == 0 in C, or kill-0 $pid in shell, since this will evalutate to true if that process is running.

This is not yet reliable though, since while killing with signal 0 will tell you whether a process with that PID is running, you can't tell if it's the correct process, since PIDs get reused fairly frequently.

For PID files to be reliable you need some way to guarantee that the PID in the file is actually referring to the correct process.

You need some way of tying the lifetime of the process to the file.

When a process terminates all the file descriptors it had open are closed, so we can know that the contents are valid if a process is holding it open.

You may be tempted to use lsof-Fp -f -- /var/run/$PROGNAME.pid, or parsing procfs manually to determine whether a process is using the file, but this is awkward for the same reason as not parsing the output of the ps(1) command to tell whether it's running.

We need to be able to do something to the file descriptor that will have an observable effect through other file descriptors to that flie.

The solution to this is to take a lock on the file.

You may see some programs use fcntl(2) with F_SETLK. This is tempting because F_GETLK will include the PID in the result instead of requiring the service to serialise and write the PID to the flie and the checking process having to read and parse the file.

F_SETLK should be avoided because the lock is removed when any file descriptor to that file owned by that process is closed, so you need to be able to guarantee that neither any of your code not any other code you use via a library will ever open that PID file again even if your process normally opens arbitrary files by user input.

So rather than using fcntl(2) with F_SETLK, use flock(2) with LOCK_EX, or fcntl(2) with F_WRLK to take a write or exclusive lock on the file, and test whether the contents are live by trying to take a read or shared lock, with flock(2)'s LOCK_SH, or fcntl(2)'s F_RDLK.

If you succeed at taking a read lock then the contents of the file aren't valid and the service isn't running.

Implementing this logic can be awkward, fortunately if you're writing a C program you can use the libbsd-pidfile functions.

It's the wrong tool for the job

Setting it up correctly is tricky

The libbsd-pidfile functions will handle locks correctly, but as permissively licensed as it is, it is not always available, perticularly if you're not writing a C program.

A brief survey of the top results for python libraries for handling PID files resulted in pid, python-pidfile and pidfile.

python-pidfile does not keep the file open or take a lock. pidfile only improves by taking the lock and holding it. pid checks the validity and sets correct permissions on the file. None of them have a mechanism to safely retrieve the PID from the file.

So if you have to do it yourself, you'll need to do the following:

1. open with O_CREAT to make the lock file if it doesn't exist.
2. Try to non-blocking take a shared lock on the file. If you fail it's running, and depending on whether you want to end it, replace it or leave it, you should either instruct it to terminate and continue, exit, or terminate then exit. If you succeed then it's not running.
3. If you want to start a new service or replace it, upgrade your shared lock to an exclusive lock with a timeout. If you take a blocking lock then your processes will deadlock waiting for their turn to start the service, and if you take a non-blocking lock then if your process is pre-empted by another process between trying to lock and releasing the shared lock, then you could end up with every process exiting.
4. If you took the lock replace the contents of the file with your PID, if you timed out unlock the file or exit.

The above is already complicated and still doesn't handle edge-cases, such as another process trying to start between taking the lock and writing the PID, which libbsd-pidfile handles with a retry loop.

It also doesn't handle the file being unlinked while starting, which would cause you to have multiple services running.

libbsd-pidfile doesn't take the shared lock then upgrade, so if many processes often want to know whether it's running, then they would be unnecessarily contesting the lock.

Views of PIDs are not universal

Process namespaces exist.

While a single process' view of its PID does not change, other processes can see it with different PIDs since processes in sub-namespaces are viewable in multiple namespaces and have a different PID in each namespace.

You can detect whether the process is running by whether locking fails, but you can't use the contents to terminate it, since the PID it knows it to be is different from the PID you can reach it with, unless you use a restricted system call to enter the namespace of the process in order to terminate it.

fcntl(2)'s F_GETLK would return the PID correctly, but is unreliable in other areas.

Services can span more than one process

The PIDfile model works acceptably if your service only runs in one process, but some programs spawn helper processes and terminating the lead process may not tidy up all the helper processes.

There are ways to have the termination of one process cause the termination of others, however this just moves the problem of "how do you know your process is running" to the supervising process without solving the general problem.

So, we know that PID files aren't necessarily the right tool for the job, in the next article we explore some alternatives.