We previously spoke about atomic file clobbering with open(2) and renameat2(2).

This can be used to perform an atomic file creation too.

At a really high level the idea is:

1. Create a temporary file.
2. Complete setting it up.
3. Rename the temporary file into place.

In more detail:

1. Decide whether creating a new file or replacing an existing one.

   This determines which flags renameat2(2) gets later. You can stat(2) the destination path before proceeding which allows you to detect whether the target was added or removed while you were building the file.

2. Pick a temporary file path.

   This needs to be on the same file system as the destination, since we are going to use renameat2(2) later.

   Ideally this would be in a directory that is cleaned up automatically so that if your program or computer crashes you won't have the temporary file left around.

   It's rare to be able to do this, since that pretty much just leaves a creating files on a tmpfs when your /tmp is a mounted tmpfs, or creating files on your root partition when /tmp is not a mounted tmpfs.

   You could find the root directory of the mount point the destination is in and put it in a temporary directory in there, but this would still require system integration to have it automatically cleaned up on mount.

   With either /tmp or a custom tempdir on the mount point you also have the problem that when you later use renameat2(2) it can fail from another process mounting a directory on top of one of the directories in the destination path.

   So even though it can leave a temporary file behind creating your temporary file in the same directory as your target is probably the best approach when atomically creating a file with a temporary file.

   linkat(2) and O_TMPFILE provide a better solution for this, but that's a topic for another article, and only works for regular files.

3. Pick a random file path in your temporary directory.

   This is what mkstemp(3) does under the hood.

   Since we intend to check whether the target file already exists we could safely use [tmpnam(3)][] or [mktemp(3)][] to make it (though not [tempnam(3)][] since it prefers the TMPDIR environment variable for creating the temp file in even if a directory is passed), however in their zeal to stop people writing insecure programs the POSIX standard has either removed or marked these functions obsolete, so you probably want your own implementation.

4. Create your new file in a way that will fail if it already exists.

   For regular files this requires passing O_CREAT|O_EXCL to the flags.

   The system calls for creating fifos, device nodes, symlinks, directories and unix sockets already default to failing if the target already exists.

   open(2), mkfifo(3), mknod(2), symlink(2) and mkdir(2) return an errno of EEXIST.

   bind(2) (for unix sockets) returns EADDRINUSE.

   open(2), mkfifo(3), mknod(2), symlink(2) and mkdir(2) have variant syscalls with the "at" suffix which take a file descriptor for the directory to create them in.

   Using these or changing directory into the destination provides resilience against the filesystem mounts changing mid-operation.

   When using bind(2) with a unix socket you probably want to chdir(2) anyway, since the maximum length path is shorter than PATH_MAX.

   If creation fails, go back to step 4.

5. Complete initialisation of the file.

   What this involves depends on the type of file and your application.

   Regular files will want their contents written and metadata set.

   Everything else probably just wants a few metadata tweaks.

6. Rename the file into place.

   This is the renameat2(2) trick mentioned in an earlier article.

   1. If you explicitly want to create the file pass RENAME_NOREPLACE.
   2. If you explicitly want to replace a file pass RENAME_EXCHANGE and [unlink(2)][] the temporary file,
   3. If you're not sure stat(2) the destination before creating and pass RENAME_NOREPLACE if it didn't exist beforehand, and RENAME_EXCHANGE as above if it did, so if you get a failure you can warn, ask or abort.
   4. If you're really sure you don't care if it either creates a new file or replaces an old one then don't pass any flags.

   Few of the errors that may happen are recoverable from, (though EEXIST when you don't care about clobbering can be handled by removing the destination while linking fails) but checking the return code can help you provide useful error messages.

   • ENOENT when using RENAME_EXCHANGE means the destination file doesn't exist when it should.
   • EEXIST when using RENAME_NOREPLACE means the destination file exists when it should not.
   • EROFS or EXDEV mean some process changed the file system mounts while the operation was in-progress. EROFS being making it read-only, and EXDEV being some mount on top of your directory.

   If you're particularly paranoid you can use stat(2) to check whether the destination file is still reachable from the root directory by checking whether the st_dev and st_ino match.

And there you have it. Atomic file updates.

What would this be useful for

This is useful if you have many processes that might make use of a file and you need to update it to a new version.

This would be useful for shared caches (such as /etc/ld.so.cache) which can't use a lock file to synchronise updates either for performance or legacy client reasons.

It can also be used to atomically upgrade a service running on a unix socket, by starting the new version of the service on a temporary socket then renaming it into place, then closing and removing the old socket.

With care this can be generalised up to a directory with no subdirectories by hard-linking the contents into a temporary directory, making whatever changes are necessary, then swapping directories and unlinking the temporary directory and contents.

This doesn't work for subdirectories because you can't have a directory hardlink and if any of those subdirectories were in use you would get very different results if they were using that directory instead of your new version of that directory.

When you write software in isolation, for yourself and no others, you are not beholden to anyone's timescales but your own. Sadly for those of us who enjoy writing software this is a very rare state of being for a project. Usually there are other users, other developers, and other timescales to consider.

In the world of free software, some projects make a thing out of aggregating a lot of software together into something larger and coherent. We often refer to these things as 'Distributions' though there are other mechanisms by which software is aggregated.

When you have some software which goes into a software distribution of some kind, you're suddenly beholden to someone else's timing. For example if your software goes into a Debian release, you can be confident that the version of the program which goes into the release will be in active use for a number of years after the release; and you ought to provide support for that. Sadly it is exactly this which encourages people to not have their software packaged into distributions; which results in more pain for users.

If you love your users, I encourage you to pick at least one distribution and get your software into it. And then set yourself up to be the best upstream you possibly can, and provide support for the duration of that distribution's release. While it is hard, it is a surprisingly gratifying thing to do and will teach you lessons which no amount of Yakking articles could do.

So now we've got an equivalent to a slow rename(2), right?

Not quite, rename(2) is atomic. It disappears from the old location and reappears whole at the new one at the same time.

Our move leaves it partially in the new location while it's copying which can confuse another program that might be looking at it while it is being copied.

Atomic creation with a temporary file

This approach was described in a previous article.

The bulk of the change is to include the clobber options, the rest is to open a temporary file in the target's directory (as seen in open_tmpfile) and using rename_file after the contents have been changed.

int open_tmpfile(const char *target, char **tmpfn_out) {
    char *template = malloc(strlen(target) + sizeof("./.tmpXXXXXX"));
    char *dir = NULL;
    int ret;
    strcpy(template, target);
    dir = dirname(template);
    if (dir != template)
        strcpy(template, dir);
    strcat(template, "/");
    strcat(template, ".tmp");
    strcat(template, basename(target));
    strcat(template, "XXXXXX");
    ret = mkstemp(template);
    if (ret >= 0)
        *tmpfn_out = template;
    else
        free(template);
    return ret;
}

int copy_file(char *source, char *target, struct stat *source_stat,
              enum clobber clobber, enum setgid setgid, int required_flags) {
    int srcfd = -1;
    int tgtfd = -1;
    int ret = -1;
    char *tmppath = NULL;

    ret = open(source, O_RDONLY);
    if (ret == -1) {
        perror("Open source file");
        goto cleanup;
    }
    srcfd = ret;

    ret = set_selinux_create_context(target, source_stat->st_mode);
    if (ret != 0) {
        perror("Set selinux create context");
        goto cleanup;
    }

    ret = open_tmpfile(target, &tmppath);
    if (ret == -1) {
        perror("Open temporary target file");
        goto cleanup;
    }
    tgtfd = ret;

    ret = copy_contents(srcfd, tgtfd);
    if (ret < 0)
        goto cleanup;

    ret = fchmod(tgtfd, source_stat->st_mode);
    if (ret < 0)
        goto cleanup;

    ret = fix_owner(target, source_stat, setgid, tgtfd);
    if (ret < 0)
        goto cleanup;

    ret = copy_flags(srcfd, tgtfd, required_flags);
    if (ret < 0)
        goto cleanup;

    ret = copy_xattrs(srcfd, tgtfd);
    if (ret < 0)
        goto cleanup;

    ret = copy_posix_acls(srcfd, tgtfd);
    if (ret < 0)
        goto cleanup;

    {
        struct timespec times[] = { source_stat->st_atim, source_stat->st_mtim, };
        ret = futimens(tgtfd, times);
        if (ret < 0)
            goto cleanup;
    }

    ret = rename_file(tmppath, target, clobber);
cleanup:
    close(srcfd);
    close(tgtfd);
    if (tmppath && ret != 0)
        (void)unlink(tmppath);
    free(tmppath);
    return ret;
}

Atomic creation with an anonymous temporary file

Astute readers may have noticed that if your program crashes before renaming, that the temporary file may be left behind.

A potential solution to this is to use the O_TMPFILE option of open(2) and then linkat(2) to link the temporary file into place.

This is handy, because the file is cleared up automatically when the file descriptor is closed.

However an issue with this approach is that linkat(2) always returns EEXIST if the target exists, so if you wanted to clobber a file you would need to fall back to the temporary file approach as above, by using linkat(2) to link the file in at a temporary path and then use renameat2(2) to put it in place, which loses the primary benefit of the linkat(2) in the first place.

This may be worthwhile as it reduces the period where the file may be left behind, but such an approach is left as an exercise to the reader.

As usual, the code may be downloaded from my-mv.c and the accompanying Makefile may be downloaded.

So we've got something we can use to move any file now?

If you define "file" as a regular file, yes.

If you say "everything is a file", then you open it up to character devices, block devices, unix sockets, symbolic links, directories and btrfs subvolumes.

However every enterprise needs a given definition of done.

For now I think copying a regular file is enough, so detail in other types of file will have to be left for a future series.

We're surrounded by people telling us that software design comes from requirements, and that code comes from design. However in the real world, particularly in hobby coding, it's very common for software to grow "organically" from the particular itches one or two people choose to scratch.

The result of this is that software, particularly at the surface, is almost entirely there to satisfy its contributors. Fortunately for the majority of us that means that early users of the software get to shape it which might mean that it meets our needs too. The better software is written by people who actually spend time working out what their users need and writing for that, rather than for themselves.

This issue can go beneath the surface too. It's very common to find that internal APIs in software projects exist only to satisfy the author and thus the code being written. It's rare to find a more complete API than the author themselves needs, unless the author is designing an API for others to use (and frankly even then it can be patchy). This is so common that even when you're aware that it happens you can fall into the trap yourself.

For this week, I'd like you to go back to some of your software which you have released (or if you haven't released any, some software you might one day want to release) and look at its functionality. Consider if there might be gaps which users would like to be filled, and look at making the codebase coherent inside and out. Then file some bugs, make some kanban cards, or whatever you need to let yourself know what you need to do. If you're feeling energetic then make the changes too.

Code that is unused has little value.

If nobody is using your code then how do you know if it is any good?

Presumably you wrote it for a reason.

Don't hide your code away, let it be free!

Collaborative projects are more successful than solo projects.

Having others use your code and provide feedback keeps the project alive.

If others take an interest and provide patches - that is excellent, but having people tell you that they are using your code is excellent motivation for you to scratch some itches yourself.

Make your code easy to use for your target audience.

I don't know the audience for my code, I hope to find out by its reception.

If it turns out I don't have one, then so be it. In this case the code itself isn't the goal, it exists as a vehicle for writing these articles.

Pick a licence

No matter how you want to make your code available you will need to provide a licence so anyone can be provably allowed to use it.

My code is currently licensed under the CC BY-SA 3.0 licence, since that is the licence of all of the blog's contents.

Copyleft

If I were to use a copyleft licence then I would be more likely to attract contributions from people who do not want it to be usable by proprietary software developers who could take it and not contribute any changes back.

This potentially provides an incentive for potential users to contribute any potential improvements back.

The current code is already copyleft since it's CC BY-SA 3.0, but it's less suitable as a licence for code.

Permissive

A permissive licence would allow proprietary software developers to use my code in closed software projects.

This is theoretically a wider audience than those who can use copyleft, but some developers may, with good reason, object to contributing to a project which allows others to use it without contributing anything back.

It also relies on the generosity of developers to contribute any changes back since with a permissive licence there is no licence-provided incentive to do so.

My choice

I sometimes need to write proprietary software. I'd like to be able to have a high quality library to use rather than have to re-implement it all again.

I'd also prefer that people writing proprietary software use high quality code than implement it themselves and get it wrong.

For this reason I'm going to use the ISC licence

Let potential users know what licence the code is under

Put a LICENSE file in your repository to make it clear how the project is licensed so they can know quickly whether they can use your project.

Add the short text of the licence to the header of your source files to make sure there is no doubt that the whole of your project is licenced the same way.

Packaging

The absolute minimum is to have the source code repository publically visible, so anyone who wants a copy can fetch it themselves.

If you then make tags it becomes easier to get a specific version.

If your source code repository has a web UI it can provide snapshots, so users don't need to be proficient with your version control tool and can avoid downloading all the history they don't care about.

You can also make source distributions, which may be different from a snapshot. These usually involve packaging up any generated or derived source files to reduce the number of dependencies required to build the code.

If you have a set of expected target environments then you can provide binary packages for those environments. This can be a set of statically linked or bundled binaries designed to run on every platform. It could be uploaded to the platform's package repository. It could be participating in a distribution's community and packaging in that.

Since my project is too young to even have its first release I'm going to just make the code available. As-is it is compatibly licensed such that users can copy it into their codebase.

For the first release I intend to tidy it up into a library that can be installed by running make install.

Put it somewhere it can be found.

Traditionally this was done by putting it in a software distribution, either yourself or having a member of the community package it.

These days it's more common to add it to the language's package manager, so CPAN for Perl modules, PyPI for Python modules, Ruby Gems for Ruby packages, NPM for Node.js packages and Crates.io for Rust packages.

There is strong feeling about whether this is a good idea, but they do provide a degree of discoverability that I would like.

Unfortunately there is no equivalent for C code, so I'm going to throw it up on GitHub and hope for the best.

GitHub has its faults, the largest of which is that it is centralising a decentralised version control, but like it or not, this is how much free-software is done and in the absence of knowing my audience I'm going to take every advantage.

There's more to a project than a code dump

Now the code is out there. Well done.

But a project is more than just code.

Communication

You need to communicate.

When your project gets big enough you will need more than GitHub providing an issue tracker and an interface for merging patches.

You might need an IRC channel (or other chat protocol) for providing help and coordinating with other developers.

You might need a mailing list to receive patches, discuss important changes, and announce new releases.

Meeting other developers and users face-to-face is important, so if you get big enough you might need to arrange hack days, dev-rooms at conferences, or eventually maybe even your own conference.

It may be advisable to have a code of conduct for your community sooner rather than later, as it may help to have a statement that you wish to handle abuse seriously and it may prevent difficulty retrofitting a code of conduct in later.

Documentation

Code documentation for how something works is handy for developers, more so if there are other developers who won't have your memory.

Documentation of interfaces is also important, both internal interfaces for how implementation details interact and external interfaces for how your users are meant to use it, be it API documentation for libraries, usage text and man pages for command-line applications, or screenshotted guides for GUI applications.

Then you may need more in-depth guides for how your code fits in to the wider context, such as blog articles describing how to integrate your library into a wider framework to achieve a particular goal, use your command-line application in a pipeline or batch script, or an example where your GUI application is used.

Then there's wider-project documentation, such as plans, documentation of how to configure your code, how to set up a developemnt environment, where all the development resources are, etc.

Note that documentation is also a disincentive to change things, since it means there is more work involved in making changes, so it is preferable to prioritise documentation of things that are the least likely to change.

Don't use it as an excuse to not write documentation though, unless it's obvious how it's meant to work or there are numerous examples to crib from a lack of documentation is a huge disincentive from using a project.

Releasing

Taking snapshots of the current version only works for so long.

Users like releases as they mark a point of stability. Users like stability since it means the code behaves consistently.

Releases are not perfect as all code has bugs, so point-releases where improvements are made that don't change behaviour are essential, since even if there is a newer version which is better, users may depend on existing behaviour.

Use semantic versioning so it's clear to users what the compatibility is, and make it easy for existing tooling to reason about your version numbers.

Testing

Having a test suite makes it easier to accept contributions and have some confidence in your code's quality.

Test for use-cases your users use, since you don't want to cause them difficulties.

When you find bugs, write a test first so you can know when it is fixed and this becomes a regression test to ensure it does not break again.

Test at multiple levels so you can make the most appropriate tests. Write unit-tests so the behaviours of your individual modules stay working, and behaviour-level tests tests for use-cases your users care about.

If you depend on a library, look to see how well tested that is, since you depend on it working the way it currently does. Provide tests for the use-cases you care about.

Make these tests able to be run by anyone, so they can be run by contributors to see if their changes break anything and downstream users can use the tests to see whether your code works.

Make these tests run automatically when changes are merged, so you can have some confidence that the code everyone is expected to use works.

Make these tests run automatically for proposed changes. People's development environments are different, not every developer has access to every environment the code is used in. If proposed changes can be run in every expected target environment then you can reduce the risk of it breaking in less common environments.

As with documentation, testing can make it harder to change code, so it's a trade-off between development speed and quality.

Termination

Know when to give up. No project is immune to competition. If you discover another project with compatible goals is competing then it may be better to swallow your pride and join that project instead.

You may have insights that will be of value to that project instead, and if the goal is to have a useful project what does it matter that it was started by someone else?

How did I get here?

I started my article series because on a project I was working on a coworker did the naïve thing and used rename(2) to move a file.

Having spent far too long reading-up on the Linux file system interface I could talk at length about what was necessary to do it properly, but I could not explain this concisely, so I decided to write instead.

I thought it would be a single article, but then it grew larger than I expected and saw no prospect of finishing, so I split it up into another article, and then another, and another.

Now, some 9000 words later I have an 8-article series and a project on GitHub. So the lessons to take away from this are:

1. Moving files sounds like a trivial thing to do, but is actually a horrible mess of complexity.
2. Nothing is ever as simple as you intend and at some point you need to call it done so you can move on and do other things.

The code is at https://github.com/fishface60/linux-fsops, yes that is my GitHub account name, legacy and branding can be hard to get rid of.