Lose three house points for that pun

thanks for another massively useful article.

I didn't know about SEEK_HOLE and SEEK_DATA at all. As far as I was aware the traditional way to do a sparse copy is set some bound on the number of zeroes that will constitute a hole and when n or more such zeroes are read, instead of copying the zeroes we seek past the current destination offset by n. This has the obvious disadvantage that any holes present in the source file that are smaller than n bytes will not be preserved as holes in the destination.

There's also the disadvantage of taking a lot longer to read since you have to copy those zeroes into userspace and scan through them to see if they are indeed all zeroes. I think this is what tar, rsync and cp --sparse=always does, but I'd have to check.

It's also not fun for disk images, beyond the size difference, since you want to avoid writing what you don't need to when writing a disk image to a real disk, but you also need to write zeroes that are in the disk image, since you can end up with an invalid file system if you don't write those zeroes. https://source.tizen.org/documentation/reference/bmaptool/introduction was written by the Tizen guys for dealing with this. It produces and uses a separate file for the map of which sectors to copy, possibly because I know of no serialisation formats other than a disk image that preserves holes without dropping zero data sectors.

My man page for lseek(2) says that SEEK_HOLE and SEEK_DATA are only supported for more recent linux kernels and even then just for a selection of file systems: btrfs (3.1), OCFS (3.2), XFS (3.5), ext4 (3.8) and tmpfs (3.8). I wonder then whether it's worthwhile making a sparse copy function that falls back to the traditional sparse copy method if the new way is unsupported? (or maybe I'm living in the past having only made the switch from ext3 in the past year or so )

As far as I'm concerned those kernel versions are old enough that I can't feasibly test them, so I can't be sure my fallback would work right. I would accept a patch if provided one.

There are a couple of alternative fallbacks. The FIEMAP ioctl can be used to get information about which parts of the file contain what types of data, and the FIBMAP ioctl can tell you where on disk the data is. FIBMAP is old, so more widely available, but of limited use since it requires elevated privileges. FIEMAP is older than SEEK_{DATA,HOLE} and newer than FIBMAP, but has been historically the cause of file corruption. I don't recall whether old versions were just used improperly or whether there were bugs with it, but I steered clear of it and opted for the simpler interface of SEEK_{HOLE,DATA}.

Thanks for this article it's really useful, it would be interesting to know why you're not fond of the stdio interface,

I don't like the buffering behaviour, it defers writes late enough that I lose context about what bit of the write failed, so when I go to flush and close I can't say how much was actually written.

and also potentially worth mentioning that EINTR is really only something that needs to be explicitly handled on Linux, from signal(7):

On Linux, even in the absence of signal handlers, certain blocking interfaces  can
fail  with the error EINTR after the process is stopped by one of the stop signals
and then resumed via SIGCONT.  This behavior is not  sanctioned  by  POSIX.1,  and
doesn't occur on other systems.

This behaviour seems less than helpful to me, it would be really good to know if there's a good reason why GNU/Linux doesn't just restart the call (as the BSDs do)

There's also the fact that some signals can be configured to restart, but not others. Signals in general are a bit of a mess, so my way of dealing with it is to wrap everything in a retry and leave signal handling for shut down and use a better form of IPC for everything else.

Also, I had no idea you could make system calls without having a C wrapper for them, so thanks for that as well!

Yep, it's a pain when the wrappers don't exist because you need to handle the error return calling convention yourself, since part of what the libc wrappers do is set errno. I tend to use negative error number returns in my own code rather than errno which makes it actually closer to what I prefer.

thanks for another massively useful article.

I didn't know about SEEK_HOLE and SEEK_DATA at all. As far as I was aware the traditional way to do a sparse copy is set some bound on the number of zeroes that will constitute a hole and when n or more such zeroes are read, instead of copying the zeroes we seek past the current destination offset by n. This has the obvious disadvantage that any holes present in the source file that are smaller than n bytes will not be preserved as holes in the destination.

My man page for lseek(2) says that SEEK_HOLE and SEEK_DATA are only supported for more recent linux kernels and even then just for a selection of file systems: btrfs (3.1), OCFS (3.2), XFS (3.5), ext4 (3.8) and tmpfs (3.8). I wonder then whether it's worthwhile making a sparse copy function that falls back to the traditional sparse copy method if the new way is unsupported? (or maybe I'm living in the past having only made the switch from ext3 in the past year or so )

Thanks for this article it's really useful, it would be interesting to know why you're not fond of the stdio interface, and also potentially worth mentioning that EINTR is really only something that needs to be explicitly handled on Linux, from signal(7):

   On Linux, even in the absence of signal handlers, certain blocking interfaces  can
   fail  with the error EINTR after the process is stopped by one of the stop signals
   and then resumed via SIGCONT.  This behavior is not  sanctioned  by  POSIX.1,  and
   doesn't occur on other systems.

This behaviour seems less than helpful to me, it would be really good to know if there's a good reason why GNU/Linux doesn't just restart the call (as the BSDs do)

Also, I had no idea you could make system calls without having a C wrapper for them, so thanks for that as well!

Sorry I didn't notice your reply sooner.

Firstly, it isn't const-correct:

  test.c: In function ‘open_tmpfile’:
  test.c:12:31: warning: passing argument 1 of ‘__xpg_basename’ discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
       strcat(template, basename(target));
                                 ^~~~~~
  In file included from test.c:1:0:
  /usr/include/libgen.h:34:14: note: expected ‘char *’ but argument is of type ‘const char *’
   extern char *__xpg_basename (char *__path) __THROW;
                ^~~~~~~~~~~~~~

This is from the awkwardness of there being two different definitions of basename. I don't get this error in the full version of the code (http://yakking.branchable.com/posts/moving-files-8-atomicity/my-mv.c) since I need to pull in the definitions with:

#include <libgen.h>          /* dirname */
#undef basename
#include <string.h>          /* basename */

If you are aware of a better way to pull in the definitions please enlighten me, I'm working from the manpages as to how to pull the definitions in.

Second, strcpy(template, dirname(template)); has undefined behaviour since dirname(template) will probably point to template and strcpy() does not work with overlapping buffers. I think you have to use memmove() or two separate buffers.

Thanks, I apparently missed the warning about overlaps when I checked the documentation. Given I only actually want to do any copy when dirname(template) doesn't point to template I've gone with:

char *dir = NULL;
strcpy(template, target);
dir = dirname(template);
if (dir != template)
    strcpy(template, dir);

Finally, it leaks the template string if mkstemp() fails, though that's not very likely.

Thanks. I have no excuse for how I missed that, I was probably writing late into the night and then didn't go back and check.

Firstly, it isn't const-correct:

test.c: In function ‘open_tmpfile’:
test.c:12:31: warning: passing argument 1 of ‘__xpg_basename’ discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
     strcat(template, basename(target));
                               ^~~~~~
In file included from test.c:1:0:
/usr/include/libgen.h:34:14: note: expected ‘char *’ but argument is of type ‘const char *’
 extern char *__xpg_basename (char *__path) __THROW;
              ^~~~~~~~~~~~~~

Second, strcpy(template, dirname(template)); has undefined behaviour since dirname(template) will probably point to template and strcpy() does not work with overlapping buffers. I think you have to use memmove() or two separate buffers.

Finally, it leaks the template string if mkstemp() fails, though that's not very likely.

It is not a problem with SED to print a previous line if the current line match something. You just copy the pattern space (current line) into the Hold Space area. If there is a match, you can copy the Hold Space back into the pattern space.

eg.

sed -n ' /MATCH/ { g ; p } ; h '

... used this technique to implement a state machine

... used this technique to implement a state machine