pages tagged renameyakkinghttp://yakking.branchable.com/tags/rename/yakkingikiwiki2017-01-09T22:34:59ZHow difficult is it to move a file atomically?http://yakking.branchable.com/posts/moving-files-8-atomicity/Richard Maw2017-01-09T22:34:59Z2016-11-16T12:00:07Z
<h1>So now we've got an equivalent to a slow <a href="http://man7.org/linux/man-pages/man2/rename.2.html">rename(2)</a>, right?</h1>
<p>Not quite, <a href="http://man7.org/linux/man-pages/man2/rename.2.html">rename(2)</a> is atomic.
It disappears from the old location and reappears whole at the new one at the same time.</p>
<p>Our move leaves it partially in the new location while it's copying
which can confuse another program that might be looking at it
while it is being copied.</p>
<h2>Atomic creation with a temporary file</h2>
<p>This approach was described in a previous article.</p>
<p>The bulk of the change is to include the clobber options,
the rest is to open a temporary file in the target's directory
(as seen in <code>open_tmpfile</code>)
and using <code>rename_file</code> after the contents have been changed.</p>
<div class="highlight-c"><pre class="hl"><span class="hl kwb">int</span> <span class="hl kwd">open_tmpfile</span><span class="hl opt">(</span><span class="hl kwb">const char</span> <span class="hl opt">*</span>target<span class="hl opt">,</span> <span class="hl kwb">char</span> <span class="hl opt">**</span>tmpfn_out<span class="hl opt">) {</span>
<span class="hl kwb">char</span> <span class="hl opt">*</span><span class="hl kwc">template</span> <span class="hl opt">=</span> <span class="hl kwd">malloc</span><span class="hl opt">(</span><span class="hl kwd">strlen</span><span class="hl opt">(</span>target<span class="hl opt">) +</span> <span class="hl kwa">sizeof</span><span class="hl opt">(</span><span class="hl str">"./.tmpXXXXXX"</span><span class="hl opt">));</span>
<span class="hl kwb">char</span> <span class="hl opt">*</span>dir <span class="hl opt">=</span> NULL<span class="hl opt">;</span>
<span class="hl kwb">int</span> ret<span class="hl opt">;</span>
<span class="hl kwd">strcpy</span><span class="hl opt">(</span><span class="hl kwc">template</span><span class="hl opt">,</span> target<span class="hl opt">);</span>
dir <span class="hl opt">=</span> <span class="hl kwd">dirname</span><span class="hl opt">(</span><span class="hl kwc">template</span><span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>dir <span class="hl opt">!=</span> <span class="hl kwc">template</span><span class="hl opt">)</span>
<span class="hl kwd">strcpy</span><span class="hl opt">(</span><span class="hl kwc">template</span><span class="hl opt">,</span> dir<span class="hl opt">);</span>
<span class="hl kwd">strcat</span><span class="hl opt">(</span><span class="hl kwc">template</span><span class="hl opt">,</span> <span class="hl str">"/"</span><span class="hl opt">);</span>
<span class="hl kwd">strcat</span><span class="hl opt">(</span><span class="hl kwc">template</span><span class="hl opt">,</span> <span class="hl str">".tmp"</span><span class="hl opt">);</span>
<span class="hl kwd">strcat</span><span class="hl opt">(</span><span class="hl kwc">template</span><span class="hl opt">,</span> <span class="hl kwd">basename</span><span class="hl opt">(</span>target<span class="hl opt">));</span>
<span class="hl kwd">strcat</span><span class="hl opt">(</span><span class="hl kwc">template</span><span class="hl opt">,</span> <span class="hl str">"XXXXXX"</span><span class="hl opt">);</span>
ret <span class="hl opt">=</span> <span class="hl kwd">mkstemp</span><span class="hl opt">(</span><span class="hl kwc">template</span><span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt">>=</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl opt">*</span>tmpfn_out <span class="hl opt">=</span> <span class="hl kwc">template</span><span class="hl opt">;</span>
<span class="hl kwa">else</span>
<span class="hl kwd">free</span><span class="hl opt">(</span><span class="hl kwc">template</span><span class="hl opt">);</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl kwb">int</span> <span class="hl kwd">copy_file</span><span class="hl opt">(</span><span class="hl kwb">char</span> <span class="hl opt">*</span>source<span class="hl opt">,</span> <span class="hl kwb">char</span> <span class="hl opt">*</span>target<span class="hl opt">,</span> <span class="hl kwb">struct</span> stat <span class="hl opt">*</span>source_stat<span class="hl opt">,</span>
<span class="hl kwb">enum</span> clobber clobber<span class="hl opt">,</span> <span class="hl kwb">enum</span> setgid setgid<span class="hl opt">,</span> <span class="hl kwb">int</span> required_flags<span class="hl opt">) {</span>
<span class="hl kwb">int</span> srcfd <span class="hl opt">= -</span><span class="hl num">1</span><span class="hl opt">;</span>
<span class="hl kwb">int</span> tgtfd <span class="hl opt">= -</span><span class="hl num">1</span><span class="hl opt">;</span>
<span class="hl kwb">int</span> ret <span class="hl opt">= -</span><span class="hl num">1</span><span class="hl opt">;</span>
<span class="hl kwb">char</span> <span class="hl opt">*</span>tmppath <span class="hl opt">=</span> NULL<span class="hl opt">;</span>
ret <span class="hl opt">=</span> <span class="hl kwd">open</span><span class="hl opt">(</span>source<span class="hl opt">,</span> O_RDONLY<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt">== -</span><span class="hl num">1</span><span class="hl opt">) {</span>
<span class="hl kwd">perror</span><span class="hl opt">(</span><span class="hl str">"Open source file"</span><span class="hl opt">);</span>
<span class="hl kwa">goto</span> cleanup<span class="hl opt">;</span>
<span class="hl opt">}</span>
srcfd <span class="hl opt">=</span> ret<span class="hl opt">;</span>
ret <span class="hl opt">=</span> <span class="hl kwd">set_selinux_create_context</span><span class="hl opt">(</span>target<span class="hl opt">,</span> source_stat<span class="hl opt">-></span>st_mode<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt">!=</span> <span class="hl num">0</span><span class="hl opt">) {</span>
<span class="hl kwd">perror</span><span class="hl opt">(</span><span class="hl str">"Set selinux create context"</span><span class="hl opt">);</span>
<span class="hl kwa">goto</span> cleanup<span class="hl opt">;</span>
<span class="hl opt">}</span>
ret <span class="hl opt">=</span> <span class="hl kwd">open_tmpfile</span><span class="hl opt">(</span>target<span class="hl opt">, &</span>tmppath<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt">== -</span><span class="hl num">1</span><span class="hl opt">) {</span>
<span class="hl kwd">perror</span><span class="hl opt">(</span><span class="hl str">"Open temporary target file"</span><span class="hl opt">);</span>
<span class="hl kwa">goto</span> cleanup<span class="hl opt">;</span>
<span class="hl opt">}</span>
tgtfd <span class="hl opt">=</span> ret<span class="hl opt">;</span>
ret <span class="hl opt">=</span> <span class="hl kwd">copy_contents</span><span class="hl opt">(</span>srcfd<span class="hl opt">,</span> tgtfd<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">goto</span> cleanup<span class="hl opt">;</span>
ret <span class="hl opt">=</span> <span class="hl kwd">fchmod</span><span class="hl opt">(</span>tgtfd<span class="hl opt">,</span> source_stat<span class="hl opt">-></span>st_mode<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">goto</span> cleanup<span class="hl opt">;</span>
ret <span class="hl opt">=</span> <span class="hl kwd">fix_owner</span><span class="hl opt">(</span>target<span class="hl opt">,</span> source_stat<span class="hl opt">,</span> setgid<span class="hl opt">,</span> tgtfd<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">goto</span> cleanup<span class="hl opt">;</span>
ret <span class="hl opt">=</span> <span class="hl kwd">copy_flags</span><span class="hl opt">(</span>srcfd<span class="hl opt">,</span> tgtfd<span class="hl opt">,</span> required_flags<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">goto</span> cleanup<span class="hl opt">;</span>
ret <span class="hl opt">=</span> <span class="hl kwd">copy_xattrs</span><span class="hl opt">(</span>srcfd<span class="hl opt">,</span> tgtfd<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">goto</span> cleanup<span class="hl opt">;</span>
ret <span class="hl opt">=</span> <span class="hl kwd">copy_posix_acls</span><span class="hl opt">(</span>srcfd<span class="hl opt">,</span> tgtfd<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">goto</span> cleanup<span class="hl opt">;</span>
<span class="hl opt">{</span>
<span class="hl kwb">struct</span> timespec times<span class="hl opt">[] = {</span> source_stat<span class="hl opt">-></span>st_atim<span class="hl opt">,</span> source_stat<span class="hl opt">-></span>st_mtim<span class="hl opt">, };</span>
ret <span class="hl opt">=</span> <span class="hl kwd">futimens</span><span class="hl opt">(</span>tgtfd<span class="hl opt">,</span> times<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">goto</span> cleanup<span class="hl opt">;</span>
<span class="hl opt">}</span>
ret <span class="hl opt">=</span> <span class="hl kwd">rename_file</span><span class="hl opt">(</span>tmppath<span class="hl opt">,</span> target<span class="hl opt">,</span> clobber<span class="hl opt">);</span>
cleanup<span class="hl opt">:</span>
<span class="hl kwd">close</span><span class="hl opt">(</span>srcfd<span class="hl opt">);</span>
<span class="hl kwd">close</span><span class="hl opt">(</span>tgtfd<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>tmppath <span class="hl opt">&&</span> ret <span class="hl opt">!=</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl opt">(</span><span class="hl kwb">void</span><span class="hl opt">)</span><span class="hl kwd">unlink</span><span class="hl opt">(</span>tmppath<span class="hl opt">);</span>
<span class="hl kwd">free</span><span class="hl opt">(</span>tmppath<span class="hl opt">);</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
<span class="hl opt">}</span>
</pre></div>
<h2>Atomic creation with an anonymous temporary file</h2>
<p>Astute readers may have noticed that if your program crashes before renaming,
that the temporary file may be left behind.</p>
<p>A potential solution to this is to use the <code>O_TMPFILE</code> option of <a href="http://man7.org/linux/man-pages/man2/open.2.html">open(2)</a>
and then <a href="http://man7.org/linux/man-pages/man2/linkat.2.html">linkat(2)</a> to link the temporary file into place.</p>
<p>This is handy, because the file is cleared up automatically
when the file descriptor is closed.</p>
<p>However an issue with this approach
is that <a href="http://man7.org/linux/man-pages/man2/linkat.2.html">linkat(2)</a> always returns <code>EEXIST</code> if the target exists,
so if you wanted to clobber a file you would need to fall back
to the temporary file approach as above,
by using <a href="http://man7.org/linux/man-pages/man2/linkat.2.html">linkat(2)</a> to link the file in at a temporary path
and then use <a href="http://man7.org/linux/man-pages/man2/renameat2.2.html">renameat2(2)</a> to put it in place,
which loses the primary benefit of the <a href="http://man7.org/linux/man-pages/man2/linkat.2.html">linkat(2)</a> in the first place.</p>
<p>This may be worthwhile as it reduces the period
where the file may be left behind,
but such an approach is left as an exercise to the reader.</p>
<p>As usual, the code may be downloaded from <a href="http://yakking.branchable.com/posts/moving-files-8-atomicity/my-mv.c">my-mv.c</a>
and the accompanying <a href="http://yakking.branchable.com/posts/moving-files-8-atomicity/Makefile">Makefile</a> may be downloaded.</p>
<h1>So we've got something we can use to move any file now?</h1>
<p>If you define "file" as a regular file, yes.</p>
<p>If you say "everything is a file",
then you open it up to character devices, block devices, unix sockets,
symbolic links, directories and btrfs subvolumes.</p>
<p>However every enterprise needs a given definition of done.</p>
<p>For now I think copying a regular file is enough,
so detail in other types of file will have to be left for a future series.</p>
Atomic file creation with temporary fileshttp://yakking.branchable.com/posts/atomic-file-creation-tmpfile/Richard Maw2016-11-02T12:00:13Z2016-11-02T12:00:07Z
<p>We previously spoke about atomic file clobbering
with <a href="http://man7.org/linux/man-pages/man2/open.2.html">open(2)</a> and <a href="http://man7.org/linux/man-pages/man2/renameat2.2.html">renameat2(2)</a>.</p>
<p>This can be used to perform an atomic file creation too.</p>
<p>At a really high level the idea is:</p>
<ol>
<li>Create a temporary file.</li>
<li>Complete setting it up.</li>
<li>Rename the temporary file into place.</li>
</ol>
<p>In more detail:</p>
<ol>
<li><p>Decide whether creating a new file or replacing an existing one.</p>
<p>This determines which flags <a href="http://man7.org/linux/man-pages/man2/renameat2.2.html">renameat2(2)</a> gets later.
You can <a href="http://man7.org/linux/man-pages/man2/stat.2.html">stat(2)</a> the destination path before proceeding
which allows you to detect whether the target was added or removed
while you were building the file.</p></li>
<li><p>Pick a temporary file path.</p>
<p>This needs to be on the same file system as the destination,
since we are going to use <a href="http://man7.org/linux/man-pages/man2/renameat2.2.html">renameat2(2)</a> later.</p>
<p>Ideally this would be in a directory that is cleaned up automatically
so that if your program or computer crashes
you won't have the temporary file left around.</p>
<p>It's rare to be able to do this, since that pretty much just leaves
a creating files on a tmpfs when your <code>/tmp</code> is a mounted tmpfs,
or creating files on your root partition when <code>/tmp</code> is not a mounted tmpfs.</p>
<p>You could find the root directory of the mount point the destination is in
and put it in a temporary directory in there,
but this would still require system integration
to have it automatically cleaned up on mount.</p>
<p>With either <code>/tmp</code> or a custom tempdir on the mount point
you also have the problem that when you later use <a href="http://man7.org/linux/man-pages/man2/renameat2.2.html">renameat2(2)</a>
it can fail from another process mounting a directory on top
of one of the directories in the destination path.</p>
<p>So even though it can leave a temporary file behind
creating your temporary file in the same directory as your target
is probably the best approach when atomically creating a file
with a temporary file.</p>
<p><a href="http://man7.org/linux/man-pages/man2/linkat.2.html">linkat(2)</a> and <code>O_TMPFILE</code> provide a better solution for this,
but that's a topic for another article,
and only works for regular files.</p></li>
<li><p>Pick a random file path in your temporary directory.</p>
<p>This is what <a href="http://man7.org/linux/man-pages/man3/mkstemp.3.html">mkstemp(3)</a> does under the hood.</p>
<p>Since we intend to check whether the target file already exists
we could safely use [tmpnam(3)][] or [mktemp(3)][] to make it
(though not [tempnam(3)][] since it prefers the <code>TMPDIR</code>
environment variable for creating the temp file in
even if a directory is passed),
however in their zeal to stop people writing insecure programs
the POSIX standard has either removed or marked these functions obsolete,
so you probably want your own implementation.</p></li>
<li><p>Create your new file in a way that will fail if it already exists.</p>
<p>For regular files this requires passing <code>O_CREAT|O_EXCL</code> to the flags.</p>
<p>The system calls for creating
fifos, device nodes, symlinks, directories and unix sockets
already default to failing if the target already exists.</p>
<p><a href="http://man7.org/linux/man-pages/man2/open.2.html">open(2)</a>, <a href="http://man7.org/linux/man-pages/man3/mkfifo.3.html">mkfifo(3)</a>, <a href="http://man7.org/linux/man-pages/man2/mknod.2.html">mknod(2)</a>, <a href="http://man7.org/linux/man-pages/man2/symlink.2.html">symlink(2)</a> and <a href="http://man7.org/linux/man-pages/man2/mkdir.2.html">mkdir(2)</a>
return an errno of <code>EEXIST</code>.</p>
<p><a href="http://man7.org/linux/man-pages/man2/bind.2.html">bind(2)</a> (for unix sockets) returns <code>EADDRINUSE</code>.</p>
<p><a href="http://man7.org/linux/man-pages/man2/open.2.html">open(2)</a>, <a href="http://man7.org/linux/man-pages/man3/mkfifo.3.html">mkfifo(3)</a>, <a href="http://man7.org/linux/man-pages/man2/mknod.2.html">mknod(2)</a>, <a href="http://man7.org/linux/man-pages/man2/symlink.2.html">symlink(2)</a> and <a href="http://man7.org/linux/man-pages/man2/mkdir.2.html">mkdir(2)</a>
have variant syscalls with the "at" suffix
which take a file descriptor for the directory to create them in.</p>
<p>Using these or changing directory into the destination provides resilience
against the filesystem mounts changing mid-operation.</p>
<p>When using <a href="http://man7.org/linux/man-pages/man2/bind.2.html">bind(2)</a> with a unix socket you probably want to <a href="http://man7.org/linux/man-pages/man2/chdir.2.html">chdir(2)</a>
anyway, since the maximum length path is shorter than <code>PATH_MAX</code>.</p>
<p>If creation fails, go back to step 4.</p></li>
<li><p>Complete initialisation of the file.</p>
<p>What this involves depends on the type of file and your application.</p>
<p>Regular files will want their contents written and metadata set.</p>
<p>Everything else probably just wants a few metadata tweaks.</p></li>
<li><p>Rename the file into place.</p>
<p>This is the <a href="http://man7.org/linux/man-pages/man2/renameat2.2.html">renameat2(2)</a> trick mentioned in an earlier article.</p>
<ol>
<li>If you explicitly want to create the file pass <code>RENAME_NOREPLACE</code>.</li>
<li>If you explicitly want to replace a file pass <code>RENAME_EXCHANGE</code>
and [unlink(2)][] the temporary file,</li>
<li>If you're not sure <a href="http://man7.org/linux/man-pages/man2/stat.2.html">stat(2)</a> the destination before creating
and pass <code>RENAME_NOREPLACE</code> if it didn't exist beforehand,
and <code>RENAME_EXCHANGE</code> as above if it did,
so if you get a failure you can warn, ask or abort.</li>
<li>If you're really sure you don't care if it either creates a new file
or replaces an old one then don't pass any flags.</li>
</ol>
<p>Few of the errors that may happen are recoverable from,
(though <code>EEXIST</code> when you don't care about clobbering can be handled
by removing the destination while linking fails)
but checking the return code can help you provide useful error messages.</p>
<ul>
<li><code>ENOENT</code> when using <code>RENAME_EXCHANGE</code> means the destination file
doesn't exist when it should.</li>
<li><code>EEXIST</code> when using <code>RENAME_NOREPLACE</code> means the destination file
exists when it should not.</li>
<li><code>EROFS</code> or <code>EXDEV</code> mean some process changed the file system mounts
while the operation was in-progress.
<code>EROFS</code> being making it read-only,
and <code>EXDEV</code> being some mount on top of your directory.</li>
</ul>
<p>If you're particularly paranoid you can use <a href="http://man7.org/linux/man-pages/man2/stat.2.html">stat(2)</a>
to check whether the destination file is still reachable
from the root directory
by checking whether the <code>st_dev</code> and <code>st_ino</code> match.</p></li>
</ol>
<p>And there you have it. Atomic file updates.</p>
<h1>What would this be useful for</h1>
<p>This is useful if you have many processes that might make use of a file
and you need to update it to a new version.</p>
<p>This would be useful for shared caches (such as <a href="http://man7.org/linux/man-pages/man8/ld.so.8.html">/etc/ld.so.cache</a>)
which can't use a lock file to synchronise updates
either for performance or legacy client reasons.</p>
<p>It can also be used to atomically upgrade a service running on a unix socket,
by starting the new version of the service on a temporary socket
then renaming it into place, then closing and removing the old socket.</p>
<p>With care this can be generalised up to a directory with no subdirectories
by hard-linking the contents into a temporary directory,
making whatever changes are necessary,
then swapping directories and unlinking the temporary directory and contents.</p>
<p>This doesn't work for subdirectories because you can't have a directory hardlink
and if any of those subdirectories were in use
you would get very different results if they were using that directory
instead of your new version of that directory.</p>
Atomically clobbering fileshttp://yakking.branchable.com/posts/clobbering/Richard Maw2016-10-26T11:00:19Z2016-10-26T11:00:12Z
<p>I mentioned atomicity at the end of my previous article
and had intended to write about how to make all the previous operations atomic,
but doing so requires understanding how to atomically clobber files
before we can atomically move them.</p>
<p>The naive approach would be to check whether the file existed
before attempting the operation.</p>
<p>However this runs the risk of the file being added or removed
between the check and the operation,
in what's known as a <a href="https://en.wikipedia.org/wiki/Time_of_check_to_time_of_use">TOCTOU</a> attack.</p>
<p>We previously mentioned how to atomically create a file without clobbering,
but we may instead explicitly want to clobber it, or not care.</p>
<p>For opening a file, this is just a matter of using different flags.</p>
<div class="highlight-c"><pre class="hl"><span class="hl kwb">enum</span> clobber <span class="hl opt">{</span>
CLOBBER_PERMITTED <span class="hl opt">=</span> <span class="hl str">'p'</span><span class="hl opt">,</span>
CLOBBER_REQUIRED <span class="hl opt">=</span> <span class="hl str">'R'</span><span class="hl opt">,</span>
CLOBBER_FORBIDDEN <span class="hl opt">=</span> <span class="hl str">'N'</span><span class="hl opt">,</span>
CLOBBER_TRY_REQUIRED <span class="hl opt">=</span> <span class="hl str">'r'</span><span class="hl opt">,</span>
CLOBBER_TRY_FORBIDDEN <span class="hl opt">=</span> <span class="hl str">'n'</span><span class="hl opt">,</span>
<span class="hl opt">};</span>
<span class="hl kwb">int</span> <span class="hl kwd">create_file</span><span class="hl opt">(</span><span class="hl kwb">const char</span> <span class="hl opt">*</span>path<span class="hl opt">,</span> mode_t mode<span class="hl opt">,</span> <span class="hl kwb">int</span> flags<span class="hl opt">,</span>
<span class="hl kwb">enum</span> clobber clobber<span class="hl opt">) {</span>
<span class="hl kwa">switch</span> <span class="hl opt">(</span>clobber<span class="hl opt">) {</span>
<span class="hl kwa">case</span> CLOBBER_PERMITTED<span class="hl opt">:</span>
flags <span class="hl opt">|=</span> O_CREAT<span class="hl opt">;</span>
<span class="hl kwa">break</span><span class="hl opt">;</span>
<span class="hl kwa">case</span> CLOBBER_REQUIRED<span class="hl opt">:</span>
<span class="hl kwa">case</span> CLOBBER_TRY_REQUIRED<span class="hl opt">:</span>
flags <span class="hl opt">&= ~</span>O_CREAT<span class="hl opt">;</span>
<span class="hl kwa">break</span><span class="hl opt">;</span>
<span class="hl kwa">case</span> CLOBBER_FORBIDDEN<span class="hl opt">:</span>
<span class="hl kwa">case</span> CLOBBER_TRY_FORBIDDEN<span class="hl opt">:</span>
flags <span class="hl opt">|=</span> O_CREAT<span class="hl opt">|</span>O_EXCL<span class="hl opt">;</span>
<span class="hl kwa">break</span><span class="hl opt">;</span>
<span class="hl kwa">default</span><span class="hl opt">:</span>
<span class="hl kwa">assert</span><span class="hl opt">(</span><span class="hl num">0</span><span class="hl opt">);</span>
<span class="hl opt">}</span>
<span class="hl kwa">return</span> <span class="hl kwd">open</span><span class="hl opt">(</span>path<span class="hl opt">,</span> flags<span class="hl opt">,</span> mode<span class="hl opt">);</span>
<span class="hl opt">}</span>
</pre></div>
<p>For renaming a file things get a bit more awkward.</p>
<p>There are flags for changing how the <a href="http://man7.org/linux/man-pages/man2/rename.2.html">rename</a> behaves when the file exists,
but there isn't one for requiring that it does so.</p>
<p>Instead there's <code>RENAME_EXCHANGE</code>
which will fail if the target does not exist
and the source file will replace the target on success,
but it has the side effect of leaving the target file behind where the source file was.</p>
<p>This can be remedied by calling <a href="http://man7.org/linux/man-pages/man2/unlink.2.html">unlink(2)</a>.</p>
<div class="highlight-c"><pre class="hl"><span class="hl kwb">int</span> <span class="hl kwd">rename_file</span><span class="hl opt">(</span><span class="hl kwb">const char</span> <span class="hl opt">*</span>src<span class="hl opt">,</span> <span class="hl kwb">const char</span> <span class="hl opt">*</span>tgt<span class="hl opt">,</span> <span class="hl kwb">enum</span> clobber clobber<span class="hl opt">) {</span>
<span class="hl kwb">int</span> ret <span class="hl opt">= -</span><span class="hl num">1</span><span class="hl opt">;</span>
<span class="hl kwb">int</span> renameflags <span class="hl opt">=</span> <span class="hl num">0</span><span class="hl opt">;</span>
<span class="hl kwa">switch</span> <span class="hl opt">(</span>clobber<span class="hl opt">) {</span>
<span class="hl kwa">case</span> CLOBBER_REQUIRED<span class="hl opt">:</span>
<span class="hl kwa">case</span> CLOBBER_TRY_REQUIRED<span class="hl opt">:</span>
renameflags <span class="hl opt">=</span> RENAME_EXCHANGE<span class="hl opt">;</span>
<span class="hl kwa">break</span><span class="hl opt">;</span>
<span class="hl kwa">case</span> CLOBBER_FORBIDDEN<span class="hl opt">:</span>
<span class="hl kwa">case</span> CLOBBER_TRY_FORBIDDEN<span class="hl opt">:</span>
renameflags <span class="hl opt">=</span> RENAME_NOREPLACE<span class="hl opt">;</span>
<span class="hl kwa">break</span><span class="hl opt">;</span>
<span class="hl kwa">default</span><span class="hl opt">:</span>
<span class="hl kwa">assert</span><span class="hl opt">(</span><span class="hl num">0</span><span class="hl opt">);</span>
<span class="hl opt">}</span>
ret <span class="hl opt">=</span> <span class="hl kwd">renameat2</span><span class="hl opt">(</span>AT_FDCWD<span class="hl opt">,</span> src<span class="hl opt">,</span> AT_FDCWD<span class="hl opt">,</span> tgt<span class="hl opt">,</span> renameflags<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt">==</span> <span class="hl num">0</span><span class="hl opt">) {</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>clobber <span class="hl opt">==</span> CLOBBER_REQUIRED <span class="hl opt">||</span> clobber <span class="hl opt">==</span> CLOBBER_TRY_REQUIRED<span class="hl opt">) {</span>
ret <span class="hl opt">=</span> <span class="hl kwd">unlink</span><span class="hl opt">(</span>src<span class="hl opt">);</span>
<span class="hl opt">}</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl kwa">if</span> <span class="hl opt">((</span>errno <span class="hl opt">==</span> ENOSYS <span class="hl opt">||</span> errno <span class="hl opt">==</span> EINVAL<span class="hl opt">)</span>
<span class="hl opt">&& (</span>clobber <span class="hl opt">!=</span> CLOBBER_REQUIRED
<span class="hl opt">&&</span> clobber <span class="hl opt">!=</span> CLOBBER_FORBIDDEN<span class="hl opt">)) {</span>
ret <span class="hl opt">=</span> <span class="hl kwd">rename</span><span class="hl opt">(</span>src<span class="hl opt">,</span> tgt<span class="hl opt">);</span>
<span class="hl opt">}</span>
cleanup<span class="hl opt">:</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
<span class="hl opt">}</span>
</pre></div>
<p>A test program,
<a href="http://yakking.branchable.com/posts/clobbering/clobbering.c">clobbering.c</a>
and the accompanying <a href="http://yakking.branchable.com/posts/clobbering/Makefile">Makefile</a> may be downloaded.</p>
<p>This test program will rename if passed two file paths
and copy standard input to a file if only passed one path.</p>
How difficult is it to preserve extended attributes when moving a file?http://yakking.branchable.com/posts/moving-files-7-difficult-xattrs/Richard Maw2016-10-19T11:00:13Z2016-10-19T11:00:06Z
<h1>Dealing with semantically important <a href="http://man7.org/linux/man-pages/man5/attr.5.html">xattrs</a></h1>
<p>We previously spoke about <a href="http://man7.org/linux/man-pages/man5/attr.5.html">extended attributes</a>
like they were just another piece of metadata attached to files.</p>
<p>However some have rather awkward interfaces as far as copying is concerned,
some because they don't depend on the file's contents itself,
and some because they are filesystem specific.</p>
<h2>Selinux labels</h2>
<p>Selinux is a complicated mandatory access control mechanism.</p>
<p>Rather than store the access control rules in the file,
like POSIX ACLs do,
the rules are stored elsewhere in the kernel
and a reference to what kind of file it is,
is stored in the file as an extended attribute
called the "security label".</p>
<p>The security label and the security context of the process accessing the file
are looked up in the ACL rules in the kernel
to determine whether the operation is permitted.</p>
<p>The details of how to define Selinux rules is complicated
and beyond the scope of this article.
We only care how we should reapply the rules when moving the file.</p>
<p>While we could copy the label from the old file into the new file,
as we did for POSIX ACLs,
Selinux contexts are defined by their file paths rather than the inodes,
so after we move a file we should relabel it
to what the file should have in the new location.</p>
<p>Using <a href="http://man7.org/linux/man-pages/man3/selinux_restorecon.3.html">selinux_restorecon(3)</a> might be tempting,
but it leaves open a race condition
where the file would be created with the wrong context
so it temporarily accessible with the wrong label.</p>
<p>If the file context should be preserved from the original file,
then you must read the context from the extended attribute,
either directly with <a href="http://man7.org/linux/man-pages/man2/fgetxattr.2.html">fgetxattr(2)</a> or <a href="http://man7.org/linux/man-pages/man3/fgetfilecon.3.html">fgetfilecon(3)</a>,
and then set the context before creating the new file
with <a href="http://man7.org/linux/man-pages/man3/setfscreatecon.3.html">setfscreatecon(3)</a>.</p>
<p>If instead it should have the label that the path database says it should be,
then the required context can be found by using <a href="http://man7.org/linux/man-pages/man3/selabel_open.3.html">selabel_open(3)</a>
with <code>SELABEL_CTX_FILE</code> to get a reference to the file contexts database,
then getting the label it should have at that path using <a href="http://man7.org/linux/man-pages/man3/selabel_lookup.3.html">selabel_lookup(3)</a>,
and setting the context for new files with <a href="http://man7.org/linux/man-pages/man3/setfscreatecon.3.html">setfscreatecon(3)</a>.</p>
<p>Existing files can have their labels changed with <a href="http://man7.org/linux/man-pages/man3/selinux_restorecon.3.html">selinux_restorecon(3)</a>.</p>
<p>The <a href="http://man7.org/linux/man-pages/man3/setfscreatecon.3.html">setfscreatecon(3)</a> API is unfortunate as it involves global state.
Recent enough versions of Linux have the <code>O_TMPFILE</code> flag for file creation,
which doesn't create a directory entry for the file when it is created,
so you can modify the file before it is visible to other processes,
and can be bound into place with <a href="http://man7.org/linux/man-pages/man2/link.2.html">linkat(2)</a>.</p>
<div class="highlight-c"><pre class="hl"><span class="hl kwb">int</span> <span class="hl kwd">set_selinux_create_context</span><span class="hl opt">(</span><span class="hl kwb">const char</span> <span class="hl opt">*</span>tgt<span class="hl opt">,</span> mode_t srcmode<span class="hl opt">) {</span>
<span class="hl kwb">int</span> ret <span class="hl opt">=</span> <span class="hl num">0</span><span class="hl opt">;</span>
<span class="hl kwb">struct</span> selabel_handle <span class="hl opt">*</span>hnd <span class="hl opt">=</span> NULL<span class="hl opt">;</span>
<span class="hl kwb">char</span> <span class="hl opt">*</span>context <span class="hl opt">=</span> NULL<span class="hl opt">;</span>
hnd <span class="hl opt">=</span> <span class="hl kwd">selabel_open</span><span class="hl opt">(</span>SELABEL_CTX_FILE<span class="hl opt">,</span> NULL<span class="hl opt">,</span> <span class="hl num">0</span><span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>hnd <span class="hl opt">==</span> NULL<span class="hl opt">) {</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>errno <span class="hl opt">!=</span> ENOENT<span class="hl opt">) {</span>
ret <span class="hl opt">=</span> <span class="hl num">1</span><span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl kwa">goto</span> cleanup<span class="hl opt">;</span>
<span class="hl opt">}</span>
ret <span class="hl opt">=</span> <span class="hl kwd">selabel_lookup</span><span class="hl opt">(</span>hnd<span class="hl opt">, &</span>context<span class="hl opt">,</span> tgt<span class="hl opt">,</span> srcmode<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt">!=</span> <span class="hl num">0</span><span class="hl opt">) {</span>
<span class="hl kwa">goto</span> cleanup<span class="hl opt">;</span>
<span class="hl opt">}</span>
ret <span class="hl opt">=</span> <span class="hl kwd">setfscreatecon</span><span class="hl opt">(</span>context<span class="hl opt">);</span>
cleanup<span class="hl opt">:</span>
<span class="hl kwd">freecon</span><span class="hl opt">(</span>context<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>hnd <span class="hl opt">!=</span> NULL<span class="hl opt">)</span>
<span class="hl kwd">selabel_close</span><span class="hl opt">(</span>hnd<span class="hl opt">);</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
<span class="hl opt">}</span>
</pre></div>
<h2><a href="https://www.kernel.org/doc/Documentation/security/Smack.txt">SMACK</a></h2>
<p>This is another security technology.</p>
<p>Like Selinux it has labels.
These are stored in extended attributes matching <code>security.SMACK64*</code>,
so require root privileges to copy faithfully.</p>
<h2>btrfs flags</h2>
<p>Only worth copying if both source and destination are on btrfs,
but if you then move a file back to btrfs you might want to restore them.</p>
<p>The only flag of real interest is <code>"btrfs.compression"</code>,
which is safe to ignore if moving to a file system which doesn't support it.</p>
<p>A "brain dead" implementation for this and SMACK is to check the prefix,
and silently accept failure if setting the attribute fails.</p>
<div class="highlight-c"><pre class="hl"><span class="hl kwb">static int</span> <span class="hl kwd">copy_xattrs</span><span class="hl opt">(</span><span class="hl kwb">int</span> srcfd<span class="hl opt">,</span> <span class="hl kwb">int</span> tgtfd<span class="hl opt">) {</span>
ssize_t ret<span class="hl opt">;</span>
<span class="hl kwb">char</span> <span class="hl opt">*</span>names <span class="hl opt">=</span> NULL<span class="hl opt">;</span>
<span class="hl kwb">void</span> <span class="hl opt">*</span>value <span class="hl opt">=</span> NULL<span class="hl opt">;</span>
<span class="hl kwb">size_t</span> names_size <span class="hl opt">=</span> <span class="hl num">0</span><span class="hl opt">,</span> value_size <span class="hl opt">=</span> <span class="hl num">0</span><span class="hl opt">;</span>
ret <span class="hl opt">=</span> <span class="hl kwd">xattr_list</span><span class="hl opt">(</span>srcfd<span class="hl opt">, &</span>names<span class="hl opt">, &</span>names_size<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">goto</span> cleanup<span class="hl opt">;</span>
<span class="hl kwa">for</span> <span class="hl opt">(</span><span class="hl kwb">char</span> <span class="hl opt">*</span>name <span class="hl opt">=</span> names<span class="hl opt">;</span> name <span class="hl opt"><</span> names <span class="hl opt">+</span> names_size<span class="hl opt">;</span>
name <span class="hl opt">=</span> <span class="hl kwd">strchrnul</span><span class="hl opt">(</span>name<span class="hl opt">,</span> <span class="hl str">'\0'</span><span class="hl opt">) +</span> <span class="hl num">1</span><span class="hl opt">) {</span>
<span class="hl com">/* Skip xattrs that need special handling */</span>
<span class="hl kwa">if</span> <span class="hl opt">(!</span><span class="hl kwd">str_starts_with</span><span class="hl opt">(</span>name<span class="hl opt">,</span> <span class="hl str">"user."</span><span class="hl opt">) &&</span>
<span class="hl opt">!</span><span class="hl kwd">str_starts_with</span><span class="hl opt">(</span>name<span class="hl opt">,</span> <span class="hl str">"security.SMACK64"</span><span class="hl opt">) &&</span>
<span class="hl opt">!</span><span class="hl kwd">str_starts_with</span><span class="hl opt">(</span>name<span class="hl opt">,</span> <span class="hl str">"btrfs."</span><span class="hl opt">)) {</span>
<span class="hl kwa">continue</span><span class="hl opt">;</span>
<span class="hl opt">}</span>
ret <span class="hl opt">=</span> <span class="hl kwd">xattr_get</span><span class="hl opt">(</span>srcfd<span class="hl opt">,</span> name<span class="hl opt">, &</span>value<span class="hl opt">, &</span>value_size<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">goto</span> cleanup<span class="hl opt">;</span>
ret <span class="hl opt">=</span> <span class="hl kwd">TEMP_FAILURE_RETRY</span><span class="hl opt">(</span><span class="hl kwd">fsetxattr</span><span class="hl opt">(</span>tgtfd<span class="hl opt">,</span> name<span class="hl opt">,</span> value<span class="hl opt">,</span> value_size<span class="hl opt">,</span> <span class="hl num">0</span><span class="hl opt">));</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">) {</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>errno <span class="hl opt">==</span> EINVAL <span class="hl opt">&&</span>
<span class="hl opt">(</span><span class="hl kwd">str_starts_with</span><span class="hl opt">(</span>name<span class="hl opt">,</span> <span class="hl str">"security.SMACK64"</span><span class="hl opt">) ||</span>
<span class="hl kwd">str_starts_with</span><span class="hl opt">(</span>name<span class="hl opt">,</span> <span class="hl str">"btrfs."</span><span class="hl opt">))) {</span>
<span class="hl kwa">continue</span><span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl kwa">goto</span> cleanup<span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl opt">}</span>
cleanup<span class="hl opt">:</span>
<span class="hl kwd">free</span><span class="hl opt">(</span>names<span class="hl opt">);</span>
<span class="hl kwd">free</span><span class="hl opt">(</span>value<span class="hl opt">);</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
<span class="hl opt">}</span>
</pre></div>
<p>As with previous articles, the full version of the
<a href="http://yakking.branchable.com/posts/moving-files-7-difficult-xattrs/my-mv.c">my-mv.c</a> source file and the <a href="http://yakking.branchable.com/posts/moving-files-7-difficult-xattrs/Makefile">Makefile</a>
may be downloaded.</p>
<p>The <a href="http://yakking.branchable.com/posts/moving-files-7-difficult-xattrs/Makefile">Makefile</a> has changed since earlier
since it now needs to link against libselinux.</p>
<h1>So now we've got an equivalent to a slow <a href="http://man7.org/linux/man-pages/man2/rename.2.html">rename(2)</a>, right?</h1>
<p>Not quite, <a href="http://man7.org/linux/man-pages/man2/rename.2.html">rename(2)</a> is atomic.
It disappears from the old location and reappears whole at the new one at the same time.</p>
How difficult is it to preserve extended attributes when moving a file?http://yakking.branchable.com/posts/moving-files-6-extended-attributes/Richard Maw2016-10-05T11:00:16Z2016-10-05T11:00:09Z
<h1>Copying <a href="http://man7.org/linux/man-pages/man5/attr.5.html">extended attributes</a></h1>
<p><a href="http://man7.org/linux/man-pages/man5/attr.5.html">Extended attributes</a> allow you to provide extra metadata for a file.</p>
<p>It's effectively a key-value store using a string for the key,
but values can be arbitrary blobs.</p>
<p><a href="http://man7.org/linux/man-pages/man2/listxattr.2.html">flistxattr(2)</a> to know which xattrs exist.
<a href="http://man7.org/linux/man-pages/man2/getxattr.2.html">fgetxattr(2)</a> to read any xattrs.
<a href="http://man7.org/linux/man-pages/man2/setxattr.2.html">fsetxattr(2)</a> to write an xattr to a file.</p>
<p>Regular users can set <a href="http://man7.org/linux/man-pages/man5/attr.5.html">xattrs</a> beginning <code>user.</code>,
and as far as Linux is concerned that's arbitrary data
that it doesn't need to care what they are.</p>
<p>However there are attributes outside the <code>user.</code> namespace
which have special meaning to Linux,
so we shouldn't try to copy everything as it is.</p>
<div class="highlight-c"><pre class="hl"><span class="hl kwb">static int</span> <span class="hl kwd">realloc_double</span><span class="hl opt">(</span><span class="hl kwb">void</span> <span class="hl opt">**</span>buf<span class="hl opt">,</span> <span class="hl kwb">size_t</span> <span class="hl opt">*</span>size<span class="hl opt">) {</span>
<span class="hl kwb">size_t</span> new_size <span class="hl opt">= *</span>size <span class="hl opt">*</span> <span class="hl num">2</span><span class="hl opt">;</span>
<span class="hl kwb">void</span> <span class="hl opt">*</span>new_buf <span class="hl opt">=</span> <span class="hl kwd">realloc</span><span class="hl opt">(*</span>buf<span class="hl opt">,</span> new_size<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>new_buf <span class="hl opt">==</span> NULL <span class="hl opt">&&</span> new_size <span class="hl opt">!=</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">return</span> <span class="hl opt">-</span><span class="hl num">1</span><span class="hl opt">;</span>
<span class="hl opt">*</span>buf <span class="hl opt">=</span> new_buf<span class="hl opt">;</span>
<span class="hl opt">*</span>size <span class="hl opt">=</span> new_size<span class="hl opt">;</span>
<span class="hl kwa">return</span> <span class="hl num">0</span><span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl kwb">static int</span> <span class="hl kwd">xattr_list</span><span class="hl opt">(</span><span class="hl kwb">int</span> fd<span class="hl opt">,</span> <span class="hl kwb">char</span> <span class="hl opt">**</span>names<span class="hl opt">,</span> <span class="hl kwb">size_t</span> <span class="hl opt">*</span>size<span class="hl opt">) {</span>
ssize_t ret<span class="hl opt">;</span>
<span class="hl kwa">if</span> <span class="hl opt">(*</span>names <span class="hl opt">==</span> NULL <span class="hl opt">&& *</span>size <span class="hl opt">==</span> <span class="hl num">0</span><span class="hl opt">) {</span>
ret <span class="hl opt">=</span> <span class="hl kwd">TEMP_FAILURE_RETRY</span><span class="hl opt">(</span><span class="hl kwd">flistxattr</span><span class="hl opt">(</span>fd<span class="hl opt">,</span> NULL<span class="hl opt">,</span> <span class="hl num">0</span><span class="hl opt">));</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">goto</span> error<span class="hl opt">;</span>
<span class="hl opt">*</span>size <span class="hl opt">=</span> ret<span class="hl opt">;</span>
<span class="hl opt">*</span>names <span class="hl opt">=</span> <span class="hl kwd">malloc</span><span class="hl opt">(*</span>size<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(*</span>names <span class="hl opt">==</span> NULL<span class="hl opt">) {</span>
ret <span class="hl opt">= -</span><span class="hl num">1</span><span class="hl opt">;</span>
<span class="hl kwa">goto</span> error<span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl opt">}</span>
<span class="hl kwa">for</span> <span class="hl opt">(;;) {</span>
ret <span class="hl opt">=</span> <span class="hl kwd">TEMP_FAILURE_RETRY</span><span class="hl opt">(</span><span class="hl kwd">flistxattr</span><span class="hl opt">(</span>fd<span class="hl opt">, *</span>names<span class="hl opt">, *</span>size<span class="hl opt">));</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt">>=</span> <span class="hl num">0</span><span class="hl opt">) {</span>
<span class="hl opt">*</span>size <span class="hl opt">=</span> ret<span class="hl opt">;</span>
<span class="hl kwa">break</span><span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>errno <span class="hl opt">!=</span> ERANGE<span class="hl opt">)</span>
<span class="hl kwa">goto</span> error<span class="hl opt">;</span>
<span class="hl com">/* New xattr added since first flistxattr */</span>
ret <span class="hl opt">=</span> <span class="hl kwd">realloc_double</span><span class="hl opt">((</span><span class="hl kwb">void</span><span class="hl opt">**)</span>names<span class="hl opt">,</span> size<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">goto</span> error<span class="hl opt">;</span>
<span class="hl opt">}</span>
ret <span class="hl opt">=</span> <span class="hl num">0</span><span class="hl opt">;</span>
error<span class="hl opt">:</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl kwb">static int</span> <span class="hl kwd">xattr_get</span><span class="hl opt">(</span><span class="hl kwb">int</span> fd<span class="hl opt">,</span> <span class="hl kwb">const char</span> <span class="hl opt">*</span>name<span class="hl opt">,</span> <span class="hl kwb">void</span> <span class="hl opt">**</span>value<span class="hl opt">,</span> <span class="hl kwb">size_t</span> <span class="hl opt">*</span>size<span class="hl opt">) {</span>
ssize_t ret<span class="hl opt">;</span>
<span class="hl kwa">if</span> <span class="hl opt">(*</span>value <span class="hl opt">==</span> NULL <span class="hl opt">&& *</span>size <span class="hl opt">==</span> <span class="hl num">0</span><span class="hl opt">) {</span>
ret <span class="hl opt">=</span> <span class="hl kwd">TEMP_FAILURE_RETRY</span><span class="hl opt">(</span><span class="hl kwd">fgetxattr</span><span class="hl opt">(</span>fd<span class="hl opt">,</span> name<span class="hl opt">,</span> NULL<span class="hl opt">,</span> <span class="hl num">0</span><span class="hl opt">));</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">goto</span> error<span class="hl opt">;</span>
<span class="hl opt">*</span>size <span class="hl opt">=</span> ret<span class="hl opt">;</span>
<span class="hl opt">*</span>value <span class="hl opt">=</span> <span class="hl kwd">malloc</span><span class="hl opt">(*</span>size<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(*</span>value <span class="hl opt">==</span> NULL<span class="hl opt">) {</span>
ret <span class="hl opt">= -</span><span class="hl num">1</span><span class="hl opt">;</span>
<span class="hl kwa">goto</span> error<span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl opt">}</span>
<span class="hl kwa">for</span> <span class="hl opt">(;;) {</span>
ret <span class="hl opt">=</span> <span class="hl kwd">TEMP_FAILURE_RETRY</span><span class="hl opt">(</span><span class="hl kwd">flistxattr</span><span class="hl opt">(</span>fd<span class="hl opt">, *</span>value<span class="hl opt">, *</span>size<span class="hl opt">));</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt">>=</span> <span class="hl num">0</span><span class="hl opt">) {</span>
<span class="hl opt">*</span>size <span class="hl opt">=</span> ret<span class="hl opt">;</span>
<span class="hl kwa">break</span><span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>errno <span class="hl opt">!=</span> ERANGE<span class="hl opt">)</span>
<span class="hl kwa">goto</span> error<span class="hl opt">;</span>
<span class="hl com">/* xattr grew since first getxattr */</span>
ret <span class="hl opt">=</span> <span class="hl kwd">realloc_double</span><span class="hl opt">(</span>value<span class="hl opt">,</span> size<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">goto</span> error<span class="hl opt">;</span>
<span class="hl opt">}</span>
ret <span class="hl opt">=</span> <span class="hl num">0</span><span class="hl opt">;</span>
error<span class="hl opt">:</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl kwb">static int</span> <span class="hl kwd">str_starts_with</span><span class="hl opt">(</span><span class="hl kwb">const char</span> <span class="hl opt">*</span>s1<span class="hl opt">,</span> <span class="hl kwb">const char</span> <span class="hl opt">*</span>s2<span class="hl opt">) {</span>
<span class="hl kwa">return</span> <span class="hl kwd">strncmp</span><span class="hl opt">(</span>s1<span class="hl opt">,</span> s2<span class="hl opt">,</span> <span class="hl kwd">strlen</span><span class="hl opt">(</span>s2<span class="hl opt">)) ==</span> <span class="hl num">0</span><span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl kwb">static int</span> <span class="hl kwd">copy_xattrs</span><span class="hl opt">(</span><span class="hl kwb">int</span> srcfd<span class="hl opt">,</span> <span class="hl kwb">int</span> tgtfd<span class="hl opt">) {</span>
ssize_t ret<span class="hl opt">;</span>
<span class="hl kwb">char</span> <span class="hl opt">*</span>names <span class="hl opt">=</span> NULL<span class="hl opt">;</span>
<span class="hl kwb">void</span> <span class="hl opt">*</span>value <span class="hl opt">=</span> NULL<span class="hl opt">;</span>
<span class="hl kwb">size_t</span> names_size <span class="hl opt">=</span> <span class="hl num">0</span><span class="hl opt">,</span> value_size <span class="hl opt">=</span> <span class="hl num">0</span><span class="hl opt">;</span>
ret <span class="hl opt">=</span> <span class="hl kwd">xattr_list</span><span class="hl opt">(</span>srcfd<span class="hl opt">, &</span>names<span class="hl opt">, &</span>names_size<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">goto</span> cleanup<span class="hl opt">;</span>
<span class="hl kwa">for</span> <span class="hl opt">(</span><span class="hl kwb">char</span> <span class="hl opt">*</span>name <span class="hl opt">=</span> names<span class="hl opt">;</span> name <span class="hl opt"><</span> names <span class="hl opt">+</span> names_size<span class="hl opt">;</span>
name <span class="hl opt">=</span> <span class="hl kwd">strchrnul</span><span class="hl opt">(</span>name<span class="hl opt">,</span> <span class="hl str">'\0'</span><span class="hl opt">) +</span> <span class="hl num">1</span><span class="hl opt">) {</span>
<span class="hl com">/* Skip xattrs that need special handling */</span>
<span class="hl kwa">if</span> <span class="hl opt">(!</span><span class="hl kwd">str_starts_with</span><span class="hl opt">(</span>name<span class="hl opt">,</span> <span class="hl str">"user."</span><span class="hl opt">)) {</span>
<span class="hl kwa">continue</span><span class="hl opt">;</span>
<span class="hl opt">}</span>
ret <span class="hl opt">=</span> <span class="hl kwd">xattr_get</span><span class="hl opt">(</span>srcfd<span class="hl opt">,</span> name<span class="hl opt">, &</span>value<span class="hl opt">, &</span>value_size<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">goto</span> cleanup<span class="hl opt">;</span>
ret <span class="hl opt">=</span> <span class="hl kwd">TEMP_FAILURE_RETRY</span><span class="hl opt">(</span><span class="hl kwd">fsetxattr</span><span class="hl opt">(</span>tgtfd<span class="hl opt">,</span> name<span class="hl opt">,</span> value<span class="hl opt">,</span> value_size<span class="hl opt">,</span> <span class="hl num">0</span><span class="hl opt">));</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">goto</span> cleanup<span class="hl opt">;</span>
<span class="hl opt">}</span>
cleanup<span class="hl opt">:</span>
<span class="hl kwd">free</span><span class="hl opt">(</span>names<span class="hl opt">);</span>
<span class="hl kwd">free</span><span class="hl opt">(</span>value<span class="hl opt">);</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
<span class="hl opt">}</span>
</pre></div>
<h2>POSIX ACLs</h2>
<p>Feature from a POSIX design specification that wasn't widely adopted,
but Linux supports the draft specification.</p>
<p>Not used much outside of NFS or SAMBA.
You would be forgiven for not knowing they exist.</p>
<p>How they work is beyond the scope of this article,
but <a href="http://man7.org/linux/man-pages/man5/acl.5.html">man7.org</a> covers some details about what it does
and <a href="https://lwn.net/Articles/661357/">lwn.net</a> covers some limitations.</p>
<p>For this article we're not concerned with how to use POSIX ACLs,
it's relevant to us because they work by storing data in a special attribute,
so if we want to preserve this we need to copy <code>system.posix_acl_access</code>.</p>
<p>Since this attribute doesn't start with <code>"user."</code>
we need root privileges to copy it faithfully.</p>
<div class="highlight-c"><pre class="hl"><span class="hl kwb">int</span> <span class="hl kwd">copy_posix_acls</span><span class="hl opt">(</span><span class="hl kwb">int</span> srcfd<span class="hl opt">,</span> <span class="hl kwb">int</span> tgtfd<span class="hl opt">) {</span>
<span class="hl kwb">static const char</span> name<span class="hl opt">[] =</span> <span class="hl str">"system.posix_acl_access"</span><span class="hl opt">;</span>
<span class="hl kwb">int</span> ret <span class="hl opt">=</span> <span class="hl num">0</span><span class="hl opt">;</span>
<span class="hl kwb">void</span> <span class="hl opt">*</span>value <span class="hl opt">=</span> NULL<span class="hl opt">;</span>
<span class="hl kwb">size_t</span> size <span class="hl opt">=</span> <span class="hl num">0</span><span class="hl opt">;</span>
ret <span class="hl opt">=</span> <span class="hl kwd">xattr_get</span><span class="hl opt">(</span>srcfd<span class="hl opt">,</span> name<span class="hl opt">, &</span>value<span class="hl opt">, &</span>size<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">) {</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>errno <span class="hl opt">==</span> ENODATA<span class="hl opt">)</span>
ret <span class="hl opt">=</span> <span class="hl num">0</span><span class="hl opt">;</span>
<span class="hl kwa">goto</span> cleanup<span class="hl opt">;</span>
<span class="hl opt">}</span>
ret <span class="hl opt">=</span> <span class="hl kwd">TEMP_FAILURE_RETRY</span><span class="hl opt">(</span><span class="hl kwd">fsetxattr</span><span class="hl opt">(</span>tgtfd<span class="hl opt">,</span> name<span class="hl opt">,</span> value<span class="hl opt">,</span> size<span class="hl opt">,</span> <span class="hl num">0</span><span class="hl opt">));</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">) {</span>
<span class="hl kwa">goto</span> cleanup<span class="hl opt">;</span>
<span class="hl opt">}</span>
cleanup<span class="hl opt">:</span>
<span class="hl kwd">free</span><span class="hl opt">(</span>value<span class="hl opt">);</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
<span class="hl opt">}</span>
</pre></div>
<p>As with previous articles, the full version of the
<a href="http://yakking.branchable.com/posts/moving-files-6-extended-attributes/my-mv.c">my-mv.c</a> source file and the <a href="http://yakking.branchable.com/posts/moving-files-6-extended-attributes/Makefile">Makefile</a>
may be downloaded.</p>
<h1>Surely that's the last of the metadata!</h1>
<p>I mentioned some extended attributes have special meaning to Linux.
POSIX ACLs can be sanely copied,
but some attributes have tricky semantics that require special handling.</p>
How difficult is it to preserve flags when moving a file?http://yakking.branchable.com/posts/moving-files-5-flags/Richard Maw2016-09-28T11:00:14Z2016-09-28T11:00:06Z
<p><a href="http://yakking.branchable.com/posts/moving-files-4-stat/">Previously</a> we spoke about the common,
POSIX file metadata.</p>
<p>This is not the only metadata that a program
that handles copying files has to worry about on Linux.</p>
<p>Files also have some additional flags for changing their behaviour,
or possibly read-only flags for providing extra information.</p>
<h1>Accessing flags.</h1>
<p>Flags were originally a feature of the <code>ext2</code> filesystem,
which means they don't have a dedicated system call,
since filesystem specific features are often implemented as <a href="http://man7.org/linux/man-pages/man2/ioctl.2.html">ioctls</a>.</p>
<p>It also explains why you might see it called <code>EXT2_IOC_GETFLAGS</code>
or <code>EXT2_IOC_SETFLAGS</code>.</p>
<p>When using ioctls, it's good to be paranoid,
since the same ioctl number can be used for different devices,
and you wouldn't want to accidentally do something unintended.</p>
<p>It's possible to check whether it's an appropriate file
by using stat and checking the file mode.</p>
<p>We previously used this pattern for the file clone ioctl on btrfs,
but included a check that it was a btrfs filesystem.</p>
<p>Since file flags are applicable to multiple filesystems
checking the filesystem type should not be necessary.</p>
<div class="highlight-c"><pre class="hl"><span class="hl ppc">#include <sys/stat.h></span>
<span class="hl ppc">#include <errno.h></span>
<span class="hl ppc">#include <linux/fs.h></span>
<span class="hl ppc">#include <sys/ioctl.h></span>
<span class="hl kwb">int</span> <span class="hl kwd">get_flags</span><span class="hl opt">(</span><span class="hl kwb">int</span> fd<span class="hl opt">,</span> <span class="hl kwb">int</span> <span class="hl opt">*</span>flags_out<span class="hl opt">) {</span>
<span class="hl kwb">struct</span> stat st<span class="hl opt">;</span>
<span class="hl kwb">int</span> ret <span class="hl opt">=</span> <span class="hl num">0</span><span class="hl opt">;</span>
ret <span class="hl opt">=</span> <span class="hl kwd">fstat</span><span class="hl opt">(</span>fd<span class="hl opt">, &</span>st<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
<span class="hl kwa">if</span> <span class="hl opt">(!</span><span class="hl kwd">S_ISREG</span><span class="hl opt">(</span>st<span class="hl opt">.</span>st_mode<span class="hl opt">) && !</span><span class="hl kwd">S_ISDIR</span><span class="hl opt">(</span>st<span class="hl opt">.</span>st_mode<span class="hl opt">)</span>
<span class="hl opt">&& !</span><span class="hl kwd">S_ISLNK</span><span class="hl opt">(</span>st<span class="hl opt">.</span>st_mode<span class="hl opt">)) {</span>
errno <span class="hl opt">=</span> ENOTTY<span class="hl opt">;</span>
<span class="hl kwa">return</span> <span class="hl opt">-</span><span class="hl num">1</span><span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl kwa">return</span> <span class="hl kwd">ioctl</span><span class="hl opt">(</span>fd<span class="hl opt">,</span> FS_IOC_GETFLAGS<span class="hl opt">,</span> flags_out<span class="hl opt">);</span>
<span class="hl opt">}</span>
<span class="hl kwb">int</span> <span class="hl kwd">set_flags</span><span class="hl opt">(</span><span class="hl kwb">int</span> fd<span class="hl opt">,</span> <span class="hl kwb">const int</span> <span class="hl opt">*</span>flags<span class="hl opt">) {</span>
<span class="hl kwb">struct</span> stat st<span class="hl opt">;</span>
<span class="hl kwb">int</span> ret <span class="hl opt">=</span> <span class="hl num">0</span><span class="hl opt">;</span>
ret <span class="hl opt">=</span> <span class="hl kwd">fstat</span><span class="hl opt">(</span>fd<span class="hl opt">, &</span>st<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
<span class="hl kwa">if</span> <span class="hl opt">(!</span><span class="hl kwd">S_ISREG</span><span class="hl opt">(</span>st<span class="hl opt">.</span>st_mode<span class="hl opt">) && !</span><span class="hl kwd">S_ISDIR</span><span class="hl opt">(</span>st<span class="hl opt">.</span>st_mode<span class="hl opt">)</span>
<span class="hl opt">&& !</span><span class="hl kwd">S_ISLNK</span><span class="hl opt">(</span>st<span class="hl opt">.</span>st_mode<span class="hl opt">)) {</span>
errno <span class="hl opt">=</span> ENOTTY<span class="hl opt">;</span>
<span class="hl kwa">return</span> <span class="hl opt">-</span><span class="hl num">1</span><span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl kwa">return</span> <span class="hl kwd">ioctl</span><span class="hl opt">(</span>fd<span class="hl opt">,</span> FS_IOC_SETFLAGS<span class="hl opt">,</span> flags<span class="hl opt">);</span>
<span class="hl opt">}</span>
</pre></div>
<p>As an aside,
I find it odd that the set flags ioctl takes a <code>const int*</code> rather than an <code>int</code>
since I know of no CPU that has shorter pointers than integers.</p>
<h1>Copying flags</h1>
<p>Since filesystems have different capabilities,
they unfortunately accept different sets of flags.</p>
<p><a href="https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/include/uapi/linux/fs.h?id=bc4dee5aa72723632a1f83fd0d3720066c93b433#n286">include/linux/fs.h</a> has definitions for all the flags
which are agreed on by every filesystem,
though they may not support them.</p>
<p>Since you can't trust flags on two filesystems to mean the same thing,
if they are on different filesystems
then you must attempt to only set flags they both agree on.</p>
<p><a href="https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/include/uapi/linux/fs.h?id=bc4dee5aa72723632a1f83fd0d3720066c93b433#n286">include/linux/fs.h</a> defines <code>FS_FL_USER_MODIFIABLE</code> for this.</p>
<p>Because filesystems may not implement every commonly defined flag
and will refuse to set flags if any provided aren't recognised,
you can either define logic for looking up the flags supported
and setting those all at once,
or try setting each flag in-turn
so you can determine whether failing to copy that flag is a problem.</p>
<p>Since the kernel doesn't expose which flags a filesystem supports at runtime
the set of flags your program thinks a filesystem supports can get out of date,
so setting the flags one at a time is the most flexible option.</p>
<p>The code below uses <a href="http://man7.org/linux/man-pages/man3/ffs.3.html">ffs(3)</a> to iterate through the bits set in the integer
since C doesn't have an operator to do it,
but <a href="http://man7.org/linux/man-pages/man3/ffs.3.html">ffs(3)</a> may be a compiler builtin which uses special instructions.</p>
<div class="highlight-c"><pre class="hl"><span class="hl kwb">int</span> <span class="hl kwd">copy_flags</span><span class="hl opt">(</span><span class="hl kwb">int</span> srcfd<span class="hl opt">,</span> <span class="hl kwb">int</span> tgtfd<span class="hl opt">,</span> <span class="hl kwb">int</span> required_flags<span class="hl opt">) {</span>
<span class="hl kwb">int</span> ret<span class="hl opt">;</span>
<span class="hl kwb">int</span> srcflags<span class="hl opt">;</span>
<span class="hl kwb">int</span> tgtflags<span class="hl opt">;</span>
<span class="hl kwb">int</span> newflags<span class="hl opt">;</span>
<span class="hl kwb">struct</span> statfs srcfs<span class="hl opt">,</span> tgtfs<span class="hl opt">;</span>
ret <span class="hl opt">=</span> <span class="hl kwd">get_flags</span><span class="hl opt">(</span>srcfd<span class="hl opt">, &</span>srcflags<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt">!=</span> <span class="hl num">0</span><span class="hl opt">) {</span>
<span class="hl com">/* If we don't support flags we have none to update. */</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>errno <span class="hl opt">==</span> EINVAL <span class="hl opt">||</span> errno <span class="hl opt">==</span> ENOTTY<span class="hl opt">)</span>
<span class="hl kwa">return</span> <span class="hl num">0</span><span class="hl opt">;</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
<span class="hl opt">}</span>
ret <span class="hl opt">=</span> <span class="hl kwd">get_flags</span><span class="hl opt">(</span>tgtfd<span class="hl opt">, &</span>tgtflags<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt">!=</span> <span class="hl num">0</span><span class="hl opt">) {</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>required_flags <span class="hl opt">==</span> <span class="hl num">0</span> <span class="hl opt">&& (</span>errno <span class="hl opt">==</span> EINVAL <span class="hl opt">||</span> errno <span class="hl opt">==</span> ENOTTY<span class="hl opt">))</span>
<span class="hl kwa">return</span> <span class="hl num">0</span><span class="hl opt">;</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
<span class="hl opt">}</span>
ret <span class="hl opt">=</span> <span class="hl kwd">fstatfs</span><span class="hl opt">(</span>srcfd<span class="hl opt">, &</span>srcfs<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt">!=</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
ret <span class="hl opt">=</span> <span class="hl kwd">fstatfs</span><span class="hl opt">(</span>tgtfd<span class="hl opt">, &</span>tgtfs<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt">!=</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
<span class="hl com">/* If on different fs need to mask to commonly agreed flags */</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>srcfs<span class="hl opt">.</span>f_type <span class="hl opt">!=</span> tgtfs<span class="hl opt">.</span>f_type<span class="hl opt">) {</span>
srcflags <span class="hl opt">&=</span> FS_FL_USER_MODIFIABLE<span class="hl opt">;</span>
tgtflags <span class="hl opt">&=</span> FS_FL_USER_MODIFIABLE<span class="hl opt">;</span>
<span class="hl kwa">if</span> <span class="hl opt">((</span>srcflags <span class="hl opt">&</span> required_flags<span class="hl opt">) !=</span> required_flags<span class="hl opt">) {</span>
errno <span class="hl opt">=</span> EINVAL<span class="hl opt">;</span>
<span class="hl kwa">return</span> <span class="hl opt">-</span><span class="hl num">1</span><span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl opt">}</span>
<span class="hl com">/* Skip setting flags if they are the same */</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>srcflags <span class="hl opt">==</span> tgtflags<span class="hl opt">)</span>
<span class="hl kwa">return</span> <span class="hl num">0</span><span class="hl opt">;</span>
<span class="hl com">/* Clear any flags that are set which we want to remove */</span>
newflags <span class="hl opt">=</span> tgtflags <span class="hl opt">&</span> srcflags<span class="hl opt">;</span>
ret <span class="hl opt">=</span> <span class="hl kwd">set_flags</span><span class="hl opt">(</span>tgtfd<span class="hl opt">, &</span>newflags<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt">!=</span> <span class="hl num">0</span><span class="hl opt">) {</span>
<span class="hl com">/* Can't set flags on the target, but we didn't require any. */</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>required_flags <span class="hl opt">==</span> <span class="hl num">0</span> <span class="hl opt">&&</span> errno <span class="hl opt">==</span> EINVAL<span class="hl opt">)</span>
<span class="hl kwa">return</span> <span class="hl num">0</span><span class="hl opt">;</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
<span class="hl opt">}</span>
tgtflags <span class="hl opt">=</span> newflags<span class="hl opt">;</span>
<span class="hl com">/* Use srcflags for flags we want to set,</span>
<span class="hl com"> which are everything not already set. */</span>
srcflags <span class="hl opt">&= ~</span>tgtflags<span class="hl opt">;</span>
<span class="hl kwa">while</span> <span class="hl opt">(</span>srcflags<span class="hl opt">) {</span>
<span class="hl kwb">int</span> flag <span class="hl opt">=</span> <span class="hl num">1</span> <span class="hl opt"><< (</span><span class="hl kwd">ffs</span><span class="hl opt">(</span>srcflags<span class="hl opt">) -</span> <span class="hl num">1</span><span class="hl opt">);</span>
newflags <span class="hl opt">=</span> tgtflags <span class="hl opt">|</span> flag<span class="hl opt">;</span>
ret <span class="hl opt">=</span> <span class="hl kwd">set_flags</span><span class="hl opt">(</span>tgtfd<span class="hl opt">, &</span>newflags<span class="hl opt">);</span>
<span class="hl com">/* Fail if this flag is required and unsettable */</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt">!=</span> <span class="hl num">0</span> <span class="hl opt">&& (</span>flag <span class="hl opt">&</span> required_flags<span class="hl opt">))</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt">==</span> <span class="hl num">0</span><span class="hl opt">)</span>
tgtflags <span class="hl opt">=</span> newflags<span class="hl opt">;</span>
srcflags <span class="hl opt">&= ~</span>flag<span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl kwa">return</span> <span class="hl num">0</span><span class="hl opt">;</span>
<span class="hl opt">}</span>
</pre></div>
<h1>Driving it</h1>
<p>Since copying flags may or may not be a problem you need a way to decide,
and, since that may depend on the context it's being called in,
feedback may be more useful than a heuristic.</p>
<p>A command-line application could ask for confirmation of
whether it's acceptable to not set a flag,
but this is awkward for programs used in batch scripts
and you may have already noticed the above code uses <code>required_flags</code>
so the user can declare which flags they consider essential.</p>
<p>Bitwise or-ing flags together is an acceptable C API,
but if it's from a command-line then that's not manageable.</p>
<p>We could do our own thing and name each flag,
but <a href="http://man7.org/linux/man-pages/man1/chattr.1.html">chattr(1)</a> exists for modifying a file's flags,
and as awkward as it can be to remember the character to flag association,
it's better to imitate something that users would be familiar with.</p>
<div class="highlight-c"><pre class="hl"><span class="hl com">/* Convert a chattr style flags string into flags */</span>
<span class="hl kwb">static int</span> <span class="hl kwd">parse_flags</span><span class="hl opt">(</span><span class="hl kwb">const char</span> <span class="hl opt">*</span>flagstr<span class="hl opt">) {</span>
<span class="hl kwb">static struct</span> flags_char <span class="hl opt">{</span>
<span class="hl kwb">int</span> flag<span class="hl opt">;</span>
<span class="hl kwb">char</span> flagchar<span class="hl opt">;</span>
<span class="hl opt">}</span> flags_chars<span class="hl opt">[] = {</span>
<span class="hl opt">{</span> FS_SECRM_FL<span class="hl opt">,</span> <span class="hl str">'s'</span> <span class="hl opt">},</span>
<span class="hl opt">{</span> FS_UNRM_FL<span class="hl opt">,</span> <span class="hl str">'u'</span> <span class="hl opt">},</span>
<span class="hl opt">{</span> FS_COMPR_FL<span class="hl opt">,</span> <span class="hl str">'c'</span> <span class="hl opt">},</span>
<span class="hl opt">{</span> FS_SYNC_FL<span class="hl opt">,</span> <span class="hl str">'S'</span> <span class="hl opt">},</span>
<span class="hl opt">{</span> FS_IMMUTABLE_FL<span class="hl opt">,</span> <span class="hl str">'i'</span> <span class="hl opt">},</span>
<span class="hl opt">{</span> FS_APPEND_FL<span class="hl opt">,</span> <span class="hl str">'a'</span> <span class="hl opt">},</span>
<span class="hl opt">{</span> FS_NODUMP_FL<span class="hl opt">,</span> <span class="hl str">'d'</span> <span class="hl opt">},</span>
<span class="hl opt">{</span> FS_NOATIME_FL<span class="hl opt">,</span> <span class="hl str">'A'</span> <span class="hl opt">},</span>
<span class="hl opt">{</span> FS_JOURNAL_DATA_FL<span class="hl opt">,</span> <span class="hl str">'j'</span> <span class="hl opt">},</span>
<span class="hl opt">{</span> FS_NOTAIL_FL<span class="hl opt">,</span> <span class="hl str">'t'</span> <span class="hl opt">},</span>
<span class="hl opt">{</span> FS_DIRSYNC_FL<span class="hl opt">,</span> <span class="hl str">'D'</span> <span class="hl opt">},</span>
<span class="hl opt">{</span> FS_TOPDIR_FL<span class="hl opt">,</span> <span class="hl str">'T'</span> <span class="hl opt">},</span>
<span class="hl ppc">#ifdef FS_EXTENTS_FL</span>
<span class="hl opt">{</span> FS_EXTENTS_FL<span class="hl opt">,</span> <span class="hl str">'e'</span><span class="hl opt">},</span>
<span class="hl ppc">#endif</span>
<span class="hl opt">{</span> FS_NOCOW_FL<span class="hl opt">,</span> <span class="hl str">'C'</span> <span class="hl opt">},</span>
<span class="hl ppc">#ifdef FS_PROJINHERIT_FL</span>
<span class="hl opt">{</span> FS_PROJINHERIT_FL<span class="hl opt">,</span> <span class="hl str">'P'</span> <span class="hl opt">},</span>
<span class="hl ppc">#endif</span>
<span class="hl opt">};</span>
<span class="hl kwb">int</span> flags <span class="hl opt">=</span> <span class="hl num">0</span><span class="hl opt">;</span>
<span class="hl kwa">for</span> <span class="hl opt">(</span><span class="hl kwb">int</span> i <span class="hl opt">=</span> <span class="hl num">0</span><span class="hl opt">;</span> i <span class="hl opt">< (</span><span class="hl kwa">sizeof</span> flags_chars <span class="hl opt">/</span> <span class="hl kwa">sizeof</span> <span class="hl opt">*</span>flags_chars<span class="hl opt">);</span> i<span class="hl opt">++) {</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span><span class="hl kwd">strchr</span><span class="hl opt">(</span>flagstr<span class="hl opt">,</span> flags_chars<span class="hl opt">[</span>i<span class="hl opt">].</span>flagchar<span class="hl opt">))</span>
flags <span class="hl opt">|=</span> flags_chars<span class="hl opt">[</span>i<span class="hl opt">].</span>flag<span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl kwa">return</span> flags<span class="hl opt">;</span>
<span class="hl opt">}</span>
</pre></div>
<p>As with previous articles, the full version of the
<a href="http://yakking.branchable.com/posts/moving-files-5-flags/my-mv.c">my-mv.c</a> source file and the <a href="http://yakking.branchable.com/posts/moving-files-5-flags/Makefile">Makefile</a>
may be downloaded.</p>
<h1>Conclusion</h1>
<p>Well that was more work than I expected,
but we've copied all the metadata now right?</p>
<p>Well, no. It turns out 32 possible flags isn't enough.</p>
<p>Flags are compact and relatively easy to set,
but 32 booleans is just not enough for everything you need,
especially when some get reserved for other filesystems
or are read-only.</p>
<p>A solution to this would be to just add more ioctls,
but that would lead to the same problems
with needing to know how to translate them between different filesystems.</p>
<p>What's needed is a unified API for setting this extra information,
theseā¦ <a href="http://man7.org/linux/man-pages/man5/attr.5.html">extended attributes</a>. We'll cover these next time.</p>
How difficult is it to preserve metadata when moving a file?http://yakking.branchable.com/posts/moving-files-4-stat/Richard Maw2016-08-31T11:00:14Z2016-08-31T11:00:07Z
<h1>So we've copied everything from the file now right?</h1>
<p>Well, we've copied all the data,
and depending on your application that might be enough,
but files also have metadata to worry about.</p>
<p>We can see this by example,
by checking the output of <code>ls -l</code>,
before and after moving the file to a different filesystem.</p>
<div class="highlight-sh"><pre class="hl">$ <span class="hl kwc">touch</span> <span class="hl opt">/</span>run<span class="hl opt">/</span>user<span class="hl opt">/</span>$<span class="hl opt">(</span>id <span class="hl kwb">-u</span><span class="hl opt">)/</span>testfile
$ <span class="hl kwc">ls</span> <span class="hl kwb">-l</span> <span class="hl opt">/</span>run<span class="hl opt">/</span>user<span class="hl opt">/</span>$<span class="hl opt">(</span>id <span class="hl kwb">-u</span><span class="hl opt">)/</span>testfile
<span class="hl kwb">-rw-rw-r--</span> <span class="hl num">1</span> richardmaw richardmaw <span class="hl num">0</span> Aug <span class="hl num">8 19</span><span class="hl opt">:</span><span class="hl num">40</span> <span class="hl opt">/</span>run<span class="hl opt">/</span>user<span class="hl opt">/</span><span class="hl num">1000</span><span class="hl opt">/</span>testfile
$ .<span class="hl opt">/</span>my-mv <span class="hl opt">/</span>run<span class="hl opt">/</span>user<span class="hl opt">/</span>$<span class="hl opt">(</span>id <span class="hl kwb">-u</span><span class="hl opt">)/</span>testfile testfile
$ <span class="hl kwc">ls</span> <span class="hl kwb">-l</span> testfile
<span class="hl kwb">-rw-------</span> <span class="hl num">1</span> richardmaw richardmaw <span class="hl num">0</span> Aug <span class="hl num">8 19</span><span class="hl opt">:</span><span class="hl num">41</span> testfile
</pre></div>
<p>You should be able to see that the <code>-rw-rw-r--</code> mode string,
which represents readable for everyone and writable for the user and group,
has become <code>-rw-------</code>, which represents read and write for the user only.</p>
<p>This is because <a href="http://man7.org/linux/man-pages/man1/ls.1.html">ls(1)</a> uses <a href="http://man7.org/linux/man-pages/man2/stat.2.html">stat(2)</a>,
which is returning different data for the file.</p>
<h2>Setting mode</h2>
<p><a href="http://man7.org/linux/man-pages/man2/stat.2.html">stat(2)</a> provided the mode of the file,
in the <code>st_mode</code> field.</p>
<p><a href="http://man7.org/linux/man-pages/man2/chmod.2.html">chmod(2)</a> can be used set mode of the new file.</p>
<p>The result of <a href="http://man7.org/linux/man-pages/man2/stat.2.html">stat(2)</a> isn't exactly the same format as <a href="http://man7.org/linux/man-pages/man2/chmod.2.html">chmod(2)</a> takes,
since in the <a href="http://man7.org/linux/man-pages/man2/stat.2.html">stat(2)</a> field it includes bits saying what type of file it is,
but <a href="http://man7.org/linux/man-pages/man2/chmod.2.html">chmod(2)</a> can't change what type a file is,
so is only interested in the portion of the mode that is the permission bits.</p>
<div class="highlight-c"><pre class="hl"><span class="hl kwb">int</span> <span class="hl kwd">copy_contents</span><span class="hl opt">(</span><span class="hl kwb">int</span> srcfd<span class="hl opt">,</span> <span class="hl kwb">int</span> tgtfd<span class="hl opt">) {</span>
<span class="hl kwb">int</span> ret <span class="hl opt">= -</span><span class="hl num">1</span><span class="hl opt">;</span>
ret <span class="hl opt">=</span> <span class="hl kwd">btrfs_clone_contents</span><span class="hl opt">(</span>srcfd<span class="hl opt">,</span> tgtfd<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt">>=</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span> <span class="hl opt">&&</span> errno <span class="hl opt">!=</span> EINVAL<span class="hl opt">) {</span>
<span class="hl com">/* Some error that wasn't from a btrfs clone,</span>
<span class="hl com"> so we can't fall back to something that would work */</span>
<span class="hl kwd">perror</span><span class="hl opt">(</span><span class="hl str">"Copy file"</span><span class="hl opt">);</span>
<span class="hl kwa">return</span> <span class="hl opt">-</span><span class="hl num">1</span><span class="hl opt">;</span>
<span class="hl opt">}</span>
ret <span class="hl opt">=</span> <span class="hl kwd">sparse_copy_contents</span><span class="hl opt">(</span>srcfd<span class="hl opt">,</span> tgtfd<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt">>=</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span> <span class="hl opt">&&</span> errno <span class="hl opt">!=</span> EINVAL<span class="hl opt">) {</span>
<span class="hl com">/* Some error that wasn't from a sparse copy,</span>
<span class="hl com"> so we can't fall back to something that would work */</span>
<span class="hl kwd">perror</span><span class="hl opt">(</span><span class="hl str">"Copy file"</span><span class="hl opt">);</span>
<span class="hl kwa">return</span> <span class="hl opt">-</span><span class="hl num">1</span><span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl kwa">return</span> <span class="hl kwd">naive_contents_copy</span><span class="hl opt">(</span>srcfd<span class="hl opt">,</span> tgtfd<span class="hl opt">);</span>
<span class="hl opt">}</span>
<span class="hl kwb">int</span> <span class="hl kwd">copy_file</span><span class="hl opt">(</span><span class="hl kwb">char</span> <span class="hl opt">*</span>source<span class="hl opt">,</span> <span class="hl kwb">char</span> <span class="hl opt">*</span>target<span class="hl opt">,</span> <span class="hl kwb">bool</span> no_clobber<span class="hl opt">) {</span>
<span class="hl kwb">int</span> srcfd <span class="hl opt">= -</span><span class="hl num">1</span><span class="hl opt">;</span>
<span class="hl kwb">int</span> tgtfd <span class="hl opt">= -</span><span class="hl num">1</span><span class="hl opt">;</span>
<span class="hl kwb">int</span> ret <span class="hl opt">= -</span><span class="hl num">1</span><span class="hl opt">;</span>
<span class="hl kwb">struct</span> stat source_stat<span class="hl opt">;</span>
ret <span class="hl opt">=</span> <span class="hl kwd">open</span><span class="hl opt">(</span>source<span class="hl opt">,</span> O_RDONLY<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt">== -</span><span class="hl num">1</span><span class="hl opt">) {</span>
<span class="hl kwd">perror</span><span class="hl opt">(</span><span class="hl str">"Open source file"</span><span class="hl opt">);</span>
<span class="hl kwa">goto</span> cleanup<span class="hl opt">;</span>
<span class="hl opt">}</span>
srcfd <span class="hl opt">=</span> ret<span class="hl opt">;</span>
ret <span class="hl opt">=</span> <span class="hl kwd">open</span><span class="hl opt">(</span>target<span class="hl opt">,</span> O_WRONLY<span class="hl opt">|</span>O_CREAT<span class="hl opt">|(</span>no_clobber <span class="hl opt">?</span> O_EXCL <span class="hl opt">:</span> <span class="hl num">0</span><span class="hl opt">),</span> <span class="hl num">0600</span><span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt">== -</span><span class="hl num">1</span><span class="hl opt">) {</span>
<span class="hl kwd">perror</span><span class="hl opt">(</span><span class="hl str">"Open target file"</span><span class="hl opt">);</span>
<span class="hl kwa">goto</span> cleanup<span class="hl opt">;</span>
<span class="hl opt">}</span>
tgtfd <span class="hl opt">=</span> ret<span class="hl opt">;</span>
ret <span class="hl opt">=</span> <span class="hl kwd">copy_contents</span><span class="hl opt">(</span>srcfd<span class="hl opt">,</span> tgtfd<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">goto</span> cleanup<span class="hl opt">;</span>
ret <span class="hl opt">=</span> <span class="hl kwd">fstat</span><span class="hl opt">(</span>srcfd<span class="hl opt">, &</span>source_stat<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">goto</span> cleanup<span class="hl opt">;</span>
ret <span class="hl opt">=</span> <span class="hl kwd">fchmod</span><span class="hl opt">(</span>tgtfd<span class="hl opt">,</span> source_stat<span class="hl opt">.</span>st_mode<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">goto</span> cleanup<span class="hl opt">;</span>
cleanup<span class="hl opt">:</span>
<span class="hl kwd">close</span><span class="hl opt">(</span>srcfd<span class="hl opt">);</span>
<span class="hl kwd">close</span><span class="hl opt">(</span>tgtfd<span class="hl opt">);</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
<span class="hl opt">}</span>
</pre></div>
<h2>User and Group</h2>
<p>User and Group are numeric IDs
that <a href="http://man7.org/linux/man-pages/man1/ls.1.html">ls(1)</a> looks up in <code>/etc/passwd</code> and <code>/etc/group</code>
to turn into a human readable name.</p>
<p>The <a href="http://man7.org/linux/man-pages/man1/chown.1.html">chown(1)</a> and <a href="http://man7.org/linux/man-pages/man1/chgrp.1.html">chgrp(1)</a> take a name,
but the <a href="http://man7.org/linux/man-pages/man2/chown.2.html">chown(2)</a> system call does both using the numeric ID.</p>
<p>The user and group can be found in the <a href="http://man7.org/linux/man-pages/man2/stat.2.html">stat(2)</a> result
in the <code>st_uid</code> and <code>st_gid</code> fields.</p>
<h3>setgid bits</h3>
<p>If the <code>setgid</code> bit is set then newly created files
have the group of the directory rather than the user that created them,
but if files are moved in, then they have the group they had before.</p>
<p>Depending on your application, it may make more sense to inherit the group
or to preserve it from the original file.</p>
<div class="highlight-c"><pre class="hl"><span class="hl kwb">enum</span> setgid <span class="hl opt">{</span>
SETGID_AUTO<span class="hl opt">,</span>
SETGID_NEVER<span class="hl opt">,</span>
SETGID_ALWAYS<span class="hl opt">,</span>
<span class="hl opt">};</span>
<span class="hl kwb">static int</span> <span class="hl kwd">fix_owner</span><span class="hl opt">(</span><span class="hl kwb">char</span> <span class="hl opt">*</span>target<span class="hl opt">,</span> <span class="hl kwb">struct</span> stat <span class="hl opt">*</span>source_stat<span class="hl opt">,</span> <span class="hl kwb">enum</span> setgid setgid<span class="hl opt">,</span> <span class="hl kwb">int</span> tgtfd<span class="hl opt">) {</span>
<span class="hl kwb">struct</span> stat target_stat<span class="hl opt">;</span>
<span class="hl kwb">struct</span> stat dirname_stat<span class="hl opt">;</span>
<span class="hl kwb">char</span> <span class="hl opt">*</span>target_dirname<span class="hl opt">;</span>
<span class="hl kwb">int</span> ret <span class="hl opt">=</span> <span class="hl num">0</span><span class="hl opt">;</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>setgid <span class="hl opt">==</span> SETGID_NEVER<span class="hl opt">)</span>
<span class="hl kwa">return</span> <span class="hl kwd">fchown</span><span class="hl opt">(</span>tgtfd<span class="hl opt">,</span> source_stat<span class="hl opt">-></span>st_uid<span class="hl opt">,</span> source_stat<span class="hl opt">-></span>st_gid<span class="hl opt">);</span>
ret <span class="hl opt">=</span> <span class="hl kwd">fstat</span><span class="hl opt">(</span>tgtfd<span class="hl opt">, &</span>target_stat<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">) {</span>
<span class="hl kwd">perror</span><span class="hl opt">(</span><span class="hl str">"Stat target file"</span><span class="hl opt">);</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
<span class="hl opt">}</span>
target_dirname <span class="hl opt">=</span> <span class="hl kwd">dirname</span><span class="hl opt">(</span>target<span class="hl opt">);</span>
ret <span class="hl opt">=</span> <span class="hl kwd">stat</span><span class="hl opt">(</span>target_dirname<span class="hl opt">, &</span>dirname_stat<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">) {</span>
<span class="hl kwd">perror</span><span class="hl opt">(</span><span class="hl str">"Stat target directory"</span><span class="hl opt">);</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl kwa">if</span> <span class="hl opt">((</span>setgid <span class="hl opt">==</span> SETGID_ALWAYS
<span class="hl opt">|| (</span>setgid <span class="hl opt">==</span> SETGID_AUTO <span class="hl opt">&&</span> dirname_stat<span class="hl opt">.</span>st_gid <span class="hl opt">&</span> S_ISGID<span class="hl opt">))</span>
<span class="hl opt">&&</span> target_stat<span class="hl opt">.</span>st_gid <span class="hl opt">!=</span> dirname_stat<span class="hl opt">.</span>st_gid<span class="hl opt">) {</span>
ret <span class="hl opt">=</span> <span class="hl kwd">fchown</span><span class="hl opt">(</span>tgtfd<span class="hl opt">,</span> target_stat<span class="hl opt">.</span>st_uid<span class="hl opt">,</span> dirname_stat<span class="hl opt">.</span>st_gid<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwd">perror</span><span class="hl opt">(</span><span class="hl str">"Chown target"</span><span class="hl opt">);</span>
<span class="hl opt">}</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl kwb">static int</span> <span class="hl kwd">fix_rename_owner</span><span class="hl opt">(</span><span class="hl kwb">char</span> <span class="hl opt">*</span>target<span class="hl opt">,</span> <span class="hl kwb">struct</span> stat <span class="hl opt">*</span>source_stat<span class="hl opt">,</span> <span class="hl kwb">enum</span> setgid setgid<span class="hl opt">) {</span>
<span class="hl kwb">int</span> tgtfd <span class="hl opt">= -</span><span class="hl num">1</span><span class="hl opt">;</span>
<span class="hl kwb">int</span> ret <span class="hl opt">= -</span><span class="hl num">1</span><span class="hl opt">;</span>
ret <span class="hl opt">=</span> <span class="hl kwd">open</span><span class="hl opt">(</span>target<span class="hl opt">,</span> O_RDWR<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt">== -</span><span class="hl num">1</span><span class="hl opt">) {</span>
<span class="hl kwd">perror</span><span class="hl opt">(</span><span class="hl str">"Open target file"</span><span class="hl opt">);</span>
<span class="hl kwa">goto</span> cleanup<span class="hl opt">;</span>
<span class="hl opt">}</span>
tgtfd <span class="hl opt">=</span> ret<span class="hl opt">;</span>
ret <span class="hl opt">=</span> <span class="hl kwd">fix_owner</span><span class="hl opt">(</span>target<span class="hl opt">,</span> source_stat<span class="hl opt">,</span> setgid<span class="hl opt">,</span> tgtfd<span class="hl opt">);</span>
cleanup<span class="hl opt">:</span>
<span class="hl kwd">close</span><span class="hl opt">(</span>tgtfd<span class="hl opt">);</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl kwb">int</span> <span class="hl kwd">move_file</span><span class="hl opt">(</span><span class="hl kwb">char</span> <span class="hl opt">*</span>source<span class="hl opt">,</span> <span class="hl kwb">char</span> <span class="hl opt">*</span>target<span class="hl opt">,</span> <span class="hl kwb">bool</span> no_clobber<span class="hl opt">,</span> <span class="hl kwb">enum</span> setgid setgid<span class="hl opt">) {</span>
<span class="hl kwb">int</span> ret<span class="hl opt">;</span>
<span class="hl kwb">struct</span> stat source_stat<span class="hl opt">;</span>
<span class="hl kwb">bool</span> have_source_stat <span class="hl opt">=</span> <span class="hl kwa">false</span><span class="hl opt">;</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>setgid <span class="hl opt">==</span> SETGID_NEVER<span class="hl opt">) {</span>
ret <span class="hl opt">=</span> <span class="hl kwd">stat</span><span class="hl opt">(</span>source<span class="hl opt">, &</span>source_stat<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
have_source_stat <span class="hl opt">=</span> <span class="hl kwa">true</span><span class="hl opt">;</span>
<span class="hl opt">}</span>
ret <span class="hl opt">=</span> <span class="hl kwd">renameat2</span><span class="hl opt">(</span>AT_FDCWD<span class="hl opt">,</span> source<span class="hl opt">,</span> AT_FDCWD<span class="hl opt">,</span> target<span class="hl opt">,</span> no_clobber <span class="hl opt">?</span> RENAME_NOREPLACE <span class="hl opt">:</span> <span class="hl num">0</span><span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt">==</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">return</span> <span class="hl kwd">fix_rename_owner</span><span class="hl opt">(</span>target<span class="hl opt">, &</span>source_stat<span class="hl opt">,</span> setgid<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>errno <span class="hl opt">==</span> EXDEV<span class="hl opt">)</span>
<span class="hl kwa">goto</span> xdev<span class="hl opt">;</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>errno <span class="hl opt">!=</span> ENOSYS<span class="hl opt">) {</span>
<span class="hl kwd">perror</span><span class="hl opt">(</span><span class="hl str">"rename2"</span><span class="hl opt">);</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl com">/* Have to skip to copy if unimplemented since rename can't detect EEXIST */</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>no_clobber<span class="hl opt">)</span>
<span class="hl kwa">goto</span> xdev<span class="hl opt">;</span>
rename<span class="hl opt">:</span>
ret <span class="hl opt">=</span> <span class="hl kwd">rename</span><span class="hl opt">(</span>source<span class="hl opt">,</span> target<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt">==</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">return</span> <span class="hl kwd">fix_rename_owner</span><span class="hl opt">(</span>target<span class="hl opt">, &</span>source_stat<span class="hl opt">,</span> setgid<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>errno <span class="hl opt">==</span> EXDEV<span class="hl opt">)</span>
<span class="hl kwa">goto</span> xdev<span class="hl opt">;</span>
<span class="hl kwd">perror</span><span class="hl opt">(</span><span class="hl str">"rename"</span><span class="hl opt">);</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
xdev<span class="hl opt">:</span>
<span class="hl kwa">if</span> <span class="hl opt">(!</span>have_source_stat<span class="hl opt">) {</span>
ret <span class="hl opt">=</span> <span class="hl kwd">stat</span><span class="hl opt">(</span>source<span class="hl opt">, &</span>source_stat<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
<span class="hl opt">}</span>
ret <span class="hl opt">=</span> <span class="hl kwd">copy_file</span><span class="hl opt">(</span>source<span class="hl opt">,</span> target<span class="hl opt">, &</span>source_stat<span class="hl opt">,</span> no_clobber<span class="hl opt">,</span> setgid<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt">!=</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
ret <span class="hl opt">=</span> <span class="hl kwd">unlink</span><span class="hl opt">(</span>source<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwd">perror</span><span class="hl opt">(</span><span class="hl str">"unlink"</span><span class="hl opt">);</span>
<span class="hl opt">}</span>
</pre></div>
<h2>Modification time</h2>
<p>mtime and <a href="https://en.wikipedia.org/wiki/Stat_(system_call)#Criticism_of_atime">atime</a> are the "last modification time" and "last access time".</p>
<p>This has classically been set with the <a href="http://man7.org/linux/man-pages/man2/utimes.2.html">utimes(2)</a> system call,
but this does not support nanosecond precision,
so the <a href="http://man7.org/linux/man-pages/man2/futimens.2.html">futimens(2)</a> system call is used.</p>
<p>This takes a pair of <code>struct timespec</code>s,
and the times from the <a href="http://man7.org/linux/man-pages/man2/stat.2.html">stat(2)</a> result
can be retrieved in <code>struct timespec</code> format
in the <code>st_atim</code> and <code>st_mtim</code> fields.</p>
<div class="highlight-c"><pre class="hl"><span class="hl kwb">int</span> <span class="hl kwd">copy_file</span><span class="hl opt">(</span><span class="hl kwb">char</span> <span class="hl opt">*</span>source<span class="hl opt">,</span> <span class="hl kwb">char</span> <span class="hl opt">*</span>target<span class="hl opt">,</span> <span class="hl kwb">struct</span> stat <span class="hl opt">*</span>source_stat<span class="hl opt">,</span> <span class="hl kwb">bool</span> no_clobber<span class="hl opt">,</span> <span class="hl kwb">enum</span> setgid setgid<span class="hl opt">) {</span>
<span class="hl kwb">int</span> srcfd <span class="hl opt">= -</span><span class="hl num">1</span><span class="hl opt">;</span>
<span class="hl kwb">int</span> tgtfd <span class="hl opt">= -</span><span class="hl num">1</span><span class="hl opt">;</span>
<span class="hl kwb">int</span> ret <span class="hl opt">= -</span><span class="hl num">1</span><span class="hl opt">;</span>
ret <span class="hl opt">=</span> <span class="hl kwd">open</span><span class="hl opt">(</span>source<span class="hl opt">,</span> O_RDONLY<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt">== -</span><span class="hl num">1</span><span class="hl opt">) {</span>
<span class="hl kwd">perror</span><span class="hl opt">(</span><span class="hl str">"Open source file"</span><span class="hl opt">);</span>
<span class="hl kwa">goto</span> cleanup<span class="hl opt">;</span>
<span class="hl opt">}</span>
srcfd <span class="hl opt">=</span> ret<span class="hl opt">;</span>
ret <span class="hl opt">=</span> <span class="hl kwd">open</span><span class="hl opt">(</span>target<span class="hl opt">,</span> O_WRONLY<span class="hl opt">|</span>O_CREAT<span class="hl opt">|(</span>no_clobber <span class="hl opt">?</span> O_EXCL <span class="hl opt">:</span> <span class="hl num">0</span><span class="hl opt">),</span> <span class="hl num">0600</span><span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt">== -</span><span class="hl num">1</span><span class="hl opt">) {</span>
<span class="hl kwd">perror</span><span class="hl opt">(</span><span class="hl str">"Open target file"</span><span class="hl opt">);</span>
<span class="hl kwa">goto</span> cleanup<span class="hl opt">;</span>
<span class="hl opt">}</span>
tgtfd <span class="hl opt">=</span> ret<span class="hl opt">;</span>
ret <span class="hl opt">=</span> <span class="hl kwd">copy_contents</span><span class="hl opt">(</span>srcfd<span class="hl opt">,</span> tgtfd<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">goto</span> cleanup<span class="hl opt">;</span>
ret <span class="hl opt">=</span> <span class="hl kwd">fchmod</span><span class="hl opt">(</span>tgtfd<span class="hl opt">,</span> source_stat<span class="hl opt">-></span>st_mode<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">goto</span> cleanup<span class="hl opt">;</span>
ret <span class="hl opt">=</span> <span class="hl kwd">fix_owner</span><span class="hl opt">(</span>target<span class="hl opt">,</span> source_stat<span class="hl opt">,</span> setgid<span class="hl opt">,</span> tgtfd<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">goto</span> cleanup<span class="hl opt">;</span>
<span class="hl opt">{</span>
<span class="hl kwb">struct</span> timespec times<span class="hl opt">[] = {</span> source_stat<span class="hl opt">-></span>st_atim<span class="hl opt">,</span> source_stat<span class="hl opt">-></span>st_mtim<span class="hl opt">, };</span>
ret <span class="hl opt">=</span> <span class="hl kwd">futimens</span><span class="hl opt">(</span>tgtfd<span class="hl opt">,</span> times<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">goto</span> cleanup<span class="hl opt">;</span>
<span class="hl opt">}</span>
cleanup<span class="hl opt">:</span>
<span class="hl kwd">close</span><span class="hl opt">(</span>srcfd<span class="hl opt">);</span>
<span class="hl kwd">close</span><span class="hl opt">(</span>tgtfd<span class="hl opt">);</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
<span class="hl opt">}</span>
</pre></div>
<p>For convenience of testing,
the full <a href="http://yakking.branchable.com/posts/moving-files-4-stat/my-mv.c">my-mv.c</a> source file and <a href="http://yakking.branchable.com/posts/moving-files-4-stat/Makefile">Makefile</a>,
including the new copy functions,
can be downloaded.</p>
<h2>Unfixable data</h2>
<h3>Link count</h3>
<p>The stat data returns how many other directory entries point to the same file
in the <code>st_nlink</code> field.</p>
<p>We could only copy this correctly by making the same number of links,
but this is unlikely to matter, and can't be fixed,
unless we're copying a whole directory tree.</p>
<h3>Creation/change time</h3>
<p>There's another time in the <a href="http://man7.org/linux/man-pages/man2/stat.2.html">stat(2)</a> result, <a href="https://en.wikipedia.org/wiki/Stat_(system_call)#ctime">ctime</a>.
This is an unchangeable last changed time.
It can only be set to an approximate value,
by changing the system clock and modifying the file.</p>
<p>This is not worth the effort,
as it requires elevated privileges and can cause problems for other programs.</p>
<h3>Device and inode</h3>
<p>There's two other fields called <code>st_dev</code> and <code>st_ino</code>,
which identify which filesystem and file on that filesystem the file is.</p>
<p>It doesn't tell you much, other than whether the file is the same as another,
which can be used to detect whether you would accidentally trash a file
if you were to copy the contents of one file into another,
or in the case of <a href="http://man7.org/linux/man-pages/man1/tar.1.html">tar(1)</a>, whether a file were replaced
in between it being created and its metadata being updated.</p>
<p><code>st_dev</code> on its own has also classically been used
to determine whether two files are on the same filesystem,
but btrfs can provide different <code>st_dev</code> values for file on the same filesystem,
but in different subvolumes,
and bind-mounts may have the same <code>st_dev</code> for logically different mounts.</p>
<p>The btrfs weirdness can be solved by using <a href="http://man7.org/linux/man-pages/man2/statfs.2.html">statfs(2)</a>
to determine whether the files are on btrfs,
and using <code>BTRFS_IOC_FS_INFO</code> and <code>BTRFS_IOC_DEV_INFO</code>
to find out which block device the filesystem was mounted from
then <a href="http://man7.org/linux/man-pages/man2/stat.2.html">stat(2)</a> on the device node to find its <code>st_dev</code>.</p>
<p>The bind-mounts can be solved by getting the mount ID,
either using <a href="http://man7.org/linux/man-pages/man2/name_to_handle_at.2.html">name_to_handle_at(2)</a>,
or opening the file and reading <code>/proc/self/fdinfo/$fd</code>
to read the <code>mnt_id</code> field,
and comparing the <code>mnd_id</code> of the two files.</p>
<h1>So we've made stat as similar as we can, that's all the metadata right?</h1>
<p>Not quite, there's some less common metadata to apply.</p>
How difficult is it to move a file quickly?http://yakking.branchable.com/posts/moving-files-3-faster/Richard Maw2016-08-17T11:00:13Z2016-08-17T11:00:06Z
<p>We previously made our file copy sparse-aware,
so it only copies the data rather than the holes between the data,
which as well as being more correct
is also a lot faster for files which happen to be sparse.</p>
<p>Files which are sparse also tend to be large,
since they are usually some form of disk image,
so while we are only copying the data there can still be a lot to copy
so it would be convenient if we could to this more quickly.</p>
<h1><a href="http://man7.org/linux/man-pages/man2/rename.2.html">rename(2)</a> was fast, can we make copying the data faster?</h1>
<p>That depends.</p>
<p><a href="http://man7.org/linux/man-pages/man2/rename.2.html">rename(2)</a> and <a href="http://man7.org/linux/man-pages/man2/link.2.html">link(2)</a> get to be fast because they don't copy any data,
they just change the directory metadata while keeping the data the same.</p>
<p>Some filesystems have an extra level of indirection
which lets multiple files share the same data though.</p>
<h2>Cloning files with btrfs</h2>
<p><a href="https://btrfs.wiki.kernel.org/index.php/Main_Page">btrfs</a> and <a href="https://en.wikipedia.org/wiki/ZFS">ZFS</a> are filesystems which support multiple files sharing data.</p>
<p>Because of <a href="https://en.wikipedia.org/wiki/ZFS">ZFS</a>' <a href="https://sfconservancy.org/blog/2016/feb/25/zfs-and-linux/">interesting</a> legal position
I am more familiar with how <a href="https://btrfs.wiki.kernel.org/index.php/Main_Page">btrfs</a> operates
so I'm not going to talk about <a href="https://en.wikipedia.org/wiki/ZFS">ZFS</a>.</p>
<p>If you want to copy a file between two btrfs file systems
that happen to be stored on the same physical hard disks
then you can "clone" the file's contents into the new file
which is nearly as fast as a <a href="http://man7.org/linux/man-pages/man2/rename.2.html">rename(2)</a>.</p>
<div class="highlight-c"><pre class="hl"><span class="hl ppc">#include <linux/btrfs.h></span> <span class="hl com">/* BTRFS_IOC_CLONE */</span><span class="hl ppc"></span>
<span class="hl ppc">#include <sys/vfs.h></span> <span class="hl com">/* ftatfs, struct statfs */</span><span class="hl ppc"></span>
<span class="hl ppc">#include <sys/stat.h></span> <span class="hl com">/* struct stat */</span><span class="hl ppc"></span>
<span class="hl ppc">#include <sys/ioctl.h></span> <span class="hl com">/* ioctl */</span><span class="hl ppc"></span>
<span class="hl ppc">#include <linux/magic.h></span> <span class="hl com">/* BTRFS_SUPER_MAGIC */</span><span class="hl ppc"></span>
<span class="hl kwb">int</span> <span class="hl kwd">btrfs_clone_contents</span><span class="hl opt">(</span><span class="hl kwb">int</span> srcfd<span class="hl opt">,</span> <span class="hl kwb">int</span> tgtfd<span class="hl opt">) {</span>
<span class="hl kwb">struct</span> statfs stfs<span class="hl opt">;</span>
<span class="hl kwb">struct</span> stat st<span class="hl opt">;</span>
<span class="hl kwb">int</span> ret<span class="hl opt">;</span>
<span class="hl com">/* Behaviour is undefined unless called on a btrfs file,</span>
<span class="hl com"> so ensure we're calling on the right file first. */</span>
ret <span class="hl opt">=</span> <span class="hl kwd">fstatfs</span><span class="hl opt">(</span>tgtfd<span class="hl opt">, &</span>stfs<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>stfs<span class="hl opt">.</span>f_type <span class="hl opt">!=</span> BTRFS_SUPER_MAGIC<span class="hl opt">) {</span>
errno <span class="hl opt">=</span> EINVAL<span class="hl opt">;</span>
<span class="hl kwa">return</span> <span class="hl opt">-</span><span class="hl num">1</span><span class="hl opt">;</span>
<span class="hl opt">}</span>
ret <span class="hl opt">=</span> <span class="hl kwd">fstat</span><span class="hl opt">(</span>tgtfd<span class="hl opt">, &</span>st<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
<span class="hl kwa">if</span> <span class="hl opt">(!</span><span class="hl kwd">S_ISREG</span><span class="hl opt">(</span>st<span class="hl opt">.</span>st_mode<span class="hl opt">)) {</span>
errno <span class="hl opt">=</span> EINVAL<span class="hl opt">;</span>
<span class="hl kwa">return</span> <span class="hl opt">-</span><span class="hl num">1</span><span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl kwa">return</span> <span class="hl kwd">ioctl</span><span class="hl opt">(</span>tgtfd<span class="hl opt">,</span> BTRFS_IOC_CLONE<span class="hl opt">,</span> srcfd<span class="hl opt">);</span>
<span class="hl opt">}</span>
<span class="hl kwb">int</span> <span class="hl kwd">copy_file</span><span class="hl opt">(</span><span class="hl kwb">const char</span> <span class="hl opt">*</span>source<span class="hl opt">,</span> <span class="hl kwb">const char</span> <span class="hl opt">*</span>target<span class="hl opt">,</span> <span class="hl kwb">bool</span> no_clobber<span class="hl opt">) {</span>
<span class="hl kwb">int</span> srcfd <span class="hl opt">= -</span><span class="hl num">1</span><span class="hl opt">;</span>
<span class="hl kwb">int</span> tgtfd <span class="hl opt">= -</span><span class="hl num">1</span><span class="hl opt">;</span>
<span class="hl kwb">int</span> ret <span class="hl opt">= -</span><span class="hl num">1</span><span class="hl opt">;</span>
srcfd <span class="hl opt">=</span> <span class="hl kwd">open</span><span class="hl opt">(</span>source<span class="hl opt">,</span> O_RDONLY<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>srcfd <span class="hl opt">== -</span><span class="hl num">1</span><span class="hl opt">) {</span>
<span class="hl kwd">perror</span><span class="hl opt">(</span><span class="hl str">"Open source file"</span><span class="hl opt">);</span>
<span class="hl kwa">return</span> srcfd<span class="hl opt">;</span>
<span class="hl opt">}</span>
tgtfd <span class="hl opt">=</span> <span class="hl kwd">open</span><span class="hl opt">(</span>target<span class="hl opt">,</span> O_WRONLY<span class="hl opt">|</span>O_CREAT<span class="hl opt">|(</span>no_clobber <span class="hl opt">?</span> O_EXCL <span class="hl opt">:</span> <span class="hl num">0</span><span class="hl opt">),</span> <span class="hl num">0600</span><span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>tgtfd <span class="hl opt">== -</span><span class="hl num">1</span><span class="hl opt">) {</span>
<span class="hl kwd">perror</span><span class="hl opt">(</span><span class="hl str">"Open target file"</span><span class="hl opt">);</span>
<span class="hl kwa">return</span> tgtfd<span class="hl opt">;</span>
<span class="hl opt">}</span>
ret <span class="hl opt">=</span> <span class="hl kwd">btrfs_clone_contents</span><span class="hl opt">(</span>srcfd<span class="hl opt">,</span> tgtfd<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt">>=</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span> <span class="hl opt">&&</span> errno <span class="hl opt">!=</span> EINVAL<span class="hl opt">) {</span>
<span class="hl com">/* Some error that wasn't from a btrfs clone,</span>
<span class="hl com"> so we can't fall back to something that would work */</span>
<span class="hl kwd">perror</span><span class="hl opt">(</span><span class="hl str">"Copy file"</span><span class="hl opt">);</span>
<span class="hl kwa">return</span> <span class="hl opt">-</span><span class="hl num">1</span><span class="hl opt">;</span>
<span class="hl opt">}</span>
ret <span class="hl opt">=</span> <span class="hl kwd">sparse_copy_contents</span><span class="hl opt">(</span>srcfd<span class="hl opt">,</span> tgtfd<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt">>=</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span> <span class="hl opt">&&</span> errno <span class="hl opt">!=</span> EINVAL<span class="hl opt">) {</span>
<span class="hl com">/* Some error that wasn't from a sparse copy,</span>
<span class="hl com"> so we can't fall back to something that would work */</span>
<span class="hl kwd">perror</span><span class="hl opt">(</span><span class="hl str">"Copy file"</span><span class="hl opt">);</span>
<span class="hl kwa">return</span> <span class="hl opt">-</span><span class="hl num">1</span><span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl kwa">return</span> <span class="hl kwd">naive_contents_copy</span><span class="hl opt">(</span>srcfd<span class="hl opt">,</span> tgtfd<span class="hl opt">);</span>
<span class="hl opt">}</span>
</pre></div>
<h2>What if I'm not using a filesystem that decouples files and their contents?</h2>
<p>You probably don't get the benefit of being able to share contents
so you are unlikely to be able to copy a file without duplicating its data.</p>
<p>It is however still possible to reduce the amount of effort involved.</p>
<p>You may have noticed that the general pattern is read the contents into memory
then write the contents from memory into the new file.</p>
<p>This means that to copy a file you are copying the data twice,
first out of the old file into your process' memory,
then again into the new file.
(The exact details are complicated.
If you've not opened the file with <code>O_DIRECT</code>
then the data may be cached so it's copied from kernel memory,
but if it's not then it's read from disk into kernel memory first.
When you write without <code>O_DIRECT</code> it queues up the data
to be written to disk at some point soon in the near future.)</p>
<p>In some circumstances it may be possible to cut this down to one copy,
straight from the source file into the target file.</p>
<p>There are a handful of system calls for copying data
from one file into another.</p>
<p>Historically there has been <a href="http://man7.org/linux/man-pages/man2/sendfile.2.html">sendfile(2)</a> and <a href="http://man7.org/linux/man-pages/man2/splice.2.html">splice(2)</a>
which copy data between two files
without reading them into userspace first.</p>
<div class="highlight-c"><pre class="hl"><span class="hl ppc">#include <sys/sendfile.h></span> <span class="hl com">/* sendfile */</span><span class="hl ppc"></span>
ssize_t <span class="hl kwd">sendfile_copy_range</span><span class="hl opt">(</span><span class="hl kwb">int</span> srcfd<span class="hl opt">,</span> <span class="hl kwb">int</span> tgtfd<span class="hl opt">,</span> <span class="hl kwb">size_t</span> range<span class="hl opt">) {</span>
<span class="hl kwb">size_t</span> to_copy <span class="hl opt">=</span> range<span class="hl opt">;</span>
<span class="hl kwa">while</span> <span class="hl opt">(</span>to_copy<span class="hl opt">) {</span>
ssize_t ret <span class="hl opt">=</span> <span class="hl kwd">sendfile</span><span class="hl opt">(</span>tgtfd<span class="hl opt">,</span> srcfd<span class="hl opt">,</span> NULL<span class="hl opt">,</span> range<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
to_copy <span class="hl opt">-=</span> ret<span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl kwa">return</span> range<span class="hl opt">;</span>
<span class="hl opt">}</span>
ssize_t <span class="hl kwd">splice_copy_range</span><span class="hl opt">(</span><span class="hl kwb">int</span> srcfd<span class="hl opt">,</span> <span class="hl kwb">int</span> tgtfd<span class="hl opt">,</span> <span class="hl kwb">size_t</span> range<span class="hl opt">) {</span>
<span class="hl kwb">size_t</span> to_copy <span class="hl opt">=</span> range<span class="hl opt">;</span>
<span class="hl kwa">while</span> <span class="hl opt">(</span>to_copy<span class="hl opt">) {</span>
ssize_t ret <span class="hl opt">=</span> <span class="hl kwd">splice</span><span class="hl opt">(</span>srcfd<span class="hl opt">,</span> NULL<span class="hl opt">,</span> tgtfd<span class="hl opt">,</span> NULL<span class="hl opt">,</span> range<span class="hl opt">,</span> <span class="hl num">0</span><span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
to_copy <span class="hl opt">-=</span> ret<span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl kwa">return</span> range<span class="hl opt">;</span>
<span class="hl opt">}</span>
</pre></div>
<p>These were originally designed for speeding up web servers serving static files
by copying data between files and pipes or sockets.</p>
<p><a href="http://man7.org/linux/man-pages/man2/copy_file_range.2.html">copy_file_range(2)</a> was added to copy the contents of one file to another,
after an a failed attempt to get it in under the name <a href="https://lwn.net/Articles/659523/">reflink</a>,
with the intention that it would be a filesystem independent way
to share the contents of files like <a href="https://btrfs.wiki.kernel.org/index.php/Main_Page">btrfs</a>'s clone ioctl.</p>
<div class="highlight-c"><pre class="hl"><span class="hl ppc">#if !HAVE_DECL_COPY_FILE_RANGE</span>
<span class="hl ppc">#ifndef __NR_copy_file_range</span>
<span class="hl ppc"># if defined(__x86_64__)</span>
<span class="hl ppc"># define __NR_copy_file_range 326</span>
<span class="hl ppc"># elif defined(__i386__)</span>
<span class="hl ppc"># define __NR_copy_file_range 377</span>
<span class="hl ppc"># endif</span>
<span class="hl ppc">#endif</span>
<span class="hl kwb">static</span> <span class="hl kwc">inline</span> <span class="hl kwb">int</span> <span class="hl kwd">copy_file_range</span><span class="hl opt">(</span><span class="hl kwb">int</span> fd_in<span class="hl opt">,</span> loff_t <span class="hl opt">*</span>off_in<span class="hl opt">,</span> <span class="hl kwb">int</span> fd_out<span class="hl opt">,</span> loff_t <span class="hl opt">*</span>off_out<span class="hl opt">,</span> <span class="hl kwb">size_t</span> len<span class="hl opt">,</span> <span class="hl kwb">unsigned int</span> flags<span class="hl opt">) {</span>
<span class="hl kwa">return</span> <span class="hl kwd">syscall</span><span class="hl opt">(</span>__NR_copy_file_range<span class="hl opt">,</span> fd_in<span class="hl opt">,</span> off_in<span class="hl opt">,</span> fd_out<span class="hl opt">,</span> off_out<span class="hl opt">,</span> len<span class="hl opt">,</span> flags<span class="hl opt">);</span>
<span class="hl opt">}</span>
<span class="hl ppc">#endif</span>
ssize_t <span class="hl kwd">cfr_copy_range</span><span class="hl opt">(</span><span class="hl kwb">int</span> srcfd<span class="hl opt">,</span> <span class="hl kwb">int</span> tgtfd<span class="hl opt">,</span> <span class="hl kwb">size_t</span> range<span class="hl opt">) {</span>
<span class="hl kwb">size_t</span> to_copy <span class="hl opt">=</span> range<span class="hl opt">;</span>
<span class="hl kwa">while</span> <span class="hl opt">(</span>to_copy<span class="hl opt">) {</span>
ssize_t ret <span class="hl opt">=</span> <span class="hl kwd">copy_file_range</span><span class="hl opt">(</span>srcfd<span class="hl opt">,</span> NULL<span class="hl opt">,</span> tgtfd<span class="hl opt">,</span> NULL<span class="hl opt">,</span> range<span class="hl opt">,</span> <span class="hl num">0</span><span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
to_copy <span class="hl opt">-=</span> ret<span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl kwa">return</span> range<span class="hl opt">;</span>
<span class="hl opt">}</span>
</pre></div>
<p>To make use of these different ways to copy a file more quickly
we need a way to dispatch between them.</p>
<p>An <code>errno</code> of <code>ENOSYS</code> means that the system call isn't supported,
so there's no value in ever calling that again,
but <code>EINVAL</code> just means it's the wrong kind of file
so you just need to fall back to a different method of copying.</p>
<div class="highlight-c"><pre class="hl">ssize_t <span class="hl kwd">naive_copy_range</span><span class="hl opt">(</span><span class="hl kwb">int</span> srcfd<span class="hl opt">,</span> <span class="hl kwb">int</span> tgtfd<span class="hl opt">,</span> <span class="hl kwb">size_t</span> range<span class="hl opt">) {</span>
<span class="hl kwb">char</span> buf<span class="hl opt">[</span><span class="hl num">4</span> <span class="hl opt">*</span> <span class="hl num">1024</span> <span class="hl opt">*</span> <span class="hl num">1024</span><span class="hl opt">];</span>
<span class="hl kwb">size_t</span> copied <span class="hl opt">=</span> <span class="hl num">0</span><span class="hl opt">;</span>
<span class="hl kwa">while</span> <span class="hl opt">(</span>range <span class="hl opt">></span> copied<span class="hl opt">) {</span>
<span class="hl kwb">size_t</span> to_copy <span class="hl opt">=</span> range <span class="hl opt">-</span> copied<span class="hl opt">;</span>
ssize_t n_read<span class="hl opt">;</span>
n_read <span class="hl opt">=</span> <span class="hl kwd">TEMP_FAILURE_RETRY</span><span class="hl opt">(</span><span class="hl kwd">read</span><span class="hl opt">(</span>srcfd<span class="hl opt">,</span> buf<span class="hl opt">,</span>
to_copy <span class="hl opt">></span> <span class="hl kwa">sizeof</span><span class="hl opt">(</span>buf<span class="hl opt">) ?</span> <span class="hl kwa">sizeof</span><span class="hl opt">(</span>buf<span class="hl opt">) :</span> to_copy<span class="hl opt">));</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>n_read <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">) {</span>
<span class="hl kwd">perror</span><span class="hl opt">(</span><span class="hl str">"Read source file"</span><span class="hl opt">);</span>
<span class="hl kwa">return</span> n_read<span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>n_read <span class="hl opt">==</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">break</span><span class="hl opt">;</span>
<span class="hl kwa">while</span> <span class="hl opt">(</span>n_read <span class="hl opt">></span> <span class="hl num">0</span><span class="hl opt">) {</span>
ssize_t n_written <span class="hl opt">=</span> <span class="hl kwd">TEMP_FAILURE_RETRY</span><span class="hl opt">(</span><span class="hl kwd">write</span><span class="hl opt">(</span>tgtfd<span class="hl opt">,</span> buf<span class="hl opt">,</span> n_read<span class="hl opt">));</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>n_written <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwd">perror</span><span class="hl opt">(</span><span class="hl str">"Write to target file"</span><span class="hl opt">);</span>
<span class="hl kwa">return</span> n_written<span class="hl opt">;</span>
n_read <span class="hl opt">-=</span> n_written<span class="hl opt">;</span>
copied <span class="hl opt">+=</span> n_written<span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl opt">}</span>
<span class="hl kwa">return</span> copied<span class="hl opt">;</span>
<span class="hl opt">}</span>
ssize_t <span class="hl kwd">copy_range</span><span class="hl opt">(</span><span class="hl kwb">int</span> srcfd<span class="hl opt">,</span> <span class="hl kwb">int</span> tgtfd<span class="hl opt">,</span> <span class="hl kwb">size_t</span> range<span class="hl opt">) {</span>
ssize_t copied<span class="hl opt">;</span>
<span class="hl kwb">static int</span> have_cfr <span class="hl opt">=</span> <span class="hl kwa">true</span><span class="hl opt">,</span> have_sendfile <span class="hl opt">=</span> <span class="hl kwa">true</span><span class="hl opt">,</span> have_splice <span class="hl opt">=</span> <span class="hl kwa">true</span><span class="hl opt">;</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>have_cfr<span class="hl opt">) {</span>
copied <span class="hl opt">=</span> <span class="hl kwd">cfr_copy_range</span><span class="hl opt">(</span>srcfd<span class="hl opt">,</span> tgtfd<span class="hl opt">,</span> range<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>copied <span class="hl opt">>=</span> <span class="hl num">0</span><span class="hl opt">) {</span>
<span class="hl kwa">return</span> copied<span class="hl opt">;</span>
<span class="hl opt">}</span> <span class="hl kwa">else if</span> <span class="hl opt">(</span>errno <span class="hl opt">==</span> ENOSYS<span class="hl opt">) {</span>
have_cfr <span class="hl opt">=</span> <span class="hl kwa">false</span><span class="hl opt">;</span>
<span class="hl opt">}</span> <span class="hl kwa">else if</span> <span class="hl opt">(</span>errno <span class="hl opt">!=</span> EINVAL<span class="hl opt">) {</span>
<span class="hl kwa">return</span> copied<span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl opt">}</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>have_sendfile<span class="hl opt">) {</span>
copied <span class="hl opt">=</span> <span class="hl kwd">sendfile_copy_range</span><span class="hl opt">(</span>srcfd<span class="hl opt">,</span> tgtfd<span class="hl opt">,</span> range<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>copied <span class="hl opt">>=</span> <span class="hl num">0</span><span class="hl opt">) {</span>
<span class="hl kwa">return</span> copied<span class="hl opt">;</span>
<span class="hl opt">}</span> <span class="hl kwa">else if</span> <span class="hl opt">(</span>errno <span class="hl opt">==</span> ENOSYS<span class="hl opt">) {</span>
have_sendfile <span class="hl opt">=</span> <span class="hl kwa">false</span><span class="hl opt">;</span>
<span class="hl opt">}</span> <span class="hl kwa">else if</span> <span class="hl opt">(</span>errno <span class="hl opt">!=</span> EINVAL<span class="hl opt">) {</span>
<span class="hl kwa">return</span> copied<span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl opt">}</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>have_splice<span class="hl opt">) {</span>
copied <span class="hl opt">=</span> <span class="hl kwd">splice_copy_range</span><span class="hl opt">(</span>srcfd<span class="hl opt">,</span> tgtfd<span class="hl opt">,</span> range<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>copied <span class="hl opt">>=</span> <span class="hl num">0</span><span class="hl opt">) {</span>
<span class="hl kwa">return</span> copied<span class="hl opt">;</span>
<span class="hl opt">}</span> <span class="hl kwa">else if</span> <span class="hl opt">(</span>errno <span class="hl opt">==</span> ENOSYS<span class="hl opt">) {</span>
have_splice <span class="hl opt">=</span> <span class="hl kwa">false</span><span class="hl opt">;</span>
<span class="hl opt">}</span> <span class="hl kwa">else if</span> <span class="hl opt">(</span>errno <span class="hl opt">!=</span> EINVAL<span class="hl opt">) {</span>
<span class="hl kwa">return</span> copied<span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl opt">}</span>
<span class="hl kwa">return</span> <span class="hl kwd">naive_copy_range</span><span class="hl opt">(</span>srcfd<span class="hl opt">,</span> tgtfd<span class="hl opt">,</span> range<span class="hl opt">);</span>
<span class="hl opt">}</span>
</pre></div>
<p>For convenience of testing,
the full <a href="http://yakking.branchable.com/posts/moving-files-3-faster/my-mv.c">my-mv.c</a> source file and <a href="http://yakking.branchable.com/posts/moving-files-3-faster/Makefile">Makefile</a>,
including the new copy functions,
can be downloaded.</p>
<h1>Conclusion</h1>
<p>So now we've got a slightly slower fall-back to when <a href="http://man7.org/linux/man-pages/man2/rename.2.html">rename(2)</a> fails right?</p>
<p>Well, not quite.
We're copying the data as quickly as we can,
but files also have associated metadata
that we're not even thinking about yet.</p>
How difficult is it to move a sparse file?http://yakking.branchable.com/posts/moving-files-2-sparseness/Richard Maw2016-08-03T11:00:14Z2016-08-03T11:00:07Z
<p>In our previous article,
we established that if <a href="http://man7.org/linux/man-pages/man2/rename.2.html">rename(2)</a> fails
then we have to fall back to copying the file then removing the old one.</p>
<p>We copied the data by just reading blocks from one file
and writing them to the new one.</p>
<p>For most files this would be sufficient,
but for those that aren't this is a bad idea.</p>
<h1>By reading and writing blocks the new file has exactly the same contents as the old one right?</h1>
<p>Not quite. Files can have "holes", where there is no data,
but reading that range returns blocks of zeroes,
so a naĆÆve copy will produce a file that has no holes.</p>
<p>This makes it take up more disk space,
and other hole-aware software will treat it differently.</p>
<p>An example would be a tool for writing disk images to disks,
it could make things quicker by only writing the data in the disk images
but if you created a copy without holes it would write a whole bunch of zeroes
that it didn't need to
and reduce the life of the hard-drive.</p>
<p>To do this properly you need to either use <a href="https://www.kernel.org/doc/Documentation/filesystems/fiemap.txt"><code>FIEMAP</code></a>,
or <a href="http://man7.org/linux/man-pages/man2/lseek.2.html"><code>SEEK_{DATA,HOLE}</code></a>.</p>
<p>While <a href="https://www.kernel.org/doc/Documentation/filesystems/fiemap.txt"><code>FIEMAP</code></a> is available in earlier kernels,
since it exposes more information than is needed for copying a file
and has historically been a source of disk corruption,
we're going to proceed with <a href="http://man7.org/linux/man-pages/man2/lseek.2.html"><code>SEEK_{DATA,HOLE}</code></a>.</p>
<h1>Copying sparsely with <a href="http://man7.org/linux/man-pages/man2/lseek.2.html">lseek(2)</a></h1>
<p>The basis of the algorithm is to use <a href="http://man7.org/linux/man-pages/man2/lseek.2.html">lseek(2)</a> with <code>SEEK_HOLE</code>
to find the end of a block of data and copy it,
then use <a href="http://man7.org/linux/man-pages/man2/lseek.2.html">lseek(2)</a> with <code>SEEK_DATA</code> to find the end of the following hole.</p>
<p>The difficulty, as always, is in the details.</p>
<div class="highlight-c"><pre class="hl">ssize_t <span class="hl kwd">copy_range</span><span class="hl opt">(</span><span class="hl kwb">int</span> srcfd<span class="hl opt">,</span> <span class="hl kwb">int</span> tgtfd<span class="hl opt">,</span> <span class="hl kwb">size_t</span> range<span class="hl opt">);</span>
ssize_t <span class="hl kwd">naive_contents_copy</span><span class="hl opt">(</span><span class="hl kwb">int</span> srcfd<span class="hl opt">,</span> <span class="hl kwb">int</span> tgtfd<span class="hl opt">);</span>
ssize_t <span class="hl kwd">sparse_copy_contents</span><span class="hl opt">(</span><span class="hl kwb">int</span> srcfd<span class="hl opt">,</span> <span class="hl kwb">int</span> tgtfd<span class="hl opt">) {</span>
<span class="hl kwb">size_t</span> copied <span class="hl opt">=</span> <span class="hl num">0</span><span class="hl opt">;</span>
off_t srcoffs <span class="hl opt">= (</span>off_t<span class="hl opt">)-</span><span class="hl num">1</span><span class="hl opt">;</span>
off_t nextoffs <span class="hl opt">= (</span>off_t<span class="hl opt">)-</span><span class="hl num">1</span><span class="hl opt">;</span>
</pre></div>
<h2>Finding the start</h2>
<p>The first thing we need to do
is to find out whether we started in a data block or a hole block.</p>
<p>To do that we need to know where "here" is though,
the way to do this is to call <code>lseek(fd, 0, SEEK_CUR)</code>,
which logically means move the position forward 0 bytes,
but has the side-effect of returning the current position.</p>
<div class="highlight-c"><pre class="hl"> srcoffs <span class="hl opt">=</span> <span class="hl kwd">TEMP_FAILURE_RETRY</span><span class="hl opt">(</span><span class="hl kwd">lseek</span><span class="hl opt">(</span>srcfd<span class="hl opt">,</span> <span class="hl num">0</span><span class="hl opt">,</span> SEEK_CUR<span class="hl opt">));</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>srcoffs <span class="hl opt">== (</span>off_t<span class="hl opt">)-</span><span class="hl num">1</span><span class="hl opt">) {</span>
<span class="hl kwd">perror</span><span class="hl opt">(</span><span class="hl str">"Find current position of file"</span><span class="hl opt">);</span>
<span class="hl com">/* Can't seek file, could be file isn't seekable,</span>
<span class="hl com"> or that the current offset would overflow. */</span>
<span class="hl kwa">return</span> <span class="hl opt">-</span><span class="hl num">1</span><span class="hl opt">;</span>
<span class="hl opt">}</span>
</pre></div>
<h2>Starting with data or a hole?</h2>
<p>Now that we've got the current offset,
we <code>lseek(fd, offset, SEEK_DATA)</code>.
If the returned value is the same as the provided offset,
then it was a data block,
but if it moved then we were in a hole
and it returned the start of the data.</p>
<p>There's also the ever-present possibility that there is no more data,
which sets <a href="http://man7.org/linux/man-pages/man3/errno.3.html">errno(3)</a> to <code>ENXIO</code>.</p>
<div class="highlight-c"><pre class="hl"> nextoffs <span class="hl opt">=</span> <span class="hl kwd">TEMP_FAILURE_RETRY</span><span class="hl opt">(</span><span class="hl kwd">lseek</span><span class="hl opt">(</span>srcfd<span class="hl opt">,</span> srcoffs<span class="hl opt">,</span> SEEK_DATA<span class="hl opt">));</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>nextoffs <span class="hl opt">== (</span>off_t<span class="hl opt">)-</span><span class="hl num">1</span><span class="hl opt">) {</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>errno <span class="hl opt">==</span> ENXIO<span class="hl opt">) {</span>
<span class="hl com">/* NXIO means EOF, there is no data to copy,</span>
<span class="hl com"> but we may need to make a hole to the end of the file */</span>
<span class="hl kwa">goto</span> end_hole<span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl kwd">perror</span><span class="hl opt">(</span><span class="hl str">"Find data or hole at beginning of file"</span><span class="hl opt">);</span>
<span class="hl com">/* Error seeking, must not support sparse seek */</span>
<span class="hl kwa">return</span> <span class="hl opt">-</span><span class="hl num">1</span><span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>srcoffs <span class="hl opt">!=</span> nextoffs<span class="hl opt">)</span>
<span class="hl com">/* Seeked to the end of a hole, can skip a data copy. */</span>
<span class="hl kwa">goto</span> hole<span class="hl opt">;</span>
</pre></div>
<h2>Copying data and holes</h2>
<p>Depending on whether we started in data or in a hole,
we either copy the contents of the data,
or use <a href="http://man7.org/linux/man-pages/man2/truncate.2.html">truncate(2)</a> to extend the file without providing data.</p>
<p>Because <a href="http://man7.org/linux/man-pages/man2/truncate.2.html">truncate(2)</a> does not advance the file offset,
we have to use <a href="http://man7.org/linux/man-pages/man2/lseek.2.html">lseek(2)</a> to do it manually.</p>
<p>As before, we can reach the end of the file when seeking,
which breaks us out of the copy data then copy hole loop.</p>
<div class="highlight-c"><pre class="hl"> <span class="hl kwa">for</span> <span class="hl opt">(;;) {</span>
ssize_t ret<span class="hl opt">;</span>
<span class="hl com">/* In data, so we must find the end of the data then copy it,</span>
<span class="hl com"> could pread/write. */</span>
nextoffs <span class="hl opt">=</span> <span class="hl kwd">TEMP_FAILURE_RETRY</span><span class="hl opt">(</span><span class="hl kwd">lseek</span><span class="hl opt">(</span>srcfd<span class="hl opt">,</span> srcoffs<span class="hl opt">,</span> SEEK_HOLE<span class="hl opt">));</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>nextoffs <span class="hl opt">== (</span>off_t<span class="hl opt">)-</span><span class="hl num">1</span><span class="hl opt">) {</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>errno <span class="hl opt">!=</span> ENXIO<span class="hl opt">) {</span>
<span class="hl kwd">perror</span><span class="hl opt">(</span><span class="hl str">"Find end of data"</span><span class="hl opt">);</span>
<span class="hl kwa">return</span> <span class="hl opt">-</span><span class="hl num">1</span><span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl com">/* EOF after data, but we still need to copy */</span>
<span class="hl kwa">goto</span> end_data<span class="hl opt">;</span>
<span class="hl opt">}</span>
srcoffs <span class="hl opt">=</span> <span class="hl kwd">TEMP_FAILURE_RETRY</span><span class="hl opt">(</span><span class="hl kwd">lseek</span><span class="hl opt">(</span>srcfd<span class="hl opt">,</span> srcoffs<span class="hl opt">,</span> SEEK_SET<span class="hl opt">));</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>srcoffs <span class="hl opt">== (</span>off_t<span class="hl opt">)-</span><span class="hl num">1</span><span class="hl opt">) {</span>
<span class="hl com">/* Rewinding failed, something is *very* strange. */</span>
<span class="hl kwd">perror</span><span class="hl opt">(</span><span class="hl str">"Rewind back to data"</span><span class="hl opt">);</span>
<span class="hl kwa">return</span> <span class="hl opt">-</span><span class="hl num">1</span><span class="hl opt">;</span>
<span class="hl opt">}</span>
ret <span class="hl opt">=</span> <span class="hl kwd">copy_range</span><span class="hl opt">(</span>srcfd<span class="hl opt">,</span> tgtfd<span class="hl opt">,</span> nextoffs <span class="hl opt">-</span> srcoffs<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">) {</span>
<span class="hl kwa">return</span> <span class="hl opt">-</span><span class="hl num">1</span><span class="hl opt">;</span>
<span class="hl opt">}</span>
copied <span class="hl opt">+=</span> ret<span class="hl opt">;</span>
srcoffs <span class="hl opt">=</span> nextoffs<span class="hl opt">;</span>
nextoffs <span class="hl opt">=</span> <span class="hl kwd">TEMP_FAILURE_RETRY</span><span class="hl opt">(</span><span class="hl kwd">lseek</span><span class="hl opt">(</span>srcfd<span class="hl opt">,</span> srcoffs<span class="hl opt">,</span> SEEK_DATA<span class="hl opt">));</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>nextoffs <span class="hl opt">== (</span>off_t<span class="hl opt">)-</span><span class="hl num">1</span><span class="hl opt">) {</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>errno <span class="hl opt">==</span> ENXIO<span class="hl opt">) {</span>
<span class="hl com">/* NXIO means EOF, there is no data to copy,</span>
<span class="hl com"> but we may need to make a hole to the end of the file */</span>
<span class="hl kwa">goto</span> end_hole<span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl kwd">perror</span><span class="hl opt">(</span><span class="hl str">"Find end of hole"</span><span class="hl opt">);</span>
<span class="hl com">/* Error seeking, must not support sparse seek */</span>
<span class="hl kwa">return</span> <span class="hl opt">-</span><span class="hl num">1</span><span class="hl opt">;</span>
<span class="hl opt">}</span>
hole<span class="hl opt">:</span>
<span class="hl com">/* Is a hole, extend the file to the offset */</span>
ret <span class="hl opt">=</span> <span class="hl kwd">TEMP_FAILURE_RETRY</span><span class="hl opt">(</span><span class="hl kwd">ftruncate</span><span class="hl opt">(</span>tgtfd<span class="hl opt">,</span> nextoffs<span class="hl opt">));</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">) {</span>
<span class="hl kwd">perror</span><span class="hl opt">(</span><span class="hl str">"Truncate file to add hole"</span><span class="hl opt">);</span>
<span class="hl kwa">return</span> <span class="hl opt">-</span><span class="hl num">1</span><span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl com">/* Move file offset for target to after the newly added hole */</span>
nextoffs <span class="hl opt">=</span> <span class="hl kwd">TEMP_FAILURE_RETRY</span><span class="hl opt">(</span><span class="hl kwd">lseek</span><span class="hl opt">(</span>tgtfd<span class="hl opt">,</span> nextoffs<span class="hl opt">,</span> SEEK_SET<span class="hl opt">));</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>nextoffs <span class="hl opt">== (</span>off_t<span class="hl opt">)-</span><span class="hl num">1</span><span class="hl opt">) {</span>
<span class="hl com">/* Something very strange happened,</span>
<span class="hl com"> either some race condition changed the file,</span>
<span class="hl com"> or the file is truncatable but not seekable</span>
<span class="hl com"> or some external memory corruption,</span>
<span class="hl com"> since EOVERFLOW can't happen with SEEK_SET */</span>
<span class="hl kwd">perror</span><span class="hl opt">(</span><span class="hl str">"Move to after newly added hole"</span><span class="hl opt">);</span>
<span class="hl kwa">return</span> <span class="hl opt">-</span><span class="hl num">1</span><span class="hl opt">;</span>
<span class="hl opt">}</span>
srcoffs <span class="hl opt">=</span> nextoffs<span class="hl opt">;</span>
<span class="hl opt">}</span>
</pre></div>
<h2>Filling it to the end</h2>
<p>When finished with the copy-hole copy-data loop,
we still have to fill the rest of the file,
either by truncating it to fill in the final hole,
or copying the rest of the data.</p>
<div class="highlight-c"><pre class="hl">end_hole<span class="hl opt">:</span>
nextoffs <span class="hl opt">=</span> <span class="hl kwd">TEMP_FAILURE_RETRY</span><span class="hl opt">(</span><span class="hl kwd">lseek</span><span class="hl opt">(</span>srcfd<span class="hl opt">,</span> <span class="hl num">0</span><span class="hl opt">,</span> SEEK_END<span class="hl opt">));</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>nextoffs <span class="hl opt">== (</span>off_t<span class="hl opt">)-</span><span class="hl num">1</span><span class="hl opt">) {</span>
<span class="hl kwd">perror</span><span class="hl opt">(</span><span class="hl str">"Seek to end of file"</span><span class="hl opt">);</span>
<span class="hl kwa">return</span> <span class="hl opt">-</span><span class="hl num">1</span><span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>srcoffs <span class="hl opt">!=</span> nextoffs<span class="hl opt">) {</span>
<span class="hl com">/* Not already at EOF, need to extend */</span>
<span class="hl kwb">int</span> ret <span class="hl opt">=</span> <span class="hl kwd">TEMP_FAILURE_RETRY</span><span class="hl opt">(</span><span class="hl kwd">ftruncate</span><span class="hl opt">(</span>tgtfd<span class="hl opt">,</span> nextoffs<span class="hl opt">));</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">) {</span>
<span class="hl kwd">perror</span><span class="hl opt">(</span><span class="hl str">"Truncate to add hole at end of file"</span><span class="hl opt">);</span>
<span class="hl kwa">return</span> <span class="hl opt">-</span><span class="hl num">1</span><span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl opt">}</span>
<span class="hl kwa">return</span> copied<span class="hl opt">;</span>
end_data<span class="hl opt">:</span>
<span class="hl opt">{</span>
ssize_t ret <span class="hl opt">=</span> <span class="hl kwd">naive_contents_copy</span><span class="hl opt">(</span>srcfd<span class="hl opt">,</span> tgtfd<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
copied <span class="hl opt">+=</span> ret<span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl kwa">return</span> copied<span class="hl opt">;</span>
<span class="hl opt">}</span>
</pre></div>
<h2>Integrating sparse copying into the program</h2>
<p>Now that we've got our <code>sparse_copy_contents</code>,
we need to amend our <code>copy_file</code> function to call it.</p>
<p>We can detect whether sparse copying is not possible
by whether <a href="http://man7.org/linux/man-pages/man3/errno.3.html">errno(3)</a> gets set to <code>EINVAL</code>,
so there's no harm in trying to sparsely copy a file first.</p>
<div class="highlight-c"><pre class="hl"><span class="hl kwb">int</span> <span class="hl kwd">copy_file</span><span class="hl opt">(</span><span class="hl kwb">const char</span> <span class="hl opt">*</span>source<span class="hl opt">,</span> <span class="hl kwb">const char</span> <span class="hl opt">*</span>target<span class="hl opt">,</span> <span class="hl kwb">bool</span> no_clobber<span class="hl opt">) {</span>
<span class="hl kwb">int</span> srcfd <span class="hl opt">= -</span><span class="hl num">1</span><span class="hl opt">;</span>
<span class="hl kwb">int</span> tgtfd <span class="hl opt">= -</span><span class="hl num">1</span><span class="hl opt">;</span>
<span class="hl kwb">int</span> ret <span class="hl opt">= -</span><span class="hl num">1</span><span class="hl opt">;</span>
srcfd <span class="hl opt">=</span> <span class="hl kwd">open</span><span class="hl opt">(</span>source<span class="hl opt">,</span> O_RDONLY<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>srcfd <span class="hl opt">== -</span><span class="hl num">1</span><span class="hl opt">) {</span>
<span class="hl kwd">perror</span><span class="hl opt">(</span><span class="hl str">"Open source file"</span><span class="hl opt">);</span>
<span class="hl kwa">return</span> srcfd<span class="hl opt">;</span>
<span class="hl opt">}</span>
tgtfd <span class="hl opt">=</span> <span class="hl kwd">open</span><span class="hl opt">(</span>target<span class="hl opt">,</span> O_WRONLY<span class="hl opt">|</span>O_CREAT<span class="hl opt">|(</span>no_clobber <span class="hl opt">?</span> O_EXCL <span class="hl opt">:</span> <span class="hl num">0</span><span class="hl opt">),</span> <span class="hl num">0600</span><span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>tgtfd <span class="hl opt">== -</span><span class="hl num">1</span><span class="hl opt">) {</span>
<span class="hl kwd">perror</span><span class="hl opt">(</span><span class="hl str">"Open target file"</span><span class="hl opt">);</span>
<span class="hl kwa">return</span> tgtfd<span class="hl opt">;</span>
<span class="hl opt">}</span>
ret <span class="hl opt">=</span> <span class="hl kwd">sparse_copy_contents</span><span class="hl opt">(</span>srcfd<span class="hl opt">,</span> tgtfd<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt">>=</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span> <span class="hl opt">&&</span> errno <span class="hl opt">!=</span> EINVAL<span class="hl opt">) {</span>
<span class="hl com">/* Some error that wasn't from a sparse copy,</span>
<span class="hl com"> so we can't fall back to something that would work */</span>
<span class="hl kwd">perror</span><span class="hl opt">(</span><span class="hl str">"Copy file"</span><span class="hl opt">);</span>
<span class="hl kwa">return</span> <span class="hl opt">-</span><span class="hl num">1</span><span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl kwa">return</span> <span class="hl kwd">naive_contents_copy</span><span class="hl opt">(</span>srcfd<span class="hl opt">,</span> tgtfd<span class="hl opt">);</span>
<span class="hl opt">}</span>
</pre></div>
<p>I have omitted the definitions of <code>copy_range</code> and <code>naive_contents_copy</code>
since they are not relevant to handling holes,
but full program listing may be downloaded from <a href="http://yakking.branchable.com/posts/moving-files-2-sparseness/my-mv.c">my-mv.c</a>.</p>
<h1>Conclusion</h1>
<p>Now when we move a file we don't end up taking more space,
we don't copy data that we don't need so it goes faster,
and files that treat holes specially, like disk images, will behave properly.</p>
<p>But files that may have holes are also likely to contain a <em>lot</em> of data,
and even only copying the data we need will take a while.</p>
<p>Can we make this as fast as <a href="http://man7.org/linux/man-pages/man2/rename.2.html">rename(2)</a>?</p>
How difficult is it to move a file?http://yakking.branchable.com/posts/moving-files-1-copying/Richard Maw2016-07-13T11:00:13Z2016-07-13T11:00:06Z
<p>You should be familiar with the <a href="http://man7.org/linux/man-pages/man1/mv.1.html">mv(1)</a> command by now,
which moves a file from one place to another.</p>
<p>If you're writing shell scripts,
or if you're using a library which lets you move files
then you don't need to worry about how it works,
but if you don't have a library available
you might be surprised by the amount of effort required.</p>
<h1>It's just a <a href="http://man7.org/linux/man-pages/man2/rename.2.html">rename(2)</a>, right?</h1>
<p>If you can guarantee that the destination is on the same filesystem
and that you don't care if it replaces some other file, then yes.</p>
<h2>Not clobbering</h2>
<p><a href="http://man7.org/linux/man-pages/man1/mv.1.html">mv(1)</a> has the <code>-n</code> or <code>--no-clobber</code> option
to prevent accidentally overwriting a file.</p>
<p>The naive way to do this would be to check whether the file exists
before calling <a href="http://man7.org/linux/man-pages/man2/rename.2.html">rename(2)</a>,
but this is a <a href="https://en.wikipedia.org/wiki/Time_of_check_to_time_of_use">TOCTTOU</a> bug
which can cause this to overwrite if another thread puts a file there.</p>
<p>To do this safely use <a href="http://man7.org/linux/man-pages/man2/rename.2.html">renameat2(2)</a> with the <code>RENAME_NOREPLACE</code> flag,
which will make it fail if the destination already exists.</p>
<h2>The destination is on another filesystem</h2>
<p>On a modern Linux distribution
your files are usually spread across multiple file systems,
so your persistent files are on a filesystem mounted from local storage,
but your operating system puts temporary files on a different file system
so they get removed when your computer shuts down.</p>
<p>Unfortunately, the <a href="http://man7.org/linux/man-pages/man2/rename.2.html">rename(2)</a> system call does not work
if the destination is on a different file system.</p>
<p>Checking ahead of time whether a path is on a different file system
is traditionally handled by calling <a href="http://man7.org/linux/man-pages/man2/stat.2.html">stat(2)</a>,
and checking whether the <code>st_dev</code> field differs,
but this is another <a href="https://en.wikipedia.org/wiki/Time_of_check_to_time_of_use">TOCTTOU</a> bug waiting to happen
and <a href="http://man7.org/linux/man-pages/man2/rename.2.html">rename(2)</a> sets <a href="http://man7.org/linux/man-pages/man3/errno.3.html">errno(3)</a> to <code>EXDEV</code>
which lets you know it failed for being on another filesystem
in the same system call you would have made anyway.</p>
<p>If you care about still being able to move the file
when its destination is on a different file system
then you need a fallback when this happens.</p>
<h1>So we fall back to copying the file and removing the old one?</h1>
<p>In principle, yes, though actually implementing this is surprisingly difficult.</p>
<p>Handling the fallback logic itself is not straight-forward,
we'll get that out of the way first.</p>
<p>We can fallback to <a href="http://man7.org/linux/man-pages/man2/rename.2.html">rename(2)</a> if <a href="http://man7.org/linux/man-pages/man2/rename.2.html">renameat2(2)</a> is not implemented
but only if we don't need to handle not clobbering the target.</p>
<p>When that happens we need to fall back to the copy,
which can use <code>O_EXCL</code> with <code>O_CREAT</code>
to only write to the file if it didn't already exist.</p>
<p>If unlinking the source file fails because the file doesn't exist,
then that means that we were able to copy its contents
while something else removed it.</p>
<p>Given the file was written to its destination
and it no longer exists where it used to
it can be argued that the operation as a whole was successful.</p>
<div class="highlight-c"><pre class="hl"><span class="hl com">/* my-mv.c */</span>
<span class="hl ppc">#include <stdbool.h></span>
<span class="hl ppc">#include <fcntl.h></span>
<span class="hl ppc">#include <stdio.h></span>
<span class="hl ppc">#include <unistd.h></span>
<span class="hl ppc">#include <errno.h></span>
<span class="hl ppc">#include <sys/syscall.h></span>
<span class="hl ppc">#if !HAVE_DECL_RENAMEAT2</span>
<span class="hl kwb">static</span> <span class="hl kwc">inline</span> <span class="hl kwb">int</span> <span class="hl kwd">renameat2</span><span class="hl opt">(</span><span class="hl kwb">int</span> oldfd<span class="hl opt">,</span> <span class="hl kwb">const char</span> <span class="hl opt">*</span>oldname<span class="hl opt">,</span> <span class="hl kwb">int</span> newfd<span class="hl opt">,</span> <span class="hl kwb">const char</span> <span class="hl opt">*</span>newname<span class="hl opt">,</span> <span class="hl kwb">unsigned</span> flags<span class="hl opt">) {</span>
<span class="hl kwa">return</span> <span class="hl kwd">syscall</span><span class="hl opt">(</span>__NR_renameat2<span class="hl opt">,</span> oldfd<span class="hl opt">,</span> oldname<span class="hl opt">,</span> newfd<span class="hl opt">,</span> newname<span class="hl opt">,</span> flags<span class="hl opt">);</span>
<span class="hl opt">}</span>
<span class="hl ppc">#endif</span>
<span class="hl ppc">#ifndef RENAME_NOREPLACE</span>
<span class="hl ppc">#define RENAME_NOREPLACE (1<<0)</span>
<span class="hl ppc">#endif</span>
<span class="hl kwb">int</span> <span class="hl kwd">copy_file</span><span class="hl opt">(</span><span class="hl kwb">const char</span> <span class="hl opt">*</span>source<span class="hl opt">,</span> <span class="hl kwb">const char</span> <span class="hl opt">*</span>target<span class="hl opt">,</span> <span class="hl kwb">bool</span> no_clobber<span class="hl opt">);</span>
<span class="hl kwb">int</span> <span class="hl kwd">move_file</span><span class="hl opt">(</span><span class="hl kwb">const char</span> <span class="hl opt">*</span>source<span class="hl opt">,</span> <span class="hl kwb">const char</span> <span class="hl opt">*</span>target<span class="hl opt">,</span> <span class="hl kwb">bool</span> no_clobber<span class="hl opt">) {</span>
<span class="hl kwb">int</span> ret<span class="hl opt">;</span>
ret <span class="hl opt">=</span> <span class="hl kwd">renameat2</span><span class="hl opt">(</span>AT_FDCWD<span class="hl opt">,</span> source<span class="hl opt">,</span> AT_FDCWD<span class="hl opt">,</span> target<span class="hl opt">,</span> no_clobber <span class="hl opt">?</span> RENAME_NOREPLACE <span class="hl opt">:</span> <span class="hl num">0</span><span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt">==</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>errno <span class="hl opt">==</span> EXDEV<span class="hl opt">)</span>
<span class="hl kwa">goto</span> xdev<span class="hl opt">;</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>errno <span class="hl opt">!=</span> ENOSYS<span class="hl opt">) {</span>
<span class="hl kwd">perror</span><span class="hl opt">(</span><span class="hl str">"renaming file"</span><span class="hl opt">);</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl com">/* Have to skip to copy if unimplemented since rename can't detect EEXIST */</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>no_clobber<span class="hl opt">)</span>
<span class="hl kwa">goto</span> xdev<span class="hl opt">;</span>
rename<span class="hl opt">:</span>
ret <span class="hl opt">=</span> <span class="hl kwd">rename</span><span class="hl opt">(</span>source<span class="hl opt">,</span> target<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt">==</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>errno <span class="hl opt">==</span> EXDEV<span class="hl opt">)</span>
<span class="hl kwa">goto</span> xdev<span class="hl opt">;</span>
<span class="hl kwd">perror</span><span class="hl opt">(</span><span class="hl str">"renaming file"</span><span class="hl opt">);</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
xdev<span class="hl opt">:</span>
ret <span class="hl opt">=</span> <span class="hl kwd">copy_file</span><span class="hl opt">(</span>source<span class="hl opt">,</span> target<span class="hl opt">,</span> no_clobber<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span><span class="hl kwd">unlink</span><span class="hl opt">(</span>source<span class="hl opt">) <</span> <span class="hl num">0</span> <span class="hl opt">&&</span> errno <span class="hl opt">!=</span> ENOENT<span class="hl opt">) {</span>
<span class="hl kwd">perror</span><span class="hl opt">(</span><span class="hl str">"unlinking source file"</span><span class="hl opt">);</span>
<span class="hl kwa">return</span> <span class="hl opt">-</span><span class="hl num">1</span><span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
<span class="hl opt">}</span>
</pre></div>
<h2>So we open both files, and loop reading data then writing it?</h2>
<p>This will produce a file that when read,
will produce the same stream of bytes as the original.</p>
<p>You could use <a href="http://man7.org/linux/man-pages/man3/stdio.3.html">stdio(3)</a> to copy the contents,
but that will have to be left as an exercise for the reader,
since I don't like its record-based interface,
I prefer to deal with file descriptors over <code>FILE*</code> handles,
and the buffering makes error handling moreā¦ interesting.</p>
<p>So, broadly, the idea is to read into a buffer,
then write from the buffer to the target file.</p>
<p>However, <a href="http://www.gnu.org/software/libc/manual/html_node/Interrupted-Primitives.html"><code>EINTR</code></a> is a problem,
many system calls can be interrupted before they do anything,
and <a href="http://man7.org/linux/man-pages/man2/read.2.html">read(2)</a> and <a href="http://man7.org/linux/man-pages/man2/write.2.html">write(2)</a> may return less than you asked for.</p>
<p>Glibc has a handy <code>TEMP_FAILURE_RETRY</code> macro for handling <a href="http://www.gnu.org/software/libc/manual/html_node/Interrupted-Primitives.html"><code>EINTR</code></a>,
but to handle the short reads and writes,
you need to always work in a loop.</p>
<div class="highlight-c"><pre class="hl"><span class="hl kwb">int</span> <span class="hl kwd">naive_contents_copy</span><span class="hl opt">(</span><span class="hl kwb">int</span> srcfd<span class="hl opt">,</span> <span class="hl kwb">int</span> tgtfd<span class="hl opt">) {</span>
<span class="hl com">/* 1MB buffer, too small makes it slow,</span>
<span class="hl com"> shrink this if you feel memory pressure on an embedded device */</span>
<span class="hl kwb">char</span> buf<span class="hl opt">[</span><span class="hl num">1</span> <span class="hl opt">*</span> <span class="hl num">1024</span> <span class="hl opt">*</span> <span class="hl num">1024</span><span class="hl opt">];</span>
ssize_t total_copied <span class="hl opt">=</span> <span class="hl num">0</span><span class="hl opt">;</span>
ssize_t ret<span class="hl opt">;</span>
<span class="hl kwa">for</span> <span class="hl opt">(;;) {</span>
ssize_t n_read<span class="hl opt">;</span>
ret <span class="hl opt">=</span> <span class="hl kwd">TEMP_FAILURE_RETRY</span><span class="hl opt">(</span><span class="hl kwd">read</span><span class="hl opt">(</span>srcfd<span class="hl opt">,</span> buf<span class="hl opt">,</span> <span class="hl kwa">sizeof</span><span class="hl opt">(</span>buf<span class="hl opt">)));</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">) {</span>
<span class="hl kwd">perror</span><span class="hl opt">(</span><span class="hl str">"Reading from source"</span><span class="hl opt">);</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
<span class="hl opt">}</span>
n_read <span class="hl opt">=</span> ret<span class="hl opt">;</span>
<span class="hl com">/* Reached the end of the file */</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>n_read <span class="hl opt">==</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">return</span> n_read<span class="hl opt">;</span>
<span class="hl kwa">while</span> <span class="hl opt">(</span>n_read <span class="hl opt">></span> <span class="hl num">0</span><span class="hl opt">) {</span>
ret <span class="hl opt">=</span> <span class="hl kwd">TEMP_FAILURE_RETRY</span><span class="hl opt">(</span><span class="hl kwd">write</span><span class="hl opt">(</span>tgtfd<span class="hl opt">,</span> buf<span class="hl opt">,</span> n_read<span class="hl opt">));</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt"><</span> <span class="hl num">0</span><span class="hl opt">) {</span>
<span class="hl kwd">perror</span><span class="hl opt">(</span><span class="hl str">"Writing to target"</span><span class="hl opt">);</span>
<span class="hl kwa">return</span> ret<span class="hl opt">;</span>
<span class="hl opt">}</span>
n_read <span class="hl opt">-=</span> ret<span class="hl opt">;</span>
total_copied <span class="hl opt">+=</span> ret<span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl opt">}</span>
<span class="hl kwa">return</span> <span class="hl num">0</span><span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl kwb">int</span> <span class="hl kwd">copy_file</span><span class="hl opt">(</span><span class="hl kwb">const char</span> <span class="hl opt">*</span>source<span class="hl opt">,</span> <span class="hl kwb">const char</span> <span class="hl opt">*</span>target<span class="hl opt">,</span> <span class="hl kwb">bool</span> no_clobber<span class="hl opt">) {</span>
<span class="hl kwb">int</span> srcfd <span class="hl opt">= -</span><span class="hl num">1</span><span class="hl opt">;</span>
<span class="hl kwb">int</span> tgtfd <span class="hl opt">= -</span><span class="hl num">1</span><span class="hl opt">;</span>
srcfd <span class="hl opt">=</span> <span class="hl kwd">open</span><span class="hl opt">(</span>source<span class="hl opt">,</span> O_RDONLY<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>srcfd <span class="hl opt">== -</span><span class="hl num">1</span><span class="hl opt">) {</span>
<span class="hl kwd">perror</span><span class="hl opt">(</span><span class="hl str">"Opening source file"</span><span class="hl opt">);</span>
<span class="hl kwa">return</span> srcfd<span class="hl opt">;</span>
<span class="hl opt">}</span>
tgtfd <span class="hl opt">=</span> <span class="hl kwd">open</span><span class="hl opt">(</span>target<span class="hl opt">,</span> O_WRONLY<span class="hl opt">|</span>O_CREAT<span class="hl opt">|(</span>no_clobber <span class="hl opt">?</span> O_EXCL <span class="hl opt">:</span> <span class="hl num">0</span><span class="hl opt">),</span> <span class="hl num">0600</span><span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>tgtfd <span class="hl opt">== -</span><span class="hl num">1</span><span class="hl opt">) {</span>
<span class="hl kwd">perror</span><span class="hl opt">(</span><span class="hl str">"Opening target file"</span><span class="hl opt">);</span>
<span class="hl kwa">return</span> tgtfd<span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl kwa">return</span> <span class="hl kwd">naive_contents_copy</span><span class="hl opt">(</span>srcfd<span class="hl opt">,</span> tgtfd<span class="hl opt">);</span>
<span class="hl opt">}</span>
</pre></div>
<h1>Making use of our new function</h1>
<p>So now we have a nice <code>move_file</code> function
that will fall back to copying it if renaming does not work.</p>
<p>But code is of no use in isolation,
we need a program for it to live in,
and the simplest way to use it is a command-line program.</p>
<div class="highlight-c"><pre class="hl"><span class="hl ppc">#include <getopt.h></span>
<span class="hl ppc">#include <string.h></span>
<span class="hl kwb">int</span> <span class="hl kwd">main</span><span class="hl opt">(</span><span class="hl kwb">int</span> argc<span class="hl opt">,</span> <span class="hl kwb">char</span> <span class="hl opt">*</span>argv<span class="hl opt">[]) {</span>
<span class="hl kwb">char</span> <span class="hl opt">*</span>source<span class="hl opt">;</span>
<span class="hl kwb">char</span> <span class="hl opt">*</span>target<span class="hl opt">;</span>
<span class="hl kwb">bool</span> no_clobber <span class="hl opt">=</span> <span class="hl kwa">false</span><span class="hl opt">;</span>
<span class="hl kwb">enum</span> opt <span class="hl opt">{</span>
OPT_NO_CLOBBER <span class="hl opt">=</span> <span class="hl str">'n'</span><span class="hl opt">,</span>
OPT_CLOBBER <span class="hl opt">=</span> <span class="hl str">'N'</span><span class="hl opt">,</span>
<span class="hl opt">};</span>
<span class="hl kwb">static const struct</span> option opts<span class="hl opt">[] = {</span>
<span class="hl opt">{ .</span>name <span class="hl opt">=</span> <span class="hl str">"no-clobber"</span><span class="hl opt">, .</span>has_arg <span class="hl opt">=</span> no_argument<span class="hl opt">, .</span>val <span class="hl opt">=</span> OPT_NO_CLOBBER<span class="hl opt">, },</span>
<span class="hl opt">{ .</span>name <span class="hl opt">=</span> <span class="hl str">"clobber"</span><span class="hl opt">, .</span>has_arg <span class="hl opt">=</span> no_argument<span class="hl opt">, .</span>val <span class="hl opt">=</span> OPT_CLOBBER<span class="hl opt">, },</span>
<span class="hl opt">{},</span>
<span class="hl opt">};</span>
<span class="hl kwa">for</span> <span class="hl opt">(;;) {</span>
<span class="hl kwb">int</span> ret <span class="hl opt">=</span> <span class="hl kwd">getopt_long</span><span class="hl opt">(</span>argc<span class="hl opt">,</span> argv<span class="hl opt">,</span> <span class="hl str">"nN"</span><span class="hl opt">,</span> opts<span class="hl opt">,</span> NULL<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>ret <span class="hl opt">== -</span><span class="hl num">1</span><span class="hl opt">)</span>
<span class="hl kwa">break</span><span class="hl opt">;</span>
<span class="hl kwa">switch</span> <span class="hl opt">(</span>ret<span class="hl opt">) {</span>
<span class="hl kwa">case</span> <span class="hl str">'?'</span><span class="hl opt">:</span>
<span class="hl kwa">return</span> <span class="hl num">1</span><span class="hl opt">;</span>
<span class="hl kwa">case</span> OPT_NO_CLOBBER<span class="hl opt">:</span>
<span class="hl kwa">case</span> OPT_CLOBBER<span class="hl opt">:</span>
no_clobber <span class="hl opt">= (</span>ret <span class="hl opt">==</span> OPT_NO_CLOBBER<span class="hl opt">);</span>
<span class="hl kwa">break</span><span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl opt">}</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>optind <span class="hl opt">==</span> argc <span class="hl opt">||</span> argc <span class="hl opt">></span> optind <span class="hl opt">+</span> <span class="hl num">2</span><span class="hl opt">) {</span>
<span class="hl kwd">fprintf</span><span class="hl opt">(</span>stderr<span class="hl opt">,</span> <span class="hl str">"1 or 2 positional arguments required</span><span class="hl esc">\n</span><span class="hl str">"</span><span class="hl opt">);</span>
<span class="hl kwa">return</span> <span class="hl num">2</span><span class="hl opt">;</span>
<span class="hl opt">}</span>
source <span class="hl opt">=</span> argv<span class="hl opt">[</span>optind<span class="hl opt">];</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>argc <span class="hl opt">==</span> optind <span class="hl opt">+</span> <span class="hl num">2</span><span class="hl opt">)</span>
target <span class="hl opt">=</span> argv<span class="hl opt">[</span>optind <span class="hl opt">+</span> <span class="hl num">1</span><span class="hl opt">];</span>
<span class="hl kwa">else</span>
<span class="hl com">/* Move into the current directory with the same name */</span>
target <span class="hl opt">=</span> <span class="hl kwd">basename</span><span class="hl opt">(</span>source<span class="hl opt">);</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span><span class="hl kwd">move_file</span><span class="hl opt">(</span>source<span class="hl opt">,</span> target<span class="hl opt">,</span> no_clobber<span class="hl opt">) >=</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl kwa">return</span> <span class="hl num">0</span><span class="hl opt">;</span>
<span class="hl kwa">return</span> <span class="hl num">1</span><span class="hl opt">;</span>
<span class="hl opt">}</span>
</pre></div>
<div class="highlight-sh"><pre class="hl">$ <span class="hl kwa">if</span> <span class="hl kwb">echo</span> <span class="hl str">'int main(){(void)renameat2;}'</span> | gcc <span class="hl kwb">-include</span> stdio.h <span class="hl kwb">-xc</span> <span class="hl opt">-</span> <span class="hl kwb">-o</span><span class="hl opt">/</span>dev<span class="hl opt">/</span>null <span class="hl num">2</span><span class="hl opt">>/</span>dev<span class="hl opt">/</span>null<span class="hl opt">;</span> <span class="hl kwa">then</span>
<span class="hl opt">></span> HAVE_DECL_RENAMEAT2<span class="hl opt">=</span><span class="hl num">1</span>
<span class="hl opt">></span> <span class="hl kwa">else</span>
<span class="hl opt">></span> HAVE_DECL_RENAMEAT2<span class="hl opt">=</span><span class="hl num">0</span>
<span class="hl opt">></span> <span class="hl kwa">fi</span>
$ <span class="hl kwc">make</span> CFLAGS<span class="hl opt">=</span><span class="hl str">"-D_GNU_SOURCE -DHAVE_DECL_RENAMEAT2=</span><span class="hl ipl">$HAVE_DECL_RENAMEAT2</span><span class="hl str">"</span> my-mv
$ .<span class="hl opt">/</span>my-mv
<span class="hl num">1</span> or <span class="hl num">2</span> positional arguments required
$ <span class="hl kwc">touch</span> test-file
$ .<span class="hl opt">/</span>my-mv test-file clobber-file
$ <span class="hl kwc">ls</span> test-file clobber-file
<span class="hl kwc">ls</span><span class="hl opt">:</span> cannot access test-file<span class="hl opt">:</span> No such <span class="hl kwc">file</span> or directory
clobber-file
$ .<span class="hl opt">/</span>my-mv <span class="hl kwb">-n</span> test-file clobber-file
rename2<span class="hl opt">:</span> No such <span class="hl kwc">file</span> or directory
$ <span class="hl kwc">touch</span> test-file
$ .<span class="hl opt">/</span>my-mv <span class="hl kwb">--no-clobber</span> test-file clobber-file
rename2<span class="hl opt">:</span> File exists
$ .<span class="hl opt">/</span>my-mv test-file clobber-file
$ <span class="hl kwc">ls</span> test-file clobber-file
<span class="hl kwc">ls</span><span class="hl opt">:</span> cannot access test-file<span class="hl opt">:</span> No such <span class="hl kwc">file</span> or directory
</pre></div>
<h1>So we've got a complete fallback for <a href="http://man7.org/linux/man-pages/man2/rename.2.html">rename(2)</a> now?</h1>
<p>Not quite.</p>
<p>For most purposes this is likely to be sufficient,
but there's a lot more to a file than the data you can read out of it,
far more than I can cover in this article,
so there will be follow-up articles to cover copying other aspects of files
including:</p>
<ol>
<li>Sparseness</li>
<li>Speed</li>
<li>Metadata</li>
<li>Atomicity</li>
<li>Other types of file</li>
</ol>