Accelerated local copy (reflink (CoW block cloning), alternatively in-kernel copy) when available

[tleedjarv]

This is an interesting experiment which, if deemed safe and working properly, can potentially offer huge gains in some situations.

This patch makes use of various OS and filesystem-specific syscalls to accelerate local file copies, making so-called reflink (COW) copies, if possible.

The following syscalls are tried:

• clonefile(2) on macOS (version >= 10.12); supported on APFS
• ioctl FICLONE on Linux (kernel >= 4.5); several filesystems are supported, at least BTRFS and XFS
  (also BTRFS_IOC_CLONE since Linux 2.6.29(?) should work)
• copy_file_range(2) on Linux (kernel >= 4.5) and FreeBSD >= 13
• sendfile(2) on Linux (kernel >= 2.2) and sendfile(3ext) on Solaris
• ioctl FSCTL_DUPLICATE_EXTENTS_TO_FILE on Windows (since Windows Server 2016 and at least Windows 11 (likely Windows 10)); supported on ReFS (including Dev Drive)

Fallback to read-write loop (the current copy function) is used if none of the above are available or supported by the filesystem.

The first three bullets provide special optimizations, while sendfile() is more of a question mark. It's supposedly more efficient due to not having to move data to userspace but I don't know if it results in any practical gains here.

Compilation on OS X < 10.12 is still possible but binaries produced by GHA CI will not be able to run on those versions. I don't know if it's going to be a problem for users.

[tleedjarv]

Rebased, with minor fixes and improvements.

[tleedjarv]

Included copy_file_range(2) support for FreeBSD >= 13.

[gdt]

FWIW, I think macOS < 10.13 is totally irrelevant now.

[tleedjarv]

Rebased, added Windows support, and other minor fixes and improvements.

The feature is ready. More testing is always welcome, of course.

The only question now is, if this feature is to be merged, should it be gated behind a user preference? Given your OS and filesystem support it, why would anyone actively disable reflinking?

[gdt]

I don't see a need for a preference. The only reason you'd want to disable it is if there is a bug in the new unison code, or in the operating system.

Presumably the existing tests would expose problems, as they do things that result in copying.

With sendfile, this is no longer reflink/COW. That's ok, but it's misnamed. It's really "use OS support for copying files" in the headline, with "OS support may involve references and copy-on-write, which should be more efficient. It may also involve data copying by the kernel instead of userland, which also should be more efficient than userland copying". That's all great, but no longer centered on reflink.

I figured it out quickly, but perhaps make it louder in the commit message and headline that it is about optimizing copying a file on a system, and not related to using a file that exists on the remote side in lieu of transferring it (which I think is also, or will be, on the table).

[tleedjarv]

With sendfile, this is no longer reflink/COW. That's ok, but it's misnamed. It's really "use OS support for copying files" in the headline, with "OS support may involve references and copy-on-write, which should be more efficient. It may also involve data copying by the kernel instead of userland, which also should be more efficient than userland copying". That's all great, but no longer centered on reflink.

All very true. I even thought about mentioning it but changed my mind. Now I can confirm that we have the same understanding.

[tleedjarv]

perhaps make it louder in the commit message and headline that it is about optimizing copying a file on a system, and not related to using a file that exists on the remote side in lieu of transferring it (which I think is also, or will be, on the table).

I think the wording in commit message is rather accurate (as it doesn't mention reflinks or CoW and is very specific about "local copies"). (Also see commit messages in #876, which complete the picture.)
The PR title is not as accurate but that's easy to change. Let me know if you have specific wording ideas you'd like to change in the commit message specifically.

[gdt]

The PR title is not as accurate but that's easy to change. Let me know if you have specific wording ideas you'd like to change in the commit message specifically.

Replacing the PR title with the single commit headline seems fine.

[gdt]

During review by unison-hackers@, I would like someone familiar with macOS to check this guard expression. I think the point is to find out if the target SDK is >= 10.12, and if so use clonefile, and if not, don't. That means that targeting 10.11 would fail to use clonefile on modern systems, but that's ok because it does not make sense to target 10.11 in 2025, unless you are building your own unison for your ancient (2008?) machine.

It seems possible to switch at runtime, but that feels like pointless work, given we're only talking about machines that are ~17 years old. (Macs from 2010 can run 10.13.)

[tleedjarv]

I don't know the correct terminology but this is not necessarily the target SDK. This is checking the compilation SDK (which I understand can still allow running on older systems (the target), but maybe I'm completely mistaken). This is a compile-time guard because we don't have <sys/clonefile.h> in < 10.12 and, ignoring that, the linker would probably not be happy. My understanding is that this guard needs to be in place no matter what.

But compiling in >= 10.12 with SDK >= 10.12, it could be possible to add a runtime switch when MAC_OS_X_VERSION_MIN_REQUIRED is defined to be < 10.12. That would theoretically instruct the linker to mark clonefile as a weak import. I can't verify it because the only macOS I have access to is the GHA runners.
If MAC_OS_X_VERSION_MIN_REQUIRED is not explicitly defined then it is implicitly set by AvailabilityMacros.h based on the target SDK, which in the GHA CI was recently set to 10.13 by 0ec31df, alternatively to whatever the default minimum is in AvailabilityMacros.h.

In summary, for GHA-built binaries this no longer matters anyway (as the minimum is set to 10.13).
For those building themselves, I don't mind adding a runtime switch if someone could test if it actually worked (not sure if clonefile can be weakly linked or not) and if the check is as simple as if (clonefile)

[gdt]

It seems on mac one chooses a target version, and then you get a binary that is like what a native build on that system would have produced, were macs normal. But there is a further concept of being able to run a binary on a system that is older, rather than justt the more normal situation of newer systems running older binaries. I'm not clear on things beyond that.

I don't think it makes a lot of sense to support the complexities of runtime switches. I'm fine with "if built for 10.11 or earlier, don't try this, and if built for 10.12 or later, rely on it. I can't really see normal usage building for anything older than 10.13. But still - if there is someone who is a mac SDK expert, I'd like to hear from them, is all I meant.

[tleedjarv]

Did some minor refactoring, no functional changes.