io: copy fails with EOPNOTSUPP on NFSv4.1

[adam-azarchs]

What version of Go are you using (go version)?

$ go version
go version go1.15 linux/amd64

Does this issue reproduce with the latest release?

Yep.

What operating system and processor architecture are you using?

linux/amd64

This seems to be an issue specific to RHEL/CentOS 7.8 (and possibly other RHEL/CentOS 7 versions), and specifically with copying between files on the same NFSv4 filesystem.

What did you do?

In a directory that is on an NFSv4.1 mount on a host running CentOS 7.8

src, err := os.Open("src.txt")
if err != nil {
    panic(err)
}
dst, err := os.Create("dst.txt")
if err != nil {
    panic(err)
}
_, err:= io.Copy(dst, src)
if err != nil {
    panic(err)
}

What did you expect to see?

No error.

What did you see instead?

Error EOPNOTSUPP. In some cases it can be intermittent, depending on server behavior.

The copy_file_range syscall is supposed to fail with ENOSYS, EXDEV, or EINVAL if there isn't appropriate support from the kernel/filesystem, in which case internal/poll.CopyFileRange will return a value indicating that the copy was not handled, and copying will proceed with a fallback implementation from there.

Unfortunately, for some versions of NFSv4, including the implementation present on CentOS/RHEL 7, if the server does not support range copies, NFSv4 will instead return EOPNOTSUPP, and due to a workaround in the kernel used by CentOS/RHEL 7.8.

There is a nearly identical issue open for the Rust standard library, which unfortunately some of our customers have already encountered. We haven't started shipping anything with Go 1.15 yet.

The conclusion of the issue thread in Rust appears to be that the best thing to do here is treat EOPNOTSUPP as the same as ENOSYS. I personally agree with that conclusion. Pedantically speaking the bug is definitely in those kernels, not in Go, but unfortunately CentOS 7 is far too popular to ignore.

[davecheney]

At the rate we have to work around broken kernels, I wonder if copy_file_range will be used at all.

[mvdan]

cc @thoeni

[thoeni]

Happy to create a CL for this one, I think I'm now familiar with the issue.

Error EOPNOTSUPP. In some cases it can be intermittent, depending on server behavior.

As a clarification on this one, I understand if it's not supported but the underlying kernel we should store this in the atomic value for future reuse instead of checking all the times, but I'm not sure on the intermittent nature of the issue...

EOPNOTSUPP seems to map to the message "operation not supported on socket" 🤔

[thoeni]

Also, I just noticed that this has been already addressed by: #40731

I got confused between the two EOPNOTSUPP and ENOTSUP in the title of the other issue. I'd say this will be fixed/resolved with the next release? Here is the CL with the changes.

[adam-azarchs]

Yes, this does seem to duplicate that issue.