net/http: improve FileServer error logging

[hullarb]

The errors returned during copying the file content in ServeContent,
ServeFile and FileServer were ignored to avoid excessive, meaningless
logging. This commit introduces error filtering and logging to ensure
that the errors occurring during reading the content are logged.

Updates #27128

[googlebot]

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it!) and we'll verify it.

---

What to do if you already signed the CLA

Individual signers

• It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

• Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
• The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
• The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

[hullarb]

@googlebot I signed it!

[googlebot]

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

[gopherbot]

This PR (HEAD: f66c1bb) has been imported to Gerrit for code review.

Please visit https://go-review.googlesource.com/c/go/+/191971 to see it.

Tip: You can toggle comments from me using the comments slash command (e.g. /comments off)
See the Wiki page for more info

[gopherbot]

Message from Gobot Gobot:

Patch Set 1:

Congratulations on opening your first change. Thank you for your contribution!

Next steps:
Within the next week or so, a maintainer will review your change and provide
feedback. See https://golang.org/doc/contribute.html#review for more info and
tips to get your patch through code review.

Most changes in the Go project go through a few rounds of revision. This can be
surprising to people new to the project. The careful, iterative review process
is our way of helping mentor contributors and ensuring that their contributions
have a lasting impact.

During May-July and Nov-Jan the Go project is in a code freeze, during which
little code gets reviewed or merged. If a reviewer responds with a comment like
R=go1.11, it means that this CL will be reviewed as part of the next development
cycle. See https://golang.org/s/release for more details.

---

Please don’t reply on this GitHub thread. Visit golang.org/cl/191971.
After addressing review feedback, remember to publish your drafts!

[gopherbot]

Message from Brad Fitzpatrick:

Patch Set 1: Code-Review-1

This would need a cleaned up commit message (read the instructions in them) and a test, but this change would require a lot more work.

This intentionally doesn't log because the majority of the failures logged would be because users closed their tab and broke their TCP connection. That's not worth logging about.

Failure to read from the source side is more interesting, though. Let me know if you're still interested in working on it.

---

Please don’t reply on this GitHub thread. Visit golang.org/cl/191971.
After addressing review feedback, remember to publish your drafts!

[gopherbot]

Message from Béla Hullár:

Patch Set 1:

Patch Set 1: Code-Review-1

This would need a cleaned up commit message (read the instructions in them) and a test, but this change would require a lot more work.

This intentionally doesn't log because the majority of the failures logged would be because users closed their tab and broke their TCP connection. That's not worth logging about.

Failure to read from the source side is more interesting, though. Let me know if you're still interested in working on it.

Thanks for the feedback.
I see the original intention and agree that the write related errors are far less interesting, although i think in some situation it can be interesting too.

I'm still happy to work on this. Could you give me further feedback what kind of solution would you like to see? do you think we could set a debug mode from env var like a GODEBUG flag to turn on this error logging?

---

Please don’t reply on this GitHub thread. Visit golang.org/cl/191971.
After addressing review feedback, remember to publish your drafts!

[gopherbot]

Message from Brad Fitzpatrick:

Patch Set 1:

Patch Set 1:

Patch Set 1: Code-Review-1

This would need a cleaned up commit message (read the instructions in them) and a test, but this change would require a lot more work.

This intentionally doesn't log because the majority of the failures logged would be because users closed their tab and broke their TCP connection. That's not worth logging about.

Failure to read from the source side is more interesting, though. Let me know if you're still interested in working on it.

Thanks for the feedback.
I see the original intention and agree that the write related errors are far less interesting, although i think in some situation it can be interesting too.

I'm still happy to work on this. Could you give me further feedback what kind of solution would you like to see? do you think we could set a debug mode from env var like a GODEBUG flag to turn on this error logging?

No, no new environment variables. Use the https://golang.org/pkg/net/http/#Server.ErrorLog (see the logf func) for logging, but only log if it's not a broken network connection as the reason.

---

Please don’t reply on this GitHub thread. Visit golang.org/cl/191971.
After addressing review feedback, remember to publish your drafts!

[gopherbot]

Message from Béla Hullár:

Patch Set 1:

Patch Set 1:

Patch Set 1:

Patch Set 1: Code-Review-1

This would need a cleaned up commit message (read the instructions in them) and a test, but this change would require a lot more work.

This intentionally doesn't log because the majority of the failures logged would be because users closed their tab and broke their TCP connection. That's not worth logging about.

Failure to read from the source side is more interesting, though. Let me know if you're still interested in working on it.

Thanks for the feedback.
I see the original intention and agree that the write related errors are far less interesting, although i think in some situation it can be interesting too.

I'm still happy to work on this. Could you give me further feedback what kind of solution would you like to see? do you think we could set a debug mode from env var like a GODEBUG flag to turn on this error logging?

No, no new environment variables. Use the https://golang.org/pkg/net/http/#Server.ErrorLog (see the logf func) for logging, but only log if it's not a broken network connection as the reason.

OK, then replacing the nil check with a type check, like this
_, err := io.CopyN(w, sendContent, sendSize)
if _, ok := err.(net.Error); err != nil && !ok {
logf(r, "http: error copy content of %s: %v", name, err)
}
would do it, right?

---

Please don’t reply on this GitHub thread. Visit golang.org/cl/191971.
After addressing review feedback, remember to publish your drafts!

[gopherbot]

Message from Brad Fitzpatrick:

Patch Set 1:

Patch Set 1:

Patch Set 1:

Patch Set 1:

Patch Set 1: Code-Review-1

This would need a cleaned up commit message (read the instructions in them) and a test, but this change would require a lot more work.

This intentionally doesn't log because the majority of the failures logged would be because users closed their tab and broke their TCP connection. That's not worth logging about.

Failure to read from the source side is more interesting, though. Let me know if you're still interested in working on it.

Thanks for the feedback.
I see the original intention and agree that the write related errors are far less interesting, although i think in some situation it can be interesting too.

I'm still happy to work on this. Could you give me further feedback what kind of solution would you like to see? do you think we could set a debug mode from env var like a GODEBUG flag to turn on this error logging?

No, no new environment variables. Use the https://golang.org/pkg/net/http/#Server.ErrorLog (see the logf func) for logging, but only log if it's not a broken network connection as the reason.

OK, then replacing the nil check with a type check, like this
_, err := io.CopyN(w, sendContent, sendSize)
if _, ok := err.(net.Error); err != nil && !ok {
logf(r, "http: error copy content of %s: %v", name, err)
}
would do it, right?

io.Copy (or io.CopyN) don't distinguish between whether the error was from the read size or the write side.

So that is likely not enough.

I'd also pull it out into a function (or two) to make it more readable & testable.

---

Please don’t reply on this GitHub thread. Visit golang.org/cl/191971.
After addressing review feedback, remember to publish your drafts!

[gopherbot]

Message from Béla Hullár:

Patch Set 1:

Patch Set 1:

Patch Set 1:

Patch Set 1:

Patch Set 1:

Patch Set 1: Code-Review-1

This would need a cleaned up commit message (read the instructions in them) and a test, but this change would require a lot more work.

This intentionally doesn't log because the majority of the failures logged would be because users closed their tab and broke their TCP connection. That's not worth logging about.

Failure to read from the source side is more interesting, though. Let me know if you're still interested in working on it.

Thanks for the feedback.
I see the original intention and agree that the write related errors are far less interesting, although i think in some situation it can be interesting too.

I'm still happy to work on this. Could you give me further feedback what kind of solution would you like to see? do you think we could set a debug mode from env var like a GODEBUG flag to turn on this error logging?

No, no new environment variables. Use the https://golang.org/pkg/net/http/#Server.ErrorLog (see the logf func) for logging, but only log if it's not a broken network connection as the reason.

OK, then replacing the nil check with a type check, like this
_, err := io.CopyN(w, sendContent, sendSize)
if _, ok := err.(net.Error); err != nil && !ok {
logf(r, "http: error copy content of %s: %v", name, err)
}
would do it, right?

io.Copy (or io.CopyN) don't distinguish between whether the error was from the read size or the write side.

So that is likely not enough.

I'd also pull it out into a function (or two) to make it more readable & testable.

Right, as FileSystem/File is an interface we cannot be sure where the net.Error comes from.
What if we wrap the writer with a thin wrapper that would register the last error?

---

Please don’t reply on this GitHub thread. Visit golang.org/cl/191971.
After addressing review feedback, remember to publish your drafts!

[gopherbot]

Message from Béla Hullár:

Patch Set 1:

Patch Set 1:

Patch Set 1:

Patch Set 1:

Patch Set 1:

Patch Set 1:

Patch Set 1: Code-Review-1

This would need a cleaned up commit message (read the instructions in them) and a test, but this change would require a lot more work.

This intentionally doesn't log because the majority of the failures logged would be because users closed their tab and broke their TCP connection. That's not worth logging about.

Failure to read from the source side is more interesting, though. Let me know if you're still interested in working on it.

Thanks for the feedback.
I see the original intention and agree that the write related errors are far less interesting, although i think in some situation it can be interesting too.

I'm still happy to work on this. Could you give me further feedback what kind of solution would you like to see? do you think we could set a debug mode from env var like a GODEBUG flag to turn on this error logging?

No, no new environment variables. Use the https://golang.org/pkg/net/http/#Server.ErrorLog (see the logf func) for logging, but only log if it's not a broken network connection as the reason.

OK, then replacing the nil check with a type check, like this
_, err := io.CopyN(w, sendContent, sendSize)
if _, ok := err.(net.Error); err != nil && !ok {
logf(r, "http: error copy content of %s: %v", name, err)
}
would do it, right?

io.Copy (or io.CopyN) don't distinguish between whether the error was from the read size or the write side.

So that is likely not enough.

I'd also pull it out into a function (or two) to make it more readable & testable.

Right, as FileSystem/File is an interface we cannot be sure where the net.Error comes from.
What if we wrap the writer with a thin wrapper that would register the last error?

Sorry, i meant wrapping the Reader

---

Please don’t reply on this GitHub thread. Visit golang.org/cl/191971.
After addressing review feedback, remember to publish your drafts!

[gopherbot]

This PR (HEAD: 7976f48) has been imported to Gerrit for code review.

Please visit https://go-review.googlesource.com/c/go/+/191971 to see it.

Tip: You can toggle comments from me using the comments slash command (e.g. /comments off)
See the Wiki page for more info

[gopherbot]

Message from Brad Fitzpatrick:

Patch Set 2:

(2 comments)

---

Please don’t reply on this GitHub thread. Visit golang.org/cl/191971.
After addressing review feedback, remember to publish your drafts!

[gopherbot]

This PR (HEAD: 2ae816e) has been imported to Gerrit for code review.

Please visit https://go-review.googlesource.com/c/go/+/191971 to see it.

Tip: You can toggle comments from me using the comments slash command (e.g. /comments off)
See the Wiki page for more info

[gopherbot]

This PR (HEAD: 3e18e1a) has been imported to Gerrit for code review.

Please visit https://go-review.googlesource.com/c/go/+/191971 to see it.

Tip: You can toggle comments from me using the comments slash command (e.g. /comments off)
See the Wiki page for more info

[gopherbot]

Message from Gobot Gobot:

Patch Set 4: TryBot-Result-1

3 of 19 TryBots failed:
Failed on linux-amd64: https://storage.googleapis.com/go-build-log/cf7a008c/linux-amd64_e1567ab9.log
Failed on linux-386: https://storage.googleapis.com/go-build-log/cf7a008c/linux-386_31a415a5.log
Failed on linux-amd64-race: https://storage.googleapis.com/go-build-log/cf7a008c/linux-amd64-race_d9a6259e.log

Consult https://build.golang.org/ to see whether they are new failures. Keep in mind that TryBots currently test exactly your git commit, without rebasing. If your commit's git parent is old, the failure might've already been fixed.

---

Please don’t reply on this GitHub thread. Visit golang.org/cl/191971.
After addressing review feedback, remember to publish your drafts!

[gopherbot]

Message from Gobot Gobot:

Patch Set 5:

TryBots beginning. Status page: https://farmer.golang.org/try?commit=815b0392

---

Please don’t reply on this GitHub thread. Visit golang.org/cl/191971.
After addressing review feedback, remember to publish your drafts!

[gopherbot]

Message from Gobot Gobot:

Patch Set 5:

Build is still in progress...
This change failed on linux-amd64:
See https://storage.googleapis.com/go-build-log/815b0392/linux-amd64_90871167.log

Other builds still in progress; subsequent failure notices suppressed until final report. Consult https://build.golang.org/ to see whether they are new failures. Keep in mind that TryBots currently test exactly your git commit, without rebasing. If your commit's git parent is old, the failure might've already been fixed.

---

Please don’t reply on this GitHub thread. Visit golang.org/cl/191971.
After addressing review feedback, remember to publish your drafts!

[gopherbot]

Message from Brad Fitzpatrick:

Patch Set 5: -Code-Review

(1 comment)

---

Please don’t reply on this GitHub thread. Visit golang.org/cl/191971.
After addressing review feedback, remember to publish your drafts!

[gopherbot]

Message from Gobot Gobot:

Patch Set 5: TryBot-Result-1

3 of 19 TryBots failed:
Failed on linux-amd64: https://storage.googleapis.com/go-build-log/815b0392/linux-amd64_90871167.log
Failed on linux-386: https://storage.googleapis.com/go-build-log/815b0392/linux-386_59207c60.log
Failed on linux-amd64-race: https://storage.googleapis.com/go-build-log/815b0392/linux-amd64-race_11bb583c.log

Consult https://build.golang.org/ to see whether they are new failures. Keep in mind that TryBots currently test exactly your git commit, without rebasing. If your commit's git parent is old, the failure might've already been fixed.

---

Please don’t reply on this GitHub thread. Visit golang.org/cl/191971.
After addressing review feedback, remember to publish your drafts!

[gopherbot]

Message from Béla Hullár:

Patch Set 5:

(1 comment)

Patch Set 5: -Code-Review

(1 comment)

---

Please don’t reply on this GitHub thread. Visit golang.org/cl/191971.
After addressing review feedback, remember to publish your drafts!

[gopherbot]

This PR (HEAD: df595d2) has been imported to Gerrit for code review.

Please visit https://go-review.googlesource.com/c/go/+/191971 to see it.

Tip: You can toggle comments from me using the comments slash command (e.g. /comments off)
See the Wiki page for more info

[gopherbot]

Message from Béla Hullár:

Patch Set 6:

sorry, i pushed early, wrapping the writer makes the same test fail

---

Please don’t reply on this GitHub thread. Visit golang.org/cl/191971.
After addressing review feedback, remember to publish your drafts!

[gopherbot]

Message from Béla Hullár:

Patch Set 6:

Unfortunately it seems that this send file optimization relies on the concrete types on multiple levels. The writer needs to implement ReadFrom (to make Copy use ReadFrom of type response), response.ReadFrom checks for regular file checking the type of the reader (here i can add the errorSaverReader to check the wrapped reader), but then the underlying TCPCon from package net also checks whether the Reader is a file or a LimitedReader, otherwise it skips the send file invocation

---

Please don’t reply on this GitHub thread. Visit golang.org/cl/191971.
After addressing review feedback, remember to publish your drafts!

[gopherbot]

Message from Go Bot:

Patch Set 1:

Congratulations on opening your first change. Thank you for your contribution!

Next steps:
Within the next week or so, a maintainer will review your change and provide
feedback. See https://golang.org/doc/contribute.html#review for more info and
tips to get your patch through code review.

Most changes in the Go project go through a few rounds of revision. This can be
surprising to people new to the project. The careful, iterative review process
is our way of helping mentor contributors and ensuring that their contributions
have a lasting impact.

During May-July and Nov-Jan the Go project is in a code freeze, during which
little code gets reviewed or merged. If a reviewer responds with a comment like
R=go1.11, it means that this CL will be reviewed as part of the next development
cycle. See https://golang.org/s/release for more details.

---

Please don’t reply on this GitHub thread. Visit golang.org/cl/191971.
After addressing review feedback, remember to publish your drafts!

[gopherbot]

Message from Go Bot:

Patch Set 4:

TryBots beginning. Status page: https://farmer.golang.org/try?commit=cf7a008c

---

Please don’t reply on this GitHub thread. Visit golang.org/cl/191971.
After addressing review feedback, remember to publish your drafts!

[gopherbot]

Message from Go Bot:

Patch Set 4:

Build is still in progress...
This change failed on linux-amd64:
See https://storage.googleapis.com/go-build-log/cf7a008c/linux-amd64_e1567ab9.log

Other builds still in progress; subsequent failure notices suppressed until final report. Consult https://build.golang.org/ to see whether they are new failures. Keep in mind that TryBots currently test exactly your git commit, without rebasing. If your commit's git parent is old, the failure might've already been fixed.

---

Please don’t reply on this GitHub thread. Visit golang.org/cl/191971.
After addressing review feedback, remember to publish your drafts!

[gopherbot]

Message from Go Bot:

Patch Set 4: TryBot-Result-1

3 of 19 TryBots failed:
Failed on linux-amd64: https://storage.googleapis.com/go-build-log/cf7a008c/linux-amd64_e1567ab9.log
Failed on linux-386: https://storage.googleapis.com/go-build-log/cf7a008c/linux-386_31a415a5.log
Failed on linux-amd64-race: https://storage.googleapis.com/go-build-log/cf7a008c/linux-amd64-race_d9a6259e.log

Consult https://build.golang.org/ to see whether they are new failures. Keep in mind that TryBots currently test exactly your git commit, without rebasing. If your commit's git parent is old, the failure might've already been fixed.

---

Please don’t reply on this GitHub thread. Visit golang.org/cl/191971.
After addressing review feedback, remember to publish your drafts!

[gopherbot]

Message from Go Bot:

Patch Set 5:

TryBots beginning. Status page: https://farmer.golang.org/try?commit=815b0392

---

Please don’t reply on this GitHub thread. Visit golang.org/cl/191971.
After addressing review feedback, remember to publish your drafts!

[gopherbot]

Message from Go Bot:

Patch Set 5:

Build is still in progress...
This change failed on linux-amd64:
See https://storage.googleapis.com/go-build-log/815b0392/linux-amd64_90871167.log

Other builds still in progress; subsequent failure notices suppressed until final report. Consult https://build.golang.org/ to see whether they are new failures. Keep in mind that TryBots currently test exactly your git commit, without rebasing. If your commit's git parent is old, the failure might've already been fixed.

---

Please don’t reply on this GitHub thread. Visit golang.org/cl/191971.
After addressing review feedback, remember to publish your drafts!

[gopherbot]

Message from Go Bot:

Patch Set 5: TryBot-Result-1

3 of 19 TryBots failed:
Failed on linux-amd64: https://storage.googleapis.com/go-build-log/815b0392/linux-amd64_90871167.log
Failed on linux-386: https://storage.googleapis.com/go-build-log/815b0392/linux-386_59207c60.log
Failed on linux-amd64-race: https://storage.googleapis.com/go-build-log/815b0392/linux-amd64-race_11bb583c.log

Consult https://build.golang.org/ to see whether they are new failures. Keep in mind that TryBots currently test exactly your git commit, without rebasing. If your commit's git parent is old, the failure might've already been fixed.

---

Please don’t reply on this GitHub thread. Visit golang.org/cl/191971.
After addressing review feedback, remember to publish your drafts!