fix(gateway)!: no duplicate payload during subdomain redirects

[hacdias]

This is based on top of #315 since that PR also has to deal with this.

Right now we send the content body on requests that include redirection. This decision was made a long time ago to ensure that tools, such as curl and wget would be able to fetch on localhost and not be redirected to subdomains that may not work with them. However, considering the published gateway standards we have, this may be a legacy thing we can clean up.

Unfortunately, we have sharness and conformance tests that check for the body on redirection requests. Those will need to be updated in order for this to work.

cc @lidel

[lidel]

@hacdias Yes, this is expected -- these tests are there to ensure we don't change this silently until we are ready.

I think we are ready, we should stop doing this and remove it from conformance, don't want others to reimplement this hack.
It is wasting bandwidth and/or increasing latency on clients that are not able to drop connection after receiving Location header.

• Update sharness and conformance to no longer return body on redirects
• Just in case anyone relies on old behavior, the body should include a text/plain or text/html message informing user about redirect and URL of the new location. This way if any automation fails, it will be easy for people to debug and fix (Location header is often not logger and if body is empty, nothing wil show up)
  • http.Redirect does that: https://pkg.go.dev/net/http#Redirect

[hacdias]

@lidel:

• Gateway conformance: fix: remove body check for subdomain redirection gateway-conformance#65
  • I would merge this first and ensure this is running everywhere so we can ensure it's ✅
• Sharness: fix(gateway)!: no duplicate payload during subdomain redirects kubo#9913

[lidel]

Lgtm, thank you for cleaning this up! ❤️
I'm merging and bubbling up to ipfs/kubo#9913