Fix indexing for k8s.io

[chenopis]

Partial revert of #9225
Partial revert of #9703
Fixes kubernetes/kubernetes#67013
Fixes #9758

[k8sio-netlify-preview-bot]

Deploy preview for kubernetes-io-master-staging ready!

Built with commit 9973900

https://deploy-preview-9767--kubernetes-io-master-staging.netlify.com

[bep]

LGTM.

[mdlinville]

Isn't this comment still unclosed?

[chenopis]

Oh, that's actually not an opening comment delimiter. It's specifying that the header tag should be for all the pages in the site, similar to file path syntax on the command line, e.g. rm -rf /* LOL. See https://www.netlify.com/docs/headers-and-basic-auth/

[mdlinville]

It seems hard to test this before going live, so ....

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mistyhacks

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [mistyhacks]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[colek42]

Did this cause google to delist the docs on kubernetes.io?

[zacharysarah]

@colek42 This fixed the reason they were de-indexed. Re-indexing will take time.

[lucperkins]

@chenopis May I ask the reasoning behind this PR? There was already a system in place for generating a _headers file and that system has been effectively overwritten by this PR (see here and here especially) because the _headers file is simply copied into the public folder, overwriting the _headers file that is currently being generated by Hugo.

Also, in the future, if a static resource needs to end up in the public folder it should be put in static, not copied from another directory.

I think the best course of action here is to modify the existing _headers generation process to continue to allow for Netlify's HTTP/2 push feature, which was the initial impetus, rather than simply overriding that process.

[zacharysarah]

@lucperkins 👋

May I ask the reasoning behind this PR?

Search results for K8s docs were de-indexed on Google (#9758, #9783), due at least partially to changes introduced in #9225. Any further iterations to _headers generation should be sensitive to the possibility of unintended downstream changes.

[chenopis]

@lucperkins Previously, we had to address the problem of other similar published branches (e.g. https://v1-9.docs.kubernetes.io/) from being indexed by Google search, which diluted the results of the main https://kubernetes.io site. Consequently, I set up a system to add the X-Robots-Tag: noindex header directive to all the branches by default via the build command. However, the master branch for the main site uses a different build context so that it does not add the noindex directive.

I'm not saying this is the best way to solve it, but I had to revert some of @bep and your changes to address the immediate de-indexing problems that were getting reported. Perhaps at the next SIG Docs meeting we can discuss a better way to handle this issue so that it still uses the HTTP/2 push feature but only adds the noindex directive to the non-master branches.