Change X-Robots-Tag header from "none" to "noindex, nofollow"

[MichaIng]

• Resolves: Support for multiple X-Robots-Tags #32102

Summary

While "none" is indeed equivalent to "noindex, nofollow" for Google, but seems to be not supported by Bing and probably other search engines.

https://developer.mozilla.org/en-US/docs/Web/HTML/Element/meta/name#other_metadata_names https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag?hl=de#comma-separated-list https://www.bing.com/webmasters/help/which-robots-metatags-does-bing-support-5198d240

TODO

• Check docs for equivalent change

Checklist

• Code is properly formatted
• Sign-off message is added to all commits
• Tests (unit, integration, api and/or acceptance) are included
• Screenshots before/after for front-end changes
• Documentation (manuals or wiki) has been updated or is not required
• Backports requested where applicable (ex: critical bugfixes)

[MichaIng]

/backport to stable25

[MichaIng]

/backport to stable24

[MichaIng]

This btw means that all admin, who did not have a warning before, do now see it, until their .htaccess or webserver config has been adjusted. Since some are sensitive about "being annoyed" by warnings, we might want to do something about it, though not sure what.

• There is no way to update the .htaccess on Nextcloud update automatically, is it?
• We could allow the none value as well, although knowing that it is ignored by Bing, this wouldn't be responsible.
• Show a different message if the current value is none, to better explain why this warning shows up now and not before? The special treatment could be removed after 1-2 Nextcloud versions, when we assume that every admin has seen it.

[szaimen]

• There is no way to update the .htaccess on Nextcloud update automatically, is it?

Actually the .htaccesss is delivered with the Nextcloud archive so it should usually be overwritten with the new one upon updating.

[MichaIng]

The bottom section is however a custom one and preserved on update, so it is not just replaced. I actually never check whether the top section is updated.

[szaimen]

The bottom section is however a custom one and preserved on update, so it is not just replaced.

It is not preserved upon update IIRC.

[MichaIng]

It is, or otherwise occ maintenance:update:htaccess (respectively the steps behind it) must run afterwards, since the directives for error documents and pretty URL (omitted index.php) with custom rewrite base (sub directory in my case) are always preserved here.

[szaimen]

I see. I thought the file would always get updated since it must have the latest changes...

[MichaIng]

Quite possible (and would be reasonable) if the bottom section of the old .htaccess would be appended to the new one, that way updating possibly required changes but preserving setup-depending and custom changes.

[nickvergessen]

Not my area of expertise, so removing my review as I'm overloaded at the moment

[kesselb]

Thank you 👍

[ChristophWurst]

Not an expert here either but looks good

Nginx doc should definitely be updated and this needs to be mentioned in the admin doc changes of 27

[MichaIng]

The .htaccess is indeed somehow radically updated. The upper part, as expected, but also custom changes below the #### DO NOT CHANGE ANYTHING ABOVE THIS LINE #### line are lost, at least this happened to some comments I added there for testing. Since the base path depending redirects are preserved, occ maintenance:update:htaccess is called, respectively the internal code behind it.

This means Apache users with AllowOverride All won't see any warning, but others will do. And all Nextcloud appliances of course need to react.

[st3iny]

Integration test failures do not seem to be related:

    �[31mAnd modify LDAP configuration�[39m         �[30m# LDAPContext::modifyLDAPConfiguration()�[39m
�[31m      | ldapExpertUsernameAttr | uid |�[39m
�[31m      Server error: `PUT http://localhost:8080/ocs/v2.php/apps/user_ldap/api/v1/config/s01` resulted in a `500 Internal Server Error` response:
      <?xml version="1.0"?>
      <ocs>
       <meta>
        <status>failure</status>
        <statuscode>500</statuscode>
        <message>Internal Server (truncated...)
       (GuzzleHttp\Exception\ServerException)�[39m

[szaimen]

CI failure unrelated

[DaphneMuller]

hello @MichaIng
Thank you so much for your work on this pull request! This ticket seems to have the tag 'missing documentation'.
Is there any chance you could clarify what documentation is missing? Is this for admins or for app developers?

[MichaIng]

Hi Daphne, the documentation change has been merged already, so not pending anymore: nextcloud/documentation#9635

[DaphneMuller]

thank you for your lightning fast reply and for removing the pending documentation label @MichaIng !❤️