Block internet search engines from indexing the mirror

[bardiharborow]

If possible, are you able to make your mirror non-indexed by internet search engines? There is very minimal benefit for clearnet users to run across three (WMF, WikiVisually and ipfs) different copies of the Wikipedia article every time they search for something.

[Stebalien]

There is very minimal benefit for clearnet users to run across three (WMF, WikiVisually and ipfs) different copies of the Wikipedia article every time they search for something.

The benefit is entirely for clearnet users. Tor users, for example, will (almost) always be able to access Wikipedia over tor so they'll see little benefit.

We should probably add a rel=canonical link pointing to Wikipedia to the head of each page but I haven't thought through the possible ramifications/downsides of this approach.

[bardiharborow]

The benefit is entirely for clearnet users. Tor users, for example, will (almost) always be able to access Wikipedia over tor so they'll see little benefit.

If anonymity is the concern, then accessing IPFS through the ipfs.io endpoint is no more anonymous to large scale surveillance than accessing Wikipedia directly, and if anything I trust the Wikimedia Foundation to handle server logs better than ipfs.io. Users of the actual IPFS software will presumably discover the mirror through different means than Google, and will not be impacted by this change.

We should probably add a rel=canonical link pointing to Wikipedia to the head of each page but I haven't thought through the possible ramifications/downsides of this approach.

Doing so would have the intended effect of removing the mirror from Google search results, and it is actually the preferred way to implement this.

[Stebalien]

@bardiharborow

If anonymity is the concern, then accessing IPFS through the ipfs.io endpoint is no more anonymous to large scale surveillance than accessing Wikipedia directly, and if anything I trust the Wikimedia Foundation to handle server logs better than ipfs.io. Users of the actual IPFS software will presumably discover the mirror through different means than Google, and will not be impacted by this change.

Ah, I think the confusion may be around the definition of "clearnet". IPFS is a clearnet. That is, it's not a darknet (it provides no anonymity at the moment). Darknets get no benefit because the exit nodes tend to be in countries with strong free speech laws.

Users of the actual IPFS software will presumably discover the mirror through different means than Google, and will not be impacted by this change.

Unlikely. We don't have any IPFS search mechanisms and rely entirely on web search engines. That's probably one of the reasons we don't use rel=canonical links.

[rameshvarun]

+1 for setting rel='canonical' links. I'm starting to see the mirror pop up frequently on the first page of Google results just from normal everyday use. Canonical links should avoid this duplication and make the mirror a good web citizen.

[nemobis]

I agree with adding the rel="canonical": it's annoying to see search duplicates. By not indexing outdated content, you'll also alleviate the concerns with other issues such as #55 #49 .

Actually, what's the purpose of indexing all the pages at all? A noindex meta tag may be appropriate.

[wesleylima]

The lack of canonical tag comes from the htmls generated by kiwix's mwoffiler. I opened an issue openzim/mwoffliner#564

[nemobis]

The lack of canonical tag comes from the htmls generated by kiwix's mwoffiler.

I understand, but you can also add a canonical link in the webserver response headers.

[lidel]

I fixed this upstream (openzim/mwoffliner#963) 👌
Old snapshots are about to be excluded via /robots.txt (ipfs-inactive/website#334)

Remaining steps before this issue can be closed:

• mwoffliner 1.9 to be released with the fix
• wiki snapshots at http://wiki.kiwix.org/wiki/Content_in_all_languages are made with updated mwoffliner and include <link rel="canonical"
• snapshots are put on ipfs + pinned on a reliable cluster
• snapshot-hashes.yml are updated to versions with canonical links

OR:

• While Add canonical link header openzim/mwoffliner#963 solves problem for new snapshots, it is still possible the script will be run against an old ZIM without the header. Before adding to IPFS the script should check if root document contains the header, and if not manually add it to every document.
  Filled Add rel="canonical" for search engines #65 to track this

I will be checking on mwoffliner/kiwix situation, but if someone has spare bandwidth and can to speed things up, please contribute upstream & post updates here.

[lidel]

This has been fixed by #65 and will be solved upstream when new snapshots are published as part of #60 #61.