Skip to content
This repository has been archived by the owner on Jul 5, 2021. It is now read-only.

Hotfix: Missing robots.txt #328

Closed
lidel opened this issue Aug 27, 2019 · 2 comments · Fixed by #334
Closed

Hotfix: Missing robots.txt #328

lidel opened this issue Aug 27, 2019 · 2 comments · Fixed by #334
Assignees
@lidel
Labels
dif/medium Prior experience is likely helpful topic/docs Documentation

Comments

@lidel
Copy link
Contributor

lidel commented Aug 27, 2019 •
edited
Loading

@andrew noted that Robots file (https://ipfs.io/robots.txt) is missing 🙃

We may want to add it and exclude things like non-canonical wikipedia snapshot (ipfs/distributed-wikipedia-mirror#48) from being crawled/cached by search engines, among other things.
@jessicaschilling jessicaschilling changed the title Missing robots.txt Hotfix: Missing robots.txt Aug 27, 2019
@jessicaschilling jessicaschilling added dif/easy Someone with a little familiarity can pick up topic/docs Documentation OKR: Hotfixes labels Aug 27, 2019
@jessicaschilling
Copy link
Contributor

Checking in with @cwaring on this (Chris, your feedback?) but this makes sense. @lidel, is this something you wouldn't mind taking on? Thank you so much!

@andrew
Copy link
Contributor

andrew commented Aug 27, 2019

It might be a con or it might be a pro, but that'll drop a lot of traffic to the gateways that come from search: https://www.alexa.com/siteinfo/ipfs.io

lidel added a commit that referenced this issue Aug 28, 2019
@lidel
fix: add robots.txt to exclude gateway paths
bc6103e 
Context: #328

License: MIT
Signed-off-by: Marcin Rataj <lidel@lidel.org>
@lidel lidel mentioned this issue Aug 28, 2019
Closed
@jessicaschilling jessicaschilling added dif/medium Prior experience is likely helpful and removed dif/easy Someone with a little familiarity can pick up labels Aug 28, 2019
@lidel lidel mentioned this issue Sep 2, 2019
Merged
@lidel lidel self-assigned this Sep 2, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@lidel lidel

Labels
dif/medium Prior experience is likely helpful topic/docs Documentation
Projects
None yet
Milestone
No milestone
3 participants
@andrew @lidel @jessicaschilling