add basic permissive robots.txt

[megahirt]

Fixes #1776

Description

It appears that one of the reasons languageforge.org does not show up in google search results is the absence of a robots.txt file (who knew?)

Screenshots

From google search console:

Checklist

• I have labeled my PR with: bug, feature, engineering, security fix or testing
• I have performed a self-review of my own code
• I have reviewed the title & description of this PR which I will use as the squashed PR commit message
• I have commented my code, particularly in hard-to-understand areas
• I have added tests that prove my fix is effective or that my feature works
• I have enabled auto-merge (optional)

Testing

This PR needs to be merged and testing can really only be done once the robots.txt has shipped to production.

[github-actions]

Unit Test Results

362 tests   362 ✔️  18s ⏱️
  37 suites      0 💤
    1 files        0 ❌

Results for commit 7c18c3a.

[rmunn]

I don't think it's a good idea to just allow every bot to index every part of our site. I think we want to consider what parts to disallow indexing, e.g. Disallow: /projects/ and several other URLs.

I also think we might want to disallow certain bots specifically, copying what other people have done. E.g., https://en.wikipedia.org/robots.txt has various bots blocked and comments about specifically why they're blocked. We don't really have time to do extensive research, so copying what bots sites have chosen to block might be a good plan.

However, if you want to just allow all bots now and edit the robots.txt later on to disallow certain ones, I won't be offended if you go ahead and dismiss this review and merge this as-is.

[megahirt]

I don't think it's a good idea to just allow every bot to index every part of our site. I think we want to consider what parts to disallow indexing, e.g. Disallow: /projects/ and several other URLs.

Our implicit policy has historically been "index what you want" since we had no specific robots.txt file with instructions otherwise. The fact that google changed their stance on "we won't index you without explicit allowance defined in robots.txt" doesn't mean that we have changed our policy. There are only a few pages that are publicly accessible (and that we want indexed). All of the the LF content is behind a login and cannot be indexed anyway. When we arrive at a point where we have public projects with public data, we will surely want that indexed as well.

So, I'd like to move forward with the simple robots.txt as proposed for expediency sake, and address other concerns in a separate PR. I do have some concerns about simply copying a robots.txt from a large org, as we may have different goals and the rules/policies are all based upon observed behavior of various bots which inevitably change over time, so I cannot say that blocking various bots which have acted badly in the past with respect to one org is necessarily a good move on our part - just some more thought on that.

[megahirt]

It's strange that there is a code formatting failure for files that weren't touched by this PR. I will ignore that for this PR and address in a separate PR.

with permission, moving forward with this PR for expediency