remove language-level UB for non-UTF-8 str

[RalfJung]

Ever since Rust 1.0, the reference said that a non-UTF-8 str causes immediate UB. In terms of today's terminology, that means that str has a validity invariant of being valid UTF-8.

However, that seems unnecessary: the compiler does not actually exploit this, nor is there any clear way it could exploit this. Making UTF-8 a library-level safety invariant is more than enough for everything str does. Most likely, it was made a validity invariant because we had not yet properly teased apart those two concepts when the document was initially written.

This is also the conclusion that the UCG WG arrived at in rust-lang/unsafe-code-guidelines#78.

I therefore propose we remove the UTF-8 clause from the language spec, so that str will have the same validity invariant as [u8]. @Centril suggested I open this a PR here to put this through FCP, so here we go.

Fixes rust-lang/rust#71033

[Centril]

Thanks!

However, I don't believe I can actually use @rfcbot in this repo,
so I think you will need to create an issue on rust-lang/rust
perhaps with most of the PR description pasted there so that I can initiate FCP.

Here are also some textual nits according to the style guide we recently merged.

[Centril]

(Approving the text itself to be merged once FCP itself is done and stuff.)

[RalfJung]

Opened rust-lang/rust#71033.

[RalfJung]

FCP passed. So can we land this?

[ehuss]

AIUI, str would be defined as [u8], correct? Would it make sense to update the str definition to clarify what it means? Like, make it clear that what str can contain, that it is conventionally interpreted as utf-8 code units, and then include a sidenote that the standard library assumes that it is valid utf-8, and violating that is a safety invariant (or whatever terminology you want to use).