Distributor write path: put httpgrpc.Errorf() calls at the topmost level

[duricanikolic]

What this PR does

This PR introduces the following error types, reppresenting the problems on the distributor’s write path that need some special handling:

• ReplicasNotMatch,
• TooManyClusters,
• ValidationError,
• IngestionRateLimited
• RequestRateLimited.
  They all belong to the newly introduced package distributorerror.

Moreover, this PR refactors the distributor's write path in such a way that all the calls to httpgrpc.Errorf() have been moved to the topmost possible level,i.e., to:

• push.handler() and
• distributor.Push().
  The semantics of the distributor's write path has not been changed, i.e., the newly introduced errors are mapped to the same HTTP status codes their corresponding problems used to be assigned before this PR.

This refactoring is just a preliminary change needed for a bigger refactoring and improving of intester's and distributor's write path error handling.

Which issue(s) this PR fixes or relates to

Related to #6008

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[colega]

Shouldn't we shorten the metric if it's too long?

[duricanikolic]

This is the same implementation we are having on master right now (here)

[colega]

Hum, okay, I'd say that's a bug.

[pstibrany]

It would be good to see the final result where this is heading, because I fail to see how this PR improves anything really.

[pstibrany]

Problems I'd like to see fixed in subsequent PRs:

• dependency from util/push to distributorerror. Push handler is currently generic and it should not call error translation from distributor (sub)package directly. To solve this we could pass "error translation function" to push handler. Later when ingester errors are supported too, this error translation function should handle both kinds of errors.

• distributorerror package -- all errors introduced here belong to distributor package. Furthermore they don't need to be exported. The only thing that needs to be exported is "distributor's error translation function".

• design of error types in distributorerror package. They should not be just wrappers around 'error' or 'format`.