Improve Retry Coverage

[andrebsguedes]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Currently some retries are only performed for requests that have an HTTP method that is considered safe by rfc7231 due to potential problems around timeouts and non-idempotent operations. The problem with this approach is that there are many idempotent requests that use methods not considered safe by this criteria, and in this case the requests may not be retried.

This problem can manifest in places where the usual workaround of wrapping the request with an extra retry layer is not efficient. One example is put_part on a multipart upload which cannot be wrapped in a retry loop without aborting the whole stream and streaming again from the start (which may not be always possible).

Another related problem is that the current checks for retrying transport errors do not cover all common cases (even for "safe" operations) which further exacerbates the first problem.

Describe the solution you'd like

One solution would be to create another version of send_retry which allows to manually specify that the operation is idempotent irrespective of the request method. This version could be used in "not safe" requests which are known to be idempotent.

Also, increase the coverage of retries for transport errors.

Describe alternatives you've considered

As explained in the initial description wrapping the request with another retry layer is not efficient in some cases.

[tustvold]

label_issue.py automatically added labels {'object-store'} from #5609