Map certs with ITUT X509 to our RSA implementation (#1754)

[nebeid]

Issues:

Addresses #P157022381

Description of changes:

This is a cherry pick of 4916fe8 to the fips-2022-11-02 branch. The changes are outside the fipsmodule and were modified from the original commit to match the return values of parse_key_type() in evp_asn1.c without changing the fipsmodule files.

---

There exists a rarer OID of RSA public keys which comes from the ITU-T version of X.509 (links below).
Links:
https://www.itu.int/ITU-T/formal-language/itu-t/x/x509/2008/AlgorithmObjectIdentifiers.html

• https://www.ivi.co/jsse/2009/06/13/0220.html

OpenSSL accepts the sign_nid and pkey_nid for this draft.

• NID_md5WithRSA
• NID_sha1WithRSA
• NID_rsa

We can parse both sign_nids correctly. But we lack the logic to interpret NID_rsa in a certificate or map the parsed sign_nids to any RSA implementation.
OpenSSL maps the corresponding nids from the same draft, which means NID_md5WithRSA and NID_sha1WithRSA are
mapped to NID_rsa and an additional EVP_PKEY_RSA2 is used to represent NID_rsa. The additional EVP_PKEY type introduces additional complexities however and the underlying functionality in EVP_PKEY_RSA2 is
identical to it's more common EVP_PKEY_RSA counterpart.

We've been burned by multiple OpenSSL cert parsing differences in the past, so we should have logic to understand certs that have NID_rsa. We don't want to duplicate all the function pointers for EVP_PKEY_RSA2 however. Since the underlying RSA functions pointers are the same, I think it should be OK for us to

1. Map NID_md5WithRSA and NID_sha1WithRSA to our RSA implementation (NID_rsa_encryption).
2. Map NID_rsa to EVP_PKEY_RSA when parsing certificates.

Unfortunately, there's some stricter parsing in rsa_pub_decode that I've had to remove since it doesn't apply to both RFCs for RSA public keys. This is aligning with OpenSSL, which happens to ignore both of the early parameters specified in either.

• Documented in f6094e0
• OpenSSL code (which ignores the parameter): https://github.com/openssl/openssl/blob/8e0d479b98357bb20ab1bd073cf75f7d42531553/crypto/rsa/rsa_ameth.c#L76-L90

Although we have EVP_PKEY_RSA2 and we could map the logic for that to EVP_PKEY_RSA in EVP_PKEY_type, I've intentionally left that out for now. We still want to discourage people from using this form, we just need to understand how to parse certificates that have it.

Relevant historic commits:

• Removed support for NID_rsa: f6094e0
• Sigalgs mapping removed: 720ff53
• RSA stricter parsing introduced: 68772b3

(cherry picked from commit 4916fe8)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and the ISC license.