Multiple upstream authority plugins support

[StupidScience]

Continuing this slack discussion I wanted to suggest support for multiple upstream authorities in spire at time. I think this feature would compliment work that's happening in #1934 and open ways for graceful handling of multiple scenarios, including but not limited:

• CA certificate rotation. When CA certificate is soon to expire there is no graceful way to renew it in spire without nondetermenistic interruptions.
• Migration from one upstream authority plugin to another. That could be desired for number of reasons: migrating to save some money, upstream authority plugin being deprecated, old plugin does not support JWT, etc...

In order to achieve this I'd like to suggest support of multiple (or strictly 2) upstream authority plugins in server's configuration while only one of them is allowed to be "active", e.g. (very roughly):

    UpstreamAuthority "disk" {
        plugin_data {
            cert_file_path = "conf/server/upstream_ca_2023.crt"
            key_file_path = "conf/server/upstream_ca_2023.key"
        }
    }
    UpstreamAuthority "disk" {
        active = false

        plugin_data {
            cert_file_path = "conf/server/upstream_ca_2024.crt"
            key_file_path = "conf/server/upstream_ca_2024.key"
        }
    }

That would mean that upstream_ca_2023 is in use to sign all SVIDs, while upstream_ca_2024 is only included to trust bundles that servers distribute to agents/clients. So once operator is ready to switch to new upstream authority they should flip active flag.
That would allow much more control on operator side to make migrations.

Please let me know wdyt?

cc @evan2645

[azdagron]

So, if I understand correctly, the desired side-effect of having the inactive upstream authority declared, is so that the root authority material for the upstream can be in the bundle? Is there anything else?

If that's it, there are other ways we can achieve that today. For example, some plugins support loading additional root CAs (see bundle_file_path here, or supplemental_bundle_path here). If we needed something more broadly scoped that would cover any plugin, this kind of thing could be implemented in SPIRE Server core so that each plugin didn't have to provide a way.

There is also the AppendBundle RPC, which can be used to add arbitrary CAs to the bundle. This could be used in advance of activating a new UpstreamAuthority to ensure the upstream authority is available in the bundle well in advance.

I'll also mention that SPIRE's prepare/activate cycle for rotating into a new authority already accommodates rolling into a new upstream authority, though the timing can be hard to control explicitly without work from the proposed LocalAuthority API, though that API is not available yet.

[StupidScience]

@azdagron yeah, I guess main point is to have new inactive upstream authority CA certificate to be in distributed to clients bundle. I probably can come up with some additional benefits of this approach is the main goal to do that this way as for me.
Thanks for pointing there are already capability to do that with certain plugins but I think it should be supported for any of them.
Listed options with API, configuration, RPC should work fine but are more error prone comparing with proposed by me solution, e.g. you need to create CA certificate and somehow feed it to spire server so should be done manually or require additional automation around.

Another point here is that while these are probably viable options to consider and move on with I didn't find any references in documentation to them or to how to safely rotate upstream authority certificate or migrate to another upstream authority.
Should AppendBundle RPC be available via bundle set command? That would also make migration path at least a bit more clear.

[azdagron]

Ah, yes, it looks like we have yet to add the bundle append subcommand. That's easy enough to implement. If that were available, would that cover your current use case?

[StupidScience]

Seems like my particular use case is already kind of covered since AWS PCA has supplemental_bundle_path property that I didn't notice (nor was looking for).
As I mentioned before this property as well as adding bundle append subcommand (while making rotation easier to achieve) are still somewhat inconvenient and error prone approaches that require "manual" bundle definition in one or another way.

[evan2645]

I guess main point is to have new inactive upstream authority CA certificate to be in distributed to clients bundle.

I think this should "just work" if you change your UpstreamCA plugin to point to the new PCA ARN? SPIRE Server will automatically prepare a cert against the new root and distribute that new root in the bundle in advance of using it to sign SVIDs. Just want to check that this isn't sufficient for your case?

[StupidScience]

I think this should "just work" if you change your UpstreamCA plugin to point to the new PCA ARN

Main concern with that is that certificate changes with this approach is uncontrollable and can lead to high severity incidents.

[evan2645]

AFAICT, the supplemental_bundle_path config has the same change control semantics as your earlier proposal - a config change to bring the new root into the bundle, and a config change to then activate it (by switching the upstream that the plugin is pointed to)?

As I mentioned before this property as well as adding bundle append subcommand (while making rotation easier to achieve) are still somewhat inconvenient and error prone approaches that require "manual" bundle definition in one or another way.

Is it that you don't have an easy way to extract the next/new CA material and configure onto the plugin?

[StupidScience]

Once again supplemental_bundle_path works for us in general since we use aws pca. I did not expect this setting to be plugin (implementation) specific so did not check that initially.

Is it that you don't have an easy way to extract the next/new CA material and configure onto the plugin?

Not really, but I want to highlight once again that it is "manual" operation or need to be automated additionally.

It is totally up to you to decide what way you're wanna go further just I do not see reasons why this scenario won't work out of the box with spire and why it does not handle rotation/upstream authority migration without additional efforts from operator side.

[evan2645]

Ok, thanks for your help through this @StupidScience. Since we do have both automatic and manual ways to handle the rotation of an upstream authority (but not via the proposed dual UpstreamAuthority approach), I'm afraid I'm misunderstanding part of your ask. I definitely think there's room for improvement, but it's not clear to me what alternate shapes would fill your requirements

I'll go ahead and close this out for now since it seems you're unblocked. Thanks again for the feedback here 🙏