-
Notifications
You must be signed in to change notification settings - Fork 22
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add cosign signature verification for plugin inventory image to ensure integrity of plugins #106
Conversation
163cf58
to
6fec923
Compare
6fec923
to
869bf39
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wasn't part of the security discussions, so I'm sorry if I'm suggesting something that was already discussed.
If the signature verification fails, CLI would show the warning message but would not throw error and users can choose to skip this validation by setting the environment variable SKIP_PLUGIN_METADATA_SIGNATURE_VERIFY
This approach implies that if a user tries to install a plugin from a hacked DB, a warning will be printed but the plugin will be installed. Part of installing a plugin is to run <plugin> info
, which means the possibly hacked plugin will be run before the user has a chance to decide NOT to install it. I think the plugin install
should fail unless the SKIP variable is set.
==
When will the real signing be done? Probably in the publishing pipeline that calls the builder plugin. Should we therefore update the builder plugin to sign the DB? The private key location would have to be specified through an env var or a flag I guess.
==
Also, can we work together to sign the local test central repo? That way we can use the full security feature while testing locally.
We probably want to add the cosign
command for each of the four DB images in hack/central-repo/upload-plugins.sh
==
Finally, how can I test the use of PLUGIN_METADATA_SIGNATURE_CUSTOM_PUBLIC_KEY_PATH
? I don't understand what the variable should be set to if I want to use a different key.
pkg/discovery/oci_dbbacked.go
Outdated
cosignVerifier := cosignhelper.NewCosignVerifier(customPublicKeyPath) | ||
if err := od.verifyInventoryImageSignature(cosignVerifier); err != nil { | ||
log.Warningf("Unable to verify Plugin metadata signature: %v", err) | ||
log.Warningf("Warning, Plugin metadata signature verification failed and therefore CLI couldn't ensure the integrity of the plugins to be installed. If you choose to ignore the validation please set environment variable SKIP_PLUGIN_METADATA_SIGNATURE_VERIFY to true ") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this is an important message. I made the below suggestion but I would like us to agree on the proper warning. @vuil @anujc25 @prkalle
I think we also need to tell the user what to do if they don't want to skip the validation.
log.Warningf("Warning, Plugin metadata signature verification failed and therefore CLI couldn't ensure the integrity of the plugins to be installed. If you choose to ignore the validation please set environment variable SKIP_PLUGIN_METADATA_SIGNATURE_VERIFY to true ") | |
log.Warningf("Warning, plugins metadata signature verification failed. The `tanzu` CLI can not ensure the integrity of the plugins to be installed. To ignore this validation please set the environment variable SKIP_PLUGIN_METADATA_SIGNATURE_VERIFY to true. This is NOT RECOMMENDED and could put your environment at risk!") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also, could we not hard-code the variable name in the message, but inject it in the message using the constant constants.SkipPluginMetadataSignatureVerification
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
think this is an important message. I made the below suggestion but I would like us to agree on the proper warning.
Thanks @marckhouzam for the suggestion. Yes, we should discuss and agree on the proper warning. I would be glad to hear suggestions.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Update: CLI would fail always if the discovery image signature fails (unless user sets TANZU_CLI_SKIP_PLUGIN_DISCOVERY_IMAGE_SIGNATURE_VERIFICATION
to true )and would show the below message when the discovery image signature verification fails.
Sample log:
❯ ./bin/tanzu plugin search
[!] Unable to verify the plugins discovery image signature: failed unmarshalling PEM encoded default public key: PEM decoding failed
[x] Fatal, plugins discovery image signature verification failed. The `tanzu` CLI can not ensure the integrity of the plugins to be installed. To ignore this validation please set the environment variable "TANZU_CLI_SKIP_PLUGIN_DISCOVERY_IMAGE_SIGNATURE_VERIFICATION" to true. This is NOT RECOMMENDED and could put your environment at risk!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Update: After discussion, the proposed global skip verification environment variable TANZU_CLI_SKIP_PLUGIN_DISCOVERY_IMAGE_SIGNATURE_VERIFICATION
is changed to TANZU_CLI_PLUGIN_DISCOVERY_IMAGE_SIGNATURE_VERIFICATION_SKIP_LIST
and would contain a comma separated list of plugin discovery image urls. CLI would skip validating the image signature for these discovery images
ex: TANZU_CLI_PLUGIN_DISCOVERY_IMAGE_SIGNATURE_VERIFICATION_SKIP_LIST=harbor-repo.vmware.com/tanzu_cli/pkalle/test/v1/plugin-inventory:latest,harbor-repo.vmware.com/tanzu_cli/pkalle/plugins/plugin-inventory:latest
@prkalle My comments are based on the assumption that this is the final iteration of this security feature. If this is instead a first version for a pre-release, some of the comments may not apply as you may have been planning to address them in a future PR. |
Very good point @marckhouzam. I am still thinking to have a prompt message in the oci_backed discovery and asking user whether to continue or else to panic (until the user sets environment variable SKIP_PLUGIN_METADATA_SIGNATURE_VERIFY to skip validation as it is a security issue). Do you think it would be fine or would it be an issue? I will look into ==
Yes, the real signing be done in publishing pipeline, but it would not be part of the builder plugin. In the publish pipeline, once the publishing is done and the DB OCI image is updated, there would be an extra step to sign the DB image with the private key as we cannot share the private key. ==
Sure, we can work together. We might need to address the issue with respect to local registry certs configuration for cosign to work with local registry. This is something we can look into.
I updated the PR description with the steps to show how users can use the environment variable |
ccc1a1d
to
c12bb6f
Compare
c12bb6f
to
2f7dfc2
Compare
I feel better giving some context about how and when we are signing the db image |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM added few comments
Please do create an issue/story for E2E tooling & test case implementation |
2f7dfc2
to
f220610
Compare
Filed Issue: #133 for E2E tests. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great @prkalle !
Once last thing: if the central repo is badly configured, the signature errors show and I feel would lead the user the wrong way, which would make it very hard to figure out they just badly configured the repo:
$ tz config set env.TANZU_CLI_PRE_RELEASE_REPO_IMAGE typo # imagine the typo is not that obvious :-)
$ tz plugin search
[!] Unable to verify the plugins discovery image signature: failed validating the signature of the image typo :GET https://index.docker.io/v2/library/typo/manifests/latest: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:library/typo Type:repository]]
failed validating the signature of the image typo :GET https://index.docker.io/v2/library/typo/manifests/latest: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:library/typo Type:repository]]
[x] Fatal, plugins discovery image signature verification failed. The `tanzu` CLI can not ensure the integrity of the plugins to be installed. To ignore this validation please append "typo" to the comma-separated list in the environment variable "TANZU_CLI_PLUGIN_DISCOVERY_IMAGE_SIGNATURE_VERIFICATION_SKIP_LIST". This is NOT RECOMMENDED and could put your environment at risk!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Thanks for all the changes and for addressing the comments.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! Great work!
f220610
to
92379ba
Compare
…e integrity of plugins -Added signature verification of plugin inventory(DB) image to ensure the integrity of plugin downloaded and installed from the repository - Also embedded the default public key in the CLI required to verify the cosign signature - If the signature verification fails, CLI would show the warning message but would not throw error and users can choose to skip this validation by setting the environment variable TANZU_CLI_PLUGIN_DISCOVERY_IMAGE_SIGNATURE_VERIFICATION_SKIP_LIST with the discovery image url. User can also choose to suppress signature verification failure warning by setting TANZU_CLI_SUPPRESS_SKIP_SIGNATURE_VERIFICATION_WARNING to true. Signed-off-by: Prem Kumar Kalle <pkalle@vmware.com>
92379ba
to
2301127
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you very much @prkalle for your work solving all my requests!
I think this feature is really great and with good UX.
LGTM
What this PR does / why we need it
This PR adds cosign signature verification functionality in CLI to ensure the plugins integrity downloaded and installed from the central repository
-Added signature verification of plugin inventory(DB) image to ensure the integrity of plugin downloaded and installed from the repository
Plugin Discovery(inventory database) Image contains index of each plugin. So, by signing the plugin discovery image and ensuring the Image is signed by CLI before installing the plugins would ensure the integrity of the plugin user is supposed to install. The plugin plublishing workflow(out of the context of this PR) would make sure to sign the Plugin Discovery Image using cosign. CLI would verify the signature of the plugin discovery image using the public key embedded in the CLI.
Which issue(s) this PR fixes
Fixes #107
Describe testing done for PR
Compiled the CLI and plugins and published to repository:
harbor-repo.vmware.com/tanzu_cli/pkalle/test/v1
Later, tried to executing
tanzu plugin search
which would validate the plugin inventory image signature and throws Warning message to User.Plugin Search command failed(below) and also shows Fatal message that Signature verification failed as the plugin discovery image is not signed yet
You can skip the discovery image signature by setting the
TANZU_CLI_PLUGIN_DISCOVERY_IMAGE_SIGNATURE_VERIFICATION_SKIP_LIST
toharbor-repo.vmware.com/tanzu_cli/pkalle/test/v1/plugin-inventory:latest
You can notice in the above log a warning message showing the discovery image signature verification is skipped. If user choose to skip this warning message, user has to set
TANZU_CLI_SUPPRESS_SKIP_SIGNATURE_VERIFICATION_WARNING
totrue
Signed the plugin inventory OCI Image with the private key using cosign
Now you can see tanzu plugin search and tanzu plugin install command can be run without any failures
If the Tanzu CLI publickey is not valid anymore, users can export the env variable
TANZU_CLI_PLUGIN_DISCOVERY_IMAGE_SIGNATURE_PUBLIC_KEY_PATH
to specify the latest valid public key. In the below logs, you can notice initially, the tanzu plugin search command fails as the key is invalid, but after the user specifies the new public key using the environment variable it doesn't show warning/errors and it is successfulRelease note
Additional information
NOTE: User can also skip the validation by exporting the environment variable:
TANZU_CLI_SKIP_PLUGIN_DISCOVERY_IMAGE_SIGNATURE_VERIFICATION
.Some additional delay is observed for validating the image signature.( This shouldn't be an issue once we add caching for plugin DB image)
Special notes for your reviewer