Add support for selective plugin migration using plugin groups for air-gapped environments #231

anujc25 · 2023-04-25T08:18:56Z

What this PR does / why we need it

This change adds support for users to do partial migration of plugins using plugin groups instead of migrating all available plugins in the plugin inventory.
Allows users to pass the --group flag with tanzu plugin download-bundle to do selective plugin download and generate tar file
Users can use the generated tar file with the tanzu plugin upload-bundle command to do partial plugin migration

Command UX:

# Downloading the plugins
$ tanzu plugin download-bundle  
      --to-tar /tmp/plugin-inventory.tar.gz
      --image projects.registry.vmware.com/tanzu_cli/plugins/plugin-inventory:latest
      --group <plugin-group-name> (optional)

# Uploading the plugins
$ tanzu plugin upload-bundle 
      --tar /tmp/plugin-inventory.tar.gz
      --to-repo private_repo.mycompany.com/plugins

Implementation Details:
Considering the original plugin inventory database image is immutable and changing the content of that image has consequences regarding signature verification, the proposed approach uses an additional plugin-inventory-metadata image containing the SQLite baked plugin_inventory_metadata.db image to store available plugins and available plugin group information.

If the actual plugin-inventory database gets published to private_repo.mycompany.com/plugins/plugin-inventory:latest
then the additional oci image containing the metadata database will be published at private_repo.mycompany.com/plugins/plugin-inventory-metadata:latest.

Workflow for tanzu plugin download-bundle command:

Download the plugin-inventory database image as plugin-inventory-image.tar.gz
Find selectedPlugins and selectedPluginGroups based on the provided --group
For each plugin binary in selectedPlugins download plugin-name_target_os_arch_version.tar.gz
Create plugin_inventory_metadata.db based on the selectedPlugins and selectedPluginGroups
Create plugin_migration_manifest.yaml with the metadata about all things that needs to be migrated
Generate tar.gz file containing all the above content which can be used with upload-bundle command.

Workflow for tanzu plugin upload-bundle command:

Extract the provided tarfile to a temp directory
Read the plugin_migration_manifest.yaml file
Upload the plugin inventory database and each plugin binary to the remote repository based on the migration manifest file
Publish plugin inventory metadata image. Before publishing the plugin inventory metadata image check if one already exists on the remote repository and if exists download the existing plugin inventory metadata database and merge the plugin inventory metadata database before publishing

Pending: (Will be done as part of a separate PR)

Verification of the image signature before downloading the plugin bundle
Documentation

Which issue(s) this PR fixes

Related to #223

Describe testing done for PR

# Point the default repository to airgapped:small
$ export TANZU_CLI_PRE_RELEASE_REPO_IMAGE=localhost:9876/tanzu-cli/plugins/airgapped:small
$ export TANZU_CLI_PLUGIN_DISCOVERY_IMAGE_SIGNATURE_VERIFICATION_SKIP_LIST=localhost:9876/tanzu-cli/plugins/airgapped:small

$ tz plugin search
[i] Reading plugin inventory for "localhost:9876/tanzu-cli/plugins/airgapped:small", this will take a few seconds.
[!] Skipping the plugins discovery image signature verification for "localhost:9876/tanzu-cli/plugins/airgapped:small"
 
  NAME              DESCRIPTION                TARGET           LATEST  
  isolated-cluster  Desc for isolated-cluster  global           v9.9.9  
  cluster           Desc for cluster           kubernetes       v9.9.9  
  account           Desc for account           mission-control  v9.9.9 

$ tz plugin group search
  GROUP              
  vmware-tkg/v0.0.1  
  vmware-tkg/v9.9.9  
  vmware-tmc/v0.0.1  
  vmware-tmc/v9.9.9

# download-bundle with plugins from group `vmware-tkg/v0.0.1`
$ tz plugin download-bundle --image localhost:9876/tanzu-cli/plugins/airgapped:small --to-tar /tmp/4/plugin_bundle_tkg_001.tar.gz --group vmware-tkg/v0.0.1
[i] downloading image "localhost:9876/tanzu-cli/plugins/airgapped:small"
.
.
[i] saving plugin bundle at: /tmp/4/plugin_bundle_tkg_001.tar.gz

# upload-bundle which was downloaded in the previous step to remote repository
$ tz plugin upload-bundle --tar /tmp/4/plugin_bundle_tkg_001.tar.gz --to-repo localhost:9876/test/airgapped/v15/tanzu-cli/plugins
[i] ---------------------------
[i] uploading image "localhost:9876/test/airgapped/v15/tanzu-cli/plugins/airgapped"
.
.
[i] ---------------------------
[i] publishing plugin inventory metadata image...
[i] plugin inventory metadata image "localhost:9876/test/airgapped/v15/tanzu-cli/plugins/airgapped-metadata:small" is not present. Skipping merging of the plugin inventory metadata
[i] uploading image "localhost:9876/test/airgapped/v15/tanzu-cli/plugins/airgapped-metadata:small"
[i] ---------------------------
[i] successfully published all plugin images to "localhost:9876/test/airgapped/v15/tanzu-cli/plugins"

# configure environment variables to point CLI to use remote repository where plugins were published
$ export TANZU_CLI_PLUGIN_DISCOVERY_IMAGE_SIGNATURE_VERIFICATION_SKIP_LIST=localhost:9876/test/airgapped/v15/tanzu-cli/plugins/airgapped:small
$ export TANZU_CLI_PRE_RELEASE_REPO_IMAGE=localhost:9876/test/airgapped/v15/tanzu-cli/plugins/airgapped:small

# Verify the available plugins in the registry by running plugin search and plugin group search
$ tz plugin search --show-details
[i] Reading plugin inventory for "localhost:9876/test/airgapped/v15/tanzu-cli/plugins/airgapped:small", this will take a few seconds.
[!] Skipping the plugins discovery image signature verification for "localhost:9876/test/airgapped/v15/tanzu-cli/plugins/airgapped:small"
 
name: cluster
description: Desc for cluster
target: kubernetes
latest: v0.0.1
versions:
    - v0.0.1

$ tz plugin group search
  GROUP              
  vmware-tkg/v0.0.1

# Now that we have downloaded and uploaded plugins from the `vmware-tkg/v0.0.1` group. Let's do the same with different plugin group `vmware-tmc/v9.9.9`
$ tz plugin download-bundle --image localhost:9876/tanzu-cli/plugins/airgapped:small --to-tar /tmp/4/plugin_bundle_tmc_999.tar.gz --group vmware-tmc/v9.9.9

$ tz plugin upload-bundle --tar /tmp/4/plugin_bundle_tmc_999.tar.gz --to-repo localhost:9876/test/airgapped/v15/tanzu-cli/plugins


# Check the plugin search and plugin group search result, it should have plugins from the `vmware-tmc/v9.9.9` group as well
$ tz plugin search --show-details
[i] Reading plugin inventory for "localhost:9876/test/airgapped/v15/tanzu-cli/plugins/airgapped:small", this will take a few seconds.
[!] Skipping the plugins discovery image signature verification for "localhost:9876/test/airgapped/v15/tanzu-cli/plugins/airgapped:small"
 
name: cluster
description: Desc for cluster
target: kubernetes
latest: v0.0.1
versions:
    - v0.0.1

name: account
description: Desc for account
target: mission-control
latest: v9.9.9
versions:
    - v9.9.9

$ tz plugin group search
  GROUP              
  vmware-tkg/v0.0.1  
  vmware-tmc/v9.9.9

# Now let's try to download the entire plugin-inventory without providing `--group` flag and upload it to the same remote registry
$ tz plugin download-bundle --image localhost:9876/tanzu-cli/plugins/airgapped:small --to-tar /tmp/4/plugin_bundle_complete.tar.gz

$ tz plugin upload-bundle --tar /tmp/4/plugin_bundle_complete.tar.gz --to-repo localhost:9876/test/airgapped/v15/tanzu-cli/plugins

# Check the plugin search and plugin group search result, it should have all plugins and plugin groups
$ tz plugin group search
[i] Reading plugin inventory for "localhost:9876/test/airgapped/v15/tanzu-cli/plugins/airgapped:small", this will take a few seconds.
[!] Skipping the plugins discovery image signature verification for "localhost:9876/test/airgapped/v15/tanzu-cli/plugins/airgapped:small"
 
  GROUP              
  vmware-tkg/v0.0.1  
  vmware-tkg/v9.9.9  
  vmware-tmc/v0.0.1  
  vmware-tmc/v9.9.9 

$ tz plugin search --show-details
name: isolated-cluster
description: Desc for isolated-cluster
target: global
latest: v9.9.9
versions:
    - v0.0.1
    - v9.9.9

name: cluster
description: Desc for cluster
target: kubernetes
latest: v9.9.9
versions:
    - v0.0.1
    - v9.9.9

name: account
description: Desc for account
target: mission-control
latest: v9.9.9
versions:
    - v0.0.1
    - v9.9.9

# Also verify that we are able to install plugins as well
$ tanzu plugin install --group vmware-tkg/v0.0.1
[i] Installing plugin 'cluster:v0.0.1' with target 'kubernetes'
[ok] successfully installed all plugins from group 'vmware-tkg/v0.0.1'

Release note

Add support for selective plugin migration using plugin groups for air-gapped environments

Additional information

Special notes for your reviewer

pkg/command/plugin_bundle.go

marckhouzam

I'm going through this. Very nice! Some first comments.

There are some comments about "hidden" plugins, but maybe for alpha.1, we can simply ignore hidden plugins/groups if we don't have time to handle them properly.

pkg/command/plugin_bundle.go

pkg/airgapped/types.go

pkg/airgapped/plugin_bundle_download.go

marckhouzam · 2023-04-26T15:00:10Z

pkg/airgapped/plugin_bundle_download.go

+		return nil, nil, errors.Errorf("incorrect plugin group %q specified", pgName)
+	}
+	pgFilter := plugininventory.PluginGroupFilter{
+		IncludeHidden: true,


Can you put a comment on why we include hidden? I assume it is something like: "If the user explicitly specified a group that is hidden, we should respect that".

Should we require the new TANZU_CLI_INCLUDE_DEACTIVATED_PLUGINS_TEST_ONLY to be set before allowing to get hidden groups/plugins? Once #219 is merged

marckhouzam · 2023-04-26T15:07:32Z

pkg/airgapped/plugin_bundle_download.go

+
+	// If groups were not provided as argument select all available plugin groups and all available plugins
+	if len(o.Groups) == 0 {
+		selectedPluginGroups, err = pi.GetPluginGroups(plugininventory.PluginGroupFilter{})


Since you allow hidden groups below, should we also allow including hidden groups if the new TANZU_CLI_INCLUDE_DEACTIVATED_PLUGINS_TEST_ONLY is set? Once #219 is merged

marckhouzam · 2023-04-26T15:16:19Z

pkg/airgapped/plugin_bundle_download.go

+			pif := &plugininventory.PluginInventoryFilter{
+				Name:    p.Name,
+				Target:  p.Target,
+				Version: p.Version,


We need to handle hidden plugins here, if we allow for hidden groups

pkg/airgapped/plugin_bundle_upload.go

pkg/command/plugin_bundle.go

marckhouzam

One more file to go, but here are the last few nits.

pkg/airgapped/helper.go

marckhouzam · 2023-04-27T12:51:25Z

pkg/airgapped/helper.go

+// `plugin-inventory` if withTag is false
+func GetImageRelativePath(image, basePath string, withTag bool) string {
+	relativePath := strings.TrimPrefix(image, basePath)
+	if withTag {


What if withTag == true but the image does not have a tag? Should we append :latest?
I don't know if this could even happen but I thought about it because the function above does add :latest if the tag is not specified.

pkg/airgapped/plugin_bundle_upload.go

pkg/registry/client.go

pkg/plugininventory/plugin_inventory_metadata.go

marckhouzam

This is really great!
Just some last comments about cleaning up temp directories and then it looks good!

pkg/discovery/oci_dbbacked.go

pkg/airgapped/plugin_bundle_test.go

pkg/plugininventory/plugin_inventory_metadata_test.go

…ents This change adds support for users to do partial migration of plugins using plugin groups instead of migrating all available plugins. - Allows users to pass `--group` flag with `tanzu plugin download-bundle` to do selective plugin download and generates tar file - Users can use the generated tar file with the `tanzu plugin upload-bundle` command to do partial plugin migration

marckhouzam

LGTM!
Thanks for this great feature in such a short time! 🚀

…r-gapped environments (#231) * Selective plugin migration using plugin groups for airgapped environments This change adds support for users to do partial migration of plugins using plugin groups instead of migrating all available plugins. - Allows users to pass `--group` flag with `tanzu plugin download-bundle` to do selective plugin download and generates tar file - Users can use the generated tar file with the `tanzu plugin upload-bundle` command to do partial plugin migration

vmwclabot added the cla-not-required label Apr 25, 2023

anujc25 force-pushed the airgap-support-with-groups branch 2 times, most recently from 5ab4263 to 87bfcf2 Compare April 26, 2023 05:49

anujc25 marked this pull request as ready for review April 26, 2023 06:15

anujc25 requested a review from a team as a code owner April 26, 2023 06:15

anujc25 commented Apr 26, 2023

View reviewed changes

pkg/command/plugin_bundle.go Outdated Show resolved Hide resolved

marckhouzam reviewed Apr 26, 2023

View reviewed changes

anujc25 commented Apr 26, 2023

View reviewed changes

pkg/command/plugin_bundle.go Show resolved Hide resolved

anujc25 commented Apr 26, 2023

View reviewed changes

pkg/command/plugin_bundle.go Show resolved Hide resolved

anujc25 force-pushed the airgap-support-with-groups branch 4 times, most recently from 080922a to e4dbf8f Compare April 26, 2023 22:22

marckhouzam reviewed Apr 27, 2023

View reviewed changes

anujc25 force-pushed the airgap-support-with-groups branch from 0dc412f to b625ab1 Compare April 27, 2023 17:41

marckhouzam reviewed Apr 27, 2023

View reviewed changes

anujc25 force-pushed the airgap-support-with-groups branch from 689843a to 205e7dd Compare April 27, 2023 18:36

anujc25 added 5 commits April 27, 2023 12:06

Address comments part-1

8d71033

Always store metadata.digest in cache even if image doesn't exists

6ef3d60

Address comments part-2

e63dddf

Address comments part-3

53cee08

anujc25 force-pushed the airgap-support-with-groups branch from 205e7dd to d822e47 Compare April 27, 2023 19:13

Update log message

bafcefc

anujc25 force-pushed the airgap-support-with-groups branch from d822e47 to bafcefc Compare April 27, 2023 19:58

marckhouzam approved these changes Apr 27, 2023

View reviewed changes

anujc25 merged commit bf44267 into vmware-tanzu:main Apr 27, 2023

marckhouzam added this to the v0.90.0 milestone Oct 20, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for selective plugin migration using plugin groups for air-gapped environments #231

Add support for selective plugin migration using plugin groups for air-gapped environments #231

anujc25 commented Apr 25, 2023 •

edited

Loading

marckhouzam left a comment

marckhouzam Apr 26, 2023

marckhouzam Apr 26, 2023

marckhouzam Apr 26, 2023

marckhouzam left a comment

marckhouzam Apr 27, 2023

marckhouzam left a comment

marckhouzam left a comment

Add support for selective plugin migration using plugin groups for air-gapped environments #231

Add support for selective plugin migration using plugin groups for air-gapped environments #231

Conversation

anujc25 commented Apr 25, 2023 • edited Loading

What this PR does / why we need it

Which issue(s) this PR fixes

Describe testing done for PR

Release note

Additional information

Special notes for your reviewer

marckhouzam left a comment

Choose a reason for hiding this comment

marckhouzam Apr 26, 2023

Choose a reason for hiding this comment

marckhouzam Apr 26, 2023

Choose a reason for hiding this comment

marckhouzam Apr 26, 2023

Choose a reason for hiding this comment

marckhouzam left a comment

Choose a reason for hiding this comment

marckhouzam Apr 27, 2023

Choose a reason for hiding this comment

marckhouzam left a comment

Choose a reason for hiding this comment

marckhouzam left a comment

Choose a reason for hiding this comment

anujc25 commented Apr 25, 2023 •

edited

Loading