Implement plugin-group publishing tooling with builder plugin #123

anujc25 · 2023-03-21T01:48:00Z

What this PR does / why we need it

Implements the following commands:

tanzu builder inventory plugin-group add
tanzu builder inventory plugin-group activate
tanzu builder inventory plugin-group deactivate
The change also updates tanzu builder plugin build to take additional
input flag --plugin-scope-association-file. If this flag is provided
the build process will automatically generate a
plugin-group-manifest.yaml file within the provided --binary-artifacts
directory.
The change also implements InsertPluginGroup and
UpdatePluginGroupActivationState APIs in the plugininventory package
Update cmd/plugin/builder/README.md with the new commands.

Which issue(s) this PR fixes

Fixes #

Describe testing done for PR

Testing using the make targets provided as part of the plugin-tooling.mk file.

# Build the plugin binaries and publish them as OCI images
$ make plugin-build-and-publish-packages

# Initialize empty plugin inventory
$ make inventory-init

# Insert plugins to the inventory database
$ make inventory-plugin-add

# Add plugin-group to the inventory database
$ make inventory-plugin-group-add

# Activate plugin-group in the inventory database
$ make inventory-plugin-group-activate

# Deactivate plugin-group in the inventory database
$ make inventory-plugin-group-deactivate

Testing using the builder plugin commands directly.

$ ./bin/builder plugin build \
          --path ./cmd/plugin \
          --binary-artifacts ./artifacts/plugins \
          --version v0.0.2 \
          --os-arch darwin_amd64 --os-arch linux_amd64 --os-arch windows_amd64 \
          --plugin-scope-association-file ./cmd/plugin/plugin-scope-association.yaml 
          
2023-03-20T18:55:11-07:00 [i] building local repository at ./artifacts/plugins, v0.0.2, [darwin_amd64 linux_amd64 windows_amd64]
2023-03-20T18:55:11-07:00 [i] 🐯 - building plugin at path "cmd/plugin/test"
2023-03-20T18:55:11-07:00 [i] 🐱 - building plugin at path "cmd/plugin/builder"
...
2023-03-20T18:55:38-07:00 [i] ========
2023-03-20T18:55:38-07:00 [i] saving plugin group manifest...
2023-03-20T18:55:38-07:00 [ok] successfully built local repository

# Building Plugin Packages
$ ./bin/builder plugin build-package --binary-artifacts ./artifacts/plugins/ --package-artifacts ./artifacts/packages --oci-registry localhost:5001

2023-02-23T15:40:59-08:00 [ℹ]  Using plugin binary artifacts from "./artifacts/plugins/"
...
2023-02-23T15:41:15-08:00 [ℹ]  Saved plugin manifest at "artifacts/packages/plugin_manifest.yaml"

# Publishing Plugin Packages
$ ./bin/builder plugin publish-package --package-artifacts ./artifacts/packages --publisher tkg --vendor vmware --repository localhost:5001/test/v1

2023-02-23T15:43:56-08:00 [ℹ]  Using plugin package artifacts from "./artifacts/packages"
...

# Initialize inventory database
$ ./bin/builder inventory init --repository localhost:5001/test/v1 --plugin-inventory-image-tag latest
2023-03-20T18:59:00-07:00 [i] created database locally at: "/var/folders/xc/12mmbpb50qxfb3ch8vv9nvwm0000gq/T/plugin_inventory.db"
2023-03-20T18:59:00-07:00 [i] publishing database at: "localhost:5001/test/v1/plugin-inventory:latest"
2023-03-20T18:59:00-07:00 [i] successfully published plugin inventory database

# Adding plugins to the inventory database
$ ./bin/builder inventory plugin add --repository localhost:5001/test/v1 --plugin-inventory-image-tag latest --publisher tkg --vendor vmware --manifest ./artifacts/packages/plugin_manifest.yaml 
2023-03-20T19:00:36-07:00 [i] Pulling plugin inventory database from: "localhost:5001/test/v1/plugin-inventory:latest"
2023-03-20T19:00:36-07:00 [i] Validating plugin 'builder_global_linux_amd64_v0.0.2'
2023-03-20T19:00:37-07:00 [i] Validating plugin 'builder_global_darwin_amd64_v0.0.2'
2023-03-20T19:00:37-07:00 [i] Validating plugin 'builder_global_windows_amd64_v0.0.2'
2023-03-20T19:00:38-07:00 [i] Validating plugin 'test_global_linux_amd64_v0.0.2'
2023-03-20T19:00:39-07:00 [i] Validating plugin 'test_global_darwin_amd64_v0.0.2'
2023-03-20T19:00:40-07:00 [i] Validating plugin 'test_global_windows_amd64_v0.0.2'
2023-03-20T19:00:41-07:00 [i] Publishing plugin inventory database
2023-03-20T19:00:41-07:00 [i] Successfully published plugin inventory database at: "localhost:5001/test/v1/plugin-inventory:latest"

Adding and verifying plugin groups:

# Point the CLI to use created provided inventory database
$ export TANZU_CLI_PRE_RELEASE_REPO_IMAGE=localhost:5001/test/v1/plugin-inventory:latest

# No group exist yet
$ tz plugin group search
  GROUP              

# Check plugin_group_manifest.yaml
$ cat ./artifacts/plugins/plugin_group_manifest.yaml 
created: 2023-03-20T18:55:38.755262-07:00
plugins:
    - name: builder
      target: global
      isContextScoped: false
      version: v0.0.2
    - name: test
      target: global
      isContextScoped: false
      version: v0.0.2

# Adding plugin-group to the inventory database
$ ./bin/builder inventory plugin-group add --name v1.0.0 --repository localhost:5001/test/v1 --plugin-inventory-image-tag latest --publisher tkg --vendor vmware --manifest ./artifacts/plugins/plugin_group_manifest.yaml
2023-03-20T19:01:54-07:00 [i] pulling plugin inventory database from: "localhost:5001/test/v1/plugin-inventory:latest"
2023-03-20T19:01:55-07:00 [i] updating plugin inventory database with plugin group entry
2023-03-20T19:01:55-07:00 [i] publishing plugin inventory database
2023-03-20T19:01:55-07:00 [i] successfully published plugin inventory database at: "localhost:5001/test/v1/plugin-inventory:latest"

# New group should be available
$ tz plugin group search
  GROUP  
  vmware-tkg/v1.0.0          

# Install plugins from the created group
$ tz plugin install all --group vmware-tkg/v1.0.0
[i] Installing plugin 'builder:v0.0.2' with target 'global'
[i] Installing plugin 'test:v0.0.2' with target 'global'
[ok] successfully installed all plugins from group 'vmware-tkg/v1.0.0'

# Running the plugin-group add command again with the same vendor, publisher, and name (it throws error)
$ ./bin/builder inventory plugin-group add --name v1.0.0 --repository localhost:5001/test/v1 --plugin-inventory-image-tag latest --publisher tkg --vendor vmware --manifest ./artifacts/plugins/plugin_group_manifest.yaml
2023-03-20T19:07:26-07:00 [i] pulling plugin inventory database from: "localhost:5001/test/v1/plugin-inventory:latest"
2023-03-20T19:07:26-07:00 [i] updating plugin inventory database with plugin group entry
Error: error while inserting plugin group 'v1.0.0': unable to insert plugin-group row {vmware tkg v1.0.0 builder global v0.0.2 true false}: UNIQUE constraint failed: PluginGroups.Vendor, PluginGroups.Publisher, PluginGroups.GroupName, PluginGroups.PluginName, PluginGroups.Target, PluginGroups.Version

# Show updated plugin-group-manifest-single-plugin.yaml
$ cat ./artifacts/plugins/plugin_group_manifest_single_plugin.yaml
created: 2023-03-20T18:55:38.755262-07:00
plugins:
    - name: builder
      target: global
      isContextScoped: false
      version: v0.0.2

# Running the plugin-group add command again with the same vendor, publisher, and name but updated plugins and with `--override` flag
$ ./bin/builder inventory plugin-group add --name v1.0.0 --repository localhost:5001/test/v1 --plugin-inventory-image-tag latest --publisher tkg --vendor vmware --manifest ./artifacts/plugins/plugin_group_manifest_single_plugin.yaml --override
2023-03-20T19:28:16-07:00 [i] pulling plugin inventory database from: "localhost:5001/test/v1/plugin-inventory:latest"
2023-03-20T19:28:16-07:00 [i] updating plugin inventory database with plugin group entry
2023-03-20T19:28:16-07:00 [i] publishing plugin inventory database
2023-03-20T19:28:17-07:00 [i] successfully published plugin inventory database at: "localhost:5001/test/v1/plugin-inventory:latest"

# Installing plugins from the group after update only installs `builder` plugin
$ tz plugin install all --group vmware-tkg/v1.0.0
[i] Installing plugin 'builder:v0.0.2' with target 'global'
[ok] successfully installed all plugins from group 'vmware-tkg/v1.0.0'

Release note

Implement plugin-group publishing tooling with builder plugin

Additional information

Special notes for your reviewer

marckhouzam

I've gone through the first chunk of changes and it all makes sense.
I'm posting come comments about style or error messages and will continue the review

pkg/plugininventory/data/sqlite/create_tables.sql

pkg/plugininventory/plugin_inventory.go

pkg/plugininventory/sqlite_inventory.go

pkg/plugininventory/sqlite_inventory_test.go

cmd/plugin/plugin-scope-association.yaml

plugin-tooling.mk

vuil

Changes look good in general. I have some nits on the docs.
Main think I also had some issue with was the PluginIdentifier overloading and I am glad @marckhouzam pointed it out.

cmd/plugin/builder/README.md

Implements following commands: - `tanzu builder inventory plugin-group add` - `tanzu builder inventory plugin-group activate` - `tanzu builder inventory plugin-group deactivate` The change also updates `tanzu builder plugin build` to take additional input flag `--plugin-scope-association-file`. If this flag is provided the build process will automatically generate a plugin-group-manifest.yaml file within the provided `--binary-artifacts` directory. The change also implements `InsertPluginGroup` and `UpdatePluginGroupActivationState` APIs in the plugininventory package Update `cmd/plugin/builder/README.md` with the new commands. Only install mandatory plugin using plugin group install

chandrareddyp · 2023-03-24T14:58:34Z

# Running the plugin-group add command again with the same vendor, publisher, and name (it throws error)
$ ./bin/builder inventory plugin-group add --name v1.0.0 --repository localhost:5001/test/v1 --plugin-inventory-image-tag latest --publisher tkg --vendor vmware --manifest ./artifacts/plugins/plugin_group_manifest.yaml
2023-03-20T19:07:26-07:00 [i] pulling plugin inventory database from: "localhost:5001/test/v1/plugin-inventory:latest"
2023-03-20T19:07:26-07:00 [i] updating plugin inventory database with plugin group entry
Error: error while inserting plugin group 'v1.0.0': unable to insert plugin-group row {vmware tkg v1.0.0 builder global v0.0.2 true false}: UNIQUE constraint failed: PluginGroups.Vendor, PluginGroups.Publisher, PluginGroups.GroupName, PluginGroups.PluginName, PluginGroups.Target, PluginGroups.Version

This makes sense for the developer, but for the end-user, we can improve it by saying, "the plugin group with the name already exists!"

chandrareddyp · 2023-03-24T15:04:10Z

what happens when we run tanzu builder inventory plugin-group deactivate, it will hide all plugin groups, which means no output for tanzu plugin group search, it would be good if you add an example.

chandrareddyp · 2023-03-24T15:05:57Z

please create an issue for E2E tests, and also add all use cases to handle as part of E2E tests.

anujc25 · 2023-03-24T16:08:49Z

what happens when we run tanzu builder inventory plugin-group deactivate, it will hide all plugin groups, which means no output for tanzu plugin group search, it would be good if you add an example.

Yes. That is the behavior we want. However, the client-side changes for the tanzu plugin group search are still not done and will be done as part of the future change. So, the e2e tests for the same should be handled as part of the same PR.
We already have an issue tracking the plugin publishing e2e tests.

chandrareddyp · 2023-03-24T16:27:07Z

what happens when we run tanzu builder inventory plugin-group deactivate, it will hide all plugin groups, which means no output for tanzu plugin group search, it would be good if you add an example.

Yes. That is the behavior we want. However, the client-side changes for the tanzu plugin group search are still not done and will be done as part of the future change. So, the e2e tests for the same should be handled as part of the same PR.
We already have an issue tracking the plugin publishing e2e tests.

Better to add some messages for tanzu plugin group search output when plugin-group is deactivated so that the end user is aware that it's been deactivated, otherwise end user may assume there is no plugin groups available

chandrareddyp

LGTM!
added few comments

anujc25 · 2023-03-24T17:08:39Z

Better to add some messages for tanzu plugin group search output when plugin-group is deactivated so that the end user is aware that it's been deactivated, otherwise end user may assume there is no plugin groups available

As mentioned above, handling the activated/deactivated state in the UX in the CLI will be handled with a separate issue. However, the goal of the deactivated state is to not even should that deactivated plugin group exist. It will only be consumed for the pre-release testing.

marckhouzam · 2023-03-24T20:25:28Z

what happens when we run tanzu builder inventory plugin-group deactivate, it will hide all plugin groups, which means no output for tanzu plugin group search, it would be good if you add an example.

I'm not sure what you really meant with this question, so I'll try to answer as I understood it.
So, no, not all plugins groups will be hidden, only the one specified by the tanzu builder inventory plugin-group deactivate which requires some flags:

$ tz builder inventory plugin-group deactivate
Error: required flag(s) "name", "publisher", "repository", "vendor" not set
2023-03-24T16:24:37-04:00 [x] : required flag(s) "name", "publisher", "repository", "vendor" not set

This means the plugin group search command will still show all the other groups

marckhouzam

Beautiful work @anujc25 !
I have one comment about cleaning up the temporary directory, but all the others are grammatical suggestions.

pkg/plugininventory/plugin_inventory.go

pkg/plugininventory/sqlite_inventory_test.go

cmd/plugin/builder/README.md

cmd/plugin/builder/inventory/inventory_helper.go

cmd/plugin/builder/inventory/inventory_plugin_group.go

anujc25 · 2023-03-24T22:09:44Z

Thanks, @marckhouzam for the detailed comments on the documentation. 👍

anujc25 requested a review from a team as a code owner March 21, 2023 01:48

vmwclabot added the cla-not-required label Mar 21, 2023

anujc25 force-pushed the plugin-groups-v2 branch 2 times, most recently from 4a87aa9 to 73d5824 Compare March 21, 2023 15:59

marckhouzam requested changes Mar 23, 2023

View reviewed changes

anujc25 force-pushed the plugin-groups-v2 branch 3 times, most recently from 52adde0 to 72c9351 Compare March 23, 2023 17:53

vuil reviewed Mar 24, 2023

View reviewed changes

cmd/plugin/builder/README.md Outdated Show resolved Hide resolved

cmd/plugin/builder/README.md Outdated Show resolved Hide resolved

anujc25 added 7 commits March 23, 2023 22:25

Address comments

a3c066f

Move Mandatory field to new PluginGroupPluginEntry from PluginIdentifier

01015d2

Add UpdatePluginGroupActivationState unit tests

a525650

Update plugin-tooling.mk

8867ccd

Fix unit test

a0b3ac6

Update builder/README.md

9dbb887

anujc25 force-pushed the plugin-groups-v2 branch from 16182cf to 9dbb887 Compare March 24, 2023 05:25

chandrareddyp reviewed Mar 24, 2023

View reviewed changes

marckhouzam approved these changes Mar 24, 2023

View reviewed changes

Address comments

8995496

anujc25 force-pushed the plugin-groups-v2 branch from ceaf39b to 8995496 Compare March 24, 2023 22:34

anujc25 merged commit 4113418 into vmware-tanzu:main Mar 24, 2023

vuil added the kind/feature Categorizes issue or PR as related to a new feature label Apr 7, 2023

marckhouzam added this to the v0.90.0 milestone Oct 20, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement plugin-group publishing tooling with builder plugin #123

Implement plugin-group publishing tooling with builder plugin #123

anujc25 commented Mar 21, 2023 •

edited

Loading

marckhouzam left a comment

vuil left a comment

chandrareddyp commented Mar 24, 2023 •

edited

Loading

chandrareddyp commented Mar 24, 2023

chandrareddyp commented Mar 24, 2023

anujc25 commented Mar 24, 2023

chandrareddyp commented Mar 24, 2023 •

edited

Loading

chandrareddyp left a comment

anujc25 commented Mar 24, 2023

marckhouzam commented Mar 24, 2023

marckhouzam left a comment

anujc25 commented Mar 24, 2023

Implement plugin-group publishing tooling with builder plugin #123

Implement plugin-group publishing tooling with builder plugin #123

Conversation

anujc25 commented Mar 21, 2023 • edited Loading

What this PR does / why we need it

Which issue(s) this PR fixes

Describe testing done for PR

Release note

Additional information

Special notes for your reviewer

marckhouzam left a comment

Choose a reason for hiding this comment

vuil left a comment

Choose a reason for hiding this comment

chandrareddyp commented Mar 24, 2023 • edited Loading

chandrareddyp commented Mar 24, 2023

chandrareddyp commented Mar 24, 2023

anujc25 commented Mar 24, 2023

chandrareddyp commented Mar 24, 2023 • edited Loading

chandrareddyp left a comment

Choose a reason for hiding this comment

anujc25 commented Mar 24, 2023

marckhouzam commented Mar 24, 2023

marckhouzam left a comment

Choose a reason for hiding this comment

anujc25 commented Mar 24, 2023

anujc25 commented Mar 21, 2023 •

edited

Loading

chandrareddyp commented Mar 24, 2023 •

edited

Loading

chandrareddyp commented Mar 24, 2023 •

edited

Loading