Central Repository: plugin install and sync #45

marckhouzam · 2023-01-26T16:41:02Z

What this PR does / why we need it

This commit creates a new OCI discovery implementation which handles the Central Repository format.

When enabling the "features.global.central-repository" feature flag, it is now possible install plugins from a Central Repository.

tanzu plugin install: working
tanzu plugin clean: working
tanzu plugin upgrade: working
tanzu plugin describe: working
tanzu plugin list: lists all plugins from the Central Repo. #53 fixed this.
tanzu plugin sync: only syncs from current contexts, but not yet installing from Central repo. This will be part of another PR, since it is a substantial change.

This first step fetches all plugins from the SQLite DB all the time. Then the plugin manager filters the list of plugins to find the correct one. I took this approach to reduce the code changes necessary, but this will optimized in a follow-up effort.

Describe testing done for PR

I've tested this with a test Central Repository from #41. You can follow these steps:

: Checkout PR 41 to start a test central repo locally in docker
make start-test-central-repo

: Checkout this PR
make build

: remove you configuration files to start fresh
tz config set features.global.central-repository true
tz plugin source add -n default -t oci -u localhost:9876/tanzu-cli/plugins/central:large
export ALLOWED_REGISTRY=localhost:9876

: Now you can start using the test Central Repo
tz plugin list # This will later not list everything
tz plugin install cluster
tz plugin install cluster --target tmc
tz tmc cluster

tz plugin install feature
tz  feature
tz k8s  feature

tz plugin list
tz plugin upgrade  feature
tz plugin delete  cluster --target tmc

tz plugin clean

To test plugin sync:

tz context create --name tmc-unstable --endpoint unstable.tmc-dev.cloud.vmware.com --staging
# See that only plugins from the context are synched, not all the ones in the central repo

tz context create --name defaultK3d --kubecontext k3d-k3s-default --kubeconfig /Users/kmarc/.config/k3d/k3s-default/kubeconfig.yaml

Release note

The Tanzu CLI now supports discovering plugins from a Centralized Repository of plugins.
All plugins can be found in that repository and are installed from that repository.
The synchronization of plugin (through "plugin sync" or setting a context) no longer
installs "standalone" plugins; instead a user must explicitly install such plugins
with the "plugin install" command.

Additional information

There are no new unit test in this PR. They will be added in a follow-up.
I'm not handling the test plugin that might be delivered with a plugin

pkg/discovery/oci_central.go

marckhouzam · 2023-01-27T13:16:47Z

BTW, with this PR, you can easily manually access the DB of the Central Repo being used, if needed:
sqlite3 ~/.cache/_tanzu/plugin_data/central.db.

anujc25 · 2023-01-30T03:58:23Z

pkg/discovery/oci_central.go

+func (od *OCIDiscoveryForCentralRepo) getPluginsFromDB(dbFilePath string) ([]Discovered, error) {
+	db, err := sql.Open("sqlite3", dbFilePath)
+	if err != nil {
+		return nil, errors.Wrapf(err, "failed to open the DB for discovery '%s'", od.Name())
+	}
+	defer db.Close()
+
+	// We need to order the results properly because the logic of processRows()
+	// expects an ordering of PluginName, then Target, then Version.
+	// The column order must also match the order used in getNextRow().
+	dbQuery := "SELECT PluginName,Target,RecommendedVersion,Version,Hidden,Description,Publisher,Vendor,OS,Architecture,Digest,URI FROM PluginBinaries ORDER BY PluginName,Target,Version;"
+	rows, err := db.Query(dbQuery)
+	if err != nil {
+		return nil, errors.Wrapf(err, "unable to setup DB query for discovery '%s'", od.Name())
+	}
+	defer rows.Close()
+
+	allPluginPtrs, err := od.processRows(rows)
+	if err != nil {
+		return nil, err
+	}
+
+	// Convert from plugin pointers to plugins
+	// TODO(khouzam): continue optimizing by converting every call
+	// to using pointers
+	var allPlugins []Discovered
+	for _, pluginPtr := range allPluginPtrs {
+		allPlugins = append(allPlugins, *pluginPtr)
+	}
+	return allPlugins, nil
+}


Should we put this into a separate golang package that handles all the SQL queries together?
We need to add more SQL queries that can be used at multiple places. Combining all SQL queries related stuff into separate packages might be better. WDYT? @marckhouzam

That is a good idea. I will work something out and update.

+1 to factoring out the db access from the logic of obtaining the 'db' via the OCI registry to minimize churn should we elect to use a different db backend in the future.

I've pushed a commit to address this.

The approach I took is to create a new DiscoveryBackend interface. A DiscoveryBackend represents the content management logic behind a plugin Discovery. The commit also creates an implementation of this new DiscoveryBackend for the SQLite backend of the Central Repository.

Note: The DiscoveryBackend support is included in the "discovery" package itself at the moment. This is so that it can use the Discovered type. Making the new DiscoveryBackend part of its own package would prevent it from using the Discovered type due to a cyclical dependency. This would then force us to create yet another type to characterize plugins for the DiscoveryBackend to use. This new type would then need to be converted to the Discovered type by the Discovery.

An alternative would be to move the Discovered type to its own package that could be imported by both the "discovery" package, and a new "discoverybackend" package.

We should discuss this.

We have discussed this and we have agreed that we will create a new type that the DiscoveryBackend can return, so as to decouple the DiscoveryBackend from the discovery package.
This will be done in a follow-up PR to allow this PR to get in now.

marckhouzam · 2023-01-30T14:20:30Z

I've added a commit to this PR to so that tz plugin sync now works as expected by only synching plugins provided by the context.

marckhouzam · 2023-01-30T15:19:28Z

I've added a commit to this PR to so that tz plugin sync now works as expected by only synching plugins provided by the context.

Actually, this is incomplete for the Central Repo feature. Context plugins need to be installed from the Central Repo, which this commit does not. I will update.

marckhouzam · 2023-01-30T16:33:59Z

I've added a commit to this PR to so that tz plugin sync now works as expected by only synching plugins provided by the context.

Actually, this is incomplete for the Central Repo feature. Context plugins need to be installed from the Central Repo, which this commit does not. I will update.

~~Properly updating the "plugin sync" logic to use the Central Repository for installation is a fairly big change as it affects Kubernetes and Rest discoveries. I will leave it for a separate PR.~~

~~So, for this PR "plugin sync" will only sync context plugins, but will still install them from the URI specified in the discovery.~~

This has been done with Anuj's guidance 👍

prkalle

The overall changes look good. Thanks!

pkg/pluginmanager/manager.go

pkg/fakes/registy.go

vuil

lgtm, thanks for the refactoring and renaming.
I have some nits that are not blocking

pkg/discovery/sqlite_backend.go

pkg/pluginmanager/manager.go

pkg/carvelhelpers/fetcher.go

pkg/discovery/sqlite_backend.go

When enabling the "features.global.central-repository" feature flag, it is now possible install plugins from a Central Repository. tanzu plugin install: working tanzu plugin clean: working tanzu plugin upgrade: working tanzu plugin describe: working tanzu plugin sync: working, now only syncs context plugins. tanzu plugin list: lists *all* plugins from the Central Repo. This will be optimized in a follow-up commit. This first implementation fetches all plugins from the SQLite DB all the time. It is then the plugin manager that filters the list of plugins to find the correct one. We took this approach to reduce the code changes necessary, but this will optimized in a follow-up effort. As part of this effort we create a new DiscoveryBackend interface. A DiscoveryBackend is the inventory management logic behind a plugin Discovery. This commit creates a DiscoveryBackend implementation for the SQLite backend of the Central Repository. Signed-off-by: Marc Khouzam <kmarc@vmware.com>

marckhouzam requested a review from a team as a code owner January 26, 2023 16:41

vmwclabot added the cla-not-required label Jan 26, 2023

marckhouzam force-pushed the feat/centralPluginInstall branch from a7a2ec1 to cb3135a Compare January 26, 2023 18:48

anujc25 reviewed Jan 27, 2023

View reviewed changes

pkg/discovery/oci_central.go Outdated Show resolved Hide resolved

marckhouzam force-pushed the feat/centralPluginInstall branch from a0016d3 to 009b63b Compare January 27, 2023 18:20

anujc25 reviewed Jan 30, 2023

View reviewed changes

marckhouzam changed the title ~~Central Repository: plugin install~~ Central Repository: plugin install and sync Jan 30, 2023

marckhouzam force-pushed the feat/centralPluginInstall branch from d9e2e95 to 3dc26f0 Compare January 30, 2023 14:19

marckhouzam force-pushed the feat/centralPluginInstall branch from 3dc26f0 to f49f84d Compare January 30, 2023 15:03

marckhouzam force-pushed the feat/centralPluginInstall branch from f49f84d to dd31e2b Compare January 30, 2023 16:06

marckhouzam force-pushed the feat/centralPluginInstall branch from dd31e2b to d198773 Compare January 30, 2023 17:04

marckhouzam mentioned this pull request Jan 31, 2023

[CentralRepo] Add tanzu plugin search command #52

Merged

marckhouzam force-pushed the feat/centralPluginInstall branch 3 times, most recently from 9b2edee to 2e6793f Compare February 1, 2023 21:26

prkalle reviewed Feb 1, 2023

View reviewed changes

pkg/pluginmanager/manager.go Show resolved Hide resolved

pkg/fakes/registy.go Outdated Show resolved Hide resolved

vuil approved these changes Feb 2, 2023

View reviewed changes

pkg/discovery/sqlite_backend.go Outdated Show resolved Hide resolved

pkg/pluginmanager/manager.go Outdated Show resolved Hide resolved

pkg/pluginmanager/manager.go Show resolved Hide resolved

marckhouzam force-pushed the feat/centralPluginInstall branch 2 times, most recently from a8689c3 to 8a8483a Compare February 2, 2023 15:43

anujc25 reviewed Feb 2, 2023

View reviewed changes

pkg/carvelhelpers/fetcher.go Outdated Show resolved Hide resolved

pkg/discovery/sqlite_backend.go Outdated Show resolved Hide resolved

marckhouzam force-pushed the feat/centralPluginInstall branch from 8a8483a to 32de66d Compare February 3, 2023 02:10

marckhouzam merged commit 88be7aa into vmware-tanzu:main Feb 3, 2023

marckhouzam deleted the feat/centralPluginInstall branch February 3, 2023 02:14

vuil added the kind/feature Categorizes issue or PR as related to a new feature label Apr 7, 2023

marckhouzam added this to the v0.90.0 milestone Oct 20, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Central Repository: plugin install and sync #45

Central Repository: plugin install and sync #45

marckhouzam commented Jan 26, 2023 •

edited

Loading

marckhouzam commented Jan 27, 2023 •

edited

Loading

anujc25 Jan 30, 2023

marckhouzam Jan 30, 2023

vuil Jan 31, 2023

marckhouzam Feb 1, 2023

marckhouzam Feb 3, 2023

marckhouzam commented Jan 30, 2023

marckhouzam commented Jan 30, 2023

marckhouzam commented Jan 30, 2023 •

edited

Loading

prkalle left a comment

vuil left a comment

Central Repository: plugin install and sync #45

Central Repository: plugin install and sync #45

Conversation

marckhouzam commented Jan 26, 2023 • edited Loading

What this PR does / why we need it

Describe testing done for PR

Release note

Additional information

marckhouzam commented Jan 27, 2023 • edited Loading

anujc25 Jan 30, 2023

Choose a reason for hiding this comment

marckhouzam Jan 30, 2023

Choose a reason for hiding this comment

vuil Jan 31, 2023

Choose a reason for hiding this comment

marckhouzam Feb 1, 2023

Choose a reason for hiding this comment

marckhouzam Feb 3, 2023

Choose a reason for hiding this comment

marckhouzam commented Jan 30, 2023

marckhouzam commented Jan 30, 2023

marckhouzam commented Jan 30, 2023 • edited Loading

prkalle left a comment

Choose a reason for hiding this comment

vuil left a comment

Choose a reason for hiding this comment

marckhouzam commented Jan 26, 2023 •

edited

Loading

marckhouzam commented Jan 27, 2023 •

edited

Loading

marckhouzam commented Jan 30, 2023 •

edited

Loading