feat(helm): Require Charts to be locally available

[sh0rez]

As proposed by this design doc, this modifies the helmTemplate native function in a way that it expects a relative path instead of the repo/name identifier for Charts.

Before, this feature relied on the system wide Helm cache, which is independent from the Tanka project and not controlled by git. This does not play well with our aim for hermeticity and strong reproducibility at all.

To overcome this, helmTemplate resolves the chart relative to the calling file. This suggests to keep Helm Charts as close to the consuming source code as possible.

Using the calling file as a fixpoint is also the only way I came up with that is independent from the current working directory Tanka was invoked in, which is important as tk show must continue to not only work from the project root, but also inside any nested folder.

Depends on #368, will be rebased once merged

[Duologic]

I need a bit more explanation on how this is supposed to work. Where is CalledFrom set? What value is it supposed to be (relative/absolute)? Does it allow me to call tk export from anywhere? Is it the same for all helm charts or does it allow a Helm chart to be vendored at arbitrary places?

[sh0rez]

Where is CalledFrom set? What value is it supposed to be (relative/absolute)?

Quoting from the design doc:

A new challenge here is local resolving of paths. Above refers to a relative path inside the library. The helmTemplate function must resolve that path for helm to use. Unfortunately it has no knowledge which file it was called from as of today. Because Jsonnet won’t provide that information magically for us, we need to tell ourselves:

local helm = (import “helm.libsonnet”).new(std.thisFile)

The helm-util library would be instructed with the callers context while importing.

Subsequent helm.template calls automatically inject above information into the conf object they hand to the native func. While we actually need the enclosing directory of std.thisFile, this information is hard to get in Jsonnet, so we pass std.thisFile instead. It is supposed to be an absolute path on disk to the calling file.

Does it allow me to call tk export from anywhere?

Yes! This maintains the current behaviour of all tk commands – because they take a directory as the first argument (environments/default, ./default, ../default, etc), you can call tk export from whereever you want. Using the callers context works well with that, because it is independent from the current working directory the tk command is invoked from (which is not guaranteed to be the project root or even in the project at all)

Is it the same for all helm charts or does it allow a Helm chart to be vendored at arbitrary places?

You can vendor it wherever you like inside the project, but you are required to reference the Chart using a relative path.

Consider:

/charts/grafana
/environments/default/main.jsonnet
/lib/mysql/mysql.libsonnet
/lib/mysql/charts/mysql

Here, environments/default/main.jsonnet would import grafana as ../../charts/grafana, while lib/mysql/mysql.libsonnet could import mysql as ./charts/mysql.

It seems reasonable to me to think of Helm Charts as yet another source format consumed by Jsonnet. Bbeing it .jsonnet, .json, .yaml, Helm Chart shouldn't matter in this mental model. In consequence, It appears straight forward to keep Chart sources as close to the consumer as possible (like lib/mysql does)

Does that answer your questions?

[Duologic]

Subsequent helm.template calls automatically inject above information into the conf object they hand to the native func. While we actually need the enclosing directory of std.thisFile, this information is hard to get in Jsonnet, so we pass std.thisFile instead. It is supposed to be an absolute path on disk to the calling file.

This clever, but I wonder if it might be too clever?

You can vendor it wherever you like inside the project, but you are required to reference the Chart using a relative path.

I think that might come and bite us when we want to take a more rigorous approach on package management.

I wonder if this PR might be running ahead of package management in Tanka in general? I think we need to slow down a bit on this track.

[sh0rez]

(mirroring the discussion from the last Tanka weekly in here)

I think that might come and bite us when we want to take a more rigorous approach on package management.

Package management is a big topic, and a decision appears fairly distant, given all what needs to be considered. We certainly need to keep this in mind and adapt as needed when the time comes :)

I wonder if this PR might be running ahead of package management in Tanka in general? I think we need to slow down a bit on this track.

We thought about this, and came to the conclusion these problems are really orthogonal. We do need to have a general discussion about package management and the path forward, but this will take considerable time, thinking and tradeoffs.

Helm Charts in the sense we think about them are resources most of the time exclusively consumed in a library, being that external to the project or not. This suggests Chart availability should be managed at the project level, similar to how a library maintainer that might use YAML or a lot of importstr underneath is responsible for that as well.

Because it integrates with the current jsonnet-bundler workflow and guarantees, it gets this issue off our back until we tackle package management in a future release.

[jdbaldry]

LGTM. Do we need the related changes to the helm-util library to support the setting of conf.calledFrom?

[sh0rez]

Yes we do. Will open that PR soon