New & Improved Package Structure: efficient OCI storage, remote & local signing, OCI import, sget deprecation

[Madeline-UX]

Discussed in #1298

^{Originally posted by jeff-mccoy January 30, 2023}

New universal package structure

This proposal seeks to define a normalized package structure for Zarf packages. In addition to providing signing without depending on cosign sget, it will also allow treating local or remote packages the same way. Additionally, this proposal discusses a new command, zarf package publish for allowing users to publish packages to a remote registry for deployment or reuse in other packages.

Related Issues (in priority order)

1. Spike: Package to OCI spec POC  #1326
2. Each component folder is now a tarball instead of a directory #1322
3. images.tar is now expanded into ./images/ #1324
4. outer tarball no longer compressed #1321
5. zarf.yaml flag metadata.uncompressed now changes compression of ./components/<name>.tar. #1323
6. Warn users about compatibility of zarf packages between Zarf verisons #1325
7. Support Importing OCI Components #1060

Package Structure Changes

• outer tarball no longer compressed
• zarf.yaml flag metadata.uncompressed now changes compression of ./components/<name>.tar.
• each component folder is now a tarball instead of a directory
• images.tar is now expanded into ./images/
• backwards compatibility is maintained via only using new capabilities if they exist:
  • if the ./images/ dir exists, it will be used instead of ./images.tar
  • if ./components/<name>.tar.? exists, it will be used instead of the ./components/<name>/ dir
  • if ./checksums.txt exists it will be processed
    • if metadata.checksumSignature is also not an empty string, it will used to verify the signature of the checksums file
  • if ./sboms/.tar.zst exists it will be used instead of the ./sboms/ dir
  • if ./zarf.yaml.sig exists it will be used to verify the signature of ./zarf.yaml

Top Level Structure

./checksums.txt
./components/<name>.tar.zst
./images/**
./sboms.tar.zst
./zarf.yaml
./zarf.yaml.sig

Old Package Structure

zarf-package-kafka-strimzi-demo-arm64.tar.zst
├── components
│   ├── baseline # -> baseline.tar.zst
│   │   ├── charts
│   │   │   └── strimzi-kafka-operator-0.29.0.tgz
│   │   ├── manifests
│   │   │   └── manifests
│   │   │       ├── kafka-topic.yaml
│   │   │       └── kafka.yaml
│   │   └── values
│   │       └── strimzi-kafka-operator-0.29.0-0
│   └── kafka-tools # -> kafka-tools.tar.zst
│       └── files
│           └── 0
├── images.tar # -> images/*
├── sboms # -> sboms.tar.zst
│   ├── compare.html
│   ├── quay.io_strimzi_kafka_0.29.0-kafka-3.2.0.json
│   ├── quay.io_strimzi_operator_0.29.0.json
│   ├── sbom-viewer-quay.io_strimzi_kafka_0.29.0-kafka-3.2.0.html
│   ├── sbom-viewer-quay.io_strimzi_operator_0.29.0.html
│   ├── sbom-viewer-zarf-component-kafka-tools.html
│   └── zarf-component-kafka-tools.json
└── zarf.yaml

New Package Structure

zarf-package-kafka-strimzi-demo-arm64.tar
├── checksums.txt # sha256sums of all files in the package except the zarf.yaml* files
├── components
│   ├── baseline.tar.zst # compressed from components/baseline
│   └── kafka-tools.tar.zst # compressed from components/kafka-tools
├── images # expanded from images.tar
│   ├── 01b1f1f1de1aa5d9a192c9308f23e3aac2f70e49b5138d20c015e903e594166f.tar.gz
│   ├── 036197be467ed17bb645dab4976d18829a54c64aad8ce4b58c50308cd1214161.tar.gz
│   ├── 374bf787cd457fd983f262c45c2dd1f1f16b4525461164d0fbfdbdd088ec9851.tar.gz
│   ├── 40c5950cb2f4d9e4a5e4f3231ffc4a80584b3a593c7a30bac482ed213e84f7f2.tar.gz
│   ├── 4b4bf95f1bc1a0d0acb9bafc4c38a1734c80b96b8a8e33464d5edeb94b50f101.tar.gz
│   ├── 4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1.tar.gz
│   ├── 52c4b995397666a08ee66ad7e5b146c93a835d5e0cef9d7f43668005de0f8721.tar.gz
│   ├── 5a3cb477fb6927486e1de4005a9f6d303105318a10f3aaf20b9ac96b75b4063d.tar.gz
│   ├── 5ece2c7bdc1cb7a7d0e8fdd82a6b718c26481bdf7106a0aa9b07bfb49f15390c.tar.gz
│   ├── 607cadd9fc076f6aa4c1c08b269c986cc6de8dd2f21a4bfbaf8fd0f1ad25f0e8.tar.gz
│   ├── 676abaef5cf46c0214d6895f00c1c9de9e39e3867b1001c70233531a06f39081.tar.gz
│   ├── 6b941698d1b1d32850d0503afc02c8d0cda80973fe3fc4557f91a815d9979791.tar.gz
│   ├── 6ea53c55475943271e40630d710aa7273879ed87f51176c5aaab07760231864e.tar.gz
│   ├── 9c9d0e98f33016ea08c3ce3e8b14bbc972210c7b4d376879ff1138a4ed76e6ea.tar.gz
│   ├── a5b9f2a044430ec591cab07d63b05c59a513bc2095cc1a0794671d9a225c2ca5.tar.gz
│   ├── a73cc92a765199bc51556a56e450dd53f3240b3a1622d6095731ba0c0caad2e0.tar.gz
│   ├── b2ad865460cc2204b2e2884a0555bb41760d4449f3fe0e36bb441d740c3e9bc6.tar.gz
│   ├── cc86879c05681cf853ca3a6773d171c33549c7e70d6a5a645ba75851a4744741.tar.gz
│   ├── ccd4421c3004e570a9ca0040b297624bf11cf8e7d6a5f4c4e2f6fe3bf3af3b2e.tar.gz
│   ├── d2ca1bcb8952a0df0ef3f48ac2e1b1c829a4c8f3a50fb8c4fe115518bf901430.tar.gz
│   ├── d7fc738c2b482c640f65d249d9a992cc3884eaf92a4f7336eee81905a15e68bf.tar.gz
│   ├── eb8600330c7fda365d3580238595890e252f1f326c133c71777a0b7f1e5a13ed.tar.gz
│   ├── ef37c8e9c16fe19c428ac96ae2f254e0bbf3ab46bc5bf6ac5fa9493746bc0dfd.tar.gz
│   ├── f91a5b4860e95380f988cdc91fb3b192cec41f3527aed80364cb7df3f1e0178c.tar.gz
│   ├── manifest.json
│   ├── sha256:7af7a0f44d24587add4881175ce4328c4743e39f044e78454a243019e2f04277
│   └── sha256:bd73ec29f4295bb439cd618b35fd8ae4cceb1d6312cd45f1864e4541784ceb69
├── sboms.tar.zst # compressed from sboms/
├── zarf.yaml
└── zarf.yaml.sig # signature of zarf.yaml

Zarf Package Config (Kafka Strimzi Example)

kind: ZarfPackageConfig
metadata:
  name: "kafka-strimzi-demo"
  description: "Demo tiny Zarf Kafka deployment"
  architecture: "arm64"
  version: "v0.1.0"
build:
  terminal: "minion.local"
  user: "jeff"
  architecture: "arm64"
  timestamp: "Mon, 09 Jan 2023 12:10:43 -0600"
  version: "v0.23.2-26-gb33622f1"
  checksumSignature: "MEQCIAU4wPBpl/U5Vtdx/eJFgR0nICiiNCgyWPWarupH0onwAiAv5ycIKgztxHNVG7bzUjqHuvK2gsc4MWxwDgtDh0JINw=="

Publishing Zarf Packages

Using the new package structure detailed above, publishing to an OCI Distribution Registry will assume that the layers will be written/optimized for reuse which means that the OCI artifact will operate in lieu of a tarball. When the package is published to a registry, two different artifacts will be published: the package and a skeleton package. The skeleton package will be a copy of the package with all layers that can be pulled remotely removed. This is intended to be used to extend a package or in the case of zarf package create. The standard package will be published for directly deploying a package.

Publish a package will follow the convention <repo>/<package-name>:<version>-<arch> and <repo>/<package-name>:<version>-skeleton. Using the package example above:

Command:

zarf package publish defenseunicorns

Result:

Create and publish OCI artifacts:
- defenseunicorns/kafka-strimzi-demo:v0.1.0-arm64
- defenseunicorns/kafka-strimzi-demo:v0.1.0-skeleton

Skeleton Package

A skeleton package is a special package that is essentially a copy/paste of the local folder where the zarf.yaml is defined. Note, in this solution referencing files outside of the current zarf.yaml folder path will not work properly. One particular note is Kustomizations, they will not be processed or converted into manifests but will simply be copied into the OCI artifact for later composition if imported.

Deploying OCI Packages

Deploying will be very similar to prior sget behavior. Using the package example above:

zarf package deploy oci://defenseunicorns/kafka-strimzi-demo:v0.1.0-arm64

Alternatively, we could look at defining a new command:

zarf pacakge deploy-oci defenseunicorns/kafka-strimzi-demo:v0.1.0-arm64

Eventually the Zarf UI could include the ability to search a registry for packages and deploy or download them.

Importing OCI Packages/Components

It can be confusing, but you cannot directly import a package in Zarf. A component is an atomic unit imported/extended within a package. We refer to this as composable components. It is possible to import a package/component that is only a single component. The challenge with OCI imports of packages/components is that we will have to copy things locally before finishing the composition. Additionally, composable components today do not have a concept of special handling for remote assets already pulled into a package/local location. This is where the skeleton component proposal comes in. The skeleton package will be loaded into a temp directory and referenced like a local package/component. This will occur at the normal composition stage, before the confirm step on zarf package create.

Example OCI import

kind: ZarfPackageConfig
metadata:
  name: "demo-with-terraform-import"

components:
  - name: terraform
    import:
      oci: defenseunicorns/terraform:1.3.7

This example would then call oci://defenseunicorns/terraform:1.3.7-skeleton and include it as a part of the package definition on zarf package create. As with local imports, this would be completely transparent on zarf package deploy.

Decision: I'm not sure if we should include a checksum or signature file for skeleton files as they will be incomplete and regenerated anyway during package create.