Implement new manifest cache design: cache manifest lists and manifest+config pairs

[chanseokoh]

(See #2707 about the new JSON data structure (ImageMetadataTemplate) for the cache file manifests_configs.json.)

The new cache supports storing and retrieving both OCI and Docker manifests and manifest lists.

• If a base image ref is not a manifest list, Jib will result in caching a single-element ImageMetadataTemplate.manifestsAndConfigs without caching a manifest list (null ImageMetadataTemplate.manifestList).
• If a base image ref is a manifest list, Jib will cache the entire original manifest list. However, for actual manifests, it will cache only a subset that are downloaded and used.

Cache API Changes

• Current manifest cache API

  • Returns and writes a single manifest and config pairs (ManifestAndConfigTemplate).

    public Optional<ManifestAndConfigTemplate> retrieveMetadata(ImageReference imageReference)

    public void writeMetadata(
        ImageReference imageReference,
        BuildableManifestTemplate manifestTemplate,
        ContainerConfigurationTemplate containerConfigurationTemplate)

• New API

  • Returns and writes a manifest list and a list of manifest and config pairs (ImageMetadataTemplate).

    public Optional<ImageMetadataTemplate> retrieveMetadata(ImageReference imageReference)

    public void writeMetadata(
        ImageReference imageReference,
        @Nullable ManifestTemplate manifestList,
        List<BuildableManifestTemplate> manifests,
        List<ContainerConfigurationTemplate> containerConfigurations)

Updating Manifest Cache File

The implementation replaces manifests_configs.json whenever running Jib online. Jib will store only those manifests and configs Jib downloaded and used in the last build–no accumulation of past manifests and configs.

Cache Invalidation in the New Release

The new cache file is manifests_configs.json. Existing manifest.json and config.json will not be used. The user upgrading Jib will lose previous manifest and config caches.

[chanseokoh]

Will update CHANGELOG and do some more manual end-to-end tests.

[loosebazooka]

I don't know if I know enough of this part of the code to review can we do in person on monday morning?

[loosebazooka]

Thanks for refactoring this, I had a much easier time following the logic. Some simple questions, but otherwise lgtm.

[loosebazooka]

Does this then throw an error for OciIndexTemplate? Perhaps we can give the user a better error message (I thought Verify is for catching programming/integration issues?)

[chanseokoh]

Basically the refactored code doesn't change the current behavior in this regard. The current code would simply break with ClassCastException if manifestTemplate could ever be of OciIndexTemplate, as it can't cast OciInexTemplate into BuildableManifestTemplate in jsonManifestToImage(). That is, it's an impossible condition; we never see OciIndexTemplate, as the manifest puller doesn't tell a registry that it can accept an OCI image index. All the registries in practice so far have been returning a Docker manifest list.

[loosebazooka]

Similarly here, what happens when schema version is not 2, does this just kill our build with a confusing error message?

[chanseokoh]

Will kill the build. It's an impossible condition the user should never hit. If we see it, it's a bug for us to fix. That is, we should pull a container config only when it's a schema 2 (i.e., when OciImageTemplate or V22ManifestTemplate), because the V2.1 manifest embeds a container config.

But actually, BuildableManifestTemplate is always schema version 2. The check line is only to help us locate places to update more easily in the future when a schema version 3 is introduced in the spec.

[chanseokoh]

Will update CHANGELOG and do some more manual end-to-end tests.

I will add CHANGELOG once this is fully implemented.