Support subresource integrity format for download() checksums.

[jmillikin]

Closes #4881

Notes for the reviewer:

• I will add tests (in the existing bash repository rule tests) after preliminary approval of the implementation.
• Supported digests are the minimal recommended set from https://www.w3.org/TR/SRI/: SHA-256, SHA-384, and SHA-512. I did not enable support for SHA-1 checksums in this PR.
• Do we want to make the struct returned by download() and download_and_extract() less strict about having a sha256 field? If we allowed the returned field to depend on the integrity list, then users of SHA-384 and SHA-512 checksums wouldn't need to re-checksum the downloaded file.

[jmillikin]

def _repo_test(ctx):
    downloaded = ctx.download(
            url = ["https://github.com/bazelbuild/bazel/releases/download/0.21.0/bazel-0.21.0-darwin-x86_64.sig"],
            output = "test-download.bin",   
            integrity = [
                "sha256-YTmQnwlYCwr3Er6ellav7BH4VTgHi9hbe5vvOfA0P00=",
                "sha384-uIEYsgRb01KvTxnqnmbAlEwedx4VBiZlGlAgjL6Qb+K2h4CzDZbo6FRjA5xKqdDr",
                "sha512-/+EGy2d+zm/rB6H5FUEzisWW6dZ+N64/8Vrl/W/h0u8oI8nrHO/wvCguUgsp3H9ZKSS13cjk1JhhU0Cfn0a6DA==",
            ],
    )
    print("downloaded: ", downloaded)

    ctx.file("WORKSPACE", "workspace(name = {name})\n".format(name = repr(ctx.name)))
    ctx.file("BUILD.bazel", "")

repo_test = repository_rule(_repo_test)

$ ~/src/bazel/bazel-bin/src/bazel query @repo_test//:*
INFO: Invocation ID: 1e019ff7-7544-44e4-bf81-62382c261a54
DEBUG: /home/john/src/testing-bazel-integrity/repo.bzl:9:5: downloaded:  struct(sha256 = "6139909f09580b0af712be9e9656afec11f85538078bd85b7b9bef39f0343f4d", sha512 = "ffe106cb677ece6feb07a1f91541338ac596e9d67e37ae3ff15ae5fd6fe1d2ef2823c9eb1ceff0bc282e520b29dc7f592924b5ddc8e4d4986153409f9f46ba0c")
@repo_test//:BUILD.bazel
Loading: 1 packages loaded

[irengrig]

/cc @aehlig

[jin]

Ping @dslomov, PTAL

[jmillikin-stripe]

@dslomov Have you had time to review this yet?

[dslomov]

ping @aehlig

[dslomov]

General idea and the API shape sounds good to me.
The return struct should probably have an integrity field, as well as sha256, sha512 fields (as appropriate)

[jmillikin]

I ran into a bunch of Git conflicts when trying to rebase this to current master. The changes to plumb checksum types through the HTTP layer have been extracted to #8714; this PR will become only the changes to repository_ctx.download().

[jmillikin]

🎉 Rebase complete, tests added, and all checks green.

I had to back off on one of the new features, multiple checksums, because the semantics were too hard to reason about. In browsers the download is allowed if any checksum matches, and I'm not 100% sure that's the right behavior for Bazel.

[jmillikin]

PR #8714 has merged, and rebase onto it was successful.

[jmillikin-stripe]

Gentle ping

[jmillikin]

@aehlig @dslomov Have you had time to review this yet?

[aehlig]

LGTM. Thanks. Will import.

[alexeagle]

packages uploaded to the npm registry using version 4 or earlier of the npm CLI are published with a SHA-1 hash. Is there a particular reason you didn't include that @jmillikin ? I'll send a PR to add it, so that Bazel can download everything that appears in an npm package-lock.json

[jmillikin-stripe]

Bazel doesn't currently support SHA1 as a download checksum, and I didn't want to change that as part of my PR. I think it would be fine to send a separate PR to enable SHA1 if that would help you.

You might also be able to use Yarn, which supports recalculating checksums when writing its lockfile. We use that functionality in our builds so that every package has at least a SHA256 checksum.

[alexeagle]

Thanks for your reply. I opened #12777 to add SHA-1 in this spot.

The feature we are working on for rules_nodejs would have bazel download the tarballs rather than npm or yarn, so we can't rewrite the lockfile (it wouldn't have the bytes on disk yet to calculate a new hash anyway).