Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Bzlmod in Bazel #13316

Closed
meteorcloudy opened this issue Apr 8, 2021 · 5 comments
Closed

Implement Bzlmod in Bazel #13316

meteorcloudy opened this issue Apr 8, 2021 · 5 comments
Assignees
Labels
area-Bzlmod Bzlmod-specific PRs, issues, and feature requests P1 I'll work on this now. (Assignee required) team-ExternalDeps External dependency handling, remote repositiories, WORKSPACE file. type: feature request

Comments

@meteorcloudy
Copy link
Member

Implement the new dependency management system proposed in https://docs.google.com/document/d/1moQfNcEIttsk6vYanNKIy3ZuK53hQUFq1b1r0rmsYVg/edit?usp=sharing

@meteorcloudy meteorcloudy added team-ExternalDeps External dependency handling, remote repositiories, WORKSPACE file. P2 We'll consider working on this in future. (Assignee optional) team-OSS Issues for the Bazel OSS team: installation, release processBazel packaging, website type: feature request labels Apr 8, 2021
@meteorcloudy
Copy link
Member Author

We'll first build a MVP in the bzlmod branch.

bazel-io pushed a commit that referenced this issue Jun 18, 2021
(#13316)

This CL implements ModuleFileFunction, a SkyFunction that takes the name and version of a module, finds and evaluates its MODULE.bazel file and returns the evaluated result.

This is not hooked into the main SkyFrame route yet (RepositoryDelegatorFunction has yet to call this).

PiperOrigin-RevId: 380141062
bazel-io pushed a commit that referenced this issue Jun 18, 2021
…erride type

(#13316)

We're "denormalizing" the override API: whereas you'd write "override_dep(name='X', Y_override(...))" before, you'd now write "Y_override(module_name='X', ...)".

PiperOrigin-RevId: 380173439
bazel-io pushed a commit that referenced this issue Jun 21, 2021
(#13316)

This SkyFunction performs module discovery by starting out from the root module and iteratively adding direct dependencies into the graph.

PiperOrigin-RevId: 380563418
bazel-io pushed a commit that referenced this issue Jun 22, 2021
RepoSpec holds information of a repository rule definition, including where the rule class is defined and the attributes of a repository rule. It's the output of Bzlmod resolution and will be used to create a repository rule instance.

Related: #13316

RELNOTES: None
PiperOrigin-RevId: 380784644
bazel-io pushed a commit that referenced this issue Jun 24, 2021
Related: #13316

RELNOTES: None
PiperOrigin-RevId: 381261666
bazel-io pushed a commit that referenced this issue Jun 28, 2021
(#13316)

This SkyFunction takes the result of DiscoveryFunction and runs MVS on it, resulting in a dep graph that contains only 1 version of each dependency (the highest requested one).

PiperOrigin-RevId: 381844359
bazel-io pushed a commit that referenced this issue Jun 30, 2021
(#13316)

To implement the index registry outlined in the design doc, we need to make HTTP GET requests to certain URLs and read their contents immediately (instead of downloading and writing them to local disk).

Also added a way to distinguish a 404 from other unrecoverable errors, since 404 is semantically different from, say a 401, in terms of the registry list (earlier registries take precedence and delegate to later registries only when the requested module is not found).

PiperOrigin-RevId: 382304312
bazel-io pushed a commit that referenced this issue Jul 2, 2021
(#13316)

Before we were requesting the ModuleFileValues one by one. This does not result in things being slow per se (since the individual ModuleFileFunction calls should be parallelized by Skyframe) but technically using getValues() can make SkyValue dependencies cleaner (since there is no dependency between the same "level" of SkyKeys being requested here).

PiperOrigin-RevId: 382700697
bazel-io pushed a commit that referenced this issue Jul 2, 2021
(#13316)

This CL implements the basic "Index" registry handling (which is what the Bazel Central Registry will be using). A registry object does two things right now: one is fetching the module file (used during ModuleFileFunction), and the other is telling later stages how to instantiate a repo rule to fetch the contents of the module (used during BzlmodRepoRuleFunction).

PiperOrigin-RevId: 382739746
bazel-io pushed a commit that referenced this issue Jul 2, 2021
- BzlmodRepoRuleHelper: calculates the RepoSpec for a given repository. The actual calculation will be implemented in following changes.

- BzlmodRepoRuleFunction: Take a given repository name, look up its RepoSpec via BzlmodRepoRuleHelper, and then create a repository rule out of it. The result will be used in RepositoryDelegatorFunction.

Related: #13316

RELNOTES: None.
PiperOrigin-RevId: 382745079
bazel-io pushed a commit that referenced this issue Jul 7, 2021
Related: #13316

RELNOTES: None.
PiperOrigin-RevId: 383380283
bazel-io pushed a commit that referenced this issue Jul 7, 2021
(#13316)

This flag is repeatable and used by ModuleFileFunction to determine where to look for modules.

This is not yet testable with an integration test since this code path isn't hooked into production yet (we'll probably need a flag for that).

PiperOrigin-RevId: 383385942
bazel-io pushed a commit that referenced this issue Jul 7, 2021
Those features are needed by the Bzlmod system to fetch Bazel module dependencies from Bazel registries.

- integrity: In addition to "sha256", users can specify the integrity value to verify the downloaded archive content.
- remote_patches, remote_patch_strip: apply patche files downloaded from given URLs. This will be used to apply patch files hosted in Bazel registries.

Related: #13316

RELNOTES: None
PiperOrigin-RevId: 383386903
bazel-io pushed a commit that referenced this issue Jul 12, 2021
Now we can get RepoSpec for non-registry overrides and Bazel module derived repositories,
module rule support will be added later.

Related: #13316

RELNOTES: None
PiperOrigin-RevId: 384178751
bazel-io pushed a commit that referenced this issue Jul 12, 2021
(#13316)

The compatibility_level is essentially the "major version" in semver, except that it's not embedded in the version string, but exists as a separate field. Modules with different compatibility levels participate in version resolution as if they're modules with different names, but the final dependency graph cannot contain multiple modules with the same name but different compatibility levels.

PiperOrigin-RevId: 384185854
bazel-io pushed a commit that referenced this issue Jul 12, 2021
(#13316)

The overrides cannot be nicely encapsulated in SelectionValue because we need the overrides when fetching modules with non-registry overrides, which happens during discovery. Any consumers of overrides should get them from the ModuleFileValue of the root module directly instead.

PiperOrigin-RevId: 384236547
bazel-io pushed a commit that referenced this issue Jul 12, 2021
…f a string

(#13316)

See motivating TODO in ModuleFileGlobals.java. This avoids double-parsing (once at input verification time and once at selection time).

PiperOrigin-RevId: 384243747
bazel-io pushed a commit that referenced this issue Jul 13, 2021
With --experimental_enable_bzlmod=true, Bazel can now load external repositories
from the new dependency management system -- Bzlmod.

Related: #13316

RELNOTES: None
PiperOrigin-RevId: 384424361
bazel-io pushed a commit that referenced this issue Jul 15, 2021
(#13316)

Now that we can properly fetch repos with non-registry overrides in RepositoryDelegatorFunction, we can also handle them correctly during discovery.

PiperOrigin-RevId: 384889161
bazel-io pushed a commit that referenced this issue Jul 15, 2021
(#13316)

Adds support for the "diamond" case of multiple_version_override: that is, multiple different modules can depend on different versions of the same module.

The "fork" case is still unchecked (so if you use `bazel_dep(B@1.0, repo_name='B1'); bazel_dep(B@2.0, repo_name='B2')` it won't throw an error). The check will be added in a next CL as I figured this one is already huge enough :)

PiperOrigin-RevId: 384894458
bazel-io pushed a commit that referenced this issue Jul 15, 2021
(#13316)

Turns out nobody needs to deal with a null version anyway, so I removed the null handling. And the unused Version#mustParse method.

PiperOrigin-RevId: 384897795
bazel-io pushed a commit that referenced this issue Jan 20, 2022
(#13316)

Use "foo.override" instead of just "foo" as the canonical repo name for the module "foo" when it has a non-registry override. Context: https://groups.google.com/a/bazel.build/g/external-deps/c/GcV1cpRd-Ls/m/o446O3vGAAAJ

PiperOrigin-RevId: 423049588
bazel-io pushed a commit that referenced this issue Jan 27, 2022
(#13316)

By recognizing that the only difference between creating a native and starlark repo rule is the code to acquire the RuleClass object, we can remove a lot of duplicate code around repo rule creation.

* BzlmodRepoRuleCreator the interface is simply replaced with the existing interface RuleFunction, which exposes the underlying RuleClass.
* Moved repo rule creation code into BzlmodRepoRuleCreator, which is now a utility class. This logic can then be shared by ModuleExtensionEvalStarlarkThreadContext and BzlmodRepoRuleFunction, instead of existing in 3 different places.

This refactor helps us eventually support rules like `native.local_repository` in module extensions.

PiperOrigin-RevId: 424627127
@Wyverald Wyverald added area-Bzlmod Bzlmod-specific PRs, issues, and feature requests P1 I'll work on this now. (Assignee required) and removed P2 We'll consider working on this in future. (Assignee optional) labels Feb 22, 2022
@Wyverald
Copy link
Member

Closing this umbrella bug in favor of the individual issues with the "area-Bzlmod" label.

@cpsauer
Copy link
Contributor

cpsauer commented May 26, 2022

@Wyverald, for toolmakers/library maintainers that should be sure to adopt before bzlmod is enabled by default:
Is there a new issue we should follow to stay abreast of when bzlmod will fully launch?

@Wyverald
Copy link
Member

Bzlmod will "fully" launch with the release of Bazel 6.0, at which point the flag --experimental_enable_bzlmod will be renamed to just --enable_bzlmod.

Bzlmod will be enabled by default in Bazel 7.0 at the earliest. That would mean deprecating WORKSPACE support, which we're still a rather long way from.

Both of these will be featured prominently in the release notes of the major releases, so hopefully it's unlikely for you to miss them :)

@cpsauer
Copy link
Contributor

cpsauer commented May 26, 2022

Definitely. Super good to know. Thank you! Great to know the forward roadmap--and thanks for all your great efforts.

fmeum pushed a commit to fmeum/bazel that referenced this issue Jul 2, 2022
(bazelbuild#13316)

Use "foo.override" instead of just "foo" as the canonical repo name for the module "foo" when it has a non-registry override. Context: https://groups.google.com/a/bazel.build/g/external-deps/c/GcV1cpRd-Ls/m/o446O3vGAAAJ

PiperOrigin-RevId: 423049588
fmeum pushed a commit to fmeum/bazel that referenced this issue Jul 2, 2022
(bazelbuild#13316)

Use "foo.override" instead of just "foo" as the canonical repo name for the module "foo" when it has a non-registry override. Context: https://groups.google.com/a/bazel.build/g/external-deps/c/GcV1cpRd-Ls/m/o446O3vGAAAJ

PiperOrigin-RevId: 423049588
fmeum pushed a commit to fmeum/bazel that referenced this issue Jul 2, 2022
(bazelbuild#13316)

Use "foo.override" instead of just "foo" as the canonical repo name for the module "foo" when it has a non-registry override. Context: https://groups.google.com/a/bazel.build/g/external-deps/c/GcV1cpRd-Ls/m/o446O3vGAAAJ

PiperOrigin-RevId: 423049588
ckolli5 pushed a commit that referenced this issue Jul 5, 2022
(#13316)

Use "foo.override" instead of just "foo" as the canonical repo name for the module "foo" when it has a non-registry override. Context: https://groups.google.com/a/bazel.build/g/external-deps/c/GcV1cpRd-Ls/m/o446O3vGAAAJ

PiperOrigin-RevId: 423049588

Co-authored-by: wyv <wyv@google.com>
luca-digrazia pushed a commit to luca-digrazia/DatasetCommitsDiffSearch that referenced this issue Sep 4, 2022
    (bazelbuild/bazel#13316)

    This is to work around a restriction in Google. The external packages we create in BzlmodRepoRuleFunction host just one rule each; so instead of storing a rule, we can just store the package and get the rule from there.

    This does require adding the rule to the package, which we weren't doing before. This involved a bit of a detour to clean up the RuleFactory#createAndAddRule method, which for some reason has had a much more sensible overload that's package private since time immemorial. This CL simply removes the overload that just gets two fields out of PackageContext.

    PiperOrigin-RevId: 387543141
luca-digrazia pushed a commit to luca-digrazia/DatasetCommitsDiffSearch that referenced this issue Sep 4, 2022
    (bazelbuild/bazel#13316)

    This makes it easier for later usage (for example, when collecting toolchains across the dep graph, BFS order makes the most sense).

    Also added some missing test assertions for the reverse lookups (which actually fixed a small bug).

    PiperOrigin-RevId: 387076182
luca-digrazia pushed a commit to luca-digrazia/DatasetCommitsDiffSearch that referenced this issue Sep 4, 2022
    (bazelbuild/bazel#13316)

    This allows us to more easily set up some tests that actually want to fetch modules, without relying on local_path_override all the time. (Useful for RegisteredExecutionPlatformsFunctionTest, for example.)

    PiperOrigin-RevId: 387064920
luca-digrazia pushed a commit to luca-digrazia/DatasetCommitsDiffSearch that referenced this issue Sep 4, 2022
    (bazelbuild/bazel#13316)

    Up until now we assume the canonical repo name of a module is simply the module name. This doesn't work with multiple-version overrides. This CL addresses this issue; the canonical repo name will be ${moduleName}.${moduleVersion} for any module.

    This does mean that we need to put in some extra work to build reverse lookup maps, and especially add extra logic in RepositoryMappingFunction to make sure that WORKSPACE references to repos generated by modules can still work (i.e. if B is a module, something that points to "@b" gets rewritten to "@B.1.0", the canonical repo name)

    PiperOrigin-RevId: 385785118
luca-digrazia pushed a commit to luca-digrazia/DatasetCommitsDiffSearch that referenced this issue Sep 4, 2022
    (bazelbuild/bazel#13316)

    Not calling module() should be the same as calling module() with no arguments. This was mistakenly disallowed before since the Module autovalue builder didn't have defaults for name and version.

    PiperOrigin-RevId: 385119757
luca-digrazia pushed a commit to luca-digrazia/DatasetCommitsDiffSearch that referenced this issue Sep 4, 2022
    (bazelbuild/bazel#13316)

    It's okay for a module depends on foo@1.2 and foo@1.3 with different repo names as long as there's a multiple_version_override for foo that allows both 1.2 and 1.3. If only 1.3 is allowed or if there's no multiple_version_override at all, then selection will succeed right now, and the module will depend on foo@1.3 twice. This CL adds a check that this doesn't happen.

    PiperOrigin-RevId: 384906574
luca-digrazia pushed a commit to luca-digrazia/DatasetCommitsDiffSearch that referenced this issue Sep 4, 2022
    (bazelbuild/bazel#13316)

    Turns out nobody needs to deal with a null version anyway, so I removed the null handling. And the unused Version#mustParse method.

    PiperOrigin-RevId: 384897795
luca-digrazia pushed a commit to luca-digrazia/DatasetCommitsDiffSearch that referenced this issue Sep 4, 2022
    (bazelbuild/bazel#13316)

    The overrides cannot be nicely encapsulated in SelectionValue because we need the overrides when fetching modules with non-registry overrides, which happens during discovery. Any consumers of overrides should get them from the ModuleFileValue of the root module directly instead.

    PiperOrigin-RevId: 384236547
luca-digrazia pushed a commit to luca-digrazia/DatasetCommitsDiffSearch that referenced this issue Sep 4, 2022
    (bazelbuild/bazel#13316)

    The compatibility_level is essentially the "major version" in semver, except that it's not embedded in the version string, but exists as a separate field. Modules with different compatibility levels participate in version resolution as if they're modules with different names, but the final dependency graph cannot contain multiple modules with the same name but different compatibility levels.

    PiperOrigin-RevId: 384185854
luca-digrazia pushed a commit to luca-digrazia/DatasetCommitsDiffSearch that referenced this issue Sep 4, 2022
    (bazelbuild/bazel#13316)

    Before we were requesting the ModuleFileValues one by one. This does not result in things being slow per se (since the individual ModuleFileFunction calls should be parallelized by Skyframe) but technically using getValues() can make SkyValue dependencies cleaner (since there is no dependency between the same "level" of SkyKeys being requested here).

    PiperOrigin-RevId: 382700697
luca-digrazia pushed a commit to luca-digrazia/DatasetCommitsDiffSearch that referenced this issue Sep 4, 2022
    (bazelbuild/bazel#13316)

    To implement the index registry outlined in the design doc, we need to make HTTP GET requests to certain URLs and read their contents immediately (instead of downloading and writing them to local disk).

    Also added a way to distinguish a 404 from other unrecoverable errors, since 404 is semantically different from, say a 401, in terms of the registry list (earlier registries take precedence and delegate to later registries only when the requested module is not found).

    PiperOrigin-RevId: 382304312
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area-Bzlmod Bzlmod-specific PRs, issues, and feature requests P1 I'll work on this now. (Assignee required) team-ExternalDeps External dependency handling, remote repositiories, WORKSPACE file. type: feature request
Projects
None yet
Development

No branches or pull requests

5 participants
@Wyverald @philwo @meteorcloudy @cpsauer and others