Crane is no longer able to cross-compile with an overridden toolchain

[szlend]

Describe the bug

After #634 crane is no longer able to cross-compile with an overridden toolchain. Instead it will always use cargo/rustc from nixpkgs.

Reproduction

1. Check out https://github.com/szlend/test-crane-gitrev-cross (based on cross-rust-overlay)
2. Run nix-tree --derivation .#default
3. Only rustc-1.73.0 is in the build closure (expected rustc-1.79.0)

Removing crossSystem fixes the issue and rustc-1.79.0 is used again.

[szlend]

I think makeScopeWithSplicing' in #634 is likely used incorrectly. Iirc self<host><target> should refer to a scope where cargo and rustc are actually variants of the toolchain we passed in and not from pkgs<host><target>.

[szlend]

I managed to work around this issue like so:

mkToolchain = p:
  let
    toolchain = p.rust-bin.stable.latest.default;
  in
  {
    cargo = toolchain;
    clippy = toolchain;
    rustc = toolchain;
    rustfmt = toolchain;
  };

otherSplices = {
  selfBuildBuild = mkToolchain pkgs.pkgsBuildBuild;
  selfBuildHost = mkToolchain pkgs.pkgsBuildHost;
  selfBuildTarget = mkToolchain pkgs.pkgsBuildTarget;
  selfHostHost = mkToolchain pkgs.pkgsHostHost;
  selfHostTarget = mkToolchain pkgs.pkgsHostTarget;
  selfTargetTarget = { }; # usually empty
};

craneLib = import "${crane}/lib" {
  inherit (pkgs) lib makeScopeWithSplicing';
  inherit otherSplices;
};

I'm not sure what the best way to expose this as an API. A few ideas:

1. Let users pass a custom otherSplices with overrideToolchain.
2. Make overrideToolchain accept a function like: craneLib.overrideToolchain (p: p.rust-bin.stable.latest.default) where p is a variant of pkgs<host><target>.

Or both? 1 is a bit low level, and 2 assumes that you overlay the toolchain over nixpkgs (which you might not want to).

[ipetkov]

Hi @szlend thanks for the report! I still don't fully grok how makeScopeWithSplicing' works end-to-end so it's entirely possible we're missing something (maybe we should be injecting self in otherSplices somehow? All the current usages in nixpkgs don't entirely make sense to me yet)

Anyways I'll try to take a look at this sometime next week, but if anyone figures things out feel free to submit a PR!

[szlend]

Here's a quick explanation of makeScopeWithSplicing' because I don't want anyone else to suffer reverse engineering this undocumented stuff from nixpkgs :D

makeScopeWithSplicing' accepts f and otherSplices. The f argument is a function that builds your package set:

pkgset = makeScopeWithSplicing' {
  f = self: {
    foo = self.callPackage ./foo.nix {};
    # ...
  };
};

# only the attributes you defined in `f` are accessible
foo = pkgset.foo.outPath; # OK
jq = pkgset.jq.outPath; # Missing attribute

# however in `pkgset.callPackage` you now have `pkgs // pkgset`:
fooAndJqPath = pkgset.callPackage ({ foo, jq }: "${foo.outPath} and ${jq.outPath}") {};

So far this is similar to makeScope you're already familiar with. Now the otherSplices attribute is used to generate __spliced.<host><target> attributes for each package you pull out with callPackage:

pkgset = makeScopeWithSplicing' {
  otherSplices = {
    # We define all the different variations of our package set here
    # `mkPackagesFor` is a placeholder for a function that builds your package set with a different `pkgs` scope
    # This could just be a recursive call into another `makeScopeWithSplicing'` with `pkgs.pkgsBuildHost` instead of `pkgs`
    selfBuildHost = mkPackagesFor pkgs.pkgsBuildHost;
    # ...
  };
};

# We can see our package now has a `__spliced` attribute when captured from `callPackage`.
# When you put `foo` into `buildInputs` / `nativeBuildInputs` / etc, the correct variant
# from `__spliced` is picked for you. This attribute only exists when `crossSystem` is set
splicedForBuildHost = pkgset.callPackage ({ foo }: foo.__spliced.buildHost) {}; 

Usually in nixpkgs they avoid building otherSplices and just use generateSplicesForMkScope <my_scope> to point to pkgs.pkgs<host><target>.<my_scope>. This only works because these packages are part of nixpkgs itself, so the pkgs<host><target> variants already exist.

If you use rust-overlay (overlayed on top of nixpkgs), you could technically do something like this:

makeScopeWithSplicing' {
  f = _self: pkgs.rust-bin.stable.latest;
  otherSplices = generateSplicesForMkScope "rust-bin.stable.latest"
}

This is basically the equivalent of pkgs // rust-bin.stable.latest within callPackage scope, but with proper splicing.

[szlend]

Happy to contribute something myself as well, but we'd need to discuss how the API for this should look like.

Imo we could support this in two ways:

1. crane.mkLib myPkgset where the user has full control over what cargo and rustc mean in the context of crane by manually crafting the package set with makeScopeWithSplicing'.
2. overrideToolchain which can now accept a function like so: crane.overrideToolchain (pkgs: pkgs.rust-bin.stable.latest). We can then generate makeScopeWithSplicing' for them and re-initialize crane, similar to the mkLib example above.

One caveat is that we can't use overrideScope for this, because overrideScope doesn't override otherSplices.

[szlend]

One caveat is that we can't use overrideScope for this, because overrideScope doesn't override otherSplices.

This also means that overrideToolchain can no longer be chained. Crane needs to be fully reinitialized through mkLib.

I hacked a quick PoC in #650, but I'm not sure what we should do about overrideToolchain. Some tests are also failing, but I think that's expected with overrideToolchain.

[szlend]

An alternative (backwards compatible) attempt is in #652.