[RFC 0040] "Ret-cont" recursive Nix

[Ericson2314]

"Ret-cont" recursive Nix is a restricted form of recursive Nix, where one builds a derivations instead of executing builds during the builds. This avoids some platform-specific contortions relating to nested sandboxing. More importantly, it prevents imperative and overly linear direct-style build scripts; easy to write but throwing away the benefits of Nix.

Rendered

[Profpatsch]

I don’t entirely understand the proposal.

From experience, naïvely recursive systems are usually extremely constrained, so I can see that problem. You should split out the problem with the current proposal from “Motivation” into its own section and start with that.

Can you frame the idea in terms of already existing systems? Scheme’s call/cc comes to mind, but from the description it is unclear to me whether the ideas are similar. We shouldn’t implement systems like that without looking at previous work and theoretical underpinnings first.

[Mic92]

So in a nutshell instead of importing nix files from a derivation, this will import derivation files from a derivation? The idea sounds to me the equivalent to autotools/automake, where before a release the configure and Makefile.in files are generated and put in the release tarball.

[edolstra]

It would be good to add an example and/or some use cases where this approach would apply. For example, I can see it working in the case where you want to build a single derivation (e.g. building Hydra from the release.nix extracted from the Hydra tarball). But it wouldn't address the scenario we have at Tweag where we want to be able to call Bazel from Nix and then have Bazel call Nix to fetch some packages.

[Ericson2314]

@Profpatsch

From experience, naïvely recursive systems are usually extremely constrained, so I can see that problem. You should split out the problem with the current proposal from “Motivation” into its own section and start with that.

Yeah, sometime I swear have to open the PR before I can see all the places I dropped in media res to no one's benefit. I'll rewrite things in the next draft to fix that.

Can you frame the idea in terms of already existing systems? Scheme’s call/cc comes to mind, but from the description it is unclear to me whether the ideas are similar. We shouldn’t implement systems like that without looking at previous work and theoretical underpinnings first.

There's a notion of applicative vs monadic build systems. The DRV langauge today is strictly applicative:

data Drv a -- derivation which makes 'a'
type CCompiler = CCode -> Binary
iCanWriteThis :: Drv (CCode -> Binary) -> Drv CCode -> Drv (CCode -> Binary)

This proposal makes it monadic by defining join:

-- | switch to inner after building outer
join :: Drv (Drv a) -> Drv a

Maybe I should make note of this in the RFC :).

---

Fun fact, operationally, that join is also a lot like Reflex FRP's switchHold. (Ignore the final m, which could be thought of as an ambient interpreter.)

[Ericson2314]

@Mic92 Yes, you can imagine trying to run Nix to the point where there are no drv-producing drvs left, which is a lot like that process those autoconf projects do before release.

[Ericson2314]

@edolstra

It would be good to add an example and/or some use cases where this approach would apply.

Sure, I'll do that with the rephrase @Profpatsch was thinking of.

For example, I can see it working in the case where you want to build a single derivation (e.g. building Hydra from the release.nix extracted from the Hydra tarball).

Indeed it would.

But it wouldn't address the scenario we have at Tweag where we want to be able to call Bazel from Nix and then have Bazel call Nix to fetch some packages.

Well, with this + intentional store, you all would hopefully have no need for Bazel anymore :). But upstream projects will still use it and so it's still worth answering.

With this RFC, there's no reason for Bazel to do any building. Instead, we can have a bazel2nix that will convert the (1st order) Bazel build graph to Nix derivations, which Nix builds as usual. We can (and should) already right this, but currently this tool could only be used "externally", and not within a derivation. With this RFC, using it "internally" is just like instantiating Hydra's release.nix, but with more hoops to jump through first (bazel2nix; nix-instantiate).

(A brief aside: A naive bazel2nix would get GCC for whatever from the PATH, like a configure script, which means we'd have to build GCC first before any bazel2nix step, which reduces parallelism. Much better is to feed bazel2nix GCC's derivation (as a Nix expression or DRV file), so it can just inject that as a dependency in the derivations produced from individual Bazel build nodes that use GCC. Then we get maximal parallelism at the finest granularity, regardless of where the package boundary is!)

[globin]

Per meeting of the @NixOS/rfc-steering-committee this RFC is open for shepherding nominations.

[nixos-discourse]

This pull request has been mentioned on Nix community. There might be relevant details there:

https://discourse.nixos.org/t/new-rfcs-40-ret-cont-recursive-nix-open-for-shepherd-nominations/2075/1

[shlevy]

I'd like to be considered for the shepherd team.

[Ericson2314]

@volth Your builtins.exec sounds like import from derivation as it exists today. Is that the case, or am I missing something?

[Ericson2314]

That's fine with me. Should we might more about this or ought we to have a prototype implementation / do we even need RFCs for experimental features or only when they become stable.

I suspect the answer is "no", so a lack of meeting on #56 is actually holding me back more than a lack of meeting in this.

[Ericson2314]

Nix finally has some competition with https://www.usenix.org/conference/atc19/presentation/fouladi .

The deal with dynamism exactly as this RFC proposes. In their words "a thunk can write another thunk as its output"; just substitute "thunk" for "derivation", and you have this RFC.

[Ericson2314]

Thanks @volth

[Mic92]

@fpletz did you to create a new meeting for this one?

[Ericson2314]

@Mic92 While I am happy to talk about this again, I was just gong to implement it after I work on CA derivations. The basic idea for both is clear to me, but in doing the programming little intricacies arise that I would never need to anticipate.

I suppose the main thing I'd like to discuss would be a new process RFC ---- getting PRs merged for Nix has been incredibly slow because it is unclear who can review and merge besides @edolstra, and Eelco is busy with many other things. We need a "chain of command" for Nix, deciding who is responsible for reviewing or merging what.

[spacekookie]

@fpletz @edolstra @shlevy @copumpkin Considering there hasn't been any activity on this RFC for a while, maybe you should consider appointing a new shepherd leader for this RFC to get things going again

[Ericson2314]

In addition to what I said above about doing this after CA derivations, there are a few things I want to revise here and an intermediate feature/RFC we could do. Given all that, I am going to close this for now to be reopened when the time is ripe.

[roberth]

I like that ret-cont avoids the problem with fixed-output derivations in recursive Nix (as implemented). See NixOS/nix#3205 (comment)

[roberth]

This proposal makes it monadic by defining join:

-- | switch to inner after building outer
join :: Drv (Drv a) -> Drv a

I believe it is more like Category and Arrow. Let me explain.

There's no Nixlang Functor for Drv because its type parameter a has a libstore kind, not a libexpr kind. The truly Monadic operation on derivations is import from derivation, which works for any Nixlang expression expr. Its type in the language is import :: forall a :: Expr. Drv a -> a, which, combined with Nix's implicit goal of producing derivations in the end gives the interpreter the ability to Drv (Drv a) -> Drv a. (import shouts Comonad, but let's skip that tangent*)

Ret-cont recursive Nix does not allow IFD. If it were implemented in the language, like RFC 92, it would look like outputOf :: forall a :: StorePath. Drv2 (Drv2 a) -> Drv2 a, where Drv2 is an awkward type that is more about what happens in the sandbox than what happens in the language. What happens in the sandbox is not actually accessible to the language proper, so it is better modeled as an opaque type with some operations that define its behavior, so we get derivation : Build a b where you can think of build as something akin to type Build a b = List a -> Drv b. Now we're really in Arrow territory. We can even define arr by running the evaluator in the sandbox, no wonder there's a separate class hierarchy, because Build just joins the list of cases where arr is not desirable. What we really want to see is (.) of Category, which is Build a b -> Build b c -> Build a c. Applying our hand-wavy type alias definition, we get (List a -> Drv b) -> (List b -> Drv c) -> List a -> Drv c, which after a function application becomes Drv b -> (List b -> Drv c) -> Drv c, so we have to build path b (build : forall p :: Path. Drv p -> p) in order to get at Drv c. This is what ret-cont does, all at the derivation level, not the Nixlang level.

The power spectrum works out the same way. By caller power we have Functor (f b) < Applicative (f b) < Arrow f < Monad (f b) and similarly traditional Nix < ret-cont recursive Nix < import from derivation. Note that callee features are inversely related to caller power. In Haskell, in-language static analysis ends at Arrow (and/or Selective); in Nix build-free evaluation ends at ret-cont recursive Nix*.

*: I'm not including full recursive Nix, I should probably look into Comonad some more and all my steps are too hand-wavy. However, my comment ends here.

[Ericson2314]

@roberth I think a could be the "type" of the derivation outputs. At least with subtyping Drv <= Bytes, Drv (Drv Bytes) -> Drv Bytes makes some sense. But yes it doesn't have the same super-linearizing feel as Monad as used in practice in Haskell, and I do think arrows are underrated if only we'd fix them up like as in https://arxiv.org/abs/1007.2885

[roberth]

I think a could be the "type" of the derivation outputs. At least with subtyping Drv <= Bytes, Drv (Drv Bytes) -> Drv Bytes makes some sense.

That's how I read it and it makes total sense at the store level. 👍

Just reading back my comment makes me dizzy though, with three 'languages' to try to keep in my head: derivations, expressions and some sort of meta-language thing consisting of types that we never really defined unambiguously. Pseudo-types as in pseudo-code? Anyway, probably best to just focus on RFC 92.

[Ericson2314]

Yes, thinking about all 3 always got me dizzy too!