[RFC 0148] Pipe operator

[piegamesde]

Rendered

Discussion notice: please try to attach all discussions to a thread by using the code review feature. If your comment doesn't refer a specific line to attach to, use the header line instead.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/pre-rfc-pipe-operator/28387/3

[roberth]

Nix is already considered to be overly complex by many. Of course we know that Nix solves hard problems and can't do so without exposing intrinsic complexity, but this needed complexity is already more than most people expect to have to deal with, coming from the broken but "easy" traditional methods.

This makes the addition of accidental complexity to Nix disproportionately harmful. It makes newcomers more likely to reject Nix, depriving themselves of a real solution to their packaging and deployment problems, while depriving us from valuable contributions.

I am opposed to the pipe function, and skeptical of a function application operator; especially one that reverses the evaluation order compared to normal function application.

[roberth]

pipe is not a good function because the order is not obvious. It's already available practically everywhere as lib.pipe. It's also hardly ever used, so doing this for performance strips it of the last argument for making it a builtin.

Builtins need to satisfy more stringent requirements, they take resources from the Nix team, and they can never be changed or removed.

I would recommend to always use a library, which can change things arbitrarily, thanks to being locked or pinned.

[KFearsoff]

It's also hardly ever used

I think this is due to documentation issues rather than due to this function itself being bad. While it has a lot of potential of misuse, it can actually make code more readable if you put in the effort for your code to be readable from top to bottom.

I would recommend to always use a library, which can change things arbitrarily, thanks to being locked or pinned

This... makes sense, but then I'd make an argument that parts of nixpkgs.lib that manipulate generic data should exist outside of builtins and outside of nixpkgs - in a separate repository. It is not weird: we already have library flakes like flake-utils or flake-parts.

This discussion is outside of the scrope of this RFC, though; I merely disagree that lib.pipe not seeing much usage is due to technical reasons.

[rhendric]

Shepherds currently believe this conversation is resolved, outcome: inclined to leave pipe in lib.

[Profpatsch]

pipe is not a good function because the order is not obvious.

@roberth for what it’s worth, the commit where I introduced it has a rationale for the order in its message.

[roberth]

It's not a problem with your contribution, but with the constraints of the Nix syntax.

lib.pipe is a necessary step, and its adoption shows that it is useful to many, so thank you for creating it!

[roberth]

I think you could call this curried style.
While it is point free in the technical sense of not having a variable name to carry the data flow, point-free is generally only used in a context where higher order functions are used for more arbitrary data flows. For many, it also carries the connotation of the synonymously used pointless style.

[piegamesde]

I've never heard that term and the resources I've read so far all talked about "point-free" programming style. Google does not seem to know much either, so some links would be appreciated.

[roberth]

The flipped pipe function is just function composition on a list, eta expanded. Eta expansion / reduction do not change the meaning, so these two functions would be indistinguishable.

That means that we can divide the problem and drop some unnecessary terminology.

1. pipe = flip composeFunctions (and composeFunctions = flip pipe by virtue of a flip law)
2. composeFunctions is isomorphic to pipe
3. composeFunctions returns a function, which makes it easier to use where a function is expected: the application of higher order functions such as map.

A lot of that is just overly formal thought. Instead we can simplify this section to:

flip pipe is equivalent to function composition applied to a list. By using a function that composes a list of functions instead of pipe, we get back a function, which can be readily used in higher order functions such as map.

[Profpatsch]

point-free is also point-less, in a semantic sense 😛

[rhendric]

Shepherds currently believe this conversation can be resolved by accepting or rejecting the suggestion below.

[piegamesde]

I am very well aware of the language's complexity. I am also aware of the fact that some of its users are very technical and into functional programming languages, and for many others this is a first contact with these concepts.

This is why I suggest picking only one of these operators as a compromise. I don't have any overly strong preferences here, but I make a point against function composition instead of argument piping, because the former leads to the so-called "point-free" programming style which tends to be more confusing for new users. I picked |> because of its similarity to the existing lib.pipe and the |> operator in Nickel, but if you are concerned about reversing control flow, then <| is a viable alternative too.

I strongly oppose calling this proposal "accidental" complexity. I spent a lot of time thinking about the options and trying to balance expressiveness and complexity, finding a compromise that makes solving real Nix problems easier. The Nix language is full of weird quirks users eventually have to face: we even have a wiki article collecting such instances, and it is far from exhaustive. Feel free to call these accidental complexity. But nothing about this proposal is accidental.

[roberth]

To be clear, I don't use accidental to describe this RFC. I only used it in accidental complexity, which I've used as a synonym for extrinsic complexity. I can see that you've put a lot of thought into it, and I respect that, but that does not necessarily make the change a net positive.

expressiveness

A language with more syntax may be easier to write, but is not more expressive unless the syntax comes with semantics that were not already covered by existing features and combinations of them.

makes solving real Nix problems easier.

What is a real Nix problem? At least the lack of a syntax won't end up on the quirks page.

Are the quirks a problem? Probably. I'd be happy to see a proposal that simplifies the language or makes the syntax easier to learn, but those are hard problems with a lot of inertia, and up-front costs that aren't "repaid" for years into the future.
Pragmatically, I think the real Nix problems aren't in the syntax. We might reach a stage where we could simplify the syntax and possibly redo it to make it more accessible to people who use more commonplace languages. That's not in the coming five years though, unless we radically expand the Nix team, which is rather resource constrained today.

[suhr]

Nix is already considered to be overly complex by many.

And more people consider Nix awkward to read and write rather than being complex. Mostly because the standard library is poor and not discoverable.

|> would definitely make Nix less awkward to use.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/pre-rfc-pipe-operator/28387/10

[piegamesde]

A language with more syntax may be easier to write, but is not more expressive unless the syntax comes with semantics that were not already covered by existing features and combinations of them.

I disagree with your interpretation of the word "expressiveness". Because you can (almost) always already solve all existing problems with existing syntax in (almost) any programming language. That's the point of being Turing complete. Therefore the question then becomes, how well can a problem be expressed in the language. And I think this holds true even for changes that are purely syntactical, like this one.

When adding new syntax to the language, one goal may indeed be to make it easier to write. Another one, and IMO much more important, is does it make the language easier to read. Depending on the feature and the language those may coincide, or one may have the other as a side benefit, or they may contradict each other. I recently learned Haskell, so I am very well aware of the trap of having a lot of powerful operators that allow it to write programs concisely and without parentheses, but which comes at the cost of readability.

What is a real Nix problem?

Problems that one might face when writing Nix code, either for some flake/shell/system configuration or when contributing to Nixpkgs. What I mean with this, is that I don't want to solve problems that for example mostly occur when trying to solve AdventOfCode in Nix.

[KFearsoff]

While the lib.pipe function is pretty cool (thanks for introducing it to me!), I think going through with this RFC would take gigantic effort, and the positive results (if any) will be seen only after a few years. And I'm thinking those positive results aren't worth it, because they won't make Nixlang significantly easier to read for beginners, and they don't solve any technical issues with Nixpkgs or Nix.

[KFearsoff]

It's also hardly ever used

I think this is due to documentation issues rather than due to this function itself being bad. While it has a lot of potential of misuse, it can actually make code more readable if you put in the effort for your code to be readable from top to bottom.

I would recommend to always use a library, which can change things arbitrarily, thanks to being locked or pinned

This... makes sense, but then I'd make an argument that parts of nixpkgs.lib that manipulate generic data should exist outside of builtins and outside of nixpkgs - in a separate repository. It is not weird: we already have library flakes like flake-utils or flake-parts.

This discussion is outside of the scrope of this RFC, though; I merely disagree that lib.pipe not seeing much usage is due to technical reasons.

[KFearsoff]

I think this is a great argument against going through with this RFC.

[piegamesde]

This would be an argument for never making any changes to improve the language at all, ever. Unless you see some shiny alternative language to which we'll realistically all migrate to within the next five years, any improvements to the Nix language are still expected to be beneficial.

I've actually thought about proposing this change for quite some time now, and the main thing that held me back was the idea is that Nix is a "lost cause" and that I should rather spending my energy towards some successor. But eventually, I realized that realistically, we'll be stuck with Nix for a long time to come, so better invest the energy to make it a bit more comfortable …

Also note that outside of Nixpkgs, the feature will be available almost immediately. There is a lot of Nix code outside of Nixpkgs, all these flakes, shells, and users' system configurations. Projects like home-manager, devenv, nur, etc. are all free to migrate to newer Nix versions at their own pace. We shouldn't ignore these.

[piegamesde]

I think going through with this RFC would take gigantic effort

Honestly, the most effort in this RFC is convincing people, and maybe also the coordination across many projects. But from the technical side of things, this is actually pretty easy to implement.

and the positive results (if any) will be seen only after a few years.

This only applies to Nixpkgs, see #148 (comment)

And I'm thinking those positive results aren't worth it, because they won't make Nixlang significantly easier to read for beginners, and they don't solve any technical issues with Nixpkgs or Nix.

They also won't make it significantly harder for beginners to read Nix code IMO, while bringing quality of life improvements to everyday Nix development.

I don't understand why only "technical" problems should be worth solving.

[edolstra]

This RFC is now open for shepherd nominations!

[piegamesde]

@AndersonTorres because I don't know every programming language and just didn't come across OCaml during my research. Furthermore, the related work section is intended to roughly cover the design space, not to be an exhaustive list of every programming language.

[piegamesde]

Updated the text according to some of the initial feedback. The more I think of it, the more I'm leaning towards also having <| in the language. I'll try to conduct a larger Nixpkgs survey soon to get some more data on that question. (Function concatenation operators are still out IMO)

[AndersonTorres]

I was thinking on it, precisely, two pipe operators! It makes the language more orthogonal: if we have arg |> function, it looks reasonable to have function <| arg as its reverse operator.

The problems are:

• if we are to elevate pipe to builtins and pipe == |>, where is the builtin for reverse pipe? The apply previously suggested looks good to me;

• how the pipes should interact when in a same statement (a |> b <| c, a <| b |> c)?
  I suggest to APLize: same precedence, right-to-left associativity (a |> (b <| c), a <| (b |> c))

• a problem for the parser team: how should the errors be reported?

[piegamesde]

I nominate @maralorn, who has a lot of experience with functional programming languages and programming language design.

[piegamesde]

@roberth, what about you, would you like to be a shepherd? I'd rather like to have critical voices on board from the beginning …

[maralorn]

I nominate @maralorn, who has a lot of experience with functional programming languages and programming language design.

I am pretty certain there are a lot of people with more experience on this in the wider community. That doesn’t prevent me from having opinions on this particular bikeshed, though. 🤣

My availability depends on the timeframe here. I will be very busy in the next few weeks, after that I’d be on board.

[piegamesde]

I've procrastinated writing this RFC for at least half a year, so waiting two more months won't be the end of the world

[roberth]

would you like to be a shepherd?

I can do that

[piegamesde]

Applied all suggestions; about the evaluation order point I'd like to delay that discussion until we have some real-world experience with implementations.

[piegamesde]

Nix implementation: NixOS/nix#11131
Lix implementation: https://gerrit.lix.systems/c/lix/+/1654/7
Nixfmt implementation: https://github.com/NixOS/nixfmt/pull/227/files

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-2-24-released/49986/6

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/rfcsc-meeting-2024-08-05/50170/1

[piegamesde]

Lix implementation is now merged into main as well. There is one small difference to the semantics of the implementation in Nix: Mixing both operators is not forbidden, instead <| binds at a lower priority. In the RFC text I wrote that this would be a bad idea, but I wanted to empirically verify that claim.

[rhendric]

Plan for moving forward/stabilizing feature in Nix

Now that the experimental feature has been released, it may be time to start talking about a plan for evaluating it.

I see a number of possibilities:

• We do nothing and hope it happens organically that enough enthusiasts try the feature out to give it a good trial. In all likelihood, people will only comment if they dislike something about the feature, which means we could receive little or no feedback and be left wondering if enough people have looked at it. This requires least effort but is likely to take the longest.
• We promote the feature in official Nix channels and set up some sort of polling for both positive and negative feedback, with an expectation of how long the poll will be running before moving this RFC to FCP. This requires a bit of up-front work and may be biased towards people who pay attention to official Nix channels, but has the advantage of resolving faster than the first possibility.
• We devise an actual experiment to measure people's ability to learn the new operators. This might take the form of a survey asking people to rate the readability of various Nix expressions, some with the new operators and some without, and testing if there is a statistically significant difference in the ratings. We could send the survey to a representative sample of community members instead of (or in addition to) the public polling from the previous possibility, to try to control for bias. I'm not aware of the Nix community undertaking this level of rigor in the past but it would help us bring a stronger case for assuaging people's concerns about the mental cost of adding new operators. It would be a substantial investment of time from a few people (I'm happy to be one of those people but it probably shouldn't be me alone).

Anyone else have thoughts?

[inclyc]

We do nothing and hope it happens organically that enough enthusiasts try the feature out to give it a good trial. In all likelihood, people will only comment if they dislike something about the feature, which means we could receive little or no feedback and be left wondering if enough people have looked at it. This requires least effort but is likely to take the longest.

I think it would be beneficial to give the new feature more visibility and promotion, rather than leaving it as an experimental feature indefinitely.
Collecting feedback through some form of polling or voting mechanism could provide valuable insights.
Additionally, introducing new features in a programming language is a significant event, especially for a community-driven project like Nix. It's likely that experienced PL experts would be needed to assess its impact and stability.

[roberth]

Latest observations:

• Friction due to lack of pipelines creates a perverse incentive to make libraries bloated, incomplete and inconsistent (more context)
• <| leads to visually unintuitive layouts when formatted according to the Nix formatting principles behind the merged RFC (see review for example and explanation)

Based on these observations and considering that a single operator could bring most of the value, I'm leaning towards accepting |> exclusively. The redundancy and "ugliness" of <| might combine to be a net negative when we have |>.

If we take this direction, perhaps |> could be simplified to |, as it matches the intuition based on shell pipes and jq pipes, which may be more familiar in part of the audience, and these tools are more likely to be used together in scripts.
It conflicts somewhat with Haskell's | for conditionals in patterns, but that should be ok because Nix is already quite a different flavor anyway. Low level languages use it for bitwise or, but Nix is not that either.

[illustris]

Here are some of my thoughts after using this operator for a while:

Partially applied pipes

Consider

nix-repl> inc = x: x+1
nix-repl> x = i: i |> inc |> toString
nix-repl> x 3
"4"

It would be nice to drop the i: i entirely, and treat x = |> inc |> toString as equivalent.

Lambdas inside pipe

nix-repl> double = x: x*2

# current syntax
nix-repl> 3 |> (x: 1 + x) |> double |> toString
"8"

# proposed alternative
nix-repl> 3 |> x: 1 + x |> double |> toString

# current syntax
nix-repl> 3 |> (x: 1 + (x |> double)) |> toString
"7"

# proposed alternative
nix-repl> 3 |> x: 1 + (x |> double) |> toString

The brackets needed around the function feel redundant.

[rhendric]

(Attempting to corral feedback into a thread.)

@illustris [#148 (comment)]

nix-repl> double = x: x*2

# current syntax
nix-repl> 3 |> (x: 1 + x) |> double |> toString
"8"

# proposed alternative
nix-repl> 3 |> x: 1 + x |> double |> toString

# current syntax
nix-repl> 3 |> (x: 1 + (x |> double)) |> toString
"7"

# proposed alternative
nix-repl> 3 |> x: 1 + (x |> double) |> toString

Both of those proposals seem, to me, to be ‘more likely’ to mean something else:

3 |> (x: (1 + x |> double |> toString))
3 |> (x: (1 + (x |> double) |> toString))

In other words, if |> binds more loosely than the lambda-forming :, it would be the first thing in the language to do so and thus would be very surprising to me. So far in the language, a lambda always extends as far rightward as it can.

[piegamesde]

Partially applied pipes

To me this smells like function composition with extra steps. Function composition is discussed in the RFC as well, and while I am not completely opposed to it, my fear is that it reduces code readability given a lack of type annotations. In x = |> inc |> toString it is a lot less intuitively clear that x is a function.

This is one of these questions where I'd like to see more usage to find out whether this is a thing that comes up sufficiently often in practice that it is worth dealing with.

(Side note: I think we would get this feature for free if Nix had operator sections)

Lambdas inside pipe

While I agree that not requiring parentheses here would be nice, especially given that one goal of the pipe operator is to reduce parentheses, I don't see any way this would be realistically implementable without throwing the entire language under the bus. Making the binding strength of abstractions context dependent just doesn't sound great

[Profpatsch]

Making the binding strength of abstractions context dependent just doesn't sound great

agreed, it’s a recipe for disaster

[illustris]

Lambdas inside pipe

Making the binding strength of abstractions context dependent

Right. I didn't think that through. This is a bad idea.

Partially applied pipes

In x = |> inc |> toString it is a lot less intuitively clear that x is a function

In my opinion it is slightly more clear than, for example,

concatLines = concatStringsSep "\n"
getBin = getOutput "bin"

or any other partially applied function. The open |> (or |) at the start suggests to me that this is a function.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/rfcsc-meeting-2024-08-19/50831/1

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/rfcsc-meeting-2024-09-02/51514/1

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/rfcsc-meeting-2024-09-16/52224/1

[jian-lin]

Is it allowed to backport the pipe operator to the Nixpkgs minimum nix version?

[MattSturgeon]

Do major feature additions like this usually get a backport?

Even if it was backported, wouldn't it defeat the purpose of having a minimal version that's likely to be installed by most users?

A backport would still be a new release, even if it was based on an old major version number.

[alyssais]

Sometimes. Support for zstd cache compression was backported to 2.3, so it's within the realms of possibility.

The reason we have a minimum version of 2.3 not so much to support people who haven't updated Nix in several years, it's because later versions of Nix have unresolved regressions in things that some people depend on.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/rfcsc-meeting-2024-09-30/53690/1