Nix flake

[nothingmuch]

per @chrisguida's request, cc @jurraca, opening a draft PR for a new flake.nix

disabling doCheck = true is enough to build from source using nix build ".?submodules=1", and in nix develop the python interpreter is set up with the poetry dependencies, but that's as far as I got, this does not yet work.

TODO:

• reuse upstream clightning package
• poetry2nix based python environment
  • figure out why callPackages python override from poetry2nix built environment is not doing the right thing, currently there's an ugly workaround (prepending to nativeBuildInputs)
  • ensure pyproject.toml & poetry.lock files are adequately imported, and poetry2nix is used correctly WRT multiple subdirectories in contrib (do they need separate derivations?)
  • use nix supplied python environment in make check
    • right now i just deleted PYTHONPATH bits from the Makefile to get the wrapped interpreter in the build environment, this works for some deps but not pyln-testing
• get doCheck in derivation working
• devshell with additional tools
  • nixpkgs.poetry (poetry itself is not supposed to be a dep of pyproject.toml)
  • bitcoin core (required for testing)
  • valgrind, etc - other tools used for --enable-developer
• crane based rust derivations
• nix flake checks
  • various extended developer tests could be set up with source filtering for efficient caching, this would make git rebase --exec 'nix flake check' pretty reasonable

[nothingmuch]

To use the unstable branch, FYI this currently causes an infinite cascade of derivations due to a breaking change in the nixpkgs python stuff that poetry2nix uses, you would need to pin nixpkgs to just before the merging PR mentioned in this thread:

https://discourse.nixos.org/t/psa-poetry2nix-is-currently-broken-with-nixpkgs-unstable/32281

[jurraca]

we probably shouldn't use unstable anyway since it would reevaluate packages every time, not a great UX. Pinning it to the latest stable release seems ok.
We could also figure out which packages we want bleeding edge for and cherry pick those from nixpkgs-unstable, like nix-bitcoin does.

[jurraca]

thanks for kicking this off!
I think you may be overcomplicating it by reusing the nixpkgs clightning package. Besides, the point of a flake is to be (mostly) self-contained to the repo. I would just reuse the nixpkgs expression in a local default.nix file or something.
Just thinking out loud about goals here:

• you can nix build for major systems (x86-64-linux, aarch64-linux, x86_64-darwin, aarch64-darwin)
• you can nix develop and have a set up dev environment
• you can run tests across the CLN test suite
• should handle setting up a local network in regtest, like startup-regtest.sh does.

[chrisguida]

Does anyone know of a good repo to look at for an example of what to do here? @jurraca @nothingmuch

The goal is to be able to type nix develop and have a dev env ready to run make.

Even something without python would work, I'm just trying to get my bearings right now.

Will add startup_regtest.sh once I have that working.

Then I'll add the ability to run this from downstream repos, ie plugins that also need the startup_regtest.sh environment.

[nothingmuch]

thanks for kicking this off! I think you may be overcomplicating it by reusing the nixpkgs clightning package.

yep, that's a hack, only intended (i should have clarified this) to delay making some decisions until later but should not be merged as such since it spreads out knowledge about what is appropriate to do to build the project in a very inconvenient way, introducing a dep on nixpkgs at the wrong level of abstraction (edit: that is, in addition to the appropriate dep on nixpkgs, namely relying on it for (native) build inputs)

• you can nix build for major systems (x86-64-linux, aarch64-linux, x86_64-darwin, aarch64-darwin)
• you can nix develop and have a set up dev environment
• you can run tests across the CLN test suite
• should handle setting up a local network in regtest, like startup-regtest.sh does.

i would only add to this that it's not just CLN proper but plugin development too that should be facilitated, regardless of whether the plugin lives in this repo or outside of it

[nothingmuch]

Does anyone know of a good repo to look at for an example of what to do here? @jurraca @nothingmuch

not off the top of my head

The goal is to be able to type nix develop and have a dev env ready to run make.

that's what i was going for in the first commit but it's somewhat incomplete

Even something without python would work, I'm just trying to get my bearings right now.

python is important IMO because of make check

[cdecker]

FWIW I was able to side-step the requirement for poetry2nix altogether by adding a shell hook that'd just invoke poetry install --with=dev --no-root in it. While not as nicely integrated into nix I think it could get us unstuck trying to get the first iteration working, and we can later start looking for better integrated solutions, or remove/replace dependencies that are hard to nixify.

What do you think @nothingmuch ?

[chrisguida]

@cdecker do you have a link to your flake handy? would love to take a look!

[cdecker]

I'm afraid I have it committed in some random branch that I am having trouble finding again 😊 But I will share as soon as I find it.