Python bindings: add `Value` type #17

roberth · 2023-03-11T16:54:42Z

@infinisil I hope this helps. A custom type doesn't seem so bad!

And fix a test failure in the sandbox due to /home existing on Darwin but not being accessible in the sandbox since it's a symlink to /System/Volumes/Data/home, see https://github.com/NixOS/nix/actions/runs/4205378453/jobs/7297384658#step:6:2127: C++ exception with description "error: getting status of /home/schnitzel/darmstadt/pommes: Operation not permitted" thrown in the test body. On Linux this wasn't a problem because there /home doesn't exist in the sandbox

Copies part of the tree from Pythonix at Mic92/pythonix@fbc8490 into the ./python subdirectory. These Python bindings don't build yet with the current Nix version, which is why they're not built yet in this commit. Mic92 confirmed that he's okay with the license being changed to the Nix one: NixOS#7735 (comment) Co-Authored-By: Jörg Thalheim <joerg@thalheim.io>

The python bindings initialised from Pythonix haven't been updated in some Nix versions and would not compile anymore. This commit still doesn't include a working python bindings build, but these are the Nix-update-caused changes that will be necessary for that

Makes the python bindings build, both incrementally and for CI. Documentation is not yet included

Using the configuration file from NixOS#6721 for less conflicts

Will be useful for documentation

And don't use the deprecated python3 meson module

Not sure why I added it originally

Mirroring what the main Nix build does

And test that null bytes currently end strings

It didn't cause a problem because it converted True to 1, which still allowed 1 == True to succeed in Python (using assertEqual)

That in fact doesn't work for importing

Using a symlink hacky and breaks in some ways

So that we don't have to rebuild Nix every time the python bindings change

- BASH_SOURCE[0] doesn't exist for evaluations under `eval` - NIX_STORE may not be set

The former is deprecated

Previously the python bindings derivation would use Nix as its source, call configure on that, just to extract the two files tests/init.sh and (the generated one) tests/common/vars-and-functions.sh The new approach is to instead have a separate derivation extracting these two files and having a small wrapper around them

Tests that the Nix bindings can be used as a Python dependency in Nix builds

Docs still needed

Creating a new one for each call is wasteful and doesn't scale to a composable interface, as the same EvalState must be used for operations on the same Values. Ideally we'd store EvalState in the python module, or make it a first class object.

yorickvP · 2023-03-14T12:10:32Z

Did you see https://gist.github.com/yorickvP/08afdaae98cad3516a9629e7e8df343d ? I think we can almost implement it with this and a python wrapper. Just need a way to

get type of a value (including unevaluated types?)
toPythonLazy
get string context

roberth · 2023-03-14T14:27:47Z

I did and I was inspired by it! Would you like to resume this?
I felt that I had to make time to make sure this direction of development isn't avoided for the wrong reasons, but I can't spend more time on this for the foreseeable future.

get type of a value

Would be nice but not essential.

get type of a value including unevaluated types?

We need to perform an undecidable amount of evaluation to return that information, and that'd be a feature to implement in the evaluator itself. I don't think it's worth the effort. Instead, you could formalize the interface of the expression with a schema expressed in python. It makes the runtime type checking at the boundary easy and consistent. Haskell hercules-ci-cnix-expr already implements this idea.

toPythonLazy

Also nice to have. A combination of v.${k} and toPythonStrict goes a long way!

get string context

Can be achieved by wrapping builtins.getContext. Can be re-done later if the error messages aren't great.

I think this PR is already a valid MVP.

fricklerhandwerk · 2023-04-16T11:07:27Z

python/doc/index.md

+
+## Build integration
+
+In the future these Python bindings will be available from Nixpkgs as `python3Packages.nix`.


Suggested change

In the future these Python bindings will be available from Nixpkgs as `python3Packages.nix`.

Let's not talk about the future here.

fricklerhandwerk · 2023-04-16T11:10:03Z

python/README.md

+
+See [index.md](./doc/index.md), which is also rendered in the HTML manual.
+
+To hack on these bindings, see [hacking.md](./doc/hacking.md), also rendered in the HTML manual.


This seems empty though. Let's not reference things that don't exist, or create those things first.

fricklerhandwerk · 2023-04-16T11:13:13Z

python/README.md

+
+## Documentation
+
+See [index.md](./doc/index.md), which is also rendered in the HTML manual.


I may be missing something but I don't see where it's rendered in the manual.

infinisil and others added 30 commits February 24, 2023 13:24

python: Integrate incremental and CI build

e3020bf

Makes the python bindings build, both incrementally and for CI. Documentation is not yet included

python: Fix for IFD

7751140

python: Format using clang-format

afa4916

Using the configuration file from NixOS#6721 for less conflicts

python: Rename pythonnix namespace to nix::python

26cb4f0

python: Add exampleEnv to try out the bindings

36a74d1

Will be useful for documentation

python: Install the bindings in hopefully the correct location

e4df267

And don't use the deprecated python3 meson module

WIP: Start writing documentation and example

402a16d

Remove .cache from .gitignore

47bd4a4

Not sure why I added it originally

Always include config.h

3fab794

Mirroring what the main Nix build does

Release the GIL for Nix evaluation

2fce5fd

Don't release GIL lock because it would segfault

e11273f

Allow getting exact throw error messages

a10ae63

Readd .cache to .gitignore

e460074

Add ThrownNixError subclass

8a9d1fe

python: Add clang-tools to dev env

6be7198

python: Properly release and reacquire GIL

813cbc4

python: Detect stack overflow from recursive data structure

fe03f3d

Insert some TODO's

7e46ddb

python: Handle null's in expressions correctly

b69d933

And test that null bytes currently end strings

python: Fix boolean to nix conversion

af8d911

It didn't cause a problem because it converted True to 1, which still allowed 1 == True to succeed in Python (using assertEqual)

python: Add test for Null conversion

bb06d43

python: Use assertRaises for tests

cd1442a

python: We don't need to install the bindings into a subdirectly

1e76e28

That in fact doesn't work for importing

Remove python from manual build again

724d1d8

Using a symlink hacky and breaks in some ways

Very simple Nix source filtering

56a0ab0

So that we don't have to rebuild Nix every time the python bindings change

Fix the buildPythonApplication example

75696bd

Fix some unset variable problems with vars-and-functions.sh

b9dc4e4

- BASH_SOURCE[0] doesn't exist for evaluations under `eval` - NIX_STORE may not be set

infinisil and others added 9 commits March 10, 2023 00:09

assertEquals -> assertEqual

090ee61

The former is deprecated

Don't depend on all files for the python bindings

1df5ea8

Make buildPythonApplication test work

754e342

Tests that the Nix bindings can be used as a Python dependency in Nix builds

Fix dev shell

29d3a70

Add debugging capability to dev shell

e412e35

Docs still needed

Convert from eval to callExprString API

c87a1b4

python: Use global EvalState for now

7cc4ac2

Creating a new one for each call is wasteful and doesn't scale to a composable interface, as the same EvalState must be used for operations on the same Values. Ideally we'd store EvalState in the python module, or make it a first class object.

python: Add nix.evalExprString() and value.toPythonStrict()

6e7eff4

roberth mentioned this pull request Mar 11, 2023

Experimental python bindings NixOS/nix#7735

Draft

14 tasks

yorickvP force-pushed the python-bindings branch from c87a1b4 to 51a003b Compare April 14, 2023 11:28

yorickvP requested a review from fricklerhandwerk as a code owner April 14, 2023 11:28

yorickvP force-pushed the python-bindings branch 2 times, most recently from a5b7b85 to 0ca1c82 Compare April 14, 2023 12:30

fricklerhandwerk reviewed Apr 16, 2023

View reviewed changes

yorickvP force-pushed the python-bindings branch from 879901f to 57e71a3 Compare April 19, 2023 12:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python bindings: add `Value` type #17

Python bindings: add `Value` type #17

roberth commented Mar 11, 2023

yorickvP commented Mar 14, 2023

roberth commented Mar 14, 2023

fricklerhandwerk Apr 16, 2023

fricklerhandwerk Apr 16, 2023

fricklerhandwerk Apr 16, 2023


		## Build integration

		In the future these Python bindings will be available from Nixpkgs as `python3Packages.nix`.


		See [index.md](./doc/index.md), which is also rendered in the HTML manual.

		To hack on these bindings, see [hacking.md](./doc/hacking.md), also rendered in the HTML manual.


		## Documentation

		See [index.md](./doc/index.md), which is also rendered in the HTML manual.

Python bindings: add Value type #17

Are you sure you want to change the base?

Python bindings: add Value type #17

Conversation

roberth commented Mar 11, 2023

yorickvP commented Mar 14, 2023

roberth commented Mar 14, 2023

fricklerhandwerk Apr 16, 2023

Choose a reason for hiding this comment

fricklerhandwerk Apr 16, 2023

Choose a reason for hiding this comment

fricklerhandwerk Apr 16, 2023

Choose a reason for hiding this comment

Python bindings: add `Value` type #17

Python bindings: add `Value` type #17