Support Python/C++ interoperability (native Python modules) #701

davidzchen · 2015-12-12T11:19:52Z

According to the documentation for py_library.deps:

The list of other libraries to be linked in to the binary target. See general comments about deps at Attributes common to all build rules. These can be py_binary rules, py_library rules or cc_library rules,

However, when I try to build a py_library target that depends on a cc_library, I get the following error:

❯❯❯ bazel build //external:markupsafe
ERROR: /private/var/tmp/_bazel_dzc/a4c09a1fcefb8b21c5b0e7d7db6f08ba/external/markupsafe_archive/BUILD:9:12: in deps attribute of py_library rule @markupsafe_archive//:markupsafe: cc_library rule '@markupsafe_archive//:speedups' is misplaced here (expected py_binary or py_library).
ERROR: Analysis of target '//external:markupsafe' failed; build aborted.
INFO: Elapsed time: 0.213s

According to BazelPyRuleClasses.java, the only allowed rule types in py_library.deps is, in fact, py_binary and py_library:

  public static final String[] ALLOWED_RULES_IN_DEPS = new String[] {
      "py_binary",
      "py_library",
  };

What is the plan for Python-C interoperability? Should we update the documentation to remove the mention of depending on cc_library in the meantime?

The background for this is that I am trying to use Jinja for generating HTML for the Skylark docgen tool that I am working on. However, Jinja depends on Markupsafe, which is partly implemented in C.

The text was updated successfully, but these errors were encountered:

davidzchen · 2016-02-07T12:33:24Z

Some related discussion here: https://groups.google.com/forum/#!searchin/bazel-discuss/pex/bazel-discuss/-DFOrVqK8aY/-7h4e69hfkEJ

ulfjack · 2016-06-15T07:01:37Z

Recategorizing this as a feature request to support Python-C++ interop. I marked the other report that asked for that as a dupe.

evanj · 2017-08-18T02:21:55Z

A workarounds I found: you can include pre-compiled .sos in the py_library's data attribute. No idea if you can somehow get the output of a cc_library to work in this way.

benley · 2017-08-18T02:23:51Z

The trick to building a .so that works as a python extension seems to be making it a cc_binary instead of a cc_library. I think you have to build it with linkshared = 1 to make it work, but I'd love to be wrong about that.

MarkusTeufelberger · 2017-09-05T16:25:28Z

I stumbled upon this one also recently. Please at least change the documentation or (ideally) increase the priority to get this fixed for good. Python libraries and binaries can and do depend on C libraries. This is currently not really easy to model in bazel.

In the meantime it would also be nice to have this workaround documented somewhere.

buchgr · 2017-09-05T21:51:42Z

@MarkusTeufelberger we love contributions 😃

ulfjack · 2017-09-06T07:04:58Z

The problem is that we don't know how to fix it. We could export our internal model, but we're unsure if that's what people want. Our internal model is that all cc_library rules in deps of py_library rules get merged into a single .so at the py_binary level, together with all cc_library dependencies of py_extension rules, except the py_extension code itself, which gets compiled into a separate .so. It sounds awkward, and it is, but there doesn't seem to be any other principled approach to avoid ODR violations in the C++ code.

ThomasColthurst · 2018-07-20T18:22:20Z

+1 for exporting the internal model. My project is considering switching from bazel to Makefiles solely to avoid the ODR violations we are currently seeing when using bazel + clif + protobufs.

hlopko · 2018-07-31T08:55:21Z

~~Work on this is blocked by #4570.~~ #4570 is fixed, nothing should block this effort.

brandjon · 2019-03-13T19:59:51Z

See related discussion in #1475.

quval · 2021-01-11T19:38:34Z

I have a draft of a Starlark implementation of something like the internal Google model (all native deps are linked together into a statically-linked shared object; the modules themselves are dynamically linked to it). Would it be interesting to share this? Perhaps on rules_python?

pvcnt · 2021-01-11T20:44:11Z

@quval It would certainly be interesting yes. :)

quval · 2021-01-17T00:23:56Z

OK, here goes: https://github.com/quval/rules_native_python/
Got it to work on both Linux and Mac, but only with a toy setup, so I'd still consider this a draft. More importantly, I'm not sure I like the hack I did to work around the circular dependency (native deps -- compiled into native module -- linked against native deps). Any thoughts and suggestions are welcome.

(Edit: just pushed a revised version. Edit 2: seems to have some issues when using gold - will update again once resolved. Edit 3: seems to be all right now. Edit 4, 11/2022: just pushed a major update after the library got some real use.)

mfarrugi · 2021-05-26T22:16:02Z

The following implementation seems to work for me without issue, are there any deficiencies I'm overlooking?

# py_native_library.bzl

def py_native_library(name, **kwargs):
	native.cc_binary(
		name = name + ".so",
		linkshared = True,
		**kwargs
	)
	native.py_library(
		name = name,
		data = [name + ".so"],
	)

Edit:

# py_native_library.bzl

def py_native_library(name, **kwargs):
	native.cc_binary(
		name = name + ".so",
		linkshared = True,
                linkstatic = False,
		**kwargs
	)
	native.py_library(
		name = name,
		data = [name + ".so"],
	)

Is more correct, covering the common case where more than one native extension is loaded into a python executable (and so shared dependencies are properly dynamically linked, rather than statically linked into each extension).

kkpattern · 2021-11-12T07:51:00Z

The problem is that we don't know how to fix it. We could export our internal model, but we're unsure if that's what people want. Our internal model is that all cc_library rules in deps of py_library rules get merged into a single .so at the py_binary level, together with all cc_library dependencies of py_extension rules, except the py_extension code itself, which gets compiled into a separate .so. It sounds awkward, and it is, but there doesn't seem to be any other principled approach to avoid ODR violations in the C++ code.

Hi ulfjack, how does google handle multiple py_library with some common cc_library depends? For example, if I have two py_library both depends on some X cc_library. Initially we may use those two py_library seperately, but at some point people may use both of these two py_library. I think in this situation there is still ODR violations?

Bazel macro py_test_module_list takes a `deps` argument, but completely ignores it instead of passes it to `native.py_test`. Fixing that as we are going to use deps of py_test_module_list in BUILD in later changes. cpp/BUILD.bazel depends on the broken behaviour: it deps-on a cc_library from a py_test, which isn't working, see upstream issue: bazelbuild/bazel#701. This is fixed by simply removing the (non-working) deps.

See #23676 for context. This is another attempt at that as I figured out what's going wrong in `bazel test`. Supersedes #24828. Now that there are Python 3.10 wheels for Ray 1.13 and this is no longer a blocker for supporting Python 3.10, I still want to make `bazel test //python/ray/tests/...` work for developing in a 3.10 env, and make it easier to add Python 3.10 tests to CI in future. The change contains three commits with rather descriptive commit message, which I repeat here: Pass deps to py_test in py_test_module_list Bazel macro py_test_module_list takes a `deps` argument, but completely ignores it instead of passes it to `native.py_test`. Fixing that as we are going to use deps of py_test_module_list in BUILD in later changes. cpp/BUILD.bazel depends on the broken behaviour: it deps-on a cc_library from a py_test, which isn't working, see upstream issue: bazelbuild/bazel#701. This is fixed by simply removing the (non-working) deps. Depend on conftest and data files in Python tests BUILD files Bazel requires that all the files used in a test run should be represented in the transitive dependencies specified for the test target. For py_test, it means srcs, deps and data. Bazel enforces this constraint by creating a "runfiles" directory, symbolic links files in the dependency closure and run the test in the "runfiles" directory, so that the test shouldn't see files not in the dependency graph. Unfortunately, the constraint does not apply for a large number of Python tests, due to pytest (>=3.9.0, <6.0) resolving these symbolic links during test collection and effectively "breaks out" of the runfiles tree. pytest >= 6.0 introduces a breaking change and removed the symbolic link resolving behaviour, see pytest pull request pytest-dev/pytest#6523 for more context. Currently, we are underspecifying dependencies in a lot of BUILD files and thus blocking us from updating to newer pytest (for Python 3.10 support). This change hopefully fixes all of them, and at least those in CI, by adding data or source dependencies (mostly for conftest.py-s) where needed. Bump pytest version from 5.4.3 to 7.0.1 We want at least pytest 6.2.5 for Python 3.10 support, but not past 7.1.0 since it drops Python 3.6 support (which Ray still supports), thus the version constraint is set to <7.1. Updating pytest, combined with earlier BUILD fixes, changed the ground truth of a few error message based unit test, these tests are updated to reflect the change. There are also two small drive-by changes for making test_traceback and test_cli pass under Python 3.10. These are discovered while debugging CI failures (on earlier Python) with a Python 3.10 install locally. Expect more such issues when adding Python 3.10 to CI.

See ray-project#23676 for context. This is another attempt at that as I figured out what's going wrong in `bazel test`. Supersedes ray-project#24828. Now that there are Python 3.10 wheels for Ray 1.13 and this is no longer a blocker for supporting Python 3.10, I still want to make `bazel test //python/ray/tests/...` work for developing in a 3.10 env, and make it easier to add Python 3.10 tests to CI in future. The change contains three commits with rather descriptive commit message, which I repeat here: Pass deps to py_test in py_test_module_list Bazel macro py_test_module_list takes a `deps` argument, but completely ignores it instead of passes it to `native.py_test`. Fixing that as we are going to use deps of py_test_module_list in BUILD in later changes. cpp/BUILD.bazel depends on the broken behaviour: it deps-on a cc_library from a py_test, which isn't working, see upstream issue: bazelbuild/bazel#701. This is fixed by simply removing the (non-working) deps. Depend on conftest and data files in Python tests BUILD files Bazel requires that all the files used in a test run should be represented in the transitive dependencies specified for the test target. For py_test, it means srcs, deps and data. Bazel enforces this constraint by creating a "runfiles" directory, symbolic links files in the dependency closure and run the test in the "runfiles" directory, so that the test shouldn't see files not in the dependency graph. Unfortunately, the constraint does not apply for a large number of Python tests, due to pytest (>=3.9.0, <6.0) resolving these symbolic links during test collection and effectively "breaks out" of the runfiles tree. pytest >= 6.0 introduces a breaking change and removed the symbolic link resolving behaviour, see pytest pull request pytest-dev/pytest#6523 for more context. Currently, we are underspecifying dependencies in a lot of BUILD files and thus blocking us from updating to newer pytest (for Python 3.10 support). This change hopefully fixes all of them, and at least those in CI, by adding data or source dependencies (mostly for conftest.py-s) where needed. Bump pytest version from 5.4.3 to 7.0.1 We want at least pytest 6.2.5 for Python 3.10 support, but not past 7.1.0 since it drops Python 3.6 support (which Ray still supports), thus the version constraint is set to <7.1. Updating pytest, combined with earlier BUILD fixes, changed the ground truth of a few error message based unit test, these tests are updated to reflect the change. There are also two small drive-by changes for making test_traceback and test_cli pass under Python 3.10. These are discovered while debugging CI failures (on earlier Python) with a Python 3.10 install locally. Expect more such issues when adding Python 3.10 to CI. Signed-off-by: Your Name <your@email.com>

See ray-project#23676 for context. This is another attempt at that as I figured out what's going wrong in `bazel test`. Supersedes ray-project#24828. Now that there are Python 3.10 wheels for Ray 1.13 and this is no longer a blocker for supporting Python 3.10, I still want to make `bazel test //python/ray/tests/...` work for developing in a 3.10 env, and make it easier to add Python 3.10 tests to CI in future. The change contains three commits with rather descriptive commit message, which I repeat here: Pass deps to py_test in py_test_module_list Bazel macro py_test_module_list takes a `deps` argument, but completely ignores it instead of passes it to `native.py_test`. Fixing that as we are going to use deps of py_test_module_list in BUILD in later changes. cpp/BUILD.bazel depends on the broken behaviour: it deps-on a cc_library from a py_test, which isn't working, see upstream issue: bazelbuild/bazel#701. This is fixed by simply removing the (non-working) deps. Depend on conftest and data files in Python tests BUILD files Bazel requires that all the files used in a test run should be represented in the transitive dependencies specified for the test target. For py_test, it means srcs, deps and data. Bazel enforces this constraint by creating a "runfiles" directory, symbolic links files in the dependency closure and run the test in the "runfiles" directory, so that the test shouldn't see files not in the dependency graph. Unfortunately, the constraint does not apply for a large number of Python tests, due to pytest (>=3.9.0, <6.0) resolving these symbolic links during test collection and effectively "breaks out" of the runfiles tree. pytest >= 6.0 introduces a breaking change and removed the symbolic link resolving behaviour, see pytest pull request pytest-dev/pytest#6523 for more context. Currently, we are underspecifying dependencies in a lot of BUILD files and thus blocking us from updating to newer pytest (for Python 3.10 support). This change hopefully fixes all of them, and at least those in CI, by adding data or source dependencies (mostly for conftest.py-s) where needed. Bump pytest version from 5.4.3 to 7.0.1 We want at least pytest 6.2.5 for Python 3.10 support, but not past 7.1.0 since it drops Python 3.6 support (which Ray still supports), thus the version constraint is set to <7.1. Updating pytest, combined with earlier BUILD fixes, changed the ground truth of a few error message based unit test, these tests are updated to reflect the change. There are also two small drive-by changes for making test_traceback and test_cli pass under Python 3.10. These are discovered while debugging CI failures (on earlier Python) with a Python 3.10 install locally. Expect more such issues when adding Python 3.10 to CI. Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>

See ray-project#23676 for context. This is another attempt at that as I figured out what's going wrong in `bazel test`. Supersedes ray-project#24828. Now that there are Python 3.10 wheels for Ray 1.13 and this is no longer a blocker for supporting Python 3.10, I still want to make `bazel test //python/ray/tests/...` work for developing in a 3.10 env, and make it easier to add Python 3.10 tests to CI in future. The change contains three commits with rather descriptive commit message, which I repeat here: Pass deps to py_test in py_test_module_list Bazel macro py_test_module_list takes a `deps` argument, but completely ignores it instead of passes it to `native.py_test`. Fixing that as we are going to use deps of py_test_module_list in BUILD in later changes. cpp/BUILD.bazel depends on the broken behaviour: it deps-on a cc_library from a py_test, which isn't working, see upstream issue: bazelbuild/bazel#701. This is fixed by simply removing the (non-working) deps. Depend on conftest and data files in Python tests BUILD files Bazel requires that all the files used in a test run should be represented in the transitive dependencies specified for the test target. For py_test, it means srcs, deps and data. Bazel enforces this constraint by creating a "runfiles" directory, symbolic links files in the dependency closure and run the test in the "runfiles" directory, so that the test shouldn't see files not in the dependency graph. Unfortunately, the constraint does not apply for a large number of Python tests, due to pytest (>=3.9.0, <6.0) resolving these symbolic links during test collection and effectively "breaks out" of the runfiles tree. pytest >= 6.0 introduces a breaking change and removed the symbolic link resolving behaviour, see pytest pull request pytest-dev/pytest#6523 for more context. Currently, we are underspecifying dependencies in a lot of BUILD files and thus blocking us from updating to newer pytest (for Python 3.10 support). This change hopefully fixes all of them, and at least those in CI, by adding data or source dependencies (mostly for conftest.py-s) where needed. Bump pytest version from 5.4.3 to 7.0.1 We want at least pytest 6.2.5 for Python 3.10 support, but not past 7.1.0 since it drops Python 3.6 support (which Ray still supports), thus the version constraint is set to <7.1. Updating pytest, combined with earlier BUILD fixes, changed the ground truth of a few error message based unit test, these tests are updated to reflect the change. There are also two small drive-by changes for making test_traceback and test_cli pass under Python 3.10. These are discovered while debugging CI failures (on earlier Python) with a Python 3.10 install locally. Expect more such issues when adding Python 3.10 to CI. Signed-off-by: Nikita Vemuri <nikitavemuri@gmail.com>

See ray-project#23676 for context. This is another attempt at that as I figured out what's going wrong in `bazel test`. Supersedes ray-project#24828. Now that there are Python 3.10 wheels for Ray 1.13 and this is no longer a blocker for supporting Python 3.10, I still want to make `bazel test //python/ray/tests/...` work for developing in a 3.10 env, and make it easier to add Python 3.10 tests to CI in future. The change contains three commits with rather descriptive commit message, which I repeat here: Pass deps to py_test in py_test_module_list Bazel macro py_test_module_list takes a `deps` argument, but completely ignores it instead of passes it to `native.py_test`. Fixing that as we are going to use deps of py_test_module_list in BUILD in later changes. cpp/BUILD.bazel depends on the broken behaviour: it deps-on a cc_library from a py_test, which isn't working, see upstream issue: bazelbuild/bazel#701. This is fixed by simply removing the (non-working) deps. Depend on conftest and data files in Python tests BUILD files Bazel requires that all the files used in a test run should be represented in the transitive dependencies specified for the test target. For py_test, it means srcs, deps and data. Bazel enforces this constraint by creating a "runfiles" directory, symbolic links files in the dependency closure and run the test in the "runfiles" directory, so that the test shouldn't see files not in the dependency graph. Unfortunately, the constraint does not apply for a large number of Python tests, due to pytest (>=3.9.0, <6.0) resolving these symbolic links during test collection and effectively "breaks out" of the runfiles tree. pytest >= 6.0 introduces a breaking change and removed the symbolic link resolving behaviour, see pytest pull request pytest-dev/pytest#6523 for more context. Currently, we are underspecifying dependencies in a lot of BUILD files and thus blocking us from updating to newer pytest (for Python 3.10 support). This change hopefully fixes all of them, and at least those in CI, by adding data or source dependencies (mostly for conftest.py-s) where needed. Bump pytest version from 5.4.3 to 7.0.1 We want at least pytest 6.2.5 for Python 3.10 support, but not past 7.1.0 since it drops Python 3.6 support (which Ray still supports), thus the version constraint is set to <7.1. Updating pytest, combined with earlier BUILD fixes, changed the ground truth of a few error message based unit test, these tests are updated to reflect the change. There are also two small drive-by changes for making test_traceback and test_cli pass under Python 3.10. These are discovered while debugging CI failures (on earlier Python) with a Python 3.10 install locally. Expect more such issues when adding Python 3.10 to CI. Signed-off-by: Your Name <your@email.com>

See ray-project#23676 for context. This is another attempt at that as I figured out what's going wrong in `bazel test`. Supersedes ray-project#24828. Now that there are Python 3.10 wheels for Ray 1.13 and this is no longer a blocker for supporting Python 3.10, I still want to make `bazel test //python/ray/tests/...` work for developing in a 3.10 env, and make it easier to add Python 3.10 tests to CI in future. The change contains three commits with rather descriptive commit message, which I repeat here: Pass deps to py_test in py_test_module_list Bazel macro py_test_module_list takes a `deps` argument, but completely ignores it instead of passes it to `native.py_test`. Fixing that as we are going to use deps of py_test_module_list in BUILD in later changes. cpp/BUILD.bazel depends on the broken behaviour: it deps-on a cc_library from a py_test, which isn't working, see upstream issue: bazelbuild/bazel#701. This is fixed by simply removing the (non-working) deps. Depend on conftest and data files in Python tests BUILD files Bazel requires that all the files used in a test run should be represented in the transitive dependencies specified for the test target. For py_test, it means srcs, deps and data. Bazel enforces this constraint by creating a "runfiles" directory, symbolic links files in the dependency closure and run the test in the "runfiles" directory, so that the test shouldn't see files not in the dependency graph. Unfortunately, the constraint does not apply for a large number of Python tests, due to pytest (>=3.9.0, <6.0) resolving these symbolic links during test collection and effectively "breaks out" of the runfiles tree. pytest >= 6.0 introduces a breaking change and removed the symbolic link resolving behaviour, see pytest pull request pytest-dev/pytest#6523 for more context. Currently, we are underspecifying dependencies in a lot of BUILD files and thus blocking us from updating to newer pytest (for Python 3.10 support). This change hopefully fixes all of them, and at least those in CI, by adding data or source dependencies (mostly for conftest.py-s) where needed. Bump pytest version from 5.4.3 to 7.0.1 We want at least pytest 6.2.5 for Python 3.10 support, but not past 7.1.0 since it drops Python 3.6 support (which Ray still supports), thus the version constraint is set to <7.1. Updating pytest, combined with earlier BUILD fixes, changed the ground truth of a few error message based unit test, these tests are updated to reflect the change. There are also two small drive-by changes for making test_traceback and test_cli pass under Python 3.10. These are discovered while debugging CI failures (on earlier Python) with a Python 3.10 install locally. Expect more such issues when adding Python 3.10 to CI. Signed-off-by: Avnish <avnishnarayan@gmail.com>

See ray-project#23676 for context. This is another attempt at that as I figured out what's going wrong in `bazel test`. Supersedes ray-project#24828. Now that there are Python 3.10 wheels for Ray 1.13 and this is no longer a blocker for supporting Python 3.10, I still want to make `bazel test //python/ray/tests/...` work for developing in a 3.10 env, and make it easier to add Python 3.10 tests to CI in future. The change contains three commits with rather descriptive commit message, which I repeat here: Pass deps to py_test in py_test_module_list Bazel macro py_test_module_list takes a `deps` argument, but completely ignores it instead of passes it to `native.py_test`. Fixing that as we are going to use deps of py_test_module_list in BUILD in later changes. cpp/BUILD.bazel depends on the broken behaviour: it deps-on a cc_library from a py_test, which isn't working, see upstream issue: bazelbuild/bazel#701. This is fixed by simply removing the (non-working) deps. Depend on conftest and data files in Python tests BUILD files Bazel requires that all the files used in a test run should be represented in the transitive dependencies specified for the test target. For py_test, it means srcs, deps and data. Bazel enforces this constraint by creating a "runfiles" directory, symbolic links files in the dependency closure and run the test in the "runfiles" directory, so that the test shouldn't see files not in the dependency graph. Unfortunately, the constraint does not apply for a large number of Python tests, due to pytest (>=3.9.0, <6.0) resolving these symbolic links during test collection and effectively "breaks out" of the runfiles tree. pytest >= 6.0 introduces a breaking change and removed the symbolic link resolving behaviour, see pytest pull request pytest-dev/pytest#6523 for more context. Currently, we are underspecifying dependencies in a lot of BUILD files and thus blocking us from updating to newer pytest (for Python 3.10 support). This change hopefully fixes all of them, and at least those in CI, by adding data or source dependencies (mostly for conftest.py-s) where needed. Bump pytest version from 5.4.3 to 7.0.1 We want at least pytest 6.2.5 for Python 3.10 support, but not past 7.1.0 since it drops Python 3.6 support (which Ray still supports), thus the version constraint is set to <7.1. Updating pytest, combined with earlier BUILD fixes, changed the ground truth of a few error message based unit test, these tests are updated to reflect the change. There are also two small drive-by changes for making test_traceback and test_cli pass under Python 3.10. These are discovered while debugging CI failures (on earlier Python) with a Python 3.10 install locally. Expect more such issues when adding Python 3.10 to CI. Signed-off-by: klwuibm <kwu888@gmail.com>

See ray-project#23676 for context. This is another attempt at that as I figured out what's going wrong in `bazel test`. Supersedes ray-project#24828. Now that there are Python 3.10 wheels for Ray 1.13 and this is no longer a blocker for supporting Python 3.10, I still want to make `bazel test //python/ray/tests/...` work for developing in a 3.10 env, and make it easier to add Python 3.10 tests to CI in future. The change contains three commits with rather descriptive commit message, which I repeat here: Pass deps to py_test in py_test_module_list Bazel macro py_test_module_list takes a `deps` argument, but completely ignores it instead of passes it to `native.py_test`. Fixing that as we are going to use deps of py_test_module_list in BUILD in later changes. cpp/BUILD.bazel depends on the broken behaviour: it deps-on a cc_library from a py_test, which isn't working, see upstream issue: bazelbuild/bazel#701. This is fixed by simply removing the (non-working) deps. Depend on conftest and data files in Python tests BUILD files Bazel requires that all the files used in a test run should be represented in the transitive dependencies specified for the test target. For py_test, it means srcs, deps and data. Bazel enforces this constraint by creating a "runfiles" directory, symbolic links files in the dependency closure and run the test in the "runfiles" directory, so that the test shouldn't see files not in the dependency graph. Unfortunately, the constraint does not apply for a large number of Python tests, due to pytest (>=3.9.0, <6.0) resolving these symbolic links during test collection and effectively "breaks out" of the runfiles tree. pytest >= 6.0 introduces a breaking change and removed the symbolic link resolving behaviour, see pytest pull request pytest-dev/pytest#6523 for more context. Currently, we are underspecifying dependencies in a lot of BUILD files and thus blocking us from updating to newer pytest (for Python 3.10 support). This change hopefully fixes all of them, and at least those in CI, by adding data or source dependencies (mostly for conftest.py-s) where needed. Bump pytest version from 5.4.3 to 7.0.1 We want at least pytest 6.2.5 for Python 3.10 support, but not past 7.1.0 since it drops Python 3.6 support (which Ray still supports), thus the version constraint is set to <7.1. Updating pytest, combined with earlier BUILD fixes, changed the ground truth of a few error message based unit test, these tests are updated to reflect the change. There are also two small drive-by changes for making test_traceback and test_cli pass under Python 3.10. These are discovered while debugging CI failures (on earlier Python) with a Python 3.10 install locally. Expect more such issues when adding Python 3.10 to CI. Signed-off-by: Frank Luan <lsf@berkeley.edu>

See ray-project#23676 for context. This is another attempt at that as I figured out what's going wrong in `bazel test`. Supersedes ray-project#24828. Now that there are Python 3.10 wheels for Ray 1.13 and this is no longer a blocker for supporting Python 3.10, I still want to make `bazel test //python/ray/tests/...` work for developing in a 3.10 env, and make it easier to add Python 3.10 tests to CI in future. The change contains three commits with rather descriptive commit message, which I repeat here: Pass deps to py_test in py_test_module_list Bazel macro py_test_module_list takes a `deps` argument, but completely ignores it instead of passes it to `native.py_test`. Fixing that as we are going to use deps of py_test_module_list in BUILD in later changes. cpp/BUILD.bazel depends on the broken behaviour: it deps-on a cc_library from a py_test, which isn't working, see upstream issue: bazelbuild/bazel#701. This is fixed by simply removing the (non-working) deps. Depend on conftest and data files in Python tests BUILD files Bazel requires that all the files used in a test run should be represented in the transitive dependencies specified for the test target. For py_test, it means srcs, deps and data. Bazel enforces this constraint by creating a "runfiles" directory, symbolic links files in the dependency closure and run the test in the "runfiles" directory, so that the test shouldn't see files not in the dependency graph. Unfortunately, the constraint does not apply for a large number of Python tests, due to pytest (>=3.9.0, <6.0) resolving these symbolic links during test collection and effectively "breaks out" of the runfiles tree. pytest >= 6.0 introduces a breaking change and removed the symbolic link resolving behaviour, see pytest pull request pytest-dev/pytest#6523 for more context. Currently, we are underspecifying dependencies in a lot of BUILD files and thus blocking us from updating to newer pytest (for Python 3.10 support). This change hopefully fixes all of them, and at least those in CI, by adding data or source dependencies (mostly for conftest.py-s) where needed. Bump pytest version from 5.4.3 to 7.0.1 We want at least pytest 6.2.5 for Python 3.10 support, but not past 7.1.0 since it drops Python 3.6 support (which Ray still supports), thus the version constraint is set to <7.1. Updating pytest, combined with earlier BUILD fixes, changed the ground truth of a few error message based unit test, these tests are updated to reflect the change. There are also two small drive-by changes for making test_traceback and test_cli pass under Python 3.10. These are discovered while debugging CI failures (on earlier Python) with a Python 3.10 install locally. Expect more such issues when adding Python 3.10 to CI. Signed-off-by: Scott Graham <scgraham@microsoft.com>

See ray-project#23676 for context. This is another attempt at that as I figured out what's going wrong in `bazel test`. Supersedes ray-project#24828. Now that there are Python 3.10 wheels for Ray 1.13 and this is no longer a blocker for supporting Python 3.10, I still want to make `bazel test //python/ray/tests/...` work for developing in a 3.10 env, and make it easier to add Python 3.10 tests to CI in future. The change contains three commits with rather descriptive commit message, which I repeat here: Pass deps to py_test in py_test_module_list Bazel macro py_test_module_list takes a `deps` argument, but completely ignores it instead of passes it to `native.py_test`. Fixing that as we are going to use deps of py_test_module_list in BUILD in later changes. cpp/BUILD.bazel depends on the broken behaviour: it deps-on a cc_library from a py_test, which isn't working, see upstream issue: bazelbuild/bazel#701. This is fixed by simply removing the (non-working) deps. Depend on conftest and data files in Python tests BUILD files Bazel requires that all the files used in a test run should be represented in the transitive dependencies specified for the test target. For py_test, it means srcs, deps and data. Bazel enforces this constraint by creating a "runfiles" directory, symbolic links files in the dependency closure and run the test in the "runfiles" directory, so that the test shouldn't see files not in the dependency graph. Unfortunately, the constraint does not apply for a large number of Python tests, due to pytest (>=3.9.0, <6.0) resolving these symbolic links during test collection and effectively "breaks out" of the runfiles tree. pytest >= 6.0 introduces a breaking change and removed the symbolic link resolving behaviour, see pytest pull request pytest-dev/pytest#6523 for more context. Currently, we are underspecifying dependencies in a lot of BUILD files and thus blocking us from updating to newer pytest (for Python 3.10 support). This change hopefully fixes all of them, and at least those in CI, by adding data or source dependencies (mostly for conftest.py-s) where needed. Bump pytest version from 5.4.3 to 7.0.1 We want at least pytest 6.2.5 for Python 3.10 support, but not past 7.1.0 since it drops Python 3.6 support (which Ray still supports), thus the version constraint is set to <7.1. Updating pytest, combined with earlier BUILD fixes, changed the ground truth of a few error message based unit test, these tests are updated to reflect the change. There are also two small drive-by changes for making test_traceback and test_cli pass under Python 3.10. These are discovered while debugging CI failures (on earlier Python) with a Python 3.10 install locally. Expect more such issues when adding Python 3.10 to CI.

See ray-project#23676 for context. This is another attempt at that as I figured out what's going wrong in `bazel test`. Supersedes ray-project#24828. Now that there are Python 3.10 wheels for Ray 1.13 and this is no longer a blocker for supporting Python 3.10, I still want to make `bazel test //python/ray/tests/...` work for developing in a 3.10 env, and make it easier to add Python 3.10 tests to CI in future. The change contains three commits with rather descriptive commit message, which I repeat here: Pass deps to py_test in py_test_module_list Bazel macro py_test_module_list takes a `deps` argument, but completely ignores it instead of passes it to `native.py_test`. Fixing that as we are going to use deps of py_test_module_list in BUILD in later changes. cpp/BUILD.bazel depends on the broken behaviour: it deps-on a cc_library from a py_test, which isn't working, see upstream issue: bazelbuild/bazel#701. This is fixed by simply removing the (non-working) deps. Depend on conftest and data files in Python tests BUILD files Bazel requires that all the files used in a test run should be represented in the transitive dependencies specified for the test target. For py_test, it means srcs, deps and data. Bazel enforces this constraint by creating a "runfiles" directory, symbolic links files in the dependency closure and run the test in the "runfiles" directory, so that the test shouldn't see files not in the dependency graph. Unfortunately, the constraint does not apply for a large number of Python tests, due to pytest (>=3.9.0, <6.0) resolving these symbolic links during test collection and effectively "breaks out" of the runfiles tree. pytest >= 6.0 introduces a breaking change and removed the symbolic link resolving behaviour, see pytest pull request pytest-dev/pytest#6523 for more context. Currently, we are underspecifying dependencies in a lot of BUILD files and thus blocking us from updating to newer pytest (for Python 3.10 support). This change hopefully fixes all of them, and at least those in CI, by adding data or source dependencies (mostly for conftest.py-s) where needed. Bump pytest version from 5.4.3 to 7.0.1 We want at least pytest 6.2.5 for Python 3.10 support, but not past 7.1.0 since it drops Python 3.6 support (which Ray still supports), thus the version constraint is set to <7.1. Updating pytest, combined with earlier BUILD fixes, changed the ground truth of a few error message based unit test, these tests are updated to reflect the change. There are also two small drive-by changes for making test_traceback and test_cli pass under Python 3.10. These are discovered while debugging CI failures (on earlier Python) with a Python 3.10 install locally. Expect more such issues when adding Python 3.10 to CI. Signed-off-by: Stefan van der Kleij <s.vanderkleij@viroteq.com>

vkresch · 2022-08-18T11:36:10Z

@mfarrugi thanks for your example. This works when I am using pybind11. Meaning if I want to call C++ functions via python this works. But how about the other way around call python functions from C++. Can I do this then:

def cc_native_library(name, py_srcs, py_deps, cc_srcs, cc_deps, **kwargs):
	native.py_library(
		name = name + ".py",
                srcs = py_srcs,
                deps = py_deps,
		**kwargs
	)
	native.cc_library(
		name = name,
                srcs = cc_srcs,
                deps = cc_deps,
		data = [name + ".py"],
		linkshared = True,
                linkstatic = False,
                deps = [":python3"], # depend on python3 path location
	)

I want to use the cc_native_library for a cc_binary.
I tried this approach for pybind11 but the problem is here with the third party packages which do not get imported correctly since they are used in the data argument of cc_library. If using only builtin libraries this works. Any suggestions or examples? Thanks in advance!

github-actions · 2024-01-13T01:28:35Z

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 90 days unless any other activity occurs. If you think this issue is still relevant and should stay open, please post any comment here and the issue will no longer be marked as stale.

github-actions · 2024-04-13T01:17:56Z

This issue has been automatically closed due to inactivity. If you're still interested in pursuing this, please post @bazelbuild/triage in a comment here and we'll take a look. Thanks!

jab · 2024-04-13T19:07:49Z

@bazelbuild/triage

davidzchen added type: documentation (cleanup) Native rules (Java / C++ / Python) labels Dec 12, 2015

damienmg added category: rules > C++ and removed category: rules > java labels Jun 14, 2016

ulfjack mentioned this issue Jun 15, 2016

support for py_library to link to a cc_library #1366

Closed

ulfjack added type: feature request category: rules > python P3 We're not considering working on this, but happy to review a PR. (No assignee) and removed category: rules > C++ type: documentation (cleanup) labels Jun 15, 2016

ulfjack added this to the 1.0 milestone Jun 15, 2016

hlopko assigned brandjon and oquenchil Jul 31, 2018

mboes mentioned this issue Aug 9, 2018

python dlopen cc_haskell_import breaks with foreign imports tweag/rules_haskell#370

Closed

brandjon mentioned this issue Oct 18, 2018

Document that py_library can't depend on cc_library #6447

Closed

brandjon added team-Rules-Python Native rules for Python and removed category: rules > python labels Oct 18, 2018

brandjon removed their assignment Oct 18, 2018

brandjon mentioned this issue Oct 18, 2018

Handle native python modules #1475

Closed

brandjon unassigned oquenchil Mar 13, 2019

lberki added P4 This is either out of scope or we don't have bandwidth to review a PR. (No assignee) and removed P3 We're not considering working on this, but happy to review a PR. (No assignee) labels Nov 18, 2020

mfarrugi mentioned this issue Sep 18, 2021

pyo3 python_calling_rust example bazelbuild/rules_rust#753

Open

1 task

github-actions bot added the stale Issues or PRs that are stale (no activity for 30 days) label Jan 13, 2024

github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Apr 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Python/C++ interoperability (native Python modules) #701

Support Python/C++ interoperability (native Python modules) #701

davidzchen commented Dec 12, 2015

davidzchen commented Feb 7, 2016

ulfjack commented Jun 15, 2016

evanj commented Aug 18, 2017

benley commented Aug 18, 2017

MarkusTeufelberger commented Sep 5, 2017

buchgr commented Sep 5, 2017

ulfjack commented Sep 6, 2017

ThomasColthurst commented Jul 20, 2018

hlopko commented Jul 31, 2018 •

edited

Loading

brandjon commented Mar 13, 2019

quval commented Jan 11, 2021

pvcnt commented Jan 11, 2021

quval commented Jan 17, 2021 •

edited

Loading

mfarrugi commented May 26, 2021 •

edited

Loading

kkpattern commented Nov 12, 2021 •

edited

Loading

vkresch commented Aug 18, 2022 •

edited

Loading

github-actions bot commented Jan 13, 2024

github-actions bot commented Apr 13, 2024

jab commented Apr 13, 2024

Support Python/C++ interoperability (native Python modules) #701

Support Python/C++ interoperability (native Python modules) #701

Comments

davidzchen commented Dec 12, 2015

davidzchen commented Feb 7, 2016

ulfjack commented Jun 15, 2016

evanj commented Aug 18, 2017

benley commented Aug 18, 2017

MarkusTeufelberger commented Sep 5, 2017

buchgr commented Sep 5, 2017

ulfjack commented Sep 6, 2017

ThomasColthurst commented Jul 20, 2018

hlopko commented Jul 31, 2018 • edited Loading

brandjon commented Mar 13, 2019

quval commented Jan 11, 2021

pvcnt commented Jan 11, 2021

quval commented Jan 17, 2021 • edited Loading

mfarrugi commented May 26, 2021 • edited Loading

kkpattern commented Nov 12, 2021 • edited Loading

vkresch commented Aug 18, 2022 • edited Loading

github-actions bot commented Jan 13, 2024

github-actions bot commented Apr 13, 2024

jab commented Apr 13, 2024

hlopko commented Jul 31, 2018 •

edited

Loading

quval commented Jan 17, 2021 •

edited

Loading

mfarrugi commented May 26, 2021 •

edited

Loading

kkpattern commented Nov 12, 2021 •

edited

Loading

vkresch commented Aug 18, 2022 •

edited

Loading