-
Notifications
You must be signed in to change notification settings - Fork 69
Handle python packages that have shared object files #55
Conversation
76e43c4
to
e8eccc1
Compare
I managed to get rid of the deps. I think its better to have hack tests than to pollute the WORKSPACE with dependencies, but I left both commits as a comparison. |
Thanks for this PR. This is very similar to what the internal Google implementation does. However, I've already been working on a slightly different implementation, with the goal of integrating more closely with the standard setuptools and pkg_resources modules, and the standard dist-info-style metadata. You can see some of the groundwork in #52. If the pkg_resource approach doesn't work for some reason, or to handle cases where pkg_resources is not available, we could still have something like this as a fallback. But I'm probably not going to merge this PR in this form, unfortunately. |
Hello! I've been helping Josh write this, with the goal of getting a runfiles-like tree so we can
Do you have any advice on how to proceed here? It would be great if we could somehow extract only the |
How badly are you blocked here? As a hack, you should be able to take this code and essentially stuff it into your main source file before importing numpy/scipy. I ask, because I'm going to be out of office for the next two weeks, so nothing's going to happen either way for a while. pkg_resources ties into the various forms of Python metadata to, for example, add shared libraries to the "eager resources" list, and/or use the "zip safe" attribute. The result is the same, extracting to a temp directory (note that considerable additional complexity is added if you try to share extractions between multiple processes safely), but is a standard mechanism, instead of one more "strange Google thing". I need to do more investigation though. This shared library problem has been a thorn in my side since 2002, so I'm loathe to go down the same path I've been down before. Inside Google, we patched libc to allow importing from shared libraries stored in zip archives, and some other hacks are used both inside Google and some other companies I know of, including various kernel patches. As an additional hurdle, people quite reasonably also want Windows and MacOS support (.dlls have some of their own peculiarities). |
I'm also having a discussion right now with the Bazel team, and it was brought up that maybe the
CC @meteorcloudy for comment and sanity check what I said above. |
We're not hard-blocked, because we can point Bazel at our local branch until this gets merged into upstream.
Are there any other maintainers of this package that could review it in the next two weeks? If you could point us toward some docs on getting C extensions working with
|
Just thought I'd mention #56 here since it is happening to me when I'm trying to build with Clearly this is not a proper fix, but it does produce a zip with grpc that I can use: https://github.com/pcn/subpar/blob/tooling_around/compiler/cli.py#L106 |
This is option is originally introduced to solve the problem of building py_binary on Windows, because we don't have runfiles on Windows(no symlink without admin right). So it's very straightforward, but if there are more requirements from other platforms, we can definitely improve it. For example, adding an attribute to py_binary for the executable zip functionality is already on my to-do list. |
Ok, setuptools eager resources was a dead end. If you don't mind, I'm going to reorganize this a bit, and make it conditional on a |
Sure! That would be much appreciated! |
Out of curiosity, would these approaches help with issues like bazelbuild/rules_python#48? |
It addresses the runtime part, we still need to figure out the build-time part. E.g. make a py_extension rule. |
Unfortunately, I don't have anything to contribute to this conversation, though I thought I'd leave a comment to register interest in getting this PR (or something like it) merged! 💞 tl;dr Currently setting up initial builds of some Python projects for internal use, was hoping to use Bazel + subpar, realized within a day or two that almost all the projects we'd be building fall under this use case, maintaining an internal fork of subpar isn't an option right now. |
Hopefully this is addressed by #65, and will be further address by long-term Bazel changes. Reopen if that is not the case. |
Some python packages have shared object files that are required to be extracted in order for the system to execute them. This patch extracts all packages that include shared objects to a tmp dir so that the so files can be run properly.