Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix external DLL loading on wheels #2811

Merged
merged 9 commits into from
Oct 15, 2020
Merged

Conversation

andfoy
Copy link
Contributor

@andfoy andfoy commented Oct 14, 2020

This PR registers the path of the main torchvision library via AddDllDirectory on Python < 3.7 and os.add_dll_directory on Python >= 3.8. The extensions are preloaded using LoadLibraryExW to circunvent the limitation imposed by AddDllDirectory without calling SetDefaultDllDirectories

Copy link
Member

@fmassa fmassa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR!

I'm not very comfortable with the current solution.

Would there be a way to fix Windows without having to do this PATH manipulation at import time?

For reference, we identified in #2775 that the Windows pip package for torchvision are still not loading image.dll correctly, and this PR is a tentative fix for it.

torchvision/io/_video_opt.py Outdated Show resolved Hide resolved
@codecov
Copy link

codecov bot commented Oct 14, 2020

Codecov Report

Merging #2811 into master will decrease coverage by 0.07%.
The diff coverage is 62.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #2811      +/-   ##
==========================================
- Coverage   73.33%   73.26%   -0.08%     
==========================================
  Files          99       99              
  Lines        8730     8778      +48     
  Branches     1374     1387      +13     
==========================================
+ Hits         6402     6431      +29     
- Misses       1909     1920      +11     
- Partials      419      427       +8     
Impacted Files Coverage Δ
torchvision/io/_video_opt.py 40.57% <56.25%> (+1.19%) ⬆️
torchvision/extension.py 67.21% <64.70%> (-0.97%) ⬇️
torchvision/io/image.py 82.25% <64.70%> (-6.88%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 16ef312...ba845f8. Read the comment docs.

@andfoy
Copy link
Contributor Author

andfoy commented Oct 14, 2020

It seems that this solution is damaging also the binary imports of other libraries, such as Scipy

@fmassa
Copy link
Member

fmassa commented Oct 14, 2020

@peterjc123 could you help review this PR?

For context, we identified in #2775 that the Windows pip package for torchvision are still not loading image.dll correctly, and this PR is a tentative fix for it.

@andfoy andfoy self-assigned this Oct 14, 2020
Copy link
Contributor

@peterjc123 peterjc123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. BTW, it would be better to refactor the code to eliminate duplicates.

@peterjc123
Copy link
Contributor

Would there be a way to fix Windows without having to do this PATH manipulation at import time?

No, unlike unix platforms, Windows doesn't have an alternative for RPATH for native executables.

@fmassa
Copy link
Member

fmassa commented Oct 15, 2020

Thanks @andfoy and @peterjc123 , I'm merging this to unblock and do some testings.

Also, one thing that I would like to point out is that there is a non-symmetric inclusion here, where we add the path to the DLLs in extension.py but not in image and video, which might add the implicit requirement that we need to import extension.py before the others. Not sure if this is a problem?

@fmassa fmassa merged commit 072d8b2 into pytorch:master Oct 15, 2020
fmassa pushed a commit that referenced this pull request Oct 16, 2020
* Fix external DLL loading on wheels

* Use SetDefaultDllDirectoriess and AddDllDirectory

* Add previous paths

* Trigger debug

* Do not call SetDefaultDllDirectories.

* Do not call loadlibrary if the extensions were not compiled

* Fix lint issues
@andfoy andfoy deleted the fix_dll_loading_win branch October 19, 2020 20:16
bryant1410 pushed a commit to bryant1410/vision-1 that referenced this pull request Nov 22, 2020
* Fix external DLL loading on wheels

* Use SetDefaultDllDirectoriess and AddDllDirectory

* Add previous paths

* Trigger debug

* Do not call SetDefaultDllDirectories.

* Do not call loadlibrary if the extensions were not compiled

* Fix lint issues
vfdev-5 pushed a commit to Quansight/vision that referenced this pull request Dec 4, 2020
* Fix external DLL loading on wheels

* Use SetDefaultDllDirectoriess and AddDllDirectory

* Add previous paths

* Trigger debug

* Do not call SetDefaultDllDirectories.

* Do not call loadlibrary if the extensions were not compiled

* Fix lint issues
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants