Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix OSX CI. #2482

Closed
wants to merge 3 commits into from
Closed

Fix OSX CI. #2482

wants to merge 3 commits into from

Conversation

beauby
Copy link
Contributor

@beauby beauby commented Sep 20, 2022

No description provided.

@beauby
Copy link
Contributor Author

beauby commented Sep 20, 2022 

$ python3 -m unittest tests/torch_test_contrib.py
OMP: Error #15: Initializing libomp.dylib, but found libomp.dylib already initialized.
OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://openmp.llvm.org/
Abort trap: 6

$ DYLD_PRINT_LIBRARIES=1 python3 -m unittest tests/torch_test_contrib.py
[...]
dyld[5861]: <4EB99AB8-B09B-3356-B58B-47F2F3411184> /usr/local/lib/python3.10/site-packages/torch/_C.cpython-310-darwin.so
dyld[5861]: <438192E1-F1DA-345D-8095-3DA12BC344B2> /usr/local/lib/python3.10/site-packages/torch/lib/libtorch_python.dylib
dyld[5861]: <084A1DEC-A2A2-3111-A4B5-57DA4487E480> /usr/local/lib/python3.10/site-packages/torch/lib/libshm.dylib
dyld[5861]: <94A706CB-B379-31D8-B3B1-AD6940E5E005> /usr/local/lib/python3.10/site-packages/torch/lib/libtorch.dylib
dyld[5861]: <4592504E-06D6-38EE-8297-6401DE3799B6> /usr/local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.dylib
dyld[5861]: <B74C52E9-94DE-310E-82BD-9826895E1BEE> /usr/local/lib/python3.10/site-packages/torch/lib/libc10.dylib
dyld[5861]: <300ECA29-CA27-320E-83E3-68DA68AE22CF> /usr/local/lib/python3.10/site-packages/torch/.dylibs/libomp.dylib
[...]
dyld[5861]: <34815B28-7115-3805-ACC0-8D34CECA95F1> /Users/distiller/project/build/faiss/python/build/lib/faiss/_swigfaiss.so
dyld[5861]: <300ECA29-CA27-320E-83E3-68DA68AE22CF> /usr/local/Cellar/libomp/14.0.6/lib/libomp.dylib
[...]

It seems torch loads a non-system libomp.dylib, which conflicts with the system libomp.dylib that Faiss attemps to load.

@beauby
Copy link
Contributor Author

beauby commented Sep 20, 2022 •
edited
Loading

Minimal repro:

$ python3 -c "import torch; import faiss; faiss.omp_get_max_threads()"

@beauby
Copy link
Contributor Author

beauby commented Sep 20, 2022

Related: pytorch/pytorch#44282

@beauby
Copy link
Contributor Author

beauby commented Sep 20, 2022 •
edited
Loading

The issue does not manifest itself in torch 1.11.0 (but it does in 1.12.0).

$ pip3 install torch==1.11.0
$ python3 -c "import faiss; import torch; faiss.omp_get_max_threads()"
$ echo $?
0

@beauby beauby changed the title Update xcode version of OSX CI. Investigate OSX CI. Sep 27, 2022
beauby added a commit to beauby/faiss that referenced this pull request Sep 28, 2022
@beauby
Pin pytorch version for OSX tests.
8d5f2a3 
Cf. facebookresearch#2482.
@beauby
Update xcode version of OSX CI.
9cecaf8
@beauby
Use conda for OSX CI.
5e37e1f
@beauby
Pin pytorch version for OSX tests.
94457af 
Current version of pytorch (12.0.1) ships a specific openmp runtime on
OSX, which conflicts with the system one. This workaround fixes Faiss'
CI until pytorch is fixed, but it does not fix the incompatibility
itself.
@beauby beauby changed the title Investigate OSX CI. Fix OSX CI. Sep 28, 2022
@facebook-github-bot
Copy link
Contributor

@beauby has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@davidmezzetti davidmezzetti mentioned this pull request Jan 27, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
CLA Signed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@beauby @facebook-github-bot