RTX failed to build nanoset package

[power10dan]

When I was trying to do pip install --r requirements.txt on my macOS, nanoset fails to build. The rootcause seems to be nanoset, which is a package that is dependent on pyo3. The problem I'm having is that my rust compiler (the latest nightly build, as suggested by the pyo3 devs) failed to successfuly compile the package, and gave me a long stack trace along with "cargo failed with code 101." I don't have much ideas on how to fix it, as it's an outdated package issue rather than a code bug that we can dive in to fix it ( I could be wrong, but this is what it looks like to me). Do we really need nanoset? Also, how did you guys install rust? Did anyone of you use the rust nightly build?

[power10dan]

Update: I was able to circumnavigate (for my own code) the issue by individually type out pip install for each package listed in requirements.txt. However, I'm still not sure why when I typed pip install -r requirements.txt, it not only installed the packages in requirements.txt but also more stuff that requires rust. Maybe somewhere along the code there exist packages that need to use a rust compiler?

[saramsey]

I'm unable to reproduce this bug. On macOS 10.14.6, I just did:

git clone https://github.com/RTXteam/RTX.git
cd RTX
virtualenv test
LDFLAGS=-L/usr/local/opt/openssl/lib test/bin/pip3 install -r requirements.txt

no error messages, and nothing about rust.

[saramsey]

Can you please post a step-by-step reproducible example for this bug, along the lines of what I showed above?

[power10dan]

These are the command that I typed:
curl https://sh.rustup.rs -sSf | sh
rustup default nightly
pip install -r requirements.txt

error: aborting due to 89 previous errors

For more information about this error, try rustc --explain E0520.
error: could not compile pyo3.

Caused by:
process didn't exit successfully: rustc --edition=2018 --crate-name pyo3 /Users/daniellin/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-0.8.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="extension-module"' --cfg 'feature="python3"' -C metadata=8b08ab2de62bfa46 -C extra-filename=-8b08ab2de62bfa46 --out-dir /private/var/folders/9s/ccb0g7tn5wl9lwtcqvpffn2c0000gn/T/pip-install-uc7nsh_6/nanoset/target/release/deps -L dependency=/private/var/folders/9s/ccb0g7tn5wl9lwtcqvpffn2c0000gn/T/pip-install-uc7nsh_6/nanoset/target/release/deps --extern indoc=/private/var/folders/9s/ccb0g7tn5wl9lwtcqvpffn2c0000gn/T/pip-install-uc7nsh_6/nanoset/target/release/deps/libindoc-28886ed2ce79f4b3.rmeta --extern inventory=/private/var/folders/9s/ccb0g7tn5wl9lwtcqvpffn2c0000gn/T/pip-install-uc7nsh_6/nanoset/target/release/deps/libinventory-dd83b543eb2d343f.rmeta --extern libc=/private/var/folders/9s/ccb0g7tn5wl9lwtcqvpffn2c0000gn/T/pip-install-uc7nsh_6/nanoset/target/release/deps/liblibc-83e79dc893784f3b.rmeta --extern num_traits=/private/var/folders/9s/ccb0g7tn5wl9lwtcqvpffn2c0000gn/T/pip-install-uc7nsh_6/nanoset/target/release/deps/libnum_traits-fc0f8927b9c24783.rmeta --extern paste=/private/var/folders/9s/ccb0g7tn5wl9lwtcqvpffn2c0000gn/T/pip-install-uc7nsh_6/nanoset/target/release/deps/libpaste-b786b0138a371e08.rmeta --extern pyo3cls=/private/var/folders/9s/ccb0g7tn5wl9lwtcqvpffn2c0000gn/T/pip-install-uc7nsh_6/nanoset/target/release/deps/libpyo3cls-9e711659cef1db7a.dylib --extern spin=/private/var/folders/9s/ccb0g7tn5wl9lwtcqvpffn2c0000gn/T/pip-install-uc7nsh_6/nanoset/target/release/deps/libspin-052b155ee7115df1.rmeta --extern unindent=/private/var/folders/9s/ccb0g7tn5wl9lwtcqvpffn2c0000gn/T/pip-install-uc7nsh_6/nanoset/target/release/deps/libunindent-57ff98b6d80d40e2.rmeta --cap-lints allow --cfg Py_3_5 --cfg Py_3_6 --cfg Py_3_7 --cfg Py_3 --cfg 'py_sys_config="WITH_THREAD"' (exit code: 1)
warning: build failed, waiting for other jobs to finish...
error: build failed
cargo rustc --lib --manifest-path Cargo.toml --features pyo3/python3 pyo3/extension-module extension-module --release --verbose -- --crate-type cdylib -C link-arg=-undefined -C link-arg=dynamic_lookup
error: cargo failed with code: 101

---

ERROR: Failed building wheel for nanoset

[saramsey]

dan, can you try what I did?

git clone https://github.com/RTXteam/RTX.git
cd RTX
virtualenv test
LDFLAGS=-L/usr/local/opt/openssl/lib test/bin/pip3 install -r requirements.txt

[saramsey]

I managed to run the above, and I don't even have rust on my system:

sramsey-laptop:bin sramsey$ locate rust | grep bin
/Users/sramsey/Work/Proj/RTX/kg2/kgx/venv/bin/jupyter-trust
/opt/cisco/anyconnect/bin/vpndownloader.app/Contents/Resources/UntrustedCertBlockedController.nib
/opt/cisco/anyconnect/bin/vpndownloader.app/Contents/Resources/UntrustedCertSettingController.nib
/opt/cisco/anyconnect/bin/vpndownloader.app/Contents/Resources/UntrustedCertWarning.rtfd
/opt/cisco/anyconnect/bin/vpndownloader.app/Contents/Resources/UntrustedCertWarning.rtfd/Caution.png
/opt/cisco/anyconnect/bin/vpndownloader.app/Contents/Resources/UntrustedCertWarning.rtfd/TXT.rtf
/opt/cisco/anyconnect/bin/vpndownloader.app/Contents/Resources/UntrustedCertWarningController.nib
/usr/local/bin/jupyter-trust

[saramsey]

Here is the output of "pip3 freeze" from the clean installation I just did:

sramsey-laptop:bin sramsey$ ./pip3 freeze
appnope==0.1.0
backcall==0.1.0
beautifulsoup4==4.8.1
biothings-client==0.2.1
CacheControl==0.12.5
certifi==2019.9.11
chardet==3.0.4
contexter==0.1.4
decorator==4.4.0
fastobo==0.6.0
frozendict==1.2
idna==2.8
ipython==7.8.0
ipython-cypher==0.2.6
ipython-genutils==0.2.0
jedi==0.15.1
joblib==0.14.0
lockfile==0.12.2
lxml==4.4.1
msgpack==0.6.2
mygene==3.1.0
mysqlclient==1.4.4
nanoset==0.1.2
neo4j-driver==1.7.5
neo4jrestclient==2.1.1
neobolt==1.7.15
neotime==1.7.4
networkx==2.4
nltk==3.4.5
nose==1.3.7
numpy==1.17.3
pandas==0.25.1
parso==0.5.1
pexpect==4.7.0
pickleshare==0.7.5
prettytable==0.7.2
prompt-toolkit==2.0.10
pronto==1.0.0
ptyprocess==0.6.0
Pygments==2.4.2
python-dateutil==2.8.0
pytz==2019.3
requests==2.22.0
requests-cache==0.5.2
scikit-learn==0.21.3
scipy==1.3.1
six==1.12.0
soupsieve==1.9.4
SQLAlchemy==1.3.10
traitlets==4.3.3
urllib3==1.25.6
wcwidth==0.1.7

[saramsey]

@power10dan what makes you think that nanoset is dependent on pyo3? I don't see pyo3 in the output of pip3 freeze above.

[power10dan]

So, I typed what you typed. Here is the installation thing that I got:
Collecting requests-cache
Using cached https://files.pythonhosted.org/packages/7f/55/9b1c40eb83c16d8fc79c5f6c2ffade04208b080670fbfc35e0a5effb5a92/requests_cache-0.5.2-py2.py3-none-any.whl
Collecting pandas
Using cached https://files.pythonhosted.org/packages/39/73/99aa822ee88cef5829607217c11bf24ecc1171ae5d49d5f780085f5da518/pandas-0.25.1-cp37-cp37m-macosx_10_9_x86_64.macosx_10_10_x86_64.whl
Processing /Users/daniellin/Library/Caches/pip/wheels/cc/c8/52/6def472f63b5bd004848d7228597cdf6c1404b189232dee631/neo4j_driver-1.7.5-cp37-none-any.whl
Collecting numpy
Using cached https://files.pythonhosted.org/packages/ea/f4/acaa005b20777fc56a1dc0cae228ab2cb5a7f09a7e7fcb6d4619ce24a1b7/numpy-1.17.3-cp37-cp37m-macosx_10_9_x86_64.whl
Processing /Users/daniellin/Library/Caches/pip/wheels/0e/1e/65/1e0128677b6e6483b90894965f43dd078966270d3dda930044/ipython_cypher-0.2.6-cp37-none-any.whl
Collecting networkx
Using cached https://files.pythonhosted.org/packages/41/8f/dd6a8e85946def36e4f2c69c84219af0fa5e832b018c970e92f2ad337e45/networkx-2.4-py3-none-any.whl
Collecting pronto
Using cached https://files.pythonhosted.org/packages/a4/7c/7185a49d142f4953bd910a399f0784e4064581cfbac08bf03130690472e8/pronto-1.0.0-py2.py3-none-any.whl
Processing /Users/daniellin/Library/Caches/pip/wheels/a0/04/57/031b9b01df38999df7dc7f4ee998a98ecdbd2d781f73e3ffbf/mysqlclient-1.4.4-cp37-cp37m-macosx_10_12_x86_64.whl
Collecting lxml
Using cached https://files.pythonhosted.org/packages/bd/9f/6cda4672d3ad1aa4cf818ab8401a763787efba751c88aaf4b38fc8f923bb/lxml-4.4.1-cp37-cp37m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Collecting mygene
Using cached https://files.pythonhosted.org/packages/10/30/596b5e4077c46add8782de6ac54c173d10b8f15db5bf57d6d8538565c468/mygene-3.1.0-py2.py3-none-any.whl
Collecting beautifulsoup4
Using cached https://files.pythonhosted.org/packages/3b/c8/a55eb6ea11cd7e5ac4bacdf92bac4693b90d3ba79268be16527555e186f0/beautifulsoup4-4.8.1-py3-none-any.whl
Processing /Users/daniellin/Library/Caches/pip/wheels/96/86/f6/68ab24c23f207c0077381a5e3904b2815136b879538a24b483/nltk-3.4.5-cp37-none-any.whl
Processing /Users/daniellin/Library/Caches/pip/wheels/4b/b2/89/cd2231ee623987c605f049df55f40a3e4252ef6a15b94836c2/SQLAlchemy-1.3.10-cp37-cp37m-macosx_10_12_x86_64.whl
Collecting scipy
Using cached https://files.pythonhosted.org/packages/d5/06/1a696649f4b2e706c509cb9333fdc6331fbe71251cede945f9e1fa13ea34/scipy-1.3.1-cp37-cp37m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Collecting scikit-learn
Using cached https://files.pythonhosted.org/packages/e9/57/8a9889d49d0d77905af5a7524fb2b468d2ef5fc723684f51f5ca63efed0d/scikit_learn-0.21.3-cp37-cp37m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Collecting requests
Using cached https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
Processing /Users/daniellin/Library/Caches/pip/wheels/36/bd/5f/dbbee4f2d51f97ecd12a363f870361179cb1fd4bc1174ea08a/CacheControl-0.12.5-cp37-none-any.whl
Collecting lockfile
Using cached https://files.pythonhosted.org/packages/c8/22/9460e311f340cb62d26a38c419b1381b8593b0bb6b5d1f056938b086d362/lockfile-0.12.2-py2.py3-none-any.whl
Requirement already satisfied: python-dateutil>=2.6.1 in ./env/lib/python3.7/site-packages (from pandas->-r requirements.txt (line 3)) (2.8.0)
Requirement already satisfied: pytz>=2017.2 in ./env/lib/python3.7/site-packages (from pandas->-r requirements.txt (line 3)) (2019.3)
Requirement already satisfied: neotime~=1.7.1 in ./env/lib/python3.7/site-packages (from neo4j-driver->-r requirements.txt (line 4)) (1.7.4)
Requirement already satisfied: neobolt~=1.7.15 in ./env/lib/python3.7/site-packages (from neo4j-driver->-r requirements.txt (line 4)) (1.7.15)
Requirement already satisfied: neo4jrestclient>=2.1.0 in ./env/lib/python3.7/site-packages (from ipython-cypher->-r requirements.txt (line 6)) (2.1.1)
Requirement already satisfied: ipython>=1.0 in ./env/lib/python3.7/site-packages (from ipython-cypher->-r requirements.txt (line 6)) (7.8.0)
Requirement already satisfied: prettytable in ./env/lib/python3.7/site-packages (from ipython-cypher->-r requirements.txt (line 6)) (0.7.2)
Requirement already satisfied: decorator>=4.3.0 in ./env/lib/python3.7/site-packages (from networkx->-r requirements.txt (line 7)) (4.4.0)
Processing /Users/daniellin/Library/Caches/pip/wheels/12/7b/af/873a21159a461afc5483eb4a56bc36a666277d7846f11bdd8d/fastobo-0.6.0-cp37-cp37m-macosx_10_12_x86_64.whl
Collecting nanoset~=0.1.2; platform_python_implementation == "CPython"
Using cached https://files.pythonhosted.org/packages/66/99/adaf669bb4fe4dc34d53c2f9bec0ed00769b7295633fdb260ef0332f808d/nanoset-0.1.2.tar.gz
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing wheel metadata ... done
Requirement already satisfied: chardet~=3.0 in ./env/lib/python3.7/site-packages (from pronto->-r requirements.txt (line 8)) (3.0.4)
Processing /Users/daniellin/Library/Caches/pip/wheels/6c/6c/e9/534386165bd12cf1885582c75eb6d0ffcb321b65c23fe0f834/frozendict-1.2-cp37-none-any.whl
Collecting contexter~=0.1.4
Using cached https://files.pythonhosted.org/packages/51/7d/ec4d7e39f2d941b37a8d629c9c5da5f350088ea8e7e145c338753739b04a/contexter-0.1.4-py2.py3-none-any.whl
Collecting biothings-client>=0.2.0
Using cached https://files.pythonhosted.org/packages/b9/fd/4954f8ef8db9a9e3fc56cf96d5c132cd85ef483c4d41710a2586798d9d99/biothings_client-0.2.1-py2.py3-none-any.whl
Collecting soupsieve>=1.2
Using cached https://files.pythonhosted.org/packages/5d/42/d821581cf568e9b7dfc5b415aa61952b0f5e3dede4f3cbd650e3a1082992/soupsieve-1.9.4-py2.py3-none-any.whl
Requirement already satisfied: six in ./env/lib/python3.7/site-packages (from nltk->-r requirements.txt (line 13)) (1.12.0)
Collecting joblib>=0.11
Using cached https://files.pythonhosted.org/packages/8f/42/155696f85f344c066e17af287359c9786b436b1bf86029bb3411283274f3/joblib-0.14.0-py2.py3-none-any.whl
Requirement already satisfied: certifi>=2017.4.17 in ./env/lib/python3.7/site-packages (from requests->-r requirements.txt (line 22)) (2019.9.11)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in ./env/lib/python3.7/site-packages (from requests->-r requirements.txt (line 22)) (1.25.6)
Requirement already satisfied: idna<2.9,>=2.5 in ./env/lib/python3.7/site-packages (from requests->-r requirements.txt (line 22)) (2.8)
Requirement already satisfied: msgpack in ./env/lib/python3.7/site-packages (from CacheControl->-r requirements.txt (line 23)) (0.6.2)
Requirement already satisfied: appnope; sys_platform == "darwin" in ./env/lib/python3.7/site-packages (from ipython>=1.0->ipython-cypher->-r requirements.txt (line 6)) (0.1.0)
Requirement already satisfied: pygments in ./env/lib/python3.7/site-packages (from ipython>=1.0->ipython-cypher->-r requirements.txt (line 6)) (2.4.2)
Requirement already satisfied: pexpect; sys_platform != "win32" in ./env/lib/python3.7/site-packages (from ipython>=1.0->ipython-cypher->-r requirements.txt (line 6)) (4.7.0)
Requirement already satisfied: traitlets>=4.2 in ./env/lib/python3.7/site-packages (from ipython>=1.0->ipython-cypher->-r requirements.txt (line 6)) (4.3.3)
Requirement already satisfied: pickleshare in ./env/lib/python3.7/site-packages (from ipython>=1.0->ipython-cypher->-r requirements.txt (line 6)) (0.7.5)
Requirement already satisfied: jedi>=0.10 in ./env/lib/python3.7/site-packages (from ipython>=1.0->ipython-cypher->-r requirements.txt (line 6)) (0.15.1)
Requirement already satisfied: backcall in ./env/lib/python3.7/site-packages (from ipython>=1.0->ipython-cypher->-r requirements.txt (line 6)) (0.1.0)
Requirement already satisfied: setuptools>=18.5 in ./env/lib/python3.7/site-packages (from ipython>=1.0->ipython-cypher->-r requirements.txt (line 6)) (41.4.0)
Requirement already satisfied: prompt-toolkit<2.1.0,>=2.0.0 in ./env/lib/python3.7/site-packages (from ipython>=1.0->ipython-cypher->-r requirements.txt (line 6)) (2.0.10)
Collecting nose
Using cached https://files.pythonhosted.org/packages/15/d8/dd071918c040f50fa1cf80da16423af51ff8ce4a0f2399b7bf8de45ac3d9/nose-1.3.7-py3-none-any.whl
Requirement already satisfied: ptyprocess>=0.5 in ./env/lib/python3.7/site-packages (from pexpect; sys_platform != "win32"->ipython>=1.0->ipython-cypher->-r requirements.txt (line 6)) (0.6.0)
Requirement already satisfied: ipython-genutils in ./env/lib/python3.7/site-packages (from traitlets>=4.2->ipython>=1.0->ipython-cypher->-r requirements.txt (line 6)) (0.2.0)
Requirement already satisfied: parso>=0.5.0 in ./env/lib/python3.7/site-packages (from jedi>=0.10->ipython>=1.0->ipython-cypher->-r requirements.txt (line 6)) (0.5.1)
Requirement already satisfied: wcwidth in ./env/lib/python3.7/site-packages (from prompt-toolkit<2.1.0,>=2.0.0->ipython>=1.0->ipython-cypher->-r requirements.txt (line 6)) (0.1.7)
*Building wheels for collected packages: nanoset
Building wheel for nanoset (PEP 517) ... *

btw, this is from https://github.com/RTXteam/RTX/blob/master/requirements.txt

[power10dan]

Like, I did the pip install thing on the requirements.txt from https://github.com/RTXteam/RTX/blob/master/requirements.txt

[saramsey]

just confirming -- you did it in a brand-new virtualenv just like I did, right?

[saramsey]

I'm still very confused about why pip3 is somehow invoking rustc on your system. any ideas?

[saramsey]

OK, I am starting to understand. The nanoset package is implemented in rust. Is your system one of the system types for which a native python wheel is available

[saramsey]

@power10dan what version of macOS are you running?

[power10dan]

Yup. I tried what the tutorial did, but for some reason the tutorial doesn't work, as for some reason, inside the pyo3 code that nanoset uses, it was not updated to the latest version. The latest version of pyo3 in docker fixes this issue. But I think for some reason the authors of nanoset didn't update their pyo3 to the latest version, and thus causing this issue. I am running on macOS Sierra.

[power10dan]

and yes, I did it in a brand-new env, and did it outside of an env.

[saramsey]

OK I am running on Mojave. Maybe that is the reason?

[saramsey]

Wait what, you are running in docker?

[saramsey]

Maybe check with the others in the Ramsey Lab to see if anyone else is running on macOS Sierra... to see if they have the same issue? IIRC, mac users in the lab include @amykglen , @zheng-liu, and Qi Wei.

[power10dan]

Nope, I'm not running on Docker. One of the solution proposed online is to use the dockerized version, but the problem lies in the nanoset package.

I think I figured out why. I was retracing the stack trace, and one of the packages pip is trying to install is called nose, which is located in https://github.com/RTXteam/RTX/blob/d756bc9fbed8e14a0777952312e43f61f21e0d63/code/UI/OpenAPI/python-flask-server/test-requirements.txt

I think the problem lies in the fact that I ran pip install -r requirements.txt in the root of the RTX directory. Magically, for some reason, pip decides to install the requirements.txt for the entire repo.
So, somewhere in this repo, somebody is using rust to write their code, or somebody is using a package that depends on rust, and thus causing the issue I'm running into. It's not a high priority bug, but it might cause issues for people who want to use our code since this bug is particularly nasty for people who want to run our code. I've worked around the problem by installing each dependency I need to test the function I want, but this solution is a bandage one rather than an actual, long term solution. Might want to have someone look at it once we clear all high-priority requirements.

[saramsey]

I think the problem lies in the fact that I ran pip install -r requirements.txt in the root of the RTX directory.

Except isn't that what I did? And I got nanoset installed, with no problem.

[power10dan]

I think it might be because you have an older version of rust. I downloaded the newest version of rust, and the compiler was spitting out a bunch of syntax errors for multiple code segments in the nanoset source code.

[zheng-liu]

Hi guys, if we have env incompatibility issues very which are very hard to solve, I also recommend trying https://github.com/pypa/pipenv, instead of pip install.

[saramsey]

@power10dan , I don't think I have rust installed at all on my macOS system. See below:

sramsey-laptop:~ sramsey$ locate rustc
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/Kernel.framework/Versions/A/Headers/kern/trustcache.h
/usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/rustc-completion.rb
/usr/share/vim/vim80/compiler/rustc.vim

Appears that I am getting a precompiled install. This doesn't seem to be an RTX bug, closing.