Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fixing Criteo Inference for TensorFlow and HugeCTR #1500

Merged
merged 33 commits into from
Apr 5, 2022
Merged

Fixing Criteo Inference for TensorFlow and HugeCTR #1500

merged 33 commits into from
Apr 5, 2022

Conversation

bschifferer
Copy link
Contributor

TensorFlow Inference Notebook had issues converting Null values - added fillna() as preliminary fix. Long term solution requires a work around for: NVIDIA-Merlin/systems#78

HugeCTR Inference Notebook did not produce a correct ps.json file. HugeCTR requires more parameter. In addition, adding short term fix for null values

benfred and others added 30 commits December 14, 2020 09:15
@benfred
API Overhaul
d708172 
First draft of the API overhauls changes. Adds most core functionality, including
defining workflow graphs with a ColumnGroup class, the workflow and dataset changes
, most operators converted to use the new api, etc.
@benfred
remove debug print statement
abc0cb4
@benfred
Fix test_io unittest
b69ad67 
Also partially fix some tests inside test_workflow
@benfred
Merge remote-tracking branch 'origin/main' into new_api
2fd3db9
@benfred
Handle multi-column joint/combo categorify
2e918cd
@benfred
Update JoinGroupby
23a4eb9
@benfred
Fix differencelag
711efd8
@rnyak
add dependencies method (NVIDIA-Merlin#498)
fd3f35a
@benfred
Convert TargetEncoding op
9b78a46
@benfred
Merge branch 'new_api' of github.com:NVIDIA/NVTabular into new_api
82f1c17
@benfred @rjzamora
Update nvtabular/workflow.py
5c28e85 
Co-authored-by: Richard (Rick) Zamora <rzamora217@gmail.com>
@benfred @rjzamora
Update nvtabular/workflow.py
6a84e7d 
Co-authored-by: Richard (Rick) Zamora <rzamora217@gmail.com>
@benfred
Remove workflow code from dataloaders
b21ffd8 
We should be doing online transforms like
```KerasSequenceLoader(workflow.transform(dataset), ...```  instead of
```KerasSequenceLoader(dataset, workflows=[workflow], ...``` now
@benfred
Merge branch 'new_api' of github.com:NVIDIA/NVTabular into new_api
5467d86
@bschifferer
Unittest ops + bugfix in Bucketize (NVIDIA-Merlin#496)
f216edf 
* test_minmix

* updates test

* unittest ops
@benfred
First draft get_embedding_sizes support
b44dfa6 
Re-add get_embedding_sizes . Note that this changes how we support multi-hot columns here
(sizes are returned same as single hot, and we don't use this method to distinguish between
multi and singlehot columns)
@benfred
isort
27b4e33
@benfred
Remove groupbystatistics
a95a4d9
@benfred
implement serialization of statistics
0e55c2a 
add save_stats/load_stats/clear_stats methods to the workflow, with each statoperator getting
called as appropiate
@benfred
Fix TF dataloader unittests
ee9367c
@benfred
test_torch_dataloader fixes
7bf624f
@bschifferer
doc strings
4c99186
@bschifferer
resovled
aee86e2
@bschifferer
Merge remote-tracking branch 'upstream/main'
ef52f5a
@bschifferer
Merge remote-tracking branch 'upstream/main'
a2cdcc6
@bschifferer
merge
1783f8c
@bschifferer
t checkout master
3ebf331 
git merge upstream/main
Merge branch 'main' of https://github.com/NVIDIA/NVTabular
@bschifferer
Merge branch 'main' of https://github.com/NVIDIA/NVTabular
a3c9695
Merge branch 'main' of https://github.com/NVIDIA-Merlin/NVTabular
4d53a25
Merge branch 'main' of https://github.com/NVIDIA/NVTabular
5f8c040
@review-notebook-app
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

@nvidia-merlin-bot
Copy link
Contributor

Click to view CI Results
GitHub pull request #1500 of commit e64e90f6fbecde92f241270efd49e448aca3faf4, no merge conflicts.
Running as SYSTEM
Setting status of e64e90f6fbecde92f241270efd49e448aca3faf4 to PENDING with url http://10.20.13.93:8080/job/nvtabular_tests/4417/ and message: 'Pending'
Using context: Jenkins Unit Test Run
Building on master in workspace /var/jenkins_home/workspace/nvtabular_tests
using credential nvidia-merlin-bot
Cloning the remote Git repository
Cloning repository https://github.com/NVIDIA-Merlin/NVTabular.git
 > git init /var/jenkins_home/workspace/nvtabular_tests/nvtabular # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/NVTabular.git
 > git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/NVTabular.git +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/NVTabular.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/NVTabular.git # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/NVTabular.git
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/NVTabular.git +refs/pull/1500/*:refs/remotes/origin/pr/1500/* # timeout=10
 > git rev-parse e64e90f6fbecde92f241270efd49e448aca3faf4^{commit} # timeout=10
Checking out Revision e64e90f6fbecde92f241270efd49e448aca3faf4 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f e64e90f6fbecde92f241270efd49e448aca3faf4 # timeout=10
Commit message: "criteo update"
 > git rev-list --no-walk 3c04458316a0df2556bcf2728896112eaa6cb95f # timeout=10
First time build. Skipping changelog.
[nvtabular_tests] $ /bin/bash /tmp/jenkins3558322435128931400.sh
rm: cannot remove '/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular.egg-link': No such file or directory
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: setuptools in /usr/local/lib/python3.8/dist-packages (62.0.0)
Installing NVTabular
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: pip in /var/jenkins_home/.local/lib/python3.8/site-packages (22.0.4)
Requirement already satisfied: setuptools in /usr/local/lib/python3.8/dist-packages (62.0.0)
Requirement already satisfied: wheel in /var/jenkins_home/.local/lib/python3.8/site-packages (0.37.1)
Requirement already satisfied: pybind11 in /var/jenkins_home/.local/lib/python3.8/site-packages (2.9.2)
Requirement already satisfied: numpy==1.20.3 in /var/jenkins_home/.local/lib/python3.8/site-packages (1.20.3)
Found existing installation: nvtabular 0.11.0+44.g50eb7ef1
Can't uninstall 'nvtabular'. No files were found to uninstall.
Found existing installation: merlin-core 0.1.1+21.g32e4695
Can't uninstall 'merlin-core'. No files were found to uninstall.
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-install-j9v4vi0a/merlin-core_22f4915adbbf457292e59cfe8b94b088
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-install-j9v4vi0a/merlin-core_22f4915adbbf457292e59cfe8b94b088
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit e98eb171171a193e31f4244e3c1c562def54a967
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (3.20.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (4.63.1)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (21.3)
Requirement already satisfied: distributed==2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (2021.11.2)
Requirement already satisfied: numba>=0.53.1 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (0.55.1)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (1.2.5)
Requirement already satisfied: dask==2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (2021.11.2)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (1.3.5)
Requirement already satisfied: pyarrow>=1.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (5.0.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (1.7.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask==2021.11.2->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask==2021.11.2->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (0.11.2)
Requirement already satisfied: pyyaml in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask==2021.11.2->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (5.4.1)
Requirement already satisfied: fsspec>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from dask==2021.11.2->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (2021.11.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask==2021.11.2->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (2.0.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed==2021.11.2->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (6.1)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed==2021.11.2->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (2.0.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed==2021.11.2->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (8.0.4)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed==2021.11.2->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (2.4.0)
Requirement already satisfied: setuptools in /usr/local/lib/python3.8/dist-packages (from distributed==2021.11.2->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (62.0.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed==2021.11.2->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (5.8.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed==2021.11.2->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (1.0.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed==2021.11.2->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (3.0.3)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed==2021.11.2->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (1.7.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (0.4.2)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (1.2.0)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in /usr/local/lib/python3.8/dist-packages (from numba>=0.53.1->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (0.38.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.53.1->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (1.20.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (3.0.7)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (2022.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (1.56.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /var/jenkins_home/.local/lib/python3.8/site-packages/absl_py-0.12.0-py3.8.egg (from tensorflow-metadata>=1.2.0->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (0.12.0)
Requirement already satisfied: six in /var/jenkins_home/.local/lib/python3.8/site-packages (from absl-py<2.0.0,>=0.9->tensorflow-metadata>=1.2.0->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (1.15.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask==2021.11.2->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (0.2.1)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed==2021.11.2->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed==2021.11.2->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (2.0.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (4.0.0)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.1.1+26.ge98eb17-py3-none-any.whl size=131890 sha256=169d7b0f881f7bf79fdca77c4f42356928b1e496b46a7dbb14579a1deb8a138c
  Stored in directory: /tmp/pip-ephem-wheel-cache-93qwrfpa/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
Successfully installed merlin-core-0.1.1+26.ge98eb17
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: natsort==8.1.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (8.1.0)
running develop
running egg_info
creating nvtabular.egg-info
writing nvtabular.egg-info/PKG-INFO
writing dependency_links to nvtabular.egg-info/dependency_links.txt
writing requirements to nvtabular.egg-info/requires.txt
writing top-level names to nvtabular.egg-info/top_level.txt
writing manifest file 'nvtabular.egg-info/SOURCES.txt'
/usr/local/lib/python3.8/dist-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
/usr/local/lib/python3.8/dist-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
package init file 'ci/__init__.py' not found (or not a regular file)
package init file 'images/__init__.py' not found (or not a regular file)
package init file 'docs/__init__.py' not found (or not a regular file)
package init file 'cpp/__init__.py' not found (or not a regular file)
package init file 'bench/__init__.py' not found (or not a regular file)
package init file 'merlin/__init__.py' not found (or not a regular file)
package init file 'examples/__init__.py' not found (or not a regular file)
package init file 'conda/__init__.py' not found (or not a regular file)
package init file 'docs/source/__init__.py' not found (or not a regular file)
package init file 'docs/source/_templates/__init__.py' not found (or not a regular file)
package init file 'docs/source/images/__init__.py' not found (or not a regular file)
package init file 'docs/source/training/__init__.py' not found (or not a regular file)
package init file 'docs/source/api/__init__.py' not found (or not a regular file)
package init file 'docs/source/examples/__init__.py' not found (or not a regular file)
package init file 'docs/source/resources/__init__.py' not found (or not a regular file)
package init file 'docs/source/api/ops/__init__.py' not found (or not a regular file)
package init file 'docs/source/api/workflow/__init__.py' not found (or not a regular file)
package init file 'docs/source/examples/getting-started-movielens/__init__.py' not found (or not a regular file)
package init file 'docs/source/examples/tabular-data-rossmann/__init__.py' not found (or not a regular file)
package init file 'docs/source/examples/advanced-ops-outbrain/__init__.py' not found (or not a regular file)
package init file 'docs/source/examples/scaling-criteo/__init__.py' not found (or not a regular file)
package init file 'docs/source/examples/winning-solution-recsys2020-twitter/__init__.py' not found (or not a regular file)
package init file 'docs/source/examples/scaling-criteo/imgs/__init__.py' not found (or not a regular file)
package init file 'cpp/nvtabular/__init__.py' not found (or not a regular file)
package init file 'cpp/nvtabular/inference/__init__.py' not found (or not a regular file)
package init file 'bench/datasets/__init__.py' not found (or not a regular file)
package init file 'bench/torch/__init__.py' not found (or not a regular file)
package init file 'bench/examples/__init__.py' not found (or not a regular file)
package init file 'bench/datasets/configs/__init__.py' not found (or not a regular file)
package init file 'bench/datasets/tools/__init__.py' not found (or not a regular file)
package init file 'bench/torch/criteo/__init__.py' not found (or not a regular file)
package init file 'examples/tensorflow/__init__.py' not found (or not a regular file)
package init file 'examples/getting-started-movielens/__init__.py' not found (or not a regular file)
package init file 'examples/multi-gpu-toy-example/__init__.py' not found (or not a regular file)
package init file 'examples/tabular-data-rossmann/__init__.py' not found (or not a regular file)
package init file 'examples/advanced-ops-outbrain/__init__.py' not found (or not a regular file)
package init file 'examples/multi-gpu-movielens/__init__.py' not found (or not a regular file)
package init file 'examples/scaling-criteo/__init__.py' not found (or not a regular file)
package init file 'examples/winning-solution-recsys2020-twitter/__init__.py' not found (or not a regular file)
package init file 'examples/tensorflow/docker/__init__.py' not found (or not a regular file)
package init file 'examples/tensorflow/imgs/__init__.py' not found (or not a regular file)
package init file 'examples/getting-started-movielens/imgs/__init__.py' not found (or not a regular file)
package init file 'examples/scaling-criteo/imgs/__init__.py' not found (or not a regular file)
package init file 'tests/integration/__init__.py' not found (or not a regular file)
package init file 'tests/integration/common/__init__.py' not found (or not a regular file)
package init file 'tests/integration/common/parsers/__init__.py' not found (or not a regular file)
package init file 'tests/unit/loader/__init__.py' not found (or not a regular file)
package init file 'tests/unit/framework_utils/__init__.py' not found (or not a regular file)
package init file 'tests/unit/ops/__init__.py' not found (or not a regular file)
package init file 'tests/unit/workflow/__init__.py' not found (or not a regular file)
package init file 'conda/environments/__init__.py' not found (or not a regular file)
package init file 'conda/recipes/__init__.py' not found (or not a regular file)
reading manifest template 'MANIFEST.in'
warning: no files found matching '*.h' under directory 'cpp'
warning: no files found matching '*.cu' under directory 'cpp'
warning: no files found matching '*.cuh' under directory 'cpp'
adding license file 'LICENSE'
writing manifest file 'nvtabular.egg-info/SOURCES.txt'
running build_ext
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/usr/include/python3.8 -c flagcheck.cpp -o flagcheck.o -std=c++17
building 'nvtabular_cpp' extension
creating build
creating build/temp.linux-x86_64-3.8
creating build/temp.linux-x86_64-3.8/cpp
creating build/temp.linux-x86_64-3.8/cpp/nvtabular
creating build/temp.linux-x86_64-3.8/cpp/nvtabular/inference
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -DVERSION_INFO=0.11.0+76.ge64e90f6 -I./cpp/ -I/var/jenkins_home/.local/lib/python3.8/site-packages/pybind11/include -I/usr/include/python3.8 -c cpp/nvtabular/__init__.cc -o build/temp.linux-x86_64-3.8/cpp/nvtabular/__init__.o -std=c++17 -fvisibility=hidden -g0
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -DVERSION_INFO=0.11.0+76.ge64e90f6 -I./cpp/ -I/var/jenkins_home/.local/lib/python3.8/site-packages/pybind11/include -I/usr/include/python3.8 -c cpp/nvtabular/inference/__init__.cc -o build/temp.linux-x86_64-3.8/cpp/nvtabular/inference/__init__.o -std=c++17 -fvisibility=hidden -g0
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -DVERSION_INFO=0.11.0+76.ge64e90f6 -I./cpp/ -I/var/jenkins_home/.local/lib/python3.8/site-packages/pybind11/include -I/usr/include/python3.8 -c cpp/nvtabular/inference/categorify.cc -o build/temp.linux-x86_64-3.8/cpp/nvtabular/inference/categorify.o -std=c++17 -fvisibility=hidden -g0
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -DVERSION_INFO=0.11.0+76.ge64e90f6 -I./cpp/ -I/var/jenkins_home/.local/lib/python3.8/site-packages/pybind11/include -I/usr/include/python3.8 -c cpp/nvtabular/inference/fill.cc -o build/temp.linux-x86_64-3.8/cpp/nvtabular/inference/fill.o -std=c++17 -fvisibility=hidden -g0
creating build/lib.linux-x86_64-3.8
x86_64-linux-gnu-g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 build/temp.linux-x86_64-3.8/cpp/nvtabular/__init__.o build/temp.linux-x86_64-3.8/cpp/nvtabular/inference/__init__.o build/temp.linux-x86_64-3.8/cpp/nvtabular/inference/categorify.o build/temp.linux-x86_64-3.8/cpp/nvtabular/inference/fill.o -L/usr/lib -o build/lib.linux-x86_64-3.8/nvtabular_cpp.cpython-38-x86_64-linux-gnu.so
copying build/lib.linux-x86_64-3.8/nvtabular_cpp.cpython-38-x86_64-linux-gnu.so -> 
Generating nvtabular/inference/triton/model_config_pb2.py from nvtabular/inference/triton/model_config.proto
Creating /var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular.egg-link (link to .)
nvtabular 0.11.0+76.ge64e90f6 is already the active version in easy-install.pth

Installed /var/jenkins_home/workspace/nvtabular_tests/nvtabular

Running black --check

All done! ✨ 🍰 ✨

131 files would be left unchanged.

Running flake8

Running isort

Skipped 2 files

Running bandit

Running pylint

************* Module bench.datasets.tools.train_hugectr

bench/datasets/tools/train_hugectr.py:28:13: I1101: Module 'hugectr' has no 'solver_parser_helper' member, but source is unavailable. Consider adding this module to extension-pkg-allow-list if you want to perform analysis based on run-time introspection of living objects. (c-extension-no-member)

bench/datasets/tools/train_hugectr.py:41:16: I1101: Module 'hugectr' has no 'optimizer' member, but source is unavailable. Consider adding this module to extension-pkg-allow-list if you want to perform analysis based on run-time introspection of living objects. (c-extension-no-member)




Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)


Running flake8-nb

Building docs

make: Entering directory '/var/jenkins_home/workspace/nvtabular_tests/nvtabular/docs'

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.9) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

/usr/local/lib/python3.8/dist-packages/recommonmark/parser.py:75: UserWarning: Container node skipped: type=document

warn("Container node skipped: type={0}".format(mdnode.t))

/usr/local/lib/python3.8/dist-packages/recommonmark/parser.py:75: UserWarning: Container node skipped: type=document

warn("Container node skipped: type={0}".format(mdnode.t))

/usr/local/lib/python3.8/dist-packages/recommonmark/parser.py:75: UserWarning: Container node skipped: type=document

warn("Container node skipped: type={0}".format(mdnode.t))

make: Leaving directory '/var/jenkins_home/workspace/nvtabular_tests/nvtabular/docs'

============================= test session starts ==============================

platform linux -- Python 3.8.10, pytest-7.1.1, pluggy-1.0.0

rootdir: /var/jenkins_home/workspace/nvtabular_tests/nvtabular, configfile: pyproject.toml

plugins: xdist-2.5.0, forked-1.4.0, cov-3.0.0

collected 1405 items / 1 skipped


tests/unit/test_dask_nvt.py ............................................ [  3%]

........................................................................ [  8%]

[  8%]

tests/unit/test_notebooks.py ......                                      [  8%]

tests/unit/test_tf4rec.py .                                              [  8%]

tests/unit/test_tools.py ......................                          [ 10%]

tests/unit/test_triton_inference.py ................................     [ 12%]

tests/unit/framework_utils/test_tf_feature_columns.py .                  [ 12%]

tests/unit/framework_utils/test_tf_layers.py ........................... [ 14%]

...................................................                      [ 18%]

tests/unit/framework_utils/test_torch_layers.py .                        [ 18%]

tests/unit/loader/test_dataloader_backend.py ......                      [ 18%]

tests/unit/loader/test_tf_dataloader.py ................................ [ 20%]

........................................s..                              [ 24%]

tests/unit/loader/test_torch_dataloader.py ............................. [ 26%]

........................................................                 [ 30%]

tests/unit/ops/test_categorify.py ...................................... [ 32%]

........................................................................ [ 37%]

................................                                         [ 40%]

tests/unit/ops/test_column_similarity.py ........................        [ 41%]

tests/unit/ops/test_drop_low_cardinality.py ..                           [ 42%]

tests/unit/ops/test_fill.py ............................................ [ 45%]

........                                                                 [ 45%]

tests/unit/ops/test_groupyby.py .........                                [ 46%]

tests/unit/ops/test_hash_bucket.py .........................             [ 48%]

tests/unit/ops/test_join.py ............................................ [ 51%]

........................................................................ [ 56%]

..................................                                       [ 58%]

tests/unit/ops/test_lambda.py ..........                                 [ 59%]

tests/unit/ops/test_normalize.py ....................................... [ 62%]

..                                                                       [ 62%]

tests/unit/ops/test_ops.py ............................................. [ 65%]

....................                                                     [ 67%]

tests/unit/ops/test_ops_schema.py ...................................... [ 69%]

........................................................................ [ 74%]

........................................................................ [ 80%]

........................................................................ [ 85%]

.......................................                                  [ 87%]

tests/unit/ops/test_reduce_dtype_size.py ..                              [ 88%]

tests/unit/ops/test_target_encode.py .....................               [ 89%]

tests/unit/workflow/test_cpu_workflow.py ......                          [ 90%]

tests/unit/workflow/test_workflow.py ................................... [ 92%]

..........................................................               [ 96%]

tests/unit/workflow/test_workflow_chaining.py ...                        [ 96%]

tests/unit/workflow/test_workflow_node.py ...........                    [ 97%]

tests/unit/workflow/test_workflow_ops.py ...                             [ 97%]

tests/unit/workflow/test_workflow_schemas.py ........................... [ 99%]

...                                                                      [100%]


=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/fsspec/spec.py:92

../../../../../usr/local/lib/python3.8/dist-packages/fsspec/spec.py:92

/usr/local/lib/python3.8/dist-packages/fsspec/spec.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

if pa_version and LooseVersion(pa_version) < LooseVersion("2.0"):


../../../../../usr/lib/python3.8/site-packages/dask_cudf/core.py:32

/usr/lib/python3.8/site-packages/dask_cudf/core.py:32: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)


../../../../../usr/local/lib/python3.8/dist-packages/setuptools/_distutils/version.py:351: 34 warnings

/usr/local/lib/python3.8/dist-packages/setuptools/_distutils/version.py:351: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)


tests/unit/test_dask_nvt.py: 2 warnings

tests/unit/test_tf4rec.py: 1 warning

tests/unit/test_tools.py: 6 warnings

tests/unit/test_triton_inference.py: 8 warnings

tests/unit/loader/test_dataloader_backend.py: 6 warnings

tests/unit/loader/test_tf_dataloader.py: 142 warnings

tests/unit/loader/test_torch_dataloader.py: 93 warnings

tests/unit/ops/test_categorify.py: 64 warnings

tests/unit/ops/test_drop_low_cardinality.py: 2 warnings

tests/unit/ops/test_fill.py: 8 warnings

tests/unit/ops/test_hash_bucket.py: 4 warnings

tests/unit/ops/test_join.py: 88 warnings

tests/unit/ops/test_lambda.py: 3 warnings

tests/unit/ops/test_normalize.py: 9 warnings

tests/unit/ops/test_ops.py: 11 warnings

tests/unit/ops/test_ops_schema.py: 17 warnings

tests/unit/workflow/test_workflow.py: 34 warnings

tests/unit/workflow/test_workflow_chaining.py: 1 warning

tests/unit/workflow/test_workflow_node.py: 1 warning

tests/unit/workflow/test_workflow_schemas.py: 1 warning

/usr/lib/python3.8/site-packages/cudf/core/dataframe.py:1253: UserWarning: The deep parameter is ignored and is only included for pandas compatibility.

warnings.warn(


tests/unit/test_dask_nvt.py: 12 warnings

/core/merlin/io/dataset.py:863: UserWarning: Only created 2 files did not have enough partitions to create 8 files.

warnings.warn(


tests/unit/test_dask_nvt.py::test_merlin_core_execution_managers

/core/merlin/core/utils.py:433: UserWarning: Existing Dask-client object detected in the current context. New cuda cluster will not be deployed. Set force_new to True to ignore running clusters.

warnings.warn(


tests/unit/test_tools.py::test_inspect_datagen[uniform-parquet]

tests/unit/test_tools.py::test_inspect_datagen[uniform-parquet]

tests/unit/ops/test_ops.py::test_data_stats[True-parquet]

tests/unit/ops/test_ops.py::test_data_stats[False-parquet]

/usr/lib/python3.8/site-packages/cudf/core/series.py:923: FutureWarning: Series.set_index is deprecated and will be removed in the future

warnings.warn(


tests/unit/loader/test_tf_dataloader.py: 2 warnings

tests/unit/loader/test_torch_dataloader.py: 12 warnings

tests/unit/workflow/test_workflow.py: 9 warnings

/core/merlin/io/dataset.py:863: UserWarning: Only created 1 files did not have enough partitions to create 2 files.

warnings.warn(


tests/unit/loader/test_torch_dataloader.py::test_horovod_multigpu

tests/unit/loader/test_torch_dataloader.py::test_distributed_multigpu

/core/merlin/io/dataset.py:863: UserWarning: Only created 1 files did not have enough partitions to create 5 files.

warnings.warn(


tests/unit/ops/test_categorify.py::test_categorify_max_size[6]

tests/unit/ops/test_categorify.py::test_categorify_max_size[max_emb_size1]

/usr/lib/python3.8/site-packages/cudf/core/frame.py:3077: FutureWarning: keep_index is deprecated and will be removed in the future.

warnings.warn(


tests/unit/ops/test_fill.py::test_fill_missing[True-True-parquet]

tests/unit/ops/test_fill.py::test_fill_missing[True-False-parquet]

tests/unit/ops/test_ops.py::test_filter[parquet-0.1-True]

/var/jenkins_home/.local/lib/python3.8/site-packages/pandas/core/indexing.py:1732: SettingWithCopyWarning:

A value is trying to be set on a copy of a slice from a DataFrame


See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy

self._setitem_single_block(indexer, value, name)


tests/unit/ops/test_ops.py::test_difference_lag[False]

/usr/lib/python3.8/site-packages/cudf/core/dataframe.py:3041: FutureWarning: The as_gpu_matrix method will be removed in a future cuDF release. Consider using to_cupy instead.

warnings.warn(


tests/unit/workflow/test_cpu_workflow.py: 6 warnings

tests/unit/workflow/test_workflow.py: 12 warnings

/core/merlin/io/dataset.py:863: UserWarning: Only created 1 files did not have enough partitions to create 10 files.

warnings.warn(


tests/unit/workflow/test_workflow.py: 48 warnings

/core/merlin/io/dataset.py:863: UserWarning: Only created 2 files did not have enough partitions to create 20 files.

warnings.warn(


tests/unit/workflow/test_workflow.py::test_parquet_output[True-Shuffle.PER_WORKER]

tests/unit/workflow/test_workflow.py::test_parquet_output[True-Shuffle.PER_PARTITION]

tests/unit/workflow/test_workflow.py::test_parquet_output[True-None]

tests/unit/workflow/test_workflow.py::test_workflow_apply[True-True-Shuffle.PER_WORKER]

tests/unit/workflow/test_workflow.py::test_workflow_apply[True-True-Shuffle.PER_PARTITION]

tests/unit/workflow/test_workflow.py::test_workflow_apply[True-True-None]

tests/unit/workflow/test_workflow.py::test_workflow_apply[False-True-Shuffle.PER_WORKER]

tests/unit/workflow/test_workflow.py::test_workflow_apply[False-True-Shuffle.PER_PARTITION]

tests/unit/workflow/test_workflow.py::test_workflow_apply[False-True-None]

/core/merlin/io/dataset.py:863: UserWarning: Only created 2 files did not have enough partitions to create 4 files.

warnings.warn(


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                                           Stmts   Miss Branch BrPart  Cover   Missing


examples/multi-gpu-movielens/torch_trainer.py                     65      0      6      1    99%   32->36

examples/multi-gpu-movielens/torch_trainer_dist.py                63      0      2      0   100%

nvtabular/init.py                                             22      2      0      0    91%   44-45

nvtabular/dispatch.py                                              3      3      0      0     0%   18-23

nvtabular/framework_utils/init.py                              0      0      0      0   100%

nvtabular/framework_utils/tensorflow/init.py                   1      0      0      0   100%

nvtabular/framework_utils/tensorflow/feature_column_utils.py     134     78     90     15    39%   30, 99, 103, 114-130, 140, 143-158, 162, 166-167, 173-198, 207-217, 220-227, 229->233, 234, 239-279, 282

nvtabular/framework_utils/tensorflow/layers/init.py            4      0      0      0   100%

nvtabular/framework_utils/tensorflow/layers/embedding.py         153     13     89      6    90%   60, 68->49, 122, 179, 231-239, 242, 335->343, 357->360, 363-364, 367

nvtabular/framework_utils/tensorflow/layers/interaction.py        47     25     22      1    45%   49, 74-103, 106-110, 113

nvtabular/framework_utils/tensorflow/layers/outer_product.py      30     24     12      0    19%   37-38, 41-60, 71-84, 87

nvtabular/framework_utils/tensorflow/tfrecords_to_parquet.py      58     58     30      0     0%   16-111

nvtabular/framework_utils/torch/init.py                        0      0      0      0   100%

nvtabular/framework_utils/torch/layers/init.py                 2      0      0      0   100%

nvtabular/framework_utils/torch/layers/embeddings.py              32      2     18      2    92%   50, 91

nvtabular/framework_utils/torch/models.py                         45      2     30     10    84%   55->57, 57->61, 61->63, 66->68, 87->89, 89->93, 93->96, 96->100, 103, 108

nvtabular/framework_utils/torch/utils.py                          75     19     34      9    69%   22, 25-33, 51->53, 53->55, 64, 71->76, 75, 107, 118-120, 124-127, 132->115

nvtabular/graph.py                                                 3      3      0      0     0%   18-23

nvtabular/inference/init.py                                    0      0      0      0   100%

nvtabular/inference/triton/init.py                            36     12     14      1    58%   42-49, 68, 72, 76-82

nvtabular/inference/triton/benchmarking_tools.py                  52     52     10      0     0%   2-103

nvtabular/inference/triton/data_conversions.py                    87      3     58      4    95%   32-33, 84

nvtabular/inference/triton/ensemble.py                           266    200     82      3    25%   73-112, 157-196, 240-288, 292-323, 381-389, 418-434, 477-520, 548-588, 594-610, 614-681, 711, 733, 739-758, 764-788, 793-796

nvtabular/inference/triton/model/init.py                       0      0      0      0   100%

nvtabular/inference/triton/model/model_pt.py                     101    101     42      0     0%   27-220

nvtabular/inference/triton/model_config_pb2.py                   299      0      2      0   100%

nvtabular/inference/triton/workflow_model.py                      55     55     24      0     0%   27-128

nvtabular/inference/workflow/init.py                           0      0      0      0   100%

nvtabular/inference/workflow/base.py                             113    113     62      0     0%   27-209

nvtabular/inference/workflow/hugectr.py                           37     37     16      0     0%   27-87

nvtabular/inference/workflow/pytorch.py                           10     10      6      0     0%   27-46

nvtabular/inference/workflow/tensorflow.py                        32     32     10      0     0%   26-68

nvtabular/io.py                                                    3      3      0      0     0%   18-23

nvtabular/loader/init.py                                       0      0      0      0   100%

nvtabular/loader/backend.py                                      371     19    154     15    94%   27-28, 112, 116->108, 125, 158-159, 299->301, 311-315, 362-363, 402->406, 403->402, 478, 482-483, 512, 588-589, 598, 624, 632

nvtabular/loader/tensorflow.py                                   179     26     60     10    85%   38-39, 74, 85-89, 101, 105, 115, 124, 329, 357, 368, 383-385, 414-416, 426-434, 437-440

nvtabular/loader/tf_utils.py                                      57     27     22      6    46%   31->34, 34->36, 41->43, 45, 46->67, 52-53, 61-63, 69-73, 88-93, 103-116

nvtabular/loader/torch.py                                         87     16     26      3    78%   28-30, 33-39, 114, 158-159, 164, 202, 205

nvtabular/ops/init.py                                         26      0      0      0   100%

nvtabular/ops/add_metadata.py                                     34      0     14      0   100%

nvtabular/ops/bucketize.py                                        40      9     20      3    73%   52-54, 58->exit, 61-64, 83-86

nvtabular/ops/categorify.py                                      657     71    350     50    86%   251, 253, 271, 275, 283, 291, 293, 320, 341-342, 389->393, 397-404, 450, 458, 481-482, 559-564, 635, 731, 748, 793, 871-872, 887-891, 892->856, 910, 918, 925->exit, 949, 952->955, 1004->1002, 1066-1068, 1071, 1073->1075, 1092->1096, 1098->1051, 1104-1107, 1119, 1123, 1127, 1134, 1139-1142, 1220, 1222, 1292->1315, 1298->1315, 1316-1321, 1366, 1379->1382, 1386->1391, 1390, 1396, 1399, 1407-1417

nvtabular/ops/clip.py                                             18      2      8      3    81%   44, 52->54, 55

nvtabular/ops/column_similarity.py                               121     26     38      5    74%   19-20, 29-30, 81->exit, 111, 206-207, 216-218, 226-242, 259->262, 263, 273

nvtabular/ops/data_stats.py                                       56      1     24      3    95%   107->109, 111, 113->103

nvtabular/ops/difference_lag.py                                   43      0     14      1    98%   73->75

nvtabular/ops/drop_low_cardinality.py                             23      0      8      1    97%   41->46

nvtabular/ops/dropna.py                                            9      0      2      0   100%

nvtabular/ops/fill.py                                             76      5     30      1    92%   63-67, 109

nvtabular/ops/filter.py                                           20      1      8      1    93%   49

nvtabular/ops/groupby.py                                         130      4     84      5    96%   72, 83, 93->95, 131, 221

nvtabular/ops/hash_bucket.py                                      43      1     22      2    95%   79, 118->124

nvtabular/ops/hashed_cross.py                                     38      3     17      3    89%   52, 64, 92

nvtabular/ops/join_external.py                                    96      8     34      8    88%   20-21, 114, 116, 118, 137->141, 150->152, 205-206, 216->227, 221

nvtabular/ops/join_groupby.py                                    128      5     57      6    94%   113, 120, 129, 136->135, 178->175, 181->175, 260-261

nvtabular/ops/lambdaop.py                                         62     13     22      3    71%   60, 64, 82, 94-109

nvtabular/ops/list_slice.py                                       89     29     42      0    64%   21-22, 146-160, 168-190

nvtabular/ops/logop.py                                            21      0      6      0   100%

nvtabular/ops/moments.py                                          69      0     24      0   100%

nvtabular/ops/normalize.py                                        93      4     22      1    94%   89, 139-140, 167

nvtabular/ops/operator.py                                         11      1      2      0    92%   52

nvtabular/ops/reduce_dtype_size.py                                49      0     20      2    97%   68->77, 74->77

nvtabular/ops/rename.py                                           29      3     14      3    86%   46, 71-73

nvtabular/ops/stat_operator.py                                     8      0      2      0   100%

nvtabular/ops/target_encoding.py                                 182      9     76      5    93%   168->172, 176->185, 274, 283-284, 297-303, 396->399

nvtabular/ops/value_counts.py                                     34      0      6      1    98%   40->38

nvtabular/tools/init.py                                        0      0      0      0   100%

nvtabular/tools/data_gen.py                                      271     25     90      7    91%   26-27, 31-32, 129-132, 142-146, 148, 170-171, 322, 332, 358, 361-370

nvtabular/tools/dataset_inspector.py                              52      8     24      2    79%   33-40, 51

nvtabular/tools/inspector_script.py                               46     46      0      0     0%   17-168

nvtabular/utils.py                                                 3      3      0      0     0%   18-23

nvtabular/worker.py                                                3      3      0      0     0%   18-23

nvtabular/workflow/init.py                                     2      0      0      0   100%

nvtabular/workflow/node.py                                         7      0      4      0   100%

nvtabular/workflow/workflow.py                                   219     17     94     12    91%   28-29, 52, 85, 206, 212->226, 239-241, 373, 388-389, 431, 508, 538, 548-550, 563


TOTAL                                                           5330   1232   2099    214    75%

Coverage XML written to file coverage.xml


Required test coverage of 70% reached. Total coverage: 74.71%

=========================== short test summary info ============================

SKIPPED [1] ../../../../../usr/lib/python3.8/site-packages/dask_cudf/io/tests/test_s3.py:16: could not import 's3fs': cannot import name 'ParamSpec' from 'typing_extensions' (/var/jenkins_home/.local/lib/python3.8/site-packages/typing_extensions.py)

SKIPPED [1] tests/unit/loader/test_tf_dataloader.py:529: not working correctly in ci environment

========== 1404 passed, 2 skipped, 661 warnings in 1422.22s (0:23:42) ==========

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/NVTabular/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[nvtabular_tests] $ /bin/bash /tmp/jenkins3089403014118660112.sh

@karlhigley karlhigley added this to the Merlin 22.04 milestone Apr 5, 2022
@karlhigley karlhigley added the bug Something isn't working label Apr 5, 2022
@github-actions
Copy link

github-actions bot commented Apr 5, 2022

Documentation preview

https://nvidia-merlin.github.io/NVTabular/review/pr-1500

@nvidia-merlin-bot
Copy link
Contributor

Click to view CI Results
GitHub pull request #1500 of commit 22660edfe7692cd1c4f56674b023f9957c01d2f1, no merge conflicts.
Running as SYSTEM
Setting status of 22660edfe7692cd1c4f56674b023f9957c01d2f1 to PENDING with url http://10.20.13.93:8080/job/nvtabular_tests/4422/ and message: 'Pending'
Using context: Jenkins Unit Test Run
Building on master in workspace /var/jenkins_home/workspace/nvtabular_tests
using credential nvidia-merlin-bot
Cloning the remote Git repository
Cloning repository https://github.com/NVIDIA-Merlin/NVTabular.git
 > git init /var/jenkins_home/workspace/nvtabular_tests/nvtabular # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/NVTabular.git
 > git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/NVTabular.git +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/NVTabular.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/NVTabular.git # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/NVTabular.git
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/NVTabular.git +refs/pull/1500/*:refs/remotes/origin/pr/1500/* # timeout=10
 > git rev-parse 22660edfe7692cd1c4f56674b023f9957c01d2f1^{commit} # timeout=10
Checking out Revision 22660edfe7692cd1c4f56674b023f9957c01d2f1 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 22660edfe7692cd1c4f56674b023f9957c01d2f1 # timeout=10
Commit message: "Merge branch 'main' into criteo_update"
 > git rev-list --no-walk 267aa739b867ba6003070f6298bba97ba2a59623 # timeout=10
First time build. Skipping changelog.
[nvtabular_tests] $ /bin/bash /tmp/jenkins3197606660815353344.sh
rm: cannot remove '/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular.egg-link': No such file or directory
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: setuptools in /usr/local/lib/python3.8/dist-packages (62.0.0)
Installing NVTabular
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: pip in /var/jenkins_home/.local/lib/python3.8/site-packages (22.0.4)
Requirement already satisfied: setuptools in /usr/local/lib/python3.8/dist-packages (62.0.0)
Requirement already satisfied: wheel in /var/jenkins_home/.local/lib/python3.8/site-packages (0.37.1)
Requirement already satisfied: pybind11 in /var/jenkins_home/.local/lib/python3.8/site-packages (2.9.2)
Requirement already satisfied: numpy==1.20.3 in /var/jenkins_home/.local/lib/python3.8/site-packages (1.20.3)
Found existing installation: nvtabular 0.11.0+44.g50eb7ef1
Can't uninstall 'nvtabular'. No files were found to uninstall.
Found existing installation: merlin-core 0.1.1+21.g32e4695
Can't uninstall 'merlin-core'. No files were found to uninstall.
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-install-uibt96oj/merlin-core_8598a2bed19e4e7d892ce0c774a0b9d5
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-install-uibt96oj/merlin-core_8598a2bed19e4e7d892ce0c774a0b9d5
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit b6508e54da2edb67f6b5fccb1a529d6422cee175
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: distributed==2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (2021.11.2)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (21.3)
Requirement already satisfied: dask==2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (2021.11.2)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (4.63.1)
Requirement already satisfied: numba>=0.53.1 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (0.55.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (1.7.0)
Requirement already satisfied: pyarrow>=1.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (5.0.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (3.20.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (1.2.5)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (1.3.5)
Requirement already satisfied: pyyaml in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask==2021.11.2->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (5.4.1)
Requirement already satisfied: fsspec>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from dask==2021.11.2->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (2021.11.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask==2021.11.2->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (2.0.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask==2021.11.2->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask==2021.11.2->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (0.11.2)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed==2021.11.2->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (1.0.3)
Requirement already satisfied: setuptools in /usr/local/lib/python3.8/dist-packages (from distributed==2021.11.2->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (62.0.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed==2021.11.2->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (2.0.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed==2021.11.2->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (6.1)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed==2021.11.2->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (8.0.4)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed==2021.11.2->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (5.8.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed==2021.11.2->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (1.7.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed==2021.11.2->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (2.4.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed==2021.11.2->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (3.0.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (0.4.2)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.53.1->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in /usr/local/lib/python3.8/dist-packages (from numba>=0.53.1->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (0.38.0)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (3.0.7)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (2022.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (1.56.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /var/jenkins_home/.local/lib/python3.8/site-packages/absl_py-0.12.0-py3.8.egg (from tensorflow-metadata>=1.2.0->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (0.12.0)
Requirement already satisfied: six in /var/jenkins_home/.local/lib/python3.8/site-packages (from absl-py<2.0.0,>=0.9->tensorflow-metadata>=1.2.0->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (1.15.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask==2021.11.2->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (0.2.1)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed==2021.11.2->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed==2021.11.2->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (2.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core@ git+https://github.com/NVIDIA-Merlin/core.git) (6.0.1)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.1.1+31.gb6508e5-py3-none-any.whl size=131948 sha256=74fdb7e614ed4aee2c5a40088082a60aab8b2320296e3212312ca6795a79c617
  Stored in directory: /tmp/pip-ephem-wheel-cache-8oqytzmi/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
Successfully installed merlin-core-0.1.1+31.gb6508e5
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: natsort==8.1.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (8.1.0)
running develop
running egg_info
creating nvtabular.egg-info
writing nvtabular.egg-info/PKG-INFO
writing dependency_links to nvtabular.egg-info/dependency_links.txt
writing requirements to nvtabular.egg-info/requires.txt
writing top-level names to nvtabular.egg-info/top_level.txt
writing manifest file 'nvtabular.egg-info/SOURCES.txt'
/usr/local/lib/python3.8/dist-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
/usr/local/lib/python3.8/dist-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
package init file 'ci/__init__.py' not found (or not a regular file)
package init file 'images/__init__.py' not found (or not a regular file)
package init file 'docs/__init__.py' not found (or not a regular file)
package init file 'cpp/__init__.py' not found (or not a regular file)
package init file 'bench/__init__.py' not found (or not a regular file)
package init file 'merlin/__init__.py' not found (or not a regular file)
package init file 'examples/__init__.py' not found (or not a regular file)
package init file 'conda/__init__.py' not found (or not a regular file)
package init file 'docs/source/__init__.py' not found (or not a regular file)
package init file 'docs/source/_templates/__init__.py' not found (or not a regular file)
package init file 'docs/source/images/__init__.py' not found (or not a regular file)
package init file 'docs/source/training/__init__.py' not found (or not a regular file)
package init file 'docs/source/api/__init__.py' not found (or not a regular file)
package init file 'docs/source/examples/__init__.py' not found (or not a regular file)
package init file 'docs/source/resources/__init__.py' not found (or not a regular file)
package init file 'docs/source/api/ops/__init__.py' not found (or not a regular file)
package init file 'docs/source/api/workflow/__init__.py' not found (or not a regular file)
package init file 'docs/source/examples/getting-started-movielens/__init__.py' not found (or not a regular file)
package init file 'docs/source/examples/tabular-data-rossmann/__init__.py' not found (or not a regular file)
package init file 'docs/source/examples/advanced-ops-outbrain/__init__.py' not found (or not a regular file)
package init file 'docs/source/examples/scaling-criteo/__init__.py' not found (or not a regular file)
package init file 'docs/source/examples/winning-solution-recsys2020-twitter/__init__.py' not found (or not a regular file)
package init file 'docs/source/examples/scaling-criteo/imgs/__init__.py' not found (or not a regular file)
package init file 'cpp/nvtabular/__init__.py' not found (or not a regular file)
package init file 'cpp/nvtabular/inference/__init__.py' not found (or not a regular file)
package init file 'bench/datasets/__init__.py' not found (or not a regular file)
package init file 'bench/torch/__init__.py' not found (or not a regular file)
package init file 'bench/examples/__init__.py' not found (or not a regular file)
package init file 'bench/datasets/configs/__init__.py' not found (or not a regular file)
package init file 'bench/datasets/tools/__init__.py' not found (or not a regular file)
package init file 'bench/torch/criteo/__init__.py' not found (or not a regular file)
package init file 'examples/tensorflow/__init__.py' not found (or not a regular file)
package init file 'examples/getting-started-movielens/__init__.py' not found (or not a regular file)
package init file 'examples/multi-gpu-toy-example/__init__.py' not found (or not a regular file)
package init file 'examples/tabular-data-rossmann/__init__.py' not found (or not a regular file)
package init file 'examples/advanced-ops-outbrain/__init__.py' not found (or not a regular file)
package init file 'examples/multi-gpu-movielens/__init__.py' not found (or not a regular file)
package init file 'examples/scaling-criteo/__init__.py' not found (or not a regular file)
package init file 'examples/winning-solution-recsys2020-twitter/__init__.py' not found (or not a regular file)
package init file 'examples/tensorflow/docker/__init__.py' not found (or not a regular file)
package init file 'examples/tensorflow/imgs/__init__.py' not found (or not a regular file)
package init file 'examples/getting-started-movielens/imgs/__init__.py' not found (or not a regular file)
package init file 'examples/scaling-criteo/imgs/__init__.py' not found (or not a regular file)
package init file 'tests/integration/__init__.py' not found (or not a regular file)
package init file 'tests/integration/common/__init__.py' not found (or not a regular file)
package init file 'tests/integration/common/parsers/__init__.py' not found (or not a regular file)
package init file 'tests/unit/loader/__init__.py' not found (or not a regular file)
package init file 'tests/unit/framework_utils/__init__.py' not found (or not a regular file)
package init file 'tests/unit/ops/__init__.py' not found (or not a regular file)
package init file 'tests/unit/workflow/__init__.py' not found (or not a regular file)
package init file 'conda/environments/__init__.py' not found (or not a regular file)
package init file 'conda/recipes/__init__.py' not found (or not a regular file)
reading manifest template 'MANIFEST.in'
warning: no files found matching '*.h' under directory 'cpp'
warning: no files found matching '*.cu' under directory 'cpp'
warning: no files found matching '*.cuh' under directory 'cpp'
adding license file 'LICENSE'
writing manifest file 'nvtabular.egg-info/SOURCES.txt'
running build_ext
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/usr/include/python3.8 -c flagcheck.cpp -o flagcheck.o -std=c++17
building 'nvtabular_cpp' extension
creating build
creating build/temp.linux-x86_64-3.8
creating build/temp.linux-x86_64-3.8/cpp
creating build/temp.linux-x86_64-3.8/cpp/nvtabular
creating build/temp.linux-x86_64-3.8/cpp/nvtabular/inference
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -DVERSION_INFO=0.11.0+81.g22660edf -I./cpp/ -I/var/jenkins_home/.local/lib/python3.8/site-packages/pybind11/include -I/usr/include/python3.8 -c cpp/nvtabular/__init__.cc -o build/temp.linux-x86_64-3.8/cpp/nvtabular/__init__.o -std=c++17 -fvisibility=hidden -g0
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -DVERSION_INFO=0.11.0+81.g22660edf -I./cpp/ -I/var/jenkins_home/.local/lib/python3.8/site-packages/pybind11/include -I/usr/include/python3.8 -c cpp/nvtabular/inference/__init__.cc -o build/temp.linux-x86_64-3.8/cpp/nvtabular/inference/__init__.o -std=c++17 -fvisibility=hidden -g0
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -DVERSION_INFO=0.11.0+81.g22660edf -I./cpp/ -I/var/jenkins_home/.local/lib/python3.8/site-packages/pybind11/include -I/usr/include/python3.8 -c cpp/nvtabular/inference/categorify.cc -o build/temp.linux-x86_64-3.8/cpp/nvtabular/inference/categorify.o -std=c++17 -fvisibility=hidden -g0
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -DVERSION_INFO=0.11.0+81.g22660edf -I./cpp/ -I/var/jenkins_home/.local/lib/python3.8/site-packages/pybind11/include -I/usr/include/python3.8 -c cpp/nvtabular/inference/fill.cc -o build/temp.linux-x86_64-3.8/cpp/nvtabular/inference/fill.o -std=c++17 -fvisibility=hidden -g0
creating build/lib.linux-x86_64-3.8
x86_64-linux-gnu-g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 build/temp.linux-x86_64-3.8/cpp/nvtabular/__init__.o build/temp.linux-x86_64-3.8/cpp/nvtabular/inference/__init__.o build/temp.linux-x86_64-3.8/cpp/nvtabular/inference/categorify.o build/temp.linux-x86_64-3.8/cpp/nvtabular/inference/fill.o -L/usr/lib -o build/lib.linux-x86_64-3.8/nvtabular_cpp.cpython-38-x86_64-linux-gnu.so
copying build/lib.linux-x86_64-3.8/nvtabular_cpp.cpython-38-x86_64-linux-gnu.so -> 
Generating nvtabular/inference/triton/model_config_pb2.py from nvtabular/inference/triton/model_config.proto
Creating /var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular.egg-link (link to .)
nvtabular 0.11.0+81.g22660edf is already the active version in easy-install.pth

Installed /var/jenkins_home/workspace/nvtabular_tests/nvtabular

Running black --check

All done! ✨ 🍰 ✨

132 files would be left unchanged.

Running flake8

Running isort

Skipped 2 files

Running bandit

Running pylint

************* Module bench.datasets.tools.train_hugectr

bench/datasets/tools/train_hugectr.py:28:13: I1101: Module 'hugectr' has no 'solver_parser_helper' member, but source is unavailable. Consider adding this module to extension-pkg-allow-list if you want to perform analysis based on run-time introspection of living objects. (c-extension-no-member)

bench/datasets/tools/train_hugectr.py:41:16: I1101: Module 'hugectr' has no 'optimizer' member, but source is unavailable. Consider adding this module to extension-pkg-allow-list if you want to perform analysis based on run-time introspection of living objects. (c-extension-no-member)




Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)


Running flake8-nb

Building docs

make: Entering directory '/var/jenkins_home/workspace/nvtabular_tests/nvtabular/docs'

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.9) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

/usr/local/lib/python3.8/dist-packages/recommonmark/parser.py:75: UserWarning: Container node skipped: type=document

warn("Container node skipped: type={0}".format(mdnode.t))

/usr/local/lib/python3.8/dist-packages/recommonmark/parser.py:75: UserWarning: Container node skipped: type=document

warn("Container node skipped: type={0}".format(mdnode.t))

/usr/local/lib/python3.8/dist-packages/recommonmark/parser.py:75: UserWarning: Container node skipped: type=document

warn("Container node skipped: type={0}".format(mdnode.t))

make: Leaving directory '/var/jenkins_home/workspace/nvtabular_tests/nvtabular/docs'

============================= test session starts ==============================

platform linux -- Python 3.8.10, pytest-7.1.1, pluggy-1.0.0

rootdir: /var/jenkins_home/workspace/nvtabular_tests/nvtabular, configfile: pyproject.toml

plugins: xdist-2.5.0, forked-1.4.0, cov-3.0.0

collected 1405 items / 1 skipped


tests/unit/test_dask_nvt.py ............................................ [  3%]

........................................................................ [  8%]

[  8%]

tests/unit/test_notebooks.py ......                                      [  8%]

tests/unit/test_tf4rec.py .                                              [  8%]

tests/unit/test_tools.py ......................                          [ 10%]

tests/unit/test_triton_inference.py ................................     [ 12%]

tests/unit/framework_utils/test_tf_feature_columns.py .                  [ 12%]

tests/unit/framework_utils/test_tf_layers.py ........................... [ 14%]

...................................................                      [ 18%]

tests/unit/framework_utils/test_torch_layers.py .                        [ 18%]

tests/unit/loader/test_dataloader_backend.py ......                      [ 18%]

tests/unit/loader/test_tf_dataloader.py ................................ [ 20%]

........................................s..                              [ 24%]

tests/unit/loader/test_torch_dataloader.py ............................. [ 26%]

........................................................                 [ 30%]

tests/unit/ops/test_categorify.py ...................................... [ 32%]

........................................................................ [ 37%]

................................                                         [ 40%]

tests/unit/ops/test_column_similarity.py ........................        [ 41%]

tests/unit/ops/test_drop_low_cardinality.py ..                           [ 42%]

tests/unit/ops/test_fill.py ............................................ [ 45%]

........                                                                 [ 45%]

tests/unit/ops/test_groupyby.py .........                                [ 46%]

tests/unit/ops/test_hash_bucket.py .........................             [ 48%]

tests/unit/ops/test_join.py ............................................ [ 51%]

........................................................................ [ 56%]

..................................                                       [ 58%]

tests/unit/ops/test_lambda.py ..........                                 [ 59%]

tests/unit/ops/test_normalize.py ....................................... [ 62%]

..                                                                       [ 62%]

tests/unit/ops/test_ops.py ............................................. [ 65%]

....................                                                     [ 67%]

tests/unit/ops/test_ops_schema.py ...................................... [ 69%]

........................................................................ [ 74%]

........................................................................ [ 80%]

........................................................................ [ 85%]

.......................................                                  [ 87%]

tests/unit/ops/test_reduce_dtype_size.py ..                              [ 88%]

tests/unit/ops/test_target_encode.py .....................               [ 89%]

tests/unit/workflow/test_cpu_workflow.py ......                          [ 90%]

tests/unit/workflow/test_workflow.py ................................... [ 92%]

..........................................................               [ 96%]

tests/unit/workflow/test_workflow_chaining.py ...                        [ 96%]

tests/unit/workflow/test_workflow_node.py ...........                    [ 97%]

tests/unit/workflow/test_workflow_ops.py ...                             [ 97%]

tests/unit/workflow/test_workflow_schemas.py ........................... [ 99%]

...                                                                      [100%]


=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/fsspec/spec.py:92

../../../../../usr/local/lib/python3.8/dist-packages/fsspec/spec.py:92

/usr/local/lib/python3.8/dist-packages/fsspec/spec.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

if pa_version and LooseVersion(pa_version) < LooseVersion("2.0"):


../../../../../usr/lib/python3.8/site-packages/dask_cudf/core.py:32

/usr/lib/python3.8/site-packages/dask_cudf/core.py:32: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)


../../../../../usr/local/lib/python3.8/dist-packages/setuptools/_distutils/version.py:351: 34 warnings

/usr/local/lib/python3.8/dist-packages/setuptools/_distutils/version.py:351: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)


nvtabular/loader/init.py:19

/var/jenkins_home/workspace/nvtabular_tests/nvtabular/nvtabular/loader/init.py:19: DeprecationWarning: The nvtabular.loader module has moved to merlin.models.loader. Support for importing from nvtabular.loader is deprecated, and will be removed in a future version. Please update your imports to refer to merlin.models.loader.

warnings.warn(


tests/unit/test_dask_nvt.py: 2 warnings

tests/unit/test_tf4rec.py: 1 warning

tests/unit/test_tools.py: 6 warnings

tests/unit/test_triton_inference.py: 8 warnings

tests/unit/loader/test_dataloader_backend.py: 6 warnings

tests/unit/loader/test_tf_dataloader.py: 142 warnings

tests/unit/loader/test_torch_dataloader.py: 93 warnings

tests/unit/ops/test_categorify.py: 64 warnings

tests/unit/ops/test_drop_low_cardinality.py: 2 warnings

tests/unit/ops/test_fill.py: 8 warnings

tests/unit/ops/test_hash_bucket.py: 4 warnings

tests/unit/ops/test_join.py: 88 warnings

tests/unit/ops/test_lambda.py: 3 warnings

tests/unit/ops/test_normalize.py: 9 warnings

tests/unit/ops/test_ops.py: 11 warnings

tests/unit/ops/test_ops_schema.py: 17 warnings

tests/unit/workflow/test_workflow.py: 34 warnings

tests/unit/workflow/test_workflow_chaining.py: 1 warning

tests/unit/workflow/test_workflow_node.py: 1 warning

tests/unit/workflow/test_workflow_schemas.py: 1 warning

/usr/lib/python3.8/site-packages/cudf/core/dataframe.py:1253: UserWarning: The deep parameter is ignored and is only included for pandas compatibility.

warnings.warn(


tests/unit/test_dask_nvt.py: 12 warnings

/core/merlin/io/dataset.py:863: UserWarning: Only created 2 files did not have enough partitions to create 8 files.

warnings.warn(


tests/unit/test_dask_nvt.py::test_merlin_core_execution_managers

/core/merlin/core/utils.py:433: UserWarning: Existing Dask-client object detected in the current context. New cuda cluster will not be deployed. Set force_new to True to ignore running clusters.

warnings.warn(


tests/unit/test_tools.py::test_inspect_datagen[uniform-parquet]

tests/unit/test_tools.py::test_inspect_datagen[uniform-parquet]

tests/unit/ops/test_ops.py::test_data_stats[True-parquet]

tests/unit/ops/test_ops.py::test_data_stats[False-parquet]

/usr/lib/python3.8/site-packages/cudf/core/series.py:923: FutureWarning: Series.set_index is deprecated and will be removed in the future

warnings.warn(


tests/unit/loader/test_tf_dataloader.py: 2 warnings

tests/unit/loader/test_torch_dataloader.py: 12 warnings

tests/unit/workflow/test_workflow.py: 9 warnings

/core/merlin/io/dataset.py:863: UserWarning: Only created 1 files did not have enough partitions to create 2 files.

warnings.warn(


tests/unit/loader/test_torch_dataloader.py::test_horovod_multigpu

tests/unit/loader/test_torch_dataloader.py::test_distributed_multigpu

/core/merlin/io/dataset.py:863: UserWarning: Only created 1 files did not have enough partitions to create 5 files.

warnings.warn(


tests/unit/ops/test_categorify.py::test_categorify_max_size[6]

tests/unit/ops/test_categorify.py::test_categorify_max_size[max_emb_size1]

/usr/lib/python3.8/site-packages/cudf/core/frame.py:3077: FutureWarning: keep_index is deprecated and will be removed in the future.

warnings.warn(


tests/unit/ops/test_fill.py::test_fill_missing[True-True-parquet]

tests/unit/ops/test_fill.py::test_fill_missing[True-False-parquet]

tests/unit/ops/test_ops.py::test_filter[parquet-0.1-True]

/var/jenkins_home/.local/lib/python3.8/site-packages/pandas/core/indexing.py:1732: SettingWithCopyWarning:

A value is trying to be set on a copy of a slice from a DataFrame


See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy

self._setitem_single_block(indexer, value, name)


tests/unit/ops/test_ops.py::test_difference_lag[False]

/usr/lib/python3.8/site-packages/cudf/core/dataframe.py:3041: FutureWarning: The as_gpu_matrix method will be removed in a future cuDF release. Consider using to_cupy instead.

warnings.warn(


tests/unit/workflow/test_cpu_workflow.py: 6 warnings

tests/unit/workflow/test_workflow.py: 12 warnings

/core/merlin/io/dataset.py:863: UserWarning: Only created 1 files did not have enough partitions to create 10 files.

warnings.warn(


tests/unit/workflow/test_workflow.py: 48 warnings

/core/merlin/io/dataset.py:863: UserWarning: Only created 2 files did not have enough partitions to create 20 files.

warnings.warn(


tests/unit/workflow/test_workflow.py::test_parquet_output[True-Shuffle.PER_WORKER]

tests/unit/workflow/test_workflow.py::test_parquet_output[True-Shuffle.PER_PARTITION]

tests/unit/workflow/test_workflow.py::test_parquet_output[True-None]

tests/unit/workflow/test_workflow.py::test_workflow_apply[True-True-Shuffle.PER_WORKER]

tests/unit/workflow/test_workflow.py::test_workflow_apply[True-True-Shuffle.PER_PARTITION]

tests/unit/workflow/test_workflow.py::test_workflow_apply[True-True-None]

tests/unit/workflow/test_workflow.py::test_workflow_apply[False-True-Shuffle.PER_WORKER]

tests/unit/workflow/test_workflow.py::test_workflow_apply[False-True-Shuffle.PER_PARTITION]

tests/unit/workflow/test_workflow.py::test_workflow_apply[False-True-None]

/core/merlin/io/dataset.py:863: UserWarning: Only created 2 files did not have enough partitions to create 4 files.

warnings.warn(


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                                           Stmts   Miss Branch BrPart  Cover   Missing


examples/multi-gpu-movielens/torch_trainer.py                     65      0      6      1    99%   32->36

examples/multi-gpu-movielens/torch_trainer_dist.py                63      0      2      0   100%

nvtabular/init.py                                             22      2      0      0    91%   44-45

nvtabular/dispatch.py                                              3      3      0      0     0%   18-23

nvtabular/framework_utils/init.py                              2      0      0      0   100%

nvtabular/framework_utils/tensorflow/init.py                   1      0      0      0   100%

nvtabular/framework_utils/tensorflow/feature_column_utils.py     134     78     90     15    39%   30, 99, 103, 114-130, 140, 143-158, 162, 166-167, 173-198, 207-217, 220-227, 229->233, 234, 239-279, 282

nvtabular/framework_utils/tensorflow/layers/init.py            4      0      0      0   100%

nvtabular/framework_utils/tensorflow/layers/embedding.py         153     13     89      6    90%   60, 68->49, 122, 179, 231-239, 242, 335->343, 357->360, 363-364, 367

nvtabular/framework_utils/tensorflow/layers/interaction.py        47     25     22      1    45%   49, 74-103, 106-110, 113

nvtabular/framework_utils/tensorflow/layers/outer_product.py      30     24     12      0    19%   37-38, 41-60, 71-84, 87

nvtabular/framework_utils/tensorflow/tfrecords_to_parquet.py      58     58     30      0     0%   16-111

nvtabular/framework_utils/torch/init.py                        0      0      0      0   100%

nvtabular/framework_utils/torch/layers/init.py                 2      0      0      0   100%

nvtabular/framework_utils/torch/layers/embeddings.py              32      2     18      2    92%   50, 91

nvtabular/framework_utils/torch/models.py                         45      2     30     10    84%   55->57, 57->61, 61->63, 66->68, 87->89, 89->93, 93->96, 96->100, 103, 108

nvtabular/framework_utils/torch/utils.py                          75     19     34      9    69%   22, 25-33, 51->53, 53->55, 64, 71->76, 75, 107, 118-120, 124-127, 132->115

nvtabular/graph.py                                                 3      3      0      0     0%   18-23

nvtabular/inference/init.py                                    2      0      0      0   100%

nvtabular/inference/triton/init.py                            36     12     14      1    58%   42-49, 68, 72, 76-82

nvtabular/inference/triton/benchmarking_tools.py                  52     52     10      0     0%   2-103

nvtabular/inference/triton/data_conversions.py                    87      3     58      4    95%   32-33, 84

nvtabular/inference/triton/ensemble.py                           266    200     82      3    25%   73-112, 157-196, 240-288, 292-323, 381-389, 418-434, 477-520, 548-588, 594-610, 614-681, 711, 733, 739-758, 764-788, 793-796

nvtabular/inference/triton/model/init.py                       0      0      0      0   100%

nvtabular/inference/triton/model/model_pt.py                     101    101     42      0     0%   27-220

nvtabular/inference/triton/model_config_pb2.py                   299      0      2      0   100%

nvtabular/inference/triton/workflow_model.py                      55     55     24      0     0%   27-128

nvtabular/inference/workflow/init.py                           0      0      0      0   100%

nvtabular/inference/workflow/base.py                             113    113     62      0     0%   27-209

nvtabular/inference/workflow/hugectr.py                           37     37     16      0     0%   27-87

nvtabular/inference/workflow/pytorch.py                           10     10      6      0     0%   27-46

nvtabular/inference/workflow/tensorflow.py                        32     32     10      0     0%   26-68

nvtabular/io.py                                                    3      3      0      0     0%   18-23

nvtabular/loader/init.py                                       2      0      0      0   100%

nvtabular/loader/backend.py                                      371     19    154     15    94%   27-28, 112, 116->108, 142, 158-159, 299->301, 311-315, 362-363, 402->406, 403->402, 478, 482-483, 512, 588-589, 598, 624, 632

nvtabular/loader/tensorflow.py                                   179     26     60     10    85%   38-39, 74, 85-89, 101, 105, 115, 124, 329, 357, 368, 383-385, 414-416, 426-434, 437-440

nvtabular/loader/tf_utils.py                                      57     27     22      6    46%   31->34, 34->36, 41->43, 45, 46->67, 52-53, 61-63, 69-73, 88-93, 103-116

nvtabular/loader/torch.py                                         87     16     26      3    78%   28-30, 33-39, 114, 158-159, 164, 202, 205

nvtabular/ops/init.py                                         26      0      0      0   100%

nvtabular/ops/add_metadata.py                                     34      0     14      0   100%

nvtabular/ops/bucketize.py                                        40      9     20      3    73%   52-54, 58->exit, 61-64, 83-86

nvtabular/ops/categorify.py                                      657     71    350     50    86%   251, 253, 271, 275, 283, 291, 293, 320, 341-342, 389->393, 397-404, 450, 458, 481-482, 559-564, 635, 731, 748, 793, 871-872, 887-891, 892->856, 910, 918, 925->exit, 949, 952->955, 1004->1002, 1066-1068, 1071, 1073->1075, 1092->1096, 1098->1051, 1104-1107, 1119, 1123, 1127, 1134, 1139-1142, 1220, 1222, 1292->1315, 1298->1315, 1316-1321, 1366, 1379->1382, 1386->1391, 1390, 1396, 1399, 1407-1417

nvtabular/ops/clip.py                                             18      2      8      3    81%   44, 52->54, 55

nvtabular/ops/column_similarity.py                               121     26     38      5    74%   19-20, 29-30, 81->exit, 111, 206-207, 216-218, 226-242, 259->262, 263, 273

nvtabular/ops/data_stats.py                                       56      1     24      3    95%   107->109, 111, 113->103

nvtabular/ops/difference_lag.py                                   43      0     14      1    98%   73->75

nvtabular/ops/drop_low_cardinality.py                             23      0      8      1    97%   41->46

nvtabular/ops/dropna.py                                            9      0      2      0   100%

nvtabular/ops/fill.py                                             76      5     30      1    92%   63-67, 109

nvtabular/ops/filter.py                                           20      1      8      1    93%   49

nvtabular/ops/groupby.py                                         130      4     84      5    96%   72, 83, 93->95, 131, 221

nvtabular/ops/hash_bucket.py                                      43      1     22      2    95%   79, 118->124

nvtabular/ops/hashed_cross.py                                     38      3     17      3    89%   52, 64, 92

nvtabular/ops/join_external.py                                    96      8     34      8    88%   20-21, 114, 116, 118, 137->141, 150->152, 205-206, 216->227, 221

nvtabular/ops/join_groupby.py                                    128      5     57      6    94%   113, 120, 129, 136->135, 178->175, 181->175, 260-261

nvtabular/ops/lambdaop.py                                         62     13     22      3    71%   60, 64, 82, 94-109

nvtabular/ops/list_slice.py                                       89     29     42      0    64%   21-22, 146-160, 168-190

nvtabular/ops/logop.py                                            21      0      6      0   100%

nvtabular/ops/moments.py                                          69      0     24      0   100%

nvtabular/ops/normalize.py                                        93      4     22      1    94%   89, 139-140, 167

nvtabular/ops/operator.py                                         11      1      2      0    92%   52

nvtabular/ops/reduce_dtype_size.py                                49      0     20      2    97%   68->77, 74->77

nvtabular/ops/rename.py                                           29      3     14      3    86%   46, 71-73

nvtabular/ops/stat_operator.py                                     8      0      2      0   100%

nvtabular/ops/target_encoding.py                                 182      9     76      5    93%   168->172, 176->185, 274, 283-284, 297-303, 396->399

nvtabular/ops/value_counts.py                                     34      0      6      1    98%   40->38

nvtabular/tools/init.py                                        0      0      0      0   100%

nvtabular/tools/data_gen.py                                      271     25     90      7    91%   26-27, 31-32, 129-132, 142-146, 148, 170-171, 322, 332, 358, 361-370

nvtabular/tools/dataset_inspector.py                              52      8     24      2    79%   33-40, 51

nvtabular/tools/inspector_script.py                               46     46      0      0     0%   17-168

nvtabular/utils.py                                                 3      3      0      0     0%   18-23

nvtabular/worker.py                                                3      3      0      0     0%   18-23

nvtabular/workflow/init.py                                     2      0      0      0   100%

nvtabular/workflow/node.py                                         7      0      4      0   100%

nvtabular/workflow/workflow.py                                   219     17     94     12    91%   28-29, 52, 85, 206, 212->226, 239-241, 373, 388-389, 431, 508, 538, 548-550, 563


TOTAL                                                           5336   1232   2099    214    75%

Coverage XML written to file coverage.xml


Required test coverage of 70% reached. Total coverage: 74.73%

=========================== short test summary info ============================

SKIPPED [1] ../../../../../usr/lib/python3.8/site-packages/dask_cudf/io/tests/test_s3.py:16: could not import 's3fs': cannot import name 'ParamSpec' from 'typing_extensions' (/var/jenkins_home/.local/lib/python3.8/site-packages/typing_extensions.py)

SKIPPED [1] tests/unit/loader/test_tf_dataloader.py:529: not working correctly in ci environment

========== 1404 passed, 2 skipped, 662 warnings in 1196.94s (0:19:56) ==========

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/NVTabular/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[nvtabular_tests] $ /bin/bash /tmp/jenkins855652275910924162.sh

@karlhigley karlhigley merged commit b6ab836 into NVIDIA-Merlin:main Apr 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@karlhigley karlhigley karlhigley approved these changes

Assignees

@bschifferer bschifferer

Labels
bug Something isn't working
Projects
None yet
Milestone
Merlin 22.04
Development

Successfully merging this pull request may close these issues.
5 participants
@bschifferer @nvidia-merlin-bot @karlhigley @benfred @rnyak