Integration Tests for Retrieval models #537

gabrielspmoreira · 2022-06-28T06:25:47Z

Fixes #482

Goals ⚽

This PR creates a set of unit and integration tests for retrieval models (based on our research scripts) to check for

Stability of our API (unit tests) - by using advanced options (hparams) of the APIs that might break if changed
Accuracy and Throughput (integration tests) - Compute accuracy and throughput metrics for training and evaluation, and log properly so that it is possible to track over time if we have a drop in those metrics.

Implementation Details 🚧

Uses Fiddle for Python-based configuration management
The training and evaluation code for the unit tests and integration tests is shared. The difference is that the unit tests run with synthetic data and integration tests run with a preprocessed LastFM dataset
This PR comprises unit and integration tests for MF and TwoTower.
The integration tests will be called in the CI by the test_container.sh script in PR Adds integration tests to Merlin Models Merlin#438 (to be merged after this PR)

Testing Details 🔍

CI

The unit tests run by CI every time code is pushed to a PR, in both CPU (GitHub Actions) and GPU (PR testing machine).
The integration tests runs nightly for the main branch (NOT for the PRs) in the Blossom machine in the container build process
Both PR testing machine and Blossom machine:
- Have available 4x P100 16GB GPUs
- Have a copy of the preprocessed LastFM dataset (parquet files) stored in /raid/data/lastfm/preprocessed for the integration test but that path can be changed by setting the CI_LASTFM_PREPROC_DATA_PATH env variable. The data folder has the following structure.

lastfm/
└── preprocessed
    ├── test
    │   ├── _file_list.txt
    │   ├── _metadata
    │   ├── _metadata.json
    │   ├── part_0.parquet
    │   └── schema.pbtxt
    ├── train
    │   ├── _file_list.txt
    │   ├── _metadata
    │   ├── _metadata.json
    │   ├── part_0.parquet
    │   └── schema.pbtxt
    └── valid
        ├── _file_list.txt
        ├── _metadata
        ├── _metadata.json
        ├── part_0.parquet
        └── schema.pbtxt

.

Logging

Both PR testing machine and Blossom machine have been setup with the WANDB_API_KEY env variable so that integration tests can log metrics to W&B. By default the merlin-ci W&B project is used for logging, but the W&B project id can be changed by setting the CI_WANDB_PROJECT env variable.
In below figure, you can see the logged runs for the integration tests of MF (1st row) and TwoTower (2nd row) models using the P100 16GB GPU . Those models are set to be trained for 1 epoch and evaluated for 2000 steps for both train and valid set and the runtime is under 10 min for each. We log to W&B the final accuracy metrics (e.g., recall_at_100-final) and throughput (average_examples_per_second-final), and also metrics over steps (e.g. loss curve) for debugging.

The final metrics should also be logged to ASVDB (AirSpeed Velocity) for tracking overtime, as done for NVTabular integration tests. That logging will not be done in this PR, but rather after the refactory of ASVDB logging is done in [Task] Set up integration tests for Merlin example notebooks Merlin#343

nvidia-merlin-bot · 2022-06-28T06:48:06Z

Click to view CI Results

GitHub pull request #537 of commit 326831a74f5f518d12082b957d312997669197e2, no merge conflicts.
Running as SYSTEM
Setting status of 326831a74f5f518d12082b957d312997669197e2 to PENDING with url https://10.20.13.93:8080/job/merlin_models/518/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_models
using credential nvidia-merlin-bot
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/models/ # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/models/
 > git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/models/ +refs/pull/537/*:refs/remotes/origin/pr/537/* # timeout=10
 > git rev-parse 326831a74f5f518d12082b957d312997669197e2^{commit} # timeout=10
Checking out Revision 326831a74f5f518d12082b957d312997669197e2 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 326831a74f5f518d12082b957d312997669197e2 # timeout=10
Commit message: "Started the integration tests for retrieval models using fiddle"
 > git rev-list --no-walk d3bf8d018c9d3267e22935bede5261551b54352d # timeout=10
[merlin_models] $ /bin/bash /tmp/jenkins18267751068169582396.sh
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: testbook in /var/jenkins_home/.local/lib/python3.8/site-packages (0.4.2)
Requirement already satisfied: nbformat>=5.0.4 in /usr/local/lib/python3.8/dist-packages (from testbook) (5.4.0)
Requirement already satisfied: nbclient>=0.4.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from testbook) (0.5.13)
Requirement already satisfied: traitlets>=5.1 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (5.3.0)
Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.6.0)
Requirement already satisfied: jupyter-core in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.10.0)
Requirement already satisfied: fastjsonschema in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (2.15.3)
Requirement already satisfied: jupyter-client>=6.1.5 in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (7.3.4)
Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (1.5.5)
Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (21.4.0)
Requirement already satisfied: importlib-resources>=1.4.0; python_version < "3.9" in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (5.8.0)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (0.18.1)
Requirement already satisfied: entrypoints in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (0.4)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (2.8.2)
Requirement already satisfied: pyzmq>=23.0 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (23.2.0)
Requirement already satisfied: tornado>=6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (6.1)
Requirement already satisfied: zipp>=3.1.0; python_version < "3.10" in /usr/local/lib/python3.8/dist-packages (from importlib-resources>=1.4.0; python_version < "3.9"->jsonschema>=2.6->nbformat>=5.0.4->testbook) (3.8.0)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.8.2->jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (1.15.0)
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/merlin_models/models, configfile: pyproject.toml
plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 465 items / 2 skipped
tests/unit/config/test_schema.py ....                                    [  0%]

tests/unit/datasets/test_advertising.py .s                               [  1%]

tests/unit/datasets/test_ecommerce.py ..sss                              [  2%]

tests/unit/datasets/test_entertainment.py ....sss.                       [  4%]

tests/unit/datasets/test_social.py .                                     [  4%]

tests/unit/datasets/test_synthetic.py .....                              [  5%]

tests/unit/tf/test_core.py ......                                        [  6%]

tests/unit/tf/test_dataset.py ...............                            [  9%]

tests/unit/tf/test_public_api.py .                                       [ 10%]

tests/unit/tf/blocks/test_cross.py ...........                           [ 12%]

tests/unit/tf/blocks/test_dlrm.py ........                               [ 14%]

tests/unit/tf/blocks/test_interactions.py .                              [ 14%]

tests/unit/tf/blocks/test_mlp.py .............................           [ 20%]

tests/unit/tf/blocks/core/test_aggregation.py .........                  [ 22%]

tests/unit/tf/blocks/core/test_base.py ..                                [ 23%]

tests/unit/tf/blocks/core/test_combinators.py .......                    [ 24%]

tests/unit/tf/blocks/core/test_index.py ...                              [ 25%]

tests/unit/tf/blocks/core/test_masking.py .......                        [ 26%]

tests/unit/tf/blocks/core/test_tabular.py ....                           [ 27%]

tests/unit/tf/blocks/core/test_transformations.py ...........            [ 29%]

tests/unit/tf/blocks/retrieval/test_matrix_factorization.py ..           [ 30%]

tests/unit/tf/blocks/retrieval/test_two_tower.py ...........             [ 32%]

tests/unit/tf/examples/test_01_getting_started.py .                      [ 32%]

tests/unit/tf/examples/test_02_dataschema.py .                           [ 33%]

tests/unit/tf/examples/test_03_exploring_different_models.py .           [ 33%]

tests/unit/tf/examples/test_04_export_ranking_models.py .                [ 33%]

tests/unit/tf/examples/test_05_export_retrieval_model.py .               [ 33%]

tests/unit/tf/examples/test_06_advanced_own_architecture.py .            [ 33%]

tests/unit/tf/examples/test_usecase_pretrained_embeddings.py .           [ 34%]

tests/unit/tf/inputs/test_continuous.py .....                            [ 35%]

tests/unit/tf/inputs/test_embedding.py ..............                    [ 38%]

tests/unit/tf/inputs/test_tabular.py .......                             [ 39%]

tests/unit/tf/layers/test_queue.py ..............                        [ 42%]

tests/unit/tf/losses/test_losses.py .......................              [ 47%]

tests/unit/tf/metrics/test_metrics_popularity.py .....                   [ 48%]

tests/unit/tf/metrics/test_metrics_topk.py ..................            [ 52%]

tests/unit/tf/models/test_base.py .........                              [ 54%]

tests/unit/tf/models/test_benchmark.py ..                                [ 55%]

tests/unit/tf/models/test_ranking.py ................                    [ 58%]

tests/unit/tf/models/test_retrieval.py .............................     [ 64%]

tests/unit/tf/prediction_tasks/test_classification.py ..                 [ 65%]

tests/unit/tf/prediction_tasks/test_multi_task.py .......                [ 66%]

tests/unit/tf/prediction_tasks/test_next_item.py ....................    [ 70%]

tests/unit/tf/prediction_tasks/test_regression.py ..                     [ 71%]

tests/unit/tf/prediction_tasks/test_sampling.py ....................     [ 75%]

tests/unit/tf/utils/test_batch.py ....                                   [ 76%]

tests/unit/torch/test_dataset.py .........                               [ 78%]

tests/unit/torch/test_public_api.py .                                    [ 78%]

tests/unit/torch/block/test_base.py ....                                 [ 79%]

tests/unit/torch/block/test_mlp.py .                                     [ 79%]

tests/unit/torch/features/test_continuous.py ..                          [ 80%]

tests/unit/torch/features/test_embedding.py ..............               [ 83%]

tests/unit/torch/features/test_tabular.py ....                           [ 84%]

tests/unit/torch/model/test_head.py ............                         [ 86%]

tests/unit/torch/model/test_model.py ..                                  [ 87%]

tests/unit/torch/tabular/test_aggregation.py ........                    [ 88%]

tests/unit/torch/tabular/test_tabular.py ...                             [ 89%]

tests/unit/torch/tabular/test_transformations.py .......                 [ 90%]

tests/unit/utils/test_schema_utils.py ................................   [ 97%]

tests/unit/xgb/test_xgboost.py ..........                                [100%]
=============================== warnings summary ===============================

../../../.local/lib/python3.8/site-packages/flatbuffers/compat.py:19

/var/jenkins_home/.local/lib/python3.8/site-packages/flatbuffers/compat.py:19: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses

import imp
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.

'nearest': pil_image.NEAREST,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.

'bilinear': pil_image.BILINEAR,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.

'bicubic': pil_image.BICUBIC,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.

'hamming': pil_image.HAMMING,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.

'box': pil_image.BOX,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.

'lanczos': pil_image.LANCZOS,
tests/unit/datasets/test_ecommerce.py::test_synthetic_aliccp_raw_data

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-True-10]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-True-9]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-True-8]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-False-10]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-False-9]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-False-8]

tests/unit/tf/test_dataset.py::test_tf_catname_ordering

tests/unit/tf/test_dataset.py::test_tf_map

/usr/local/lib/python3.8/dist-packages/cudf/core/frame.py:384: UserWarning: The deep parameter is ignored and is only included for pandas compatibility.

warnings.warn(
tests/unit/tf/blocks/core/test_index.py: 7 warnings

tests/unit/tf/models/test_retrieval.py: 311 warnings

tests/unit/tf/prediction_tasks/test_next_item.py: 100 warnings

tests/unit/tf/utils/test_batch.py: 4 warnings

/tmp/autograph_generated_filec39p3k23.py:8: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead

ag.converted_call(ag__.ld(warnings).warn, ("The 'warn' method is deprecated, use 'warning' instead", ag__.ld(DeprecationWarning), 2), None, fscope)
tests/unit/tf/blocks/core/test_transformations.py::test_stochastic_swap_noise[0.1]

tests/unit/tf/blocks/core/test_transformations.py::test_stochastic_swap_noise[0.3]

tests/unit/tf/blocks/core/test_transformations.py::test_stochastic_swap_noise[0.5]

tests/unit/tf/blocks/core/test_transformations.py::test_stochastic_swap_noise[0.7]

/usr/local/lib/python3.8/dist-packages/tensorflow/python/util/dispatch.py:1082: UserWarning: tf.keras.backend.random_binomial is deprecated, and will be removed in a future version.Please use tf.keras.backend.random_bernoulli instead.

return dispatch_target(*args, **kwargs)
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py::test_matrix_factorization_embedding_export

tests/unit/tf/blocks/retrieval/test_matrix_factorization.py::test_matrix_factorization_embedding_export

tests/unit/tf/blocks/retrieval/test_two_tower.py::test_matrix_factorization_embedding_export

tests/unit/tf/blocks/retrieval/test_two_tower.py::test_matrix_factorization_embedding_export

tests/unit/tf/inputs/test_embedding.py::test_embedding_features_exporting_and_loading_pretrained_initializer

/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py:324: DeprecationWarning: This function is deprecated in favor of cupy.from_dlpack

embeddings_cupy = cupy.fromDlpack(to_dlpack(tf.convert_to_tensor(embeddings)))
tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[True]

tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[True]

tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[False]

tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/numpy/core/numeric.py:2453: DeprecationWarning: elementwise comparison failed; this will raise an error in the future.

return bool(asarray(a1 == a2).all())
tests/unit/torch/block/test_mlp.py::test_mlp_block

/var/jenkins_home/workspace/merlin_models/models/tests/unit/torch/_conftest.py:151: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:210.)

return {key: torch.tensor(value) for key, value in data.items()}
tests/unit/xgb/test_xgboost.py::test_without_dask_client

tests/unit/xgb/test_xgboost.py::TestXGBoost::test_music_regression

tests/unit/xgb/test_xgboost.py::test_gpu_hist_dmatrix[fit_kwargs0-DaskDeviceQuantileDMatrix]

tests/unit/xgb/test_xgboost.py::test_gpu_hist_dmatrix[fit_kwargs1-DaskDMatrix]

/var/jenkins_home/workspace/merlin_models/models/merlin/models/xgb/init.py:249: UserWarning: Ignoring list columns as inputs to XGBoost model: ['item_genres', 'user_genres'].

warnings.warn(f"Ignoring list columns as inputs to XGBoost model: {list_column_names}.")
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

=========================== short test summary info ============================

SKIPPED [1] tests/unit/implicit/init.py:18: could not import 'implicit': No module named 'implicit'

SKIPPED [1] tests/unit/lightfm/init.py:18: could not import 'lightfm': No module named 'lightfm'

SKIPPED [1] tests/unit/datasets/test_advertising.py:20: No data-dir available, pass it through env variable $INPUT_DATA_DIR

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:62: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:78: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:92: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP

SKIPPED [3] tests/unit/datasets/test_entertainment.py:44: No data-dir available, pass it through env variable $INPUT_DATA_DIR

========== 458 passed, 9 skipped, 456 warnings in 1310.30s (0:21:50) ===========

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/models/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_models] $ /bin/bash /tmp/jenkins10764431366641289326.sh

nvidia-merlin-bot · 2022-07-06T23:33:47Z

Click to view CI Results

GitHub pull request #537 of commit 08a38456830f4bca72fb3f4289e2b333453ee6af, no merge conflicts.
Running as SYSTEM
Setting status of 08a38456830f4bca72fb3f4289e2b333453ee6af to PENDING with url https://10.20.13.93:8080/job/merlin_models/583/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_models
using credential nvidia-merlin-bot
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/models/ # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/models/
 > git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/models/ +refs/pull/537/*:refs/remotes/origin/pr/537/* # timeout=10
 > git rev-parse 08a38456830f4bca72fb3f4289e2b333453ee6af^{commit} # timeout=10
Checking out Revision 08a38456830f4bca72fb3f4289e2b333453ee6af (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 08a38456830f4bca72fb3f4289e2b333453ee6af # timeout=10
Commit message: "Finished first version of retrieval integration tests"
 > git rev-list --no-walk 2a93c3274728a98cb80df37894684baa4ae3eda1 # timeout=10
[merlin_models] $ /bin/bash /tmp/jenkins12995762093889843755.sh
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: testbook in /var/jenkins_home/.local/lib/python3.8/site-packages (0.4.2)
Requirement already satisfied: nbformat>=5.0.4 in /usr/local/lib/python3.8/dist-packages (from testbook) (5.4.0)
Requirement already satisfied: nbclient>=0.4.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from testbook) (0.5.13)
Requirement already satisfied: traitlets>=5.1 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (5.3.0)
Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.6.1)
Requirement already satisfied: jupyter-core in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.10.0)
Requirement already satisfied: fastjsonschema in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (2.15.3)
Requirement already satisfied: jupyter-client>=6.1.5 in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (7.3.4)
Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (1.5.5)
Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (21.4.0)
Requirement already satisfied: importlib-resources>=1.4.0; python_version < "3.9" in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (5.8.0)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (0.18.1)
Requirement already satisfied: entrypoints in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (0.4)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (2.8.2)
Requirement already satisfied: pyzmq>=23.0 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (23.2.0)
Requirement already satisfied: tornado>=6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (6.1)
Requirement already satisfied: zipp>=3.1.0; python_version < "3.10" in /usr/local/lib/python3.8/dist-packages (from importlib-resources>=1.4.0; python_version < "3.9"->jsonschema>=2.6->nbformat>=5.0.4->testbook) (3.8.0)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.8.2->jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (1.15.0)
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/merlin_models/models, configfile: pyproject.toml
plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 489 items / 2 skipped
tests/unit/config/test_schema.py ....                                    [  0%]

tests/unit/datasets/test_advertising.py .s                               [  1%]

tests/unit/datasets/test_ecommerce.py ..sss                              [  2%]

tests/unit/datasets/test_entertainment.py ....sss.                       [  3%]

tests/unit/datasets/test_social.py .                                     [  4%]

tests/unit/datasets/test_synthetic.py .....                              [  5%]

tests/unit/tf/test_core.py ......                                        [  6%]

tests/unit/tf/test_dataset.py ...............                            [  9%]

tests/unit/tf/test_public_api.py .                                       [  9%]

tests/unit/tf/blocks/test_cross.py ...........                           [ 11%]

tests/unit/tf/blocks/test_dlrm.py ........                               [ 13%]

tests/unit/tf/blocks/test_interactions.py .                              [ 13%]

tests/unit/tf/blocks/test_mlp.py .............................           [ 19%]

tests/unit/tf/blocks/core/test_aggregation.py .........                  [ 21%]

tests/unit/tf/blocks/core/test_base.py ..                                [ 21%]

tests/unit/tf/blocks/core/test_combinators.py .......                    [ 23%]

tests/unit/tf/blocks/core/test_index.py ...                              [ 23%]

tests/unit/tf/blocks/core/test_masking.py .......                        [ 25%]

tests/unit/tf/blocks/core/test_tabular.py ....                           [ 26%]

tests/unit/tf/blocks/core/test_transformations.py ...........            [ 28%]

tests/unit/tf/blocks/retrieval/test_matrix_factorization.py ..           [ 28%]

tests/unit/tf/blocks/retrieval/test_two_tower.py ...........             [ 31%]

tests/unit/tf/examples/test_01_getting_started.py .                      [ 31%]

tests/unit/tf/examples/test_02_dataschema.py .                           [ 31%]

tests/unit/tf/examples/test_03_exploring_different_models.py .           [ 31%]

tests/unit/tf/examples/test_04_export_ranking_models.py .                [ 31%]

tests/unit/tf/examples/test_05_export_retrieval_model.py .               [ 32%]

tests/unit/tf/examples/test_06_advanced_own_architecture.py .            [ 32%]

tests/unit/tf/examples/test_usecase_pretrained_embeddings.py .           [ 32%]

tests/unit/tf/inputs/test_continuous.py .....                            [ 33%]

tests/unit/tf/inputs/test_embedding.py ..........................        [ 38%]

tests/unit/tf/inputs/test_tabular.py .......                             [ 40%]

tests/unit/tf/layers/test_queue.py ..............                        [ 43%]

tests/unit/tf/losses/test_losses.py .......................              [ 47%]

tests/unit/tf/metrics/test_metrics_popularity.py .....                   [ 48%]

tests/unit/tf/metrics/test_metrics_topk.py ..................            [ 52%]

tests/unit/tf/models/test_base.py ............                           [ 55%]

tests/unit/tf/models/test_benchmark.py ..                                [ 55%]

tests/unit/tf/models/test_ranking.py ................                    [ 58%]

tests/unit/tf/models/test_retrieval.py .............................     [ 64%]

tests/unit/tf/prediction_tasks/test_classification.py ..                 [ 65%]

tests/unit/tf/prediction_tasks/test_multi_task.py .......                [ 66%]

tests/unit/tf/prediction_tasks/test_next_item.py ....................    [ 70%]

tests/unit/tf/prediction_tasks/test_regression.py ..                     [ 70%]

tests/unit/tf/prediction_tasks/test_sampling.py ....................     [ 75%]

tests/unit/tf/utils/test_batch.py ....                                   [ 75%]

tests/unit/tf/utils/test_tf_utils.py ..                                  [ 76%]

tests/unit/torch/test_dataset.py .........                               [ 78%]

tests/unit/torch/test_public_api.py .                                    [ 78%]

tests/unit/torch/block/test_base.py ....                                 [ 79%]

tests/unit/torch/block/test_mlp.py .                                     [ 79%]

tests/unit/torch/features/test_continuous.py ..                          [ 79%]

tests/unit/torch/features/test_embedding.py ..............               [ 82%]

tests/unit/torch/features/test_tabular.py ....                           [ 83%]

tests/unit/torch/model/test_head.py ............                         [ 85%]

tests/unit/torch/model/test_model.py ..                                  [ 86%]

tests/unit/torch/tabular/test_aggregation.py ........                    [ 87%]

tests/unit/torch/tabular/test_tabular.py ...                             [ 88%]

tests/unit/torch/tabular/test_transformations.py .......                 [ 89%]

tests/unit/utils/test_schema_utils.py ................................   [ 96%]

tests/unit/xgb/test_xgboost.py .................                         [100%]
=============================== warnings summary ===============================

../../../.local/lib/python3.8/site-packages/flatbuffers/compat.py:19

/var/jenkins_home/.local/lib/python3.8/site-packages/flatbuffers/compat.py:19: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses

import imp
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.

'nearest': pil_image.NEAREST,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.

'bilinear': pil_image.BILINEAR,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.

'bicubic': pil_image.BICUBIC,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.

'hamming': pil_image.HAMMING,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.

'box': pil_image.BOX,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.

'lanczos': pil_image.LANCZOS,
tests/unit/datasets/test_ecommerce.py::test_synthetic_aliccp_raw_data

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-True-10]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-True-9]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-True-8]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-False-10]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-False-9]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-False-8]

tests/unit/tf/test_dataset.py::test_tf_catname_ordering

tests/unit/tf/test_dataset.py::test_tf_map

/usr/local/lib/python3.8/dist-packages/cudf/core/frame.py:384: UserWarning: The deep parameter is ignored and is only included for pandas compatibility.

warnings.warn(
tests/unit/tf/blocks/core/test_index.py: 7 warnings

tests/unit/tf/models/test_retrieval.py: 311 warnings

tests/unit/tf/prediction_tasks/test_next_item.py: 100 warnings

tests/unit/tf/utils/test_batch.py: 4 warnings

/tmp/autograph_generated_filee4hji5zo.py:8: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead

ag.converted_call(ag__.ld(warnings).warn, ("The 'warn' method is deprecated, use 'warning' instead", ag__.ld(DeprecationWarning), 2), None, fscope)
tests/unit/tf/blocks/core/test_transformations.py: 4 warnings

tests/unit/tf/models/test_base.py: 6 warnings

/usr/local/lib/python3.8/dist-packages/tensorflow/python/util/dispatch.py:1082: UserWarning: tf.keras.backend.random_binomial is deprecated, and will be removed in a future version.Please use tf.keras.backend.random_bernoulli instead.

return dispatch_target(*args, **kwargs)
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py::test_matrix_factorization_embedding_export

tests/unit/tf/blocks/retrieval/test_matrix_factorization.py::test_matrix_factorization_embedding_export

tests/unit/tf/blocks/retrieval/test_two_tower.py::test_matrix_factorization_embedding_export

tests/unit/tf/blocks/retrieval/test_two_tower.py::test_matrix_factorization_embedding_export

tests/unit/tf/inputs/test_embedding.py::test_embedding_features_exporting_and_loading_pretrained_initializer

/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py:566: DeprecationWarning: This function is deprecated in favor of cupy.from_dlpack

embeddings_cupy = cupy.fromDlpack(to_dlpack(tf.convert_to_tensor(embeddings)))
tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[True]

tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[True]

tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[False]

tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/numpy/core/numeric.py:2453: DeprecationWarning: elementwise comparison failed; this will raise an error in the future.

return bool(asarray(a1 == a2).all())
tests/unit/torch/block/test_mlp.py::test_mlp_block

/var/jenkins_home/workspace/merlin_models/models/tests/unit/torch/_conftest.py:151: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)

return {key: torch.tensor(value) for key, value in data.items()}
tests/unit/xgb/test_xgboost.py::test_without_dask_client

tests/unit/xgb/test_xgboost.py::TestXGBoost::test_music_regression

tests/unit/xgb/test_xgboost.py::test_gpu_hist_dmatrix[fit_kwargs0-DaskDeviceQuantileDMatrix]

tests/unit/xgb/test_xgboost.py::test_gpu_hist_dmatrix[fit_kwargs1-DaskDMatrix]

tests/unit/xgb/test_xgboost.py::TestEvals::test_multiple

tests/unit/xgb/test_xgboost.py::TestEvals::test_default

tests/unit/xgb/test_xgboost.py::TestEvals::test_train_and_valid

tests/unit/xgb/test_xgboost.py::TestEvals::test_invalid_data

/var/jenkins_home/workspace/merlin_models/models/merlin/models/xgb/init.py:266: UserWarning: Ignoring list columns as inputs to XGBoost model: ['item_genres', 'user_genres'].

warnings.warn(f"Ignoring list columns as inputs to XGBoost model: {list_column_names}.")
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

=========================== short test summary info ============================

SKIPPED [1] tests/unit/implicit/init.py:18: could not import 'implicit': No module named 'implicit'

SKIPPED [1] tests/unit/lightfm/init.py:18: could not import 'lightfm': No module named 'lightfm'

SKIPPED [1] tests/unit/datasets/test_advertising.py:20: No data-dir available, pass it through env variable $INPUT_DATA_DIR

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:62: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:78: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:92: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP

SKIPPED [3] tests/unit/datasets/test_entertainment.py:44: No data-dir available, pass it through env variable $INPUT_DATA_DIR

========== 482 passed, 9 skipped, 466 warnings in 1449.30s (0:24:09) ===========

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/models/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_models] $ /bin/bash /tmp/jenkins17654370569026588615.sh

github-actions · 2022-07-06T23:37:37Z

Documentation preview

https://nvidia-merlin.github.io/models/review/pr-537

nvidia-merlin-bot · 2022-07-07T01:15:52Z

Click to view CI Results

GitHub pull request #537 of commit 9e169cca4f5185f7d3b24e24e9e867347a88302c, no merge conflicts.
Running as SYSTEM
Setting status of 9e169cca4f5185f7d3b24e24e9e867347a88302c to PENDING with url https://10.20.13.93:8080/job/merlin_models/584/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_models
using credential nvidia-merlin-bot
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/models/ # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/models/
 > git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/models/ +refs/pull/537/*:refs/remotes/origin/pr/537/* # timeout=10
 > git rev-parse 9e169cca4f5185f7d3b24e24e9e867347a88302c^{commit} # timeout=10
Checking out Revision 9e169cca4f5185f7d3b24e24e9e867347a88302c (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 9e169cca4f5185f7d3b24e24e9e867347a88302c # timeout=10
Commit message: "Added CI script for integration tests"
 > git rev-list --no-walk 08a38456830f4bca72fb3f4289e2b333453ee6af # timeout=10
[merlin_models] $ /bin/bash /tmp/jenkins15823996260727284071.sh
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: testbook in /var/jenkins_home/.local/lib/python3.8/site-packages (0.4.2)
Requirement already satisfied: nbformat>=5.0.4 in /usr/local/lib/python3.8/dist-packages (from testbook) (5.4.0)
Requirement already satisfied: nbclient>=0.4.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from testbook) (0.5.13)
Requirement already satisfied: traitlets>=5.1 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (5.3.0)
Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.6.1)
Requirement already satisfied: jupyter-core in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.10.0)
Requirement already satisfied: fastjsonschema in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (2.15.3)
Requirement already satisfied: jupyter-client>=6.1.5 in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (7.3.4)
Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (1.5.5)
Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (21.4.0)
Requirement already satisfied: importlib-resources>=1.4.0; python_version < "3.9" in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (5.8.0)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (0.18.1)
Requirement already satisfied: entrypoints in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (0.4)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (2.8.2)
Requirement already satisfied: pyzmq>=23.0 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (23.2.0)
Requirement already satisfied: tornado>=6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (6.1)
Requirement already satisfied: zipp>=3.1.0; python_version < "3.10" in /usr/local/lib/python3.8/dist-packages (from importlib-resources>=1.4.0; python_version < "3.9"->jsonschema>=2.6->nbformat>=5.0.4->testbook) (3.8.0)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.8.2->jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (1.15.0)
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/merlin_models/models, configfile: pyproject.toml
plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 489 items / 2 skipped
tests/unit/config/test_schema.py ....                                    [  0%]

tests/unit/datasets/test_advertising.py .s                               [  1%]

tests/unit/datasets/test_ecommerce.py ..sss                              [  2%]

tests/unit/datasets/test_entertainment.py ....sss.                       [  3%]

tests/unit/datasets/test_social.py .                                     [  4%]

tests/unit/datasets/test_synthetic.py .....                              [  5%]

tests/unit/tf/test_core.py ......                                        [  6%]

tests/unit/tf/test_dataset.py ...............                            [  9%]

tests/unit/tf/test_public_api.py .                                       [  9%]

tests/unit/tf/blocks/test_cross.py ...........                           [ 11%]

tests/unit/tf/blocks/test_dlrm.py ........                               [ 13%]

tests/unit/tf/blocks/test_interactions.py .                              [ 13%]

tests/unit/tf/blocks/test_mlp.py .............................           [ 19%]

tests/unit/tf/blocks/core/test_aggregation.py .........                  [ 21%]

tests/unit/tf/blocks/core/test_base.py ..                                [ 21%]

tests/unit/tf/blocks/core/test_combinators.py .......                    [ 23%]

tests/unit/tf/blocks/core/test_index.py ...                              [ 23%]

tests/unit/tf/blocks/core/test_masking.py .......                        [ 25%]

tests/unit/tf/blocks/core/test_tabular.py ....                           [ 26%]

tests/unit/tf/blocks/core/test_transformations.py ...........            [ 28%]

tests/unit/tf/blocks/retrieval/test_matrix_factorization.py ..           [ 28%]

tests/unit/tf/blocks/retrieval/test_two_tower.py ...........             [ 31%]

tests/unit/tf/examples/test_01_getting_started.py .                      [ 31%]

tests/unit/tf/examples/test_02_dataschema.py .                           [ 31%]

tests/unit/tf/examples/test_03_exploring_different_models.py .           [ 31%]

tests/unit/tf/examples/test_04_export_ranking_models.py .                [ 31%]

tests/unit/tf/examples/test_05_export_retrieval_model.py .               [ 32%]

tests/unit/tf/examples/test_06_advanced_own_architecture.py .            [ 32%]

tests/unit/tf/examples/test_usecase_pretrained_embeddings.py .           [ 32%]

tests/unit/tf/inputs/test_continuous.py .....                            [ 33%]

tests/unit/tf/inputs/test_embedding.py ..........................        [ 38%]

tests/unit/tf/inputs/test_tabular.py .......                             [ 40%]

tests/unit/tf/layers/test_queue.py ..............                        [ 43%]

tests/unit/tf/losses/test_losses.py .......................              [ 47%]

tests/unit/tf/metrics/test_metrics_popularity.py .....                   [ 48%]

tests/unit/tf/metrics/test_metrics_topk.py ..................            [ 52%]

tests/unit/tf/models/test_base.py ............                           [ 55%]

tests/unit/tf/models/test_benchmark.py ..                                [ 55%]

tests/unit/tf/models/test_ranking.py ................                    [ 58%]

tests/unit/tf/models/test_retrieval.py .............................     [ 64%]

tests/unit/tf/prediction_tasks/test_classification.py ..                 [ 65%]

tests/unit/tf/prediction_tasks/test_multi_task.py .......                [ 66%]

tests/unit/tf/prediction_tasks/test_next_item.py ....................    [ 70%]

tests/unit/tf/prediction_tasks/test_regression.py ..                     [ 70%]

tests/unit/tf/prediction_tasks/test_sampling.py ....................     [ 75%]

tests/unit/tf/utils/test_batch.py ....                                   [ 75%]

tests/unit/tf/utils/test_tf_utils.py ..                                  [ 76%]

tests/unit/torch/test_dataset.py .........                               [ 78%]

tests/unit/torch/test_public_api.py .                                    [ 78%]

tests/unit/torch/block/test_base.py ....                                 [ 79%]

tests/unit/torch/block/test_mlp.py .                                     [ 79%]

tests/unit/torch/features/test_continuous.py ..                          [ 79%]

tests/unit/torch/features/test_embedding.py ..............               [ 82%]

tests/unit/torch/features/test_tabular.py ....                           [ 83%]

tests/unit/torch/model/test_head.py ............                         [ 85%]

tests/unit/torch/model/test_model.py ..                                  [ 86%]

tests/unit/torch/tabular/test_aggregation.py ........                    [ 87%]

tests/unit/torch/tabular/test_tabular.py ...                             [ 88%]

tests/unit/torch/tabular/test_transformations.py .......                 [ 89%]

tests/unit/utils/test_schema_utils.py ................................   [ 96%]

tests/unit/xgb/test_xgboost.py .................                         [100%]
=============================== warnings summary ===============================

../../../.local/lib/python3.8/site-packages/flatbuffers/compat.py:19

/var/jenkins_home/.local/lib/python3.8/site-packages/flatbuffers/compat.py:19: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses

import imp
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.

'nearest': pil_image.NEAREST,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.

'bilinear': pil_image.BILINEAR,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.

'bicubic': pil_image.BICUBIC,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.

'hamming': pil_image.HAMMING,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.

'box': pil_image.BOX,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.

'lanczos': pil_image.LANCZOS,
tests/unit/datasets/test_ecommerce.py::test_synthetic_aliccp_raw_data

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-True-10]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-True-9]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-True-8]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-False-10]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-False-9]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-False-8]

tests/unit/tf/test_dataset.py::test_tf_catname_ordering

tests/unit/tf/test_dataset.py::test_tf_map

/usr/local/lib/python3.8/dist-packages/cudf/core/frame.py:384: UserWarning: The deep parameter is ignored and is only included for pandas compatibility.

warnings.warn(
tests/unit/tf/blocks/core/test_index.py: 7 warnings

tests/unit/tf/models/test_retrieval.py: 311 warnings

tests/unit/tf/prediction_tasks/test_next_item.py: 100 warnings

tests/unit/tf/utils/test_batch.py: 4 warnings

/tmp/autograph_generated_filecfp4d1o2.py:8: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead

ag.converted_call(ag__.ld(warnings).warn, ("The 'warn' method is deprecated, use 'warning' instead", ag__.ld(DeprecationWarning), 2), None, fscope)
tests/unit/tf/blocks/core/test_transformations.py: 4 warnings

tests/unit/tf/models/test_base.py: 6 warnings

/usr/local/lib/python3.8/dist-packages/tensorflow/python/util/dispatch.py:1082: UserWarning: tf.keras.backend.random_binomial is deprecated, and will be removed in a future version.Please use tf.keras.backend.random_bernoulli instead.

return dispatch_target(*args, **kwargs)
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py::test_matrix_factorization_embedding_export

tests/unit/tf/blocks/retrieval/test_matrix_factorization.py::test_matrix_factorization_embedding_export

tests/unit/tf/blocks/retrieval/test_two_tower.py::test_matrix_factorization_embedding_export

tests/unit/tf/blocks/retrieval/test_two_tower.py::test_matrix_factorization_embedding_export

tests/unit/tf/inputs/test_embedding.py::test_embedding_features_exporting_and_loading_pretrained_initializer

/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py:566: DeprecationWarning: This function is deprecated in favor of cupy.from_dlpack

embeddings_cupy = cupy.fromDlpack(to_dlpack(tf.convert_to_tensor(embeddings)))
tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[True]

tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[True]

tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[False]

tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/numpy/core/numeric.py:2453: DeprecationWarning: elementwise comparison failed; this will raise an error in the future.

return bool(asarray(a1 == a2).all())
tests/unit/torch/block/test_mlp.py::test_mlp_block

/var/jenkins_home/workspace/merlin_models/models/tests/unit/torch/_conftest.py:151: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)

return {key: torch.tensor(value) for key, value in data.items()}
tests/unit/xgb/test_xgboost.py::test_without_dask_client

tests/unit/xgb/test_xgboost.py::TestXGBoost::test_music_regression

tests/unit/xgb/test_xgboost.py::test_gpu_hist_dmatrix[fit_kwargs0-DaskDeviceQuantileDMatrix]

tests/unit/xgb/test_xgboost.py::test_gpu_hist_dmatrix[fit_kwargs1-DaskDMatrix]

tests/unit/xgb/test_xgboost.py::TestEvals::test_multiple

tests/unit/xgb/test_xgboost.py::TestEvals::test_default

tests/unit/xgb/test_xgboost.py::TestEvals::test_train_and_valid

tests/unit/xgb/test_xgboost.py::TestEvals::test_invalid_data

/var/jenkins_home/workspace/merlin_models/models/merlin/models/xgb/init.py:266: UserWarning: Ignoring list columns as inputs to XGBoost model: ['item_genres', 'user_genres'].

warnings.warn(f"Ignoring list columns as inputs to XGBoost model: {list_column_names}.")
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

=========================== short test summary info ============================

SKIPPED [1] tests/unit/implicit/init.py:18: could not import 'implicit': No module named 'implicit'

SKIPPED [1] tests/unit/lightfm/init.py:18: could not import 'lightfm': No module named 'lightfm'

SKIPPED [1] tests/unit/datasets/test_advertising.py:20: No data-dir available, pass it through env variable $INPUT_DATA_DIR

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:62: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:78: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:92: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP

SKIPPED [3] tests/unit/datasets/test_entertainment.py:44: No data-dir available, pass it through env variable $INPUT_DATA_DIR

========== 482 passed, 9 skipped, 466 warnings in 1459.23s (0:24:19) ===========

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/models/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_models] $ /bin/bash /tmp/jenkins15119829467177249500.sh

nvidia-merlin-bot · 2022-07-07T01:40:29Z

Click to view CI Results

GitHub pull request #537 of commit 386892343bb8b86d77a1f082caf825f486059305, no merge conflicts.
Running as SYSTEM
Setting status of 386892343bb8b86d77a1f082caf825f486059305 to PENDING with url https://10.20.13.93:8080/job/merlin_models/585/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_models
using credential nvidia-merlin-bot
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/models/ # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/models/
 > git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/models/ +refs/pull/537/*:refs/remotes/origin/pr/537/* # timeout=10
 > git rev-parse 386892343bb8b86d77a1f082caf825f486059305^{commit} # timeout=10
Checking out Revision 386892343bb8b86d77a1f082caf825f486059305 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 386892343bb8b86d77a1f082caf825f486059305 # timeout=10
Commit message: "Fixed CI integration commands"
 > git rev-list --no-walk 9e169cca4f5185f7d3b24e24e9e867347a88302c # timeout=10
[merlin_models] $ /bin/bash /tmp/jenkins18276078802222716052.sh
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: testbook in /var/jenkins_home/.local/lib/python3.8/site-packages (0.4.2)
Requirement already satisfied: nbformat>=5.0.4 in /usr/local/lib/python3.8/dist-packages (from testbook) (5.4.0)
Requirement already satisfied: nbclient>=0.4.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from testbook) (0.5.13)
Requirement already satisfied: traitlets>=5.1 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (5.3.0)
Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.6.1)
Requirement already satisfied: jupyter-core in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.10.0)
Requirement already satisfied: fastjsonschema in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (2.15.3)
Requirement already satisfied: jupyter-client>=6.1.5 in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (7.3.4)
Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (1.5.5)
Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (21.4.0)
Requirement already satisfied: importlib-resources>=1.4.0; python_version < "3.9" in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (5.8.0)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (0.18.1)
Requirement already satisfied: entrypoints in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (0.4)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (2.8.2)
Requirement already satisfied: pyzmq>=23.0 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (23.2.0)
Requirement already satisfied: tornado>=6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (6.1)
Requirement already satisfied: zipp>=3.1.0; python_version < "3.10" in /usr/local/lib/python3.8/dist-packages (from importlib-resources>=1.4.0; python_version < "3.9"->jsonschema>=2.6->nbformat>=5.0.4->testbook) (3.8.0)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.8.2->jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (1.15.0)
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/merlin_models/models, configfile: pyproject.toml
plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 489 items / 2 skipped
tests/unit/config/test_schema.py ....                                    [  0%]

tests/unit/datasets/test_advertising.py .s                               [  1%]

tests/unit/datasets/test_ecommerce.py ..sss                              [  2%]

tests/unit/datasets/test_entertainment.py ....sss.                       [  3%]

tests/unit/datasets/test_social.py .                                     [  4%]

tests/unit/datasets/test_synthetic.py .....                              [  5%]

tests/unit/tf/test_core.py ......                                        [  6%]

tests/unit/tf/test_dataset.py ...............                            [  9%]

tests/unit/tf/test_public_api.py .                                       [  9%]

tests/unit/tf/blocks/test_cross.py ...........                           [ 11%]

tests/unit/tf/blocks/test_dlrm.py ........                               [ 13%]

tests/unit/tf/blocks/test_interactions.py .                              [ 13%]

tests/unit/tf/blocks/test_mlp.py .............................           [ 19%]

tests/unit/tf/blocks/core/test_aggregation.py .........                  [ 21%]

tests/unit/tf/blocks/core/test_base.py ..                                [ 21%]

tests/unit/tf/blocks/core/test_combinators.py .......                    [ 23%]

tests/unit/tf/blocks/core/test_index.py ...                              [ 23%]

tests/unit/tf/blocks/core/test_masking.py .......                        [ 25%]

tests/unit/tf/blocks/core/test_tabular.py ....                           [ 26%]

tests/unit/tf/blocks/core/test_transformations.py ...........            [ 28%]

tests/unit/tf/blocks/retrieval/test_matrix_factorization.py ..           [ 28%]

tests/unit/tf/blocks/retrieval/test_two_tower.py ...........             [ 31%]

tests/unit/tf/examples/test_01_getting_started.py .                      [ 31%]

tests/unit/tf/examples/test_02_dataschema.py .                           [ 31%]

tests/unit/tf/examples/test_03_exploring_different_models.py .           [ 31%]

tests/unit/tf/examples/test_04_export_ranking_models.py .                [ 31%]

tests/unit/tf/examples/test_05_export_retrieval_model.py .               [ 32%]

tests/unit/tf/examples/test_06_advanced_own_architecture.py .            [ 32%]

tests/unit/tf/examples/test_usecase_pretrained_embeddings.py .           [ 32%]

tests/unit/tf/inputs/test_continuous.py .....                            [ 33%]

tests/unit/tf/inputs/test_embedding.py ..........................        [ 38%]

tests/unit/tf/inputs/test_tabular.py .......                             [ 40%]

tests/unit/tf/layers/test_queue.py ..............                        [ 43%]

tests/unit/tf/losses/test_losses.py .......................              [ 47%]

tests/unit/tf/metrics/test_metrics_popularity.py .....                   [ 48%]

tests/unit/tf/metrics/test_metrics_topk.py ..................            [ 52%]

tests/unit/tf/models/test_base.py ............                           [ 55%]

tests/unit/tf/models/test_benchmark.py ..                                [ 55%]

tests/unit/tf/models/test_ranking.py ................                    [ 58%]

tests/unit/tf/models/test_retrieval.py .............................     [ 64%]

tests/unit/tf/prediction_tasks/test_classification.py ..                 [ 65%]

tests/unit/tf/prediction_tasks/test_multi_task.py .......                [ 66%]

tests/unit/tf/prediction_tasks/test_next_item.py ....................    [ 70%]

tests/unit/tf/prediction_tasks/test_regression.py ..                     [ 70%]

tests/unit/tf/prediction_tasks/test_sampling.py ....................     [ 75%]

tests/unit/tf/utils/test_batch.py ....                                   [ 75%]

tests/unit/tf/utils/test_tf_utils.py ..                                  [ 76%]

tests/unit/torch/test_dataset.py .........                               [ 78%]

tests/unit/torch/test_public_api.py .                                    [ 78%]

tests/unit/torch/block/test_base.py ....                                 [ 79%]

tests/unit/torch/block/test_mlp.py .                                     [ 79%]

tests/unit/torch/features/test_continuous.py ..                          [ 79%]

tests/unit/torch/features/test_embedding.py ..............               [ 82%]

tests/unit/torch/features/test_tabular.py ....                           [ 83%]

tests/unit/torch/model/test_head.py ............                         [ 85%]

tests/unit/torch/model/test_model.py ..                                  [ 86%]

tests/unit/torch/tabular/test_aggregation.py ........                    [ 87%]

tests/unit/torch/tabular/test_tabular.py ...                             [ 88%]

tests/unit/torch/tabular/test_transformations.py .......                 [ 89%]

tests/unit/utils/test_schema_utils.py ................................   [ 96%]

tests/unit/xgb/test_xgboost.py .................                         [100%]
=============================== warnings summary ===============================

../../../.local/lib/python3.8/site-packages/flatbuffers/compat.py:19

/var/jenkins_home/.local/lib/python3.8/site-packages/flatbuffers/compat.py:19: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses

import imp
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.

'nearest': pil_image.NEAREST,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.

'bilinear': pil_image.BILINEAR,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.

'bicubic': pil_image.BICUBIC,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.

'hamming': pil_image.HAMMING,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.

'box': pil_image.BOX,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.

'lanczos': pil_image.LANCZOS,
tests/unit/datasets/test_ecommerce.py::test_synthetic_aliccp_raw_data

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-True-10]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-True-9]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-True-8]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-False-10]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-False-9]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-False-8]

tests/unit/tf/test_dataset.py::test_tf_catname_ordering

tests/unit/tf/test_dataset.py::test_tf_map

/usr/local/lib/python3.8/dist-packages/cudf/core/frame.py:384: UserWarning: The deep parameter is ignored and is only included for pandas compatibility.

warnings.warn(
tests/unit/tf/blocks/core/test_index.py: 7 warnings

tests/unit/tf/models/test_retrieval.py: 311 warnings

tests/unit/tf/prediction_tasks/test_next_item.py: 100 warnings

tests/unit/tf/utils/test_batch.py: 4 warnings

/tmp/autograph_generated_filer2hu9l29.py:8: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead

ag.converted_call(ag__.ld(warnings).warn, ("The 'warn' method is deprecated, use 'warning' instead", ag__.ld(DeprecationWarning), 2), None, fscope)
tests/unit/tf/blocks/core/test_transformations.py: 4 warnings

tests/unit/tf/models/test_base.py: 6 warnings

/usr/local/lib/python3.8/dist-packages/tensorflow/python/util/dispatch.py:1082: UserWarning: tf.keras.backend.random_binomial is deprecated, and will be removed in a future version.Please use tf.keras.backend.random_bernoulli instead.

return dispatch_target(*args, **kwargs)
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py::test_matrix_factorization_embedding_export

tests/unit/tf/blocks/retrieval/test_matrix_factorization.py::test_matrix_factorization_embedding_export

tests/unit/tf/blocks/retrieval/test_two_tower.py::test_matrix_factorization_embedding_export

tests/unit/tf/blocks/retrieval/test_two_tower.py::test_matrix_factorization_embedding_export

tests/unit/tf/inputs/test_embedding.py::test_embedding_features_exporting_and_loading_pretrained_initializer

/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py:566: DeprecationWarning: This function is deprecated in favor of cupy.from_dlpack

embeddings_cupy = cupy.fromDlpack(to_dlpack(tf.convert_to_tensor(embeddings)))
tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[True]

tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[True]

tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[False]

tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/numpy/core/numeric.py:2453: DeprecationWarning: elementwise comparison failed; this will raise an error in the future.

return bool(asarray(a1 == a2).all())
tests/unit/torch/block/test_mlp.py::test_mlp_block

/var/jenkins_home/workspace/merlin_models/models/tests/unit/torch/_conftest.py:151: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)

return {key: torch.tensor(value) for key, value in data.items()}
tests/unit/xgb/test_xgboost.py::test_without_dask_client

tests/unit/xgb/test_xgboost.py::TestXGBoost::test_music_regression

tests/unit/xgb/test_xgboost.py::test_gpu_hist_dmatrix[fit_kwargs0-DaskDeviceQuantileDMatrix]

tests/unit/xgb/test_xgboost.py::test_gpu_hist_dmatrix[fit_kwargs1-DaskDMatrix]

tests/unit/xgb/test_xgboost.py::TestEvals::test_multiple

tests/unit/xgb/test_xgboost.py::TestEvals::test_default

tests/unit/xgb/test_xgboost.py::TestEvals::test_train_and_valid

tests/unit/xgb/test_xgboost.py::TestEvals::test_invalid_data

/var/jenkins_home/workspace/merlin_models/models/merlin/models/xgb/init.py:266: UserWarning: Ignoring list columns as inputs to XGBoost model: ['item_genres', 'user_genres'].

warnings.warn(f"Ignoring list columns as inputs to XGBoost model: {list_column_names}.")
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

=========================== short test summary info ============================

SKIPPED [1] tests/unit/implicit/init.py:18: could not import 'implicit': No module named 'implicit'

SKIPPED [1] tests/unit/lightfm/init.py:18: could not import 'lightfm': No module named 'lightfm'

SKIPPED [1] tests/unit/datasets/test_advertising.py:20: No data-dir available, pass it through env variable $INPUT_DATA_DIR

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:62: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:78: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:92: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP

SKIPPED [3] tests/unit/datasets/test_entertainment.py:44: No data-dir available, pass it through env variable $INPUT_DATA_DIR

========== 482 passed, 9 skipped, 466 warnings in 1460.33s (0:24:20) ===========

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/models/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_models] $ /bin/bash /tmp/jenkins2674751604004063899.sh

nvidia-merlin-bot · 2022-07-07T15:51:07Z

Click to view CI Results

GitHub pull request #537 of commit 790e2eadebfcd1a276ebf51532d3c0b9dde3f26e, no merge conflicts.
Running as SYSTEM
Setting status of 790e2eadebfcd1a276ebf51532d3c0b9dde3f26e to PENDING with url https://10.20.13.93:8080/job/merlin_models/592/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_models
using credential nvidia-merlin-bot
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/models/ # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/models/
 > git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/models/ +refs/pull/537/*:refs/remotes/origin/pr/537/* # timeout=10
 > git rev-parse 790e2eadebfcd1a276ebf51532d3c0b9dde3f26e^{commit} # timeout=10
Checking out Revision 790e2eadebfcd1a276ebf51532d3c0b9dde3f26e (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 790e2eadebfcd1a276ebf51532d3c0b9dde3f26e # timeout=10
Commit message: "Enabling only retrieval integration tests"
 > git rev-list --no-walk 012b1f031ea524d705ec212446885d1dfd42076b # timeout=10
[merlin_models] $ /bin/bash /tmp/jenkins14153993953834989652.sh
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: testbook in /var/jenkins_home/.local/lib/python3.8/site-packages (0.4.2)
Requirement already satisfied: nbformat>=5.0.4 in /usr/local/lib/python3.8/dist-packages (from testbook) (5.4.0)
Requirement already satisfied: nbclient>=0.4.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from testbook) (0.5.13)
Requirement already satisfied: traitlets>=5.1 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (5.3.0)
Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.6.1)
Requirement already satisfied: jupyter-core in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.10.0)
Requirement already satisfied: fastjsonschema in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (2.15.3)
Requirement already satisfied: jupyter-client>=6.1.5 in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (7.3.4)
Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (1.5.5)
Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (21.4.0)
Requirement already satisfied: importlib-resources>=1.4.0; python_version < "3.9" in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (5.8.0)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (0.18.1)
Requirement already satisfied: entrypoints in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (0.4)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (2.8.2)
Requirement already satisfied: pyzmq>=23.0 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (23.2.0)
Requirement already satisfied: tornado>=6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (6.1)
Requirement already satisfied: zipp>=3.1.0; python_version < "3.10" in /usr/local/lib/python3.8/dist-packages (from importlib-resources>=1.4.0; python_version < "3.9"->jsonschema>=2.6->nbformat>=5.0.4->testbook) (3.8.0)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.8.2->jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (1.15.0)
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/merlin_models/models, configfile: pyproject.toml
plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 489 items / 2 skipped
tests/unit/config/test_schema.py ....                                    [  0%]

tests/unit/datasets/test_advertising.py .s                               [  1%]

tests/unit/datasets/test_ecommerce.py ..sss                              [  2%]

tests/unit/datasets/test_entertainment.py ....sss.                       [  3%]

tests/unit/datasets/test_social.py .                                     [  4%]

tests/unit/datasets/test_synthetic.py .....                              [  5%]

tests/unit/tf/test_core.py ......                                        [  6%]

tests/unit/tf/test_dataset.py ...............                            [  9%]

tests/unit/tf/test_public_api.py .                                       [  9%]

tests/unit/tf/blocks/test_cross.py ...........                           [ 11%]

tests/unit/tf/blocks/test_dlrm.py ........                               [ 13%]

tests/unit/tf/blocks/test_interactions.py .                              [ 13%]

tests/unit/tf/blocks/test_mlp.py .............................           [ 19%]

tests/unit/tf/blocks/core/test_aggregation.py .........                  [ 21%]

tests/unit/tf/blocks/core/test_base.py ..                                [ 21%]

tests/unit/tf/blocks/core/test_combinators.py .......                    [ 23%]

tests/unit/tf/blocks/core/test_index.py ...                              [ 23%]

tests/unit/tf/blocks/core/test_masking.py .......                        [ 25%]

tests/unit/tf/blocks/core/test_tabular.py ....                           [ 26%]

tests/unit/tf/blocks/core/test_transformations.py ...........            [ 28%]

tests/unit/tf/blocks/retrieval/test_matrix_factorization.py ..           [ 28%]

tests/unit/tf/blocks/retrieval/test_two_tower.py ...........             [ 31%]

tests/unit/tf/examples/test_01_getting_started.py .                      [ 31%]

tests/unit/tf/examples/test_02_dataschema.py .                           [ 31%]

tests/unit/tf/examples/test_03_exploring_different_models.py .           [ 31%]

tests/unit/tf/examples/test_04_export_ranking_models.py .                [ 31%]

tests/unit/tf/examples/test_05_export_retrieval_model.py .               [ 32%]

tests/unit/tf/examples/test_06_advanced_own_architecture.py .            [ 32%]

tests/unit/tf/examples/test_usecase_pretrained_embeddings.py .           [ 32%]

tests/unit/tf/inputs/test_continuous.py .....                            [ 33%]

tests/unit/tf/inputs/test_embedding.py ..........................        [ 38%]

tests/unit/tf/inputs/test_tabular.py .......                             [ 40%]

tests/unit/tf/layers/test_queue.py ..............                        [ 43%]

tests/unit/tf/losses/test_losses.py .......................              [ 47%]

tests/unit/tf/metrics/test_metrics_popularity.py .....                   [ 48%]

tests/unit/tf/metrics/test_metrics_topk.py ..................            [ 52%]

tests/unit/tf/models/test_base.py ............                           [ 55%]

tests/unit/tf/models/test_benchmark.py ..                                [ 55%]

tests/unit/tf/models/test_ranking.py ................                    [ 58%]

tests/unit/tf/models/test_retrieval.py .............................     [ 64%]

tests/unit/tf/prediction_tasks/test_classification.py ..                 [ 65%]

tests/unit/tf/prediction_tasks/test_multi_task.py .......                [ 66%]

tests/unit/tf/prediction_tasks/test_next_item.py ....................    [ 70%]

tests/unit/tf/prediction_tasks/test_regression.py ..                     [ 70%]

tests/unit/tf/prediction_tasks/test_sampling.py ....................     [ 75%]

tests/unit/tf/utils/test_batch.py ....                                   [ 75%]

tests/unit/tf/utils/test_tf_utils.py ..                                  [ 76%]

tests/unit/torch/test_dataset.py .........                               [ 78%]

tests/unit/torch/test_public_api.py .                                    [ 78%]

tests/unit/torch/block/test_base.py ....                                 [ 79%]

tests/unit/torch/block/test_mlp.py .                                     [ 79%]

tests/unit/torch/features/test_continuous.py ..                          [ 79%]

tests/unit/torch/features/test_embedding.py ..............               [ 82%]

tests/unit/torch/features/test_tabular.py ....                           [ 83%]

tests/unit/torch/model/test_head.py ............                         [ 85%]

tests/unit/torch/model/test_model.py ..                                  [ 86%]

tests/unit/torch/tabular/test_aggregation.py ........                    [ 87%]

tests/unit/torch/tabular/test_tabular.py ...                             [ 88%]

tests/unit/torch/tabular/test_transformations.py .......                 [ 89%]

tests/unit/utils/test_schema_utils.py ................................   [ 96%]

tests/unit/xgb/test_xgboost.py .................                         [100%]
=============================== warnings summary ===============================

../../../.local/lib/python3.8/site-packages/flatbuffers/compat.py:19

/var/jenkins_home/.local/lib/python3.8/site-packages/flatbuffers/compat.py:19: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses

import imp
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.

'nearest': pil_image.NEAREST,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.

'bilinear': pil_image.BILINEAR,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.

'bicubic': pil_image.BICUBIC,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.

'hamming': pil_image.HAMMING,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.

'box': pil_image.BOX,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.

'lanczos': pil_image.LANCZOS,
tests/unit/datasets/test_ecommerce.py::test_synthetic_aliccp_raw_data

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-True-10]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-True-9]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-True-8]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-False-10]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-False-9]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-False-8]

tests/unit/tf/test_dataset.py::test_tf_catname_ordering

tests/unit/tf/test_dataset.py::test_tf_map

/usr/local/lib/python3.8/dist-packages/cudf/core/frame.py:384: UserWarning: The deep parameter is ignored and is only included for pandas compatibility.

warnings.warn(
tests/unit/tf/blocks/core/test_index.py: 7 warnings

tests/unit/tf/models/test_retrieval.py: 311 warnings

tests/unit/tf/prediction_tasks/test_next_item.py: 100 warnings

tests/unit/tf/utils/test_batch.py: 4 warnings

/tmp/autograph_generated_file6lqaf52z.py:8: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead

ag.converted_call(ag__.ld(warnings).warn, ("The 'warn' method is deprecated, use 'warning' instead", ag__.ld(DeprecationWarning), 2), None, fscope)
tests/unit/tf/blocks/core/test_transformations.py: 4 warnings

tests/unit/tf/models/test_base.py: 6 warnings

/usr/local/lib/python3.8/dist-packages/tensorflow/python/util/dispatch.py:1082: UserWarning: tf.keras.backend.random_binomial is deprecated, and will be removed in a future version.Please use tf.keras.backend.random_bernoulli instead.

return dispatch_target(*args, **kwargs)
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py::test_matrix_factorization_embedding_export

tests/unit/tf/blocks/retrieval/test_matrix_factorization.py::test_matrix_factorization_embedding_export

tests/unit/tf/blocks/retrieval/test_two_tower.py::test_matrix_factorization_embedding_export

tests/unit/tf/blocks/retrieval/test_two_tower.py::test_matrix_factorization_embedding_export

tests/unit/tf/inputs/test_embedding.py::test_embedding_features_exporting_and_loading_pretrained_initializer

/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py:566: DeprecationWarning: This function is deprecated in favor of cupy.from_dlpack

embeddings_cupy = cupy.fromDlpack(to_dlpack(tf.convert_to_tensor(embeddings)))
tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[True]

tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[True]

tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[False]

tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/numpy/core/numeric.py:2453: DeprecationWarning: elementwise comparison failed; this will raise an error in the future.

return bool(asarray(a1 == a2).all())
tests/unit/torch/block/test_mlp.py::test_mlp_block

/var/jenkins_home/workspace/merlin_models/models/tests/unit/torch/_conftest.py:151: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)

return {key: torch.tensor(value) for key, value in data.items()}
tests/unit/xgb/test_xgboost.py::test_without_dask_client

tests/unit/xgb/test_xgboost.py::TestXGBoost::test_music_regression

tests/unit/xgb/test_xgboost.py::test_gpu_hist_dmatrix[fit_kwargs0-DaskDeviceQuantileDMatrix]

tests/unit/xgb/test_xgboost.py::test_gpu_hist_dmatrix[fit_kwargs1-DaskDMatrix]

tests/unit/xgb/test_xgboost.py::TestEvals::test_multiple

tests/unit/xgb/test_xgboost.py::TestEvals::test_default

tests/unit/xgb/test_xgboost.py::TestEvals::test_train_and_valid

tests/unit/xgb/test_xgboost.py::TestEvals::test_invalid_data

/var/jenkins_home/workspace/merlin_models/models/merlin/models/xgb/init.py:266: UserWarning: Ignoring list columns as inputs to XGBoost model: ['item_genres', 'user_genres'].

warnings.warn(f"Ignoring list columns as inputs to XGBoost model: {list_column_names}.")
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

=========================== short test summary info ============================

SKIPPED [1] tests/unit/implicit/init.py:18: could not import 'implicit': No module named 'implicit'

SKIPPED [1] tests/unit/lightfm/init.py:18: could not import 'lightfm': No module named 'lightfm'

SKIPPED [1] tests/unit/datasets/test_advertising.py:20: No data-dir available, pass it through env variable $INPUT_DATA_DIR

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:62: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:78: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:92: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP

SKIPPED [3] tests/unit/datasets/test_entertainment.py:44: No data-dir available, pass it through env variable $INPUT_DATA_DIR

========== 482 passed, 9 skipped, 466 warnings in 1469.69s (0:24:29) ===========

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/models/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_models] $ /bin/bash /tmp/jenkins14364508015315862096.sh

nvidia-merlin-bot · 2022-07-07T15:51:24Z

Click to view CI Results

GitHub pull request #537 of commit 17df36e442d64dd8a7c40796917eebb4bef05005, no merge conflicts.
Running as SYSTEM
Setting status of 17df36e442d64dd8a7c40796917eebb4bef05005 to PENDING with url https://10.20.13.93:8080/job/merlin_models/594/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_models
using credential nvidia-merlin-bot
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/models/ # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/models/
 > git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/models/ +refs/pull/537/*:refs/remotes/origin/pr/537/* # timeout=10
 > git rev-parse 17df36e442d64dd8a7c40796917eebb4bef05005^{commit} # timeout=10
 > git rev-parse origin/17df36e442d64dd8a7c40796917eebb4bef05005^{commit} # timeout=10
 > git rev-parse 17df36e442d64dd8a7c40796917eebb4bef05005^{commit} # timeout=10
ERROR: Couldn't find any revision to build. Verify the repository and branch configuration for this job.
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash
cd /var/jenkins_home/
CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/models/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log" 
[merlin_models] $ /bin/bash /tmp/jenkins10713747182221278361.sh

nvidia-merlin-bot · 2022-07-07T16:16:09Z

Click to view CI Results

GitHub pull request #537 of commit 5fc561ac63c03cb83a6533d46ee2dd5b23af3c62, no merge conflicts.
Running as SYSTEM
Setting status of 5fc561ac63c03cb83a6533d46ee2dd5b23af3c62 to PENDING with url https://10.20.13.93:8080/job/merlin_models/595/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_models
using credential nvidia-merlin-bot
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/models/ # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/models/
 > git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/models/ +refs/pull/537/*:refs/remotes/origin/pr/537/* # timeout=10
 > git rev-parse 5fc561ac63c03cb83a6533d46ee2dd5b23af3c62^{commit} # timeout=10
Checking out Revision 5fc561ac63c03cb83a6533d46ee2dd5b23af3c62 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 5fc561ac63c03cb83a6533d46ee2dd5b23af3c62 # timeout=10
Commit message: "Enabling only retrieval integration tests"
 > git rev-list --no-walk 3644b6e0bf231b0324b647e5cbf0a48d4adbf25a # timeout=10
[merlin_models] $ /bin/bash /tmp/jenkins17986127781425149000.sh
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: testbook in /var/jenkins_home/.local/lib/python3.8/site-packages (0.4.2)
Requirement already satisfied: nbformat>=5.0.4 in /usr/local/lib/python3.8/dist-packages (from testbook) (5.4.0)
Requirement already satisfied: nbclient>=0.4.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from testbook) (0.5.13)
Requirement already satisfied: traitlets>=5.1 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (5.3.0)
Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.6.1)
Requirement already satisfied: jupyter-core in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.10.0)
Requirement already satisfied: fastjsonschema in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (2.15.3)
Requirement already satisfied: jupyter-client>=6.1.5 in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (7.3.4)
Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (1.5.5)
Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (21.4.0)
Requirement already satisfied: importlib-resources>=1.4.0; python_version < "3.9" in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (5.8.0)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (0.18.1)
Requirement already satisfied: entrypoints in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (0.4)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (2.8.2)
Requirement already satisfied: pyzmq>=23.0 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (23.2.0)
Requirement already satisfied: tornado>=6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (6.1)
Requirement already satisfied: zipp>=3.1.0; python_version < "3.10" in /usr/local/lib/python3.8/dist-packages (from importlib-resources>=1.4.0; python_version < "3.9"->jsonschema>=2.6->nbformat>=5.0.4->testbook) (3.8.0)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.8.2->jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (1.15.0)
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/merlin_models/models, configfile: pyproject.toml
plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 489 items / 2 skipped
tests/unit/config/test_schema.py ....                                    [  0%]

tests/unit/datasets/test_advertising.py .s                               [  1%]

tests/unit/datasets/test_ecommerce.py ..sss                              [  2%]

tests/unit/datasets/test_entertainment.py ....sss.                       [  3%]

tests/unit/datasets/test_social.py .                                     [  4%]

tests/unit/datasets/test_synthetic.py .....                              [  5%]

tests/unit/tf/test_core.py ......                                        [  6%]

tests/unit/tf/test_dataset.py ...............                            [  9%]

tests/unit/tf/test_public_api.py .                                       [  9%]

tests/unit/tf/blocks/test_cross.py ...........                           [ 11%]

tests/unit/tf/blocks/test_dlrm.py ........                               [ 13%]

tests/unit/tf/blocks/test_interactions.py .                              [ 13%]

tests/unit/tf/blocks/test_mlp.py .............................           [ 19%]

tests/unit/tf/blocks/core/test_aggregation.py .........                  [ 21%]

tests/unit/tf/blocks/core/test_base.py ..                                [ 21%]

tests/unit/tf/blocks/core/test_combinators.py .......                    [ 23%]

tests/unit/tf/blocks/core/test_index.py ...                              [ 23%]

tests/unit/tf/blocks/core/test_masking.py .......                        [ 25%]

tests/unit/tf/blocks/core/test_tabular.py ....                           [ 26%]

tests/unit/tf/blocks/core/test_transformations.py ...........            [ 28%]

tests/unit/tf/blocks/retrieval/test_matrix_factorization.py ..           [ 28%]

tests/unit/tf/blocks/retrieval/test_two_tower.py ...........             [ 31%]

tests/unit/tf/examples/test_01_getting_started.py .                      [ 31%]

tests/unit/tf/examples/test_02_dataschema.py .                           [ 31%]

tests/unit/tf/examples/test_03_exploring_different_models.py .           [ 31%]

tests/unit/tf/examples/test_04_export_ranking_models.py .                [ 31%]

tests/unit/tf/examples/test_05_export_retrieval_model.py .               [ 32%]

tests/unit/tf/examples/test_06_advanced_own_architecture.py .            [ 32%]

tests/unit/tf/examples/test_usecase_pretrained_embeddings.py .           [ 32%]

tests/unit/tf/inputs/test_continuous.py .....                            [ 33%]

tests/unit/tf/inputs/test_embedding.py ..........................        [ 38%]

tests/unit/tf/inputs/test_tabular.py .......                             [ 40%]

tests/unit/tf/layers/test_queue.py ..............                        [ 43%]

tests/unit/tf/losses/test_losses.py .......................              [ 47%]

tests/unit/tf/metrics/test_metrics_popularity.py .....                   [ 48%]

tests/unit/tf/metrics/test_metrics_topk.py ..................            [ 52%]

tests/unit/tf/models/test_base.py ............                           [ 55%]

tests/unit/tf/models/test_benchmark.py ..                                [ 55%]

tests/unit/tf/models/test_ranking.py ................                    [ 58%]

tests/unit/tf/models/test_retrieval.py .............................     [ 64%]

tests/unit/tf/prediction_tasks/test_classification.py ..                 [ 65%]

tests/unit/tf/prediction_tasks/test_multi_task.py .......                [ 66%]

tests/unit/tf/prediction_tasks/test_next_item.py ....................    [ 70%]

tests/unit/tf/prediction_tasks/test_regression.py ..                     [ 70%]

tests/unit/tf/prediction_tasks/test_sampling.py ....................     [ 75%]

tests/unit/tf/utils/test_batch.py ....                                   [ 75%]

tests/unit/tf/utils/test_tf_utils.py ..                                  [ 76%]

tests/unit/torch/test_dataset.py .........                               [ 78%]

tests/unit/torch/test_public_api.py .                                    [ 78%]

tests/unit/torch/block/test_base.py ....                                 [ 79%]

tests/unit/torch/block/test_mlp.py .                                     [ 79%]

tests/unit/torch/features/test_continuous.py ..                          [ 79%]

tests/unit/torch/features/test_embedding.py ..............               [ 82%]

tests/unit/torch/features/test_tabular.py ....                           [ 83%]

tests/unit/torch/model/test_head.py ............                         [ 85%]

tests/unit/torch/model/test_model.py ..                                  [ 86%]

tests/unit/torch/tabular/test_aggregation.py ........                    [ 87%]

tests/unit/torch/tabular/test_tabular.py ...                             [ 88%]

tests/unit/torch/tabular/test_transformations.py .......                 [ 89%]

tests/unit/utils/test_schema_utils.py ................................   [ 96%]

tests/unit/xgb/test_xgboost.py .................                         [100%]
=============================== warnings summary ===============================

../../../.local/lib/python3.8/site-packages/flatbuffers/compat.py:19

/var/jenkins_home/.local/lib/python3.8/site-packages/flatbuffers/compat.py:19: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses

import imp
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.

'nearest': pil_image.NEAREST,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.

'bilinear': pil_image.BILINEAR,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.

'bicubic': pil_image.BICUBIC,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.

'hamming': pil_image.HAMMING,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.

'box': pil_image.BOX,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.

'lanczos': pil_image.LANCZOS,
tests/unit/datasets/test_ecommerce.py::test_synthetic_aliccp_raw_data

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-True-10]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-True-9]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-True-8]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-False-10]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-False-9]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-False-8]

tests/unit/tf/test_dataset.py::test_tf_catname_ordering

tests/unit/tf/test_dataset.py::test_tf_map

/usr/local/lib/python3.8/dist-packages/cudf/core/frame.py:384: UserWarning: The deep parameter is ignored and is only included for pandas compatibility.

warnings.warn(
tests/unit/tf/blocks/core/test_index.py: 7 warnings

tests/unit/tf/models/test_retrieval.py: 311 warnings

tests/unit/tf/prediction_tasks/test_next_item.py: 100 warnings

tests/unit/tf/utils/test_batch.py: 4 warnings

/tmp/autograph_generated_file7i8ehoqk.py:8: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead

ag.converted_call(ag__.ld(warnings).warn, ("The 'warn' method is deprecated, use 'warning' instead", ag__.ld(DeprecationWarning), 2), None, fscope)
tests/unit/tf/blocks/core/test_transformations.py: 4 warnings

tests/unit/tf/models/test_base.py: 6 warnings

/usr/local/lib/python3.8/dist-packages/tensorflow/python/util/dispatch.py:1082: UserWarning: tf.keras.backend.random_binomial is deprecated, and will be removed in a future version.Please use tf.keras.backend.random_bernoulli instead.

return dispatch_target(*args, **kwargs)
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py::test_matrix_factorization_embedding_export

tests/unit/tf/blocks/retrieval/test_matrix_factorization.py::test_matrix_factorization_embedding_export

tests/unit/tf/blocks/retrieval/test_two_tower.py::test_matrix_factorization_embedding_export

tests/unit/tf/blocks/retrieval/test_two_tower.py::test_matrix_factorization_embedding_export

tests/unit/tf/inputs/test_embedding.py::test_embedding_features_exporting_and_loading_pretrained_initializer

/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py:566: DeprecationWarning: This function is deprecated in favor of cupy.from_dlpack

embeddings_cupy = cupy.fromDlpack(to_dlpack(tf.convert_to_tensor(embeddings)))
tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[True]

tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[True]

tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[False]

tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/numpy/core/numeric.py:2453: DeprecationWarning: elementwise comparison failed; this will raise an error in the future.

return bool(asarray(a1 == a2).all())
tests/unit/torch/block/test_mlp.py::test_mlp_block

/var/jenkins_home/workspace/merlin_models/models/tests/unit/torch/_conftest.py:151: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)

return {key: torch.tensor(value) for key, value in data.items()}
tests/unit/xgb/test_xgboost.py::test_without_dask_client

tests/unit/xgb/test_xgboost.py::TestXGBoost::test_music_regression

tests/unit/xgb/test_xgboost.py::test_gpu_hist_dmatrix[fit_kwargs0-DaskDeviceQuantileDMatrix]

tests/unit/xgb/test_xgboost.py::test_gpu_hist_dmatrix[fit_kwargs1-DaskDMatrix]

tests/unit/xgb/test_xgboost.py::TestEvals::test_multiple

tests/unit/xgb/test_xgboost.py::TestEvals::test_default

tests/unit/xgb/test_xgboost.py::TestEvals::test_train_and_valid

tests/unit/xgb/test_xgboost.py::TestEvals::test_invalid_data

/var/jenkins_home/workspace/merlin_models/models/merlin/models/xgb/init.py:266: UserWarning: Ignoring list columns as inputs to XGBoost model: ['item_genres', 'user_genres'].

warnings.warn(f"Ignoring list columns as inputs to XGBoost model: {list_column_names}.")
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

=========================== short test summary info ============================

SKIPPED [1] tests/unit/implicit/init.py:18: could not import 'implicit': No module named 'implicit'

SKIPPED [1] tests/unit/lightfm/init.py:18: could not import 'lightfm': No module named 'lightfm'

SKIPPED [1] tests/unit/datasets/test_advertising.py:20: No data-dir available, pass it through env variable $INPUT_DATA_DIR

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:62: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:78: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:92: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP

SKIPPED [3] tests/unit/datasets/test_entertainment.py:44: No data-dir available, pass it through env variable $INPUT_DATA_DIR

========== 482 passed, 9 skipped, 466 warnings in 1468.81s (0:24:28) ===========

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/models/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_models] $ /bin/bash /tmp/jenkins4099655643709819532.sh

nvidia-merlin-bot · 2022-07-07T16:44:49Z

Click to view CI Results

GitHub pull request #537 of commit 134f8b7ab615bc33233ac52332a4d658c3017c13, no merge conflicts.
Running as SYSTEM
Setting status of 134f8b7ab615bc33233ac52332a4d658c3017c13 to PENDING with url https://10.20.13.93:8080/job/merlin_models/596/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_models
using credential nvidia-merlin-bot
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/models/ # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/models/
 > git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/models/ +refs/pull/537/*:refs/remotes/origin/pr/537/* # timeout=10
 > git rev-parse 134f8b7ab615bc33233ac52332a4d658c3017c13^{commit} # timeout=10
Checking out Revision 134f8b7ab615bc33233ac52332a4d658c3017c13 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 134f8b7ab615bc33233ac52332a4d658c3017c13 # timeout=10
Commit message: "Fixing wandb import order"
 > git rev-list --no-walk 5fc561ac63c03cb83a6533d46ee2dd5b23af3c62 # timeout=10
[merlin_models] $ /bin/bash /tmp/jenkins2015098421208784611.sh
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: testbook in /var/jenkins_home/.local/lib/python3.8/site-packages (0.4.2)
Requirement already satisfied: nbformat>=5.0.4 in /usr/local/lib/python3.8/dist-packages (from testbook) (5.4.0)
Requirement already satisfied: nbclient>=0.4.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from testbook) (0.5.13)
Requirement already satisfied: traitlets>=5.1 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (5.3.0)
Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.6.1)
Requirement already satisfied: jupyter-core in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.10.0)
Requirement already satisfied: fastjsonschema in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (2.15.3)
Requirement already satisfied: jupyter-client>=6.1.5 in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (7.3.4)
Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (1.5.5)
Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (21.4.0)
Requirement already satisfied: importlib-resources>=1.4.0; python_version < "3.9" in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (5.8.0)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (0.18.1)
Requirement already satisfied: entrypoints in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (0.4)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (2.8.2)
Requirement already satisfied: pyzmq>=23.0 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (23.2.0)
Requirement already satisfied: tornado>=6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (6.1)
Requirement already satisfied: zipp>=3.1.0; python_version < "3.10" in /usr/local/lib/python3.8/dist-packages (from importlib-resources>=1.4.0; python_version < "3.9"->jsonschema>=2.6->nbformat>=5.0.4->testbook) (3.8.0)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.8.2->jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (1.15.0)
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/merlin_models/models, configfile: pyproject.toml
plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 489 items / 2 skipped
tests/unit/config/test_schema.py ....                                    [  0%]

tests/unit/datasets/test_advertising.py .s                               [  1%]

tests/unit/datasets/test_ecommerce.py ..sss                              [  2%]

tests/unit/datasets/test_entertainment.py ....sss.                       [  3%]

tests/unit/datasets/test_social.py .                                     [  4%]

tests/unit/datasets/test_synthetic.py .....                              [  5%]

tests/unit/tf/test_core.py ......                                        [  6%]

tests/unit/tf/test_dataset.py ...............                            [  9%]

tests/unit/tf/test_public_api.py .                                       [  9%]

tests/unit/tf/blocks/test_cross.py ...........                           [ 11%]

tests/unit/tf/blocks/test_dlrm.py ........                               [ 13%]

tests/unit/tf/blocks/test_interactions.py .                              [ 13%]

tests/unit/tf/blocks/test_mlp.py .............................           [ 19%]

tests/unit/tf/blocks/core/test_aggregation.py .........                  [ 21%]

tests/unit/tf/blocks/core/test_base.py ..                                [ 21%]

tests/unit/tf/blocks/core/test_combinators.py .......                    [ 23%]

tests/unit/tf/blocks/core/test_index.py ...                              [ 23%]

tests/unit/tf/blocks/core/test_masking.py .......                        [ 25%]

tests/unit/tf/blocks/core/test_tabular.py ....                           [ 26%]

tests/unit/tf/blocks/core/test_transformations.py ...........            [ 28%]

tests/unit/tf/blocks/retrieval/test_matrix_factorization.py ..           [ 28%]

tests/unit/tf/blocks/retrieval/test_two_tower.py ...........             [ 31%]

tests/unit/tf/examples/test_01_getting_started.py .                      [ 31%]

tests/unit/tf/examples/test_02_dataschema.py .                           [ 31%]

tests/unit/tf/examples/test_03_exploring_different_models.py .           [ 31%]

tests/unit/tf/examples/test_04_export_ranking_models.py .                [ 31%]

tests/unit/tf/examples/test_05_export_retrieval_model.py .               [ 32%]

tests/unit/tf/examples/test_06_advanced_own_architecture.py .            [ 32%]

tests/unit/tf/examples/test_usecase_pretrained_embeddings.py .           [ 32%]

tests/unit/tf/inputs/test_continuous.py .....                            [ 33%]

tests/unit/tf/inputs/test_embedding.py ..........................        [ 38%]

tests/unit/tf/inputs/test_tabular.py .......                             [ 40%]

tests/unit/tf/layers/test_queue.py ..............                        [ 43%]

tests/unit/tf/losses/test_losses.py .......................              [ 47%]

tests/unit/tf/metrics/test_metrics_popularity.py .....                   [ 48%]

tests/unit/tf/metrics/test_metrics_topk.py ..................            [ 52%]

tests/unit/tf/models/test_base.py ............                           [ 55%]

tests/unit/tf/models/test_benchmark.py ..                                [ 55%]

tests/unit/tf/models/test_ranking.py ................                    [ 58%]

tests/unit/tf/models/test_retrieval.py .............................     [ 64%]

tests/unit/tf/prediction_tasks/test_classification.py ..                 [ 65%]

tests/unit/tf/prediction_tasks/test_multi_task.py .......                [ 66%]

tests/unit/tf/prediction_tasks/test_next_item.py ....................    [ 70%]

tests/unit/tf/prediction_tasks/test_regression.py ..                     [ 70%]

tests/unit/tf/prediction_tasks/test_sampling.py ....................     [ 75%]

tests/unit/tf/utils/test_batch.py ....                                   [ 75%]

tests/unit/tf/utils/test_tf_utils.py ..                                  [ 76%]

tests/unit/torch/test_dataset.py .........                               [ 78%]

tests/unit/torch/test_public_api.py .                                    [ 78%]

tests/unit/torch/block/test_base.py ....                                 [ 79%]

tests/unit/torch/block/test_mlp.py .                                     [ 79%]

tests/unit/torch/features/test_continuous.py ..                          [ 79%]

tests/unit/torch/features/test_embedding.py ..............               [ 82%]

tests/unit/torch/features/test_tabular.py ....                           [ 83%]

tests/unit/torch/model/test_head.py ............                         [ 85%]

tests/unit/torch/model/test_model.py ..                                  [ 86%]

tests/unit/torch/tabular/test_aggregation.py ........                    [ 87%]

tests/unit/torch/tabular/test_tabular.py ...                             [ 88%]

tests/unit/torch/tabular/test_transformations.py .......                 [ 89%]

tests/unit/utils/test_schema_utils.py ................................   [ 96%]

tests/unit/xgb/test_xgboost.py .................                         [100%]
=============================== warnings summary ===============================

../../../.local/lib/python3.8/site-packages/flatbuffers/compat.py:19

/var/jenkins_home/.local/lib/python3.8/site-packages/flatbuffers/compat.py:19: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses

import imp
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.

'nearest': pil_image.NEAREST,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.

'bilinear': pil_image.BILINEAR,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.

'bicubic': pil_image.BICUBIC,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.

'hamming': pil_image.HAMMING,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.

'box': pil_image.BOX,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.

'lanczos': pil_image.LANCZOS,
tests/unit/datasets/test_ecommerce.py::test_synthetic_aliccp_raw_data

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-True-10]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-True-9]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-True-8]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-False-10]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-False-9]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-False-8]

tests/unit/tf/test_dataset.py::test_tf_catname_ordering

tests/unit/tf/test_dataset.py::test_tf_map

/usr/local/lib/python3.8/dist-packages/cudf/core/frame.py:384: UserWarning: The deep parameter is ignored and is only included for pandas compatibility.

warnings.warn(
tests/unit/tf/blocks/core/test_index.py: 7 warnings

tests/unit/tf/models/test_retrieval.py: 311 warnings

tests/unit/tf/prediction_tasks/test_next_item.py: 100 warnings

tests/unit/tf/utils/test_batch.py: 4 warnings

/tmp/autograph_generated_filevess8fmq.py:8: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead

ag.converted_call(ag__.ld(warnings).warn, ("The 'warn' method is deprecated, use 'warning' instead", ag__.ld(DeprecationWarning), 2), None, fscope)
tests/unit/tf/blocks/core/test_transformations.py: 4 warnings

tests/unit/tf/models/test_base.py: 6 warnings

/usr/local/lib/python3.8/dist-packages/tensorflow/python/util/dispatch.py:1082: UserWarning: tf.keras.backend.random_binomial is deprecated, and will be removed in a future version.Please use tf.keras.backend.random_bernoulli instead.

return dispatch_target(*args, **kwargs)
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py::test_matrix_factorization_embedding_export

tests/unit/tf/blocks/retrieval/test_matrix_factorization.py::test_matrix_factorization_embedding_export

tests/unit/tf/blocks/retrieval/test_two_tower.py::test_matrix_factorization_embedding_export

tests/unit/tf/blocks/retrieval/test_two_tower.py::test_matrix_factorization_embedding_export

tests/unit/tf/inputs/test_embedding.py::test_embedding_features_exporting_and_loading_pretrained_initializer

/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py:566: DeprecationWarning: This function is deprecated in favor of cupy.from_dlpack

embeddings_cupy = cupy.fromDlpack(to_dlpack(tf.convert_to_tensor(embeddings)))
tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[True]

tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[True]

tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[False]

tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/numpy/core/numeric.py:2453: DeprecationWarning: elementwise comparison failed; this will raise an error in the future.

return bool(asarray(a1 == a2).all())
tests/unit/torch/block/test_mlp.py::test_mlp_block

/var/jenkins_home/workspace/merlin_models/models/tests/unit/torch/_conftest.py:151: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)

return {key: torch.tensor(value) for key, value in data.items()}
tests/unit/xgb/test_xgboost.py::test_without_dask_client

tests/unit/xgb/test_xgboost.py::TestXGBoost::test_music_regression

tests/unit/xgb/test_xgboost.py::test_gpu_hist_dmatrix[fit_kwargs0-DaskDeviceQuantileDMatrix]

tests/unit/xgb/test_xgboost.py::test_gpu_hist_dmatrix[fit_kwargs1-DaskDMatrix]

tests/unit/xgb/test_xgboost.py::TestEvals::test_multiple

tests/unit/xgb/test_xgboost.py::TestEvals::test_default

tests/unit/xgb/test_xgboost.py::TestEvals::test_train_and_valid

tests/unit/xgb/test_xgboost.py::TestEvals::test_invalid_data

/var/jenkins_home/workspace/merlin_models/models/merlin/models/xgb/init.py:266: UserWarning: Ignoring list columns as inputs to XGBoost model: ['item_genres', 'user_genres'].

warnings.warn(f"Ignoring list columns as inputs to XGBoost model: {list_column_names}.")
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

=========================== short test summary info ============================

SKIPPED [1] tests/unit/implicit/init.py:18: could not import 'implicit': No module named 'implicit'

SKIPPED [1] tests/unit/lightfm/init.py:18: could not import 'lightfm': No module named 'lightfm'

SKIPPED [1] tests/unit/datasets/test_advertising.py:20: No data-dir available, pass it through env variable $INPUT_DATA_DIR

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:62: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:78: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:92: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP

SKIPPED [3] tests/unit/datasets/test_entertainment.py:44: No data-dir available, pass it through env variable $INPUT_DATA_DIR

========== 482 passed, 9 skipped, 466 warnings in 1466.58s (0:24:26) ===========

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/models/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_models] $ /bin/bash /tmp/jenkins6729719812518053167.sh

nvidia-merlin-bot · 2022-07-07T21:08:46Z

Click to view CI Results

GitHub pull request #537 of commit bddcb091fdd7a097b693922d1be3bb0a24f3825e, no merge conflicts.
Running as SYSTEM
Setting status of bddcb091fdd7a097b693922d1be3bb0a24f3825e to PENDING with url https://10.20.13.93:8080/job/merlin_models/603/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_models
using credential nvidia-merlin-bot
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/models/ # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/models/
 > git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/models/ +refs/pull/537/*:refs/remotes/origin/pr/537/* # timeout=10
 > git rev-parse bddcb091fdd7a097b693922d1be3bb0a24f3825e^{commit} # timeout=10
Checking out Revision bddcb091fdd7a097b693922d1be3bb0a24f3825e (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f bddcb091fdd7a097b693922d1be3bb0a24f3825e # timeout=10
Commit message: "Refactored retrieval tests to run both as integration and unit tests"
 > git rev-list --no-walk 156284b84e65d105da4e8491a2921524a77075f0 # timeout=10
[merlin_models] $ /bin/bash /tmp/jenkins15993686129055987073.sh
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: testbook in /var/jenkins_home/.local/lib/python3.8/site-packages (0.4.2)
Requirement already satisfied: nbformat>=5.0.4 in /usr/local/lib/python3.8/dist-packages (from testbook) (5.4.0)
Requirement already satisfied: nbclient>=0.4.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from testbook) (0.5.13)
Requirement already satisfied: traitlets>=5.1 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (5.3.0)
Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.6.1)
Requirement already satisfied: jupyter-core in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.10.0)
Requirement already satisfied: fastjsonschema in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (2.15.3)
Requirement already satisfied: jupyter-client>=6.1.5 in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (7.3.4)
Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (1.5.5)
Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (21.4.0)
Requirement already satisfied: importlib-resources>=1.4.0; python_version < "3.9" in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (5.8.0)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (0.18.1)
Requirement already satisfied: entrypoints in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (0.4)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (2.8.2)
Requirement already satisfied: pyzmq>=23.0 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (23.2.0)
Requirement already satisfied: tornado>=6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (6.1)
Requirement already satisfied: zipp>=3.1.0; python_version < "3.10" in /usr/local/lib/python3.8/dist-packages (from importlib-resources>=1.4.0; python_version < "3.9"->jsonschema>=2.6->nbformat>=5.0.4->testbook) (3.8.0)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.8.2->jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (1.15.0)
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/merlin_models/models, configfile: pyproject.toml
plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 460 items / 1 error / 2 skipped
==================================== ERRORS ====================================

___________ ERROR collecting tests/unit/tf/models/test_retrieval.py ____________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_models/models/tests/unit/tf/models/test_retrieval.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/tf/models/test_retrieval.py:17: in 

from tests.common.tf.retrieval import retrieval_tests_common

tests/common/tf/retrieval/retrieval_tests_common.py:19: in 

import fiddle as fdl

E   ModuleNotFoundError: No module named 'fiddle'

=============================== warnings summary ===============================

../../../.local/lib/python3.8/site-packages/flatbuffers/compat.py:19

/var/jenkins_home/.local/lib/python3.8/site-packages/flatbuffers/compat.py:19: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses

import imp
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.

'nearest': pil_image.NEAREST,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.

'bilinear': pil_image.BILINEAR,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.

'bicubic': pil_image.BICUBIC,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.

'hamming': pil_image.HAMMING,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.

'box': pil_image.BOX,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.

'lanczos': pil_image.LANCZOS,
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

=========================== short test summary info ============================

SKIPPED [1] tests/unit/implicit/init.py:18: could not import 'implicit': No module named 'implicit'

SKIPPED [1] tests/unit/lightfm/init.py:18: could not import 'lightfm': No module named 'lightfm'

!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!

=================== 2 skipped, 7 warnings, 1 error in 3.29s ====================

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/models/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_models] $ /bin/bash /tmp/jenkins11951728044823492179.sh

nvidia-merlin-bot · 2022-07-07T21:29:45Z

Click to view CI Results

GitHub pull request #537 of commit 70265b44872106b044ffa158b32ec50d4abf0ced, no merge conflicts.
Running as SYSTEM
Setting status of 70265b44872106b044ffa158b32ec50d4abf0ced to PENDING with url https://10.20.13.93:8080/job/merlin_models/604/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_models
using credential nvidia-merlin-bot
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/models/ # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/models/
 > git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/models/ +refs/pull/537/*:refs/remotes/origin/pr/537/* # timeout=10
 > git rev-parse 70265b44872106b044ffa158b32ec50d4abf0ced^{commit} # timeout=10
Checking out Revision 70265b44872106b044ffa158b32ec50d4abf0ced (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 70265b44872106b044ffa158b32ec50d4abf0ced # timeout=10
Commit message: "Fixed wandb import"
 > git rev-list --no-walk bddcb091fdd7a097b693922d1be3bb0a24f3825e # timeout=10
[merlin_models] $ /bin/bash /tmp/jenkins21449374532572595.sh
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: testbook in /var/jenkins_home/.local/lib/python3.8/site-packages (0.4.2)
Requirement already satisfied: nbformat>=5.0.4 in /usr/local/lib/python3.8/dist-packages (from testbook) (5.4.0)
Requirement already satisfied: nbclient>=0.4.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from testbook) (0.5.13)
Requirement already satisfied: traitlets>=5.1 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (5.3.0)
Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.6.1)
Requirement already satisfied: jupyter-core in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.10.0)
Requirement already satisfied: fastjsonschema in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (2.15.3)
Requirement already satisfied: jupyter-client>=6.1.5 in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (7.3.4)
Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (1.5.5)
Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (21.4.0)
Requirement already satisfied: importlib-resources>=1.4.0; python_version < "3.9" in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (5.8.0)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (0.18.1)
Requirement already satisfied: entrypoints in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (0.4)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (2.8.2)
Requirement already satisfied: pyzmq>=23.0 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (23.2.0)
Requirement already satisfied: tornado>=6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (6.1)
Requirement already satisfied: zipp>=3.1.0; python_version < "3.10" in /usr/local/lib/python3.8/dist-packages (from importlib-resources>=1.4.0; python_version < "3.9"->jsonschema>=2.6->nbformat>=5.0.4->testbook) (3.8.0)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.8.2->jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (1.15.0)
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/merlin_models/models, configfile: pyproject.toml
plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 460 items / 1 error / 2 skipped
==================================== ERRORS ====================================

___________ ERROR collecting tests/unit/tf/models/test_retrieval.py ____________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_models/models/tests/unit/tf/models/test_retrieval.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/tf/models/test_retrieval.py:17: in 

from tests.common.tf.retrieval import retrieval_tests_common

tests/common/tf/retrieval/retrieval_tests_common.py:19: in 

import fiddle as fdl

E   ModuleNotFoundError: No module named 'fiddle'

=============================== warnings summary ===============================

../../../.local/lib/python3.8/site-packages/flatbuffers/compat.py:19

/var/jenkins_home/.local/lib/python3.8/site-packages/flatbuffers/compat.py:19: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses

import imp
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.

'nearest': pil_image.NEAREST,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.

'bilinear': pil_image.BILINEAR,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.

'bicubic': pil_image.BICUBIC,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.

'hamming': pil_image.HAMMING,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.

'box': pil_image.BOX,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.

'lanczos': pil_image.LANCZOS,
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

=========================== short test summary info ============================

SKIPPED [1] tests/unit/implicit/init.py:18: could not import 'implicit': No module named 'implicit'

SKIPPED [1] tests/unit/lightfm/init.py:18: could not import 'lightfm': No module named 'lightfm'

!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!

=================== 2 skipped, 7 warnings, 1 error in 3.23s ====================

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/models/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_models] $ /bin/bash /tmp/jenkins5034085445218266901.sh

gabrielspmoreira · 2022-07-07T23:03:13Z

rerun tests

nvidia-merlin-bot · 2022-07-07T23:03:35Z

Click to view CI Results

GitHub pull request #537 of commit 70265b44872106b044ffa158b32ec50d4abf0ced, no merge conflicts.
Running as SYSTEM
Setting status of 70265b44872106b044ffa158b32ec50d4abf0ced to PENDING with url https://10.20.13.93:8080/job/merlin_models/605/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_models
using credential nvidia-merlin-bot
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/models/ # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/models/
 > git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/models/ +refs/pull/537/*:refs/remotes/origin/pr/537/* # timeout=10
 > git rev-parse 70265b44872106b044ffa158b32ec50d4abf0ced^{commit} # timeout=10
Checking out Revision 70265b44872106b044ffa158b32ec50d4abf0ced (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 70265b44872106b044ffa158b32ec50d4abf0ced # timeout=10
Commit message: "Fixed wandb import"
 > git rev-list --no-walk 70265b44872106b044ffa158b32ec50d4abf0ced # timeout=10
[merlin_models] $ /bin/bash /tmp/jenkins12936353116818878306.sh
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: testbook in /var/jenkins_home/.local/lib/python3.8/site-packages (0.4.2)
Requirement already satisfied: nbformat>=5.0.4 in /usr/local/lib/python3.8/dist-packages (from testbook) (5.4.0)
Requirement already satisfied: nbclient>=0.4.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from testbook) (0.5.13)
Requirement already satisfied: traitlets>=5.1 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (5.3.0)
Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.6.1)
Requirement already satisfied: jupyter-core in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.10.0)
Requirement already satisfied: fastjsonschema in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (2.15.3)
Requirement already satisfied: jupyter-client>=6.1.5 in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (7.3.4)
Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (1.5.5)
Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (21.4.0)
Requirement already satisfied: importlib-resources>=1.4.0; python_version < "3.9" in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (5.8.0)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (0.18.1)
Requirement already satisfied: entrypoints in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (0.4)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (2.8.2)
Requirement already satisfied: pyzmq>=23.0 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (23.2.0)
Requirement already satisfied: tornado>=6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (6.1)
Requirement already satisfied: zipp>=3.1.0; python_version < "3.10" in /usr/local/lib/python3.8/dist-packages (from importlib-resources>=1.4.0; python_version < "3.9"->jsonschema>=2.6->nbformat>=5.0.4->testbook) (3.8.0)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.8.2->jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (1.15.0)
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/merlin_models/models, configfile: pyproject.toml
plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 460 items / 1 error / 2 skipped
==================================== ERRORS ====================================

___________ ERROR collecting tests/unit/tf/models/test_retrieval.py ____________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_models/models/tests/unit/tf/models/test_retrieval.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/tf/models/test_retrieval.py:17: in 

from tests.common.tf.retrieval import retrieval_tests_common

tests/common/tf/retrieval/retrieval_tests_common.py:19: in 

import fiddle as fdl

E   ModuleNotFoundError: No module named 'fiddle'

=============================== warnings summary ===============================

../../../.local/lib/python3.8/site-packages/flatbuffers/compat.py:19

/var/jenkins_home/.local/lib/python3.8/site-packages/flatbuffers/compat.py:19: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses

import imp
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.

'nearest': pil_image.NEAREST,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.

'bilinear': pil_image.BILINEAR,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.

'bicubic': pil_image.BICUBIC,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.

'hamming': pil_image.HAMMING,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.

'box': pil_image.BOX,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.

'lanczos': pil_image.LANCZOS,
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

=========================== short test summary info ============================

SKIPPED [1] tests/unit/implicit/init.py:18: could not import 'implicit': No module named 'implicit'

SKIPPED [1] tests/unit/lightfm/init.py:18: could not import 'lightfm': No module named 'lightfm'

!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!

=================== 2 skipped, 7 warnings, 1 error in 3.24s ====================

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/models/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_models] $ /bin/bash /tmp/jenkins5829438308839265233.sh

gabrielspmoreira · 2022-07-08T12:16:35Z

rerun tests

nvidia-merlin-bot · 2022-07-08T12:16:55Z

Click to view CI Results

GitHub pull request #537 of commit 70265b44872106b044ffa158b32ec50d4abf0ced, no merge conflicts.
Running as SYSTEM
Setting status of 70265b44872106b044ffa158b32ec50d4abf0ced to PENDING with url https://10.20.13.93:8080/job/merlin_models/612/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_models
using credential nvidia-merlin-bot
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/models/ # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/models/
 > git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/models/ +refs/pull/537/*:refs/remotes/origin/pr/537/* # timeout=10
 > git rev-parse 70265b44872106b044ffa158b32ec50d4abf0ced^{commit} # timeout=10
Checking out Revision 70265b44872106b044ffa158b32ec50d4abf0ced (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 70265b44872106b044ffa158b32ec50d4abf0ced # timeout=10
Commit message: "Fixed wandb import"
 > git rev-list --no-walk b177515f04536a088d7f2129e9e60cc6015381f6 # timeout=10
[merlin_models] $ /bin/bash /tmp/jenkins12493541900259494419.sh
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: testbook in /usr/local/lib/python3.8/dist-packages (0.4.2)
Requirement already satisfied: nbformat>=5.0.4 in /usr/local/lib/python3.8/dist-packages (from testbook) (5.4.0)
Requirement already satisfied: nbclient>=0.4.0 in /usr/local/lib/python3.8/dist-packages (from testbook) (0.6.5)
Requirement already satisfied: traitlets>=5.1 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (5.3.0)
Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.6.1)
Requirement already satisfied: jupyter-core in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.10.0)
Requirement already satisfied: fastjsonschema in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (2.15.3)
Requirement already satisfied: jupyter-client>=6.1.5 in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (7.3.4)
Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (1.5.5)
Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (21.4.0)
Requirement already satisfied: importlib-resources>=1.4.0; python_version < "3.9" in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (5.8.0)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (0.18.1)
Requirement already satisfied: entrypoints in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (0.4)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (2.8.2)
Requirement already satisfied: pyzmq>=23.0 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (23.2.0)
Requirement already satisfied: tornado>=6.0 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (6.1)
Requirement already satisfied: zipp>=3.1.0; python_version < "3.10" in /usr/local/lib/python3.8/dist-packages (from importlib-resources>=1.4.0; python_version < "3.9"->jsonschema>=2.6->nbformat>=5.0.4->testbook) (3.8.0)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.8.2->jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (1.15.0)
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/merlin_models/models, configfile: pyproject.toml
plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 460 items / 1 error / 2 skipped
==================================== ERRORS ====================================

___________ ERROR collecting tests/unit/tf/models/test_retrieval.py ____________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_models/models/tests/unit/tf/models/test_retrieval.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/tf/models/test_retrieval.py:17: in 

from tests.common.tf.retrieval import retrieval_tests_common

tests/common/tf/retrieval/retrieval_tests_common.py:19: in 

import fiddle as fdl

E   ModuleNotFoundError: No module named 'fiddle'

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.

'nearest': pil_image.NEAREST,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.

'bilinear': pil_image.BILINEAR,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.

'bicubic': pil_image.BICUBIC,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.

'hamming': pil_image.HAMMING,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.

'box': pil_image.BOX,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.

'lanczos': pil_image.LANCZOS,
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

=========================== short test summary info ============================

SKIPPED [1] tests/unit/implicit/init.py:18: could not import 'implicit': No module named 'implicit'

SKIPPED [1] tests/unit/lightfm/init.py:18: could not import 'lightfm': No module named 'lightfm'

!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!

=================== 2 skipped, 6 warnings, 1 error in 3.25s ====================

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/models/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_models] $ /bin/bash /tmp/jenkins12149609240052428998.sh

gabrielspmoreira · 2022-07-08T14:58:53Z

rerun tests

nvidia-merlin-bot · 2022-07-08T16:28:09Z

Click to view CI Results

GitHub pull request #537 of commit 5382ff213c819c11c5ef8c61e020b67bf33de6e5, no merge conflicts.
Running as SYSTEM
Setting status of 5382ff213c819c11c5ef8c61e020b67bf33de6e5 to PENDING with url https://10.20.13.93:8080/job/merlin_models/619/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_models
using credential nvidia-merlin-bot
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/models/ # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/models/
 > git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/models/ +refs/pull/537/*:refs/remotes/origin/pr/537/* # timeout=10
 > git rev-parse 5382ff213c819c11c5ef8c61e020b67bf33de6e5^{commit} # timeout=10
Checking out Revision 5382ff213c819c11c5ef8c61e020b67bf33de6e5 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 5382ff213c819c11c5ef8c61e020b67bf33de6e5 # timeout=10
Commit message: "Changed retrieval unit test to train for a full epoch to check the runtime using the P100 machine available for PR testing"
 > git rev-list --no-walk b1d22554248a676e36cc94b2df7b1b5f0c035941 # timeout=10
[merlin_models] $ /bin/bash /tmp/jenkins15734170389674513421.sh
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: testbook in /usr/local/lib/python3.8/dist-packages (0.4.2)
Requirement already satisfied: nbformat>=5.0.4 in /usr/local/lib/python3.8/dist-packages (from testbook) (5.4.0)
Requirement already satisfied: nbclient>=0.4.0 in /usr/local/lib/python3.8/dist-packages (from testbook) (0.6.5)
Requirement already satisfied: traitlets>=5.1 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (5.3.0)
Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.6.1)
Requirement already satisfied: jupyter-core in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.10.0)
Requirement already satisfied: fastjsonschema in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (2.15.3)
Requirement already satisfied: jupyter-client>=6.1.5 in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (7.3.4)
Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (1.5.5)
Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (21.4.0)
Requirement already satisfied: importlib-resources>=1.4.0; python_version < "3.9" in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (5.8.0)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (0.18.1)
Requirement already satisfied: entrypoints in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (0.4)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (2.8.2)
Requirement already satisfied: pyzmq>=23.0 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (23.2.0)
Requirement already satisfied: tornado>=6.0 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (6.1)
Requirement already satisfied: zipp>=3.1.0; python_version < "3.10" in /usr/local/lib/python3.8/dist-packages (from importlib-resources>=1.4.0; python_version < "3.9"->jsonschema>=2.6->nbformat>=5.0.4->testbook) (3.8.0)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.8.2->jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (1.15.0)
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/merlin_models/models, configfile: pyproject.toml
plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 460 items / 1 error / 2 skipped
==================================== ERRORS ====================================

___________ ERROR collecting tests/unit/tf/models/test_retrieval.py ____________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_models/models/tests/unit/tf/models/test_retrieval.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/tf/models/test_retrieval.py:17: in 

from tests.common.tf.retrieval import retrieval_tests_common

tests/common/tf/retrieval/retrieval_tests_common.py:22: in 

from tests.common.tf.retrieval.retrieval_config import config_retrieval_train_eval_runner

tests/common/tf/retrieval/retrieval_config.py:19: in 

from tests.common.tf.retrieval.retrieval_utils import (

tests/common/tf/retrieval/retrieval_utils.py:27: in 

from wandb import wdb

E   ImportError: cannot import name 'wdb' from 'wandb' (/var/jenkins_home/.local/lib/python3.8/site-packages/wandb/init.py)

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.

'nearest': pil_image.NEAREST,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.

'bilinear': pil_image.BILINEAR,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.

'bicubic': pil_image.BICUBIC,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.

'hamming': pil_image.HAMMING,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.

'box': pil_image.BOX,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.

'lanczos': pil_image.LANCZOS,
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

=========================== short test summary info ============================

SKIPPED [1] tests/unit/implicit/init.py:18: could not import 'implicit': No module named 'implicit'

SKIPPED [1] tests/unit/lightfm/init.py:18: could not import 'lightfm': No module named 'lightfm'

!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!

=================== 2 skipped, 6 warnings, 1 error in 3.92s ====================

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/models/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_models] $ /bin/bash /tmp/jenkins530544502548032698.sh

nvidia-merlin-bot · 2022-07-11T22:28:17Z

Click to view CI Results

GitHub pull request #537 of commit c54d4177f00a7f605b77e583c9e8459fa1753423, has merge conflicts. Running as SYSTEM Setting status of c54d4177f00a7f605b77e583c9e8459fa1753423 to PENDING with url https://10.20.13.93:8080/job/merlin_models/643/console and message: 'Pending' Using context: Jenkins Building on master in workspace /var/jenkins_home/workspace/merlin_models using credential nvidia-merlin-bot > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/NVIDIA-Merlin/models/ # timeout=10 Fetching upstream changes from https://github.com/NVIDIA-Merlin/models/ > git --version # timeout=10 using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/models/ +refs/pull/537/*:refs/remotes/origin/pr/537/* # timeout=10 > git rev-parse c54d4177f00a7f605b77e583c9e8459fa1753423^{commit} # timeout=10 Checking out Revision c54d4177f00a7f605b77e583c9e8459fa1753423 (detached) > git config core.sparsecheckout # timeout=10 > git checkout -f c54d4177f00a7f605b77e583c9e8459fa1753423 # timeout=10 Commit message: "Changing memory management of unit tests to test retrieval on P100" > git rev-list --no-walk 731988e1ca64fd327479c3523c884c107e7fc5f0 # timeout=10 [merlin_models] $ /bin/bash /tmp/jenkins13416869784070619666.sh Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com Requirement already satisfied: testbook in /usr/local/lib/python3.8/dist-packages (0.4.2) Requirement already satisfied: nbformat>=5.0.4 in /usr/local/lib/python3.8/dist-packages (from testbook) (5.4.0) Requirement already satisfied: nbclient>=0.4.0 in /usr/local/lib/python3.8/dist-packages (from testbook) (0.6.5) Requirement already satisfied: traitlets>=5.1 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (5.3.0) Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.6.1) Requirement already satisfied: jupyter-core in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.10.0) Requirement already satisfied: fastjsonschema in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (2.15.3) Requirement already satisfied: jupyter-client>=6.1.5 in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (7.3.4) Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (1.5.5) Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (21.4.0) Requirement already satisfied: importlib-resources>=1.4.0; python_version < "3.9" in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (5.8.0) Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (0.18.1) Requirement already satisfied: entrypoints in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (0.4) Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (2.8.2) Requirement already satisfied: pyzmq>=23.0 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (23.2.0) Requirement already satisfied: tornado>=6.0 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (6.1) Requirement already satisfied: zipp>=3.1.0; python_version < "3.10" in /usr/local/lib/python3.8/dist-packages (from importlib-resources>=1.4.0; python_version < "3.9"->jsonschema>=2.6->nbformat>=5.0.4->testbook) (3.8.0) Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.8.2->jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (1.15.0) ============================= test session starts ============================== platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0 rootdir: /var/jenkins_home/workspace/merlin_models/models, configfile: pyproject.toml plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, cov-3.0.0 collected 478 items / 2 skipped

tests/unit/config/test_schema.py .... [ 0%]
tests/unit/datasets/test_advertising.py .s [ 1%]
tests/unit/datasets/test_ecommerce.py ..sss [ 2%]
tests/unit/datasets/test_entertainment.py ....sss. [ 3%]
tests/unit/datasets/test_social.py . [ 4%]
tests/unit/datasets/test_synthetic.py ..... [ 5%]
tests/unit/tf/test_core.py ...... [ 6%]
tests/unit/tf/test_dataset.py ............... [ 9%]
tests/unit/tf/test_public_api.py . [ 9%]
tests/unit/tf/blocks/test_cross.py ........... [ 12%]
tests/unit/tf/blocks/test_dlrm.py ........ [ 13%]
tests/unit/tf/blocks/test_interactions.py . [ 14%]
tests/unit/tf/blocks/test_mlp.py ............................. [ 20%]
tests/unit/tf/blocks/core/test_aggregation.py ......... [ 21%]
tests/unit/tf/blocks/core/test_base.py .. [ 22%]
tests/unit/tf/blocks/core/test_combinators.py ............... [ 25%]
tests/unit/tf/blocks/core/test_index.py ... [ 26%]
tests/unit/tf/blocks/core/test_masking.py ....... [ 27%]
tests/unit/tf/blocks/core/test_tabular.py .... [ 28%]
tests/unit/tf/blocks/core/test_transformations.py ........... [ 30%]
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py .. [ 31%]
tests/unit/tf/blocks/retrieval/test_two_tower.py ........... [ 33%]
tests/unit/tf/examples/test_01_getting_started.py . [ 33%]
tests/unit/tf/examples/test_02_dataschema.py . [ 33%]
tests/unit/tf/examples/test_03_exploring_different_models.py . [ 34%]
tests/unit/tf/examples/test_04_export_ranking_models.py . [ 34%]
tests/unit/tf/examples/test_05_export_retrieval_model.py . [ 34%]
tests/unit/tf/examples/test_06_advanced_own_architecture.py . [ 34%]
tests/unit/tf/examples/test_usecase_pretrained_embeddings.py . [ 34%]
tests/unit/tf/inputs/test_continuous.py ..... [ 35%]
tests/unit/tf/inputs/test_embedding.py .......................... [ 41%]
tests/unit/tf/inputs/test_tabular.py ....... [ 42%]
tests/unit/tf/layers/test_queue.py .............. [ 45%]
tests/unit/tf/losses/test_losses.py ....................... [ 50%]
tests/unit/tf/metrics/test_metrics_popularity.py ..... [ 51%]
tests/unit/tf/metrics/test_metrics_topk.py .................. [ 55%]
tests/unit/tf/models/test_base.py ............ [ 57%]
tests/unit/tf/models/test_benchmark.py .. [ 58%]
tests/unit/tf/models/test_ranking.py ................ [ 61%]
tests/unit/tf/models/test_retrieval.py ..............................F [ 68%]
tests/unit/tf/prediction_tasks/test_classification.py .. [ 68%]
tests/unit/tf/prediction_tasks/test_multi_task.py ....... [ 70%]
tests/unit/tf/prediction_tasks/test_next_item.py ............. [ 72%]
tests/unit/tf/prediction_tasks/test_regression.py .. [ 73%]
tests/unit/tf/prediction_tasks/test_sampling.py ...... [ 74%]
tests/unit/tf/utils/test_batch.py .... [ 75%]
tests/unit/tf/utils/test_tf_utils.py .. [ 75%]
tests/unit/torch/test_dataset.py ......... [ 77%]
tests/unit/torch/test_public_api.py . [ 77%]
tests/unit/torch/block/test_base.py .... [ 78%]
tests/unit/torch/block/test_mlp.py . [ 78%]
tests/unit/torch/features/test_continuous.py .. [ 79%]
tests/unit/torch/features/test_embedding.py .............. [ 82%]
tests/unit/torch/features/test_tabular.py .... [ 83%]
tests/unit/torch/model/test_head.py ............ [ 85%]
tests/unit/torch/model/test_model.py .. [ 85%]
tests/unit/torch/tabular/test_aggregation.py ........ [ 87%]
tests/unit/torch/tabular/test_tabular.py ... [ 88%]
tests/unit/torch/tabular/test_transformations.py ....... [ 89%]
tests/unit/utils/test_schema_utils.py ................................ [ 96%]
tests/unit/xgb/test_xgboost.py ................. [100%]

=================================== FAILURES ===================================
___________________________ test_mf_advanced_options ___________________________

train_eval_datasets_last_fm = (<merlin.io.dataset.Dataset object at 0x7f6f272bc1f0>, <merlin.io.dataset.Dataset object at 0x7f6c83ca4ca0>)

def test_mf_advanced_options(train_eval_datasets_last_fm):
    train_ds, eval_ds = train_eval_datasets_last_fm

  metrics = retrieval_tests_common.train_eval_two_tower_for_lastfm(

        train_ds,
        eval_ds,
        # train_epochs=1,
        # train_steps_per_epoch=500,
        # eval_steps=100,
        train_epochs=1,
        train_steps_per_epoch=500,
        eval_steps=5000,
        log_to_wandb=True,
        wandb_project="merlin-ci",
    )

tests/unit/tf/models/test_retrieval.py:325:

tests/common/tf/retrieval/retrieval_tests_common.py:161: in train_eval_two_tower_for_lastfm
return train_eval_two_tower(
tests/common/tf/retrieval/retrieval_tests_common.py:114: in train_eval_two_tower
metrics = runner.run(hparams)
tests/common/tf/retrieval/retrieval_utils.py:531: in run
train_metrics = self.model.evaluate(
merlin/models/tf/models/base.py:861: in evaluate
return super().evaluate(
merlin/models/tf/models/base.py:567: in evaluate
return super().evaluate(
/usr/local/lib/python3.8/dist-packages/keras/utils/traceback_utils.py:67: in error_handler
raise e.with_traceback(filtered_tb) from None

op_name = '__inference_test_function_2432696', num_outputs = 11
inputs = [<tf.Tensor: shape=(), dtype=resource, value=<ResourceHandle(name="Resource-15787-at-0x8121f3c0", device="/job:localho...GPU:0", container="Anonymous", type="tensorflow::Var", dtype and shapes : "[ DType enum: 1, Shape: [148,64] ]")>>, ...]
attrs = ('executor_type', '', 'config_proto', b'\n\x07\n\x03CPU\x10\x01\n\x07\n\x03GPU\x10\x022\x11*\x030,1J\n\n\x06\n\x04\x00`\xcbD\n\x008\x01\x82\x01\x00')
ctx = <tensorflow.python.eager.context.Context object at 0x7f6ddd58a070>
name = None

def quick_execute(op_name, num_outputs, inputs, attrs, ctx, name=None):
  """Execute a TensorFlow operation.

  Args:
    op_name: Name of the TensorFlow operation (see REGISTER_OP in C++ code) to
      execute.
    num_outputs: The number of outputs of the operation to fetch. (Explicitly
      provided instead of being inferred for performance reasons).
    inputs: A list of inputs to the operation. Each entry should be a Tensor, or
      a value which can be passed to the Tensor constructor to create one.
    attrs: A tuple with alternating string attr names and attr values for this
      operation.
    ctx: The value of context.context().
    name: Customized name for the operation.

  Returns:
    List of output Tensor objects. The list is empty if there are no outputs

  Raises:
    An exception on error.
  """
  device_name = ctx.device_name
  # pylint: disable=protected-access
  try:
    ctx.ensure_initialized()

  tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,

                                        inputs, attrs, num_outputs)

E tensorflow.python.framework.errors_impl.ResourceExhaustedError: Graph execution error:
E
E Detected at node 'top_k_index_block/TopKV2' defined at (most recent call last):
E File "/usr/local/bin/pytest", line 8, in
E sys.exit(console_main())
E File "/usr/local/lib/python3.8/dist-packages/_pytest/config/init.py", line 187, in console_main
E code = main()
E File "/usr/local/lib/python3.8/dist-packages/_pytest/config/init.py", line 164, in main
E ret: Union[ExitCode, int] = config.hook.pytest_cmdline_main(
E File "/usr/local/lib/python3.8/dist-packages/pluggy/_hooks.py", line 265, in call
E return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
E File "/usr/local/lib/python3.8/dist-packages/pluggy/_manager.py", line 80, in _hookexec
E return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
E File "/usr/local/lib/python3.8/dist-packages/pluggy/_callers.py", line 39, in _multicall
E res = hook_impl.function(*args)
E File "/usr/local/lib/python3.8/dist-packages/_pytest/main.py", line 315, in pytest_cmdline_main
E return wrap_session(config, _main)
E File "/usr/local/lib/python3.8/dist-packages/_pytest/main.py", line 268, in wrap_session
E session.exitstatus = doit(config, session) or 0
E File "/usr/local/lib/python3.8/dist-packages/_pytest/main.py", line 322, in _main
E config.hook.pytest_runtestloop(session=session)
E File "/usr/local/lib/python3.8/dist-packages/pluggy/_hooks.py", line 265, in call
E return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
E File "/usr/local/lib/python3.8/dist-packages/pluggy/_manager.py", line 80, in _hookexec
E return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
E File "/usr/local/lib/python3.8/dist-packages/pluggy/_callers.py", line 39, in _multicall
E res = hook_impl.function(*args)
E File "/usr/local/lib/python3.8/dist-packages/_pytest/main.py", line 347, in pytest_runtestloop
E item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
E File "/usr/local/lib/python3.8/dist-packages/pluggy/_hooks.py", line 265, in call
E return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
E File "/usr/local/lib/python3.8/dist-packages/pluggy/_manager.py", line 80, in _hookexec
E return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
E File "/usr/local/lib/python3.8/dist-packages/pluggy/_callers.py", line 39, in _multicall
E res = hook_impl.function(*args)
E File "/usr/local/lib/python3.8/dist-packages/_pytest/runner.py", line 111, in pytest_runtest_protocol
E runtestprotocol(item, nextitem=nextitem)
E File "/usr/local/lib/python3.8/dist-packages/_pytest/runner.py", line 130, in runtestprotocol
E reports.append(call_and_report(item, "call", log))
E File "/usr/local/lib/python3.8/dist-packages/_pytest/runner.py", line 219, in call_and_report
E call = call_runtest_hook(item, when, **kwds)
E File "/usr/local/lib/python3.8/dist-packages/_pytest/runner.py", line 258, in call_runtest_hook
E return CallInfo.from_call(
E File "/usr/local/lib/python3.8/dist-packages/_pytest/runner.py", line 338, in from_call
E result: Optional[TResult] = func()
E File "/usr/local/lib/python3.8/dist-packages/_pytest/runner.py", line 259, in
E lambda: ihook(item=item, **kwds), when=when, reraise=reraise
E File "/usr/local/lib/python3.8/dist-packages/pluggy/_hooks.py", line 265, in call
E return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
E File "/usr/local/lib/python3.8/dist-packages/pluggy/_manager.py", line 80, in _hookexec
E return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
E File "/usr/local/lib/python3.8/dist-packages/pluggy/_callers.py", line 39, in _multicall
E res = hook_impl.function(*args)
E File "/usr/local/lib/python3.8/dist-packages/_pytest/runner.py", line 166, in pytest_runtest_call
E item.runtest()
E File "/usr/local/lib/python3.8/dist-packages/_pytest/python.py", line 1761, in runtest
E self.ihook.pytest_pyfunc_call(pyfuncitem=self)
E File "/usr/local/lib/python3.8/dist-packages/pluggy/_hooks.py", line 265, in call
E return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
E File "/usr/local/lib/python3.8/dist-packages/pluggy/_manager.py", line 80, in _hookexec
E return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
E File "/usr/local/lib/python3.8/dist-packages/pluggy/_callers.py", line 39, in _multicall
E res = hook_impl.function(*args)
E File "/usr/local/lib/python3.8/dist-packages/_pytest/python.py", line 192, in pytest_pyfunc_call
E result = testfunction(**testargs)
E File "/var/jenkins_home/workspace/merlin_models/models/tests/unit/tf/models/test_retrieval.py", line 325, in test_mf_advanced_options
E metrics = retrieval_tests_common.train_eval_two_tower_for_lastfm(
E File "/var/jenkins_home/workspace/merlin_models/models/tests/common/tf/retrieval/retrieval_tests_common.py", line 161, in train_eval_two_tower_for_lastfm
E return train_eval_two_tower(
E File "/var/jenkins_home/workspace/merlin_models/models/tests/common/tf/retrieval/retrieval_tests_common.py", line 114, in train_eval_two_tower
E metrics = runner.run(hparams)
E File "/var/jenkins_home/workspace/merlin_models/models/tests/common/tf/retrieval/retrieval_utils.py", line 531, in run
E train_metrics = self.model.evaluate(
E File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/models/base.py", line 861, in evaluate
E return super().evaluate(
E File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/models/base.py", line 567, in evaluate
E return super().evaluate(
E File "/usr/local/lib/python3.8/dist-packages/keras/utils/traceback_utils.py", line 64, in error_handler
E return fn(*args, **kwargs)
E File "/usr/local/lib/python3.8/dist-packages/keras/engine/training.py", line 1756, in evaluate
E tmp_logs = self.test_function(iterator)
E File "/usr/local/lib/python3.8/dist-packages/keras/engine/training.py", line 1557, in test_function
E return step_function(self, iterator)
E File "/usr/local/lib/python3.8/dist-packages/keras/engine/training.py", line 1546, in step_function
E outputs = model.distribute_strategy.run(run_step, args=(data,))
E File "/usr/local/lib/python3.8/dist-packages/keras/engine/training.py", line 1535, in run_step
E outputs = model.test_step(data)
E File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/models/base.py", line 407, in test_step
E outputs = self.pre_eval_topk.call_outputs(outputs)
E File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/blocks/core/index.py", line 253, in call_outputs
E pred_top_scores, top_ids = self(queries, k=self._k)
E File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/config/schema.py", line 58, in call
E return super().call(*args, **kwargs)
E File "/usr/local/lib/python3.8/dist-packages/keras/utils/traceback_utils.py", line 64, in error_handler
E return fn(*args, **kwargs)
E File "/usr/local/lib/python3.8/dist-packages/keras/engine/base_layer.py", line 1014, in call
E outputs = call_fn(inputs, *args, **kwargs)
E File "/usr/local/lib/python3.8/dist-packages/keras/utils/traceback_utils.py", line 92, in error_handler
E return fn(*args, **kwargs)
E File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/blocks/core/index.py", line 228, in call
E top_scores, top_indices = tf.math.top_k(scores, k=k)
E Node: 'top_k_index_block/TopKV2'
E Detected at node 'top_k_index_block/TopKV2' defined at (most recent call last):
E File "/usr/local/bin/pytest", line 8, in
E sys.exit(console_main())
E File "/usr/local/lib/python3.8/dist-packages/_pytest/config/init.py", line 187, in console_main
E code = main()
E File "/usr/local/lib/python3.8/dist-packages/_pytest/config/init.py", line 164, in main
E ret: Union[ExitCode, int] = config.hook.pytest_cmdline_main(
E File "/usr/local/lib/python3.8/dist-packages/pluggy/_hooks.py", line 265, in call
E return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
E File "/usr/local/lib/python3.8/dist-packages/pluggy/_manager.py", line 80, in _hookexec
E return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
E File "/usr/local/lib/python3.8/dist-packages/pluggy/_callers.py", line 39, in _multicall
E res = hook_impl.function(*args)
E File "/usr/local/lib/python3.8/dist-packages/_pytest/main.py", line 315, in pytest_cmdline_main
E return wrap_session(config, _main)
E File "/usr/local/lib/python3.8/dist-packages/_pytest/main.py", line 268, in wrap_session
E session.exitstatus = doit(config, session) or 0
E File "/usr/local/lib/python3.8/dist-packages/_pytest/main.py", line 322, in _main
E config.hook.pytest_runtestloop(session=session)
E File "/usr/local/lib/python3.8/dist-packages/pluggy/_hooks.py", line 265, in call
E return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
E File "/usr/local/lib/python3.8/dist-packages/pluggy/_manager.py", line 80, in _hookexec
E return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
E File "/usr/local/lib/python3.8/dist-packages/pluggy/_callers.py", line 39, in _multicall
E res = hook_impl.function(*args)
E File "/usr/local/lib/python3.8/dist-packages/_pytest/main.py", line 347, in pytest_runtestloop
E item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
E File "/usr/local/lib/python3.8/dist-packages/pluggy/_hooks.py", line 265, in call
E return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
E File "/usr/local/lib/python3.8/dist-packages/pluggy/_manager.py", line 80, in _hookexec
E return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
E File "/usr/local/lib/python3.8/dist-packages/pluggy/_callers.py", line 39, in _multicall
E res = hook_impl.function(*args)
E File "/usr/local/lib/python3.8/dist-packages/_pytest/runner.py", line 111, in pytest_runtest_protocol
E runtestprotocol(item, nextitem=nextitem)
E File "/usr/local/lib/python3.8/dist-packages/_pytest/runner.py", line 130, in runtestprotocol
E reports.append(call_and_report(item, "call", log))
E File "/usr/local/lib/python3.8/dist-packages/_pytest/runner.py", line 219, in call_and_report
E call = call_runtest_hook(item, when, **kwds)
E File "/usr/local/lib/python3.8/dist-packages/_pytest/runner.py", line 258, in call_runtest_hook
E return CallInfo.from_call(
E File "/usr/local/lib/python3.8/dist-packages/_pytest/runner.py", line 338, in from_call
E result: Optional[TResult] = func()
E File "/usr/local/lib/python3.8/dist-packages/_pytest/runner.py", line 259, in
E lambda: ihook(item=item, **kwds), when=when, reraise=reraise
E File "/usr/local/lib/python3.8/dist-packages/pluggy/_hooks.py", line 265, in call
E return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
E File "/usr/local/lib/python3.8/dist-packages/pluggy/_manager.py", line 80, in _hookexec
E return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
E File "/usr/local/lib/python3.8/dist-packages/pluggy/_callers.py", line 39, in _multicall
E res = hook_impl.function(*args)
E File "/usr/local/lib/python3.8/dist-packages/_pytest/runner.py", line 166, in pytest_runtest_call
E item.runtest()
E File "/usr/local/lib/python3.8/dist-packages/_pytest/python.py", line 1761, in runtest
E self.ihook.pytest_pyfunc_call(pyfuncitem=self)
E File "/usr/local/lib/python3.8/dist-packages/pluggy/_hooks.py", line 265, in call
E return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
E File "/usr/local/lib/python3.8/dist-packages/pluggy/_manager.py", line 80, in _hookexec
E return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
E File "/usr/local/lib/python3.8/dist-packages/pluggy/_callers.py", line 39, in _multicall
E res = hook_impl.function(*args)
E File "/usr/local/lib/python3.8/dist-packages/_pytest/python.py", line 192, in pytest_pyfunc_call
E result = testfunction(**testargs)
E File "/var/jenkins_home/workspace/merlin_models/models/tests/unit/tf/models/test_retrieval.py", line 325, in test_mf_advanced_options
E metrics = retrieval_tests_common.train_eval_two_tower_for_lastfm(
E File "/var/jenkins_home/workspace/merlin_models/models/tests/common/tf/retrieval/retrieval_tests_common.py", line 161, in train_eval_two_tower_for_lastfm
E return train_eval_two_tower(
E File "/var/jenkins_home/workspace/merlin_models/models/tests/common/tf/retrieval/retrieval_tests_common.py", line 114, in train_eval_two_tower
E metrics = runner.run(hparams)
E File "/var/jenkins_home/workspace/merlin_models/models/tests/common/tf/retrieval/retrieval_utils.py", line 531, in run
E train_metrics = self.model.evaluate(
E File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/models/base.py", line 861, in evaluate
E return super().evaluate(
E File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/models/base.py", line 567, in evaluate
E return super().evaluate(
E File "/usr/local/lib/python3.8/dist-packages/keras/utils/traceback_utils.py", line 64, in error_handler
E return fn(*args, **kwargs)
E File "/usr/local/lib/python3.8/dist-packages/keras/engine/training.py", line 1756, in evaluate
E tmp_logs = self.test_function(iterator)
E File "/usr/local/lib/python3.8/dist-packages/keras/engine/training.py", line 1557, in test_function
E return step_function(self, iterator)
E File "/usr/local/lib/python3.8/dist-packages/keras/engine/training.py", line 1546, in step_function
E outputs = model.distribute_strategy.run(run_step, args=(data,))
E File "/usr/local/lib/python3.8/dist-packages/keras/engine/training.py", line 1535, in run_step
E outputs = model.test_step(data)
E File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/models/base.py", line 407, in test_step
E outputs = self.pre_eval_topk.call_outputs(outputs)
E File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/blocks/core/index.py", line 253, in call_outputs
E pred_top_scores, top_ids = self(queries, k=self._k)
E File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/config/schema.py", line 58, in call
E return super().call(*args, **kwargs)
E File "/usr/local/lib/python3.8/dist-packages/keras/utils/traceback_utils.py", line 64, in error_handler
E return fn(*args, **kwargs)
E File "/usr/local/lib/python3.8/dist-packages/keras/engine/base_layer.py", line 1014, in call
E outputs = call_fn(inputs, *args, **kwargs)
E File "/usr/local/lib/python3.8/dist-packages/keras/utils/traceback_utils.py", line 92, in error_handler
E return fn(*args, **kwargs)
E File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/blocks/core/index.py", line 228, in call
E top_scores, top_indices = tf.math.top_k(scores, k=k)
E Node: 'top_k_index_block/TopKV2'
E 2 root error(s) found.
E (0) RESOURCE_EXHAUSTED: OOM when allocating tensor with shape[2678366463] and type int8 on /job:localhost/replica:0/task:0/device:GPU:0 by allocator gpu_async_0
E [[{{node top_k_index_block/TopKV2}}]]
E Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.
E
E [[StatefulPartitionedCall/cond_1/pivot_t/_67/_153]]
E Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.
E
E (1) RESOURCE_EXHAUSTED: OOM when allocating tensor with shape[2678366463] and type int8 on /job:localhost/replica:0/task:0/device:GPU:0 by allocator gpu_async_0
E [[{{node top_k_index_block/TopKV2}}]]
E Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.
E
E 0 successful operations.
E 0 derived errors ignored. [Op:__inference_test_function_2432696]

/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py:54: ResourceExhaustedError
----------------------------- Captured stdout call -----------------------------

1/500 [..............................] - ETA: 54:44 - loss: 9.5823 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.0232 - _timestamp: 1657578244.0000 - _runtime: 10.0000��
2/500 [..............................] - ETA: 27s - loss: 9.4951 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.0233 - _timestamp: 1657578244.0000 - _runtime: 10.0000 ��
3/500 [..............................] - ETA: 28s - loss: 9.4324 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.0237 - _timestamp: 1657578244.0000 - _runtime: 10.0000��
4/500 [..............................] - ETA: 34s - loss: 9.3666 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.0246 - _timestamp: 1657578244.0000 - _runtime: 10.0000��
5/500 [..............................] - ETA: 32s - loss: 9.3095 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.0258 - _timestamp: 1657578244.0000 - _runtime: 10.0000��
6/500 [..............................] - ETA: 31s - loss: 9.2664 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.0270 - _timestamp: 1657578244.0000 - _runtime: 10.0000��
7/500 [..............................] - ETA: 30s - loss: 9.2250 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.0284 - _timestamp: 1657578244.0000 - _runtime: 10.0000��
9/500 [..............................] - ETA: 28s - loss: 9.1655 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.0313 - _timestamp: 1657578244.0000 - _runtime: 10.0000��
10/500 [..............................] - ETA: 31s - loss: 9.1367 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.0328 - _timestamp: 1657578244.0000 - _runtime: 10.0000��
11/500 [..............................] - ETA: 31s - loss: 9.1141 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.0344 - _timestamp: 1657578244.5000 - _runtime: 10.5000��
12/500 [..............................] - ETA: 31s - loss: 9.0906 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.0360 - _timestamp: 1657578244.5000 - _runtime: 10.5000��
13/500 [..............................] - ETA: 32s - loss: 9.0675 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.0376 - _timestamp: 1657578244.5000 - _runtime: 10.5000��
14/500 [..............................] - ETA: 31s - loss: 9.0465 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.0393 - _timestamp: 1657578244.5000 - _runtime: 10.5000��
16/500 [..............................] - ETA: 30s - loss: 9.0066 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.0426 - _timestamp: 1657578244.5000 - _runtime: 10.5000��
18/500 [>.............................] - ETA: 29s - loss: 8.9738 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.0460 - _timestamp: 1657578244.5000 - _runtime: 10.5000��
20/500 [>.............................] - ETA: 28s - loss: 8.9431 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.0494 - _timestamp: 1657578244.5000 - _runtime: 10.5000��
22/500 [>.............................] - ETA: 28s - loss: 8.9139 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.0529 - _timestamp: 1657578244.6667 - _runtime: 10.6667��
24/500 [>.............................] - ETA: 27s - loss: 8.8867 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.0563 - _timestamp: 1657578244.6667 - _runtime: 10.6667��
26/500 [>.............................] - ETA: 26s - loss: 8.8610 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.0596 - _timestamp: 1657578244.6667 - _runtime: 10.6667��
28/500 [>.............................] - ETA: 26s - loss: 8.8372 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.0630 - _timestamp: 1657578244.6667 - _runtime: 10.6667��
30/500 [>.............................] - ETA: 25s - loss: 8.8142 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.0663 - _timestamp: 1657578244.6667 - _runtime: 10.6667��
32/500 [>.............................] - ETA: 25s - loss: 8.7917 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.0696 - _timestamp: 1657578245.0000 - _runtime: 11.0000��
34/500 [=>............................] - ETA: 24s - loss: 8.7695 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.0728 - _timestamp: 1657578245.0000 - _runtime: 11.0000��
36/500 [=>............................] - ETA: 24s - loss: 8.7493 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.0761 - _timestamp: 1657578245.0000 - _runtime: 11.0000��
38/500 [=>............................] - ETA: 24s - loss: 8.7293 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.0793 - _timestamp: 1657578245.0000 - _runtime: 11.0000��
39/500 [=>............................] - ETA: 24s - loss: 8.7193 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.0808 - _timestamp: 1657578245.0000 - _runtime: 11.0000��
40/500 [=>............................] - ETA: 24s - loss: 8.7092 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.0824 - _timestamp: 1657578245.0000 - _runtime: 11.0000��
42/500 [=>............................] - ETA: 24s - loss: 8.6890 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.0856 - _timestamp: 1657578245.2000 - _runtime: 11.2000��
44/500 [=>............................] - ETA: 23s - loss: 8.6717 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.0887 - _timestamp: 1657578245.2000 - _runtime: 11.2000��
45/500 [=>............................] - ETA: 23s - loss: 8.6629 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.0903 - _timestamp: 1657578245.2000 - _runtime: 11.2000��
47/500 [=>............................] - ETA: 23s - loss: 8.6455 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.0933 - _timestamp: 1657578245.2000 - _runtime: 11.2000��
49/500 [=>............................] - ETA: 23s - loss: 8.6286 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.0964 - _timestamp: 1657578245.2000 - _runtime: 11.2000��
51/500 [==>...........................] - ETA: 23s - loss: 8.6123 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.0994 - _timestamp: 1657578245.5000 - _runtime: 11.5000��
53/500 [==>...........................] - ETA: 22s - loss: 8.5957 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.1024 - _timestamp: 1657578245.5000 - _runtime: 11.5000��
55/500 [==>...........................] - ETA: 22s - loss: 8.5799 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.1053 - _timestamp: 1657578245.5000 - _runtime: 11.5000��
57/500 [==>...........................] - ETA: 22s - loss: 8.5645 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.1082 - _timestamp: 1657578245.5000 - _runtime: 11.5000��
59/500 [==>...........................] - ETA: 22s - loss: 8.5499 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.1111 - _timestamp: 1657578245.5000 - _runtime: 11.5000��
61/500 [==>...........................] - ETA: 21s - loss: 8.5351 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.1139 - _timestamp: 1657578245.7143 - _runtime: 11.7143��
63/500 [==>...........................] - ETA: 21s - loss: 8.5207 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.1167 - _timestamp: 1657578245.7143 - _runtime: 11.7143��
65/500 [==>...........................] - ETA: 21s - loss: 8.5069 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.1195 - _timestamp: 1657578245.7143 - _runtime: 11.7143��
67/500 [===>..........................] - ETA: 21s - loss: 8.4933 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.1222 - _timestamp: 1657578245.7143 - _runtime: 11.7143��
69/500 [===>..........................] - ETA: 21s - loss: 8.4798 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.1249 - _timestamp: 1657578245.7143 - _runtime: 11.7143��
71/500 [===>..........................] - ETA: 21s - loss: 8.4669 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.1276 - _timestamp: 1657578246.0000 - _runtime: 12.0000��
73/500 [===>..........................] - ETA: 20s - loss: 8.4546 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.1302 - _timestamp: 1657578246.0000 - _runtime: 12.0000��
75/500 [===>..........................] - ETA: 20s - loss: 8.4425 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.1328 - _timestamp: 1657578246.0000 - _runtime: 12.0000��
77/500 [===>..........................] - ETA: 20s - loss: 8.4311 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.1354 - _timestamp: 1657578246.0000 - _runtime: 12.0000��
79/500 [===>..........................] - ETA: 20s - loss: 8.4196 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.1379 - _timestamp: 1657578246.0000 - _runtime: 12.0000��
81/500 [===>..........................] - ETA: 20s - loss: 8.4081 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.1404 - _timestamp: 1657578246.2222 - _runtime: 12.2222��
82/500 [===>..........................] - ETA: 20s - loss: 8.4025 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.1416 - _timestamp: 1657578246.2222 - _runtime: 12.2222��
84/500 [====>.........................] - ETA: 20s - loss: 8.3920 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.1441 - _timestamp: 1657578246.2222 - _runtime: 12.2222��
86/500 [====>.........................] - ETA: 20s - loss: 8.3815 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.1465 - _timestamp: 1657578246.2222 - _runtime: 12.2222��
88/500 [====>.........................] - ETA: 20s - loss: 8.3706 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.1489 - _timestamp: 1657578246.2222 - _runtime: 12.2222��
89/500 [====>.........................] - ETA: 19s - loss: 8.3652 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.1500 - _timestamp: 1657578246.2222 - _runtime: 12.2222��
91/500 [====>.........................] - ETA: 19s - loss: 8.3556 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.1524 - _timestamp: 1657578246.4000 - _runtime: 12.4000��
93/500 [====>.........................] - ETA: 19s - loss: 8.3461 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.1547 - _timestamp: 1657578246.4000 - _runtime: 12.4000��
95/500 [====>.........................] - ETA: 19s - loss: 8.3371 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.1570 - _timestamp: 1657578246.4000 - _runtime: 12.4000��
97/500 [====>.........................] - ETA: 19s - loss: 8.3276 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.1592 - _timestamp: 1657578246.4000 - _runtime: 12.4000��
99/500 [====>.........................] - ETA: 19s - loss: 8.3186 - recall_at_10: 0.0029 - mrr_at_10: 4.0370e-04 - ndcg_at_10: 9.5864e-04 - recall_at_50: 0.0125 - mrr_at_50: 8.1821e-04 - ndcg_at_50: 0.0030 - recall_at_100: 0.0249 - mrr_at_100: 9.9422e-04 - ndcg_at_100: 0.0050 - regularization_loss: 0.1614 - _timestamp: 1657578246.4000 - _runtime: 12.4000��
101/500 [=====>........................] - ETA: 19s - loss: 8.3098 - recall_at_10: 0.0040 - mrr_at_10: 0.0010 - ndcg_at_10: 0.0017 - recall_at_50: 0.0141 - mrr_at_50: 0.0015 - ndcg_at_50: 0.0038 - recall_at_100: 0.0268 - mrr_at_100: 0.0016 - ndcg_at_100: 0.0059 - regularization_loss: 0.1636 - _timestamp: 1657578246.6364 - _runtime: 12.6364 ��
103/500 [=====>........................] - ETA: 19s - loss: 8.3012 - recall_at_10: 0.0060 - mrr_at_10: 0.0022 - ndcg_at_10: 0.0030 - recall_at_50: 0.0173 - mrr_at_50: 0.0027 - ndcg_at_50: 0.0055 - recall_at_100: 0.0306 - mrr_at_100: 0.0029 - ndcg_at_100: 0.0076 - regularization_loss: 0.1658 - _timestamp: 1657578246.6364 - _runtime: 12.6364��
105/500 [=====>........................] - ETA: 18s - loss: 8.2929 - recall_at_10: 0.0079 - mrr_at_10: 0.0033 - ndcg_at_10: 0.0044 - recall_at_50: 0.0204 - mrr_at_50: 0.0039 - ndcg_at_50: 0.0071 - recall_at_100: 0.0343 - mrr_at_100: 0.0041 - ndcg_at_100: 0.0093 - regularization_loss: 0.1680 - _timestamp: 1657578246.6364 - _runtime: 12.6364��
107/500 [=====>........................] - ETA: 18s - loss: 8.2845 - recall_at_10: 0.0098 - mrr_at_10: 0.0044 - ndcg_at_10: 0.0057 - recall_at_50: 0.0234 - mrr_at_50: 0.0050 - ndcg_at_50: 0.0086 - recall_at_100: 0.0378 - mrr_at_100: 0.0052 - ndcg_at_100: 0.0109 - regularization_loss: 0.1701 - _timestamp: 1657578246.6364 - _runtime: 12.6364��
109/500 [=====>........................] - ETA: 18s - loss: 8.2762 - recall_at_10: 0.0116 - mrr_at_10: 0.0055 - ndcg_at_10: 0.0069 - recall_at_50: 0.0262 - mrr_at_50: 0.0061 - ndcg_at_50: 0.0100 - recall_at_100: 0.0411 - mrr_at_100: 0.0063 - ndcg_at_100: 0.0125 - regularization_loss: 0.1722 - _timestamp: 1657578246.6364 - _runtime: 12.6364��
111/500 [=====>........................] - ETA: 18s - loss: 8.2683 - recall_at_10: 0.0133 - mrr_at_10: 0.0065 - ndcg_at_10: 0.0081 - recall_at_50: 0.0290 - mrr_at_50: 0.0072 - ndcg_at_50: 0.0115 - recall_at_100: 0.0444 - mrr_at_100: 0.0074 - ndcg_at_100: 0.0139 - regularization_loss: 0.1743 - _timestamp: 1657578246.8333 - _runtime: 12.8333��
113/500 [=====>........................] - ETA: 18s - loss: 8.2604 - recall_at_10: 0.0150 - mrr_at_10: 0.0075 - ndcg_at_10: 0.0092 - recall_at_50: 0.0316 - mrr_at_50: 0.0082 - ndcg_at_50: 0.0128 - recall_at_100: 0.0475 - mrr_at_100: 0.0084 - ndcg_at_100: 0.0154 - regularization_loss: 0.1763 - _timestamp: 1657578246.8333 - _runtime: 12.8333��
115/500 [=====>........................] - ETA: 18s - loss: 8.2525 - recall_at_10: 0.0166 - mrr_at_10: 0.0084 - ndcg_at_10: 0.0103 - recall_at_50: 0.0342 - mrr_at_50: 0.0092 - ndcg_at_50: 0.0141 - recall_at_100: 0.0505 - mrr_at_100: 0.0094 - ndcg_at_100: 0.0168 - regularization_loss: 0.1784 - _timestamp: 1657578246.8333 - _runtime: 12.8333��
117/500 [======>.......................] - ETA: 18s - loss: 8.2448 - recall_at_10: 0.0182 - mrr_at_10: 0.0093 - ndcg_at_10: 0.0114 - recall_at_50: 0.0367 - mrr_at_50: 0.0101 - ndcg_at_50: 0.0154 - recall_at_100: 0.0534 - mrr_at_100: 0.0104 - ndcg_at_100: 0.0181 - regularization_loss: 0.1804 - _timestamp: 1657578246.8333 - _runtime: 12.8333��
119/500 [======>.......................] - ETA: 18s - loss: 8.2376 - recall_at_10: 0.0197 - mrr_at_10: 0.0102 - ndcg_at_10: 0.0124 - recall_at_50: 0.0391 - mrr_at_50: 0.0111 - ndcg_at_50: 0.0166 - recall_at_100: 0.0563 - mrr_at_100: 0.0113 - ndcg_at_100: 0.0194 - regularization_loss: 0.1824 - _timestamp: 1657578246.8333 - _runtime: 12.8333��
121/500 [======>.......................] - ETA: 17s - loss: 8.2302 - recall_at_10: 0.0212 - mrr_at_10: 0.0111 - ndcg_at_10: 0.0134 - recall_at_50: 0.0414 - mrr_at_50: 0.0120 - ndcg_at_50: 0.0178 - recall_at_100: 0.0590 - mrr_at_100: 0.0122 - ndcg_at_100: 0.0207 - regularization_loss: 0.1843 - _timestamp: 1657578247.0769 - _runtime: 13.0769��
123/500 [======>.......................] - ETA: 17s - loss: 8.2232 - recall_at_10: 0.0226 - mrr_at_10: 0.0119 - ndcg_at_10: 0.0144 - recall_at_50: 0.0436 - mrr_at_50: 0.0128 - ndcg_at_50: 0.0190 - recall_at_100: 0.0616 - mrr_at_100: 0.0131 - ndcg_at_100: 0.0219 - regularization_loss: 0.1862 - _timestamp: 1657578247.0769 - _runtime: 13.0769��
125/500 [======>.......................] - ETA: 17s - loss: 8.2163 - recall_at_10: 0.0240 - mrr_at_10: 0.0127 - ndcg_at_10: 0.0153 - recall_at_50: 0.0458 - mrr_at_50: 0.0137 - ndcg_at_50: 0.0201 - recall_at_100: 0.0642 - mrr_at_100: 0.0139 - ndcg_at_100: 0.0230 - regularization_loss: 0.1882 - _timestamp: 1657578247.0769 - _runtime: 13.0769��
126/500 [======>.......................] - ETA: 17s - loss: 8.2131 - recall_at_10: 0.0246 - mrr_at_10: 0.0131 - ndcg_at_10: 0.0158 - recall_at_50: 0.0469 - mrr_at_50: 0.0141 - ndcg_at_50: 0.0206 - recall_at_100: 0.0654 - mrr_at_100: 0.0143 - ndcg_at_100: 0.0236 - regularization_loss: 0.1891 - _timestamp: 1657578247.0769 - _runtime: 13.0769��
128/500 [======>.......................] - ETA: 17s - loss: 8.2066 - recall_at_10: 0.0259 - mrr_at_10: 0.0138 - ndcg_at_10: 0.0167 - recall_at_50: 0.0489 - mrr_at_50: 0.0149 - ndcg_at_50: 0.0217 - recall_at_100: 0.0679 - mrr_at_100: 0.0151 - ndcg_at_100: 0.0247 - regularization_loss: 0.1910 - _timestamp: 1657578247.0769 - _runtime: 13.0769��
130/500 [======>.......................] - ETA: 17s - loss: 8.2001 - recall_at_10: 0.0272 - mrr_at_10: 0.0146 - ndcg_at_10: 0.0175 - recall_at_50: 0.0509 - mrr_at_50: 0.0156 - ndcg_at_50: 0.0227 - recall_at_100: 0.0702 - mrr_at_100: 0.0159 - ndcg_at_100: 0.0258 - regularization_loss: 0.1928 - _timestamp: 1657578247.0769 - _runtime: 13.0769��
132/500 [======>.......................] - ETA: 17s - loss: 8.1937 - recall_at_10: 0.0284 - mrr_at_10: 0.0153 - ndcg_at_10: 0.0184 - recall_at_50: 0.0529 - mrr_at_50: 0.0164 - ndcg_at_50: 0.0237 - recall_at_100: 0.0725 - mrr_at_100: 0.0167 - ndcg_at_100: 0.0269 - regularization_loss: 0.1947 - _timestamp: 1657578247.2857 - _runtime: 13.2857��
133/500 [======>.......................] - ETA: 17s - loss: 8.1902 - recall_at_10: 0.0290 - mrr_at_10: 0.0156 - ndcg_at_10: 0.0188 - recall_at_50: 0.0538 - mrr_at_50: 0.0167 - ndcg_at_50: 0.0242 - recall_at_100: 0.0736 - mrr_at_100: 0.0170 - ndcg_at_100: 0.0274 - regularization_loss: 0.1956 - _timestamp: 1657578247.2857 - _runtime: 13.2857��
135/500 [=======>......................] - ETA: 17s - loss: 8.1842 - recall_at_10: 0.0302 - mrr_at_10: 0.0163 - ndcg_at_10: 0.0196 - recall_at_50: 0.0557 - mrr_at_50: 0.0175 - ndcg_at_50: 0.0251 - recall_at_100: 0.0758 - mrr_at_100: 0.0177 - ndcg_at_100: 0.0284 - regularization_loss: 0.1974 - _timestamp: 1657578247.2857 - _runtime: 13.2857��
137/500 [=======>......................] - ETA: 17s - loss: 8.1775 - recall_at_10: 0.0313 - mrr_at_10: 0.0170 - ndcg_at_10: 0.0204 - recall_at_50: 0.0575 - mrr_at_50: 0.0182 - ndcg_at_50: 0.0260 - recall_at_100: 0.0779 - mrr_at_100: 0.0184 - ndcg_at_100: 0.0293 - regularization_loss: 0.1992 - _timestamp: 1657578247.2857 - _runtime: 13.2857��
139/500 [=======>......................] - ETA: 16s - loss: 8.1715 - recall_at_10: 0.0324 - mrr_at_10: 0.0176 - ndcg_at_10: 0.0211 - recall_at_50: 0.0592 - mrr_at_50: 0.0188 - ndcg_at_50: 0.0269 - recall_at_100: 0.0800 - mrr_at_100: 0.0191 - ndcg_at_100: 0.0303 - regularization_loss: 0.2010 - _timestamp: 1657578247.2857 - _runtime: 13.2857��
141/500 [=======>......................] - ETA: 16s - loss: 8.1651 - recall_at_10: 0.0335 - mrr_at_10: 0.0182 - ndcg_at_10: 0.0218 - recall_at_50: 0.0609 - mrr_at_50: 0.0195 - ndcg_at_50: 0.0278 - recall_at_100: 0.0820 - mrr_at_100: 0.0198 - ndcg_at_100: 0.0312 - regularization_loss: 0.2027 - _timestamp: 1657578247.5333 - _runtime: 13.5333��
143/500 [=======>......................] - ETA: 16s - loss: 8.1592 - recall_at_10: 0.0345 - mrr_at_10: 0.0189 - ndcg_at_10: 0.0225 - recall_at_50: 0.0626 - mrr_at_50: 0.0201 - ndcg_at_50: 0.0286 - recall_at_100: 0.0840 - mrr_at_100: 0.0204 - ndcg_at_100: 0.0321 - regularization_loss: 0.2044 - _timestamp: 1657578247.5333 - _runtime: 13.5333��
145/500 [=======>......................] - ETA: 16s - loss: 8.1535 - recall_at_10: 0.0355 - mrr_at_10: 0.0194 - ndcg_at_10: 0.0232 - recall_at_50: 0.0642 - mrr_at_50: 0.0207 - ndcg_at_50: 0.0295 - recall_at_100: 0.0859 - mrr_at_100: 0.0210 - ndcg_at_100: 0.0330 - regularization_loss: 0.2061 - _timestamp: 1657578247.5333 - _runtime: 13.5333��
147/500 [=======>......................] - ETA: 16s - loss: 8.1480 - recall_at_10: 0.0365 - mrr_at_10: 0.0200 - ndcg_at_10: 0.0239 - recall_at_50: 0.0658 - mrr_at_50: 0.0213 - ndcg_at_50: 0.0303 - recall_at_100: 0.0877 - mrr_at_100: 0.0216 - ndcg_at_100: 0.0338 - regularization_loss: 0.2078 - _timestamp: 1657578247.5333 - _runtime: 13.5333��
149/500 [=======>......................] - ETA: 16s - loss: 8.1424 - recall_at_10: 0.0375 - mrr_at_10: 0.0206 - ndcg_at_10: 0.0246 - recall_at_50: 0.0673 - mrr_at_50: 0.0219 - ndcg_at_50: 0.0310 - recall_at_100: 0.0895 - mrr_at_100: 0.0222 - ndcg_at_100: 0.0346 - regularization_loss: 0.2095 - _timestamp: 1657578247.5333 - _runtime: 13.5333��
151/500 [========>.....................] - ETA: 16s - loss: 8.1370 - recall_at_10: 0.0384 - mrr_at_10: 0.0211 - ndcg_at_10: 0.0252 - recall_at_50: 0.0688 - mrr_at_50: 0.0225 - ndcg_at_50: 0.0318 - recall_at_100: 0.0912 - mrr_at_100: 0.0228 - ndcg_at_100: 0.0354 - regularization_loss: 0.2112 - _timestamp: 1657578247.7500 - _runtime: 13.7500��
153/500 [========>.....................] - ETA: 16s - loss: 8.1315 - recall_at_10: 0.0393 - mrr_at_10: 0.0217 - ndcg_at_10: 0.0258 - recall_at_50: 0.0702 - mrr_at_50: 0.0231 - ndcg_at_50: 0.0325 - recall_at_100: 0.0929 - mrr_at_100: 0.0234 - ndcg_at_100: 0.0362 - regularization_loss: 0.2128 - _timestamp: 1657578247.7500 - _runtime: 13.7500��
155/500 [========>.....................] - ETA: 16s - loss: 8.1263 - recall_at_10: 0.0402 - mrr_at_10: 0.0222 - ndcg_at_10: 0.0264 - recall_at_50: 0.0716 - mrr_at_50: 0.0236 - ndcg_at_50: 0.0333 - recall_at_100: 0.0946 - mrr_at_100: 0.0239 - ndcg_at_100: 0.0370 - regularization_loss: 0.2144 - _timestamp: 1657578247.7500 - _runtime: 13.7500��
157/500 [========>.....................] - ETA: 15s - loss: 8.1212 - recall_at_10: 0.0411 - mrr_at_10: 0.0227 - ndcg_at_10: 0.0270 - recall_at_50: 0.0730 - mrr_at_50: 0.0241 - ndcg_at_50: 0.0340 - recall_at_100: 0.0962 - mrr_at_100: 0.0244 - ndcg_at_100: 0.0377 - regularization_loss: 0.2160 - _timestamp: 1657578247.7500 - _runtime: 13.7500��
159/500 [========>.....................] - ETA: 15s - loss: 8.1160 - recall_at_10: 0.0419 - mrr_at_10: 0.0232 - ndcg_at_10: 0.0276 - recall_at_50: 0.0743 - mrr_at_50: 0.0246 - ndcg_at_50: 0.0346 - recall_at_100: 0.0978 - mrr_at_100: 0.0250 - ndcg_at_100: 0.0384 - regularization_loss: 0.2176 - _timestamp: 1657578247.7500 - _runtime: 13.7500��
160/500 [========>.....................] - ETA: 15s - loss: 8.1135 - recall_at_10: 0.0423 - mrr_at_10: 0.0234 - ndcg_at_10: 0.0279 - recall_at_50: 0.0750 - mrr_at_50: 0.0249 - ndcg_at_50: 0.0350 - recall_at_100: 0.0986 - mrr_at_100: 0.0252 - ndcg_at_100: 0.0388 - regularization_loss: 0.2184 - _timestamp: 1657578247.7500 - _runtime: 13.7500��
162/500 [========>.....................] - ETA: 15s - loss: 8.1085 - recall_at_10: 0.0432 - mrr_at_10: 0.0239 - ndcg_at_10: 0.0284 - recall_at_50: 0.0763 - mrr_at_50: 0.0254 - ndcg_at_50: 0.0356 - recall_at_100: 0.1001 - mrr_at_100: 0.0257 - ndcg_at_100: 0.0395 - regularization_loss: 0.2200 - _timestamp: 1657578248.0000 - _runtime: 14.0000��
164/500 [========>.....................] - ETA: 15s - loss: 8.1036 - recall_at_10: 0.0439 - mrr_at_10: 0.0244 - ndcg_at_10: 0.0290 - recall_at_50: 0.0775 - mrr_at_50: 0.0259 - ndcg_at_50: 0.0363 - recall_at_100: 0.1016 - mrr_at_100: 0.0262 - ndcg_at_100: 0.0402 - regularization_loss: 0.2215 - _timestamp: 1657578248.0000 - _runtime: 14.0000��
166/500 [========>.....................] - ETA: 15s - loss: 8.0989 - recall_at_10: 0.0447 - mrr_at_10: 0.0248 - ndcg_at_10: 0.0295 - recall_at_50: 0.0787 - mrr_at_50: 0.0263 - ndcg_at_50: 0.0369 - recall_at_100: 0.1030 - mrr_at_100: 0.0267 - ndcg_at_100: 0.0408 - regularization_loss: 0.2231 - _timestamp: 1657578248.0000 - _runtime: 14.0000��
168/500 [=========>....................] - ETA: 15s - loss: 8.0938 - recall_at_10: 0.0455 - mrr_at_10: 0.0252 - ndcg_at_10: 0.0300 - recall_at_50: 0.0799 - mrr_at_50: 0.0268 - ndcg_at_50: 0.0375 - recall_at_100: 0.1044 - mrr_at_100: 0.0271 - ndcg_at_100: 0.0415 - regularization_loss: 0.2246 - _timestamp: 1657578248.0000 - _runtime: 14.0000��
170/500 [=========>....................] - ETA: 15s - loss: 8.0892 - recall_at_10: 0.0462 - mrr_at_10: 0.0257 - ndcg_at_10: 0.0305 - recall_at_50: 0.0811 - mrr_at_50: 0.0272 - ndcg_at_50: 0.0381 - recall_at_100: 0.1058 - mrr_at_100: 0.0276 - ndcg_at_100: 0.0421 - regularization_loss: 0.2261 - _timestamp: 1657578248.0000 - _runtime: 14.0000��
172/500 [=========>....................] - ETA: 15s - loss: 8.0845 - recall_at_10: 0.0469 - mrr_at_10: 0.0261 - ndcg_at_10: 0.0310 - recall_at_50: 0.0823 - mrr_at_50: 0.0277 - ndcg_at_50: 0.0387 - recall_at_100: 0.1071 - mrr_at_100: 0.0280 - ndcg_at_100: 0.0427 - regularization_loss: 0.2276 - _timestamp: 1657578248.2222 - _runtime: 14.2222��
174/500 [=========>....................] - ETA: 15s - loss: 8.0797 - recall_at_10: 0.0476 - mrr_at_10: 0.0265 - ndcg_at_10: 0.0315 - recall_at_50: 0.0834 - mrr_at_50: 0.0281 - ndcg_at_50: 0.0393 - recall_at_100: 0.1084 - mrr_at_100: 0.0285 - ndcg_at_100: 0.0433 - regularization_loss: 0.2291 - _timestamp: 1657578248.2222 - _runtime: 14.2222��
176/500 [=========>....................] - ETA: 15s - loss: 8.0748 - recall_at_10: 0.0483 - mrr_at_10: 0.0269 - ndcg_at_10: 0.0320 - recall_at_50: 0.0845 - mrr_at_50: 0.0285 - ndcg_at_50: 0.0398 - recall_at_100: 0.1097 - mrr_at_100: 0.0289 - ndcg_at_100: 0.0439 - regularization_loss: 0.2305 - _timestamp: 1657578248.2222 - _runtime: 14.2222��
178/500 [=========>....................] - ETA: 14s - loss: 8.0703 - recall_at_10: 0.0490 - mrr_at_10: 0.0273 - ndcg_at_10: 0.0324 - recall_at_50: 0.0855 - mrr_at_50: 0.0289 - ndcg_at_50: 0.0404 - recall_at_100: 0.1110 - mrr_at_100: 0.0293 - ndcg_at_100: 0.0445 - regularization_loss: 0.2320 - _timestamp: 1657578248.2222 - _runtime: 14.2222��
180/500 [=========>....................] - ETA: 14s - loss: 8.0659 - recall_at_10: 0.0496 - mrr_at_10: 0.0277 - ndcg_at_10: 0.0329 - recall_at_50: 0.0866 - mrr_at_50: 0.0293 - ndcg_at_50: 0.0409 - recall_at_100: 0.1122 - mrr_at_100: 0.0297 - ndcg_at_100: 0.0451 - regularization_loss: 0.2334 - _timestamp: 1657578248.2222 - _runtime: 14.2222��
182/500 [=========>....................] - ETA: 14s - loss: 8.0618 - recall_at_10: 0.0503 - mrr_at_10: 0.0281 - ndcg_at_10: 0.0333 - recall_at_50: 0.0876 - mrr_at_50: 0.0297 - ndcg_at_50: 0.0414 - recall_at_100: 0.1134 - mrr_at_100: 0.0301 - ndcg_at_100: 0.0456 - regularization_loss: 0.2348 - _timestamp: 1657578248.4211 - _runtime: 14.4211��
184/500 [==========>...................] - ETA: 14s - loss: 8.0575 - recall_at_10: 0.0509 - mrr_at_10: 0.0284 - ndcg_at_10: 0.0337 - recall_at_50: 0.0886 - mrr_at_50: 0.0301 - ndcg_at_50: 0.0419 - recall_at_100: 0.1146 - mrr_at_100: 0.0305 - ndcg_at_100: 0.0461 - regularization_loss: 0.2362 - _timestamp: 1657578248.4211 - _runtime: 14.4211��
186/500 [==========>...................] - ETA: 14s - loss: 8.0529 - recall_at_10: 0.0515 - mrr_at_10: 0.0288 - ndcg_at_10: 0.0342 - recall_at_50: 0.0895 - mrr_at_50: 0.0305 - ndcg_at_50: 0.0424 - recall_at_100: 0.1157 - mrr_at_100: 0.0309 - ndcg_at_100: 0.0467 - regularization_loss: 0.2376 - _timestamp: 1657578248.4211 - _runtime: 14.4211��
188/500 [==========>...................] - ETA: 14s - loss: 8.0489 - recall_at_10: 0.0521 - mrr_at_10: 0.0291 - ndcg_at_10: 0.0346 - recall_at_50: 0.0905 - mrr_at_50: 0.0309 - ndcg_at_50: 0.0429 - recall_at_100: 0.1168 - mrr_at_100: 0.0312 - ndcg_at_100: 0.0472 - regularization_loss: 0.2390 - _timestamp: 1657578248.4211 - _runtime: 14.4211��
190/500 [==========>...................] - ETA: 14s - loss: 8.0449 - recall_at_10: 0.0527 - mrr_at_10: 0.0295 - ndcg_at_10: 0.0350 - recall_at_50: 0.0914 - mrr_at_50: 0.0312 - ndcg_at_50: 0.0434 - recall_at_100: 0.1179 - mrr_at_100: 0.0316 - ndcg_at_100: 0.0477 - regularization_loss: 0.2403 - _timestamp: 1657578248.4211 - _runtime: 14.4211��
191/500 [==========>...................] - ETA: 14s - loss: 8.0429 - recall_at_10: 0.0530 - mrr_at_10: 0.0296 - ndcg_at_10: 0.0352 - recall_at_50: 0.0919 - mrr_at_50: 0.0314 - ndcg_at_50: 0.0436 - recall_at_100: 0.1185 - mrr_at_100: 0.0318 - ndcg_at_100: 0.0479 - regularization_loss: 0.2410 - _timestamp: 1657578248.6500 - _runtime: 14.6500��
192/500 [==========>...................] - ETA: 14s - loss: 8.0408 - recall_at_10: 0.0533 - mrr_at_10: 0.0298 - ndcg_at_10: 0.0354 - recall_at_50: 0.0924 - mrr_at_50: 0.0316 - ndcg_at_50: 0.0439 - recall_at_100: 0.1190 - mrr_at_100: 0.0319 - ndcg_at_100: 0.0482 - regularization_loss: 0.2417 - _timestamp: 1657578248.6500 - _runtime: 14.6500��
193/500 [==========>...................] - ETA: 14s - loss: 8.0387 - recall_at_10: 0.0536 - mrr_at_10: 0.0300 - ndcg_at_10: 0.0356 - recall_at_50: 0.0928 - mrr_at_50: 0.0317 - ndcg_at_50: 0.0441 - recall_at_100: 0.1195 - mrr_at_100: 0.0321 - ndcg_at_100: 0.0484 - regularization_loss: 0.2424 - _timestamp: 1657578248.6500 - _runtime: 14.6500��
195/500 [==========>...................] - ETA: 14s - loss: 8.0348 - recall_at_10: 0.0541 - mrr_at_10: 0.0303 - ndcg_at_10: 0.0359 - recall_at_50: 0.0937 - mrr_at_50: 0.0321 - ndcg_at_50: 0.0446 - recall_at_100: 0.1206 - mrr_at_100: 0.0325 - ndcg_at_100: 0.0489 - regularization_loss: 0.2437 - _timestamp: 1657578248.6500 - _runtime: 14.6500��
197/500 [==========>...................] - ETA: 14s - loss: 8.0309 - recall_at_10: 0.0547 - mrr_at_10: 0.0306 - ndcg_at_10: 0.0363 - recall_at_50: 0.0946 - mrr_at_50: 0.0324 - ndcg_at_50: 0.0450 - recall_at_100: 0.1216 - mrr_at_100: 0.0328 - ndcg_at_100: 0.0494 - regularization_loss: 0.2450 - _timestamp: 1657578248.6500 - _runtime: 14.6500��
199/500 [==========>...................] - ETA: 13s - loss: 8.0271 - recall_at_10: 0.0552 - mrr_at_10: 0.0309 - ndcg_at_10: 0.0367 - recall_at_50: 0.0954 - mrr_at_50: 0.0327 - ndcg_at_50: 0.0454 - recall_at_100: 0.1226 - mrr_at_100: 0.0331 - ndcg_at_100: 0.0498 - regularization_loss: 0.2463 - _timestamp: 1657578248.6500 - _runtime: 14.6500��
201/500 [===========>..................] - ETA: 13s - loss: 8.0234 - recall_at_10: 0.0561 - mrr_at_10: 0.0314 - ndcg_at_10: 0.0372 - recall_at_50: 0.0967 - mrr_at_50: 0.0333 - ndcg_at_50: 0.0461 - recall_at_100: 0.1241 - mrr_at_100: 0.0336 - ndcg_at_100: 0.0505 - regularization_loss: 0.2476 - _timestamp: 1657578248.8571 - _runtime: 14.8571��
203/500 [===========>..................] - ETA: 13s - loss: 8.0197 - recall_at_10: 0.0572 - mrr_at_10: 0.0321 - ndcg_at_10: 0.0380 - recall_at_50: 0.0983 - mrr_at_50: 0.0339 - ndcg_at_50: 0.0470 - recall_at_100: 0.1260 - mrr_at_100: 0.0343 - ndcg_at_100: 0.0514 - regularization_loss: 0.2489 - _timestamp: 1657578248.8571 - _runtime: 14.8571��
205/500 [===========>..................] - ETA: 13s - loss: 8.0159 - recall_at_10: 0.0583 - mrr_at_10: 0.0327 - ndcg_at_10: 0.0388 - recall_at_50: 0.1000 - mrr_at_50: 0.0346 - ndcg_at_50: 0.0478 - recall_at_100: 0.1279 - mrr_at_100: 0.0350 - ndcg_at_100: 0.0523 - regularization_loss: 0.2501 - _timestamp: 1657578248.8571 - _runtime: 14.8571��
207/500 [===========>..................] - ETA: 13s - loss: 8.0123 - recall_at_10: 0.0594 - mrr_at_10: 0.0333 - ndcg_at_10: 0.0395 - recall_at_50: 0.1016 - mrr_at_50: 0.0352 - ndcg_at_50: 0.0487 - recall_at_100: 0.1297 - mrr_at_100: 0.0356 - ndcg_at_100: 0.0532 - regularization_loss: 0.2514 - _timestamp: 1657578248.8571 - _runtime: 14.8571��
209/500 [===========>..................] - ETA: 13s - loss: 8.0087 - recall_at_10: 0.0604 - mrr_at_10: 0.0339 - ndcg_at_10: 0.0402 - recall_at_50: 0.1031 - mrr_at_50: 0.0359 - ndcg_at_50: 0.0495 - recall_at_100: 0.1315 - mrr_at_100: 0.0363 - ndcg_at_100: 0.0541 - regularization_loss: 0.2526 - _timestamp: 1657578248.8571 - _runtime: 14.8571��
211/500 [===========>..................] - ETA: 13s - loss: 8.0049 - recall_at_10: 0.0615 - mrr_at_10: 0.0345 - ndcg_at_10: 0.0409 - recall_at_50: 0.1047 - mrr_at_50: 0.0365 - ndcg_at_50: 0.0503 - recall_at_100: 0.1333 - mrr_at_100: 0.0369 - ndcg_at_100: 0.0549 - regularization_loss: 0.2539 - _timestamp: 1657578249.0909 - _runtime: 15.0909��
212/500 [===========>..................] - ETA: 13s - loss: 8.0030 - recall_at_10: 0.0620 - mrr_at_10: 0.0348 - ndcg_at_10: 0.0413 - recall_at_50: 0.1054 - mrr_at_50: 0.0368 - ndcg_at_50: 0.0507 - recall_at_100: 0.1341 - mrr_at_100: 0.0372 - ndcg_at_100: 0.0554 - regularization_loss: 0.2545 - _timestamp: 1657578249.0909 - _runtime: 15.0909��
214/500 [===========>..................] - ETA: 13s - loss: 7.9996 - recall_at_10: 0.0630 - mrr_at_10: 0.0354 - ndcg_at_10: 0.0419 - recall_at_50: 0.1069 - mrr_at_50: 0.0374 - ndcg_at_50: 0.0515 - recall_at_100: 0.1359 - mrr_at_100: 0.0378 - ndcg_at_100: 0.0562 - regularization_loss: 0.2557 - _timestamp: 1657578249.0909 - _runtime: 15.0909��
216/500 [===========>..................] - ETA: 13s - loss: 7.9962 - recall_at_10: 0.0640 - mrr_at_10: 0.0360 - ndcg_at_10: 0.0426 - recall_at_50: 0.1084 - mrr_at_50: 0.0380 - ndcg_at_50: 0.0523 - recall_at_100: 0.1376 - mrr_at_100: 0.0384 - ndcg_at_100: 0.0570 - regularization_loss: 0.2569 - _timestamp: 1657578249.0909 - _runtime: 15.0909��
217/500 [============>.................] - ETA: 13s - loss: 7.9945 - recall_at_10: 0.0645 - mrr_at_10: 0.0363 - ndcg_at_10: 0.0429 - recall_at_50: 0.1091 - mrr_at_50: 0.0383 - ndcg_at_50: 0.0527 - recall_at_100: 0.1384 - mrr_at_100: 0.0387 - ndcg_at_100: 0.0574 - regularization_loss: 0.2575 - _timestamp: 1657578249.0909 - _runtime: 15.0909��
218/500 [============>.................] - ETA: 13s - loss: 7.9929 - recall_at_10: 0.0650 - mrr_at_10: 0.0365 - ndcg_at_10: 0.0433 - recall_at_50: 0.1099 - mrr_at_50: 0.0386 - ndcg_at_50: 0.0531 - recall_at_100: 0.1392 - mrr_at_100: 0.0390 - ndcg_at_100: 0.0578 - regularization_loss: 0.2581 - _timestamp: 1657578249.0909 - _runtime: 15.0909��
220/500 [============>.................] - ETA: 12s - loss: 7.9897 - recall_at_10: 0.0659 - mrr_at_10: 0.0371 - ndcg_at_10: 0.0439 - recall_at_50: 0.1113 - mrr_at_50: 0.0391 - ndcg_at_50: 0.0538 - recall_at_100: 0.1408 - mrr_at_100: 0.0396 - ndcg_at_100: 0.0586 - regularization_loss: 0.2592 - _timestamp: 1657578249.0909 - _runtime: 15.0909��
222/500 [============>.................] - ETA: 12s - loss: 7.9865 - recall_at_10: 0.0669 - mrr_at_10: 0.0376 - ndcg_at_10: 0.0446 - recall_at_50: 0.1127 - mrr_at_50: 0.0397 - ndcg_at_50: 0.0545 - recall_at_100: 0.1424 - mrr_at_100: 0.0401 - ndcg_at_100: 0.0594 - regularization_loss: 0.2604 - _timestamp: 1657578249.3043 - _runtime: 15.3043��
224/500 [============>.................] - ETA: 12s - loss: 7.9832 - recall_at_10: 0.0678 - mrr_at_10: 0.0382 - ndcg_at_10: 0.0452 - recall_at_50: 0.1140 - mrr_at_50: 0.0403 - ndcg_at_50: 0.0553 - recall_at_100: 0.1440 - mrr_at_100: 0.0407 - ndcg_at_100: 0.0601 - regularization_loss: 0.2615 - _timestamp: 1657578249.3043 - _runtime: 15.3043��
226/500 [============>.................] - ETA: 12s - loss: 7.9799 - recall_at_10: 0.0687 - mrr_at_10: 0.0387 - ndcg_at_10: 0.0458 - recall_at_50: 0.1154 - mrr_at_50: 0.0408 - ndcg_at_50: 0.0560 - recall_at_100: 0.1455 - mrr_at_100: 0.0412 - ndcg_at_100: 0.0609 - regularization_loss: 0.2627 - _timestamp: 1657578249.3043 - _runtime: 15.3043��
228/500 [============>.................] - ETA: 12s - loss: 7.9766 - recall_at_10: 0.0696 - mrr_at_10: 0.0392 - ndcg_at_10: 0.0464 - recall_at_50: 0.1167 - mrr_at_50: 0.0413 - ndcg_at_50: 0.0567 - recall_at_100: 0.1471 - mrr_at_100: 0.0418 - ndcg_at_100: 0.0616 - regularization_loss: 0.2638 - _timestamp: 1657578249.3043 - _runtime: 15.3043��
230/500 [============>.................] - ETA: 12s - loss: 7.9731 - recall_at_10: 0.0705 - mrr_at_10: 0.0397 - ndcg_at_10: 0.0470 - recall_at_50: 0.1180 - mrr_at_50: 0.0419 - ndcg_at_50: 0.0574 - recall_at_100: 0.1485 - mrr_at_100: 0.0423 - ndcg_at_100: 0.0623 - regularization_loss: 0.2650 - _timestamp: 1657578249.3043 - _runtime: 15.3043��
232/500 [============>.................] - ETA: 12s - loss: 7.9699 - recall_at_10: 0.0714 - mrr_at_10: 0.0402 - ndcg_at_10: 0.0476 - recall_at_50: 0.1193 - mrr_at_50: 0.0424 - ndcg_at_50: 0.0580 - recall_at_100: 0.1500 - mrr_at_100: 0.0428 - ndcg_at_100: 0.0630 - regularization_loss: 0.2661 - _timestamp: 1657578249.5417 - _runtime: 15.5417��
234/500 [=============>................] - ETA: 12s - loss: 7.9666 - recall_at_10: 0.0722 - mrr_at_10: 0.0407 - ndcg_at_10: 0.0481 - recall_at_50: 0.1205 - mrr_at_50: 0.0429 - ndcg_at_50: 0.0587 - recall_at_100: 0.1514 - mrr_at_100: 0.0433 - ndcg_at_100: 0.0637 - regularization_loss: 0.2671 - _timestamp: 1657578249.5417 - _runtime: 15.5417��
236/500 [=============>................] - ETA: 12s - loss: 7.9636 - recall_at_10: 0.0730 - mrr_at_10: 0.0412 - ndcg_at_10: 0.0487 - recall_at_50: 0.1218 - mrr_at_50: 0.0434 - ndcg_at_50: 0.0593 - recall_at_100: 0.1529 - mrr_at_100: 0.0438 - ndcg_at_100: 0.0644 - regularization_loss: 0.2682 - _timestamp: 1657578249.5417 - _runtime: 15.5417��
237/500 [=============>................] - ETA: 12s - loss: 7.9622 - recall_at_10: 0.0734 - mrr_at_10: 0.0414 - ndcg_at_10: 0.0490 - recall_at_50: 0.1224 - mrr_at_50: 0.0436 - ndcg_at_50: 0.0597 - recall_at_100: 0.1536 - mrr_at_100: 0.0441 - ndcg_at_100: 0.0647 - regularization_loss: 0.2688 - _timestamp: 1657578249.5417 - _runtime: 15.5417��
239/500 [=============>................] - ETA: 12s - loss: 7.9640 - recall_at_10: 0.0743 - mrr_at_10: 0.0419 - ndcg_at_10: 0.0495 - recall_at_50: 0.1236 - mrr_at_50: 0.0441 - ndcg_at_50: 0.0603 - recall_at_100: 0.1549 - mrr_at_100: 0.0445 - ndcg_at_100: 0.0654 - regularization_loss: 0.2693 - _timestamp: 1657578249.5417 - _runtime: 15.5417��
240/500 [=============>................] - ETA: 12s - loss: 7.9672 - recall_at_10: 0.0747 - mrr_at_10: 0.0421 - ndcg_at_10: 0.0498 - recall_at_50: 0.1242 - mrr_at_50: 0.0443 - ndcg_at_50: 0.0606 - recall_at_100: 0.1556 - mrr_at_100: 0.0448 - ndcg_at_100: 0.0657 - regularization_loss: 0.2693 - _timestamp: 1657578249.5417 - _runtime: 15.5417��
241/500 [=============>................] - ETA: 12s - loss: 7.9701 - recall_at_10: 0.0751 - mrr_at_10: 0.0423 - ndcg_at_10: 0.0501 - recall_at_50: 0.1247 - mrr_at_50: 0.0446 - ndcg_at_50: 0.0609 - recall_at_100: 0.1563 - mrr_at_100: 0.0450 - ndcg_at_100: 0.0660 - regularization_loss: 0.2693 - _timestamp: 1657578249.7600 - _runtime: 15.7600��
242/500 [=============>................] - ETA: 12s - loss: 7.9729 - recall_at_10: 0.0755 - mrr_at_10: 0.0425 - ndcg_at_10: 0.0503 - recall_at_50: 0.1253 - mrr_at_50: 0.0448 - ndcg_at_50: 0.0612 - recall_at_100: 0.1570 - mrr_at_100: 0.0453 - ndcg_at_100: 0.0663 - regularization_loss: 0.2693 - _timestamp: 1657578249.7600 - _runtime: 15.7600��
243/500 [=============>................] - ETA: 12s - loss: 7.9754 - recall_at_10: 0.0758 - mrr_at_10: 0.0428 - ndcg_at_10: 0.0506 - recall_at_50: 0.1259 - mrr_at_50: 0.0450 - ndcg_at_50: 0.0615 - recall_at_100: 0.1576 - mrr_at_100: 0.0455 - ndcg_at_100: 0.0667 - regularization_loss: 0.2693 - _timestamp: 1657578249.7600 - _runtime: 15.7600��
244/500 [=============>................] - ETA: 12s - loss: 7.9775 - recall_at_10: 0.0762 - mrr_at_10: 0.0430 - ndcg_at_10: 0.0509 - recall_at_50: 0.1265 - mrr_at_50: 0.0453 - ndcg_at_50: 0.0618 - recall_at_100: 0.1583 - mrr_at_100: 0.0457 - ndcg_at_100: 0.0670 - regularization_loss: 0.2694 - _timestamp: 1657578249.7600 - _runtime: 15.7600��
245/500 [=============>................] - ETA: 12s - loss: 7.9796 - recall_at_10: 0.0766 - mrr_at_10: 0.0432 - ndcg_at_10: 0.0511 - recall_at_50: 0.1270 - mrr_at_50: 0.0455 - ndcg_at_50: 0.0621 - recall_at_100: 0.1589 - mrr_at_100: 0.0460 - ndcg_at_100: 0.0673 - regularization_loss: 0.2695 - _timestamp: 1657578249.7600 - _runtime: 15.7600��
247/500 [=============>................] - ETA: 11s - loss: 7.9834 - recall_at_10: 0.0774 - mrr_at_10: 0.0436 - ndcg_at_10: 0.0516 - recall_at_50: 0.1282 - mrr_at_50: 0.0460 - ndcg_at_50: 0.0627 - recall_at_100: 0.1602 - mrr_at_100: 0.0464 - ndcg_at_100: 0.0679 - regularization_loss: 0.2697 - _timestamp: 1657578249.7600 - _runtime: 15.7600��
249/500 [=============>................] - ETA: 11s - loss: 7.9865 - recall_at_10: 0.0781 - mrr_at_10: 0.0441 - ndcg_at_10: 0.0521 - recall_at_50: 0.1293 - mrr_at_50: 0.0464 - ndcg_at_50: 0.0633 - recall_at_100: 0.1615 - mrr_at_100: 0.0469 - ndcg_at_100: 0.0685 - regularization_loss: 0.2700 - _timestamp: 1657578249.7600 - _runtime: 15.7600��
251/500 [==============>...............] - ETA: 11s - loss: 7.9892 - recall_at_10: 0.0789 - mrr_at_10: 0.0445 - ndcg_at_10: 0.0526 - recall_at_50: 0.1304 - mrr_at_50: 0.0468 - ndcg_at_50: 0.0639 - recall_at_100: 0.1627 - mrr_at_100: 0.0473 - ndcg_at_100: 0.0691 - regularization_loss: 0.2705 - _timestamp: 1657578250.0000 - _runtime: 16.0000��
253/500 [==============>...............] - ETA: 11s - loss: 7.9913 - recall_at_10: 0.0796 - mrr_at_10: 0.0449 - ndcg_at_10: 0.0531 - recall_at_50: 0.1314 - mrr_at_50: 0.0473 - ndcg_at_50: 0.0645 - recall_at_100: 0.1640 - mrr_at_100: 0.0477 - ndcg_at_100: 0.0697 - regularization_loss: 0.2711 - _timestamp: 1657578250.0000 - _runtime: 16.0000��
255/500 [==============>...............] - ETA: 11s - loss: 7.9933 - recall_at_10: 0.0803 - mrr_at_10: 0.0453 - ndcg_at_10: 0.0536 - recall_at_50: 0.1325 - mrr_at_50: 0.0477 - ndcg_at_50: 0.0650 - recall_at_100: 0.1652 - mrr_at_100: 0.0482 - ndcg_at_100: 0.0703 - regularization_loss: 0.2718 - _timestamp: 1657578250.0000 - _runtime: 16.0000��
257/500 [==============>...............] - ETA: 11s - loss: 7.9952 - recall_at_10: 0.0810 - mrr_at_10: 0.0457 - ndcg_at_10: 0.0541 - recall_at_50: 0.1335 - mrr_at_50: 0.0481 - ndcg_at_50: 0.0656 - recall_at_100: 0.1664 - mrr_at_100: 0.0486 - ndcg_at_100: 0.0709 - regularization_loss: 0.2726 - _timestamp: 1657578250.0000 - _runtime: 16.0000��
259/500 [==============>...............] - ETA: 11s - loss: 7.9967 - recall_at_10: 0.0817 - mrr_at_10: 0.0461 - ndcg_at_10: 0.0546 - recall_at_50: 0.1345 - mrr_at_50: 0.0485 - ndcg_at_50: 0.0661 - recall_at_100: 0.1675 - mrr_at_100: 0.0490 - ndcg_at_100: 0.0714 - regularization_loss: 0.2734 - _timestamp: 1657578250.0000 - _runtime: 16.0000��
261/500 [==============>...............] - ETA: 11s - loss: 7.9978 - recall_at_10: 0.0824 - mrr_at_10: 0.0465 - ndcg_at_10: 0.0550 - recall_at_50: 0.1355 - mrr_at_50: 0.0489 - ndcg_at_50: 0.0666 - recall_at_100: 0.1687 - mrr_at_100: 0.0494 - ndcg_at_100: 0.0720 - regularization_loss: 0.2744 - _timestamp: 1657578250.2222 - _runtime: 16.2222��
263/500 [==============>...............] - ETA: 11s - loss: 7.9990 - recall_at_10: 0.0831 - mrr_at_10: 0.0469 - ndcg_at_10: 0.0555 - recall_at_50: 0.1365 - mrr_at_50: 0.0493 - ndcg_at_50: 0.0672 - recall_at_100: 0.1698 - mrr_at_100: 0.0498 - ndcg_at_100: 0.0725 - regularization_loss: 0.2754 - _timestamp: 1657578250.2222 - _runtime: 16.2222��
265/500 [==============>...............] - ETA: 11s - loss: 8.0000 - recall_at_10: 0.0837 - mrr_at_10: 0.0473 - ndcg_at_10: 0.0559 - recall_at_50: 0.1375 - mrr_at_50: 0.0497 - ndcg_at_50: 0.0677 - recall_at_100: 0.1709 - mrr_at_100: 0.0502 - ndcg_at_100: 0.0731 - regularization_loss: 0.2765 - _timestamp: 1657578250.2222 - _runtime: 16.2222��
267/500 [===============>..............] - ETA: 10s - loss: 8.0008 - recall_at_10: 0.0844 - mrr_at_10: 0.0476 - ndcg_at_10: 0.0564 - recall_at_50: 0.1385 - mrr_at_50: 0.0501 - ndcg_at_50: 0.0682 - recall_at_100: 0.1720 - mrr_at_100: 0.0506 - ndcg_at_100: 0.0736 - regularization_loss: 0.2777 - _timestamp: 1657578250.2222 - _runtime: 16.2222��
269/500 [===============>..............] - ETA: 10s - loss: 8.0015 - recall_at_10: 0.0850 - mrr_at_10: 0.0480 - ndcg_at_10: 0.0568 - recall_at_50: 0.1394 - mrr_at_50: 0.0505 - ndcg_at_50: 0.0687 - recall_at_100: 0.1731 - mrr_at_100: 0.0510 - ndcg_at_100: 0.0741 - regularization_loss: 0.2789 - _timestamp: 1657578250.2222 - _runtime: 16.2222��
271/500 [===============>..............] - ETA: 10s - loss: 8.0021 - recall_at_10: 0.0856 - mrr_at_10: 0.0484 - ndcg_at_10: 0.0572 - recall_at_50: 0.1404 - mrr_at_50: 0.0509 - ndcg_at_50: 0.0692 - recall_at_100: 0.1742 - mrr_at_100: 0.0514 - ndcg_at_100: 0.0747 - regularization_loss: 0.2801 - _timestamp: 1657578250.4643 - _runtime: 16.4643��
273/500 [===============>..............] - ETA: 10s - loss: 8.0026 - recall_at_10: 0.0863 - mrr_at_10: 0.0487 - ndcg_at_10: 0.0576 - recall_at_50: 0.1413 - mrr_at_50: 0.0512 - ndcg_at_50: 0.0697 - recall_at_100: 0.1753 - mrr_at_100: 0.0517 - ndcg_at_100: 0.0752 - regularization_loss: 0.2814 - _timestamp: 1657578250.4643 - _runtime: 16.4643��
275/500 [===============>..............] - ETA: 10s - loss: 8.0029 - recall_at_10: 0.0869 - mrr_at_10: 0.0491 - ndcg_at_10: 0.0580 - recall_at_50: 0.1422 - mrr_at_50: 0.0516 - ndcg_at_50: 0.0701 - recall_at_100: 0.1763 - mrr_at_100: 0.0521 - ndcg_at_100: 0.0757 - regularization_loss: 0.2827 - _timestamp: 1657578250.4643 - _runtime: 16.4643��
277/500 [===============>..............] - ETA: 10s - loss: 8.0030 - recall_at_10: 0.0875 - mrr_at_10: 0.0494 - ndcg_at_10: 0.0585 - recall_at_50: 0.1431 - mrr_at_50: 0.0520 - ndcg_at_50: 0.0706 - recall_at_100: 0.1773 - mrr_at_100: 0.0525 - ndcg_at_100: 0.0762 - regularization_loss: 0.2840 - _timestamp: 1657578250.4643 - _runtime: 16.4643��
279/500 [===============>..............] - ETA: 10s - loss: 8.0032 - recall_at_10: 0.0881 - mrr_at_10: 0.0498 - ndcg_at_10: 0.0589 - recall_at_50: 0.1440 - mrr_at_50: 0.0523 - ndcg_at_50: 0.0711 - recall_at_100: 0.1783 - mrr_at_100: 0.0528 - ndcg_at_100: 0.0766 - regularization_loss: 0.2853 - _timestamp: 1657578250.4643 - _runtime: 16.4643��
281/500 [===============>..............] - ETA: 10s - loss: 8.0031 - recall_at_10: 0.0887 - mrr_at_10: 0.0501 - ndcg_at_10: 0.0593 - recall_at_50: 0.1448 - mrr_at_50: 0.0527 - ndcg_at_50: 0.0715 - recall_at_100: 0.1793 - mrr_at_100: 0.0532 - ndcg_at_100: 0.0771 - regularization_loss: 0.2866 - _timestamp: 1657578250.6897 - _runtime: 16.6897��
283/500 [===============>..............] - ETA: 10s - loss: 8.0030 - recall_at_10: 0.0892 - mrr_at_10: 0.0504 - ndcg_at_10: 0.0596 - recall_at_50: 0.1457 - mrr_at_50: 0.0530 - ndcg_at_50: 0.0720 - recall_at_100: 0.1803 - mrr_at_100: 0.0535 - ndcg_at_100: 0.0776 - regularization_loss: 0.2879 - _timestamp: 1657578250.6897 - _runtime: 16.6897��
284/500 [================>.............] - ETA: 10s - loss: 8.0029 - recall_at_10: 0.0895 - mrr_at_10: 0.0506 - ndcg_at_10: 0.0598 - recall_at_50: 0.1461 - mrr_at_50: 0.0532 - ndcg_at_50: 0.0722 - recall_at_100: 0.1808 - mrr_at_100: 0.0537 - ndcg_at_100: 0.0778 - regularization_loss: 0.2886 - _timestamp: 1657578250.6897 - _runtime: 16.6897��
286/500 [================>.............] - ETA: 10s - loss: 8.0026 - recall_at_10: 0.0901 - mrr_at_10: 0.0509 - ndcg_at_10: 0.0602 - recall_at_50: 0.1469 - mrr_at_50: 0.0535 - ndcg_at_50: 0.0727 - recall_at_100: 0.1817 - mrr_at_100: 0.0540 - ndcg_at_100: 0.0783 - regularization_loss: 0.2899 - _timestamp: 1657578250.6897 - _runtime: 16.6897��
288/500 [================>.............] - ETA: 9s - loss: 8.0024 - recall_at_10: 0.0906 - mrr_at_10: 0.0512 - ndcg_at_10: 0.0606 - recall_at_50: 0.1478 - mrr_at_50: 0.0539 - ndcg_at_50: 0.0731 - recall_at_100: 0.1827 - mrr_at_100: 0.0544 - ndcg_at_100: 0.0787 - regularization_loss: 0.2912 - _timestamp: 1657578250.6897 - _runtime: 16.6897 ��
290/500 [================>.............] - ETA: 9s - loss: 8.0019 - recall_at_10: 0.0912 - mrr_at_10: 0.0516 - ndcg_at_10: 0.0610 - recall_at_50: 0.1486 - mrr_at_50: 0.0542 - ndcg_at_50: 0.0735 - recall_at_100: 0.1836 - mrr_at_100: 0.0547 - ndcg_at_100: 0.0792 - regularization_loss: 0.2925 - _timestamp: 1657578250.6897 - _runtime: 16.6897��
292/500 [================>.............] - ETA: 9s - loss: 8.0015 - recall_at_10: 0.0917 - mrr_at_10: 0.0519 - ndcg_at_10: 0.0613 - recall_at_50: 0.1494 - mrr_at_50: 0.0545 - ndcg_at_50: 0.0739 - recall_at_100: 0.1845 - mrr_at_100: 0.0550 - ndcg_at_100: 0.0796 - regularization_loss: 0.2938 - _timestamp: 1657578250.9333 - _runtime: 16.9333��
294/500 [================>.............] - ETA: 9s - loss: 8.0010 - recall_at_10: 0.0923 - mrr_at_10: 0.0522 - ndcg_at_10: 0.0617 - recall_at_50: 0.1502 - mrr_at_50: 0.0548 - ndcg_at_50: 0.0744 - recall_at_100: 0.1855 - mrr_at_100: 0.0553 - ndcg_at_100: 0.0801 - regularization_loss: 0.2951 - _timestamp: 1657578250.9333 - _runtime: 16.9333��
296/500 [================>.............] - ETA: 9s - loss: 8.0002 - recall_at_10: 0.0928 - mrr_at_10: 0.0525 - ndcg_at_10: 0.0621 - recall_at_50: 0.1509 - mrr_at_50: 0.0552 - ndcg_at_50: 0.0748 - recall_at_100: 0.1864 - mrr_at_100: 0.0557 - ndcg_at_100: 0.0805 - regularization_loss: 0.2964 - _timestamp: 1657578250.9333 - _runtime: 16.9333��
298/500 [================>.............] - ETA: 9s - loss: 7.9994 - recall_at_10: 0.0933 - mrr_at_10: 0.0528 - ndcg_at_10: 0.0624 - recall_at_50: 0.1517 - mrr_at_50: 0.0555 - ndcg_at_50: 0.0752 - recall_at_100: 0.1872 - mrr_at_100: 0.0560 - ndcg_at_100: 0.0809 - regularization_loss: 0.2977 - _timestamp: 1657578250.9333 - _runtime: 16.9333��
300/500 [=================>............] - ETA: 9s - loss: 7.9986 - recall_at_10: 0.0938 - mrr_at_10: 0.0531 - ndcg_at_10: 0.0628 - recall_at_50: 0.1525 - mrr_at_50: 0.0558 - ndcg_at_50: 0.0756 - recall_at_100: 0.1881 - mrr_at_100: 0.0563 - ndcg_at_100: 0.0813 - regularization_loss: 0.2990 - _timestamp: 1657578250.9333 - _runtime: 16.9333��
301/500 [=================>............] - ETA: 9s - loss: 7.9981 - recall_at_10: 0.0942 - mrr_at_10: 0.0533 - ndcg_at_10: 0.0630 - recall_at_50: 0.1530 - mrr_at_50: 0.0560 - ndcg_at_50: 0.0758 - recall_at_100: 0.1887 - mrr_at_100: 0.0565 - ndcg_at_100: 0.0816 - regularization_loss: 0.2997 - _timestamp: 1657578251.1613 - _runtime: 17.1613��
303/500 [=================>............] - ETA: 9s - loss: 7.9971 - recall_at_10: 0.0948 - mrr_at_10: 0.0536 - ndcg_at_10: 0.0634 - recall_at_50: 0.1540 - mrr_at_50: 0.0563 - ndcg_at_50: 0.0763 - recall_at_100: 0.1898 - mrr_at_100: 0.0568 - ndcg_at_100: 0.0821 - regularization_loss: 0.3009 - _timestamp: 1657578251.1613 - _runtime: 17.1613��
304/500 [=================>............] - ETA: 9s - loss: 7.9967 - recall_at_10: 0.0951 - mrr_at_10: 0.0538 - ndcg_at_10: 0.0636 - recall_at_50: 0.1545 - mrr_at_50: 0.0565 - ndcg_at_50: 0.0766 - recall_at_100: 0.1904 - mrr_at_100: 0.0570 - ndcg_at_100: 0.0824 - regularization_loss: 0.3016 - _timestamp: 1657578251.1613 - _runtime: 17.1613��
306/500 [=================>............] - ETA: 9s - loss: 7.9958 - recall_at_10: 0.0958 - mrr_at_10: 0.0541 - ndcg_at_10: 0.0640 - recall_at_50: 0.1555 - mrr_at_50: 0.0569 - ndcg_at_50: 0.0771 - recall_at_100: 0.1916 - mrr_at_100: 0.0574 - ndcg_at_100: 0.0829 - regularization_loss: 0.3028 - _timestamp: 1657578251.1613 - _runtime: 17.1613��
308/500 [=================>............] - ETA: 8s - loss: 7.9947 - recall_at_10: 0.0964 - mrr_at_10: 0.0545 - ndcg_at_10: 0.0644 - recall_at_50: 0.1564 - mrr_at_50: 0.0572 - ndcg_at_50: 0.0776 - recall_at_100: 0.1927 - mrr_at_100: 0.0577 - ndcg_at_100: 0.0834 - regularization_loss: 0.3040 - _timestamp: 1657578251.1613 - _runtime: 17.1613��
310/500 [=================>............] - ETA: 8s - loss: 7.9936 - recall_at_10: 0.0970 - mrr_at_10: 0.0548 - ndcg_at_10: 0.0648 - recall_at_50: 0.1574 - mrr_at_50: 0.0576 - ndcg_at_50: 0.0780 - recall_at_100: 0.1938 - mrr_at_100: 0.0581 - ndcg_at_100: 0.0839 - regularization_loss: 0.3053 - _timestamp: 1657578251.1613 - _runtime: 17.1613��
312/500 [=================>............] - ETA: 8s - loss: 7.9924 - recall_at_10: 0.0977 - mrr_at_10: 0.0552 - ndcg_at_10: 0.0652 - recall_at_50: 0.1583 - mrr_at_50: 0.0579 - ndcg_at_50: 0.0785 - recall_at_100: 0.1949 - mrr_at_100: 0.0585 - ndcg_at_100: 0.0844 - regularization_loss: 0.3065 - _timestamp: 1657578251.4062 - _runtime: 17.4062��
314/500 [=================>............] - ETA: 8s - loss: 7.9912 - recall_at_10: 0.0983 - mrr_at_10: 0.0555 - ndcg_at_10: 0.0656 - recall_at_50: 0.1592 - mrr_at_50: 0.0583 - ndcg_at_50: 0.0790 - recall_at_100: 0.1959 - mrr_at_100: 0.0588 - ndcg_at_100: 0.0849 - regularization_loss: 0.3077 - _timestamp: 1657578251.4062 - _runtime: 17.4062��
316/500 [=================>............] - ETA: 8s - loss: 7.9900 - recall_at_10: 0.0989 - mrr_at_10: 0.0558 - ndcg_at_10: 0.0660 - recall_at_50: 0.1602 - mrr_at_50: 0.0586 - ndcg_at_50: 0.0794 - recall_at_100: 0.1970 - mrr_at_100: 0.0591 - ndcg_at_100: 0.0854 - regularization_loss: 0.3089 - _timestamp: 1657578251.4062 - _runtime: 17.4062��
318/500 [==================>...........] - ETA: 8s - loss: 7.9887 - recall_at_10: 0.0995 - mrr_at_10: 0.0561 - ndcg_at_10: 0.0664 - recall_at_50: 0.1611 - mrr_at_50: 0.0590 - ndcg_at_50: 0.0799 - recall_at_100: 0.1980 - mrr_at_100: 0.0595 - ndcg_at_100: 0.0859 - regularization_loss: 0.3100 - _timestamp: 1657578251.4062 - _runtime: 17.4062��
320/500 [==================>...........] - ETA: 8s - loss: 7.9875 - recall_at_10: 0.1000 - mrr_at_10: 0.0564 - ndcg_at_10: 0.0668 - recall_at_50: 0.1620 - mrr_at_50: 0.0593 - ndcg_at_50: 0.0803 - recall_at_100: 0.1991 - mrr_at_100: 0.0598 - ndcg_at_100: 0.0864 - regularization_loss: 0.3112 - _timestamp: 1657578251.4062 - _runtime: 17.4062��
322/500 [==================>...........] - ETA: 8s - loss: 7.9863 - recall_at_10: 0.1006 - mrr_at_10: 0.0568 - ndcg_at_10: 0.0672 - recall_at_50: 0.1628 - mrr_at_50: 0.0596 - ndcg_at_50: 0.0808 - recall_at_100: 0.2001 - mrr_at_100: 0.0601 - ndcg_at_100: 0.0868 - regularization_loss: 0.3123 - _timestamp: 1657578251.6364 - _runtime: 17.6364��
324/500 [==================>...........] - ETA: 8s - loss: 7.9851 - recall_at_10: 0.1012 - mrr_at_10: 0.0571 - ndcg_at_10: 0.0675 - recall_at_50: 0.1637 - mrr_at_50: 0.0599 - ndcg_at_50: 0.0812 - recall_at_100: 0.2011 - mrr_at_100: 0.0605 - ndcg_at_100: 0.0873 - regularization_loss: 0.3135 - _timestamp: 1657578251.6364 - _runtime: 17.6364��
326/500 [==================>...........] - ETA: 8s - loss: 7.9837 - recall_at_10: 0.1017 - mrr_at_10: 0.0574 - ndcg_at_10: 0.0679 - recall_at_50: 0.1646 - mrr_at_50: 0.0603 - ndcg_at_50: 0.0817 - recall_at_100: 0.2021 - mrr_at_100: 0.0608 - ndcg_at_100: 0.0877 - regularization_loss: 0.3146 - _timestamp: 1657578251.6364 - _runtime: 17.6364��
328/500 [==================>...........] - ETA: 7s - loss: 7.9825 - recall_at_10: 0.1023 - mrr_at_10: 0.0577 - ndcg_at_10: 0.0683 - recall_at_50: 0.1654 - mrr_at_50: 0.0606 - ndcg_at_50: 0.0821 - recall_at_100: 0.2031 - mrr_at_100: 0.0611 - ndcg_at_100: 0.0882 - regularization_loss: 0.3157 - _timestamp: 1657578251.6364 - _runtime: 17.6364��
330/500 [==================>...........] - ETA: 7s - loss: 7.9810 - recall_at_10: 0.1029 - mrr_at_10: 0.0580 - ndcg_at_10: 0.0686 - recall_at_50: 0.1663 - mrr_at_50: 0.0609 - ndcg_at_50: 0.0825 - recall_at_100: 0.2041 - mrr_at_100: 0.0614 - ndcg_at_100: 0.0886 - regularization_loss: 0.3168 - _timestamp: 1657578251.6364 - _runtime: 17.6364��
332/500 [==================>...........] - ETA: 7s - loss: 7.9796 - recall_at_10: 0.1034 - mrr_at_10: 0.0583 - ndcg_at_10: 0.0690 - recall_at_50: 0.1671 - mrr_at_50: 0.0612 - ndcg_at_50: 0.0829 - recall_at_100: 0.2050 - mrr_at_100: 0.0617 - ndcg_at_100: 0.0891 - regularization_loss: 0.3179 - _timestamp: 1657578251.8529 - _runtime: 17.8529��
334/500 [===================>..........] - ETA: 7s - loss: 7.9781 - recall_at_10: 0.1039 - mrr_at_10: 0.0586 - ndcg_at_10: 0.0693 - recall_at_50: 0.1679 - mrr_at_50: 0.0615 - ndcg_at_50: 0.0833 - recall_at_100: 0.2060 - mrr_at_100: 0.0620 - ndcg_at_100: 0.0895 - regularization_loss: 0.3190 - _timestamp: 1657578251.8529 - _runtime: 17.8529��
336/500 [===================>..........] - ETA: 7s - loss: 7.9767 - recall_at_10: 0.1045 - mrr_at_10: 0.0589 - ndcg_at_10: 0.0697 - recall_at_50: 0.1687 - mrr_at_50: 0.0618 - ndcg_at_50: 0.0838 - recall_at_100: 0.2069 - mrr_at_100: 0.0624 - ndcg_at_100: 0.0899 - regularization_loss: 0.3200 - _timestamp: 1657578251.8529 - _runtime: 17.8529��
338/500 [===================>..........] - ETA: 7s - loss: 7.9751 - recall_at_10: 0.1050 - mrr_at_10: 0.0591 - ndcg_at_10: 0.0700 - recall_at_50: 0.1695 - mrr_at_50: 0.0621 - ndcg_at_50: 0.0842 - recall_at_100: 0.2079 - mrr_at_100: 0.0627 - ndcg_at_100: 0.0904 - regularization_loss: 0.3211 - _timestamp: 1657578251.8529 - _runtime: 17.8529��
340/500 [===================>..........] - ETA: 7s - loss: 7.9735 - recall_at_10: 0.1055 - mrr_at_10: 0.0594 - ndcg_at_10: 0.0704 - recall_at_50: 0.1703 - mrr_at_50: 0.0624 - ndcg_at_50: 0.0846 - recall_at_100: 0.2088 - mrr_at_100: 0.0629 - ndcg_at_100: 0.0908 - regularization_loss: 0.3221 - _timestamp: 1657578251.8529 - _runtime: 17.8529��
342/500 [===================>..........] - ETA: 7s - loss: 7.9720 - recall_at_10: 0.1060 - mrr_at_10: 0.0597 - ndcg_at_10: 0.0707 - recall_at_50: 0.1711 - mrr_at_50: 0.0627 - ndcg_at_50: 0.0849 - recall_at_100: 0.2097 - mrr_at_100: 0.0632 - ndcg_at_100: 0.0912 - regularization_loss: 0.3231 - _timestamp: 1657578252.0857 - _runtime: 18.0857��
344/500 [===================>..........] - ETA: 7s - loss: 7.9705 - recall_at_10: 0.1065 - mrr_at_10: 0.0600 - ndcg_at_10: 0.0710 - recall_at_50: 0.1719 - mrr_at_50: 0.0630 - ndcg_at_50: 0.0853 - recall_at_100: 0.2106 - mrr_at_100: 0.0635 - ndcg_at_100: 0.0916 - regularization_loss: 0.3241 - _timestamp: 1657578252.0857 - _runtime: 18.0857��
346/500 [===================>..........] - ETA: 7s - loss: 7.9690 - recall_at_10: 0.1070 - mrr_at_10: 0.0602 - ndcg_at_10: 0.0714 - recall_at_50: 0.1726 - mrr_at_50: 0.0633 - ndcg_at_50: 0.0857 - recall_at_100: 0.2115 - mrr_at_100: 0.0638 - ndcg_at_100: 0.0920 - regularization_loss: 0.3251 - _timestamp: 1657578252.0857 - _runtime: 18.0857��
348/500 [===================>..........] - ETA: 7s - loss: 7.9673 - recall_at_10: 0.1075 - mrr_at_10: 0.0605 - ndcg_at_10: 0.0717 - recall_at_50: 0.1734 - mrr_at_50: 0.0635 - ndcg_at_50: 0.0861 - recall_at_100: 0.2123 - mrr_at_100: 0.0641 - ndcg_at_100: 0.0924 - regularization_loss: 0.3261 - _timestamp: 1657578252.0857 - _runtime: 18.0857��
350/500 [====================>.........] - ETA: 6s - loss: 7.9658 - recall_at_10: 0.1080 - mrr_at_10: 0.0608 - ndcg_at_10: 0.0720 - recall_at_50: 0.1741 - mrr_at_50: 0.0638 - ndcg_at_50: 0.0865 - recall_at_100: 0.2132 - mrr_at_100: 0.0644 - ndcg_at_100: 0.0928 - regularization_loss: 0.3271 - _timestamp: 1657578252.0857 - _runtime: 18.0857��
352/500 [====================>.........] - ETA: 6s - loss: 7.9642 - recall_at_10: 0.1085 - mrr_at_10: 0.0610 - ndcg_at_10: 0.0723 - recall_at_50: 0.1749 - mrr_at_50: 0.0641 - ndcg_at_50: 0.0868 - recall_at_100: 0.2141 - mrr_at_100: 0.0647 - ndcg_at_100: 0.0932 - regularization_loss: 0.3281 - _timestamp: 1657578252.3056 - _runtime: 18.3056��
354/500 [====================>.........] - ETA: 6s - loss: 7.9626 - recall_at_10: 0.1090 - mrr_at_10: 0.0613 - ndcg_at_10: 0.0726 - recall_at_50: 0.1756 - mrr_at_50: 0.0644 - ndcg_at_50: 0.0872 - recall_at_100: 0.2149 - mrr_at_100: 0.0649 - ndcg_at_100: 0.0936 - regularization_loss: 0.3290 - _timestamp: 1657578252.3056 - _runtime: 18.3056��
356/500 [====================>.........] - ETA: 6s - loss: 7.9610 - recall_at_10: 0.1094 - mrr_at_10: 0.0616 - ndcg_at_10: 0.0729 - recall_at_50: 0.1763 - mrr_at_50: 0.0646 - ndcg_at_50: 0.0876 - recall_at_100: 0.2157 - mrr_at_100: 0.0652 - ndcg_at_100: 0.0939 - regularization_loss: 0.3300 - _timestamp: 1657578252.3056 - _runtime: 18.3056��
358/500 [====================>.........] - ETA: 6s - loss: 7.9595 - recall_at_10: 0.1099 - mrr_at_10: 0.0618 - ndcg_at_10: 0.0732 - recall_at_50: 0.1770 - mrr_at_50: 0.0649 - ndcg_at_50: 0.0879 - recall_at_100: 0.2166 - mrr_at_100: 0.0655 - ndcg_at_100: 0.0943 - regularization_loss: 0.3309 - _timestamp: 1657578252.3056 - _runtime: 18.3056��
360/500 [====================>.........] - ETA: 6s - loss: 7.9579 - recall_at_10: 0.1104 - mrr_at_10: 0.0621 - ndcg_at_10: 0.0735 - recall_at_50: 0.1777 - mrr_at_50: 0.0652 - ndcg_at_50: 0.0883 - recall_at_100: 0.2174 - mrr_at_100: 0.0657 - ndcg_at_100: 0.0947 - regularization_loss: 0.3318 - _timestamp: 1657578252.3056 - _runtime: 18.3056��
362/500 [====================>.........] - ETA: 6s - loss: 7.9562 - recall_at_10: 0.1108 - mrr_at_10: 0.0623 - ndcg_at_10: 0.0738 - recall_at_50: 0.1784 - mrr_at_50: 0.0654 - ndcg_at_50: 0.0886 - recall_at_100: 0.2182 - mrr_at_100: 0.0660 - ndcg_at_100: 0.0951 - regularization_loss: 0.3328 - _timestamp: 1657578252.5405 - _runtime: 18.5405��
364/500 [====================>.........] - ETA: 6s - loss: 7.9546 - recall_at_10: 0.1113 - mrr_at_10: 0.0626 - ndcg_at_10: 0.0741 - recall_at_50: 0.1791 - mrr_at_50: 0.0657 - ndcg_at_50: 0.0890 - recall_at_100: 0.2190 - mrr_at_100: 0.0662 - ndcg_at_100: 0.0954 - regularization_loss: 0.3337 - _timestamp: 1657578252.5405 - _runtime: 18.5405��
366/500 [====================>.........] - ETA: 6s - loss: 7.9528 - recall_at_10: 0.1117 - mrr_at_10: 0.0628 - ndcg_at_10: 0.0744 - recall_at_50: 0.1798 - mrr_at_50: 0.0659 - ndcg_at_50: 0.0893 - recall_at_100: 0.2198 - mrr_at_100: 0.0665 - ndcg_at_100: 0.0958 - regularization_loss: 0.3346 - _timestamp: 1657578252.5405 - _runtime: 18.5405��
368/500 [=====================>........] - ETA: 6s - loss: 7.9513 - recall_at_10: 0.1122 - mrr_at_10: 0.0630 - ndcg_at_10: 0.0747 - recall_at_50: 0.1805 - mrr_at_50: 0.0662 - ndcg_at_50: 0.0897 - recall_at_100: 0.2206 - mrr_at_100: 0.0668 - ndcg_at_100: 0.0961 - regularization_loss: 0.3355 - _timestamp: 1657578252.5405 - _runtime: 18.5405��
370/500 [=====================>........] - ETA: 5s - loss: 7.9496 - recall_at_10: 0.1126 - mrr_at_10: 0.0633 - ndcg_at_10: 0.0750 - recall_at_50: 0.1812 - mrr_at_50: 0.0664 - ndcg_at_50: 0.0900 - recall_at_100: 0.2213 - mrr_at_100: 0.0670 - ndcg_at_100: 0.0965 - regularization_loss: 0.3363 - _timestamp: 1657578252.5405 - _runtime: 18.5405��
371/500 [=====================>........] - ETA: 5s - loss: 7.9487 - recall_at_10: 0.1128 - mrr_at_10: 0.0634 - ndcg_at_10: 0.0751 - recall_at_50: 0.1815 - mrr_at_50: 0.0666 - ndcg_at_50: 0.0902 - recall_at_100: 0.2217 - mrr_at_100: 0.0671 - ndcg_at_100: 0.0967 - regularization_loss: 0.3368 - _timestamp: 1657578252.7632 - _runtime: 18.7632��
373/500 [=====================>........] - ETA: 5s - loss: 7.9471 - recall_at_10: 0.1132 - mrr_at_10: 0.0636 - ndcg_at_10: 0.0754 - recall_at_50: 0.1821 - mrr_at_50: 0.0668 - ndcg_at_50: 0.0905 - recall_at_100: 0.2225 - mrr_at_100: 0.0674 - ndcg_at_100: 0.0970 - regularization_loss: 0.3376 - _timestamp: 1657578252.7632 - _runtime: 18.7632��
375/500 [=====================>........] - ETA: 5s - loss: 7.9456 - recall_at_10: 0.1137 - mrr_at_10: 0.0639 - ndcg_at_10: 0.0757 - recall_at_50: 0.1828 - mrr_at_50: 0.0670 - ndcg_at_50: 0.0908 - recall_at_100: 0.2232 - mrr_at_100: 0.0676 - ndcg_at_100: 0.0974 - regularization_loss: 0.3385 - _timestamp: 1657578252.7632 - _runtime: 18.7632��
377/500 [=====================>........] - ETA: 5s - loss: 7.9438 - recall_at_10: 0.1141 - mrr_at_10: 0.0641 - ndcg_at_10: 0.0760 - recall_at_50: 0.1834 - mrr_at_50: 0.0673 - ndcg_at_50: 0.0912 - recall_at_100: 0.2240 - mrr_at_100: 0.0679 - ndcg_at_100: 0.0977 - regularization_loss: 0.3393 - _timestamp: 1657578252.7632 - _runtime: 18.7632��
379/500 [=====================>........] - ETA: 5s - loss: 7.9421 - recall_at_10: 0.1145 - mrr_at_10: 0.0643 - ndcg_at_10: 0.0762 - recall_at_50: 0.1841 - mrr_at_50: 0.0675 - ndcg_at_50: 0.0915 - recall_at_100: 0.2247 - mrr_at_100: 0.0681 - ndcg_at_100: 0.0980 - regularization_loss: 0.3401 - _timestamp: 1657578252.7632 - _runtime: 18.7632��
381/500 [=====================>........] - ETA: 5s - loss: 7.9405 - recall_at_10: 0.1149 - mrr_at_10: 0.0645 - ndcg_at_10: 0.0765 - recall_at_50: 0.1847 - mrr_at_50: 0.0678 - ndcg_at_50: 0.0918 - recall_at_100: 0.2254 - mrr_at_100: 0.0683 - ndcg_at_100: 0.0984 - regularization_loss: 0.3410 - _timestamp: 1657578253.0000 - _runtime: 19.0000��
383/500 [=====================>........] - ETA: 5s - loss: 7.9388 - recall_at_10: 0.1153 - mrr_at_10: 0.0648 - ndcg_at_10: 0.0768 - recall_at_50: 0.1853 - mrr_at_50: 0.0680 - ndcg_at_50: 0.0921 - recall_at_100: 0.2262 - mrr_at_100: 0.0686 - ndcg_at_100: 0.0987 - regularization_loss: 0.3418 - _timestamp: 1657578253.0000 - _runtime: 19.0000��
385/500 [======================>.......] - ETA: 5s - loss: 7.9370 - recall_at_10: 0.1157 - mrr_at_10: 0.0650 - ndcg_at_10: 0.0770 - recall_at_50: 0.1859 - mrr_at_50: 0.0682 - ndcg_at_50: 0.0924 - recall_at_100: 0.2269 - mrr_at_100: 0.0688 - ndcg_at_100: 0.0990 - regularization_loss: 0.3426 - _timestamp: 1657578253.0000 - _runtime: 19.0000��
387/500 [======================>.......] - ETA: 5s - loss: 7.9355 - recall_at_10: 0.1161 - mrr_at_10: 0.0652 - ndcg_at_10: 0.0773 - recall_at_50: 0.1866 - mrr_at_50: 0.0684 - ndcg_at_50: 0.0927 - recall_at_100: 0.2276 - mrr_at_100: 0.0690 - ndcg_at_100: 0.0994 - regularization_loss: 0.3434 - _timestamp: 1657578253.0000 - _runtime: 19.0000��
389/500 [======================>.......] - ETA: 5s - loss: 7.9337 - recall_at_10: 0.1165 - mrr_at_10: 0.0654 - ndcg_at_10: 0.0776 - recall_at_50: 0.1872 - mrr_at_50: 0.0687 - ndcg_at_50: 0.0930 - recall_at_100: 0.2283 - mrr_at_100: 0.0693 - ndcg_at_100: 0.0997 - regularization_loss: 0.3442 - _timestamp: 1657578253.0000 - _runtime: 19.0000��
391/500 [======================>.......] - ETA: 5s - loss: 7.9319 - recall_at_10: 0.1169 - mrr_at_10: 0.0656 - ndcg_at_10: 0.0778 - recall_at_50: 0.1878 - mrr_at_50: 0.0689 - ndcg_at_50: 0.0933 - recall_at_100: 0.2290 - mrr_at_100: 0.0695 - ndcg_at_100: 0.1000 - regularization_loss: 0.3450 - _timestamp: 1657578253.2250 - _runtime: 19.2250��
393/500 [======================>.......] - ETA: 4s - loss: 7.9301 - recall_at_10: 0.1173 - mrr_at_10: 0.0658 - ndcg_at_10: 0.0781 - recall_at_50: 0.1884 - mrr_at_50: 0.0691 - ndcg_at_50: 0.0936 - recall_at_100: 0.2297 - mrr_at_100: 0.0697 - ndcg_at_100: 0.1003 - regularization_loss: 0.3457 - _timestamp: 1657578253.2250 - _runtime: 19.2250��
395/500 [======================>.......] - ETA: 4s - loss: 7.9284 - recall_at_10: 0.1177 - mrr_at_10: 0.0661 - ndcg_at_10: 0.0783 - recall_at_50: 0.1889 - mrr_at_50: 0.0693 - ndcg_at_50: 0.0939 - recall_at_100: 0.2303 - mrr_at_100: 0.0699 - ndcg_at_100: 0.1006 - regularization_loss: 0.3465 - _timestamp: 1657578253.2250 - _runtime: 19.2250��
396/500 [======================>.......] - ETA: 4s - loss: 7.9276 - recall_at_10: 0.1179 - mrr_at_10: 0.0662 - ndcg_at_10: 0.0784 - recall_at_50: 0.1892 - mrr_at_50: 0.0694 - ndcg_at_50: 0.0941 - recall_at_100: 0.2307 - mrr_at_100: 0.0700 - ndcg_at_100: 0.1008 - regularization_loss: 0.3469 - _timestamp: 1657578253.2250 - _runtime: 19.2250��
398/500 [======================>.......] - ETA: 4s - loss: 7.9259 - recall_at_10: 0.1182 - mrr_at_10: 0.0664 - ndcg_at_10: 0.0787 - recall_at_50: 0.1898 - mrr_at_50: 0.0697 - ndcg_at_50: 0.0944 - recall_at_100: 0.2313 - mrr_at_100: 0.0702 - ndcg_at_100: 0.1011 - regularization_loss: 0.3476 - _timestamp: 1657578253.2250 - _runtime: 19.2250��
400/500 [=======================>......] - ETA: 4s - loss: 7.9242 - recall_at_10: 0.1186 - mrr_at_10: 0.0666 - ndcg_at_10: 0.0789 - recall_at_50: 0.1904 - mrr_at_50: 0.0699 - ndcg_at_50: 0.0946 - recall_at_100: 0.2320 - mrr_at_100: 0.0705 - ndcg_at_100: 0.1014 - regularization_loss: 0.3484 - _timestamp: 1657578253.2250 - _runtime: 19.2250��
401/500 [=======================>......] - ETA: 4s - loss: 7.9233 - recall_at_10: 0.1189 - mrr_at_10: 0.0667 - ndcg_at_10: 0.0791 - recall_at_50: 0.1908 - mrr_at_50: 0.0700 - ndcg_at_50: 0.0948 - recall_at_100: 0.2324 - mrr_at_100: 0.0706 - ndcg_at_100: 0.1016 - regularization_loss: 0.3488 - _timestamp: 1657578253.4634 - _runtime: 19.4634��
403/500 [=======================>......] - ETA: 4s - loss: 7.9217 - recall_at_10: 0.1194 - mrr_at_10: 0.0670 - ndcg_at_10: 0.0794 - recall_at_50: 0.1915 - mrr_at_50: 0.0703 - ndcg_at_50: 0.0952 - recall_at_100: 0.2333 - mrr_at_100: 0.0709 - ndcg_at_100: 0.1020 - regularization_loss: 0.3495 - _timestamp: 1657578253.4634 - _runtime: 19.4634��
405/500 [=======================>......] - ETA: 4s - loss: 7.9200 - recall_at_10: 0.1199 - mrr_at_10: 0.0672 - ndcg_at_10: 0.0797 - recall_at_50: 0.1923 - mrr_at_50: 0.0706 - ndcg_at_50: 0.0956 - recall_at_100: 0.2342 - mrr_at_100: 0.0712 - ndcg_at_100: 0.1024 - regularization_loss: 0.3502 - _timestamp: 1657578253.4634 - _runtime: 19.4634��
407/500 [=======================>......] - ETA: 4s - loss: 7.9184 - recall_at_10: 0.1204 - mrr_at_10: 0.0675 - ndcg_at_10: 0.0801 - recall_at_50: 0.1930 - mrr_at_50: 0.0709 - ndcg_at_50: 0.0960 - recall_at_100: 0.2350 - mrr_at_100: 0.0714 - ndcg_at_100: 0.1028 - regularization_loss: 0.3509 - _timestamp: 1657578253.4634 - _runtime: 19.4634��
409/500 [=======================>......] - ETA: 4s - loss: 7.9167 - recall_at_10: 0.1209 - mrr_at_10: 0.0678 - ndcg_at_10: 0.0804 - recall_at_50: 0.1937 - mrr_at_50: 0.0711 - ndcg_at_50: 0.0963 - recall_at_100: 0.2359 - mrr_at_100: 0.0717 - ndcg_at_100: 0.1032 - regularization_loss: 0.3517 - _timestamp: 1657578253.4634 - _runtime: 19.4634��
411/500 [=======================>......] - ETA: 4s - loss: 7.9151 - recall_at_10: 0.1213 - mrr_at_10: 0.0680 - ndcg_at_10: 0.0807 - recall_at_50: 0.1945 - mrr_at_50: 0.0714 - ndcg_at_50: 0.0967 - recall_at_100: 0.2367 - mrr_at_100: 0.0720 - ndcg_at_100: 0.1035 - regularization_loss: 0.3524 - _timestamp: 1657578253.6905 - _runtime: 19.6905��
413/500 [=======================>......] - ETA: 3s - loss: 7.9135 - recall_at_10: 0.1218 - mrr_at_10: 0.0683 - ndcg_at_10: 0.0810 - recall_at_50: 0.1952 - mrr_at_50: 0.0717 - ndcg_at_50: 0.0971 - recall_at_100: 0.2375 - mrr_at_100: 0.0723 - ndcg_at_100: 0.1039 - regularization_loss: 0.3531 - _timestamp: 1657578253.6905 - _runtime: 19.6905��
415/500 [=======================>......] - ETA: 3s - loss: 7.9118 - recall_at_10: 0.1223 - mrr_at_10: 0.0685 - ndcg_at_10: 0.0813 - recall_at_50: 0.1959 - mrr_at_50: 0.0719 - ndcg_at_50: 0.0974 - recall_at_100: 0.2384 - mrr_at_100: 0.0725 - ndcg_at_100: 0.1043 - regularization_loss: 0.3538 - _timestamp: 1657578253.6905 - _runtime: 19.6905��
417/500 [========================>.....] - ETA: 3s - loss: 7.9101 - recall_at_10: 0.1228 - mrr_at_10: 0.0688 - ndcg_at_10: 0.0816 - recall_at_50: 0.1966 - mrr_at_50: 0.0722 - ndcg_at_50: 0.0978 - recall_at_100: 0.2392 - mrr_at_100: 0.0728 - ndcg_at_100: 0.1047 - regularization_loss: 0.3544 - _timestamp: 1657578253.6905 - _runtime: 19.6905��
419/500 [========================>.....] - ETA: 3s - loss: 7.9083 - recall_at_10: 0.1232 - mrr_at_10: 0.0690 - ndcg_at_10: 0.0819 - recall_at_50: 0.1973 - mrr_at_50: 0.0725 - ndcg_at_50: 0.0981 - recall_at_100: 0.2400 - mrr_at_100: 0.0731 - ndcg_at_100: 0.1050 - regularization_loss: 0.3551 - _timestamp: 1657578253.6905 - _runtime: 19.6905��
421/500 [========================>.....] - ETA: 3s - loss: 7.9067 - recall_at_10: 0.1237 - mrr_at_10: 0.0693 - ndcg_at_10: 0.0822 - recall_at_50: 0.1980 - mrr_at_50: 0.0727 - ndcg_at_50: 0.0985 - recall_at_100: 0.2408 - mrr_at_100: 0.0733 - ndcg_at_100: 0.1054 - regularization_loss: 0.3558 - _timestamp: 1657578253.9070 - _runtime: 19.9070��
423/500 [========================>.....] - ETA: 3s - loss: 7.9050 - recall_at_10: 0.1241 - mrr_at_10: 0.0695 - ndcg_at_10: 0.0825 - recall_at_50: 0.1987 - mrr_at_50: 0.0730 - ndcg_at_50: 0.0988 - recall_at_100: 0.2416 - mrr_at_100: 0.0736 - ndcg_at_100: 0.1058 - regularization_loss: 0.3565 - _timestamp: 1657578253.9070 - _runtime: 19.9070��
425/500 [========================>.....] - ETA: 3s - loss: 7.9033 - recall_at_10: 0.1246 - mrr_at_10: 0.0698 - ndcg_at_10: 0.0828 - recall_at_50: 0.1994 - mrr_at_50: 0.0732 - ndcg_at_50: 0.0992 - recall_at_100: 0.2424 - mrr_at_100: 0.0738 - ndcg_at_100: 0.1061 - regularization_loss: 0.3571 - _timestamp: 1657578253.9070 - _runtime: 19.9070��
427/500 [========================>.....] - ETA: 3s - loss: 7.9016 - recall_at_10: 0.1250 - mrr_at_10: 0.0700 - ndcg_at_10: 0.0831 - recall_at_50: 0.2001 - mrr_at_50: 0.0735 - ndcg_at_50: 0.0995 - recall_at_100: 0.2431 - mrr_at_100: 0.0741 - ndcg_at_100: 0.1065 - regularization_loss: 0.3578 - _timestamp: 1657578253.9070 - _runtime: 19.9070��
429/500 [========================>.....] - ETA: 3s - loss: 7.8999 - recall_at_10: 0.1255 - mrr_at_10: 0.0703 - ndcg_at_10: 0.0834 - recall_at_50: 0.2007 - mrr_at_50: 0.0737 - ndcg_at_50: 0.0999 - recall_at_100: 0.2439 - mrr_at_100: 0.0743 - ndcg_at_100: 0.1068 - regularization_loss: 0.3584 - _timestamp: 1657578253.9070 - _runtime: 19.9070��
431/500 [========================>.....] - ETA: 3s - loss: 7.8982 - recall_at_10: 0.1259 - mrr_at_10: 0.0705 - ndcg_at_10: 0.0837 - recall_at_50: 0.2014 - mrr_at_50: 0.0740 - ndcg_at_50: 0.1002 - recall_at_100: 0.2447 - mrr_at_100: 0.0746 - ndcg_at_100: 0.1072 - regularization_loss: 0.3590 - _timestamp: 1657578254.1364 - _runtime: 20.1364��
433/500 [========================>.....] - ETA: 3s - loss: 7.8964 - recall_at_10: 0.1264 - mrr_at_10: 0.0707 - ndcg_at_10: 0.0839 - recall_at_50: 0.2020 - mrr_at_50: 0.0742 - ndcg_at_50: 0.1005 - recall_at_100: 0.2454 - mrr_at_100: 0.0748 - ndcg_at_100: 0.1075 - regularization_loss: 0.3597 - _timestamp: 1657578254.1364 - _runtime: 20.1364��
435/500 [=========================>....] - ETA: 2s - loss: 7.8948 - recall_at_10: 0.1268 - mrr_at_10: 0.0710 - ndcg_at_10: 0.0842 - recall_at_50: 0.2027 - mrr_at_50: 0.0745 - ndcg_at_50: 0.1009 - recall_at_100: 0.2462 - mrr_at_100: 0.0751 - ndcg_at_100: 0.1079 - regularization_loss: 0.3603 - _timestamp: 1657578254.1364 - _runtime: 20.1364��
437/500 [=========================>....] - ETA: 2s - loss: 7.8930 - recall_at_10: 0.1272 - mrr_at_10: 0.0712 - ndcg_at_10: 0.0845 - recall_at_50: 0.2033 - mrr_at_50: 0.0747 - ndcg_at_50: 0.1012 - recall_at_100: 0.2469 - mrr_at_100: 0.0753 - ndcg_at_100: 0.1082 - regularization_loss: 0.3609 - _timestamp: 1657578254.1364 - _runtime: 20.1364��
439/500 [=========================>....] - ETA: 2s - loss: 7.8913 - recall_at_10: 0.1276 - mrr_at_10: 0.0714 - ndcg_at_10: 0.0848 - recall_at_50: 0.2040 - mrr_at_50: 0.0750 - ndcg_at_50: 0.1015 - recall_at_100: 0.2476 - mrr_at_100: 0.0756 - ndcg_at_100: 0.1086 - regularization_loss: 0.3615 - _timestamp: 1657578254.1364 - _runtime: 20.1364��
441/500 [=========================>....] - ETA: 2s - loss: 7.8897 - recall_at_10: 0.1281 - mrr_at_10: 0.0717 - ndcg_at_10: 0.0850 - recall_at_50: 0.2046 - mrr_at_50: 0.0752 - ndcg_at_50: 0.1018 - recall_at_100: 0.2484 - mrr_at_100: 0.0758 - ndcg_at_100: 0.1089 - regularization_loss: 0.3621 - _timestamp: 1657578254.3556 - _runtime: 20.3556��
442/500 [=========================>....] - ETA: 2s - loss: 7.8889 - recall_at_10: 0.1283 - mrr_at_10: 0.0718 - ndcg_at_10: 0.0852 - recall_at_50: 0.2049 - mrr_at_50: 0.0753 - ndcg_at_50: 0.1020 - recall_at_100: 0.2487 - mrr_at_100: 0.0759 - ndcg_at_100: 0.1091 - regularization_loss: 0.3624 - _timestamp: 1657578254.3556 - _runtime: 20.3556��
444/500 [=========================>....] - ETA: 2s - loss: 7.8872 - recall_at_10: 0.1287 - mrr_at_10: 0.0720 - ndcg_at_10: 0.0855 - recall_at_50: 0.2055 - mrr_at_50: 0.0755 - ndcg_at_50: 0.1023 - recall_at_100: 0.2494 - mrr_at_100: 0.0762 - ndcg_at_100: 0.1094 - regularization_loss: 0.3630 - _timestamp: 1657578254.3556 - _runtime: 20.3556��
446/500 [=========================>....] - ETA: 2s - loss: 7.8856 - recall_at_10: 0.1291 - mrr_at_10: 0.0722 - ndcg_at_10: 0.0857 - recall_at_50: 0.2062 - mrr_at_50: 0.0758 - ndcg_at_50: 0.1026 - recall_at_100: 0.2501 - mrr_at_100: 0.0764 - ndcg_at_100: 0.1097 - regularization_loss: 0.3636 - _timestamp: 1657578254.3556 - _runtime: 20.3556��
447/500 [=========================>....] - ETA: 2s - loss: 7.8848 - recall_at_10: 0.1293 - mrr_at_10: 0.0723 - ndcg_at_10: 0.0859 - recall_at_50: 0.2065 - mrr_at_50: 0.0759 - ndcg_at_50: 0.1028 - recall_at_100: 0.2505 - mrr_at_100: 0.0765 - ndcg_at_100: 0.1099 - regularization_loss: 0.3639 - _timestamp: 1657578254.3556 - _runtime: 20.3556��
449/500 [=========================>....] - ETA: 2s - loss: 7.8833 - recall_at_10: 0.1297 - mrr_at_10: 0.0725 - ndcg_at_10: 0.0861 - recall_at_50: 0.2071 - mrr_at_50: 0.0761 - ndcg_at_50: 0.1031 - recall_at_100: 0.2512 - mrr_at_100: 0.0767 - ndcg_at_100: 0.1102 - regularization_loss: 0.3645 - _timestamp: 1657578254.3556 - _runtime: 20.3556��
451/500 [==========================>...] - ETA: 2s - loss: 7.8816 - recall_at_10: 0.1301 - mrr_at_10: 0.0728 - ndcg_at_10: 0.0864 - recall_at_50: 0.2077 - mrr_at_50: 0.0763 - ndcg_at_50: 0.1034 - recall_at_100: 0.2519 - mrr_at_100: 0.0770 - ndcg_at_100: 0.1105 - regularization_loss: 0.3651 - _timestamp: 1657578254.5870 - _runtime: 20.5870��
453/500 [==========================>...] - ETA: 2s - loss: 7.8800 - recall_at_10: 0.1305 - mrr_at_10: 0.0730 - ndcg_at_10: 0.0866 - recall_at_50: 0.2083 - mrr_at_50: 0.0766 - ndcg_at_50: 0.1037 - recall_at_100: 0.2526 - mrr_at_100: 0.0772 - ndcg_at_100: 0.1108 - regularization_loss: 0.3656 - _timestamp: 1657578254.5870 - _runtime: 20.5870��
455/500 [==========================>...] - ETA: 2s - loss: 7.8785 - recall_at_10: 0.1309 - mrr_at_10: 0.0732 - ndcg_at_10: 0.0869 - recall_at_50: 0.2089 - mrr_at_50: 0.0768 - ndcg_at_50: 0.1040 - recall_at_100: 0.2533 - mrr_at_100: 0.0774 - ndcg_at_100: 0.1112 - regularization_loss: 0.3662 - _timestamp: 1657578254.5870 - _runtime: 20.5870��
457/500 [==========================>...] - ETA: 1s - loss: 7.8768 - recall_at_10: 0.1313 - mrr_at_10: 0.0734 - ndcg_at_10: 0.0871 - recall_at_50: 0.2095 - mrr_at_50: 0.0770 - ndcg_at_50: 0.1043 - recall_at_100: 0.2539 - mrr_at_100: 0.0776 - ndcg_at_100: 0.1115 - regularization_loss: 0.3668 - _timestamp: 1657578254.5870 - _runtime: 20.5870��
459/500 [==========================>...] - ETA: 1s - loss: 7.8750 - recall_at_10: 0.1317 - mrr_at_10: 0.0736 - ndcg_at_10: 0.0874 - recall_at_50: 0.2100 - mrr_at_50: 0.0772 - ndcg_at_50: 0.1046 - recall_at_100: 0.2546 - mrr_at_100: 0.0779 - ndcg_at_100: 0.1118 - regularization_loss: 0.3673 - _timestamp: 1657578254.5870 - _runtime: 20.5870��
461/500 [==========================>...] - ETA: 1s - loss: 7.8732 - recall_at_10: 0.1321 - mrr_at_10: 0.0738 - ndcg_at_10: 0.0876 - recall_at_50: 0.2106 - mrr_at_50: 0.0774 - ndcg_at_50: 0.1049 - recall_at_100: 0.2553 - mrr_at_100: 0.0781 - ndcg_at_100: 0.1121 - regularization_loss: 0.3679 - _timestamp: 1657578254.8085 - _runtime: 20.8085��
463/500 [==========================>...] - ETA: 1s - loss: 7.8716 - recall_at_10: 0.1324 - mrr_at_10: 0.0740 - ndcg_at_10: 0.0879 - recall_at_50: 0.2112 - mrr_at_50: 0.0777 - ndcg_at_50: 0.1052 - recall_at_100: 0.2559 - mrr_at_100: 0.0783 - ndcg_at_100: 0.1124 - regularization_loss: 0.3684 - _timestamp: 1657578254.8085 - _runtime: 20.8085��
465/500 [==========================>...] - ETA: 1s - loss: 7.8699 - recall_at_10: 0.1328 - mrr_at_10: 0.0742 - ndcg_at_10: 0.0881 - recall_at_50: 0.2118 - mrr_at_50: 0.0779 - ndcg_at_50: 0.1054 - recall_at_100: 0.2566 - mrr_at_100: 0.0785 - ndcg_at_100: 0.1127 - regularization_loss: 0.3690 - _timestamp: 1657578254.8085 - _runtime: 20.8085��
467/500 [===========================>..] - ETA: 1s - loss: 7.8683 - recall_at_10: 0.1332 - mrr_at_10: 0.0744 - ndcg_at_10: 0.0884 - recall_at_50: 0.2123 - mrr_at_50: 0.0781 - ndcg_at_50: 0.1057 - recall_at_100: 0.2572 - mrr_at_100: 0.0787 - ndcg_at_100: 0.1130 - regularization_loss: 0.3695 - _timestamp: 1657578254.8085 - _runtime: 20.8085��
469/500 [===========================>..] - ETA: 1s - loss: 7.8668 - recall_at_10: 0.1336 - mrr_at_10: 0.0746 - ndcg_at_10: 0.0886 - recall_at_50: 0.2129 - mrr_at_50: 0.0783 - ndcg_at_50: 0.1060 - recall_at_100: 0.2579 - mrr_at_100: 0.0789 - ndcg_at_100: 0.1133 - regularization_loss: 0.3700 - _timestamp: 1657578254.8085 - _runtime: 20.8085��
471/500 [===========================>..] - ETA: 1s - loss: 7.8652 - recall_at_10: 0.1339 - mrr_at_10: 0.0748 - ndcg_at_10: 0.0889 - recall_at_50: 0.2134 - mrr_at_50: 0.0785 - ndcg_at_50: 0.1063 - recall_at_100: 0.2585 - mrr_at_100: 0.0791 - ndcg_at_100: 0.1136 - regularization_loss: 0.3706 - _timestamp: 1657578255.0417 - _runtime: 21.0417��
472/500 [===========================>..] - ETA: 1s - loss: 7.8644 - recall_at_10: 0.1341 - mrr_at_10: 0.0749 - ndcg_at_10: 0.0890 - recall_at_50: 0.2137 - mrr_at_50: 0.0786 - ndcg_at_50: 0.1064 - recall_at_100: 0.2588 - mrr_at_100: 0.0792 - ndcg_at_100: 0.1137 - regularization_loss: 0.3708 - _timestamp: 1657578255.0417 - _runtime: 21.0417��
474/500 [===========================>..] - ETA: 1s - loss: 7.8628 - recall_at_10: 0.1345 - mrr_at_10: 0.0751 - ndcg_at_10: 0.0892 - recall_at_50: 0.2143 - mrr_at_50: 0.0788 - ndcg_at_50: 0.1067 - recall_at_100: 0.2595 - mrr_at_100: 0.0794 - ndcg_at_100: 0.1140 - regularization_loss: 0.3713 - _timestamp: 1657578255.0417 - _runtime: 21.0417��
476/500 [===========================>..] - ETA: 1s - loss: 7.8613 - recall_at_10: 0.1348 - mrr_at_10: 0.0753 - ndcg_at_10: 0.0894 - recall_at_50: 0.2148 - mrr_at_50: 0.0790 - ndcg_at_50: 0.1070 - recall_at_100: 0.2601 - mrr_at_100: 0.0796 - ndcg_at_100: 0.1143 - regularization_loss: 0.3718 - _timestamp: 1657578255.0417 - _runtime: 21.0417��
477/500 [===========================>..] - ETA: 1s - loss: 7.8631 - recall_at_10: 0.1350 - mrr_at_10: 0.0754 - ndcg_at_10: 0.0896 - recall_at_50: 0.2151 - mrr_at_50: 0.0791 - ndcg_at_50: 0.1071 - recall_at_100: 0.2604 - mrr_at_100: 0.0797 - ndcg_at_100: 0.1144 - regularization_loss: 0.3717 - _timestamp: 1657578255.0417 - _runtime: 21.0417��
478/500 [===========================>..] - ETA: 1s - loss: 7.8649 - recall_at_10: 0.1352 - mrr_at_10: 0.0755 - ndcg_at_10: 0.0897 - recall_at_50: 0.2154 - mrr_at_50: 0.0792 - ndcg_at_50: 0.1072 - recall_at_100: 0.2607 - mrr_at_100: 0.0798 - ndcg_at_100: 0.1146 - regularization_loss: 0.3716 - _timestamp: 1657578255.0417 - _runtime: 21.0417��
479/500 [===========================>..] - ETA: 0s - loss: 7.8664 - recall_at_10: 0.1354 - mrr_at_10: 0.0756 - ndcg_at_10: 0.0898 - recall_at_50: 0.2156 - mrr_at_50: 0.0793 - ndcg_at_50: 0.1074 - recall_at_100: 0.2610 - mrr_at_100: 0.0799 - ndcg_at_100: 0.1147 - regularization_loss: 0.3715 - _timestamp: 1657578255.0417 - _runtime: 21.0417��
480/500 [===========================>..] - ETA: 0s - loss: 7.8679 - recall_at_10: 0.1355 - mrr_at_10: 0.0757 - ndcg_at_10: 0.0899 - recall_at_50: 0.2159 - mrr_at_50: 0.0794 - ndcg_at_50: 0.1075 - recall_at_100: 0.2613 - mrr_at_100: 0.0800 - ndcg_at_100: 0.1149 - regularization_loss: 0.3714 - _timestamp: 1657578255.0417 - _runtime: 21.0417��
481/500 [===========================>..] - ETA: 0s - loss: 7.8692 - recall_at_10: 0.1357 - mrr_at_10: 0.0758 - ndcg_at_10: 0.0900 - recall_at_50: 0.2161 - mrr_at_50: 0.0795 - ndcg_at_50: 0.1076 - recall_at_100: 0.2616 - mrr_at_100: 0.0801 - ndcg_at_100: 0.1150 - regularization_loss: 0.3714 - _timestamp: 1657578255.2653 - _runtime: 21.2653��
483/500 [===========================>..] - ETA: 0s - loss: 7.8715 - recall_at_10: 0.1361 - mrr_at_10: 0.0760 - ndcg_at_10: 0.0902 - recall_at_50: 0.2167 - mrr_at_50: 0.0797 - ndcg_at_50: 0.1079 - recall_at_100: 0.2622 - mrr_at_100: 0.0803 - ndcg_at_100: 0.1153 - regularization_loss: 0.3712 - _timestamp: 1657578255.2653 - _runtime: 21.2653��
484/500 [============================>.] - ETA: 0s - loss: 7.8725 - recall_at_10: 0.1362 - mrr_at_10: 0.0761 - ndcg_at_10: 0.0904 - recall_at_50: 0.2169 - mrr_at_50: 0.0798 - ndcg_at_50: 0.1080 - recall_at_100: 0.2625 - mrr_at_100: 0.0804 - ndcg_at_100: 0.1154 - regularization_loss: 0.3712 - _timestamp: 1657578255.2653 - _runtime: 21.2653��
485/500 [============================>.] - ETA: 0s - loss: 7.8734 - recall_at_10: 0.1364 - mrr_at_10: 0.0762 - ndcg_at_10: 0.0905 - recall_at_50: 0.2172 - mrr_at_50: 0.0799 - ndcg_at_50: 0.1082 - recall_at_100: 0.2628 - mrr_at_100: 0.0805 - ndcg_at_100: 0.1156 - regularization_loss: 0.3712 - _timestamp: 1657578255.2653 - _runtime: 21.2653��
487/500 [============================>.] - ETA: 0s - loss: 7.8749 - recall_at_10: 0.1368 - mrr_at_10: 0.0763 - ndcg_at_10: 0.0907 - recall_at_50: 0.2177 - mrr_at_50: 0.0801 - ndcg_at_50: 0.1084 - recall_at_100: 0.2634 - mrr_at_100: 0.0807 - ndcg_at_100: 0.1158 - regularization_loss: 0.3712 - _timestamp: 1657578255.2653 - _runtime: 21.2653��
489/500 [============================>.] - ETA: 0s - loss: 7.8763 - recall_at_10: 0.1371 - mrr_at_10: 0.0765 - ndcg_at_10: 0.0909 - recall_at_50: 0.2182 - mrr_at_50: 0.0803 - ndcg_at_50: 0.1087 - recall_at_100: 0.2640 - mrr_at_100: 0.0809 - ndcg_at_100: 0.1161 - regularization_loss: 0.3714 - _timestamp: 1657578255.2653 - _runtime: 21.2653��
491/500 [============================>.] - ETA: 0s - loss: 7.8774 - recall_at_10: 0.1374 - mrr_at_10: 0.0767 - ndcg_at_10: 0.0911 - recall_at_50: 0.2187 - mrr_at_50: 0.0805 - ndcg_at_50: 0.1090 - recall_at_100: 0.2646 - mrr_at_100: 0.0811 - ndcg_at_100: 0.1164 - regularization_loss: 0.3716 - _timestamp: 1657578255.5000 - _runtime: 21.5000��
493/500 [============================>.] - ETA: 0s - loss: 7.8783 - recall_at_10: 0.1378 - mrr_at_10: 0.0769 - ndcg_at_10: 0.0914 - recall_at_50: 0.2192 - mrr_at_50: 0.0807 - ndcg_at_50: 0.1092 - recall_at_100: 0.2652 - mrr_at_100: 0.0813 - ndcg_at_100: 0.1166 - regularization_loss: 0.3718 - _timestamp: 1657578255.5000 - _runtime: 21.5000��
495/500 [============================>.] - ETA: 0s - loss: 7.8792 - recall_at_10: 0.1381 - mrr_at_10: 0.0771 - ndcg_at_10: 0.0916 - recall_at_50: 0.2197 - mrr_at_50: 0.0808 - ndcg_at_50: 0.1095 - recall_at_100: 0.2658 - mrr_at_100: 0.0815 - ndcg_at_100: 0.1169 - regularization_loss: 0.3722 - _timestamp: 1657578255.5000 - _runtime: 21.5000��
497/500 [============================>.] - ETA: 0s - loss: 7.8800 - recall_at_10: 0.1384 - mrr_at_10: 0.0773 - ndcg_at_10: 0.0918 - recall_at_50: 0.2202 - mrr_at_50: 0.0810 - ndcg_at_50: 0.1097 - recall_at_100: 0.2663 - mrr_at_100: 0.0817 - ndcg_at_100: 0.1172 - regularization_loss: 0.3726 - _timestamp: 1657578255.5000 - _runtime: 21.5000��
499/500 [============================>.] - ETA: 0s - loss: 7.8806 - recall_at_10: 0.1388 - mrr_at_10: 0.0774 - ndcg_at_10: 0.0920 - recall_at_50: 0.2207 - mrr_at_50: 0.0812 - ndcg_at_50: 0.1100 - recall_at_100: 0.2669 - mrr_at_100: 0.0819 - ndcg_at_100: 0.1174 - regularization_loss: 0.3730 - _timestamp: 1657578255.5000 - _runtime: 21.5000��
500/500 [==============================] - 30s 46ms/step - loss: 7.8809 - recall_at_10: 0.1391 - mrr_at_10: 0.0776 - ndcg_at_10: 0.0922 - recall_at_50: 0.2212 - mrr_at_50: 0.0814 - ndcg_at_50: 0.1102 - recall_at_100: 0.2675 - mrr_at_100: 0.0821 - ndcg_at_100: 0.1177 - regularization_loss: 0.3736 - _timestamp: 1657578255.7255 - _runtime: 21.7255
----------------------------- Captured stderr call -----------------------------
wandb: Tracking run with wandb version 0.12.21
wandb: Run data is saved locally in /var/jenkins_home/workspace/merlin_models/models/wandb/run-20220711_222354-2dcwmxqd
wandb: Run wandb offline to turn off syncing.
wandb: Syncing run ruby-tree-44
wandb: ⭐️ View project at https://wandb.ai/merlin-research/merlin-ci
wandb: 🚀 View run at https://wandb.ai/merlin-research/merlin-ci/runs/2dcwmxqd
wandb: WARNING Unable to compute FLOPs for this model.
WARNING:tensorflow:No training configuration found in save file, so the model was not compiled. Compile it manually.
2022-07-11 22:24:41.451103: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:288] gpu_async_0 cuMemAllocAsync failed to allocate 2678366463 bytes: CUDA error: out of memory (CUDA_ERROR_OUT_OF_MEMORY)
Reported by CUDA: Free memory/Total memory: 918093824/17069309952
2022-07-11 22:24:41.451161: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:293] Stats: Limit: 1706033152
InUse: 9600641406
MaxInUse: 11587752099
NumAllocs: 3100553
MaxAllocSize: 2678366463
Reserved: 0
PeakReserved: 0
LargestFreeBlock: 0

2022-07-11 22:24:41.453711: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:56] Histogram of current allocation: (allocation_size_in_bytes, nb_allocation_of_that_sizes), ...;
2022-07-11 22:24:41.453729: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 0, 1
2022-07-11 22:24:41.453738: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 1, 15
2022-07-11 22:24:41.453749: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 4, 242
2022-07-11 22:24:41.453760: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 8, 16
2022-07-11 22:24:41.453770: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 12, 6
2022-07-11 22:24:41.453778: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 16, 1
2022-07-11 22:24:41.453785: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 20, 1
2022-07-11 22:24:41.453791: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 24, 2
2022-07-11 22:24:41.453798: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 40, 5
2022-07-11 22:24:41.453804: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 48, 1
2022-07-11 22:24:41.453810: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 128, 24
2022-07-11 22:24:41.453817: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 160, 30
2022-07-11 22:24:41.453823: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 192, 12
2022-07-11 22:24:41.453829: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 200, 4
2022-07-11 22:24:41.453836: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 256, 43
2022-07-11 22:24:41.453842: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 288, 12
2022-07-11 22:24:41.453851: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 328, 44
2022-07-11 22:24:41.453858: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 400, 10
2022-07-11 22:24:41.453865: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 440, 1
2022-07-11 22:24:41.453877: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 480, 12
2022-07-11 22:24:41.453883: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 512, 14
2022-07-11 22:24:41.453890: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 672, 12
2022-07-11 22:24:41.453896: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 832, 12
2022-07-11 22:24:41.453903: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 1024, 54
2022-07-11 22:24:41.453909: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 1028, 1
2022-07-11 22:24:41.453916: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 1280, 30
2022-07-11 22:24:41.453922: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 1472, 6
2022-07-11 22:24:41.453929: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 1536, 15
2022-07-11 22:24:41.453935: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 2048, 12
2022-07-11 22:24:41.453942: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 2304, 15
2022-07-11 22:24:41.453949: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 3264, 12
2022-07-11 22:24:41.453955: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 3840, 15
2022-07-11 22:24:41.453962: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 5376, 15
2022-07-11 22:24:41.453968: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 6464, 12
2022-07-11 22:24:41.453975: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 6656, 15
2022-07-11 22:24:41.453984: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 8000, 4
2022-07-11 22:24:41.453992: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 8192, 121307
2022-07-11 22:24:41.453999: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 10200, 1
2022-07-11 22:24:41.454006: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 13056, 21
2022-07-11 22:24:41.454012: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 16064, 24
2022-07-11 22:24:41.454019: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 16128, 1
2022-07-11 22:24:41.454025: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 19264, 12
2022-07-11 22:24:41.454032: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 25600, 2
2022-07-11 22:24:41.454038: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 25856, 15
2022-07-11 22:24:41.454044: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 32064, 24
2022-07-11 22:24:41.454051: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 32768, 62812
2022-07-11 22:24:41.454057: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 37888, 4
2022-07-11 22:24:41.454064: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 49152, 7
2022-07-11 22:24:41.454070: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 64256, 30
2022-07-11 22:24:41.454077: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 70912, 1
2022-07-11 22:24:41.454083: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 77056, 15
2022-07-11 22:24:41.454089: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 81920, 5
2022-07-11 22:24:41.454096: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 84992, 1
2022-07-11 22:24:41.454102: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 96096, 24
2022-07-11 22:24:41.454109: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 128256, 30
2022-07-11 22:24:41.454116: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 163840, 22
2022-07-11 22:24:41.454123: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 256256, 30
2022-07-11 22:24:41.454129: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 262144, 2
2022-07-11 22:24:41.454136: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 409600, 2
2022-07-11 22:24:41.454145: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 460288, 1
2022-07-11 22:24:41.454152: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 524288, 12
2022-07-11 22:24:41.454158: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 1307804, 1
2022-07-11 22:24:41.454165: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 6266176, 6
2022-07-11 22:24:41.454171: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 13311232, 1
2022-07-11 22:24:41.454178: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 20991744, 6
2022-07-11 22:24:41.454184: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 21657312, 6
2022-07-11 22:24:41.454191: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 83698944, 1
2022-07-11 22:24:41.454197: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 83699456, 6
2022-07-11 22:24:41.454204: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 1339183104, 4
2022-07-11 22:24:41.454265: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at topk_op.cc:102 : RESOURCE_EXHAUSTED: OOM when allocating tensor with shape[2678366463] and type int8 on /job:localhost/replica:0/task:0/device:GPU:0 by allocator gpu_async_0
------------------------------ Captured log call -------------------------------
WARNING merlin_models:api.py:446 The sampler InBatchSampler returned no samples for this batch.
WARNING merlin_models:api.py:446 The sampler InBatchSampler returned no samples for this batch.
WARNING absl:save.py:233 Found untraced functions such as model_context_layer_call_fn, model_context_layer_call_and_return_conditional_losses, parallel_block_layer_call_fn, parallel_block_layer_call_and_return_conditional_losses, sequential_block_3_layer_call_fn while saving (showing 5 of 32). These functions will not be directly callable after loading.
WARNING tensorflow:load.py:167 No training configuration found in save file, so the model was not compiled. Compile it manually.
WARNING merlin_models:api.py:446 The sampler InBatchSampler returned no samples for this batch.
=============================== warnings summary ===============================
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36
/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.
'nearest': pil_image.NEAREST,

../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37
/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.
'bilinear': pil_image.BILINEAR,

../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38
/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.
'bicubic': pil_image.BICUBIC,

../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39
/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.
'hamming': pil_image.HAMMING,

../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40
/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.
'box': pil_image.BOX,

../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41
/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.
'lanczos': pil_image.LANCZOS,

tests/unit/datasets/test_ecommerce.py: 1 warning
tests/unit/tf/test_dataset.py: 8 warnings
tests/unit/tf/models/test_retrieval.py: 2 warnings
/usr/local/lib/python3.8/dist-packages/cudf/core/frame.py:384: UserWarning: The deep parameter is ignored and is only included for pandas compatibility.
warnings.warn(

tests/unit/tf/blocks/core/test_index.py: 7 warnings
tests/unit/tf/models/test_retrieval.py: 317 warnings
tests/unit/tf/prediction_tasks/test_next_item.py: 23 warnings
tests/unit/tf/utils/test_batch.py: 4 warnings
/tmp/autograph_generated_file5lu6cqhq.py:8: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
ag.converted_call(ag__.ld(warnings).warn, ("The 'warn' method is deprecated, use 'warning' instead", ag__.ld(DeprecationWarning), 2), None, fscope)

tests/unit/tf/blocks/core/test_transformations.py: 4 warnings
tests/unit/tf/models/test_base.py: 6 warnings
/usr/local/lib/python3.8/dist-packages/tensorflow/python/util/dispatch.py:1082: UserWarning: tf.keras.backend.random_binomial is deprecated, and will be removed in a future version.Please use tf.keras.backend.random_bernoulli instead.
return dispatch_target(*args, **kwargs)

tests/unit/tf/blocks/retrieval/test_matrix_factorization.py::test_matrix_factorization_embedding_export
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py::test_matrix_factorization_embedding_export
tests/unit/tf/blocks/retrieval/test_two_tower.py::test_matrix_factorization_embedding_export
tests/unit/tf/blocks/retrieval/test_two_tower.py::test_matrix_factorization_embedding_export
tests/unit/tf/inputs/test_embedding.py::test_embedding_features_exporting_and_loading_pretrained_initializer
/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py:566: DeprecationWarning: This function is deprecated in favor of cupy.from_dlpack
embeddings_cupy = cupy.fromDlpack(to_dlpack(tf.convert_to_tensor(embeddings)))

tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[True]
tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[True]
tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[False]
tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[False]
/usr/local/lib/python3.8/dist-packages/numpy/core/numeric.py:2446: DeprecationWarning: elementwise comparison failed; this will raise an error in the future.
return bool(asarray(a1 == a2).all())

tests/unit/torch/block/test_mlp.py::test_mlp_block
/var/jenkins_home/workspace/merlin_models/models/tests/unit/torch/_conftest.py:151: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at ../torch/csrc/utils/tensor_new.cpp:201.)
return {key: torch.tensor(value) for key, value in data.items()}

tests/unit/xgb/test_xgboost.py::test_without_dask_client
tests/unit/xgb/test_xgboost.py::TestXGBoost::test_music_regression
tests/unit/xgb/test_xgboost.py::test_gpu_hist_dmatrix[fit_kwargs0-DaskDeviceQuantileDMatrix]
tests/unit/xgb/test_xgboost.py::test_gpu_hist_dmatrix[fit_kwargs1-DaskDMatrix]
tests/unit/xgb/test_xgboost.py::TestEvals::test_multiple
tests/unit/xgb/test_xgboost.py::TestEvals::test_default
tests/unit/xgb/test_xgboost.py::TestEvals::test_train_and_valid
tests/unit/xgb/test_xgboost.py::TestEvals::test_invalid_data
/var/jenkins_home/workspace/merlin_models/models/merlin/models/xgb/init.py:266: UserWarning: Ignoring list columns as inputs to XGBoost model: ['item_genres', 'user_genres'].
warnings.warn(f"Ignoring list columns as inputs to XGBoost model: {list_column_names}.")

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
SKIPPED [1] tests/unit/implicit/init.py:18: could not import 'implicit': No module named 'implicit'
SKIPPED [1] tests/unit/lightfm/init.py:18: could not import 'lightfm': No module named 'lightfm'
SKIPPED [1] tests/unit/datasets/test_advertising.py:20: No data-dir available, pass it through env variable $INPUT_DATA_DIR
SKIPPED [1] tests/unit/datasets/test_ecommerce.py:62: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP
SKIPPED [1] tests/unit/datasets/test_ecommerce.py:78: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP
SKIPPED [1] tests/unit/datasets/test_ecommerce.py:92: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP
SKIPPED [3] tests/unit/datasets/test_entertainment.py:44: No data-dir available, pass it through env variable $INPUT_DATA_DIR
===== 1 failed, 470 passed, 9 skipped, 396 warnings in 3496.80s (0:58:16) ======
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script : #!/bin/bash
cd /var/jenkins_home/
CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/models/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"
[merlin_models] $ /bin/bash /tmp/jenkins17787774992359273889.sh

…ntime using the P100 machine available for PR testing

…ck for runtime

…rmed

…data, which will be used only by the retrieval integration tests

nvidia-merlin-bot · 2022-07-13T18:10:07Z

Click to view CI Results

GitHub pull request #537 of commit c63333d4c5db66112d2200ab95c88847fdec576b, no merge conflicts.
Running as SYSTEM
Setting status of c63333d4c5db66112d2200ab95c88847fdec576b to PENDING with url https://10.20.13.93:8080/job/merlin_models/685/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_models
using credential nvidia-merlin-bot
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/models/ # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/models/
 > git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/models/ +refs/pull/537/*:refs/remotes/origin/pr/537/* # timeout=10
 > git rev-parse c63333d4c5db66112d2200ab95c88847fdec576b^{commit} # timeout=10
Checking out Revision c63333d4c5db66112d2200ab95c88847fdec576b (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f c63333d4c5db66112d2200ab95c88847fdec576b # timeout=10
Commit message: "Fix retrieval test"
 > git rev-list --no-walk 3a5e07026d51efdb4f23084b338f48cb0600482a # timeout=10
[merlin_models] $ /bin/bash /tmp/jenkins7549807890358762269.sh
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: testbook in /usr/local/lib/python3.8/dist-packages (0.4.2)
Requirement already satisfied: nbformat>=5.0.4 in /usr/local/lib/python3.8/dist-packages (from testbook) (5.4.0)
Requirement already satisfied: nbclient>=0.4.0 in /usr/local/lib/python3.8/dist-packages (from testbook) (0.6.5)
Requirement already satisfied: traitlets>=5.1 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (5.3.0)
Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.6.1)
Requirement already satisfied: jupyter-core in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.10.0)
Requirement already satisfied: fastjsonschema in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (2.15.3)
Requirement already satisfied: jupyter-client>=6.1.5 in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (7.3.4)
Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (1.5.5)
Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (21.4.0)
Requirement already satisfied: importlib-resources>=1.4.0; python_version < "3.9" in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (5.8.0)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (0.18.1)
Requirement already satisfied: entrypoints in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (0.4)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (2.8.2)
Requirement already satisfied: pyzmq>=23.0 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (23.2.0)
Requirement already satisfied: tornado>=6.0 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (6.1)
Requirement already satisfied: zipp>=3.1.0; python_version < "3.10" in /usr/local/lib/python3.8/dist-packages (from importlib-resources>=1.4.0; python_version < "3.9"->jsonschema>=2.6->nbformat>=5.0.4->testbook) (3.8.0)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.8.2->jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (1.15.0)
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/merlin_models/models, configfile: pyproject.toml
plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 474 items / 2 skipped
tests/unit/config/test_schema.py .... tests/unit/datasets/test_advertising.py .s tests/unit/datasets/test_ecommerce.py ..sss tests/unit/datasets/test_entertainment.py ....sss. tests/unit/datasets/test_social.py . tests/unit/datasets/test_synthetic.py ..... tests/unit/tf/test_core.py ...... tests/unit/tf/test_dataset.py ............... tests/unit/tf/test_public_api.py . tests/unit/tf/blocks/test_cross.py ........... tests/unit/tf/blocks/test_dlrm.py ........ tests/unit/tf/blocks/test_interactions.py . tests/unit/tf/blocks/test_mlp.py ............... tests/unit/tf/blocks/retrieval/test_matrix_factorization.py tests/unit/tf/blocks/retrieval/test_two_tower.py ........... tests/unit/tf/core/test_aggregation.py ......... tests/unit/tf/core/test_base.py .. tests/unit/tf/core/test_combinators.py ............... tests/unit/tf/core/test_index.py ... tests/unit/tf/core/test_prediction.py .. tests/unit/tf/core/test_tabular.py .... tests/unit/tf/core/test_transformations.py ...... tests/unit/tf/data_augmentation/test_misc.py . tests/unit/tf/data_augmentation/test_negative_sampling.py tests/unit/tf/data_augmentation/test_noise.py ..... tests/unit/tf/examples/test_01_getting_started.py . tests/unit/tf/examples/test_02_dataschema.py . tests/unit/tf/examples/test_03_exploring_different_models.py tests/unit/tf/examples/test_04_export_ranking_models.py . tests/unit/tf/examples/test_05_export_retrieval_model.py . tests/unit/tf/examples/test_06_advanced_own_architecture.py tests/unit/tf/examples/test_usecase_pretrained_embeddings.py tests/unit/tf/inputs/test_continuous.py ..... tests/unit/tf/inputs/test_embedding.py tests/unit/tf/inputs/test_tabular.py ....... tests/unit/tf/layers/test_queue.py .............. tests/unit/tf/losses/test_losses.py ....................... tests/unit/tf/metrics/test_metrics_popularity.py ..... tests/unit/tf/metrics/test_metrics_topk.py tests/unit/tf/models/test_base.py ........... tests/unit/tf/models/test_benchmark.py .. tests/unit/tf/models/test_ranking.py ................ tests/unit/tf/models/test_retrieval.py tests/unit/tf/prediction_tasks/test_classification.py .. tests/unit/tf/prediction_tasks/test_multi_task.py ....... tests/unit/tf/prediction_tasks/test_next_item.py ..... tests/unit/tf/prediction_tasks/test_regression.py .. tests/unit/tf/prediction_tasks/test_sampling.py ...... tests/unit/tf/utils/test_batch.py .... tests/unit/tf/utils/test_tf_utils.py .. tests/unit/torch/test_dataset.py ......... tests/unit/torch/test_public_api.py . tests/unit/torch/block/test_base.py .... tests/unit/torch/block/test_mlp.py . tests/unit/torch/features/test_continuous.py .. tests/unit/torch/features/test_embedding.py .............. tests/unit/torch/features/test_tabular.py .... tests/unit/torch/model/test_head.py ............ tests/unit/torch/model/test_model.py .. tests/unit/torch/tabular/test_aggregation.py ........ tests/unit/torch/tabular/test_tabular.py ... tests/unit/torch/tabular/test_transformations.py ....... tests/unit/utils/test_schema_utils.py .......... tests/unit/xgb/test_xgboost.py .................

[  0%]
 [  1%]
 [  2%]
 [  4%]
 [  4%]
 [  5%]
 [  6%]
 [  9%]
 [  9%]
 [ 12%]
 [ 13%]
 [ 14%]
 ..................       [ 21%]
 ..           [ 21%]
 [ 23%]
 [ 25%]
 [ 26%]
 [ 29%]
 [ 29%]
 [ 30%]
 [ 31%]
 [ 32%]
 [ 32%]
 .....          [ 33%]
 [ 34%]
 [ 35%]
 [ 35%]
 .           [ 35%]
 [ 35%]
 [ 35%]
 .            [ 36%]
 .           [ 36%]
 [ 37%]
 ..........................        [ 42%]
 [ 44%]
 [ 47%]
 [ 52%]
 [ 53%]
 ..................            [ 56%]
 [ 59%]
 [ 59%]
 [ 63%]
 ...............................   [ 69%]
 [ 70%]
 [ 71%]
 [ 72%]
 [ 72%]
 [ 74%]
 [ 75%]
 [ 75%]
 [ 77%]
 [ 77%]
 [ 78%]
 [ 78%]
 [ 79%]
 [ 82%]
 [ 82%]
 [ 85%]
 [ 85%]
 [ 87%]
 [ 88%]
 [ 89%]
 ......................   [ 96%]
 [100%]
=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.

'nearest': pil_image.NEAREST,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.

'bilinear': pil_image.BILINEAR,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.

'bicubic': pil_image.BICUBIC,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.

'hamming': pil_image.HAMMING,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.

'box': pil_image.BOX,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.

'lanczos': pil_image.LANCZOS,
tests/unit/datasets/test_ecommerce.py::test_synthetic_aliccp_raw_data

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-True-10]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-True-9]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-True-8]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-False-10]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-False-9]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-False-8]

tests/unit/tf/test_dataset.py::test_tf_catname_ordering

tests/unit/tf/test_dataset.py::test_tf_map

/usr/local/lib/python3.8/dist-packages/cudf/core/frame.py:384: UserWarning: The deep parameter is ignored and is only included for pandas compatibility.

warnings.warn(
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py::test_matrix_factorization_embedding_export

tests/unit/tf/blocks/retrieval/test_matrix_factorization.py::test_matrix_factorization_embedding_export

tests/unit/tf/blocks/retrieval/test_two_tower.py::test_matrix_factorization_embedding_export

tests/unit/tf/blocks/retrieval/test_two_tower.py::test_matrix_factorization_embedding_export

tests/unit/tf/inputs/test_embedding.py::test_embedding_features_exporting_and_loading_pretrained_initializer

/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py:566: DeprecationWarning: This function is deprecated in favor of cupy.from_dlpack

embeddings_cupy = cupy.fromDlpack(to_dlpack(tf.convert_to_tensor(embeddings)))
tests/unit/tf/core/test_index.py: 4 warnings

tests/unit/tf/models/test_retrieval.py: 53 warnings

tests/unit/tf/prediction_tasks/test_next_item.py: 3 warnings

tests/unit/tf/utils/test_batch.py: 2 warnings

/tmp/autograph_generated_fileinf3617r.py:8: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead

ag.converted_call(ag__.ld(warnings).warn, ("The 'warn' method is deprecated, use 'warning' instead", ag__.ld(DeprecationWarning), 2), None, fscope)
tests/unit/tf/data_augmentation/test_noise.py: 4 warnings

tests/unit/tf/models/test_base.py: 8 warnings

/usr/local/lib/python3.8/dist-packages/tensorflow/python/util/dispatch.py:1082: UserWarning: tf.keras.backend.random_binomial is deprecated, and will be removed in a future version.Please use tf.keras.backend.random_bernoulli instead.

return dispatch_target(*args, **kwargs)
tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[True]

tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[True]

tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[False]

tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[False]

/usr/local/lib/python3.8/dist-packages/numpy/core/numeric.py:2446: DeprecationWarning: elementwise comparison failed; this will raise an error in the future.

return bool(asarray(a1 == a2).all())
tests/unit/torch/block/test_mlp.py::test_mlp_block

/var/jenkins_home/workspace/merlin_models/models/tests/unit/torch/_conftest.py:151: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)

return {key: torch.tensor(value) for key, value in data.items()}
tests/unit/xgb/test_xgboost.py::test_without_dask_client

tests/unit/xgb/test_xgboost.py::TestXGBoost::test_music_regression

tests/unit/xgb/test_xgboost.py::test_gpu_hist_dmatrix[fit_kwargs0-DaskDeviceQuantileDMatrix]

tests/unit/xgb/test_xgboost.py::test_gpu_hist_dmatrix[fit_kwargs1-DaskDMatrix]

tests/unit/xgb/test_xgboost.py::TestEvals::test_multiple

tests/unit/xgb/test_xgboost.py::TestEvals::test_default

tests/unit/xgb/test_xgboost.py::TestEvals::test_train_and_valid

tests/unit/xgb/test_xgboost.py::TestEvals::test_invalid_data

/var/jenkins_home/workspace/merlin_models/models/merlin/models/xgb/init.py:266: UserWarning: Ignoring list columns as inputs to XGBoost model: ['item_genres', 'user_genres'].

warnings.warn(f"Ignoring list columns as inputs to XGBoost model: {list_column_names}.")
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

=========================== short test summary info ============================

SKIPPED [1] tests/unit/implicit/init.py:18: could not import 'implicit': No module named 'implicit'

SKIPPED [1] tests/unit/lightfm/init.py:18: could not import 'lightfm': No module named 'lightfm'

SKIPPED [1] tests/unit/datasets/test_advertising.py:20: No data-dir available, pass it through env variable $INPUT_DATA_DIR

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:62: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:78: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:92: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP

SKIPPED [3] tests/unit/datasets/test_entertainment.py:44: No data-dir available, pass it through env variable $INPUT_DATA_DIR

========== 467 passed, 9 skipped, 107 warnings in 1639.74s (0:27:19) ===========

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/models/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_models] $ /bin/bash /tmp/jenkins3415442025633188346.sh

nvidia-merlin-bot · 2022-07-13T18:38:01Z

Click to view CI Results

GitHub pull request #537 of commit be9fe221c2e5fc64ab8e087bb7814412af18daf9, no merge conflicts.
Running as SYSTEM
Setting status of be9fe221c2e5fc64ab8e087bb7814412af18daf9 to PENDING with url https://10.20.13.93:8080/job/merlin_models/686/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_models
using credential nvidia-merlin-bot
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/models/ # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/models/
 > git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/models/ +refs/pull/537/*:refs/remotes/origin/pr/537/* # timeout=10
 > git rev-parse be9fe221c2e5fc64ab8e087bb7814412af18daf9^{commit} # timeout=10
Checking out Revision be9fe221c2e5fc64ab8e087bb7814412af18daf9 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f be9fe221c2e5fc64ab8e087bb7814412af18daf9 # timeout=10
Commit message: "Fix retrieval test"
 > git rev-list --no-walk c63333d4c5db66112d2200ab95c88847fdec576b # timeout=10
[merlin_models] $ /bin/bash /tmp/jenkins15425560071084506418.sh
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: testbook in /usr/local/lib/python3.8/dist-packages (0.4.2)
Requirement already satisfied: nbformat>=5.0.4 in /usr/local/lib/python3.8/dist-packages (from testbook) (5.4.0)
Requirement already satisfied: nbclient>=0.4.0 in /usr/local/lib/python3.8/dist-packages (from testbook) (0.6.5)
Requirement already satisfied: traitlets>=5.1 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (5.3.0)
Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.6.1)
Requirement already satisfied: jupyter-core in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.10.0)
Requirement already satisfied: fastjsonschema in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (2.15.3)
Requirement already satisfied: jupyter-client>=6.1.5 in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (7.3.4)
Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (1.5.5)
Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (21.4.0)
Requirement already satisfied: importlib-resources>=1.4.0; python_version < "3.9" in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (5.8.0)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (0.18.1)
Requirement already satisfied: entrypoints in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (0.4)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (2.8.2)
Requirement already satisfied: pyzmq>=23.0 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (23.2.0)
Requirement already satisfied: tornado>=6.0 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (6.1)
Requirement already satisfied: zipp>=3.1.0; python_version < "3.10" in /usr/local/lib/python3.8/dist-packages (from importlib-resources>=1.4.0; python_version < "3.9"->jsonschema>=2.6->nbformat>=5.0.4->testbook) (3.8.0)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.8.2->jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (1.15.0)
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/merlin_models/models, configfile: pyproject.toml
plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 474 items / 2 skipped
tests/unit/config/test_schema.py ....                                    [  0%]

tests/unit/datasets/test_advertising.py .s                               [  1%]

tests/unit/datasets/test_ecommerce.py ..sss                              [  2%]

tests/unit/datasets/test_entertainment.py ....sss.                       [  4%]

tests/unit/datasets/test_social.py .                                     [  4%]

tests/unit/datasets/test_synthetic.py .....                              [  5%]

tests/unit/tf/test_core.py ......                                        [  6%]

tests/unit/tf/test_dataset.py ...............                            [  9%]

tests/unit/tf/test_public_api.py .                                       [  9%]

tests/unit/tf/blocks/test_cross.py ...........                           [ 12%]

tests/unit/tf/blocks/test_dlrm.py ........                               [ 13%]

tests/unit/tf/blocks/test_interactions.py .                              [ 14%]

tests/unit/tf/blocks/test_mlp.py .................................       [ 21%]

tests/unit/tf/blocks/retrieval/test_matrix_factorization.py ..           [ 21%]

tests/unit/tf/blocks/retrieval/test_two_tower.py ...........             [ 23%]

tests/unit/tf/core/test_aggregation.py .........                         [ 25%]

tests/unit/tf/core/test_base.py ..                                       [ 26%]

tests/unit/tf/core/test_combinators.py ...............                   [ 29%]

tests/unit/tf/core/test_index.py ...                                     [ 29%]

tests/unit/tf/core/test_prediction.py ..                                 [ 30%]

tests/unit/tf/core/test_tabular.py ....                                  [ 31%]

tests/unit/tf/core/test_transformations.py ......                        [ 32%]

tests/unit/tf/data_augmentation/test_misc.py .                           [ 32%]

tests/unit/tf/data_augmentation/test_negative_sampling.py .....          [ 33%]

tests/unit/tf/data_augmentation/test_noise.py .....                      [ 34%]

tests/unit/tf/examples/test_01_getting_started.py .                      [ 35%]

tests/unit/tf/examples/test_02_dataschema.py .                           [ 35%]

tests/unit/tf/examples/test_03_exploring_different_models.py .           [ 35%]

tests/unit/tf/examples/test_04_export_ranking_models.py .                [ 35%]

tests/unit/tf/examples/test_05_export_retrieval_model.py .               [ 35%]

tests/unit/tf/examples/test_06_advanced_own_architecture.py .            [ 36%]

tests/unit/tf/examples/test_usecase_pretrained_embeddings.py .           [ 36%]

tests/unit/tf/inputs/test_continuous.py .....                            [ 37%]

tests/unit/tf/inputs/test_embedding.py ..........................        [ 42%]

tests/unit/tf/inputs/test_tabular.py .......                             [ 44%]

tests/unit/tf/layers/test_queue.py ..............                        [ 47%]

tests/unit/tf/losses/test_losses.py .......................              [ 52%]

tests/unit/tf/metrics/test_metrics_popularity.py .....                   [ 53%]

tests/unit/tf/metrics/test_metrics_topk.py ..................            [ 56%]

tests/unit/tf/models/test_base.py ...........                            [ 59%]

tests/unit/tf/models/test_benchmark.py ..                                [ 59%]

tests/unit/tf/models/test_ranking.py ................                    [ 63%]

tests/unit/tf/models/test_retrieval.py ...............................   [ 69%]

tests/unit/tf/prediction_tasks/test_classification.py ..                 [ 70%]

tests/unit/tf/prediction_tasks/test_multi_task.py .......                [ 71%]

tests/unit/tf/prediction_tasks/test_next_item.py .....                   [ 72%]

tests/unit/tf/prediction_tasks/test_regression.py ..                     [ 72%]

tests/unit/tf/prediction_tasks/test_sampling.py ......                   [ 74%]

tests/unit/tf/utils/test_batch.py ....                                   [ 75%]

tests/unit/tf/utils/test_tf_utils.py ..                                  [ 75%]

tests/unit/torch/test_dataset.py .........                               [ 77%]

tests/unit/torch/test_public_api.py .                                    [ 77%]

tests/unit/torch/block/test_base.py ....                                 [ 78%]

tests/unit/torch/block/test_mlp.py .                                     [ 78%]

tests/unit/torch/features/test_continuous.py ..                          [ 79%]

tests/unit/torch/features/test_embedding.py ..............               [ 82%]

tests/unit/torch/features/test_tabular.py ....                           [ 82%]

tests/unit/torch/model/test_head.py ............                         [ 85%]

tests/unit/torch/model/test_model.py ..                                  [ 85%]

tests/unit/torch/tabular/test_aggregation.py ........                    [ 87%]

tests/unit/torch/tabular/test_tabular.py ...                             [ 88%]

tests/unit/torch/tabular/test_transformations.py .......                 [ 89%]

tests/unit/utils/test_schema_utils.py ................................   [ 96%]

tests/unit/xgb/test_xgboost.py .................                         [100%]
=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.

'nearest': pil_image.NEAREST,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.

'bilinear': pil_image.BILINEAR,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.

'bicubic': pil_image.BICUBIC,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.

'hamming': pil_image.HAMMING,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.

'box': pil_image.BOX,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.

'lanczos': pil_image.LANCZOS,
tests/unit/datasets/test_ecommerce.py::test_synthetic_aliccp_raw_data

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-True-10]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-True-9]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-True-8]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-False-10]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-False-9]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-False-8]

tests/unit/tf/test_dataset.py::test_tf_catname_ordering

tests/unit/tf/test_dataset.py::test_tf_map

/usr/local/lib/python3.8/dist-packages/cudf/core/frame.py:384: UserWarning: The deep parameter is ignored and is only included for pandas compatibility.

warnings.warn(
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py::test_matrix_factorization_embedding_export

tests/unit/tf/blocks/retrieval/test_matrix_factorization.py::test_matrix_factorization_embedding_export

tests/unit/tf/blocks/retrieval/test_two_tower.py::test_matrix_factorization_embedding_export

tests/unit/tf/blocks/retrieval/test_two_tower.py::test_matrix_factorization_embedding_export

tests/unit/tf/inputs/test_embedding.py::test_embedding_features_exporting_and_loading_pretrained_initializer

/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py:566: DeprecationWarning: This function is deprecated in favor of cupy.from_dlpack

embeddings_cupy = cupy.fromDlpack(to_dlpack(tf.convert_to_tensor(embeddings)))
tests/unit/tf/core/test_index.py: 4 warnings

tests/unit/tf/models/test_retrieval.py: 53 warnings

tests/unit/tf/prediction_tasks/test_next_item.py: 3 warnings

tests/unit/tf/utils/test_batch.py: 2 warnings

/tmp/autograph_generated_filej386dzi2.py:8: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead

ag.converted_call(ag__.ld(warnings).warn, ("The 'warn' method is deprecated, use 'warning' instead", ag__.ld(DeprecationWarning), 2), None, fscope)
tests/unit/tf/data_augmentation/test_noise.py: 4 warnings

tests/unit/tf/models/test_base.py: 8 warnings

/usr/local/lib/python3.8/dist-packages/tensorflow/python/util/dispatch.py:1082: UserWarning: tf.keras.backend.random_binomial is deprecated, and will be removed in a future version.Please use tf.keras.backend.random_bernoulli instead.

return dispatch_target(*args, **kwargs)
tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[True]

tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[True]

tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[False]

tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[False]

/usr/local/lib/python3.8/dist-packages/numpy/core/numeric.py:2446: DeprecationWarning: elementwise comparison failed; this will raise an error in the future.

return bool(asarray(a1 == a2).all())
tests/unit/torch/block/test_mlp.py::test_mlp_block

/var/jenkins_home/workspace/merlin_models/models/tests/unit/torch/_conftest.py:151: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)

return {key: torch.tensor(value) for key, value in data.items()}
tests/unit/xgb/test_xgboost.py::test_without_dask_client

tests/unit/xgb/test_xgboost.py::TestXGBoost::test_music_regression

tests/unit/xgb/test_xgboost.py::test_gpu_hist_dmatrix[fit_kwargs0-DaskDeviceQuantileDMatrix]

tests/unit/xgb/test_xgboost.py::test_gpu_hist_dmatrix[fit_kwargs1-DaskDMatrix]

tests/unit/xgb/test_xgboost.py::TestEvals::test_multiple

tests/unit/xgb/test_xgboost.py::TestEvals::test_default

tests/unit/xgb/test_xgboost.py::TestEvals::test_train_and_valid

tests/unit/xgb/test_xgboost.py::TestEvals::test_invalid_data

/var/jenkins_home/workspace/merlin_models/models/merlin/models/xgb/init.py:266: UserWarning: Ignoring list columns as inputs to XGBoost model: ['item_genres', 'user_genres'].

warnings.warn(f"Ignoring list columns as inputs to XGBoost model: {list_column_names}.")
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

=========================== short test summary info ============================

SKIPPED [1] tests/unit/implicit/init.py:18: could not import 'implicit': No module named 'implicit'

SKIPPED [1] tests/unit/lightfm/init.py:18: could not import 'lightfm': No module named 'lightfm'

SKIPPED [1] tests/unit/datasets/test_advertising.py:20: No data-dir available, pass it through env variable $INPUT_DATA_DIR

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:62: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:78: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:92: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP

SKIPPED [3] tests/unit/datasets/test_entertainment.py:44: No data-dir available, pass it through env variable $INPUT_DATA_DIR

========== 467 passed, 9 skipped, 107 warnings in 1656.32s (0:27:36) ===========

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/models/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_models] $ /bin/bash /tmp/jenkins2985472034136254228.sh

nvidia-merlin-bot · 2022-07-14T15:08:20Z

Click to view CI Results

GitHub pull request #537 of commit 6678dbb6070a69004988f05ab0c8078880e951ca, no merge conflicts.
Running as SYSTEM
Setting status of 6678dbb6070a69004988f05ab0c8078880e951ca to PENDING with url https://10.20.13.93:8080/job/merlin_models/691/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_models
using credential nvidia-merlin-bot
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/models/ # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/models/
 > git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/models/ +refs/pull/537/*:refs/remotes/origin/pr/537/* # timeout=10
 > git rev-parse 6678dbb6070a69004988f05ab0c8078880e951ca^{commit} # timeout=10
Checking out Revision 6678dbb6070a69004988f05ab0c8078880e951ca (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 6678dbb6070a69004988f05ab0c8078880e951ca # timeout=10
Commit message: "Merge branch 'main' into retrieval_integration_tests"
 > git rev-list --no-walk 0d1684ede81be0e4a5e5f24fce00768041f9889f # timeout=10
[merlin_models] $ /bin/bash /tmp/jenkins10807057964207328951.sh
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: testbook in /usr/local/lib/python3.8/dist-packages (0.4.2)
Requirement already satisfied: nbformat>=5.0.4 in /usr/local/lib/python3.8/dist-packages (from testbook) (5.4.0)
Requirement already satisfied: nbclient>=0.4.0 in /usr/local/lib/python3.8/dist-packages (from testbook) (0.6.5)
Requirement already satisfied: traitlets>=5.1 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (5.3.0)
Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.6.1)
Requirement already satisfied: jupyter-core in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.10.0)
Requirement already satisfied: fastjsonschema in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (2.15.3)
Requirement already satisfied: jupyter-client>=6.1.5 in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (7.3.4)
Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (1.5.5)
Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (21.4.0)
Requirement already satisfied: importlib-resources>=1.4.0; python_version < "3.9" in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (5.8.0)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (0.18.1)
Requirement already satisfied: entrypoints in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (0.4)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (2.8.2)
Requirement already satisfied: pyzmq>=23.0 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (23.2.0)
Requirement already satisfied: tornado>=6.0 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (6.1)
Requirement already satisfied: zipp>=3.1.0; python_version < "3.10" in /usr/local/lib/python3.8/dist-packages (from importlib-resources>=1.4.0; python_version < "3.9"->jsonschema>=2.6->nbformat>=5.0.4->testbook) (3.8.0)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.8.2->jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (1.15.0)
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/merlin_models/models, configfile: pyproject.toml
plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 475 items / 2 skipped
tests/unit/config/test_schema.py ....                                    [  0%]

tests/unit/datasets/test_advertising.py .s                               [  1%]

tests/unit/datasets/test_ecommerce.py ..sss                              [  2%]

tests/unit/datasets/test_entertainment.py ....sss.                       [  4%]

tests/unit/datasets/test_social.py .                                     [  4%]

tests/unit/datasets/test_synthetic.py .....                              [  5%]

tests/unit/tf/test_core.py ......                                        [  6%]

tests/unit/tf/test_dataset.py ...............                            [  9%]

tests/unit/tf/test_public_api.py .                                       [  9%]

tests/unit/tf/blocks/test_cross.py ...........                           [ 12%]

tests/unit/tf/blocks/test_dlrm.py ........                               [ 13%]

tests/unit/tf/blocks/test_interactions.py .                              [ 14%]

tests/unit/tf/blocks/test_mlp.py .................................       [ 21%]

tests/unit/tf/blocks/retrieval/test_matrix_factorization.py ..           [ 21%]

tests/unit/tf/blocks/retrieval/test_two_tower.py ...........             [ 23%]

tests/unit/tf/core/test_aggregation.py .........                         [ 25%]

tests/unit/tf/core/test_base.py ..                                       [ 26%]

tests/unit/tf/core/test_combinators.py ...............                   [ 29%]

tests/unit/tf/core/test_index.py ...                                     [ 29%]

tests/unit/tf/core/test_prediction.py ..                                 [ 30%]

tests/unit/tf/core/test_tabular.py ....                                  [ 31%]

tests/unit/tf/core/test_transformations.py ......                        [ 32%]

tests/unit/tf/data_augmentation/test_misc.py .                           [ 32%]

tests/unit/tf/data_augmentation/test_negative_sampling.py ......         [ 33%]

tests/unit/tf/data_augmentation/test_noise.py .....                      [ 34%]

tests/unit/tf/examples/test_01_getting_started.py .                      [ 35%]

tests/unit/tf/examples/test_02_dataschema.py .                           [ 35%]

tests/unit/tf/examples/test_03_exploring_different_models.py .           [ 35%]

tests/unit/tf/examples/test_04_export_ranking_models.py .                [ 35%]

tests/unit/tf/examples/test_05_export_retrieval_model.py .               [ 36%]

tests/unit/tf/examples/test_06_advanced_own_architecture.py .            [ 36%]

tests/unit/tf/examples/test_usecase_pretrained_embeddings.py .           [ 36%]

tests/unit/tf/inputs/test_continuous.py .....                            [ 37%]

tests/unit/tf/inputs/test_embedding.py ..........................        [ 42%]

tests/unit/tf/inputs/test_tabular.py .......                             [ 44%]

tests/unit/tf/layers/test_queue.py ..............                        [ 47%]

tests/unit/tf/losses/test_losses.py .......................              [ 52%]

tests/unit/tf/metrics/test_metrics_popularity.py .....                   [ 53%]

tests/unit/tf/metrics/test_metrics_topk.py ..................            [ 57%]

tests/unit/tf/models/test_base.py ...........                            [ 59%]

tests/unit/tf/models/test_benchmark.py ..                                [ 59%]

tests/unit/tf/models/test_ranking.py ................                    [ 63%]

tests/unit/tf/models/test_retrieval.py ...............................   [ 69%]

tests/unit/tf/prediction_tasks/test_classification.py ..                 [ 70%]

tests/unit/tf/prediction_tasks/test_multi_task.py .......                [ 71%]

tests/unit/tf/prediction_tasks/test_next_item.py .....                   [ 72%]

tests/unit/tf/prediction_tasks/test_regression.py ..                     [ 73%]

tests/unit/tf/prediction_tasks/test_sampling.py ......                   [ 74%]

tests/unit/tf/utils/test_batch.py ....                                   [ 75%]

tests/unit/tf/utils/test_tf_utils.py ..                                  [ 75%]

tests/unit/torch/test_dataset.py .........                               [ 77%]

tests/unit/torch/test_public_api.py .                                    [ 77%]

tests/unit/torch/block/test_base.py ....                                 [ 78%]

tests/unit/torch/block/test_mlp.py .                                     [ 78%]

tests/unit/torch/features/test_continuous.py ..                          [ 79%]

tests/unit/torch/features/test_embedding.py ..............               [ 82%]

tests/unit/torch/features/test_tabular.py ....                           [ 82%]

tests/unit/torch/model/test_head.py ............                         [ 85%]

tests/unit/torch/model/test_model.py ..                                  [ 85%]

tests/unit/torch/tabular/test_aggregation.py ........                    [ 87%]

tests/unit/torch/tabular/test_tabular.py ...                             [ 88%]

tests/unit/torch/tabular/test_transformations.py .......                 [ 89%]

tests/unit/utils/test_schema_utils.py ................................   [ 96%]

tests/unit/xgb/test_xgboost.py .................                         [100%]
=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.

'nearest': pil_image.NEAREST,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.

'bilinear': pil_image.BILINEAR,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.

'bicubic': pil_image.BICUBIC,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.

'hamming': pil_image.HAMMING,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.

'box': pil_image.BOX,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.

'lanczos': pil_image.LANCZOS,
tests/unit/datasets/test_ecommerce.py::test_synthetic_aliccp_raw_data

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-True-10]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-True-9]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-True-8]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-False-10]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-False-9]

tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-False-8]

tests/unit/tf/test_dataset.py::test_tf_catname_ordering

tests/unit/tf/test_dataset.py::test_tf_map

/usr/local/lib/python3.8/dist-packages/cudf/core/frame.py:384: UserWarning: The deep parameter is ignored and is only included for pandas compatibility.

warnings.warn(
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py::test_matrix_factorization_embedding_export

tests/unit/tf/blocks/retrieval/test_matrix_factorization.py::test_matrix_factorization_embedding_export

tests/unit/tf/blocks/retrieval/test_two_tower.py::test_matrix_factorization_embedding_export

tests/unit/tf/blocks/retrieval/test_two_tower.py::test_matrix_factorization_embedding_export

tests/unit/tf/inputs/test_embedding.py::test_embedding_features_exporting_and_loading_pretrained_initializer

/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py:566: DeprecationWarning: This function is deprecated in favor of cupy.from_dlpack

embeddings_cupy = cupy.fromDlpack(to_dlpack(tf.convert_to_tensor(embeddings)))
tests/unit/tf/core/test_index.py: 4 warnings

tests/unit/tf/models/test_retrieval.py: 53 warnings

tests/unit/tf/prediction_tasks/test_next_item.py: 3 warnings

tests/unit/tf/utils/test_batch.py: 2 warnings

/tmp/autograph_generated_filezhhk4_6.py:8: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead

ag_.converted_call(ag__.ld(warnings).warn, ("The 'warn' method is deprecated, use 'warning' instead", ag__.ld(DeprecationWarning), 2), None, fscope)
tests/unit/tf/data_augmentation/test_noise.py: 4 warnings

tests/unit/tf/models/test_base.py: 8 warnings

/usr/local/lib/python3.8/dist-packages/tensorflow/python/util/dispatch.py:1082: UserWarning: tf.keras.backend.random_binomial is deprecated, and will be removed in a future version.Please use tf.keras.backend.random_bernoulli instead.

return dispatch_target(*args, **kwargs)
tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[True]

tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[True]

tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[False]

tests/unit/tf/models/test_ranking.py::test_dlrm_model_multi_task[False]

/usr/local/lib/python3.8/dist-packages/numpy/core/numeric.py:2446: DeprecationWarning: elementwise comparison failed; this will raise an error in the future.

return bool(asarray(a1 == a2).all())
tests/unit/torch/block/test_mlp.py::test_mlp_block

/var/jenkins_home/workspace/merlin_models/models/tests/unit/torch/_conftest.py:151: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)

return {key: torch.tensor(value) for key, value in data.items()}
tests/unit/xgb/test_xgboost.py::test_without_dask_client

tests/unit/xgb/test_xgboost.py::TestXGBoost::test_music_regression

tests/unit/xgb/test_xgboost.py::test_gpu_hist_dmatrix[fit_kwargs0-DaskDeviceQuantileDMatrix]

tests/unit/xgb/test_xgboost.py::test_gpu_hist_dmatrix[fit_kwargs1-DaskDMatrix]

tests/unit/xgb/test_xgboost.py::TestEvals::test_multiple

tests/unit/xgb/test_xgboost.py::TestEvals::test_default

tests/unit/xgb/test_xgboost.py::TestEvals::test_train_and_valid

tests/unit/xgb/test_xgboost.py::TestEvals::test_invalid_data

/var/jenkins_home/workspace/merlin_models/models/merlin/models/xgb/init.py:266: UserWarning: Ignoring list columns as inputs to XGBoost model: ['item_genres', 'user_genres'].

warnings.warn(f"Ignoring list columns as inputs to XGBoost model: {list_column_names}.")
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

=========================== short test summary info ============================

SKIPPED [1] tests/unit/implicit/init.py:18: could not import 'implicit': No module named 'implicit'

SKIPPED [1] tests/unit/lightfm/init.py:18: could not import 'lightfm': No module named 'lightfm'

SKIPPED [1] tests/unit/datasets/test_advertising.py:20: No data-dir available, pass it through env variable $INPUT_DATA_DIR

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:62: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:78: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:92: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP

SKIPPED [3] tests/unit/datasets/test_entertainment.py:44: No data-dir available, pass it through env variable $INPUT_DATA_DIR

========== 468 passed, 9 skipped, 107 warnings in 1667.31s (0:27:47) ===========

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/models/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_models] $ /bin/bash /tmp/jenkins18262449675760801083.sh

* Started the integration tests for retrieval models using fiddle * Finished first version of retrieval integration tests * Fixing precommit checks * Added CI script for integration tests * Fixed CI integration commands * Enabling only retrieval integration tests * Fixing wandb import order * Refactored retrieval tests to run both as integration and unit tests * Fixed wandb import * Changed retrieval unit test to train for a full epoch to check the runtime using the P100 machine available for PR testing * Fixing wandb import * Changing memory management for running the retrieval integration tests * Limiting training steps of unit test on retrieval to 500 steps to check for runtime * Changing memory management of unit tests to test retrieval on P100 * Increasing test to train for a full epoch with LastFM * Sets W&B execution as successful only when the run is completely peformed * Evaluating MF retrieval test with smaller batch size and for fewer steps * Reducing the eval steps and eval batch size of two-tower retrieval test * Changed unit tests to use synthetic data rather than the real LastFM data, which will be used only by the retrieval integration tests * Fixed retrieval unit tests after rebasing * Fixed wandb import * Adjusting retrieval unit tests to run faster * Fixing tests * Fix retrieval test

gabrielspmoreira added area/tests ci labels Jun 28, 2022

gabrielspmoreira self-assigned this Jun 28, 2022

gabrielspmoreira marked this pull request as draft June 28, 2022 06:25

gabrielspmoreira force-pushed the retrieval_integration_tests branch from 326831a to 08a3845 Compare July 6, 2022 23:09

gabrielspmoreira marked this pull request as ready for review July 7, 2022 00:13

gabrielspmoreira force-pushed the retrieval_integration_tests branch 2 times, most recently from 17df36e to 5fc561a Compare July 7, 2022 15:33

gabrielspmoreira requested review from marcromeyn and sararb July 7, 2022 17:12

gabrielspmoreira force-pushed the retrieval_integration_tests branch from 70265b4 to b1d2255 Compare July 8, 2022 15:39

gabrielspmoreira force-pushed the retrieval_integration_tests branch from bd001a4 to 8c63e06 Compare July 11, 2022 14:40

gabrielspmoreira added 18 commits July 13, 2022 15:08

Fixing wandb import order

a1f4f4c

Refactored retrieval tests to run both as integration and unit tests

daa132d

Fixed wandb import

b6fe381

Changed retrieval unit test to train for a full epoch to check the ru…

dce1c38

…ntime using the P100 machine available for PR testing

Fixing wandb import

e776e39

Changing memory management for running the retrieval integration tests

d3fc414

Limiting training steps of unit test on retrieval to 500 steps to che…

086b47b

…ck for runtime

Changing memory management of unit tests to test retrieval on P100

48b20c5

Increasing test to train for a full epoch with LastFM

511b3ea

Sets W&B execution as successful only when the run is completely pefo…

805f122

…rmed

Evaluating MF retrieval test with smaller batch size and for fewer steps

2033e6c

Reducing the eval steps and eval batch size of two-tower retrieval test

fb38034

Changed unit tests to use synthetic data rather than the real LastFM …

8d17f07

…data, which will be used only by the retrieval integration tests

Fixed retrieval unit tests after rebasing

6b9d277

Fixed wandb import

3e9e03f

Adjusting retrieval unit tests to run faster

dd6369b

Fixing tests

9d4967c

Fix retrieval test

be9fe22

gabrielspmoreira force-pushed the retrieval_integration_tests branch from c63333d to be9fe22 Compare July 13, 2022 18:09

gabrielspmoreira requested a review from marcromeyn July 14, 2022 00:23

marcromeyn approved these changes Jul 14, 2022

View reviewed changes

Merge branch 'main' into retrieval_integration_tests

6678dbb

gabrielspmoreira merged commit 2356e4c into main Jul 14, 2022

gabrielspmoreira mentioned this pull request Jul 14, 2022

Adds integration tests to Merlin Models NVIDIA-Merlin/Merlin#438

Merged

gabrielspmoreira mentioned this pull request Aug 19, 2022

[RMP] Provide to our customers best practices guidance for training Retrieval, Ranking and Multi-Stage RecSys models NVIDIA-Merlin/Merlin#553

Closed

25 tasks

gabrielspmoreira mentioned this pull request Oct 4, 2022

[Task] Implement ranking models integration tests #785

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Integration Tests for Retrieval models #537

Integration Tests for Retrieval models #537

gabrielspmoreira commented Jun 28, 2022 •

edited

Loading

nvidia-merlin-bot commented Jun 28, 2022

nvidia-merlin-bot commented Jul 6, 2022

github-actions bot commented Jul 6, 2022

nvidia-merlin-bot commented Jul 7, 2022

nvidia-merlin-bot commented Jul 7, 2022

nvidia-merlin-bot commented Jul 7, 2022

nvidia-merlin-bot commented Jul 7, 2022

nvidia-merlin-bot commented Jul 7, 2022

nvidia-merlin-bot commented Jul 7, 2022

nvidia-merlin-bot commented Jul 7, 2022

nvidia-merlin-bot commented Jul 7, 2022

gabrielspmoreira commented Jul 7, 2022

nvidia-merlin-bot commented Jul 7, 2022

gabrielspmoreira commented Jul 8, 2022

nvidia-merlin-bot commented Jul 8, 2022

gabrielspmoreira commented Jul 8, 2022

nvidia-merlin-bot commented Jul 8, 2022

nvidia-merlin-bot commented Jul 11, 2022

nvidia-merlin-bot commented Jul 13, 2022

nvidia-merlin-bot commented Jul 13, 2022

nvidia-merlin-bot commented Jul 14, 2022

Integration Tests for Retrieval models #537

Integration Tests for Retrieval models #537

Conversation

gabrielspmoreira commented Jun 28, 2022 • edited Loading

Goals ⚽

Implementation Details 🚧

Testing Details 🔍

CI

Logging

nvidia-merlin-bot commented Jun 28, 2022

nvidia-merlin-bot commented Jul 6, 2022

github-actions bot commented Jul 6, 2022

Documentation preview

nvidia-merlin-bot commented Jul 7, 2022

nvidia-merlin-bot commented Jul 7, 2022

nvidia-merlin-bot commented Jul 7, 2022

nvidia-merlin-bot commented Jul 7, 2022

nvidia-merlin-bot commented Jul 7, 2022

nvidia-merlin-bot commented Jul 7, 2022

nvidia-merlin-bot commented Jul 7, 2022

nvidia-merlin-bot commented Jul 7, 2022

gabrielspmoreira commented Jul 7, 2022

nvidia-merlin-bot commented Jul 7, 2022

gabrielspmoreira commented Jul 8, 2022

nvidia-merlin-bot commented Jul 8, 2022

gabrielspmoreira commented Jul 8, 2022

nvidia-merlin-bot commented Jul 8, 2022

nvidia-merlin-bot commented Jul 11, 2022

nvidia-merlin-bot commented Jul 13, 2022

nvidia-merlin-bot commented Jul 13, 2022

nvidia-merlin-bot commented Jul 14, 2022

gabrielspmoreira commented Jun 28, 2022 •

edited

Loading