Add Merlin example notebooks as integration tests with real data

[karlhigley]

No description provided.

[rnyak]

@karlhigley do we add integration tests for 22.05 release? or for 22.06? Basically, we are adding unit tests for the notebooks but should we do something extra for the integration tests.

[EvenOldridge]

On real data, not synthetic

[bschifferer]

@radekosmulski can you add an integration test for Merlin examples (unittest are from here): #288

Example for integration tests can be found here: NVIDIA-Merlin/models#414

[radekosmulski]

will do @bschifferer! 🙂

[radekosmulski]

I would like to give a big shoutout to @bschifferer and @rnyak -- I am starting to get a hang of the containers and how to work with them 🥳

I am making some progress on this, but I have a question. @bschifferer provided me with the real data I can use, but he shared it with me via google drive.

In CI could I please ask where or if the data will be mounted anywhere or do we need to pull it?

[bschifferer]

Hello @radekosmulski

we will assume that the data will be in this folder /raid/data/aliccp/raw/ - @jperez999 is working on adding it to the CI machine. The request is in this ticket: #212

/raid/data/aliccp/raw/
├── train
│   ├── train_0.parquet
│   ├── train_1.parquet
│   ├── train_2.parquet
│   ├── train_3.parquet
│   ├── train_4.parquet
│   ├── train_5.parquet
│   ├── train_6.parquet
│   └── train_7.parquet
└── valid
    ├── test_0.parquet
    ├── test_1.parquet
    ├── test_2.parquet
    ├── test_3.parquet
    ├── test_4.parquet
    ├── test_5.parquet
    ├── test_6.parquet
    └── test_7.parquet

Note that the test data is in the folder valid

[radekosmulski]

That is all I needed to know, @bschifferer! 🙂 Thank you very much!!!!

[radekosmulski]

@bschifferer if I am reading this right the transfrom_aliccp function expects the directories to be named train and test.

Should we keep the original naming or accommodate this via some other means?

[radekosmulski]

I restructured the test into two tests (one for each notebook).

I started to get OOM errors, but I am not completely sure if it was indeed because there was a single test (and I suppose state from the first notebook was retained while the 2nd one was being run) or if it was something that got messed up with my container

Nonetheless, with 2 test, a fresh container, everything works fine. No issue with first running the unit test and subsequently the integration test.

Data for the integration test lives in /raid/data/aliccp/raw/ but it doesn't follow the train/valid naming convention from above. Using train/test to accommodate transform_aliccp but can change that if need be.

What might be important for running this in CI -- even with splitting the test, GPU memory consumption still reaches around 32 GBs. The total runtime is also quite high for the integration test at 20 minutes