Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MNT] Update skbase to run sktime's contract validation tests #47

Closed
felipeangelimvieira opened this issue May 16, 2024 · 7 comments
Closed
Assignees
Labels
maintenance Continuous Integration, unit testing & package distribution

Comments

@felipeangelimvieira
Copy link
Owner

As discussed with @fkiraly, the previous version of skbase wasn't properly validating jax.numpy attributes (see sktime/skbase#324)

The tests are now being skipped, but can be properly executed with this update.

@fkiraly
Copy link
Contributor

fkiraly commented May 28, 2024

This specific issue should be resolved by sktime/skbase#323, right?

#324 is a "bonus" feature and we should keep the issue open, but for the purpose of prophetverse, this feels resolved?

@felipeangelimvieira
Copy link
Owner Author

Yes! It's working with #323. This week I expect to create a PR with some changes in code and without skipping the sktime contract tests. The tests passed in my local machine.

@felipeangelimvieira
Copy link
Owner Author

The test is creating some problems in the CI workflow... it seems to be a common issue with github actions (actions/runner-images#6680), and it may be related to CPU / memory usage. Have you ever experienced it in sktime, @fkiraly ?

Example of CI failure

@fkiraly
Copy link
Contributor

fkiraly commented Jun 12, 2024

Yes, and we have not fully diagnosed it.

Our suspicion is that this happens for memory intensive models, and the approach is selecting test parameters in get_test_params that lead to low memory and runtime requirements.

@fkiraly
Copy link
Contributor

fkiraly commented Jun 12, 2024

Plus, it seems that test_sktime_contract is being skipped, so the failure is coming from the other tests.

@felipeangelimvieira
Copy link
Owner Author

Plus, it seems that test_sktime_contract is being skipped, so the failure is coming from the other tests.

Locally, the tests are being executed partially but then skipped manually by a particular test case:

Output of pytest tests/sktime/test_sktime_check_estimator.py -vvrs

================================================================== short test summary info ===================================================================
SKIPPED [4] .venv/lib/python3.11/site-packages/sktime/forecasting/tests/test_all_forecasters.py:215: ForecastingHorizon with timedelta values is currently experimental and not supported everywhere
======================================================= 4 skipped, 1930 warnings in 619.14s (0:10:19) ========================================================

@fkiraly
Copy link
Contributor

fkiraly commented Jun 15, 2024

Should be addressed and explained by sktime/sktime#6587 - let's wait with closing this until the release though.

@felipeangelimvieira felipeangelimvieira added the maintenance Continuous Integration, unit testing & package distribution label Jun 21, 2024
@felipeangelimvieira felipeangelimvieira self-assigned this Jun 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
maintenance Continuous Integration, unit testing & package distribution
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants