Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ENH] set seed in test_score #6629

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

XinyuWuu
Copy link
Member

This PR add seed setting in test_score in TestAllForecasters.

The DeepAR model in PR #6551 is giving different output every call to predict which causes problem with test_score.

As suggested by @benHeid, maybe we can pass some parameters to the predict function in pytorch-forecasting model to control the sampling behavior, howerver I can't find such parameters. Luckily after setting seed for some fundamental packages before every call to predict, DeepAR gives the same output every time.

As mentioned by @fkiraly, more than one deep learning models are facing the same problem. Setting seed in test_score for some fundamental packages could be helpful for models using sampling or other non-deterministic methods to generate prediction.

@XinyuWuu
Copy link
Member Author

As discussed in the tech session, we might need to reset the seed back to original seed. We can do it for numpy, random and torch but not tensorflow.

Copy link
Collaborator

@fkiraly fkiraly left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To quickly summarize my feedback from the tech session:

  • we need to make sure that setting of seeds is not global and affects other tests or estimators
  • numpy.seed is afaik deprecated, generators should be used instead. This works via random_state for model that support it. Most deep learning models do not.
  • related to the above but not this PR directly, most tensorflow based models fail tests that also assume we can derandomize, e.g., test_multiprocessing_idempotent, hence this test is skipped for them in tests._config. If we can find a way to set the seed on a per-estimator-instance basis for tf or torch, that would be great. See [ENH] randomization/derandomization tag and conditional test logic #6274

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: PR in progress
Development

Successfully merging this pull request may close these issues.

2 participants