You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The "official" implementation of perplexity metric does not cache language model [1]. It seems that metric instance should fetch model and prepare it for further usage in _download_and_prepare. I suppose that there should be clear API on caching and cache resetting. Also, it is totally unclear how to configure metrics on loading (there is only config_name but kwargs are just ignored).
The text was updated successfully, but these errors were encountered:
The "official" implementation of
perplexity
metric does not cache language model [1]. It seems that metric instance should fetch model and prepare it for further usage in_download_and_prepare
. I suppose that there should be clear API on caching and cache resetting. Also, it is totally unclear how to configure metrics on loading (there is onlyconfig_name
butkwargs
are just ignored).The text was updated successfully, but these errors were encountered: