You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently the perplexity metric and measurement both instantiate an entire model object within the _compute() function and run inference, which breaks the pattern where only predictions, references, and other metadata is passed in and only the metric computation is performed (e.g. inference is already run before the metric hits its _compute function, which is the case for most other metrics).
Evaluators already instantiate model-like objects with model_or_pipeline, which makes it redundant to reinstantiate another copy of the model/pipeline within the _compute() function. It seems like this is done because perplexity can be used in a standalone manner to calculate the perplexity of some text with regards to a pretrained model (e.g. perplexity.compute(model_id='gpt2', add_start_token=False, input_texts=input_texts), trading off developer flexibility for user-friendliness.
For background, this is relevant since I'm in the middle of implementing a "text-generation" evaluator, and it makes sense for perplexity to be the default metric for text generation. We could alternatively write one-off logic for the text generation evaluator to compute perplexity with the model instantiated in the evaluator instead of using the metric implemented in evaluate, but that seems suboptimally out-of-pattern. Additionally, perplexity metrics right now are limited to be used with models which can easily instantiated via AutoModelForCausalLM.from_pretrained(model_id), but does not support a generic call to a language model.
The text was updated successfully, but these errors were encountered:
Currently the
perplexity
metric and measurement both instantiate an entire model object within the_compute()
function and run inference, which breaks the pattern where only predictions, references, and other metadata is passed in and only the metric computation is performed (e.g. inference is already run before the metric hits its_compute
function, which is the case for most other metrics).Evaluators already instantiate model-like objects with
model_or_pipeline
, which makes it redundant to reinstantiate another copy of the model/pipeline within the_compute()
function. It seems like this is done because perplexity can be used in a standalone manner to calculate the perplexity of some text with regards to a pretrained model (e.g.perplexity.compute(model_id='gpt2', add_start_token=False, input_texts=input_texts)
, trading off developer flexibility for user-friendliness.For background, this is relevant since I'm in the middle of implementing a "text-generation" evaluator, and it makes sense for perplexity to be the default metric for text generation. We could alternatively write one-off logic for the text generation evaluator to compute perplexity with the model instantiated in the evaluator instead of using the metric implemented in
evaluate
, but that seems suboptimally out-of-pattern. Additionally, perplexity metrics right now are limited to be used with models which can easily instantiated viaAutoModelForCausalLM.from_pretrained(model_id)
, but does not support a generic call to a language model.The text was updated successfully, but these errors were encountered: