-
Notifications
You must be signed in to change notification settings - Fork 248
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Is TextGenerationEvaluator incomplete? #537
Comments
Hey @inf3rnus, did you figure it out? |
Its crazy that its not solved yet. |
@DiabolicDev Nope, It's been 84 years since I've looked at this, but I believe if the dataset doesn't have an evaluator defined in it, then you need to implement the code for whatever metric you're trying to evaluate yourself. Depending on how bad you need this to work, I'd try to step what I did via a debugger, or I'd checkout https://github.com/EleutherAI/lm-evaluation-harness in the meantime. |
Yeah definitely man, I've been looking at other evaluation libraries, just thought it'd be super easy to run these evaluations on Colab using models from huggingface and their evaluation library if they had made it properly. Anyways, thanks for letting me know! |
@DiabolicDev Anytime! Happy hunting |
Hey all, fascinating library you got going on here.
Was trying to get a working example for perplexity on
EleutherAI/lambada_openai
withgpt2
Unfortunately, the only way I could get it working was by doing something like:
Okay, so that works, what doesn't is the task evaluator class for text generation. This code throws
I get the error:
And this is likely because compute in evaluter/base.py needs to have metric inputs returned from self.prepare_data() in the TextGenerationEvaluator class
e.g. first element of the tuple returned in this guy is empty
but is used by this code, which has nothing to compare against when evaluating perplexity, which is why the error was produced bc
never gets called before
self._finalize()
in module.py (formetric.compute()
)Wondering if it's still in progress, only works with some metrics for the time being, or what the deal is?
Final question - all the logic for determining how a metric is computed is held within the metric itself correct?
So e.,g. if I were trying to compute accuracy on lambada, I'd have to implement that myself?
Many thanks!
The text was updated successfully, but these errors were encountered: