-
Notifications
You must be signed in to change notification settings - Fork 248
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature: integration standard libraries #11
Comments
And to Skimage |
And to Torch Fidelity -- for generative metrics |
I would rather reimplement these metrics than depend on other libs. My heuristic:
|
So this would be quite a deviation from the current status of metrics: most of them are wrappers around existing libraies. Just to name a few:
Would you reimplement all of them? As an example, there are O(100) metrics in scikit-learn and we only have a handful of them inside My main concern is that we can't move very fast and adopt a wide range of modalities if we implement most of them ourselves. I feel like a criteria for adoption is how fast we cover a wide range of metrics which avoids a user to leave the library for a missing one. On the other hand with community metrics users would probably quickly add missing metrics by creating wrappers since that's the fastest way (which is probably what kind of happened in Regarding dependencies: the idea was that each metric's repository has a requirement.txt file, so we don't need to add the dependencies to What do you think about the following:
|
Just giving my half-cent opinion here… if you are going to rewrite the metrics, it could be done in a way that you compute the metrics from a confusion matrix. I know this could be done for metrics used in computer vision, I don't know if the same would be true for NLP. But as already noted, this would probably take more time than third-party integrations 😅 |
Create a uniform integration for metrics libraries such as
scikit-learn
,keras.metrics
ortorchmetrics
instead of integrating them metrics from these frameworks one-by-one. The goal of the integration is to be able to load the metrics from these frameworks with a syntax as follows:This way there is also more transparency where the metrics come from especially if they are just a wrapper around existing libraries. This would also simplify maintaining them by avoiding the need to change every metric separately if needed.
Popular metrics libraries:
scikit-learn
keras.metrics
torchmetrics
Others (maybe a bit more niche):
PyDML
networkX
MLFlow
If all metrics (incl. "canonical" ones) will be on the hub such an integration could just be a script that automatically creates (or updates) the necessary repositories on the hub.
The text was updated successfully, but these errors were encountered: