-
Notifications
You must be signed in to change notification settings - Fork 251
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature: add community metrics #4
Comments
Also cc @julien-c |
maybe just name the file
Yes, you can just add a |
Thanks, that's a good idea. Maybe we keep |
To clarify: I think there is a small thing left to implement in moon-landing (cc @cbensimon) but then you will be able to query https://hf.co/api/spaces?filter=metric i.e. same as for other repo types But in the meantime you can already add tags to README.md "space cards" |
Agree these should be two separate files. If you want to use a metric on the hub, you should just need to download |
We would like a way for community members to easily add custom or new metrics. These custom metrics could be hosted on the hub either in a dedicated repository type (like datasets or models) or on Spaces.
Spaces comes with a few advantages, especially in short term:
@osanseviero made a PoC here: https://huggingface.co/spaces/osanseviero/accuracy_metric
A few things that we should add to make this as user friendly and useful as possible:
README.md
(like model/dataset card) with basic structuremy_metric.py
file (similar todatasets
loading scripts)app.py
that loads all information fromREADME.md
andmy_metric.py
to create a default widget. This does probably not need to be changed.evaluate-cli
command to test if everything is fineevaluate-cli test-metric PATH_TO_REPO
list_metrics()
function in theevaluate
without needing to look at the content of each repo.The goal is to enable behaviour (example with the space above):
The text was updated successfully, but these errors were encountered: