[Docs] Benchmark docs #5360
[Docs] Benchmark docs #5360
Conversation
examples/benchmarking/README.md
Outdated
|
|
||
| If you would like to list benchmark results on your favorite models of the [model hub](https://huggingface.co/models) here, please open a Pull Request and add it below. | ||
|
|
||
| | Benchmark description | Results | Environment info | Author | |
There was a problem hiding this comment.
Is there one number we could put in the actual table. Like tokens/second or MB/1000 tokens so that this table is more than just a ton of links?
There was a problem hiding this comment.
I'd also have a Framework column instead of including in description. And maybe also Date
There was a problem hiding this comment.
I love this idea though!
There was a problem hiding this comment.
How good is GitHub at diff'ing large CSV files? I was thinking of copying all results into a single, monster (multi-dimensional?) spreadsheet like @LysandreJik's spreadsheet
| >>> python examples/benchmarking/run_benchmark_tf.py --help | ||
|
|
||
|
|
||
| An instantiated benchmark object can then simply be run by calling ``benchmark.run()``. |
There was a problem hiding this comment.
this seems to not linearly follow from the command line example above.
|
|
||
| Here, three arguments are given to the benchmark argument data classes, namely ``models``, ``batch_sizes``, and ``sequence_lengths``. The argument ``models`` is required and expects a :obj:`list` of model identifiers from the `model hub <https://huggingface.co/models>`__ | ||
| The :obj:`list` arguments ``batch_sizes`` and ``sequence_lengths`` define the size of the ``input_ids`` on which the model is benchmarked. | ||
| There are many more parameters that can be configured via the benchmark argument data classes. For more detail on these one can either directly consult the files |
There was a problem hiding this comment.
I would have one example with all reasonable clargs. It's easier to delete than add.
docs/source/benchmarks.rst
Outdated
| How to benchmark 🤗 Transformer models | ||
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||
|
|
||
| The classes :obj:`PyTorchBenchmark` and :obj:`TensorflowBenchmark` allow to flexibly benchmark 🤗 Transformer models. |
There was a problem hiding this comment.
Use :class:~transformers.PyTorchBenchmark and :class:~transformers.TensorflowBenchmark so that we get a link to the doc (once it's added ;-) ) here and in the rest of the document.
|
Actually will make a separate PR for the examples README.md - to have the docs in v3.0.0. |
docs/source/benchmarks.rst
Outdated
| How to benchmark 🤗 Transformer models | ||
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||
|
|
||
| The classes :obj:`PyTorchBenchmark` and :obj:`TensorflowBenchmark` allow to flexibly benchmark 🤗 Transformer models. |
There was a problem hiding this comment.
not sure where this comes from actually :D -> was supposed to be a HF smiley
Co-authored-by: Lysandre Debut <lysandre@huggingface.co>
There was a problem hiding this comment.
@LysandreJik Are the doctest included in the tests now? I think this file should be ignored if that's the case because:
- is it fast enough?
- the output seems very dependent of the system
docs/source/benchmarks.rst
Outdated
|
|
||
| Hereby, `inference` is defined by a single forward pass, and `training` is defined by a single forward pass and backward pass. | ||
|
|
||
| The benchmark classes :obj:`PyTorchBenchmark` and :obj:`TensorflowBenchmark` expect an object of type :obj:`PyTorchBenchmarkArguments` and :obj:`TensorflowBenchmarkArguments`, respectively, for instantiation. :obj:`PyTorchBenchmarkArguments` and :obj:`TensorflowBenchmarkArguments` are data classes and contain all relevant configurations for their corresponding benchmark class. |
There was a problem hiding this comment.
Same for :obj:PyTorchBenchmarkArguments and :obj:TensorflowBenchmarkArguments
(PS: if this is still experimental and can be renamed, Tensorflow is spelled TensorFlow normally, PyTorchBenchmark has the right spelling)
|
@sgugger right now it's not ignored, so the slow test will fail because the output isn't the same. I don't think it's too big of a deal though, we can fix that after the release with only partial testing of the file. |
…vonplaten/transformers into update_doc_for_benchmark
|
Thanks for the review. Addressed them and also renamed the classes for consistency. |
This PR updates the docs for benchmarks and adds a README.md where the community can post their benchmark results.
Would be happy about Feedback from @sgugger and @LysandreJik .
@LysandreJik - I deleted the part about "This work was done by Timothy Liu." because the links were broken.