Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add LLaMA end-to-end benchmarking #19985

Merged
merged 18 commits into from
Mar 22, 2024
Merged

Add LLaMA end-to-end benchmarking #19985

Changes from 1 commit
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
cad5aab
Add E2E benchmarking
kunal-vaishnavi Mar 17, 2024
3abc6fe
Add license to files and changes suggested by linter
kunal-vaishnavi Mar 18, 2024
72c92b8
Fix runtime errors
kunal-vaishnavi Mar 18, 2024
ac1f266
Update README
kunal-vaishnavi Mar 18, 2024
f3d9036
Update CSV labels
kunal-vaishnavi Mar 19, 2024
45774c2
Add cache dir
kunal-vaishnavi Mar 19, 2024
8172c28
Add changes suggested by linter
kunal-vaishnavi Mar 19, 2024
5ff17cf
Fix comment formatting
kunal-vaishnavi Mar 20, 2024
33897a2
Fix linter error in CI that differs locally
kunal-vaishnavi Mar 20, 2024
5736e14
Update README
kunal-vaishnavi Mar 20, 2024
559a738
Fix bug with input shapes
kunal-vaishnavi Mar 20, 2024
c44005b
Merge branch 'main' into kvaishnavi/llama-e2e
kunal-vaishnavi Mar 20, 2024
f0ebf4d
Add changes suggested by linter
kunal-vaishnavi Mar 20, 2024
b3971d1
Add longer prompts
kunal-vaishnavi Mar 21, 2024
0b21f14
Add error message for unrecognized prompt lengths
kunal-vaishnavi Mar 21, 2024
30a5973
Add local directory option for loading Hugging Face models
kunal-vaishnavi Mar 21, 2024
2dc0762
Clarify arg descriptions
kunal-vaishnavi Mar 21, 2024
34110f5
Update local directory arg description
kunal-vaishnavi Mar 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update README
  • Loading branch information
@kunal-vaishnavi
kunal-vaishnavi committed Mar 18, 2024
commit ac1f266f9e2dbb08b67bbed019c7dc8de5025360
6 changes: 6 additions & 0 deletions onnxruntime/python/tools/transformers/models/llama/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,12 @@
# Contents
- [LLaMA-2](#llama-2)
- [Prerequisites](#prerequisites)
- [Exporting LLaMA-2](#exporting-llama-2)
- [Benchmarking LLaMA-2](#benchmark-llama-2)
- [Variants](#variants)
- [Benchmark All](#benchmark-all)
- [Benchmark E2E](#benchmark-e2e)
- [E2E Inference with LLaMA-2](#e2e-inference-with-llama-2)
- [Mistral](#mistral)
- [Exporting Mistral](#exporting-mistral)
- [Optimizing and Quantizing Mistral](#optimizing-and-quantizing-mistral)
Expand Down Expand Up @@ -427,6 +429,10 @@ CUDA_VISIBLE_DEVICES=0 python3 -m models.llama.benchmark_e2e \
--auth
```

## E2E Inference with LLaMA-2

For end-to-end inference, please visit [this folder](https://github.com/microsoft/onnxruntime-inference-examples/tree/main/python/models/llama) in ONNX Runtime Inference Examples for a step-by-step walkthrough, code examples, and performance metrics.

# Mistral

## Introduction
Expand Down