Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

【benchmark】 fix model_zoo path #8643

Merged
merged 16 commits into from
Jun 20, 2024
Merged

【benchmark】 fix model_zoo path #8643

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
ef5bd1d
add llama-7b_auto_dp2mp2pp2 benchmark sh
mmglove May 10, 2024
fc252a0
add llama-7b_auto_dp2mp2pp2 benchmark script for cinn
mmglove May 11, 2024
bcee7ed
update llama-7b_auto_dp2mp2pp2 benchmark script
mmglove May 11, 2024
25d0681
Update run_benchmark.sh
mmglove May 28, 2024
09cd337
Update run_benchmark.sh
mmglove May 28, 2024
893fc0c
Merge branch 'PaddlePaddle:develop' into develop
mmglove May 28, 2024
0ea8f23
fix llama-7b_auto_dp2mp2pp2/benchmark_common
mmglove May 28, 2024
638839c
Merge branch 'PaddlePaddle:develop' into develop
mmglove May 31, 2024
26fabaa
Update run_benchmark.sh
mmglove May 31, 2024
f75436d
Merge branch 'PaddlePaddle:develop' into develop
mmglove Jun 6, 2024
26fa8bf
Merge branch 'PaddlePaddle:develop' into develop
mmglove Jun 20, 2024
c49b7e1
Update prepare.sh
mmglove Jun 20, 2024
f895c9c
Update prepare.sh
mmglove Jun 20, 2024
55754a1
Update prepare.sh
mmglove Jun 20, 2024
564ea16
Update prepare.sh
mmglove Jun 20, 2024
41c33fc
Update prepare.sh
mmglove Jun 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions tests/test_tipc/auto_tuner/llama_pretrain/benchmark_common/prepare.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,11 @@ python -m pip install -r ../requirements.txt
python -m pip install -r ../requirements-dev.txt

# install fused_ln custom ops
cd ../model_zoo/gpt-3/external_ops/
cd ../legacy/model_zoo/gpt-3/external_ops/
python setup.py install

# install tool_helpers
cd ../../../llm/llama
cd ../../../../llm/llama
python -m pip install tool_helpers

rm -rf data && mkdir data
Expand Down
6 changes: 3 additions & 3 deletions tests/test_tipc/dygraph/hybrid_parallelism/llama/benchmark_common/prepare.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,16 +16,16 @@ python -m pip install -r ../requirements.txt
python -m pip install -r ../requirements-dev.txt

# install fused_ln custom ops
cd ../model_zoo/gpt-3/external_ops/
cd ../legacy/model_zoo/gpt-3/external_ops/
python setup.py install

# install tool_helpers
cd ../../../llm/llama
cd ../../../../llm/llama
python -m pip install tool_helpers

wget https://bj.bcebos.com/paddlenlp/models/transformers/llama/data/llama_openwebtext_100k_ids.npy
wget https://bj.bcebos.com/paddlenlp/models/transformers/llama/data/llama_openwebtext_100k_idx.npz

mkdir data
mv llama_openwebtext_100k_ids.npy ./data
mv llama_openwebtext_100k_idx.npz ./data
mv llama_openwebtext_100k_idx.npz ./data
4 changes: 2 additions & 2 deletions tests/test_tipc/dygraph/hybrid_parallelism/llama2/benchmark_common/prepare.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,11 @@ python -m pip install -r ../requirements.txt
python -m pip install -r ../requirements-dev.txt

# install fused_ln custom ops
cd ../model_zoo/gpt-3/external_ops/
cd ../legacy/model_zoo/gpt-3/external_ops/
python setup.py install

# install tool_helpers
cd ../../../llm/llama
cd ../../../../llm/llama
python -m pip install tool_helpers

# download data
Expand Down
6 changes: 3 additions & 3 deletions tests/test_tipc/dygraph/hybrid_parallelism/qwen/benchmark_common/prepare.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,16 +18,16 @@ python -m pip install -r ../requirements-dev.txt
python -m pip install tiktoken

# install fused_ln custom ops
cd ../model_zoo/gpt-3/external_ops/
cd ../legacy/model_zoo/gpt-3/external_ops/
python setup.py install

# install tool_helpers
cd ../../../llm/qwen
cd ../../../../llm/qwen
python -m pip install tool_helpers

wget https://bj.bcebos.com/paddlenlp/models/transformers/llama/data/llama_openwebtext_100k_ids.npy
wget https://bj.bcebos.com/paddlenlp/models/transformers/llama/data/llama_openwebtext_100k_idx.npz

mkdir data
mv llama_openwebtext_100k_ids.npy ./data
mv llama_openwebtext_100k_idx.npz ./data
mv llama_openwebtext_100k_idx.npz ./data
4 changes: 2 additions & 2 deletions tests/test_tipc/static/auto_parallel/llama2/benchmark_common/prepare.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,11 @@ python -m pip install -r ../requirements.txt
python -m pip install -r ../requirements-dev.txt

# install fused_ln custom ops
cd ../model_zoo/gpt-3/external_ops/
cd ../legacy/model_zoo/gpt-3/external_ops/
python setup.py install

# install tool_helpers
cd ../../../llm/llama
cd ../../../../llm/llama
python -m pip install tool_helpers

# download data
Expand Down
Loading