Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Export of Openai Whisper with batched prompts #19854

Merged
merged 9 commits into from
Apr 3, 2024
Merged

Export of Openai Whisper with batched prompts #19854

Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
439211c
Add logic for padding prompts
shubhambhokare1 Mar 11, 2024
9ffae5b
Fix lint
shubhambhokare1 Mar 11, 2024
b34a04a
Modularize verify_onnx
shubhambhokare1 Mar 15, 2024
44efe35
minor edits
shubhambhokare1 Mar 28, 2024
d225e4d
remove looping logic
shubhambhokare1 Mar 29, 2024
94a0ddb
Avoid model cloning
shubhambhokare1 Mar 30, 2024
e7a48be
lint
shubhambhokare1 Mar 30, 2024
068f673
Apply suggestions from code review
shubhambhokare1 Apr 1, 2024
d33fb3a
Add cache dir
shubhambhokare1 Apr 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
remove looping logic
  • Loading branch information
@shubhambhokare1
shubhambhokare1 committed Mar 29, 2024
commit d225e4d5fecdba0e86c37e15ae8e3ff749cdef06
9 changes: 5 additions & 4 deletions onnxruntime/python/tools/transformers/models/whisper/whisper_helper.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -363,6 +363,9 @@ def pt_transcription_for_verify_onnx(
prompt_ids = [processor.get_prompt_ids(p) for p in prompts]
pt_transcription = []
pt_outputs = []
# The looping for model.generate is necessary here due to the limitation as per
# https://huggingface.co/docs/transformers/model_doc/whisper#transformers.WhisperForConditionalGeneration.generate.prompt_ids
# prompt_ids input requires a tensor of rank 1
for i in range(batch_size):
inputs["prompt_ids"] = torch.from_numpy(prompt_ids[i])
inputs["input_features"] = input_features_[i].to(device)
Expand Down Expand Up @@ -493,10 +496,8 @@ def verify_onnx(
inputs[name] = np.array([1.0], dtype=ort_to_np[dtype])
else:
inputs[name] = np.array([inputs[name]], dtype=ort_to_np[dtype])
ort_outputs = ort_session.run(None, inputs)[0]
ort_transcription = []
for o in ort_outputs:
ort_transcription.append(processor.batch_decode(o, skip_special_tokens=True)[0])
ort_outputs = ort_session.run(None, inputs)[0][:, 0, :]
ort_transcription = processor.batch_decode(ort_outputs, skip_special_tokens=True)
expected_transcription_options = WhisperHelper.select_transcription_options(batch_size, prompt_mode)

parity = 1
Expand Down
Loading