Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Core] Support serving encoder/decoder models #7258

Merged
merged 35 commits into from
Aug 9, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
33c9e25
Introduce `is_list_of`
DarkLight1337 Aug 7, 2024
e6dd6f5
Avoid circular imports
DarkLight1337 Aug 7, 2024
f938c86
Refactor prompt parsing and extend this to async engine
DarkLight1337 Aug 7, 2024
6332d1e
Remove unnecessary comments
DarkLight1337 Aug 7, 2024
07b4d21
Enable full async
DarkLight1337 Aug 7, 2024
e29864c
grammar
DarkLight1337 Aug 7, 2024
c9dfb40
Add description
DarkLight1337 Aug 7, 2024
1233192
Fix wrong type annotations
DarkLight1337 Aug 7, 2024
f332275
Merge branch 'upstream' into inputs-parser
DarkLight1337 Aug 7, 2024
dcdebee
Remove redundant docs
DarkLight1337 Aug 7, 2024
65db3f1
Be more strict
DarkLight1337 Aug 7, 2024
9ffeb22
Fix docs
DarkLight1337 Aug 7, 2024
c9e0b08
Fix 2
DarkLight1337 Aug 7, 2024
14bca1f
Disallow multi-modal data for enc/dec models
DarkLight1337 Aug 7, 2024
8fc7099
Improve type narrowing behavior using `TypeIs`
DarkLight1337 Aug 7, 2024
3a8a072
Avoid sequential await
DarkLight1337 Aug 7, 2024
ef5327c
Fix type annotations based on test files
DarkLight1337 Aug 7, 2024
8a835cc
Properly handle `inputs["decoder_prompt"]=None`
DarkLight1337 Aug 7, 2024
e0024c2
Clean
DarkLight1337 Aug 7, 2024
76af172
Clean
DarkLight1337 Aug 7, 2024
5c16f2e
Fix incorrect decoder inputs in singleton case
DarkLight1337 Aug 7, 2024
e239ba9
Clean
DarkLight1337 Aug 7, 2024
4b0e3df
Move functions to a more appropriate place
DarkLight1337 Aug 7, 2024
53f7f50
Remove outdated comment
DarkLight1337 Aug 7, 2024
3afdbc5
Fix mismatch between hf and vllm output text
DarkLight1337 Aug 7, 2024
c61b01f
Factor out duplicate code
DarkLight1337 Aug 7, 2024
f8ed373
Factor out more duplicate code
DarkLight1337 Aug 7, 2024
a4df70a
Remove default values to avoid accidentally miss those arguments
DarkLight1337 Aug 7, 2024
5240bb3
Add test for serving encoder/decoder model with OpenAI server
DarkLight1337 Aug 7, 2024
d321c82
Use two type variables
DarkLight1337 Aug 7, 2024
931d1f6
Merge branch 'upstream' into inputs-parser
DarkLight1337 Aug 7, 2024
a06c67f
Merge branch 'upstream' into inputs-parser
DarkLight1337 Aug 7, 2024
9f64a05
Merge branch 'upstream' into inputs-parser
DarkLight1337 Aug 7, 2024
e4c5c21
Update error message
DarkLight1337 Aug 8, 2024
68fbf5a
Merge branch 'upstream' into inputs-parser
DarkLight1337 Aug 8, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix incorrect decoder inputs in singleton case
  • Loading branch information
DarkLight1337 committed Aug 7, 2024
commit 5c16f2e90f4c93c782676f4580eca2ec5f7c3c3b
2 changes: 1 addition & 1 deletion vllm/engine/async_llm_engine.py
Original file line number Diff line number Diff line change
Expand Up @@ -375,7 +375,7 @@ async def _process_encoder_decoder_prompt_async(
request_id=request_id,
)

decoder_comps = encoder_comps
decoder_comps = None, None, None

encoder_prompt, encoder_prompt_ids, encoder_mm_data = encoder_comps
decoder_prompt, decoder_prompt_ids, decoder_mm_data = decoder_comps
Expand Down
2 changes: 1 addition & 1 deletion vllm/engine/llm_engine.py
Original file line number Diff line number Diff line change
Expand Up @@ -847,7 +847,7 @@ def _process_encoder_decoder_prompt(
request_id=request_id,
)

decoder_comps = encoder_comps
decoder_comps = None, None, None

encoder_prompt, encoder_prompt_ids, encoder_mm_data = encoder_comps
decoder_prompt, decoder_prompt_ids, decoder_mm_data = decoder_comps
Expand Down
2 changes: 1 addition & 1 deletion vllm/inputs/data.py
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ class ExplicitEncoderDecoderPrompt(TypedDict, Generic[_T]):

Only the encoder prompt may have multi-modal data.

Note that an ExplicitEncoderDecoderPrompt may not
Note that an :class:`ExplicitEncoderDecoderPrompt` may not
be used as an input to a decoder-only model,
and that the `encoder_prompt` and `decoder_prompt`
fields of this data structure themselves must be
Expand Down
Loading