Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Core] Logprobs support in Multi-step #7652

Merged
merged 49 commits into from
Aug 30, 2024
Merged
Changes from 1 commit
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
f97e0ae
added example
afeldman-nm Aug 21, 2024
f969241
wip:
afeldman-nm Aug 21, 2024
642d31b
first working attempt at logprobs
afeldman-nm Aug 21, 2024
a0ca262
merge; format
afeldman-nm Aug 21, 2024
ed97288
passing test; dataclass
afeldman-nm Aug 21, 2024
861e1b9
refactoring
afeldman-nm Aug 21, 2024
8bc0765
Merge branch 'main' into logprobs_merge
afeldman-nm Aug 21, 2024
a34d1ac
refactoring
afeldman-nm Aug 21, 2024
4cda5c0
Merge branch 'logprobs' into logprobs_merge
afeldman-nm Aug 21, 2024
ac8a39a
Merge branch 'main' into logprobs_merge
afeldman-nm Aug 21, 2024
1284327
removing example
afeldman-nm Aug 21, 2024
a6c1207
removed example from build pipeline
afeldman-nm Aug 21, 2024
fe42995
fixed one docstring; embedded NUM_LOGPROBS
afeldman-nm Aug 21, 2024
9fb5bbe
test refactor
afeldman-nm Aug 21, 2024
046a8b1
incremental refactors
afeldman-nm Aug 21, 2024
fa86efd
remove unnecessary conftest change
afeldman-nm Aug 21, 2024
1c0ffb6
Update vllm/model_executor/layers/sampler.py
afeldman-nm Aug 21, 2024
3babadb
refactor
afeldman-nm Aug 21, 2024
f502029
Merge branch 'afeldman-nm/logprobs' of https://github.com/neuralmagic…
afeldman-nm Aug 21, 2024
1875b37
test_multi_step comment
afeldman-nm Aug 21, 2024
3760a95
utils function docstrings
afeldman-nm Aug 21, 2024
d43308c
docstring refactors
afeldman-nm Aug 21, 2024
54db498
merge
afeldman-nm Aug 21, 2024
dfbbaf0
passing tests & formatted
afeldman-nm Aug 21, 2024
5eebfca
Merge branch 'main' into logprobs_merge
afeldman-nm Aug 21, 2024
5e23d9a
Merge branch 'main' into logprobs_merge
afeldman-nm Aug 22, 2024
717efa3
merge; format
afeldman-nm Aug 22, 2024
e0d59ce
removed incorrect SamplerOutput imports
afeldman-nm Aug 22, 2024
102fd92
formatting
afeldman-nm Aug 22, 2024
948f4ef
Update tests/multi_step/test_correctness.py
afeldman-nm Aug 22, 2024
6e6711f
fixed comment
afeldman-nm Aug 22, 2024
f61163e
merge; format
afeldman-nm Aug 23, 2024
1cc93dd
rename
afeldman-nm Aug 23, 2024
4995204
Merge branch 'logprobs' into logprobs_merge
afeldman-nm Aug 23, 2024
da5826b
test modification
afeldman-nm Aug 26, 2024
d4fb430
merge; format
afeldman-nm Aug 26, 2024
b6752e0
merge
afeldman-nm Aug 27, 2024
1e42656
formatting
afeldman-nm Aug 27, 2024
cd0fdf9
disabled logprobs pythonization when logprobs are disabled
afeldman-nm Aug 27, 2024
3fecbc4
wip
afeldman-nm Aug 27, 2024
67bd035
skip logprobs processing entirely when logprobs are not enabled; form…
afeldman-nm Aug 27, 2024
419659d
multi-step output processing; formatting
afeldman-nm Aug 27, 2024
55eaab9
wip
afeldman-nm Aug 27, 2024
bae1fb9
small fixes
afeldman-nm Aug 27, 2024
fbb75b7
reverting to no prompt-logprobs support; merged in main
afeldman-nm Aug 28, 2024
63c5582
timeout increase
afeldman-nm Aug 28, 2024
8191571
refactoring
afeldman-nm Aug 28, 2024
9a708f8
Merge branch 'main' into logprobs_no_prompt
afeldman-nm Aug 28, 2024
e54606d
upstream merge
afeldman-nm Aug 29, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
utils function docstrings
  • Loading branch information
afeldman-nm committed Aug 21, 2024
commit 3760a95628c7488d1a6727c6d114f94c616fb9ab
32 changes: 14 additions & 18 deletions tests/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -423,19 +423,16 @@ async def completions_with_server_args(
server_cli_args: List[str],
num_logprobs: Optional[int],
) -> Completion:
'''
Construct a remote OpenAI server, obtain an async client to the
'''Construct a remote OpenAI server, obtain an async client to the
server & invoke the completions API to obtain completions.

Arguments:

* prompts: test prompts
* model_name: model to spin up on the vLLM server
* server_cli_args: CLI args for starting the server
Args:
prompts: test prompts
model_name: model to spin up on the vLLM server
server_cli_args: CLI args for starting the server

Returns:

* OpenAI Completion instance
OpenAI Completion instance
'''

outputs = None
Expand All @@ -454,7 +451,8 @@ async def completions_with_server_args(

def get_client_text_generations(completions: Completion):
'''Extract generated tokens from the output of a
request made to an Open-AI-protocol completions endpoint.'''
request made to an Open-AI-protocol completions endpoint.
'''
return [x.text for x in completions.choices]


Expand Down Expand Up @@ -486,8 +484,7 @@ def assert_all_close_logprobs(
atol: float = 1e-3,
rtol: float = 1e-3,
) -> None:
'''
Asserts that logprobs produced by the vLLM engine instance under test
'''Asserts that logprobs produced by the vLLM engine instance under test
are very close to a set of ground-truth reference values.

If the completions API was invoked with a non-`None` `logprobs` argument,
Expand All @@ -502,12 +499,11 @@ def assert_all_close_logprobs(
Else, if the completions API was invoked with `logprobs=None`, then
both the reference & test log probs should be List[None].

Arguments:

* ref_logprobs: ground-truth logprobs
* test_logprobs: logprobs produced by vLLM engine under test
* atol: absolute mismatch tolerance when comparing single logprobs
* rtol: relative mismatch tolerance when comparing single logprobs
Args:
ref_logprobs: ground-truth logprobs
test_logprobs: logprobs produced by vLLM engine under test
atol: absolute mismatch tolerance when comparing single logprobs
rtol: relative mismatch tolerance when comparing single logprobs
'''

assert len(ref_logprobs) == len(test_logprobs), (
Expand Down