Beam search seems not work as expected

[goodbai-nlp]

Hi,

I try to use the beam search of vllm with the following code, but I found it processes and generates much more results than input, could anyone kindly tell me how to fix that?

model = LLM(model=args.model_name_or_path)
    prompts = [
        "Hello, my name is",
        "The president of the United States is",
        "The capital of France is",
        "The future of AI is",
    ]
    gen_params = SamplingParams(n=1, use_beam_search=True, best_of=5, temperature=0, top_p=1, top_k=-1, max_tokens=args.max_new_tokens, stop=["</s>"])
    outputs = model.generate(prompts, gen_params)
    print(f"generate {len(outputs)} outputs!!!")

[WoosukKwon]

Hi @goodbai-nlp thanks for trying out vLLM! According to your output, it seems you put 173 prompts as the input to LLM. Could you check again?

[goodbai-nlp]

@WoosukKwon yeah that's weird, I am sure that only put 4 prompts are taken as input (and I print the request number to confirm that). Could you reproduce this bug?

[WoosukKwon]

@goodbai-nlp Could you try running https://github.com/vllm-project/vllm/blob/main/examples/offline_inference.py and let me know the output?

When I ran the attached code, I got 4 outputs.

[goodbai-nlp]

@WoosukKwon, I got 4 outputs from the attached code. However, when I try to use beam search and set a larger value for max_tokens, the bug occurs.

The output when max_tokens=64:

The output when max_tokens=256:

In my opinion, max_tokens should only affect the max number of generated token.

[WoosukKwon]

@goodbai-nlp I reproduced the bug and fixed it in #202. Thanks for the bug report!