[Feature]: Pipeline Parallelism support for the Vision Language Models

[Manikandan-Thangaraj-ZS0321]

🚀 The feature, motivation and pitch

If I am not wrong, currently vllm supports only the Language models not the Vision models.

NotImplementedError: Pipeline parallelism is only supported for the following architectures: ['AquilaModel', 'AquilaForCausalLM', 'DeepseekV2ForCausalLM', 'InternLMForCausalLM', 'JAISLMHeadModel', 'LlamaForCausalLM', 'LLaMAForCausalLM', 'MistralForCausalLM', 'Phi3ForCausalLM', 'GPT2LMHeadModel', 'MixtralForCausalLM', 'NemotronForCausalLM', 'Qwen2ForCausalLM', 'Qwen2MoeForCausalLM', 'QWenLMHeadModel'].

This feature would greatly benefit teams and projects working with vision-language models, allowing them to scale out their workloads efficiently and maintain performance as model sizes continue to grow.

Also It would be greatly helpful, if someone can point me out on other possibilities for pipeline parallelism. Thanks in advance

Alternatives

No response

Additional context

No response

[youkaichao]

You can follow #7168 to add some if you want.

[Manikandan-Thangaraj-ZS0321]

Hi @youkaichao,

Thank a lot for the response. I was wondering if there's any way to know when the PR will be merged, as it doesn't seem to be updated with the latest changes from the main branch.

Additionally, could you please suggest any other frameworks that support pipeline parallelism? I need to implement the scale-out (pipeline parallelism) feature in my project as soon as possible.

Thanks again!

[youkaichao]

that pr #7168 should be merged after @andoorve comes back.

[Manikandan-Thangaraj-ZS0321]

@youkaichao, any ideas on frameworks that support pipeline parallelism other than vllm.

[youkaichao]

any ideas on frameworks that support pipeline parallelism other than vllm

sorry I don't know

[DarkLight1337]

The rest of the vision language models still don't support PP yet, so I'm reopening this.