Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Model] Initialize support for InternVL2 series models #6514

Merged
merged 42 commits into from
Jul 29, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
4102e8c
init internvl support
Isotr0py Jul 10, 2024
2feadc8
add internvl2 support for 1B and 2B
Isotr0py Jul 13, 2024
958a926
add internvl2 example
Isotr0py Jul 17, 2024
7142c60
add internvl2 test
Isotr0py Jul 17, 2024
26c2651
fix internvl2 test
Isotr0py Jul 17, 2024
6e8135c
Merge branch 'vllm-project:main' into internvl
Isotr0py Jul 17, 2024
9d8cb20
format code
Isotr0py Jul 17, 2024
45b0cd7
add docs
Isotr0py Jul 17, 2024
1dafa5e
remove unused code in test
Isotr0py Jul 17, 2024
1fbe958
fix dummy data for internvl
Isotr0py Jul 17, 2024
91572b9
format code
Isotr0py Jul 17, 2024
56e171f
update internvl example
Isotr0py Jul 20, 2024
5971c11
fix internvl-2B test
Isotr0py Jul 21, 2024
cf1784c
Merge branch 'vllm-project:main' into internvl
Isotr0py Jul 25, 2024
ea61600
fix internvl test
Isotr0py Jul 25, 2024
49bdf60
format internvl2 test
Isotr0py Jul 25, 2024
f2d6bdd
add timm to test requirements
Isotr0py Jul 25, 2024
8aa0ac7
fix internvl test
Isotr0py Jul 25, 2024
95d8b4f
Merge branch 'main' into internvl
Isotr0py Jul 26, 2024
9bee8a8
port and format internvl config
Isotr0py Jul 26, 2024
4d9946c
format code
Isotr0py Jul 26, 2024
4dae318
isort
Isotr0py Jul 26, 2024
eea984f
format stacked_params_mapping
Isotr0py Jul 26, 2024
1b7c795
remove broken 4B test
Isotr0py Jul 27, 2024
6fb37ec
Merge branch 'main' into internvl
Isotr0py Jul 27, 2024
9f3cd25
fix a typo
Isotr0py Jul 27, 2024
b861119
Merge branch 'vllm-project:main' into internvl
Isotr0py Jul 27, 2024
405162e
migrate internvl example
Isotr0py Jul 27, 2024
c2e8cba
use sdpa for internvl attention
Isotr0py Jul 27, 2024
de8573d
update chat_utils and model_type
Isotr0py Jul 27, 2024
aeb10a9
reorganize internvl code
Isotr0py Jul 27, 2024
443650e
Merge branch 'main' into internvl
Isotr0py Jul 27, 2024
a32fcbe
refactor intern_vit
Isotr0py Jul 27, 2024
61df94f
fix example template
Isotr0py Jul 27, 2024
daa2233
fix test internvl prompt format
Isotr0py Jul 27, 2024
72d8350
format code
Isotr0py Jul 27, 2024
eb914ff
Fix internvl input processor
Isotr0py Jul 27, 2024
c243d71
separate llm_class selection
Isotr0py Jul 28, 2024
10c11e3
isort
Isotr0py Jul 28, 2024
e17c0be
revert llm backbone selection
Isotr0py Jul 28, 2024
a846154
remove template and fix typo
Isotr0py Jul 29, 2024
26fe0b0
update
ywang96 Jul 29, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
format code
  • Loading branch information
Isotr0py committed Jul 27, 2024
commit 72d83505c764b505aad06f33980322840632bce4
4 changes: 2 additions & 2 deletions tests/models/test_internvl.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,9 @@

HF_IMAGE_PROMPTS = IMAGE_ASSETS.prompts({
"stop_sign":
"<|im_start|>User\n<image>\nWhat's the content in the center of the image?<|im_end|>\n<|im_start|>Assistant\n",
"<|im_start|>User\n<image>\nWhat's the content in the center of the image?<|im_end|>\n<|im_start|>Assistant\n", # noqa: E501
"cherry_blossom":
"<|im_start|>User\n<image>\nWhat is the season?<|im_end|>\n<|im_start|>Assistant\n",
"<|im_start|>User\n<image>\nWhat is the season?<|im_end|>\n<|im_start|>Assistant\n", # noqa: E501
})

# we use snapshot_download to prevent conflicts between
Expand Down
2 changes: 1 addition & 1 deletion vllm/model_executor/models/internvl.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,10 @@
from vllm.attention import AttentionMetadata
from vllm.config import CacheConfig, MultiModalConfig
from vllm.inputs import INPUT_REGISTRY, InputContext, LLMInputs
from vllm.model_executor.models.intern_vit import InternVisionModel
from vllm.model_executor.layers.quantization import QuantizationConfig
from vllm.model_executor.model_loader.weight_utils import default_weight_loader
from vllm.model_executor.models import ModelRegistry
from vllm.model_executor.models.intern_vit import InternVisionModel
from vllm.model_executor.sampling_metadata import SamplingMetadata
from vllm.multimodal import MULTIMODAL_REGISTRY, BatchedTensors
from vllm.multimodal.base import MultiModalInputs
Expand Down
Loading