-
-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add minimum capability requirement for AWQ #1064
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -68,6 +68,14 @@ def get_model(model_config: ModelConfig) -> nn.Module: | |
quant_config = get_quant_config(model_config.quantization, | ||
model_config.model, | ||
model_config.download_dir) | ||
capability = torch.cuda.get_device_capability() | ||
capability = capability[0] * 10 + capability[1] | ||
if capability < quant_config.get_min_capability(): | ||
raise ValueError( | ||
f"The quantization method {model_config.quantization} is not " | ||
"supported for the current GPU. " | ||
f"Minimum capability: {quant_config.get_min_capability()}. " | ||
f"Current capability: {capability}.") | ||
Comment on lines
+71
to
+78
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why do we need the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Just saw the comments in the PR. Can we just change There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It's problematic when we want to build the wheel for all GPU architectures (e.g., for pypi publication or building docker image). In such a case, we cannot selectively include the extension according to the architecture. Therefore, I believe this is an easier solution, and in fact we already used this kind of guard for bfloat16 attention kernels, which do not support Turing and Volta GPUs. |
||
supported_dtypes = quant_config.get_supported_act_dtypes() | ||
if model_config.dtype not in supported_dtypes: | ||
raise ValueError( | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is this namespace needed?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
They are optional, but for better coding convention. From google cpp style guide:
Namespace prevents naming conflicts, so it's pretty useful for external code like the AWQ kernels.