Adds CLIP to models exportable with ONNX

[unography]

This isn't currently working, getting an error while validating the model -

onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Unexpected input data type. Actual: (tensor(float)) , expected: (tensor(int64))

Environment:
Pytorch: 1.13.0.dev20220806
onnxruntime: 1.12.0

Would love some guidance here!

@chainyo @patrickvonplaten

[chainyo]

Hi, @unography. Could you give us a more detailed traceback, please?

It's hard to say without the script command and the full traceback.

[unography]

Hi, @unography. Could you give us a more detailed traceback, please?

It's hard to say without the script command and the full traceback.

this is the full traceback -

(transformers) ➜  transformers git:(main) python -m transformers.onnx --model=openai/clip-vit-base-patch32 onnx/
vocab_file vocab.json
merges_file merges.txt
tokenizer_file tokenizer.json
added_tokens_file added_tokens.json
special_tokens_map_file special_tokens_map.json
tokenizer_config_file tokenizer_config.json
Using framework PyTorch: 1.13.0.dev20220806
/Users/dhruv/Documents/code/transformers/src/transformers/models/clip/modeling_clip.py:222: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attn_weights.size() != (bsz * self.num_heads, tgt_len, src_len):
/Users/dhruv/Documents/code/transformers/src/transformers/models/clip/modeling_clip.py:262: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attn_output.size() != (bsz * self.num_heads, tgt_len, self.head_dim):
/Users/dhruv/Documents/code/transformers/src/transformers/models/clip/modeling_clip.py:680: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
  mask.fill_(torch.tensor(torch.finfo(dtype).min))
/Users/dhruv/Documents/code/transformers/src/transformers/models/clip/modeling_clip.py:230: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if causal_attention_mask.size() != (bsz, 1, tgt_len, src_len):
/Users/dhruv/Documents/code/transformers/src/transformers/models/clip/modeling_clip.py:239: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attention_mask.size() != (bsz, 1, tgt_len, src_len):
/Users/dhruv/Documents/code/transformers/.venv/lib/python3.8/site-packages/torch/onnx/symbolic_opset9.py:4592: UserWarning: Exporting aten::index operator of advanced indexing in opset 14 is achieved by combination of multiple ONNX operators, including Reshape, Transpose, Concat, and Gather. If indices include negative values, the exported graph will produce incorrect results.
  warnings.warn(
Validating ONNX model...
Traceback (most recent call last):
  File "/Users/dhruv/.pyenv/versions/3.8.12/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/dhruv/.pyenv/versions/3.8.12/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/dhruv/Documents/code/transformers/src/transformers/onnx/__main__.py", line 107, in <module>
    main()
  File "/Users/dhruv/Documents/code/transformers/src/transformers/onnx/__main__.py", line 100, in main
    validate_model_outputs(onnx_config, preprocessor, model, args.output, onnx_outputs, args.atol)
  File "/Users/dhruv/Documents/code/transformers/src/transformers/onnx/convert.py", line 405, in validate_model_outputs
    onnx_outputs = session.run(onnx_named_outputs, onnx_inputs)
  File "/Users/dhruv/Documents/code/transformers/.venv/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 200, in run
    return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Unexpected input data type. Actual: (tensor(float)) , expected: (tensor(int64))

[regisss]

@unography You need to pass the ONNX inputs in the same order as they are registered in the forward method of the model. We can see here that pixel_values comes before attention_mask, so in the ONNX config you must return:

OrderedDict(
  [
      ("input_ids", {0: "batch", 1: "sequence"}),
      ("pixel_values", {0: "batch"}),
      ("attention_mask", {0: "batch", 1: "sequence"}),
  ]
)

Note that it is an OrderedDict so the order matters :)

To explain a bit the error message, what was happening is that it expected the second input to be int64 since that is how you defined it in the ONNX config. But it actually got a float tensor because pixel_values is passed before attention_mask in the forward method.

[unography]

@unography You need to pass the ONNX inputs in the same order as they are registered in the forward method of the model. We can see here that pixel_values comes before attention_mask, so in the ONNX config you must return:

OrderedDict(
  [
      ("input_ids", {0: "batch", 1: "sequence"}),
      ("pixel_values", {0: "batch"}),
      ("attention_mask", {0: "batch", 1: "sequence"}),
  ]
)

Note that it is an OrderedDict so the order matters :)

To explain a bit the error message, what was happening is that it expected the second input to be int64 since that is how you defined it in the ONNX config. But it actually got a float tensor because pixel_values is passed before attention_mask in the forward method.

ah yes, understood. able to resolve this issue, getting an error on the output values now

(transformers) ➜  transformers git:(main) python -m transformers.onnx --model=openai/clip-vit-base-patch32 onnx/
vocab_file vocab.json
merges_file merges.txt
tokenizer_file tokenizer.json
added_tokens_file added_tokens.json
special_tokens_map_file special_tokens_map.json
tokenizer_config_file tokenizer_config.json
Using framework PyTorch: 1.13.0.dev20220806
/Users/dhruv/Documents/code/transformers/src/transformers/models/clip/modeling_clip.py:222: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attn_weights.size() != (bsz * self.num_heads, tgt_len, src_len):
/Users/dhruv/Documents/code/transformers/src/transformers/models/clip/modeling_clip.py:262: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attn_output.size() != (bsz * self.num_heads, tgt_len, self.head_dim):
/Users/dhruv/Documents/code/transformers/src/transformers/models/clip/modeling_clip.py:680: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
  mask.fill_(torch.tensor(torch.finfo(dtype).min))
/Users/dhruv/Documents/code/transformers/src/transformers/models/clip/modeling_clip.py:230: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if causal_attention_mask.size() != (bsz, 1, tgt_len, src_len):
/Users/dhruv/Documents/code/transformers/src/transformers/models/clip/modeling_clip.py:239: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attention_mask.size() != (bsz, 1, tgt_len, src_len):
/Users/dhruv/Documents/code/transformers/.venv/lib/python3.8/site-packages/torch/onnx/symbolic_opset9.py:4592: UserWarning: Exporting aten::index operator of advanced indexing in opset 14 is achieved by combination of multiple ONNX operators, including Reshape, Transpose, Concat, and Gather. If indices include negative values, the exported graph will produce incorrect results.
  warnings.warn(
Validating ONNX model...
Traceback (most recent call last):
  File "/Users/dhruv/.pyenv/versions/3.8.12/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/dhruv/.pyenv/versions/3.8.12/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/dhruv/Documents/code/transformers/src/transformers/onnx/__main__.py", line 107, in <module>
    main()
  File "/Users/dhruv/Documents/code/transformers/src/transformers/onnx/__main__.py", line 100, in main
    validate_model_outputs(onnx_config, preprocessor, model, args.output, onnx_outputs, args.atol)
  File "/Users/dhruv/Documents/code/transformers/src/transformers/onnx/convert.py", line 405, in validate_model_outputs
    onnx_outputs = session.run(onnx_named_outputs, onnx_inputs)
  File "/Users/dhruv/Documents/code/transformers/.venv/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 200, in run
    return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Unexpected input data type. Actual: (tensor(float)) , expected: (tensor(int64))
(transformers) ➜  transformers git:(clip_onnx) ✗ xx
(transformers) ➜  transformers git:(clip_onnx) ✗ python -m transformers.onnx --model=openai/clip-vit-base-patch32 onnx/
vocab_file vocab.json
merges_file merges.txt
tokenizer_file tokenizer.json
added_tokens_file added_tokens.json
special_tokens_map_file special_tokens_map.json
tokenizer_config_file tokenizer_config.json
Using framework PyTorch: 1.13.0.dev20220806
/Users/dhruv/Documents/code/transformers/src/transformers/models/clip/modeling_clip.py:222: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attn_weights.size() != (bsz * self.num_heads, tgt_len, src_len):
/Users/dhruv/Documents/code/transformers/src/transformers/models/clip/modeling_clip.py:262: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attn_output.size() != (bsz * self.num_heads, tgt_len, self.head_dim):
/Users/dhruv/Documents/code/transformers/src/transformers/models/clip/modeling_clip.py:680: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
  mask.fill_(torch.tensor(torch.finfo(dtype).min))
/Users/dhruv/Documents/code/transformers/src/transformers/models/clip/modeling_clip.py:230: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if causal_attention_mask.size() != (bsz, 1, tgt_len, src_len):
/Users/dhruv/Documents/code/transformers/src/transformers/models/clip/modeling_clip.py:239: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attention_mask.size() != (bsz, 1, tgt_len, src_len):
/Users/dhruv/Documents/code/transformers/.venv/lib/python3.8/site-packages/torch/onnx/symbolic_opset9.py:4592: UserWarning: Exporting aten::index operator of advanced indexing in opset 14 is achieved by combination of multiple ONNX operators, including Reshape, Transpose, Concat, and Gather. If indices include negative values, the exported graph will produce incorrect results.
  warnings.warn(
Validating ONNX model...
        -[x] ONNX model output names {'last_hidden_state'} do not match reference model {'text_embeds', 'logits_per_image', 'text_model_output', 'logits_per_text', 'image_embeds', 'vision_model_output'}
Traceback (most recent call last):
  File "/Users/dhruv/.pyenv/versions/3.8.12/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/dhruv/.pyenv/versions/3.8.12/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/dhruv/Documents/code/transformers/src/transformers/onnx/__main__.py", line 107, in <module>
    main()
  File "/Users/dhruv/Documents/code/transformers/src/transformers/onnx/__main__.py", line 100, in main
    validate_model_outputs(onnx_config, preprocessor, model, args.output, onnx_outputs, args.atol)
  File "/Users/dhruv/Documents/code/transformers/src/transformers/onnx/convert.py", line 414, in validate_model_outputs
    raise ValueError(
ValueError: Outputs doesn't match between reference model and ONNX exported model: {'last_hidden_state'}

I'm guessing in the onnx config I have to define a separate function for outputs as well?

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[regisss]

@unography Yes, you have to define outputs the same way you did for inputs. Inputs are not defined in the parent class because they usually vary from one model to another, which is why it is mandatory to define them for each model. However, some default outputs are already defined depending on the tasks. For the default task, you can see here that it expects last_hiddent_state as the output, which is not returned by CLIP. So you can override this to specify the outputs you want.

Not sure though which outputs we would like to have here. Maybe text_embeds and image_embeds since this is basically feature extraction?

[unography]

@unography Yes, you have to define outputs the same way you did for inputs. Inputs are not defined in the parent class because they usually vary from one model to another, which is why it is mandatory to define them for each model. However, some default outputs are already defined depending on the tasks. For the default task, you can see here that it expects last_hiddent_state as the output, which is not returned by CLIP. So you can override this to specify the outputs you want.

Not sure though which outputs we would like to have here. Maybe text_embeds and image_embeds since this is basically feature extraction?

so if I define in the onnx config, say text_embeds and image_embeds, but the model is actually returning more outputs, like vision_model_output, will these extra outputs create any conflict or will it get handled by the onnxconfig automatically?

[regisss]

@unography Yes, you have to define outputs the same way you did for inputs. Inputs are not defined in the parent class because they usually vary from one model to another, which is why it is mandatory to define them for each model. However, some default outputs are already defined depending on the tasks. For the default task, you can see here that it expects last_hiddent_state as the output, which is not returned by CLIP. So you can override this to specify the outputs you want.
Not sure though which outputs we would like to have here. Maybe text_embeds and image_embeds since this is basically feature extraction?

so if I define in the onnx config, say text_embeds and image_embeds, but the model is actually returning more outputs, like vision_model_output, will these extra outputs create any conflict or will it get handled by the onnxconfig automatically?

Not sure about this. I think it should work the same as inputs, i.e. the order will matter and the inputs that are not specified in the config will just be skipped.

[unography]

@regisss sure, i'll try it out, thank you so much for your help!

[unography]

how do I make the test cases pass? Is it only formatting issues or something else?

[regisss]

I took a deeper look @unography! There are a few things to correct regarding imports and type hints to make tests pass. I also have comments about the changes to the forward method of the model.

Besides, which version of black have you used to format the code? Did you use the command make style to do it? Because there are many files unrelated to this PR that have been reformatted and this should not have happened.

[regisss]

The cast is only needed for the argmax, so it would be better to do it in the same call.

[regisss]

I prefer not modifying the source code like this. Why did you need to do that?

[unography]

I can revert this, this was from the original repo
openai/CLIP@1937532

[regisss]

Ah okay I see. Yes I would prefer to keep it as it is.

[regisss]

TensorType and ProcessorMixin are only needed for type hinting. The way we manage imports in Transformers will return an error with this implementation. That is why some tests failed :)

To solve this, you need to put these two imports in a TYPE_CHECKING conditional statement, here is an example.

Also, it's better to use relative imports because absolute imports can lead to weird errors.

I made the exact same mistakes in the PR of LayoutLMv3 haha

[unography]

I took a deeper look @unography! There are a few things to correct regarding imports and type hints to make tests pass. I also have comments about the changes to the forward method of the model.

Besides, which version of black have you used to format the code? Did you use the command make style to do it? Because there are many files unrelated to this PR that have been reformatted and this should not have happened.

my black version is 22.6.0, I think I did make fix-copies but tests were still failing so I did black . from inside the CLIP folder.
what is the correct way to fix this?

[regisss]

Try with black 22.3, and use the command make style from the root of the repo to format the code.

[regisss]

You should use a forward reference here

[regisss]

You should use a forward reference here

[unography]

by forward reference do you mean making it like processor: "ProcessorMixin"?

[regisss]

Yes exactly

[unography]

@regisss I made the changes, apart from the code formatting. How do I format my code correctly? And do I need to run make fix-copies ?

[regisss]

@regisss I made the changes, apart from the code formatting. How do I format my code correctly? And do I need to run make fix-copies ?

I don't think make fix-copies is necessary anymore because you already updated the doc.

[regisss]

@unography Not sure why modeling_groupvit.py is still in the changes.

Also, can you make sure that the test pytest tests/onnx/test_onnx_v2.py -v -k "clip" pass?

[unography]

@regisss .T is working for me while using pytorch's nightly release, but it fails on pytorch 1.12.1, the stable version.
I've made the change to make it .t(), this is working in the nightly release version as well

[regisss]

Thanks @unography, it looks good to me!

Looking at the failed tests, it seems you need to run make fix-copies one more time. Could you do it please?

[unography]

@regisss ah, my mistake. pushed. there are now additional changes to owlvit, groupvit and vision_text_dual_encoder, I'm assuming we copy over code to these files from the actual CLIP model?

[regisss]

@regisss ah, my mistake. pushed. there are now additional changes to owlvit, groupvit and vision_text_dual_encoder, I'm assuming we copy over code to these files from the actual CLIP model?

Yes that is what happens. Actually you removed those changes because they were among all the formatting changes that you got the first time you ran black, and I told you to remove them, sorry. I did not pay attention to those.

It looks good to me @unography :)

[regisss]

Gently pinging @sgugger for approval

[sgugger]

Thanks for the PR! There is only a need to add comments when you do something that is surprising to the reader (like casting those input IDs to int). Changing the .T to .t() does not call for one IMO.

Left a suggestion and then you'll need to run make fix-copies.

[unography]

@sgugger sure, removed the comment and pushed, but some tests are failing right now, I can't understand why.

[sgugger]

This is a flaky test, don't worry.
Thanks again for your contribution!

[regisss]

Congrats @unography for this PR!

[unography]

Thanks @regisss for all the help!

[chainyo]

Huge contribution! That's awesome!

[NielsRogge]

Awesome work @unography - are you planning to add support for GroupViT and OWL-ViT as well?

[unography]

@NielsRogge sure, if they're open I'll add a draft PR for them and get started!