Bug in TinyBert data augmentation?

[gowtham1997]

Hello,

Pretrained-Language-Model/TinyBERT/data_augmentation.py

Lines 147 to 154 in 54ca698

	tokenized_text = self.tokenizer.tokenize(sent)
	tokenized_text = ['[CLS]'] + tokenized_text
	tokenized_len = len(tokenized_text)
	tokenized_text = word_pieces + ['[SEP]'] + tokenized_text[1:] + ['[SEP]']
	if len(tokenized_text) > 512:
	tokenized_text = tokenized_text[:512]

In line 154, the tokenized text is sliced to len <= 512, if it exceeds 512 tokens but the corresponding tokenized_len in line 149 is not updated.

The segment_ids in the subsequent lines seem to be using the un-updated tokenized_len and causing errors in the forward pass.

 File "/lusnlsas/paramsiddhi/iitm/vinodg/glue_data_generation/plm/TinyBERT/transformer/modeling.py", line 361, in forward
    embeddings = words_embeddings + position_embeddings + token_type_embeddings
RuntimeError: The size of tensor a (512) must match the size of tensor b (763) at non-singleton dimension 1

^ this bug occurs when I try to generate data augmentations using the bert-base-cased model.

[zwjyyc]

Thanks! We agree with your comment and will fix this bug. Your pull request is also welcome.

[gowtham1997]

@zwjyyc I've submitted a pull request for the same. Can you please help review this?