Skip to content
This repository has been archived by the owner on Dec 16, 2022. It is now read-only.

Add custom_dummy_tokens to PretrainedTransformerTokenizer #5608

Merged
merged 7 commits into from
Apr 8, 2022

Conversation

tarohi24
Copy link
Contributor

@tarohi24 tarohi24 commented Mar 26, 2022

This commit aims to support #5597

Changes proposed in this pull request:

  • Add custom_dummy_tokens argument to PretrainedTransformerTokenizer. Without that argument, initializing PretrainedTransformerTokenizer of some non-English models fail.

Before submitting

  • I've read and followed all steps in the Making a pull request
    section of the CONTRIBUTING docs.
  • I've updated or added any relevant docstrings following the syntax described in the
    Writing docstrings section of the CONTRIBUTING docs.
  • If this PR fixes a bug, I've added a test that will fail without my fix.
  • If this PR adds a new feature, I've added tests that sufficiently cover my new functionality.

After submitting

  • All GitHub Actions jobs for my pull request have passed.
  • codecov/patch reports high test coverage (at least 90%).
    You can find this under the "Actions" tab of the pull request once the other checks have finished.

Copy link
Member

@epwalsh epwalsh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @tarohi24! This looks good, I just have two small suggestions

CHANGELOG.md Outdated Show resolved Hide resolved
tarohi24 and others added 2 commits April 2, 2022 08:33
Co-authored-by: Pete <epwalsh10@gmail.com>
@tarohi24 tarohi24 marked this pull request as draft April 1, 2022 23:34
@tarohi24
Copy link
Contributor Author

tarohi24 commented Apr 1, 2022

Thanks, @epwalsh for reviewing! The suggestions look good to me. I made commits to reflect these suggestions.

@tarohi24 tarohi24 marked this pull request as ready for review April 1, 2022 23:45
Copy link
Member

@epwalsh epwalsh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks @tarohi24!

@epwalsh
Copy link
Member

epwalsh commented Apr 6, 2022

Looks like we'll need to resolve #5613 before merging this.

@dirkgr dirkgr enabled auto-merge (squash) April 8, 2022 00:16
@dirkgr dirkgr merged commit 1be8855 into allenai:main Apr 8, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants