Add BigBirdPegasus #10991

thevasudevgupta · 2021-03-31T11:49:26Z

What does this PR do?

This PR will add Google's BigBird-Pegasus. Extending #10183

Following checkpoints will be added:

It is verified that uploaded models work correctly, see:

BigBird Pegasus Arxiv: https://colab.research.google.com/drive/1ntBBkiDgccbKwKmOECB8VWEFeFmZebLN?usp=sharing
BigBird Pegasus BigPatent: https://colab.research.google.com/drive/1RKI0BG3JUy4Hn8VdIzNLE5QduwtaiXYZ?usp=sharing
BigBird Pegasus Pubmed: https://colab.research.google.com/drive/1LebnFVp4unqZWRx5gez1hVyqR9cibIoH?usp=sharing

Here a notebook showing how well BigBirdPegasus works on long-document summarization: https://colab.research.google.com/github/vasudevgupta7/bigbird/blob/main/notebooks/bigbird_pegasus_evaluation.ipynb

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline, Pull Request section?
Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the documentation guidelines, and here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

@patrickvonplaten

thevasudevgupta · 2021-04-10T07:24:25Z

For running conversion script for BigBirdPegasus:

python3 src/transformers/models/bigbird_pegasus/convert_bigbird_pegasus_tf_to_pytorch.py --tf_ckpt_path src/tf_ckpt/bigbird-pegasus-large-arxiv/model.ckpt-300000 --save_dir src/google/bigbird-pegasus-large-arxiv

For running conversion script for bigbird-roberta EncoderDecoderModel:

python3 src/transformers/models/bigbird_pegasus/convert_bigbird_roberta_tf_to_pytorch.py --tf_ckpt_path src/tf_ckpt/bigbird-roberta-arxiv/model.ckpt-300000 --save_dir src/google/bigbird-roberta-arxiv

tests/test_modeling_bigbird_pegasus.py

src/transformers/__init__.py

src/transformers/models/bigbird_pegasus/modeling_bigbird_pegasus.py

…7/transformers into add_bigbird_pegasus

patrickvonplaten

Amazing contribution @vasudevgupta7 !

patil-suraj

Amazing work @vasudevgupta7 !

Just left a few nits.

src/transformers/models/bigbird_pegasus/__init__.py

src/transformers/models/bigbird_pegasus/modeling_bigbird_pegasus.py

tests/test_modeling_bigbird_pegasus.py

sgugger

Great work! Thanks a lot for this addition. I left a few nits.

src/transformers/convert_slow_tokenizer.py

src/transformers/models/bigbird_pegasus/convert_bigbird_pegasus_tf_to_pytorch.py

src/transformers/models/bigbird_pegasus/modeling_bigbird_pegasus.py

src/transformers/models/pegasus/tokenization_pegasus.py

tests/test_modeling_bigbird_pegasus.py

LysandreJik

Great job @vasudevgupta7! Really cool to have BigBirdPegasus in the library as well.

Nice notebook too. Are you planning a notebook like last time as well?

src/transformers/models/pegasus/tokenization_pegasus.py

src/transformers/models/bigbird_pegasus/modeling_bigbird_pegasus.py

Co-authored-by: Suraj Patil <surajp815@gmail.com> Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

thevasudevgupta · 2021-05-06T23:53:33Z

@LysandreJik, yes we are planning to add this notebook with a few modifications.

thevasudevgupta · 2021-05-07T00:19:32Z

@patrickvonplaten, Test failing on CircleCi: tests/test_modeling_bigbird_pegasus.py::BigBirdPegasusStandaloneDecoderModelTest::test_decoder_model_attn_mask_past is passing for me locally.

Everything else is fixed!!

* init bigbird pegasus * add debugging nb ; update config * init conversion * update conversion script * complete conversion script * init forward() * complete forward() * add tokenizer * add some slow tests * commit current * fix copies * add docs * add conversion script for bigbird-roberta-summarization * remove TODO * small fixups * correct tokenizer * add bigbird core for now * fix config * fix more * revert pegasus-tokenizer back * make style * everything working for pubmed; yayygit status * complete tests finally * remove bigbird pegasus tok * correct tokenizer * correct tests * add tokenizer files * finish make style * fix test * update * make style * fix tok utils base file * make fix-copies * clean a bit * small update * fix some suggestions * add to readme * fix a bit, clean tests * fix more tests * Update src/transformers/__init__.py * Update src/transformers/__init__.py * make fix-copies * complete attn switching, auto-padding left * make style * fix auto-padding test * make style * fix batched attention tests * put tolerance at 1e-1 for stand-alone decoder test * fix docs * fix tests * correct slow tokenizer conversion * Apply suggestions from code review Co-authored-by: Suraj Patil <surajp815@gmail.com> Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> * complete remaining suggestions * fix test Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> Co-authored-by: Suraj Patil <surajp815@gmail.com> Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

thevasudevgupta added 6 commits March 30, 2021 19:15

init bigbird pegasus

84b8ab3

add debugging nb ; update config

5c0307b

init conversion

ee47797

update conversion script

677b72a

complete conversion script

94861d6

init forward()

7529516

thevasudevgupta mentioned this pull request Apr 7, 2021

BigBird #10183

Merged

5 tasks

complete forward()

2ce502b

thevasudevgupta added 8 commits April 10, 2021 14:30

add tokenizer

72807a7

add some slow tests

0f00e43

commit current

3bee313

fix copies

2f72512

add docs

af47e3c

add conversion script for bigbird-roberta-summarization

bbe710f

remove TODO

7b1ecce

small fixups

1de6954

patrickvonplaten reviewed Apr 19, 2021

View reviewed changes

tests/test_modeling_bigbird_pegasus.py Show resolved Hide resolved

correct tokenizer

fb10be0

lefnire mentioned this pull request Apr 20, 2021

🌟 BigBird #6113

Closed

3 tasks

thevasudevgupta added 7 commits April 23, 2021 02:45

add bigbird core for now

c78bf6f

fix config

38c7638

fix more

0b87a70

revert pegasus-tokenizer back

c403796

make style

30a586a

everything working for pubmed; yayygit status

9618f8b

complete tests finally

53d99b5

patrickvonplaten reviewed May 3, 2021

View reviewed changes

tests/test_modeling_bigbird_pegasus.py Show resolved Hide resolved

patrickvonplaten reviewed May 3, 2021

View reviewed changes

tests/test_modeling_bigbird_pegasus.py Show resolved Hide resolved

patrickvonplaten reviewed May 3, 2021

View reviewed changes

tests/test_modeling_bigbird_pegasus.py Show resolved Hide resolved

Update src/transformers/__init__.py

a066db7

patrickvonplaten reviewed May 6, 2021

View reviewed changes

src/transformers/__init__.py Outdated Show resolved Hide resolved

patrickvonplaten and others added 3 commits May 6, 2021 10:02

Update src/transformers/__init__.py

9c0b312

make fix-copies

4d706f7

complete attn switching, auto-padding left

9a52751

patrickvonplaten reviewed May 6, 2021

View reviewed changes

src/transformers/models/bigbird_pegasus/modeling_bigbird_pegasus.py Show resolved Hide resolved

patrickvonplaten and others added 8 commits May 6, 2021 09:02

make style

b13e76a

fix auto-padding test

97e1f71

make style

0ba203a

fix batched attention tests

0a85bb2

Merge branch 'add_bigbird_pegasus' of https://github.com/vasudevgupta…

89e9d5c

…7/transformers into add_bigbird_pegasus

put tolerance at 1e-1 for stand-alone decoder test

7d2cae9

fix docs

34c3a63

fix tests

c229de1

patrickvonplaten changed the title ~~[WIP] Add BigBirdPegasus~~ Add BigBirdPegasus May 6, 2021

patrickvonplaten requested review from sgugger and LysandreJik May 6, 2021 11:50

patrickvonplaten approved these changes May 6, 2021

View reviewed changes

patrickvonplaten requested a review from patil-suraj May 6, 2021 11:57

patil-suraj approved these changes May 6, 2021

View reviewed changes

sgugger approved these changes May 6, 2021

View reviewed changes

LysandreJik approved these changes May 6, 2021

View reviewed changes

src/transformers/models/pegasus/tokenization_pegasus.py Show resolved Hide resolved

src/transformers/models/pegasus/tokenization_pegasus.py Show resolved Hide resolved

src/transformers/models/bigbird_pegasus/modeling_bigbird_pegasus.py Show resolved Hide resolved

patrickvonplaten and others added 2 commits May 6, 2021 16:44

correct slow tokenizer conversion

1d8d60d

Apply suggestions from code review

dfb74ea

Co-authored-by: Suraj Patil <surajp815@gmail.com> Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

complete remaining suggestions

ca77ad3

fix test

2eb7563

patrickvonplaten merged commit dc3f675 into huggingface:master May 7, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add BigBirdPegasus #10991

Add BigBirdPegasus #10991

thevasudevgupta commented Mar 31, 2021 •

edited

Loading

thevasudevgupta commented Apr 10, 2021 •

edited

Loading

patrickvonplaten left a comment

patil-suraj left a comment

sgugger left a comment

LysandreJik left a comment

thevasudevgupta commented May 6, 2021 •

edited

Loading

thevasudevgupta commented May 7, 2021

Add BigBirdPegasus #10991

Add BigBirdPegasus #10991

Conversation

thevasudevgupta commented Mar 31, 2021 • edited Loading

What does this PR do?

Before submitting

Who can review?

thevasudevgupta commented Apr 10, 2021 • edited Loading

patrickvonplaten left a comment

Choose a reason for hiding this comment

patil-suraj left a comment

Choose a reason for hiding this comment

sgugger left a comment

Choose a reason for hiding this comment

LysandreJik left a comment

Choose a reason for hiding this comment

thevasudevgupta commented May 6, 2021 • edited Loading

thevasudevgupta commented May 7, 2021

thevasudevgupta commented Mar 31, 2021 •

edited

Loading

thevasudevgupta commented Apr 10, 2021 •

edited

Loading

thevasudevgupta commented May 6, 2021 •

edited

Loading