🌟 BigBird

[astariul]

🌟 New model addition

Model description

Paper : https://arxiv.org/pdf/2007.14062.pdf

Abstract :

Transformers-based models, such as BERT, have been one of the most successful deep learning
models for NLP. Unfortunately, one of their core limitations is the quadratic dependency
(mainly in terms of memory) on the sequence length due to their full attention mechanism.
To remedy this, we propose, BigBird, a sparse attention mechanism that reduces this
quadratic dependency to linear. We show that BigBird is a universal approximator of
sequence functions and is Turing complete, thereby preserving these properties of the
quadratic, full attention model. Along the way, our theoretical analysis reveals some of the
benefits of having O(1) global tokens (such as CLS), that attend to the entire sequence
as part of the sparse attention mechanism. The proposed sparse attention can handle
sequences of length up to 8x of what was previously possible using similar hardware. As
a consequence of the capability to handle longer context, BigBird drastically improves
performance on various NLP tasks such as question answering and summarization. We also
propose novel applications to genomics data.

Open source status

• the model implementation is available: No
• the model weights are available: No
• who are the authors: ?

[sathvikask0]

When will be getting this model?

[patrickvonplaten]

Until the weights and code are not published I think we won't focus too much on adding the model

[sathvikask0]

I am planning to start a small tight group of individuals who will work on implementing research papers for proper business use cases.
Please let me know if anyone is interested for the same.
Project 1 : BigBert for Genomics Research

[nikhilbyte]

I am planning to start a small tight group of individuals who will work on implementing research papers for proper business use cases.
Please let me know if anyone is interested for the same.
Project 1 : BigBert for Genomics Research

I'll be up for this project

[seduerr91]

I'll be up for this project too. I got a slightly different use case idea, tho. :)

[ptynecki]

@sathvikask0
I am super interesting about the BigBird for Genomics Research. Are you planning to release the fixed-length embedding part as well?

[nikhilbyte]

I'm also doing some research on using Google BigBird for genomics research. There's a competition going on right now and we can definitely leverage BigBird for genomics sequencing.

[ptynecki]

@sathvikask0 @nikhilbyte @seduerr91
What if we could meet together and talk about the BigBert implementation for Genomics Research?

[seduerr91]

Sure do you want to set up a google meet?

[nikhilbyte]

I'm in.

[sathvikask0]

Hello @nikhilbyte @seduerr91 @ptynecki are we still doing this, I want to be a part of it!

[nikhilbyte]

Hello @nikhilbyte @seduerr91 @ptynecki are we still doing this, I want to be a part of it!

I'm up for this. Let me know how to connect with you.

[marcoabrate]

@patrickvonplaten actually you can read on the paper (appendix E, section E.4) that for summarization, "For the large size model, we lift weight from the state-of-the-art Pegasus model [107], which is pretrained using an objective designed for summarization task". Do you think it would be possible to include the new architecture, using the weights already available of google/pegasus-large?

[patrickvonplaten]

Is there an official code base by now?

[patrickvonplaten]

As soon as weights and codebase is out, we'll integrate! But it does not make much sense IMO to do it before that

[tbrittoborges]

I am planning to start a small tight group of individuals who will work on implementing research papers for proper business use cases.
Please let me know if anyone is interested for the same.
Project 1 : BigBert for Genomics Research

I would like to join the effort as well

[sachinrajoria]

It seems BigBird official code and pretrained models are finally out (well partially). The code seems to be written for TPUs mainly so not sure how easy to port to huggingface. Also I see a keras based BigBird implementation as part of Tensorflow official models, which might be easier to port. So let's start working on it!

[patrickvonplaten]

will try to allocate some time next week to start porting the model :-)

[gsrilathatata]

Can you please add me to this group, I would also like to work on this project.

[isollid]

@patrickvonplaten, do you know when it will be ready? 🐦

[LoganHollmer]

Any update?

[Cerebrock]

Has there been any progress on this? :)

[cmcapellan]

@patrickvonplaten I see #10183 is passing all its checks, is it close to being able to merge? Looking forward to using with my project!

[thevasudevgupta]

Hi, it will be merged by next week.

[ZrW00]

Is this model available before this weekend?

[LysandreJik]

@DarthAnakin BidBird is available as of this morning on the master branch and will be in the next release

[ZrW00]

@LysandreJik Thanks!

[cmcapellan]

@LysandreJik very excited to see this complete. When will the next release happen?

[LysandreJik]

We expect to do it early next week!

[tanmaylaud]

Any plans to add a Fast Tokenizer for this model ?
I would be happy to help integrate it.
@patrickvonplaten

[LysandreJik]

@tanmaylaud we would welcome an effort to add a fast tokenizer for this model!

[slvcsl]

Thanks a lot! Is any example script available at the moment? I'm particularly looking for summarization.

[lefnire]

Also looking for summarization support. Seems it needs a Pegasus decoder to work. I see a few such BigBird->Pegasus models at https://huggingface.co/vasudevgupta, following from discussions at #10991

[patrickvonplaten]

@vasudevgupta7 is working very hard on it to merge it soon :-) Think, it should be ready in ~2 weeks the latest

[thevasudevgupta]

Hi all, BigBird pegasus is available in master of 🤗Transformers now. Give it a try ...