Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add TransformerLayer, TransformerBlock, C3TR modules #2333

Merged
merged 18 commits into from
Apr 1, 2021
Merged

Add TransformerLayer, TransformerBlock, C3TR modules #2333

Changes from 1 commit
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
4fa9326
yolotr
dingyiwei Feb 6, 2021
b479d24
transformer block
dingyiwei Feb 11, 2021
55a8f21
Merge branch 'master' into Transformer
dingyiwei Feb 11, 2021
9a1dee8
Remove bias in Transformer
dingyiwei Feb 27, 2021
d27066f
Remove C3T
dingyiwei Mar 1, 2021
3a2b403
Remove a deprecated class
dingyiwei Mar 1, 2021
e3bdf31
Merge branch 'master' into Transformer
dingyiwei Mar 2, 2021
258f6b0
put the 2nd LayerNorm into the 2nd residual block
Mar 6, 2021
85297a1
merge ultralytics:master
glenn-jocher Mar 29, 2021
2b0fdc1
move example model to models/hub, rename to -transformer
glenn-jocher Mar 29, 2021
33e17af
Add module comments and TODOs
glenn-jocher Mar 29, 2021
23c9ee3
Remove LN in Transformer
dingyiwei Mar 31, 2021
a5b8346
Add comments for Transformer
dingyiwei Mar 31, 2021
e6e5f0e
Solve the problem of MA with DDP
dingyiwei Mar 31, 2021
feb4e76
cleanup
glenn-jocher Apr 1, 2021
d38b59b
cleanup find_unused_parameters
glenn-jocher Apr 1, 2021
4d58290
merge master
glenn-jocher Apr 1, 2021
5fd5b86
PEP8 reformat
glenn-jocher Apr 1, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Remove bias in Transformer
  • Loading branch information
@dingyiwei
dingyiwei committed Feb 27, 2021
commit 9a1dee87f0f1b8215aa29d0b36f9f6f2493c819d
13 changes: 6 additions & 7 deletions models/common.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,20 +83,19 @@ def __init__(self, c, num_heads):
super().__init__()

self.ln1 = nn.LayerNorm(c)
self.q = nn.Linear(c, c)
self.k = nn.Linear(c, c)
self.v = nn.Linear(c, c)
self.q = nn.Linear(c, c, bias=False)
self.k = nn.Linear(c, c, bias=False)
self.v = nn.Linear(c, c, bias=False)
self.ma = nn.MultiheadAttention(embed_dim=c, num_heads=num_heads)
self.ln2 = nn.LayerNorm(c)
self.fc1 = nn.Linear(c, c)
self.fc2 = nn.Linear(c, c)
self.act = nn.SiLU()
self.fc1 = nn.Linear(c, c, bias=False)
self.fc2 = nn.Linear(c, c, bias=False)

def forward(self, x):
x_ = self.ln1(x)
x = self.ma(self.q(x_), self.k(x_), self.v(x_))[0] + x
x = self.ln2(x)
x = self.fc2(self.act(self.fc1(x))) + x
x = self.fc2(self.fc1(x)) + x
return x


Expand Down