Skip to content
This repository has been archived by the owner on Mar 19, 2024. It is now read-only.

Vit deit #195

Closed
wants to merge 19 commits into from
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
linting
  • Loading branch information
growlix committed Feb 20, 2021
commit c881d4d581d2e8b07affb7123fe90f8814f60dfb
60 changes: 38 additions & 22 deletions vissl/data/collators/cutmixup_collator.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
Mixup and Cutmix
Papers:
mixup: Beyond Empirical Risk Minimization (https://arxiv.org/abs/1710.09412)
CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features (https://arxiv.org/abs/1905.04899)
CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features (https://arxiv.org/abs/1905.04899) # NOQA
Code Reference:
CutMix: https://github.com/clovaai/CutMix-PyTorch
Hacked together by / Copyright 2020 Ross Wightman
Expand Down Expand Up @@ -48,15 +48,19 @@ def cutmixup_collator(batch, **kwargs):
implementation (link when publicly available).

kwargs:
mixup_alpha (float): mixup alpha value, mixup is active if > 0.
cutmix_alpha (float): cutmix alpha value, cutmix is active if > 0.
cutmix_minmax (List[float]): cutmix min/max image ratio, cutmix is active and uses this vs alpha if not None.
prob (float): probability of applying mixup or cutmix per batch or element
switch_prob (float): probability of switching to cutmix instead of mixup when both are active
mode (str): how to apply mixup/cutmix params (per 'batch', 'pair' (pair of elements), 'elem' (element)
correct_lam (bool): apply lambda correction when cutmix bbox clipped by image borders
label_smoothing (float): apply label smoothing to the mixed target tensor
num_classes (int): number of classes for target
:mixup_alpha (float): mixup alpha value, mixup is active if > 0.
:cutmix_alpha (float): cutmix alpha value, cutmix is active if > 0.
:cutmix_minmax (List[float]): cutmix min/max image ratio, cutmix is active
and uses this vs alpha if not None.
:prob (float): probability of applying mixup or cutmix per batch or element
:switch_prob (float): probability of switching to cutmix instead of mixup
when both are active
:mode (str): how to apply mixup/cutmix params (per 'batch', 'pair' (pair of
elements), 'elem' (element)
:correct_lam (bool): apply lambda correction when cutmix bbox clipped by
image borders
:label_smoothing (float): apply label smoothing to the mixed target tensor
:num_classes (int): number of classes for target


The collators collates the batch for the following input (assuming k-copies of image):
Expand All @@ -71,7 +75,8 @@ def cutmixup_collator(batch, **kwargs):

Returns: Example output:
output = {
"data": torch.tensor([img1_0, ..., imgN_0], [img1_k, ..., imgN_k]) ..
"data": torch.tensor([img1_0, ..., imgN_0],
[img1_k, ..., imgN_k]) ..
}
"""
assert "data" in batch[0], "data not found in sample"
Expand Down Expand Up @@ -250,7 +255,8 @@ def rand_bbox(img_shape, lam, margin=0.0, count=None):
Args:
img_shape (tuple): Image shape as tuple
lam (float): Cutmix lambda value
margin (float): Percentage of bbox dimension to enforce as margin (reduce amount of box outside image)
margin (float): Percentage of bbox dimension to enforce as margin
(reduce amount of box outside image)
count (int): Number of bbox to generate
"""
ratio = np.sqrt(1 - lam)
Expand All @@ -270,10 +276,12 @@ def rand_bbox_minmax(img_shape, minmax, count=None):
"""Min-Max CutMix bounding-box
Inspired by Darknet cutmix impl, generates a random rectangular bbox
based on min/max percent values applied to each dimension of the input image.
Typical defaults for minmax are usually in the .2-.3 for min and .8-.9 range for max.
Typical defaults for minmax are usually in the .2-.3 for min and .8-.9
range for max.
Args:
img_shape (tuple): Image shape as tuple
minmax (tuple or list): Min and max bbox ratios (as percent of image size)
minmax (tuple or list): Min and max bbox ratios (as percent of image
size)
count (int): Number of bbox to generate
"""
assert len(minmax) == 2
Expand Down Expand Up @@ -310,12 +318,18 @@ class Mixup:
Args:
mixup_alpha (float): mixup alpha value, mixup is active if > 0.
cutmix_alpha (float): cutmix alpha value, cutmix is active if > 0.
cutmix_minmax (List[float]): cutmix min/max image ratio, cutmix is active and uses this vs alpha if not None.
prob (float): probability of applying mixup or cutmix per batch or element
switch_prob (float): probability of switching to cutmix instead of mixup when both are active
mode (str): how to apply mixup/cutmix params (per 'batch', 'pair' (pair of elements), 'elem' (element)
correct_lam (bool): apply lambda correction when cutmix bbox clipped by image borders
label_smoothing (float): apply label smoothing to the mixed target tensor
cutmix_minmax (List[float]): cutmix min/max image ratio, cutmix is
active and uses this vs alpha if not None.
prob (float): probability of applying mixup or cutmix per batch or
element
switch_prob (float): probability of switching to cutmix instead of
mixup when both are active
mode (str): how to apply mixup/cutmix params (per 'batch', 'pair' (pair
of elements), 'elem' (element)
correct_lam (bool): apply lambda correction when cutmix bbox clipped by
image borders
label_smoothing (float): apply label smoothing to the mixed target
tensor
num_classes (int): number of classes for target
"""

Expand Down Expand Up @@ -375,7 +389,8 @@ def _params_per_elem(self, batch_size):
else:
assert (
AssertionError
), "One of mixup_alpha > 0., cutmix_alpha > 0., cutmix_minmax not None should be true."
), "One of mixup_alpha > 0., cutmix_alpha > 0.," \
"cutmix_minmax not None should be true."
lam = np.where(
np.random.rand(batch_size) < self.mix_prob,
lam_mix.astype(np.float32),
Expand All @@ -402,7 +417,8 @@ def _params_per_batch(self):
else:
assert (
AssertionError
), "One of mixup_alpha > 0., cutmix_alpha > 0., cutmix_minmax not None should be true."
), "One of mixup_alpha > 0., cutmix_alpha > 0.," \
"cutmix_minmax not None should be true."
lam = float(lam_mix)
return lam, use_cutmix

Expand Down
6 changes: 4 additions & 2 deletions vissl/data/ssl_transforms/rand_auto_aug.py
Original file line number Diff line number Diff line change
Expand Up @@ -615,8 +615,10 @@ def auto_augment_policy(name="v0", hparams=None):
def auto_augment_transform(config_str, hparams):
"""
Create a AutoAugment transform
:param config_str: String defining configuration of auto augmentation. Consists of multiple sections separated by
dashes ('-'). The first section defines the AutoAugment policy (one of 'v0', 'v0r', 'original', 'originalr').
:param config_str: String defining configuration of auto augmentation.
Consists of multiple sections separated by dashes ('-'). The first
section defines the AutoAugment policy (one of 'v0', 'v0r', 'original',
'originalr').
The remaining sections, not order sepecific determine
'mstd' - float std deviation of magnitude noise applied
Ex 'original-mstd0.5' results in AutoAugment with original policy, magnitude_std 0.5
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ def _create_loss_function(self):
logging.info(
"Instantiating "
"CrossEntropyMultipleOutputSingleTargetLoss, which"
"internally uses SmoothCrossEntropy loss to accomodate"
"internally uses SmoothCrossEntropy loss to accommodate"
"label smoothing, but defaults to vanilla cross-entropy "
"if provided single-target labels."
)
Expand Down
2 changes: 1 addition & 1 deletion vissl/models/model_helpers.py
Original file line number Diff line number Diff line change
Expand Up @@ -421,7 +421,7 @@ def lecun_normal_init(tensor, fan_in):
# Matthew # Leavitt (ito@fb.com, matthew.l.leavitt@gmail.com) and Vedanuj
# Goswami (vedanuj@fb.com).
# trunc_normal_ and _no_grad_trunc_normal_ from:
# https://github.com/rwightman/pytorch-image-models/blob/678ba4e0a2c0b52c5e7b2ec0ba689399840282ee/timm/models/layers/weight_init.py # #NOQA
# https://github.com/rwightman/pytorch-image-models/blob/678ba4e0a2c0b52c5e7b2ec0ba689399840282ee/timm/models/layers/weight_init.py # NOQA
def trunc_normal_(tensor, mean=0.0, std=1.0, a=-2.0, b=2.0):
r"""Supposedly should be available in PyTorch soon. Replace when available.
Fills the input Tensor with values drawn
Expand Down
2 changes: 1 addition & 1 deletion vissl/utils/checkpoint.py
Original file line number Diff line number Diff line change
Expand Up @@ -445,7 +445,7 @@ def init_model_from_weights(
and config.MODEL.FEATURE_EVAL_SETTINGS.EVAL_TRUNK_AND_HEAD
)
):
# Accomodate changing position embeddings. Fine-tuning at a
# Accommodate changing position embeddings. Fine-tuning at a
# different resolution than that which a model was pretrained
# at requires interpolating the learned position embeddings.
if "pos_embedding" in layername:
Expand Down