Bag of Tricks for Learning with Noisy Labels

This project is a collection of common tricks for learning with noisy labels in the field of machine learning. Noisy labels refer to labels that contain errors or inaccuracies, which can significantly affect the performance of machine learning models. The tricks included in this project aim to mitigate the negative impact of noisy labels and improve the robustness of machine learning models.

In this repo:

• We provide code for constructing noisy data, including symmetric, asymmetric, instance, and real noise.

• All experiments are conducted on the CIFAR dataset. The CIFAR dataset has 50,000 images, and the image size is 32x32, so it can be run on a single GPU.

The following is a brief introduction to the tricks used in this project:

• EMA: a technique for updating the weights of a teacher model by taking an exponential moving average of the current student's weights and the teacher's previous weights.
• Mix-up: a technique for combining two training examples and their labels to create a convex combination of the two examples and labels.
• Label smoothing: a regularization technique that replaces the hard 0/1 targets with soft targets.
• RandAugment: a data augmentation method that applies a series of random image transformations with different intensities.
• AutoAugment: a data augmentation method that uses a reinforcement learning algorithm to learn the best data augmentation policy for a given dataset.
• Regularization: include class balance regularization.

Experiment

The project uses ResNet + cross-entropy as baseline, and all tricks are evaluated on top of this baseline. This section provides a brief introduction, and detailed descriptions and experimental results can be found in the trick-gallery.md file in the Document directory.

We conducted extensive experiments on the CIFAR dataset, which is a popular benchmark dataset for image classification. The results were the mean and standard deviation after three repetitions and the noise types and corresponding noise rates of the constructed dataset can be found in the table below.

Dataset	CIFAR-10								CIFAR-100
Noise type	Symmetric			Asym.	Instance		Real		Symmetric			Asym.	Instance		Real
Noise rate	0.2	0.4	0.6	0.4	0.2	0.4	aggre	worst	0.2	0.4	0.6	0.4	0.2	0.4	noisy100

- Symmetric Noise is generated by uniformly flipping labels for a percentage of the training dataset for all classes, as well as Asymmetric Noise by flipping labels for particular pairs of classes

- Instance-dependent Noise is generated by image features. For more infomation, please refer to Part-dependent Label Noise: Towards Instance-dependent Label Noise

- Real Noise is a re-annotation of CIFAR-10/CIFAR-100 with human workers. For more infomation, please refer to Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations

- Noise Rate is defined as $\alpha = \frac{\text{Noise Samples}}{\text{Total Samples}}$.

Usage

In this section, we will show you how to run these tricks in detail.

cd bag-of-tricks

DS={cifar-10, cifar-100}; TYPE={symmetric, asymmetric, pairflip, aggre, worst, noisy100}; RATE :Noise Rate

• Baseline

  python main.py --dataset {DS} --noise_type {TYPE} --noise_rate {RATE}

• PreResNet

  python main.py --dataset {DS} --noise_type {TYPE} --noise_rate {RATE} --net 'PreResNet18'

• Mixup

  python main.py --dataset {DS} --noise_type {TYPE} --noise_rate {RATE} --alpha 1

• Label Smoothing

  python main.py --dataset {DS} --noise_type {TYPE} --noise_rate {RATE} --label-smoothing 0.1

• Autoaug

  python main.py --dataset {DS} --noise_type {TYPE} --noise_rate {RATE} --aug_type 'autoaug'

• Randaug

  python main.py --dataset {DS} --noise_type {TYPE} --noise_rate {RATE} --aug_type 'randaug'

• Classbalance

  python main.py --dataset {DS} --noise_type {TYPE} --noise_rate {RATE} --class_balance 1

Results

Vallina is an ordinary model, and EMA is a student model obtained by exponential moving average. The experiment was repeated three times, and mean±std was taken as the final result.

CIFAR-10

Dataset		CIFAR-10
Noise type		Symmetric			Asym.	Instance		Real
Noise rate		0.2	0.4	0.6	0.4	0.2	0.4	aggre	worst
Baseline	Vanilla	86.82±0.12	82.58±0.50	75.13±0.21	85.49±0.75	86.14±0.21	76.25±0.64	90.40±0.13	79.53±0.26
	EMA	91.06±0.03	87.92±0.20	81.17±0.35	88.85±0.08	90.80±0.11	82.55±0.07	91.46±0.03	83.64±0.10
PreResNet	Vanilla	86.82±0.04	82.25±0.41	74.34±0.10	84.98±1.16	86.67±0.48	77.28±0.83	89.95±0.13	78.76±0.74
	EMA	90.92±0.11	87.67±0.27	80.39±0.26	89.42±0.33	90.72±0.10	80.94±0.43	91.33±0.09	83.60±0.17
Mixup	Vanilla	90.07±0.07	86.44±0.13	79.28±0.14	88.22±0.40	90.30±0.10	78.82±0.42	92.08±0.13	82.50±0.25
	EMA	92.69±0.13	89.98±0.00	84.13±0.05	90.68±0.05	92.97±0.10	83.28±0.28	93.29±0.09	85.54±0.10
Label Smoothing	Vanilla	87.37±0.23	83.09±0.57	76.04±0.43	86.15±0.29	86.70±0.29	76.94±0.82	90.67±0.05	79.52±0.29
	EMA	91.18±0.15	88.05±0.22	81.76±0.25	90.11±0.20	91.13±0.07	83.29±0.25	91.85±0.09	83.55±0.21
Autoaug	Vanilla	91.37±0.03	87.18±0.12	80.30±0.22	88.54±0.44	90.47±0.43	76.47±0.73	93.66±0.08	83.79±0.59
	EMA	93.26±0.13	90.43±0.16	84.60±0.12	90.70±0.06	92.57±0.09	78.75±0.30	94.08±0.04	86.41±0.06
Randaug	Vanilla	91.06±0.10	86.92±0.53	79.13±0.58	88.38±0.23	89.42±0.24	74.89±2.00	93.44±0.01	83.50±0.36
	EMA	93.18±0.03	90.21±0.15	83.68±0.14	90.41±0.18	92.09±0.12	76.10±0.19	93.96±0.05	85.97±0.10
Class Balance	Vanilla	86.61±0.55	82.71±0.16	75.54±0.88	86.90±0.18	87.20±0.38	81.53±0.31	90.64±0.10	79.59±0.34
	EMA	90.91±0.10	88.10±0.28	81.29±0.36	89.74±0.12	91.44±0.12	86.15±0.14	91.50±0.04	83.80±0.24

CIFAR-100

Dataset		CIFAR-100
Noise type		Symmetric			Asym.	Instance		Real
Noise rate		0.2	0.4	0.6	0.4	0.2	0.4	noisy100
baseline	Vanilla	63.94±0.39	52.39±0.63	41.20±0.25	45.60±0.17	62.54±0.50	44.48±0.15	54.55±0.13
	EMA	67.98±0.13	61.86±0.11	49.32±0.65	52.99±0.51	66.14±0.16	53.51±0.16	60.43±0.07
PreResNet	Vanilla	62.46±0.36	52.54±0.18	43.38±0.37	45.98±0.20	62.46±0.30	44.84±0.11	53.38±0.52
	EMA	67.91±0.06	62.43±0.26	52.26±0.24	55.12±0.16	66.11±0.09	53.15±0.18	60.24±0.15
Mixup	Vanilla	66.33±0.26	58.74±0.49	48.89±0.77	52.32±0.36	62.28±0.55	44.41±0.67	58.34±0.29
	EMA	71.08±0.31	64.86±0.08	55.23±0.40	58.59±0.15	67.75±0.07	51.10±0.13	63.17±0.22
Label Smoothing	Vanilla	65.81±0.54	53.61±0.32	41.86±0.71	47.91±0.09	62.82±0.36	41.62±0.44	55.71±0.44
	EMA	68.73±0.14	62.38±0.59	49.93±0.48	57.31±0.27	66.90±0.12	52.68±0.35	61.13±0.27
Autoaug	Vanilla	68.44±0.04	59.10±0.35	48.16±0.59	53.53±0.45	66.63±0.09	46.48±0.48	58.30±0.25
	EMA	71.80±0.17	66.03±0.17	55.74±0.38	60.30±0.29	68.03±0.04	48.78±0.13	63.49±0.09
Randaug	Vanilla	66.98±0.13	57.78±0.28	46.82±0.47	52.39±0.05	65.17±0.13	44.62±0.37	57.11±0.07
	EMA	71.11±0.08	64.67±0.21	53.95±0.35	59.45±0.29	66.88±0.08	47.47±0.45	62.98±0.23
Class Balance	Vanilla	64.15±0.16	52.74±0.59	41.03±0.35	47.87±0.33	65.12±0.20	53.42±0.13	55.28±0.15
	EMA	67.43±0.08	61.34±0.34	48.83±0.52	54.69±0.30	67.46±0.20	59.84±0.46	60.50±0.21

Related link

Noisy Labels Competition

Bag of Tricks for LongTailed Image Classification

Bag of Tricks for Image Classification with Convolutional Neural Networks

Contribution

We welcome contributions and suggestions for new tricks to be added to this project. Please feel free to submit a pull request or open an issue to discuss your ideas.