Matthew's Correlation Coefficient (MCC) always zero

[mirix]

Hello,

I am fine-tuning a multiclass audio classifier by using the template script with two GPUs:

https://github.com/huggingface/transformers/blob/main/examples/pytorch/audio-classification/README.md

I have seven imbalanced classes.

The model learns, in a few epochs the loss drops from ca. 2 to ca. 1 and F1 weighted goes from 0.26 to ca. 0.56.

Accuracy values are initially higher than F1 (it starts at 0.46), but also improve significantly during training.

However, Matthew's Correlation Coefficient (MCC) stays always at 0.0 during the training.

I have changed the command line to:

--metric_for_best_model matthews_correlation \

And the script to:

metric = evaluate.load("matthews_correlation") 

...

    def compute_metrics(eval_pred):
        """Computes accuracy on a batch of predictions"""
        predictions = np.argmax(eval_pred.predictions, axis=1)
        return metric.compute(predictions=predictions, references=eval_pred.label_ids, average='macro')

The average argument makes no difference, I have tried both macro and none (as well as removing it).

Am I doing anything wrong, is this behavior to be expected or is it a bug?

Best,

Ed

[mirix]

It was not zero. It was a problem with rounding. When there is only one metric, the trainer seems to report one decimal number (0.0), however, when I load several metrics and choose MCC as the one for evaluation, the trainer reports several decimal places for each metric.