You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I am experimenting with weighted custom loss functions for a binary classification problem but I am having trouble replicating the exact behaviour when using the default loss function in conjunction with sample weights. If there are no sample weights, then the custom loss function (same as in the advanced example notebook) works as expected and the training/validation loss as well as final predictions are exactly the same.
However, if I weight the samples (doesn't matter what weights - any weights), the training loss in the custom function becomes slightly different to that of the default loss function. The difference is very minor but I still do not understand why there is any difference at all. This looks like it might be a bug to me. It's not clear which is right and which is wrong. Intriguingly, the final predictions however are still exactly the same.
I have looked at the C++ source code and I believe I have followed the same logic. When sample weights are provided, the gradient and the hessian are simply multiplied by the sample weights before being returned - that is the only difference between when sample weights are provided and when they are not. This is exactly what I'm doing.
Please let me know if my loss function is wrong or if this is a bug. Thanks
Simply run the above code in python. This will output the training and validation losses as well as the first 20 predictions for the model which uses the default loss function and the model which uses the custom loss function. The printed training loss numbers will be different but the final predictions will nevertheless be the same
However if you now comment out the line which sets the weights (train.set_weight(weight=weights)), the training losses for the two models will now be exactly the same.
The text was updated successfully, but these errors were encountered:
Thanks very much Guolin, I thought since I am not passing weights to the validation set, the logloss_eval function doesn't matter but I now realise it's used to compute the metrics for the training set as well.
In case anyone else comes across this, the following function for training metrics on training and validation sets works:
I am experimenting with weighted custom loss functions for a binary classification problem but I am having trouble replicating the exact behaviour when using the default loss function in conjunction with sample weights. If there are no sample weights, then the custom loss function (same as in the advanced example notebook) works as expected and the training/validation loss as well as final predictions are exactly the same.
However, if I weight the samples (doesn't matter what weights - any weights), the training loss in the custom function becomes slightly different to that of the default loss function. The difference is very minor but I still do not understand why there is any difference at all. This looks like it might be a bug to me. It's not clear which is right and which is wrong. Intriguingly, the final predictions however are still exactly the same.
I have looked at the C++ source code and I believe I have followed the same logic. When sample weights are provided, the gradient and the hessian are simply multiplied by the sample weights before being returned - that is the only difference between when sample weights are provided and when they are not. This is exactly what I'm doing.
Please let me know if my loss function is wrong or if this is a bug. Thanks
Environment info
Operating System: Ubuntu 18.04.4 LTS
CPU/GPU model: Intel Core i7-8565U CPU
C++/Python/R version: Python 3.7.6
LightGBM version or commit hash: 2.3.0
Error message
No error message.
Reproducible examples
Steps to reproduce
Simply run the above code in python. This will output the training and validation losses as well as the first 20 predictions for the model which uses the default loss function and the model which uses the custom loss function. The printed training loss numbers will be different but the final predictions will nevertheless be the same
However if you now comment out the line which sets the weights (
train.set_weight(weight=weights)
), the training losses for the two models will now be exactly the same.The text was updated successfully, but these errors were encountered: