Fix mAP calculations

[comlhj1114]

Fix mAP calculations.
Originally, there were extrapolations to generate precision curve, recall curve, and PR curve.
The extrapolations generate large differenced with cocoapi.
This commit removes extrapolations to accurately calculate mAP, especially false-negative related mAP errors when conf-thres is high.

related issue : #1466

Dataset: coco128
Model: yolov5s.pt

Original
mAP@0.50 (conf-thres=0.001) : 0.719
mAP@0.50 (conf-thres=0.25) : 0.687

Fixed
mAP@0.50 (conf-thres=0.001) : 0.712
mAP@0.50 (conf-thres=0.25) : 0.594

Original Curves (conf-thres=0.25)

Fixed Curves (conf-thres=0.25)

🛠️ PR Summary

_{Made with ❤️ by Ultralytics Actions}

🌟 Summary

Improved precision-recall curve interpolation and AP calculation for object detection metrics.

📊 Key Changes

• Updated the interpolation method to specify right=0 for both recall and precision in ap_per_class function.
• Altered computation of sentinel values in compute_ap function for precision-recall curves.

🎯 Purpose & Impact

• Consistency in Border Handling: Adding right=0 ensures that extrapolation for values outside the data range is consistent, avoiding assumed default values.
• Better Precision-Recall Curves: Modifying the sentinel values leads to more accurate precision-recall curves, especially towards the limits where recall is high.
• Impact on Users: Users will experience more accurate Average Precision (AP) scores, which are crucial for evaluating object detection models, leading to improved trust in the metrics reported by YOLOv5 models.📈🔍

[github-actions]

👋 Hello @comlhj1114, thank you for submitting a YOLOv5 🚀 PR! To allow your work to be integrated as seamlessly as possible, we advise you to:

• ✅ Verify your PR is up-to-date with upstream/master. If your PR is behind upstream/master an automatic GitHub Actions merge may be attempted by writing /rebase in a new comment, or by running the following code, replacing 'feature' with the name of your local branch:

git remote add upstream https://github.com/ultralytics/yolov5.git
git fetch upstream
# git checkout feature  # <--- replace 'feature' with local branch name
git merge upstream/master
git push -u origin -f

• ✅ Verify all Continuous Integration (CI) checks are passing.
• ✅ Reduce changes to the absolute minimum required for your bug fix or feature addition. "It is not daily increase but daily decrease, hack away the unessential. The closer to the source, the less wastage there is." -Bruce Lee

[glenn-jocher]

@comlhj1114 I appreciate the effort here, but the existing implementation is in place because there is no such thing as a valid mAP calculation that does not extend all the way from near zero to 1.0 confidence. It's simply a user error to assume you can compute mAP metrics at anything other than near zero --conf.

[comlhj1114]

@glenn-jocher Thank you for your quick reply!
Even though there is no valid mAP calculation, I think yolov5 repo should follow cocoapi calculations.
Because in many other repos (e.g. SAHI) they gives compatibilities with yolov5 and they use cocoapi to caculate mAPs. So, different mAP calculations can generate confusions.

[glenn-jocher]

@comlhj1114 SAHI is a derivative work of YOLOv5, there's no reason for us to modify our behavior to theirs. If anything it should be the other way around.

[glenn-jocher]

@comlhj1114 also what's missing here is any before and after change results. This modification will clearly affect metrics, yet I see no testing to quantify the before and after of what you've proposed.

[glenn-jocher]

@comlhj1114 just to be perfectly clear, this thing you have written "mAP@0.50 (conf-thres=0.25) : 0.687" is not a valid metric. By definition it can not exist. It's like me saying "the temperature on May negative-3 is 25C."

It is impossible to compute mAP as anything other than the integral of the PR curve that extends from confidence=0 to confidence=1.

[comlhj1114]

@glenn-jocher
Thanks for your reply.
Then, what about using cocoapi's mAP cacluations instead of yolov5's mAP calucations?
Why did you make custom mAP calculations instead of using original coco's suggestion?
Is there any special reason?

[comlhj1114]

My PR intends to follow cocoapi's mAP calculations.
So better solution for me is just using cocoapi for mAP calculation.

[glenn-jocher]

@comlhj1114 pycocotools is already applied automatically to compute COCO mAP, and if you have a valid COCO-format JSON for a custom dataset then you can also supply it to val.py:

That being said, pycocotools is about 100 times slower than the YOLOv5 mAP code, so if you like waiting around a hundred times longer than you need to then use pycocotools as much as you'd like.

[comlhj1114]

@glenn-jocher Speed optimization is always right.
What I wanted is even with high conf-tresh, yolov5's mAP calculation should follow cocoapi's calculation.

[comlhj1114]

After download coco dataset, I'll test and share the results.

[glenn-jocher]

@comlhj1114 ok! You can also validate COCO using these two cells in the Colab notebook (after running Setup cell):
https://colab.research.google.com/github/ultralytics/yolov5/blob/master/tutorial.ipynb?hl=en

[comlhj1114]

@glenn-jocher Thank you for your support!

[zhiqwang]

Hi @comlhj1114 and @glenn-jocher , If you want to easily use coco128 as a test, you can use this https://github.com/zhiqwang/yolov5-rt-stack/releases/download/v0.3.0/coco128.zip , here I converted the yolov5 coco128 txt format to coco json for testing on cocoapi.

[comlhj1114]

@zhiqwang Thanks for your support :)
But, even after change some code in val.py, I couldnt utilize your dataset.. sorry
anno_json = str(Path(data.get('path', '../coco128')) / 'annotations/instances_train2017.json') # annotations json

[comlhj1114]

@glenn-jocher this is my results.
Original code
mAP@0.50 (conf=0.001) = 0.683/0.688 (yolov5/cocoapi)
mAP@0.50 (conf=0.250) = 0.708/0.621
mAP@0.50:0.95 (conf=0.001) = 0.496/0.506
mAP@0.50:0.95 (conf=0.250) = 0.554/0.466

[comlhj1114]

@glenn-jocher
PR code
mAP@0.50 (conf=0.001) = 0.682/0.688 (PR/cocoapi)
mAP@0.50 (conf=0.250) = 0.619/0.621
mAP@0.50:0.95 (conf=0.001) = 0.493/0.506
mAP@0.50:0.95 (conf=0.250) = 0.462/0.466

[comlhj1114]

@glenn-jocher
mAP calculations in the PR still have some gap, but much more smaller than the original and the trend of mAPs are same in the PR code, which are smaller when the conf-thres is higher, while in the original code, the trend is opposite.

[glenn-jocher]

@comlhj1114 ok thanks for the results! It looks like under the default scenario the PR is reducing the YOLO mAP further away from pycocotools. We really need solutions that go in the opposite direction to close the gap. We even have an open competition on the topic here: #2258

If we are using pycocotools as the standard then ideally the YOLO mAP calculation should also be 0.506, yet for some reason it is coming in lower on COCO for us, i.e. 0.496 in master and 0.493 in this PR.

[comlhj1114]

#7732 is much better solution.
so,this pr is closed

[glenn-jocher]

@comlhj1114 got it! Thanks for your update. If you have any other questions or need further assistance, feel free to ask. Good luck with your future evaluations!