YoloV7 serverless detector feature for auto annotation (cvat-ai#5552)

### Motivation and context Integration of YOLOv7 as a serverless nuclio function that can be used for auto-labeling. YoloV7 is the SOTA at the time of this PR therefore it would make sense to support it in CVAT. The integration is quite simple into CVAT as docker based on Ultralytics YoloV5 with coco pretrained model (https://github.com/WongKinYiu/yolov7) and a docker image (https://hub.docker.com/r/ultralytics/yolov5). related issue: cvat-ai#5548 ### How has this been tested? Automatic annotation was run using YOLOv7 on a custom dataset. The serverless function was deployed using ``` nuctl deploy --project-name cvat \ --path serverless/onnx/WongKinYiu/yolov7/nuclio \ --volume `pwd`/serverless/common:/opt/nuclio/common \ --platform local ``` Then using the 'Automatic annotation' action the function was tested and the auto-generated labels were controlled to check that no coordinates misfit is happening. ### Use custom model: 1. Export your model with NMS for image resolution of 640x640 (preferable). 2. Copy your custom model yolov7-custom.onnx to /serverless/common 3. Modify function.yaml file according to your labels. 4. Modify model_handler.py as follow: ``` self.model_path = "yolov7-custom.onnx" ``` Co-authored-by: Nikita Manovich <nikita@cvat.ai> Co-authored-by: yasakova-anastasia <yasakova_anastasiya@mail.ru>
amjadsaadeh · Jan 10, 2023 · f3843aa · f3843aa
1 parent bcaca03
commit f3843aa
Show file tree

Hide file tree

Showing 6 changed files with 421 additions and 0 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -16,6 +16,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   (<https://github.com/opencv/cvat/pull/5523>)
 - \[SDK\] A PyTorch adapter setting to disable cache updates
   (<https://github.com/opencv/cvat/pull/5549>)
+- YOLO v7 serverless feature added using ONNX backend (<https://github.com/opencv/cvat/pull/5552>)
 
 ### Changed
 - The Docker Compose files now use the Compose Specification version

diff --git a/README.md b/README.md
@@ -178,6 +178,7 @@ up to 10x. Here is a list of the algorithms we support, and the platforms they c
 | [Faster RCNN](/serverless/openvino/omz/public/faster_rcnn_inception_v2_coco/nuclio)                     | detector   | OpenVINO   | ✔️  |     |
 | [Mask RCNN](/serverless/openvino/omz/public/mask_rcnn_inception_resnet_v2_atrous_coco/nuclio)           | detector   | OpenVINO   | ✔️  |     |
 | [YOLO v3](/serverless/openvino/omz/public/yolo-v3-tf/nuclio)                                            | detector   | OpenVINO   | ✔️  |     |
+| [YOLO v7](/serverless/onnx/WongKinYiu/yolov7/nuclio)                                                    | detector   | ONNX       | ✔️  | ✔️  |
 | [Object reidentification](/serverless/openvino/omz/intel/person-reidentification-retail-300/nuclio)     | reid       | OpenVINO   | ✔️  |     |
 | [Semantic segmentation for ADAS](/serverless/openvino/omz/intel/semantic-segmentation-adas-0001/nuclio) | detector   | OpenVINO   | ✔️  |     |
 | [Text detection v4](/serverless/openvino/omz/intel/text-detection-0004/nuclio)                          | detector   | OpenVINO   | ✔️  |     |

diff --git a/serverless/onnx/WongKinYiu/yolov7/nuclio/function-gpu.yaml b/serverless/onnx/WongKinYiu/yolov7/nuclio/function-gpu.yaml
@@ -0,0 +1,135 @@
+metadata:
+  name: onnx-yolov7
+  namespace: cvat
+  annotations:
+    name: YOLO v7
+    type: detector
+    framework: onnx
+    spec: |
+      [
+        { "id": 0, "name": "person" },
+        { "id": 1, "name": "bicycle" },
+        { "id": 2, "name": "car" },
+        { "id": 3, "name": "motorbike" },
+        { "id": 4, "name": "aeroplane" },
+        { "id": 5, "name": "bus" },
+        { "id": 6, "name": "train" },
+        { "id": 7, "name": "truck" },
+        { "id": 8, "name": "boat" },
+        { "id": 9, "name": "traffic light" },
+        { "id": 10, "name": "fire hydrant" },
+        { "id": 11, "name": "stop sign" },
+        { "id": 12, "name": "parking meter" },
+        { "id": 13, "name": "bench" },
+        { "id": 14, "name": "bird" },
+        { "id": 15, "name": "cat" },
+        { "id": 16, "name": "dog" },
+        { "id": 17, "name": "horse" },
+        { "id": 18, "name": "sheep" },
+        { "id": 19, "name": "cow" },
+        { "id": 20, "name": "elephant" },
+        { "id": 21, "name": "bear" },
+        { "id": 22, "name": "zebra" },
+        { "id": 23, "name": "giraffe" },
+        { "id": 24, "name": "backpack" },
+        { "id": 25, "name": "umbrella" },
+        { "id": 26, "name": "handbag" },
+        { "id": 27, "name": "tie" },
+        { "id": 28, "name": "suitcase" },
+        { "id": 29, "name": "frisbee" },
+        { "id": 30, "name": "skis" },
+        { "id": 31, "name": "snowboard" },
+        { "id": 32, "name": "sports ball" },
+        { "id": 33, "name": "kite" },
+        { "id": 34, "name": "baseball bat" },
+        { "id": 35, "name": "baseball glove" },
+        { "id": 36, "name": "skateboard" },
+        { "id": 37, "name": "surfboard" },
+        { "id": 38, "name": "tennis racket" },
+        { "id": 39, "name": "bottle" },
+        { "id": 40, "name": "wine glass" },
+        { "id": 41, "name": "cup" },
+        { "id": 42, "name": "fork" },
+        { "id": 43, "name": "knife" },
+        { "id": 44, "name": "spoon" },
+        { "id": 45, "name": "bowl" },
+        { "id": 46, "name": "banana" },
+        { "id": 47, "name": "apple" },
+        { "id": 48, "name": "sandwich" },
+        { "id": 49, "name": "orange" },
+        { "id": 50, "name": "broccoli" },
+        { "id": 51, "name": "carrot" },
+        { "id": 52, "name": "hot dog" },
+        { "id": 53, "name": "pizza" },
+        { "id": 54, "name": "donut" },
+        { "id": 55, "name": "cake" },
+        { "id": 56, "name": "chair" },
+        { "id": 57, "name": "sofa" },
+        { "id": 58, "name": "pottedplant" },
+        { "id": 59, "name": "bed" },
+        { "id": 60, "name": "diningtable" },
+        { "id": 61, "name": "toilet" },
+        { "id": 62, "name": "tvmonitor" },
+        { "id": 63, "name": "laptop" },
+        { "id": 64, "name": "mouse" },
+        { "id": 65, "name": "remote" },
+        { "id": 66, "name": "keyboard" },
+        { "id": 67, "name": "cell phone" },
+        { "id": 68, "name": "microwave" },
+        { "id": 69, "name": "oven" },
+        { "id": 70, "name": "toaster" },
+        { "id": 71, "name": "sink" },
+        { "id": 72, "name": "refrigerator" },
+        { "id": 73, "name": "book" },
+        { "id": 74, "name": "clock" },
+        { "id": 75, "name": "vase" },
+        { "id": 76, "name": "scissors" },
+        { "id": 77, "name": "teddy bear" },
+        { "id": 78, "name": "hair drier" },
+        { "id": 79, "name": "toothbrush" }
+      ]
+
+spec:
+  description: YOLO v7 via onnx-runtime
+  runtime: 'python:3.8'
+  handler: main:handler
+  eventTimeout: 30s
+  build:
+    image: cvat/onnx-yolov7
+    baseImage: ultralytics/yolov5:latest
+
+    directives:
+      preCopy:
+        - kind: USER
+          value: root
+        - kind: RUN
+          value: apt update && apt install --no-install-recommends -y libglib2.0-0
+        - kind: WORKDIR
+          value: /opt/nuclio
+        - kind: RUN
+          value: pip install onnxruntime
+        - kind: WORKDIR
+          value: /opt/nuclio
+        - kind: RUN
+          value: wget https://github.com/WongKinYiu/yolov7/releases/download/v0.1/yolov7-nms-640.onnx
+        - kind: RUN
+          value: ln -s /usr/bin/python3 /usr/bin/python
+
+  triggers:
+    myHttpTrigger:
+      maxWorkers: 1
+      kind: 'http'
+      workerAvailabilityTimeoutMilliseconds: 10000
+      attributes:
+        maxRequestBodySize: 33554432 # 32MB
+
+  resources:
+    limits:
+      nvidia.com/gpu: 1
+
+  platform:
+    attributes:
+      restartPolicy:
+        name: always
+        maximumRetryCount: 3
+      mountMode: volume
diff --git a/serverless/onnx/WongKinYiu/yolov7/nuclio/function.yaml b/serverless/onnx/WongKinYiu/yolov7/nuclio/function.yaml
@@ -0,0 +1,128 @@
+metadata:
+  name: onnx-yolov7
+  namespace: cvat
+  annotations:
+    name: YOLO v7
+    type: detector
+    framework: onnx
+    spec: |
+      [
+        { "id": 0, "name": "person" },
+        { "id": 1, "name": "bicycle" },
+        { "id": 2, "name": "car" },
+        { "id": 3, "name": "motorbike" },
+        { "id": 4, "name": "aeroplane" },
+        { "id": 5, "name": "bus" },
+        { "id": 6, "name": "train" },
+        { "id": 7, "name": "truck" },
+        { "id": 8, "name": "boat" },
+        { "id": 9, "name": "traffic light" },
+        { "id": 10, "name": "fire hydrant" },
+        { "id": 11, "name": "stop sign" },
+        { "id": 12, "name": "parking meter" },
+        { "id": 13, "name": "bench" },
+        { "id": 14, "name": "bird" },
+        { "id": 15, "name": "cat" },
+        { "id": 16, "name": "dog" },
+        { "id": 17, "name": "horse" },
+        { "id": 18, "name": "sheep" },
+        { "id": 19, "name": "cow" },
+        { "id": 20, "name": "elephant" },
+        { "id": 21, "name": "bear" },
+        { "id": 22, "name": "zebra" },
+        { "id": 23, "name": "giraffe" },
+        { "id": 24, "name": "backpack" },
+        { "id": 25, "name": "umbrella" },
+        { "id": 26, "name": "handbag" },
+        { "id": 27, "name": "tie" },
+        { "id": 28, "name": "suitcase" },
+        { "id": 29, "name": "frisbee" },
+        { "id": 30, "name": "skis" },
+        { "id": 31, "name": "snowboard" },
+        { "id": 32, "name": "sports ball" },
+        { "id": 33, "name": "kite" },
+        { "id": 34, "name": "baseball bat" },
+        { "id": 35, "name": "baseball glove" },
+        { "id": 36, "name": "skateboard" },
+        { "id": 37, "name": "surfboard" },
+        { "id": 38, "name": "tennis racket" },
+        { "id": 39, "name": "bottle" },
+        { "id": 40, "name": "wine glass" },
+        { "id": 41, "name": "cup" },
+        { "id": 42, "name": "fork" },
+        { "id": 43, "name": "knife" },
+        { "id": 44, "name": "spoon" },
+        { "id": 45, "name": "bowl" },
+        { "id": 46, "name": "banana" },
+        { "id": 47, "name": "apple" },
+        { "id": 48, "name": "sandwich" },
+        { "id": 49, "name": "orange" },
+        { "id": 50, "name": "broccoli" },
+        { "id": 51, "name": "carrot" },
+        { "id": 52, "name": "hot dog" },
+        { "id": 53, "name": "pizza" },
+        { "id": 54, "name": "donut" },
+        { "id": 55, "name": "cake" },
+        { "id": 56, "name": "chair" },
+        { "id": 57, "name": "sofa" },
+        { "id": 58, "name": "pottedplant" },
+        { "id": 59, "name": "bed" },
+        { "id": 60, "name": "diningtable" },
+        { "id": 61, "name": "toilet" },
+        { "id": 62, "name": "tvmonitor" },
+        { "id": 63, "name": "laptop" },
+        { "id": 64, "name": "mouse" },
+        { "id": 65, "name": "remote" },
+        { "id": 66, "name": "keyboard" },
+        { "id": 67, "name": "cell phone" },
+        { "id": 68, "name": "microwave" },
+        { "id": 69, "name": "oven" },
+        { "id": 70, "name": "toaster" },
+        { "id": 71, "name": "sink" },
+        { "id": 72, "name": "refrigerator" },
+        { "id": 73, "name": "book" },
+        { "id": 74, "name": "clock" },
+        { "id": 75, "name": "vase" },
+        { "id": 76, "name": "scissors" },
+        { "id": 77, "name": "teddy bear" },
+        { "id": 78, "name": "hair drier" },
+        { "id": 79, "name": "toothbrush" }
+      ]
+spec:
+  description: YOLO v7 via onnx
+  runtime: 'python:3.8'
+  handler: main:handler
+  eventTimeout: 30s
+  build:
+    image: cvat/onnx-yolov7
+    baseImage: ultralytics/yolov5:latest-cpu
+
+    directives:
+      preCopy:
+        - kind: USER
+          value: root
+        - kind: RUN
+          value: apt update && apt install --no-install-recommends -y libglib2.0-0 && apt install wget
+        - kind: RUN
+          value: pip install onnxruntime
+        - kind: WORKDIR
+          value: /opt/nuclio
+        - kind: RUN
+          value: wget https://github.com/WongKinYiu/yolov7/releases/download/v0.1/yolov7-nms-640.onnx
+        - kind: RUN
+          value: ln -s /usr/bin/python3 /usr/bin/python
+
+  triggers:
+    myHttpTrigger:
+      maxWorkers: 2
+      kind: 'http'
+      workerAvailabilityTimeoutMilliseconds: 10000
+      attributes:
+        maxRequestBodySize: 33554432 # 32MB
+
+  platform:
+    attributes:
+      restartPolicy:
+        name: always
+        maximumRetryCount: 3
+      mountMode: volume
diff --git a/serverless/onnx/WongKinYiu/yolov7/nuclio/main.py b/serverless/onnx/WongKinYiu/yolov7/nuclio/main.py
@@ -0,0 +1,37 @@
+import base64
+import io
+import json
+
+import yaml
+from model_handler import ModelHandler
+from PIL import Image
+
+
+def init_context(context):
+    context.logger.info("Init context...  0%")
+
+    # Read labels
+    with open("/opt/nuclio/function.yaml", 'rb') as function_file:
+        functionconfig = yaml.safe_load(function_file)
+
+    labels_spec = functionconfig['metadata']['annotations']['spec']
+    labels = {item['id']: item['name'] for item in json.loads(labels_spec)}
+
+    # Read the DL model
+    model = ModelHandler(labels)
+    context.user_data.model = model
+
+    context.logger.info("Init context...100%")
+
+
+def handler(context, event):
+    context.logger.info("Run YoloV7 ONNX model")
+    data = event.body
+    buf = io.BytesIO(base64.b64decode(data["image"]))
+    threshold = float(data.get("threshold", 0.5))
+    image = Image.open(buf)
+
+    results = context.user_data.model.infer(image, threshold)
+
+    return context.Response(body=json.dumps(results), headers={},
+        content_type='application/json', status_code=200)