3D Multi Object Tracking with Pixhawk, Ardupilot Copter, NVIDIA Jetson, ROS and stereo camera

Papers

Multi Object Tracking

• Simple Online and Realtime Tracking with a Deep Association Metric
• Towards Real-Time Multi-Object Tracking

Object Detection models

• YOLOv3
• CenterNet (DLA or Resnet)

REID models

• In Defense of the Triplet Loss for Person Re-Identification
• Triplet Loss and Online Triplet Mining in TensorFlow

UAV specific

• Object detection on aerial imagery using CenterNet
• VisDrone-MOT2019: The Vision Meets DroneMultiple Object Tracking Challenge Results
• Eye in the Sky: Drone-Based Object Tracking and 3DLocalization
• Super accurate low latency object detection on a surveillance UAV

Datasets

MOT

• VisDrone MOT
• UAV123
• UAVDT
• SDD

Projects

• https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0225092

• https://github.com/ArduPilot/redtail

QuadCopter Object Tracking on a budget - OpenMV + RPi0 + OpenCV + DroneKit

• Indoor autonomous flight with Arducopter, ROS and Aruco Boards Detection

Implementation

Object Detection, REID and Tracking

Pytorch

• https://github.com/Qidian213/deep_sort_yolov3 - Deep SORT + YOLOv3

• https://github.com/kimyoon-young/centerNet-deep-sort - Deep SORT + CenterNet

• https://github.com/Zhongdao/Towards-Realtime-MOT - YOLOv3 with combined Detection and REID

• https://github.com/pytorch/vision/tree/master/references/similarity - REID

• https://github.com/ultralytics/yolov3 - YOLOv3 ONNX

• https://github.com/xingyizhou/CenterNet - CenterNet

Tensorflow 2

• https://github.com/zzh8829/yolov3-tf2
• https://github.com/xuannianz/keras-CenterNet

TensorRT C++:

https://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/index.html

INT8 precision

PyTorch + ONNYX

0. onnx installation

• docker

• pip

pip install onnx onnxruntime

• git

1. torch.onnx.export()

2. onnx-simplifier

https://github.com/daquexian/onnx-simplifier

3. ONNX -> TensorRT

https://github.com/onnx/onnx-tensorrt

https://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/index.html#import_onnx_c

Tensorflow + UFF

https://www.tensorflow.org/versions/r2.1/api_docs/python/tf/experimental/tensorrt/Converter

https://github.com/dusty-nv/jetson-inference

tensorNet subclass

https://github.com/NVIDIA/TensorRT/tree/release/7.0/samples/opensource/sampleUffSSD

Serialization

Must be done on Jetson

https://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/index.html#serial_model_c

DLib

Kalman Filter / Particle Filter

http://dlib.net/algorithms.html#kalman_filter

Optimization

http://dlib.net/optimization.html#max_cost_assignment

ROS

1. https://github.com/dusty-nv/ros_deep_learning
2. Implement Nodelets
3. MAVROS: https://dev.px4.io/v1.9.0/en/ros/mavros_installation.html
4. Stereo camera wrappers 4.1. Realsense D435: https://github.com/IntelRealSense/realsense-ros 4.2. ZED Stereo: https://github.com/stereolabs/zed-ros-wrapper

Other

OpenKAI

https://github.com/yankailab/OpenKAI -

NVIDIA DeepStrem

https://developer.nvidia.com/deepstream-sdk

Navigation

PID Controller

https://github.com/ArduPilot/ardupilot/tree/master/libraries/PID

https://github.com/yankailab/OpenKAI/tree/master/src/Control

https://github.com/yans0ng/Venom-ROS/tree/master/venom_offb/include/venom_offb

Reinforcement learning

• Generalization through Simulation: Integrating Simulated and Real Data into Deep Reinforcement Learning for Vision-Based Autonomous Flight

RL + AirSim

• https://github.com/kshitiz38/autonomous_drone_for_tracking
• https://github.com/microsoft/AirSim/blob/master/docs/reinforcement_learning.md#rl-with-quadrotor

AirSim Simulation

AirSim Stereo Camera -> ROS MOT -> MAVROS -> AirSim Ardupilot SITL

Ardupilot integration

https://ardupilot.org/dev/docs/sitl-with-airsim.html

https://discuss.ardupilot.org/t/gsoc-2019-airsim-simulator-support-for-ardupilot-sitl/42890

https://discuss.ardupilot.org/t/airsim-multiple-arducopter-sitl-mavros/49613/4

ROS C++ wrapper

https://github.com/microsoft/AirSim/tree/master/ros/src/airsim_ros_pkgs

Hardware

NVIDIA Jetson

TX2

Xavier NX

https://developer.nvidia.com/embedded/jetson-xavier-nx

Installation

TX2:

• Jetpack 4.3 (CUDA, TensorRT)
• ROS
• Stereo Camera SDK

PC

• PyTorch
• AirSim