MtpNet 💪: Multitask Panoptic Driving 🚗 Perception Network

English | 简体中文

Introduction

Mtpnet Network Architecture.

• MtpNet is an efficient multitasking network used to solve three important tasks in autonomous driving: traffic object detection, drivable area segmentation, and lane detection.
• In all three tasks, both accuracy and speed have reached advanced levels.

Web Demo

• Integrated into Huggingface Spaces 🤗 using Gradio. Try out the Web Demo !

Results

We used the BDD100K as our datasets,and experiments are run on NVIDIA 3090.

Visualization

Model : trained on the BDD100k dataset and test on the bilibili: Cam 看世界 video.

input	output by yolop	output by mtpnet

Example: output by mtpnet

Traffic Object Detection Result

Result	Visualization
Model mAP@0.5 (%) Recall (%) MultiNet 60.2 81.3 DLT-Net 68.4 89.4 Faster R-CNN 55.6 77.2 YOLOv5s 77.2 86.8 YOLOP 76.5 89.2 HybridNets 77.3 92.8 YOLOPv2 83.4 91.1 MtpNet 89.8(+6.4) 94.1(+1.3)

Drivable Area Segmentation

Result	Visualization
Model Drivable mIoU (%) MultiNet 71.6 DLT-Net 71.3 PSPNet 89.6 YOLOP 91.5 HybridNets 90.5 YOLOPv2 93.2 MtpNet 96.2(+3.0)

Lane Line Detection

Result	Visualization
Model Accuracy (%) Lane Line IoU (%) Enet 34.12 14.64 SCNN 35.79 15.84 Enet-SAD 36.56 16.02 YOLOP 70.5 26.2 HybridNets 85.4 31.6 YOLOPv2 87.3 27.2 MtpNet 88.7(+1.4) 27.8(-3.8)

Model parameter and inference speed(fps NVIDIA 3090)

Model	Size	Params	Batch4	Batch8	Batch16	Batch32	Average
YOLOP	640	7.9M	50	47	50	44	47
HybridNets	640	12.8M	29	32	47	59	41
YOLOPv2	640	38.9M	66	79	79	75	74
MtpNet	640	50.7M	55	63	65	66	62

Day-time and Night-time visualization results

Models

You can get the model from here.

Demo Test

You can detect images or videos.

python demo_onnx.py  --source demo/images/example.png # Identify the specified image
python demo_onnx.py  --source demo/images             # Identify all images in the folder
python demo_onnx.py  --source demo/videos/example.mp4 # Identify designated videos
python demo_onnx.py  --source demo/videos             # Identify all videos in the folder
python demo_onnx.py  --source 0                       # Real recognition through camera

Data preparation

Download

• Download the images from images.
• Download the annotations of detection from det_annotations.
• Download the annotations of drivable area segmentation from da_seg_annotations.
• Download the annotations of lane line segmentation from ll_seg_annotations.

License

MtpNet is released under the MIT Licence.