LiDUT-Depth

A Lightweight Self-supervised Depth Estimation Model featuring Dynamic Upsampling and Triplet Loss Optimization

Hao Jiang$^{[0009-0000-7962-6732]^1}$, Zhijun Fang$^{*[0000-0001-8563-5678]^{1, 2}}$, Xuan Shao$^{**[0000-0002-4096-9428]^1}$
Xiaoyan Jiang$^{[0000-0002-1946-576X]^2}$ and Jenq-Neng Hwang$^{[0000−0002−8877−2421]^3}$
$^1$ School of Computer Science and Technology, Donghua University, Shanghai, China
$^2$ School of Electronic and Electrical Engineering, Shanghai University of Engineering Science, Shanghai, China
$^3$ Department of Electrical & Computer Engineering, University of Washington, Washington D.C., United States

This work was supported in part by the Natural Science Foundation of China under Grant U2033218, in part by the Shanghai Sailing Program under Grant 23YF1400900, and in part by the Fundamental Research Funds for the Central Universities under Grants CF221223219597.
The paper was accepted by ICPR 2024 (27th International Conference on Pattern Recognition)

(LiDUT-Depth 1024x320)

Table of Contents

• LiDUT-Depth
  • Table of Contents
  • Overview
  • Results
    • KITTI
  • Setup
  • Data Preparation
  • Single Image Test
  • Evaluation
  • Training
    • start training
    • tensorboard visualization
  • Citation

Overview

Results

KITTI

You can download the trained models using the links below.

Model	Params	ImageNet Pretrained	Input size	Abs Rel	Sq Rel	RMSE	RMSE log	delta < 1.25	delta < 1.25^2	delta < 1.25^3
LiDUT-Depth_640x192	3.1M	yes	640x192	0.103	0.723	4.469	0.178	0.889	0.964	0.984
LiDUT-Depth_1024x320	3.1M	yes	1024x320	0.101	0.703	4.354	0.175	0.895	0.966	0.985

Setup

Assuming a fresh Anaconda distribution, you can install the dependencies with:

conda create -n lidut-depth python=3.10
conda activate lidut-depth

pip install -r requirements.txt
pip install 'git+https://github.com/saadnaeem-dev/pytorch-linear-warmup-cosine-annealing-warm-restarts-weight-decay'

We ran our experiments with PyTorch 2.0.1, CUDA 11.8, Python 3.10.12 and Ubuntu 20.04.

We recommend using a conda environment to avoid dependency conflicts.

Data Preparation

You can download the entire raw KITTI dataset by running:

wget -i splits/kitti_archives_to_download.txt -P kitti_data/

Then unzip with

cd kitti_data
unzip "*.zip"
cd ..

Warning: it weighs about 175GB, so make sure you have enough space to unzip too!

Our default settings expect that you have converted the png images to jpeg with this command, which also deletes the raw KITTI .png files:

find kitti_data/ -name '*.png' | parallel 'convert -quality 92 -sampling-factor 2x2,1x1,1x1 {.}.png {.}.jpg && rm {}'

or you can skip this conversion step and train from raw png files by adding the flag --png when training, at the expense of slower load times.

Pre-computed segmentation images provided by FSRE-Depth are also needed for training.

Note: The pre-computed segmentation is NOT needed for evaluating.

KITTI-DATA/
    ├── 2011_09_26/             
    ├── 2011_09_28/                    
    ├── 2011_09_29/
    ├── 2011_09_30/
    ├── 2011_10_03/
    └── segmentation/   # download and unzip "segmentation.zip" 

Single Image Test

python test_simple.py --load_weights_folder path/to/your/weights/folder --image_path path/to/your/test/image

Evaluation

To prepare the ground truth depth maps run:

python export_gt_depth.py --data_path kitti_data --split eigen
python export_gt_depth.py --data_path kitti_data --split eigen_benchmark

...assuming that you have placed the KITTI dataset in the default location of ./kitti_data/.

Evaluate the weights of a single epoch:

python evaluate_depth.py --load_weights_folder path/to/your/weights/folder --data_path path/to/kitti_data/

Or evaluate the weights of every epochs together (will print all the evaluate results to a txt file in your model folder):

python evaluate_depth_all.py --load_weights_folder path/to/your/model/folder --data_path path/to/kitti_data/ --num_epochs number/of/your/epochs/to/evaluate

To evaluate the model's generalization ability, you'll need to prepare the Make3D dataset first. Download Test134 images and depths, simply uncompress them to ./splits/make3d, then run ./splits/make3d/make3d_extraction.py to get the image filenames, finally run ./evaluate_depth_make3d.py to get the generalization ability result (you may need to modify the data path in the code):

python evaluate_depth_make3d.py

Training

start training

python train.py --data_path path/to/your/data --model_name test --num_epochs 30 --batch_size 12 --num_workers 16 --mypretrain path/to/your/pretrained/weights

tensorboard visualization

tensorboard --log_dir ./tmp/mytrain

Citation

@incollection{Jiang2024LiDUT,
	booktitle = {Lecture Notes in Computer Science},
	doi = {10.1007/978-3-031-78444-6_12},
	isbn = {9783031784439},
	issn = {0302-9743},
	language = {en},
	publisher = {Springer Nature Switzerland},
	title = {LiDUT-Depth: A Lightweight Self-supervised Depth Estimation Model Featuring Dynamic Upsampling and Triplet Loss Optimization},
	url = {http://dx.doi.org/10.1007/978-3-031-78444-6_12},
	author = {Hao Jiang, and Zhijun Fang, and Xuan Shao, and Xiaoyan Jiang, and Jenq-Neng Hwang},
	pages = {176--189},
	date = {2024-12-04},
	year = {2024},
	month = {12},
	day = {4},
}