Segment then Splat: Unified 3D Open-Vocabulary Segmentation via Gaussian Splatting

Project page | Paper

This repository contains the official implementation associated with the paper "Segment then Splat: Unified 3D Open-Vocabulary Segmentation via Gaussian Splatting".

Dataset

We organize the datasets as follows, take lerf-ovs as an example:

├── lerf_ovs_langsplat
│   | ramen 
│     ├── images
│     ├── sparse 
│     ├── train.txt
│     ├── test.txt
│     ├── ...
│   | teatime 
│     ├── images
│     ├── sparse 
│     ├── train.txt
│     ├── test.txt
│     ├── ...
│    ...

We provide preprocessed lerf-ovs dataset here.

Run

Environment

git clone https://github.com/luyr/Segment-then-Splat --recursive
cd Segment-then-Splat

conda create -n segment_then_splat python=3.10
conda activate segment_then_splat

pip install torch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 --index-url https://download.pytorch.org/whl/cu121

pip install -r requirements.txt --no-build-isolation

Follow the instructions in thrid_party/AutoSeg-SAM2/ to install segment-anything-1 nad segment-anything-2, and download checkpoints.

Object Tracking & Object-specific Initialization (To be verified)

Remember to modify the data path inside autoseg.sh. For object_specific_initialization.py, you will need to run comap first. Refer to 3DGS for more details.

# Object Tracking
cd thrid_party/AutoSeg-SAM2/
bash autoseg.sh
# Object-specific Initialization
cd ../..
python ./helpers/preprocess_mask.py --mask_root third_party/AutoSeg-SAM2/output/ramen --out_root <data_dir>/lerf_ovs/ramen/ --image_path <data_dir>/lerf_ovs/ramen/images
python ./helpers/object_specific_initialization.py --scene_root <data_dir>/lerf_ovs/ramen/

We provide preprocessed lerf-ovs dataset with tracking results here.

Scene Reconstuction

python train.py -s <data_dir>/lerf_ovs_langsplat/ramen/ -m output/ramen  --eval --iterations 40000 --num_sample_objects 3 --densify_until_iter 20000 --partial_mask_iou 0.3

For dynamic scenes add --is_6dof and --deform.

---

CLIP Embedding Association & Evaluation

# Render object test images
python render_objs.py -m ./output/ramen/ --mode render --skip_train # For dynamic scenes add --is_6dof and --deform.

# Associate CLIP embedding and calculate mIoU
python ./helpers/evaluation.py --scene <data_dir>/lerf_ovs_langsplat/ramen/ --render_dir ./output/ramen/ --label_dir <data_dir>/lerf_ovs_langsplat/label/ramen/gt

Acknowledgments

We sincerely thank the authors of Deformable-3DGS, AutoSeg-SAM2, LERF, HyperNeRF, 3DOVS, and Neu3D, whose codes and datasets were used in our work.

BibTex

@article{Lu2025Segment,
  title={Segment then Splat: Unified 3D Open-Vocabulary Segmentation via Gaussian Splatting},
  author={Yiren Lu and Yunlai Zhou and Yiran Qiao and Chaoda Song and Tuo Liang and Jing Ma and Huan Wang and Yu Yin},
  journal={Advances in Neural Information Processing Systems},
  year={2025},
}