TextSAM-EUS: Text Prompt Learning for SAM to Accurately Segment Pancreatic Tumor in Endoscopic Ultrasound

Health-X Lab | IMPACT Lab

Pascal Spiegler, Taha Koleilat, Arash Harirpoush, Corey S. Miller, Hassan Rivaz, Marta Kersten-Oertel, Yiming Xiao

Overview

Abstract: Pancreatic cancer carries a poor prognosis and relies on endoscopic ultrasound (EUS) for targeted biopsy and radiotherapy. However, the speckle noise, low contrast, and unintuitive appearance of EUS make segmentation of pancreatic tumors with fully supervised deep learning (DL) models both error-prone and dependent on large, expert-curated annotation datasets. To address these challenges, we present TextSAM-EUS, a novel, lightweight, text-driven adaptation of the Segment Anything Model (SAM) that requires no manual geometric prompts at inference. Our approach leverages text prompt learning (context optimization) through the BiomedCLIP text encoder in conjunction with a LoRA-based adaptation of SAM’s architecture to enable automatic pancreatic tumor segmentation in EUS, tuning only 0.86% of the total parameters. On the public Endoscopic Ultrasound Database of the Pancreas, TextSAM-EUS with automatic prompts attains 82.69% Dice and 85.28% normalized surface distance (NSD), and with manual geometric prompts reaches 83.10% Dice and 85.70% NSD, outperforming both existing state-of-the-art (SOTA) supervised DL models and foundation models (e.g., SAM and its variants). As the first attempt to incorporate prompt learning in SAM-based medical image segmentation, TextSAM-EUS offers a practical option for efficient and robust automatic EUS segmentation. Our code will be publicly available upon acceptance.

Method

Segmentation Results

Installation

This codebase is tested on Ubuntu 20.04.2 LTS with python 3.10. Follow the below steps to create environment and install dependencies.

• Setup conda environment (recommended).

# Create a conda environment
conda create -n textsam_eus python=3.10 -y

# Activate the environment
conda activate textsam_eus

# Install torch (requires version >= 2.1.2) and torchvision
# Please refer to https://pytorch.org/ if you need a different cuda version
pip install torch==2.1.2 torchvision==0.16.2 --index-url https://download.pytorch.org/whl/cu118

• Clone TextSAM-EUS code repository and install requirements

# Clone MaPLe code base
git clone https://github.com/HealthX-Lab/TextSAM-EUS

cd TextSAM-EUS/
# Install requirements

pip install -e .

Data preparation

• Download the dataset following the paper of Jaramillo et al. (2020).

• Place dataset under data like the following:

data/
|–– EUS/
|   |–– train/
|   |   |–– images/
|   |   |–– masks/
|   |–– val/
|   |   |–– images/
|   |   |–– masks/
|   |–– test/
|   |   |–– images/
|   |   |–– masks/

Training and Evaluation

• Run the training and evaluation script

bash scripts/pipeline.sh EUS outputs

• You can change some design settings in the config.

Citation

If you use our work, please consider citing:

@inproceedings{spiegler2025textsam,
  title={Textsam-eus: Text prompt learning for sam to accurately segment pancreatic tumor in endoscopic ultrasound},
  author={Spiegler, Pascal and Koleilat, Taha and Harirpoush, Arash and Miller, Corey S and Rivaz, Hassan and Kersten-Oertel, Marta and Xiao, Yiming},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={948--957},
  year={2025}
}

Acknowledgements

Our code builds upon the open_clip, segment-anything, and MaPLe repositories. We are grateful to the authors for making their code publicly available. If you use our model or code, we kindly request that you also consider citing these foundational works.