Foundation Model Empowered Synesthesia of Machines (SoM): AI-native Intelligent Multi-Modal Sensing-Communication Integration

Cheng, Xiang, et al. "Foundation Model Empowered Synesthesia of Machines (SoM): AI-native Intelligent Multi-Modal Sensing-Communication Integration." IEEE Transactions on Network Science and Engineering, 2025, (early acess). [paper]

---

📌 Overview

In our survey, we propose two roadmaps for enabling AI-native multi-modal sensing-communication integration using Foundation Models (FMs):

• Roadmap 1: Large Language Models Empowered SoM System Design
• Roadmap 2: Wireless Foundation Models Empowered SoM System Design

A total of five case studies are discussed in the paper.

Case Study	Roadmap Origin	Folder Path
Case Study 1.1	Roadmap 1	./Case_Study_1.1/
Case Study 1.2	Roadmap 1	./Case_Study_1.2/
Case Study 2.1	Roadmap 2	./Case_Study_2.1/
Case Study 2.2	Roadmap 2	./Case_Study_2.2/
Case Study 2.3	Roadmap 2	./Case_Study_2.3/

---

📁 Directory Structure

Each case study is structured as follows:

Case Study X.Y/
├── Codes/              # Main source code and inference script
│   ├── Inference.py   # Entry point for reproduction
│   ├── Weights/       # Pretrained model weights (download separately)
│   └── data/          # Input data (download separately)
├── requirements.txt   # Python dependency list

📥 Weights and data are provided via Baidu Cloud Disk Download link. Please download and place them under the corresponding Codes/ folder as described below.

---

🚀 Quick Start

1. Clone the repository

git clone https://github.com/liuboxun/FM4SoM.git
cd FM4SoM/Case_Study_1.2/Codes    # Or cd Case_Study_2.2/Codes

2. Install dependencies

pip install -r requirements.txt

💡 Tip: It is recommended to use a virtual environment like conda.

3. Prepare weights and data

Download the Weights/ and data/ folders from the link above and place them under the Codes/ directory.

4. Run the inference script

python Inference.py

This will reproduce the results for the selected case study as described in the paper.

---

📄 Citation

If you find our work helpful, please consider citing:

@article{cheng2025fm4som,
  title     = {Foundation Model Empowered Synesthesia of Machines (SoM): AI-native Intelligent Multi-Modal Sensing-Communication Integration},
  author    = {Cheng, Xiang and Liu, Boxun and Liu, Xuanyu and Liu, Ensong and Huang, Ziwei},
  journal   = {IEEE Transactions on Network Science and Engineering},
  year      = {2025},
  note      = {Early Access},
}