Belief Coevolution in a Social Network of Generalist and Specialist Large Language Models
CoevolveSim is an agent-based simulation framework for studying how beliefs coevolve in a social network of large language models (LLMs) with varying domain expertise and social cues. The framework operationalizes coevolution as a multi-round interaction process in which LLM agents exchange beliefs, respond to neighbor summaries, and update their internal stance on factual statements.
CoevolveSim enables systematic evaluation of how expertise, roles and network structure jointly shape collective reasoning in LLM populations. The framework underlies all analyses in our paper TO BE ANNOUNCED.
- Overview
- Installation
- Quick Start
- Project Structure
- Running Experiments
- Analyzing Results
- Documentation
- Citation
- License
This framework simulates belief dynamics in social networks where agents are LLM-based entities that:
- Initialize with an independent assessment of a statement
- Observe the beliefs of their network neighbors
- Update their own beliefs through social influence
- Propagate beliefs across the network over multiple rounds
The simulation uses neural probes to extract belief scores from LLM hidden states, enabling quantitative tracking of belief evolution without requiring explicit probabilistic outputs from the models.
Each agent in CoevolveSim is an LLM with:
- Expertise: determined by its underlying model (e.g., biomedical, math, cybersecurity).
- Role: a social label such as “Clinical Physician,” “Chemist,” or “Human Participant”.
Agents interact through three key steps:
- Initial belief formation Agents receive a factual statement (e.g., “Methyl nicotinate is indicated for the treatment of aches”) and produce an initial classification via a zero-shot probe.
- Social exposure On each subsequent round, agents receive aggregated neighbor summaries containing the expressed beliefs and roles of adjacent agents (see Fig. 1, page 2).
- Belief updating Agents re-evaluate the statement conditioned on their own knowledge plus the neighborhood summary, yielding new belief scores across {true, false, neither}.
Over multiple rounds, belief states propagate through the network, enabling the study of convergence, cascades, fragmentation, and influence.
CoevolveSim is designed to study questions such as:
- How do LLM beliefs change when agents interact through a network rather than in isolation?
- When do expert agents stabilize collective beliefs?
- How do social identities (roles) influence agent susceptibility or influence?
- Under what conditions do belief cascades emerge?
- Are there persistent opinion leaders in LLM networks?
- Python: 3.10 or higher (tested on 3.10, 3.11, 3.12)
- PyTorch: 2.6.0 (for GPU support, install appropriate CUDA version)
- GPU: CUDA-capable GPU recommended (optional, CPU mode available)
- Memory: At least 16GB RAM (32GB+ recommended for larger models)
-
Clone the repository:
git clone https://github.com/carlomarxdk/coevolve-sim.git cd coevolve-sim -
Create a virtual environment (recommended):
python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate
-
Install dependencies:
pip install -r requirements.txt
Note on GPU Support: If you need CUDA support, ensure you install PyTorch with the correct CUDA version for your system. See PyTorch installation guide.
-
Verify installation:
# Run tests to verify the installation pytest tests/ -v # Run a quick test experiment (uses dummy backend, no GPU required) python experiment.py backend=dummy experiment.max_rounds=2 network.params.n=5
If you prefer Pipenv:
pipenv install
pipenv shellSee this guide on Pipenv setup for macOS users.
The framework uses Hydra for configuration management. The main entry point is experiment.py.
You can run one of the default experiment (16 agents, Erdős–Rényi network, 10 rounds):
python experiment.pyCustomize parameters using Hydra:
python experiment.py \
network.params.n=20 \
network.params.p=0.5 \
experiment.max_rounds=15 \
seed=42Reproduce the configuration used in the paper:
python experiment.py \
catalog=random_roles \
prompt=wR_L \
network.params.n=16 \
network.params.p=0.3 \
experiment.max_rounds=10 \
seed=814183Important
Some of the LLMs specified in the configs/model requires personal HuggingFace token.
For example, in case of llama-base.yaml, you need to update the access:token: field with a valid Access Token.
coevolve-sim/
├── configs/ # Hydra configuration files
│ ├── experiment.yaml # Main experiment configuration
│ ├── catalog/ # Agent role configurations
│ ├── network/ # Network topology settings
│ ├── probe/ # Belief probe configurations
│ ├── prompt/ # Prompt template definitions
│ └── statement/ # Evaluation statements
├── src/ # Core source code
│ ├── Agent.py # Agent classes (LLM, Expert)
│ ├── InferenceScheduler.py # Model loading and caching
│ ├── Message.py # Prompt generation system
│ ├── MetricsTracker.py # Metrics collection
│ ├── Network.py # Network topology management
│ ├── Probe.py # Belief scoring probes
│ └── metrics/ # Metric computation modules
├── tests/ # Comprehensive test suite
├── experiment.py # Main experiment runner
├── plot_*.py # Plotting and visualization scripts
├── utils.py # Utility functions
└── requirements.txt # Python dependencies
experiment.py: Main entry point for running simulationsutils.py: Experiment checkpointing, I/O management, and utilitiesplot_*.py: Various plotting scripts for visualizing results
The experiment uses Hydra's compositional configuration with several modular groups:
Defines the social network topology connecting agents.
erdos-renyi: Random graph generatorn: Number of nodes/agents (default: 16)p: Edge probability between any two nodes (default: 0.3)
# Create a network with 20 nodes and 50% edge probability
python experiment.py network.params.n=20 network.params.p=0.5Defines the composition of agents in the simulation.
simple: All agents have "LLM" role (default)only_llms: All agents have "LLM" role (same as simple)only_participants: All agents have "Human Participant" rolellm_and_participant: 50% LLM role, 50% Participant rolerandom_roles: Random role assignment from diverse role poolexperts: Expert agents with predefined knowledgerandom_experts: Mix of LLM and expert agents
python experiment.py catalog=random_rolesThese catalogs pull information for each single LLM from configs/models.
Important
Some of the LLMs specified in the configs/model requires personal HuggingFace token.
For example, in case of llama-base.yaml, you need to update the access:token: field with a valid Access Token.
Controls how agents are prompted and how neighbor information is presented.
woR_C: Without role context, count-based neighbor aggregationwR_C: With role context, count-based neighbor aggregationwR_L: With role context, detailed list of neighbor beliefs
python experiment.py prompt=wR_LThe claims that agents evaluate. Includes both true and false medical/scientific statements.
python experiment.py statement=true_doc_1
python experiment.py statement=false_rest_1Experiments are automatically saved in a hierarchical structure:
outputs/runs/
└── <catalog>/
└── <prompt>/
└── <statement>/
└── YYYY-MM-DD_HH-MM-SS/
├── config.json # Complete configuration
├── rounds/ # Per-round data
│ ├── round_0/
│ │ ├── beliefs.jsonl
│ │ ├── beliefs_detailed.json
│ │ └── metrics.json
│ └── ...
└── results/ # Final results
├── agents_data.json
├── network_edges.json
├── final_metrics.json
└── per_round_metrics.json
Run the test suite to verify your installation:
# Run all tests
pytest tests/ -v
# Run specific test modules
pytest tests/test_agent.py -v
pytest tests/test_network.py -v
pytest tests/test_probe.py -vTest coverage includes:
- Agent initialization and belief tracking
- Network topology and edge operations
- Model loading and caching
- Probe scoring and calibration
- Message generation and prompting
- Metrics tracking and aggregation
-
Always set a seed:
python experiment.py seed=814183
Note
Seed.txt contains a list of seed we use for the reruns of the experiments.
-
Document your configuration: Each experiment automatically saves its complete configuration to
config.jsonin the output directory. -
Version control your configs: Custom configuration files in
configs/should be version controlled. -
Check for completed experiments: The framework automatically checks if an experiment with the same configuration and seed has already been completed.
If you use this framework in your research, please cite our paper:
TO BE ANNOUNCEDThis project is licensed under the MIT License - see the LICENSE file for details.
We welcome contributions involving:
- New expert LLMs
- Alternative network topologies (e.g., Watts–Strogatz, Barabási–Albert)
- Additional belief extraction methods (e.g., activation-based probes)
- Human–LLM coevolution extensions
Please open an issue before submitting a PR.
Germans Savcisens*, Samantha Dies, Courtney Maynard, and Tina Eliassi-Rad
Khoury College of Computer Sciences, Northeastern University
Santa Fe Institute (T.E.-R.)
*Correspondence about the code or paper: g.savcisens@northeastern.edu
Note
- This is research software. While we strive for correctness and reproducibility, please verify results for your specific use case.
- GitHub Copilot contributed to code annotations, docstrings, and formatting. All algorithmic logic, methodological design, and scientific claims were developed and reviewed by the authors.