STAR-LQG: Set-Theoretic Adaptive Robust Linear-Quadratic Games

Overview

STAR-LQG is a MATLAB framework for learning-based robust control in linear-quadratic (LQ) games. It enables a controlled agent to iteratively identify and adapt to unknown adversary strategies while ensuring robust stability and near-optimal performance under bounded disturbances.

This approach combines set-membership estimation with robust LQR design via linear matrix inequalities (LMIs) to achieve an $\epsilon$-Nash equilibrium in continuous-time LQ games.

---

Motivation

Traditional methods in multi-agent learning and differential games often require players to alternate between learning and control phases, which is unrealistic in dynamic environments where agents must continuously adapt in real time. Existing adaptive schemes frequently rely on restrictive assumptions, such as requiring all agents to follow the same adaptation law or neglecting external disturbances.

STAR-LQG addresses these limitations by:

• Learning adversary strategies online using set-membership estimation.
• Ensuring robust closed-loop stability for all plausible adversary strategies.
• Adapting policies in real time, even under unknown adversary behavior and bounded disturbances.
• Maintaining convergence to a neighborhood-optimal solution, formalized as an $\epsilon$-Nash equilibrium.

---

Problem Formulation

The system dynamics are governed by a linear model:

$$ \dot{\mathbf{x}}(t) = \mathbf{A}\mathbf{x}(t) + \mathbf{B}_1\mathbf{u}_1(t) + \mathbf{B}_2\mathbf{u}_2(t) + \mathbf{w}(t), $$

where:

• $\mathbf{x} \in \mathbb{R}^{n_x}$: system state
• $u_1 \in \mathbb{R}^{n_{u_1}}$: controlled agent input
• $u_2 \in \mathbb{R}^{n_{u_2}}$: adversary input (unknown)
• $\mathbf{w}(t) \in \mathbb{R}^{n_x}$: bounded external disturbance
• $\mathbf{A}$, $\mathbf{B}_1$: known system matrices
• $\mathbf{B}_2$: unknown adversary input matrix

Each player minimizes an infinite-horizon quadratic cost:

$$ J_i(\mathbf{x}(\cdot),\mathbf{u}_i(\cdot)) = \int_{t_0}^{\infty} \left( \mathbf{x}(t)^T \mathbf{Q}_i \mathbf{x}(t) + \mathbf{u}_i(t)^T \mathbf{R}_i \mathbf{u}_i(t) \right) dt, \quad i \in {1,2}. $$

The controlled agent seeks a feedback gain $\mathbf{K}_1$ that minimizes its cost while guaranteeing robust stability against all consistent adversary strategies, which are iteratively estimated from observed data:

$$ \mathbf{u}_2(t) = -\mathbf{K}_2 \mathbf{x}(t) + \tilde{\mathbf{u}}_2(t), $$

where $\tilde{\mathbf{u}}_2(t)$ represents deviations from the unknown equilibrium strategy.

The set of admissible disturbances is modeled as a convex polytope:

$$ \mathcal{W} = {\tilde{\mathbf{w}} \in \mathbb{R}^{n_x} \mid \mathbf{G}_w \tilde{\mathbf{w}} \leq \mathbf{g}_w }. $$

---

Features

• Infinite-horizon LQ game solver with analytical Nash equilibrium computation.
• Set-membership estimation of adversary strategies from online data.
• Robust LQR synthesis via LMIs to guarantee closed-loop stability.
• Real-time policy adaptation under unknown adversary behavior and disturbances.
• MATLAB implementation with simulations and visualization scripts.

---

Installation

git clone https://github.com/TUM-ITR/STAR-LQG.git
cd STAR-LQG

Prerequisites

Ensure you have MATLAB installed with:

• Optimization Toolbox
• Control System Toolbox
• CVX (for convex optimization)

Code Structure

STAR-LQG/
│
├── main/                # Main simulation scripts
├── functions/           # Core algorithms: LQ solvers, LS, set-membership
├── plots/               # Scripts to visualize simulation results
├── examples/            # Example systems and parameter setups
└── README.md