LLM05 — Improper Output Handling (POC)

Reference: OWASP LLM05:2025 Improper Output Handling

Goal: demonstrate how unsafe rendering or execution of LLM outputs leads to XSS/data risks, and how simple guards (sanitization, schema validation, policies) mitigate them.

Quick Start

This PoC consists of a React frontend (Vite) and a Python backend (Flask) using uv for Python package management.

Prerequisites

• Node.js 18+ and npm
• Python 3.11+
• uv (Python package manager)
• An LLM API key (OpenAI, Anthropic, etc.)

Installation

1. Clone the repository:

git clone <repo-url>
cd owasp-LLM05

2. Install frontend dependencies:

cd frontend
npm install
cd ..

3. Install backend dependencies with uv:

cd backend
uv sync
cd ..

4. Configure environment variables:

cp .env.example .env

Running the Application

Terminal 1 - Start backend:

cd backend
uv run python main.py

Backend runs on http://127.0.0.1:5000

Terminal 2 - Start frontend:

cd frontend
npm run dev

Frontend runs on http://localhost:5173

Open http://localhost:5173 in your browser to see the PoC.

Available Commands

# Frontend
cd frontend
npm run dev          # Start development server
npm run build        # Build for production
npm test             # Run unit tests

# Backend
cd backend
uv run python main.py        # Start API server

Running Tests

The frontend includes user interaction tests that verify the components work correctly:

cd frontend
npm test             # Run tests in watch mode
npm test -- --run    # Run tests once and exit

Test Coverage:

• User typing prompts into forms
• Form submission and API calls
• Loading states
• Error handling
• Button enable/disable states

These tests simulate real user behavior to ensure the renderers function correctly.

What's the Vulnerability?

Treating LLM output as trusted and directly rendering or executing it creates serious security risks. This PoC demonstrates XSS vulnerabilities, but these represent just a sample of the many issues that can arise from improper output handling.

Core principle: LLM output must be validated and sanitized like any other untrusted user input. Just because it comes from an AI model doesn't make it safe.

The Problem: Rendering HTML/Markdown Without Sanitization → XSS Attacks

When LLM-generated content containing malicious scripts is rendered directly in the browser, attackers can:

• Execute arbitrary JavaScript in users' browsers
• Steal sessions and cookies
• Manipulate the DOM
• Redirect users to malicious sites
• Perform actions on behalf of the user

Why this happens: LLMs are designed to generate helpful content, including HTML. Attackers can use prompt injection techniques to coerce the model into generating malicious code, which then executes when rendered unsafely.

Other Potential Risks (Not Demonstrated in This PoC)

While this PoC focuses on XSS, improper output handling can lead to other vulnerabilities:

• SQL Injection: If an application runs LLM-generated database queries without validation or parameterization, attackers can manipulate the LLM to produce malicious SQL that deletes data, exfiltrates sensitive information, or compromises the database.
• Remote Code Execution: When LLM output is passed to eval() or system shells
• CSRF Attacks: Malicious links or forms in generated content
• Path Traversal: File system access through generated file paths

See the OWASP reference for additional attack scenarios.

How It Works

User ──> UI Form ──> /api/generate (Backend) ──> LLM API
                              │
                              └──> Returns Markdown/HTML
                                   │
                                   ├─> ❌ Unsafe Path: UI renders raw HTML
                                   │   └─> XSS vulnerability
                                   │
                                   └─> ✅ Safe Path: UI uses DOMPurify + CSP
                                       └─> User sees sanitized content

The application demonstrates both vulnerable and secure implementations side-by-side, allowing you to see the difference in behavior and understand how proper output handling prevents attacks.

Demonstrating the XSS Vulnerability

Understanding the Attack Vector

The application shows two renderers side-by-side:

• Unsafe Renderer (left): Renders raw LLM output using dangerouslySetInnerHTML → XSS executes
• Safe Renderer (right): Sanitizes with DOMPurify before rendering → XSS blocked

Why <img onerror> instead of <script> tags?

Browsers block <script> tags inserted via innerHTML (a security feature), but event handler attributes like onerror do execute. This is why DOMPurify is critical - it strips dangerous attributes that browsers don't block.

Testing the Vulnerability

See DEMO_PROMPTS.md for ready-to-use prompts that demonstrate the XSS vulnerability. The prompts show how attackers can bypass LLM safety filters using educational framing and indirect injection techniques.

Mitigations Implemented

This PoC demonstrates the two essential security controls that prevent XSS from improper output handling:

1. Sanitize Before Render (DOMPurify)

Strip dangerous elements before displaying LLM output in the browser:

DOMPurify.sanitize(html, {
  ALLOWED_TAGS: ['p', 'b', 'i', 'em', 'strong', 'a', 'ul', 'ol', 'li'],
  ALLOWED_ATTR: ['href', 'title']
})

This removes <script> tags, event handlers, and other XSS vectors while preserving safe formatting.

2. Content Security Policy (CSP)

Default-deny inline scripts and restrict resource loading. Set via meta tag in frontend/index.html:

<meta http-equiv="Content-Security-Policy" content="default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self' data: https:; font-src 'self'; connect-src 'self' http://127.0.0.1:5000;">

To test CSP:

1. Open frontend/index.html
2. Uncomment the CSP meta tag (remove <!-- and --> around line 18)
3. Restart the frontend dev server

What CSP Blocks:

• Inline <script> tags (when unsafe-inline is not present)
• javascript: URLs
• eval() and similar dynamic code execution
• Event handler attributes (onerror, onclick, etc.) when unsafe-inline is not present

CSP Limitations:

• unsafe-inline keyword: If your CSP includes unsafe-inline, it allows inline scripts and event handlers, making CSP ineffective against XSS
• Meta tag limitations: CSP in meta tags can be bypassed if malicious content is injected before the meta tag loads
• DOM-based XSS: CSP doesn't prevent XSS if malicious code is already present in the page's HTML
• Server-side XSS: CSP only protects the browser, not server-side rendering vulnerabilities
• Other attacks: CSP doesn't prevent CSRF, SQL injection, or clickjacking (use frame-ancestors for that)

Important: CSP provides defense-in-depth and blocks both inline scripts and event handlers when properly configured (without unsafe-inline). DOMPurify is also critical as it sanitizes content before it reaches the browser, providing multiple layers of protection. Use both together for maximum security.

Additional Hardening Options

For production applications, consider layering additional defenses:

• Structured output validation (validate JSON schema before rendering)
• Human-in-the-loop review for high-risk content
• Parameterized queries if executing LLM-generated SQL

What This Repo Demonstrates

Vulnerability: Trusting LLM output (HTML/Markdown, code, queries) without validation/sanitization.

Impact: XSS in a dashboard when LLM returns HTML with malicious event handlers.

Fixes: Sanitize rendered content (DOMPurify), enforce content policies (CSP), validate structured output, parameterize any queries, add "human-in-the-loop" for risky actions.

Resources

• OWASP LLM Top 10
• DOMPurify Documentation
• Content Security Policy Guide