Skip to content

Security: Zero-Trust-AI-Security/ai-forensics

Security

docs/SECURITY.md

Security Overview

This project implements a SafeTensors inspection and analysis tool designed to operate on untrusted model artifacts. The primary security goals are:

  • Prevent code execution and unsafe deserialization
  • Enforce strict structural validation of SafeTensors files
  • Resist denial-of-service and resource-exhaustion attacks
  • Avoid unsafe memory access or unchecked arithmetic
  • Provide transparent, auditable security guarantees

This tool is intended for inspection, validation, and analysis only. It is not designed for training, inference, or execution of model code.

Security Reporting

Security issues should be reported via the project's designated security contact or disclosure process. Reports should include:

  • Affected version or commit
  • Reproduction steps or sample file (if possible)
  • Observed vs expected behavior

References

  • OWASP Secure Coding Practices (SCP) checklist

Threat Model

Trust Boundaries

  • Untrusted input: All .safetensors files, including headers, metadata, tensor declarations, and raw tensor bytes
  • Trusted components: Tool binary, runtime environment, standard libraries

Attacker Goals

  • Trigger memory corruption or unsafe behavior
  • Exhaust memory, CPU, or disk resources
  • Bypass validation to hide malformed or malicious tensor data
  • Confuse downstream analysis via misleading metadata

Non-Goals

  • Authenticity or provenance verification of model files
  • Protection against malicious model behavior during inference
  • Sandboxing or execution isolation (no execution occurs)

Secure Design Principles

  • Fail closed: Invalid or ambiguous files are rejected
  • Capability-based compatibility: Files are accepted based on supported format features, not declared version
  • Minimal attack surface: No dynamic loading, code execution, or reflection
  • Defensive parsing: All offsets, sizes, and counts are validated before use

SafeTensors Security Rules

Spec-Driven Validation (Format Correctness)

These rules enforce the SafeTensors file format as documented in the project specification.

  • Validate header length field and enforce size limits
  • Enforce UTF-8 encoding and JSON object structure
  • Validate tensor schema (dtype, shape, data_offsets)
  • Ensure tensor byte ranges are in-bounds and non-overlapping
  • Validate that shape * dtype_size == data_offsets length
  • Enforce little-endian interpretation of numeric data

Security-Driven Validation (Abuse Resistance)

These rules address malicious or adversarial inputs beyond basic correctness.

  • Cap header size, tensor count, and dimensionality
  • Enforce integer overflow checks on all size and offset calculations
  • Reject unknown or unsupported dtypes in strict mode
  • Reject malformed or control-character-containing names and metadata
  • Avoid allocations proportional to attacker-controlled sizes
  • Restrict or encapsulate any use of unsafe code

Trust Model Limitations

  • SafeTensors ensures structural safety, not data trustworthiness
  • Metadata is treated as untrusted and advisory only
  • The tool does not infer intent, safety, or correctness of model behavior
  • Hashing is provided for integrity checking, not trust establishment

CWE mapping

Input Validation & Parsing

  • CWE-20: Improper Input Validation

  • Why: The SafeTensors file format relies on structured, attacker-controlled input (binary header length, UTF-8 JSON header, tensor schema fields). Improper validation could allow malformed headers, inconsistent tensor declarations, or unsafe assumptions about file structure.

    • Untrusted input includes header length, JSON keys/values, tensor names, shapes, dtypes, and byte offsets.
    • CWE-20 is addressed explicitly because SafeTensors parsing is security-critical and operates on untrusted artifacts.
    • Failure to validate all fields consistently can enable downstream memory safety issues or logic bypasses.
  • Mitigation:

    • Validate all inputs against a strict, documented SafeTensors schema
    • Reject unknown, missing, or malformed fields
    • Enforce type, range, and structural constraints before any data access
    • Prefer allowlists over denylists for keys and values
    • Fail closed on ambiguity or parse errors
  • OWASP SCP alignment: Input Validation, Data Validation, Fail Securely

  • CWE-116: Improper Encoding or Escaping of Output

    • Why: Tensor names and metadata values may be rendered in logs, CLI output, JSON exports, or audit artifacts.
      • Unescaped control characters or terminal sequences could cause log injection, misleading output, or downstream parsing issues.
      • CWE-116 is addressed to prevent untrusted data from influencing how output is interpreted.
    • Mitigation:
      • Escape or sanitize control characters before output
      • Truncate excessively long untrusted strings
      • Avoid emitting raw metadata without sanitization
      • Use structured output formats where possible
    • OWASP SCP alignment: Output Encoding, Logging and Monitoring
  • CWE-176: Improper Handling of Unicode Encoding

    • Why: SafeTensors headers and tensor names are expected to be UTF-8 encoded.
      • Malformed UTF-8 sequences can cause parser confusion, inconsistent behavior, or downstream tooling errors.
      • Unicode edge cases can be abused for obfuscation or log manipulation.
    • Mitigation:
      • Reject invalid UTF-8 at parse time
      • Normalize or restrict Unicode classes if required
      • Avoid lossy decoding or silent replacement
    • OWASP SCP alignment: Input Validation, Data Encoding
  • CWE-444: Inconsistent Interpretation of HTTP Requests

    Reframed here as inconsistent interpretation of structured data

    • Why: Model artifact formats rely on structured input that may be interpreted differently by different tools or libraries.
      • Ambiguities such as duplicate keys, ordering dependence, or permissive parsing can be exploited to produce different interpretations across systems.
      • Attackers may craft inputs that appear valid to one parser but are rejected or interpreted differently by another.
      • CWE-444 is addressed to ensure deterministic, canonical interpretation of SafeTensors and future supported formats.
    • Mitigation:
      • Enforce a single, canonical parsing strategy
      • Reject duplicate keys or ambiguous structures
      • Avoid order-dependent parsing logic
      • Fail closed on ambiguous or non-canonical input
    • OWASP SCP alignment: Input Validation, Canonicalization, Fail Securely
  • CWE-502: Deserialization of Untrusted Data

    • Why: The tool deserializes untrusted structured data (e.g., JSON headers) from model artifacts.
      • Although SafeTensors avoids executable serialization formats, deserialization remains a security-sensitive operation.
      • Future supported formats may include additional structured metadata requiring deserialization.
      • CWE-502 is explicitly addressed to ensure no code execution, object instantiation side effects, or unsafe behavior occurs during deserialization.
    • Mitigation:
      • Deserialize only into strict, typed data structures
      • Disallow polymorphic or executable deserialization
      • Enforce schema validation before use
      • Treat all deserialized data as untrusted
    • OWASP SCP alignment: Secure Deserialization, Input Validation, Least Privilege
  • CWE-915: Improperly Controlled Modification of Dynamically-Determined Object Attributes

    • Why: Dynamic construction or extension of objects based on untrusted input can lead to logic confusion or unsafe state.
      • This CWE is closely related to CWE-1321 but applies more generally to dynamic attribute creation.
      • It is relevant for JSON parsing and for potential future formats with extensible metadata.
      • CWE-915 is addressed defensively to prevent untrusted input from altering program behavior.
    • Mitigation:
      • Use fixed schemas and typed structures
      • Reject unexpected or dynamically introduced attributes
      • Avoid merging untrusted data into program state objects
    • OWASP SCP alignment: Input Validation, Secure Object Handling
  • CWE-1284: Improper Validation of Specified Quantity in Input

    • Why: SafeTensors files declare tensor shapes, data types, and byte ranges independently.
      • A mismatch between shape * dtype_size and data_offsets length could hide malformed or truncated data.
      • CWE-1284 is addressed to ensure internal consistency across related fields
    • Mitigation:
      • Cross-validate tensor shape, dtype size, and byte range length
      • Reject any inconsistency between declared and computed sizes
      • Perform validation before any data access
    • OWASP SCP alignment: Data Validation, Integrity Controls
  • CWE-1321: Improperly Controlled Modification of Object Prototype Attributes (JSON-specific abuse class)

    • Why: SafeTensors headers are JSON objects supplied by untrusted sources.
      • Naive deserialization into dynamic maps may allow reserved keys (e.g., __proto__, constructor) to influence program behavior.
      • CWE-1321 is addressed defensively because the project processes untrusted JSON input.
      • While Rust mitigates classic prototype pollution, explicit schema enforcement prevents logic confusion, unsafe refactors, and tooling misuse.
    • Mitigation:
      • Deserialize into strict structs or maps
      • Reject unexpected or reserved keys
      • Do not merge metadata objects into program control structures
    • OWASP SCP alignment: Input Validation, Secure Deserialization

Memory Safety & Arithmetic

  • CWE-125: Out-of-Bounds Read

  • Why: Malformed data_offsets may reference bytes outside the file boundary.

    • Out-of-bounds reads can lead to crashes or information disclosure.
    • CWE-125 is relevant despite memory safety guarantees because logic errors can still cause invalid reads.
  • Mitigation:

    • Explicit bounds checking for every read
    • Enforce [start, end) semantics within file bounds
    • Reject overlapping or negative ranges
  • OWASP SCP alignment: Memory Safety, Defensive Coding

  • CWE-190: Integer Overflow or Wraparound

  • Why: SafeTensors parsing requires arithmetic on attacker-controlled integers, including header lengths, tensor shapes, byte offsets, and total sizes.

    • Integer overflow during offset or size calculation could bypass bounds checks or cause incorrect memory access.
    • Conversions between file-sized integers (u64) and platform-sized integers (usize) are particularly risky.
    • CWE-190 is explicitly addressed due to the use of arithmetic in security-sensitive parsing paths.
  • Mitigation:

    • Use checked arithmetic (checked_add, checked_mul, checked_sub) for all size and offset calculations
    • Reject values that overflow or exceed configured safety limits
    • Avoid implicit integer casts; require explicit, checked conversions
    • Enforce maximum limits on header size, tensor count, and total declared data size
  • OWASP SCP alignment: Integer Safety, Error Handling, Defense in Depth

  • CWE-416: Use After Free

    • Why: Relevant if memory-mapped files or unsafe code is used for performance.
      • Incorrect lifetime management could lead to use-after-free bugs.
    • Mitigation:
      • Minimize and isolate unsafe usage
      • Encapsulate mmap lifetimes in safe abstractions
      • Prefer streaming reads for inspection workflows
    • OWASP SCP alignment: Memory Safety, Secure Coding Practices
  • CWE-681: Incorrect Conversion Between Numeric Types

    • Why: Converting large file offsets to platform-dependent types can truncate values.
      • This may bypass validation or cause incorrect indexing.
    • Mitigation:
      • Use explicit, checked numeric conversions
      • Enforce platform-independent limits
      • Reject values that cannot be safely represented
    • OWASP SCP alignment: Integer Safety, Portability
  • CWE-787: Out-of-Bounds Write

    • Why: Relevant during tensor extraction or file export.
      • Writing beyond allocated buffers could corrupt memory or files.
    • Mitigation:
      • Never trust declared sizes.
      • Use bounded writes only
      • Avoid unsafe blocks where possible
    • OWASP SCP alignment: Memory Safety, Defensive Coding

Denial of Service & Resource Management

  • CWE-400: Uncontrolled Resource Consumption

    • Why: Attackers may craft files with large headers, excessive tensor counts, or pathological shapes.
    • Mitigation:
      • Enforce hard caps on header size, tensor count, and total declared data
      • Reject files exceeding resource limits early
    • OWASP SCP alignment: Availability, DoS Prevention
  • CWE-409: Improper Handling of Highly Compressed Data (Zip Bomb)

    • Why: Some model artifact formats (e.g., .pth, archives, or container formats) may include compressed data.
      • Highly compressed or nested compressed data can be used to trigger resource exhaustion (e.g., "decompression bombs").
      • While SafeTensors does not currently support compression, CWE-409 is documented to guide safe handling as new formats are added.
    • Mitigation:
      • Explicitly identify and control supported compression formats
      • Enforce strict limits on:
        • decompressed size
        • compression ratio
        • nesting depth
      • Prefer streaming decompression with hard caps
      • Reject unsupported or opaque compression schemes
    • OWASP SCP alignment: Resource Management, Availability, Defense in Depth
  • CWE-674: Uncontrolled Recursion

    • Why: Deeply nested structured input (e.g., JSON objects or arrays) can trigger excessive recursion or stack exhaustion.
      • This applies to SafeTensors headers and to future formats that may embed structured metadata.
      • CWE-674 is addressed to prevent stack-based denial-of-service conditions.
    • Mitigation:
      • Enforce maximum nesting depth for structured input
      • Prefer iterative parsing where feasible
      • Reject inputs exceeding configured depth limits
    • OWASP SCP alignment: Resource Management, Availability, Denial-of-Service Prevention
  • CWE-770: Allocation of Resources Without Limits or Throttling

    • Why: Large or deeply nested JSON structures can cause excessive memory or CPU usage.
    • Mitigation:
      • Stream parsing where feasible
      • Enforce depth and size limits
        • header size
        • tensor count
        • shape dimensionality
      • Avoid recursive or unbounded parsing
    • OWASP SCP alignment: Resource Management, Resilience
  • CWE-789: Uncontrolled Memory Allocation

    • Why: SafeTensors headers may declare extremely large tensors or many tensors without immediately containing equivalent data.
      • Naively allocating buffers based on declared sizes can lead to memory exhaustion.
      • CWE-789 is relevant because attackers may attempt to force large heap allocations via metadata alone.
      • This applies even in memory-safe languages if allocation size is attacker-controlled.
    • Mitigation:
      • Never allocate memory proportional to declared tensor sizes without enforcing hard limits
      • Use streaming or bounded reads instead of bulk allocation
      • Enforce caps on:
        • header size
        • tensor count
        • tensor dimensionality
        • total declared tensor byte size
      • Reject files that exceed configured resource limits early
    • OWASP SCP alignment: Resource Management, Availability, Denial-of-Service Prevention

Logic & Trust Boundary Issues

  • CWE-73: External Control of File Name or Path

    • Why: SafeTensors metadata may contain arbitrary strings that could be misused as file paths, filenames, or references to external resources.
      • If metadata were interpreted as filesystem paths, attackers could attempt path traversal, overwrites, or unintended file access.
      • CWE-73 is addressed defensively to ensure metadata remains purely informational.
      • This is particularly relevant in inspection or extraction workflows.
    • Mitigation:
      • Treat all metadata fields as opaque, untrusted data
      • Reject absolute paths, relative paths (../), or path separators in metadata values
      • Never use metadata-derived strings for filesystem operations without explicit validation
      • Require explicit, user-supplied paths for any file output operations
    • OWASP SCP alignment: Path Traversal Prevention, Trust Boundary Enforcement
  • CWE-345: Insufficient Verification of Data Authenticity

    • Why: SafeTensors provides safety, not authenticity.
    • Mitigation (documentation-level):
      • Document authenticity as a non-goal
      • Provide hashing for integrity checks only
      • Rely on external provenance systems
      • Explicitly state that:
        • SafeTensors is not trusted
        • Hashing and provenance checks are external
    • OWASP SCP alignment: Cryptography Usage, Security Documentation
  • CWE-807: Reliance on Untrusted Inputs in a Security Decision

    • Why: Using metadata fields to make trust decisions.
      • Metadata fields could be misleading or malicious.
    • Mitigation:
      • Treat metadata as advisory only
      • Never gate security logic on metadata contents
    • OWASP SCP alignment: Trust Boundary Enforcement
  • CWE-843: Access of Resource Using Incompatible Type

    • Why: Treating tensor bytes as a dtype they do not represent (e.g., malformed dtype string).
      • Incorrect dtype interpretation could cause invalid data handling.
    • Mitigation:
      • Enforce strict dtype allowlists
      • Reject unknown or unsupported types
      • Avoid implicit coercion
    • OWASP SCP alignment: Type Safety, Defensive Coding
  • CWE-917: Improper Neutralization of Special Elements in Expression Language

    • Why: Some model formats or metadata systems may embed expressions, configuration directives, or special syntax.
      • If such expressions were evaluated or interpreted, they could influence control flow or execution.
      • Even when currently unsupported, CWE-917 is documented defensively to prevent unsafe future expansion.
      • SafeTensors metadata is currently treated as inert data only.
    • Mitigation:
      • Do not evaluate or interpret expressions from model artifacts
      • Treat all metadata fields as opaque strings
      • Explicitly prohibit expression evaluation or templating
      • Require explicit design review before supporting any evaluative syntax
    • OWASP SCP alignment: Input Validation, Trust Boundary Enforcement, Secure Design

CWEs Not covered in the design

  • CWE-78 / CWE-94 - Command or Code Injection
  • CWE-79 - XSS
  • CWE-89 - SQL Injection
  • CWE-306 / CWE-287 - Authentication / Authorization
  • CWE-22 - Path Traversal (already covered via CWE-73, don't duplicate)

Not applicable - the tool performs no execution, evaluation, or authentication.

Secure Development Practices

  • Prefer safe Rust constructs; minimize and isolate unsafe
  • All file parsing performed with explicit bounds and limits
  • Errors include context but avoid leaking sensitive system details
  • Code is structured to support fuzzing and property testing
  • Security-relevant behavior is documented and auditable

Summary

This tool is designed to safely inspect untrusted SafeTensors files by combining:

  • A stable, well-defined format specification
  • Defensive parsing and strict validation
  • Explicit acknowledgment of trust limitations

The result is a robust, auditable inspection tool suitable for security-sensitive environments.

There aren’t any published security advisories