PayloadStash – README YAML Spec

YAML‑driven HTTP fetch‑and‑stash for Python. Define your run once, execute a sequence of REST calls, and write every response to a clean, predictable folder structure—with defaults, forced fields, anchors/aliases, concurrency, resilient error handling, and run‑level reporting.

Note: For the authoritative, formal definition of the YAML configuration grammar (types, constraints, and resolution semantics), see “README - Config File Formal Specification.md” at the repository root. This README summarizes usage and examples and is kept consistent with that specification.

Concept

PayloadStash reads a YAML config, resolves YAML anchors, merges Defaults and Forced values into each request, and executes requests sequentially or concurrently according to your Sequences. Responses are saved to disk with file extensions based on Content‑Type.

Defaults – Used only if a request does not provide that section.
Forced – Injected into every request; overrides request and defaults when keys collide.
Resolved copy – After anchor resolution & merges, the effective config is written to *-resolved.yml next to the output.
Failure handling – A failing request does not stop the stash run. Its HTTP status and timing are recorded, output is written (error body if available), and subsequent requests continue.

Quick Start - Native (recommended for most users)

# 1) Install (venv will be created in ./.venv)
# Regular install: (recommended for users)
python3 bootstrap.py
# Editable (dev) install: (recommended for developers)
python3 bootstrap.py --editable

# 2) Run a config (you can run without activating the venv)
./payloadstash run path/to/config.yml --out ./out

# 3) Validate only (no requests)
./payloadstash validate path/to/config.yml

# 4) Emit the fully-resolved config (after anchors & merges)
./payloadstash resolve path/to/config.yml --out ./out

If you prefer to manage venvs yourself:

python -m venv .venv && .venv/bin/python -m pip install -U pip && .venv/bin/pip install -e .

Quick Start - Docker Execution

Prerequisites: Docker and Docker Compose (v2 preferred).
Build the image (one-time or when sources change): ./x-docker-build-payloadstash.sh
Validate a config inside the container (from the host's ./config dir:) ./x-docker-run-payloadstash.sh validate config-example.yml
Run a config (writes outputs under ./output on the host): ./x-docker-run-payloadstash.sh run config-example.yml

Notes:

The run script mounts ./config -> /app/config and ./output -> /app/output.
If you pass --out/-o, it is rewritten to /app/output automatically; otherwise the script adds --out /app/output for 'run'.
Relative config paths are assumed to be under ./config and are rewritten to /app/config/ in the container.
Before the app starts, the script prints the exact host directories being mounted.

Exporting to an Air-Gapped server using a Docker Image

Package all the files (whenever sources or dependencies change) on a host with internet access: sudo ./x-docker-package-payloadstash.sh This will create a ./packaged/payloadstash.zip file.
Copy the ./packaged/payloadstash.zip file to the air-gapped server.
On the air-gapped server, extract the package: unzip payloadstash.zip This will create a ./payloadstash/ directory.
Switch into ./payloadstash then load the image to Docker with: sudo ./x-docker-load-payloadstash.sh
Still from inside this directory, run the cli on the air-gapped server via Docker: x-docker-run-payloadstash.sh run config-example.yml

Quick Start - Developer Install

If you are developing PayloadStash, the recommended editable install is via the bootstrap script:

Editable install (creates .venv and installs -e): python3 bootstrap.py --editable
Reinstall only when:
- dependencies change, or
- entry point names change.
Otherwise, edit code and rerun the CLI payloadstash, no reinstall needed.

If you prefer to manage venvs yourself:

Do once per environment: python -m venv .venv && .venv/bin/python -m pip install -U pip && .venv/bin/pip install -e .

Directory Layout

PayloadStash writes responses to a deterministic path with a timestamped run folder:

<out>/
  <StashConfig.Name>/
    <RunTimestamp>/
      seqNNN-<Sequence.Name>/
        reqNNN-<RequestKey>-response.<ext>
      <original-config>-results.csv
      <original-config>-resolved.yml
      <original-config>-log.txt

Example

out/
  PXXX-Tester-01/
    2025-09-17T15-42-10Z/
      seq001-GetGeneralData/
        req001-GetConfig-response.json
        req002-GetCatalog-response.json
      seq002-GetPlayer01/
        req001-GetState-response.json
        req002-GrantItem-response.json
      PXXX-Tester-01-results.csv
      PXXX-Tester-01-resolved.yml
      PXXX-Tester-01-log.txt

Configuration Overview

A PayloadStash YAML contains two major areas:

Header Groups (Anchors / Aliases) – optional convenience blocks for DRY configs.
StashConfig – the actual run definition: name, defaults, forced values, and sequences.

###########################################################
# Header Groups (Anchors / Aliases)
###########################################################

common_headers: &common_headers
  Content-Type: application/json
  Accept: application/json

common_headers_players: &common_headers_players
  X-App-Client: PayloadStash/1.0
  X-Player-API: v2

###########################################################
# Stash Configuration
###########################################################

StashConfig:
  Name: PXXX-Tester-01

  #########################################################
  # Defaults
  #########################################################
  Defaults:
    URLRoot: https://somehost.com/api/v1
    Headers: *common_headers

  #########################################################
  # Forced Values
  #########################################################
  Forced:
    Headers: {}
    Body:
      someprop: abc
      anotherprop: your value here
    Query: {}

  #########################################################
  # Sequences
  #########################################################
  Sequences:
    - Name: GetGeneralData
      Type: Concurrent
      ConcurrencyLimit: 4
      Requests:
        - GetConfig:
            Method: POST
            URLPath: /getGameConfig
            Headers:
              <<: *common_headers
              X-Request-Scope: config

        - GetCatalog:
            Method: POST
            URLPath: /getGameConfig
            Headers:
              <<: *common_headers
              X-Request-Scope: catalog

    - Name: GetPlayer01
      Type: Sequential
      Requests:
        - GetState:
            Method: POST
            URLPath: /getState
            Headers:
              <<: [*common_headers, *common_headers_players]
              X-Request-Scope: state

        - GrantItem:
            Method: POST
            URLPath: /grantItem
            Headers:
              <<: [*common_headers, *common_headers_players]
              X-Request-Scope: grant

Full YAML Schema (informal)

# 0) Optional header groups for anchors/aliases
<alias_name>: &<alias_name>
  <HeaderKey>: <string>
  # ...repeat as needed...

Dynamics:
  patterns:
    userid_hex_prefixed:
      template: "1234${hex:20}"
    userid_hex_structured:
      template: "1234${hex:22}${choice:teams}${hex:4}${hex:2}00"
    userid_uuid_v4:
      template: "${uuidv4}"

  sets:
    teams: ["0","1","2","3"]

StashConfig:
  Name: <string>

  Defaults:
    # Required Defaults
    URLRoot: <string>
    FlowControl:
      # Number of seconds in between sequences and requests (default: 0)
      DelaySeconds: <int>
      TimeoutSeconds: <int>
    
    # Allow Insecure TLS
    InsecureTLS: <bool>
    
    # Optional Defaults
    Headers?: { <k>: <v>, ... }
    Body?:    { <k>: <v>, ... }
    # Example: compute a timestamp at resolve-time
    # Body.timestamp can call the built-in timestamp() helper:
    #   timestamp: { $func: timestamp, format: iso_8601 }
    Query?:   { <k>: <v>, ... }

    # Allow Response Processing
    Response?:
      PrettyPrint?: <bool>
      Sort?: <bool>

    # Optional global retry policy (applies when a request omits Retry)
    Retry?:
      Attempts: <int>                 # total tries including the first (e.g., 3)
      BackoffStrategy: <fixed|exponential>
      BackoffSeconds: <number>        # base delay (e.g., 0.5)
      Multiplier?: <number>           # exponential growth factor (e.g., 2.0)
      MaxBackoffSeconds?: <number>    # cap per-try backoff
      MaxElapsedSeconds?: <number>    # overall cap across all retries (optional)
      Jitter?: <bool|string>         # boolean or "min"/"max"; see Formal Specification for jitter semantics
      RetryOnStatus?: [<int>, ...]    # HTTP codes to retry (e.g., [429, 500, 502, 503, 504])
      RetryOnNetworkErrors?: <bool>   # retry on DNS/connect/reset/timeouts (default: true)
      RetryOnTimeouts?: <bool>        # retry when client timeout occurs (default: true)

  Forced?:
    Headers?: { <k>: <v>, ... }
    Body?:    { <k>: <v>, ... }
    Query?:   { <k>: <v>, ... }

  Sequences:
    - Name: <string>
      Type: <Sequential|Concurrent>
      ConcurrencyLimit?: <int>
      Requests:
        - <RequestKey>:
            Method: <GET|POST|PUT|PATCH|DELETE|HEAD|OPTIONS>
            URLPath: <string>
            Headers?: { <k>: <v>, ... }
            Body?:    { <k>: <v>, ... }
            Query?:   { <k>: <v>, ... }
            FlowControl?:
              DelaySeconds?: <int>
              TimeoutSeconds?: <int>
            # Optional per-request retry policy (overrides Defaults.Retry if present)
            Retry?:
              Attempts: <int>
              BackoffStrategy: <fixed|exponential>
              BackoffSeconds: <number>
              Multiplier?: <number>
              MaxBackoffSeconds?: <number>
              MaxElapsedSeconds?: <number>
              Jitter?: <false|min|max|true>
              RetryOnStatus?: [<int>, ...]
              RetryOnNetworkErrors?: <bool>
              RetryOnTimeouts?: <bool>
            Response:
              PrettyPrint?: <bool>
              Sort?: <bool>

Functions & Dynamics inside configs

You can compute certain values using a special object syntax. Prefer the concise shorthand mapping forms for $ operators where available (e.g., { $timestamp: epoch_ms }, { $dynamic: name }). The following helpers are available:

Functions:

timestamp: returns the current UTC time in one of several formats.

Usage forms:

{ $func: timestamp, format: iso_8601 }
With control over when it is evaluated:
- { $func: timestamp, format: iso_8601, when: request }

Supported formats for timestamp: epoch_ms, epoch_s, iso_8601.

When parameter:

when: resolve (default) – evaluate during config resolution (e.g., when writing *-resolved.yml).
when: request – defer evaluation until the moment you are about to send the request.

How deferral works:

During config resolution, request-time functions are preserved as a marker like: {"$deferred": {"func": "timestamp", "format": "epoch_ms"}}
At send-time, call the helper to resolve these markers in the object you are about to send:

from payload_stash.config_utility import resolve_deferred

# Example inputs with deferred markers (as they would appear after resolve-time)
headers = {"X-Request-Id": {"$deferred": {"func": "timestamp", "format": "epoch_ms"}}}
query = {"q": "value"}
body = {"now": {"$deferred": {"func": "timestamp", "format": "iso_8601"}}}

ready_headers = resolve_deferred(headers)
ready_query = resolve_deferred(query)
ready_body = resolve_deferred(body)

Example adding timestamps with both timings:

StashConfig:
  Name: WithTimestamp
  Defaults:
    Body:
      ts_resolve: { $func: timestamp, format: iso_8601 }
      ts_request: { $func: timestamp, format: epoch_ms, when: request }
  Sequences:
    - Name: OnlySeq
      Type: Sequential
      Requests:
        - Ping:
            Method: GET
            URLPath: /health
            Body:
              now_req: { $func: timestamp, format: iso_8601, when: request }

Dynamics:

Dynamics let you declare reusable ID or token patterns once and then materialize them anywhere in Headers, Query, or Body using a special object form: { $dynamic: <patternName>, ... }.
Unlike functions (which call code), dynamics expand a named template you define under the top-level dynamics: section of your YAML.

Where you declare them:

Top-level section (sibling of StashConfig):

dynamics:
  patterns:
    user_structured:
      template: "1234${hex:22}${choice:teams}00"
    user_uuid:
      template: "user-${uuidv4}"
  sets:
    teams: ["00", "01", "02", "03"]

patterns: A map of pattern names to a template string.
sets: Named lists used by ${choice:<setName>} placeholders inside templates.

Template placeholders supported:

${hex:N} — N random hexadecimal characters, uppercase A–F guaranteed.
${alphanumeric:N} — N random characters from 0-9, A-Z, a-z.
${numeric:N} — N random digits 0-9.
${alpha:N} — N random letters A-Z, a-z.
${uuidv4} — A standard UUID v4 string, e.g., 3f2b0b9a-3c03-4b0a-b4ad-5d9d3f6e45a7.
${choice:setName} — Pick one random element from a named set, e.g., teams above returns one of ["00", "01", "02", "03"].
${timestamp[:format]} — Current UTC timestamp; format options: iso_8601 (default), epoch_ms, epoch_s.
${secrets:secretName} — Secret value from the secrets file, e.g., ${secrets:api_key}.

Using a dynamic in a request:

Resolve-time (default): materialize immediately during config resolution.
```
Body:
  userId: { $dynamic: user_structured }
```
Request-time deferral: keep as a marker until the HTTP call is about to be sent, so every request gets a fresh value.
```
Body:
  userId: { $dynamic: user_structured, when: request }
```

When parameter for dynamics:

when: resolve (default) — expand the template while creating the *-resolved.yml file. The resulting literal string appears in that file.
when: request — store a deferred marker in the resolved file; the CLI will expand it right before sending the request so each attempt (or each request in a loop) can get a unique value.

Notes and behavior:

Scope: dynamics is top-level and applies to the entire file. Pattern names must be unique. You can keep multiple pattern families in one file by namespacing, e.g., player_uuid_v4.
Determinism: expansions are random by design (hex/choice). Use resolve-time if you want to audit exact values in *-resolved.yml; use request-time to get per-request variety.
Coexistence with Functions: you can freely mix $func and $dynamic in the same object. Both support when with the same semantics.
Resolved output: request-time dynamics are preserved in *-resolved.yml as a generic $deferred marker and expanded by the runner at send-time (similar to $func deferral).

Static Dynamics in resolved files:

In every -resolved.yml, a top-of-file section named "Static Dynamics" lists the resolve-time values for each defined dynamic pattern.
Any use of { $dynamic: name } that does not specify when: request will use the single value shown for that pattern in "Static Dynamics" throughout the resolved file.
If a use specifies when: request, the resolved file will contain a $deferred marker for that field and a fresh value will be generated right before each request is sent.

End-to-end example:

dynamics:
  patterns:
    player_id:
      template: "p-${hex:8}-${choice:teams}-${hex:4}"
  sets:
    teams: ["NORTH", "SOUTH", "EAST", "WEST"]

StashConfig:
  Name: WithDynamics
  Defaults:
    Body:
      # resolve-time value: appears literal in *-resolved.yml
      example_id_resolve: { $dynamic: player_id }
      # request-time: shows as deferred in *-resolved.yml and is generated at send-time
      example_id_request: { $dynamic: player_id, when: request }
  Sequences:
    - Name: OnlySeq
      Type: Sequential
      Requests:
        - Ping:
            Method: GET
            URLPath: /health
            Body:
              per_call_id: { $dynamic: player_id, when: request }

Secrets ($secrets) and the --secrets argument

PayloadStash supports injecting secrets into your YAML config at resolve-time using a separate .env-style file. This keeps sensitive values out of versioned configs while still allowing convenient use in headers, query strings, or bodies.

Key points:

Secrets are provided via a separate file passed to the CLI with --secrets <path>.
The secrets file uses KEY=VALUE lines and is case-sensitive for both keys and values.
Secret values are injected during config resolution. The resolved config written to disk and all run logs will have secrets redacted as ***REDACTED***.
If a config references a secret but no secrets file is provided, or the key is missing, validation fails with an informative error.

Secrets file format

Example .env-style file (any filename is fine):

# Comments and blank lines are ignored
AUTH_TOKEN=abc123DEF
MixedCaseKey=ValueWithMixedCASE
QUOTED_VALUE="Bearer abc-123-XYZ"
SINGLE_QUOTED='value with spaces'

Rules:

Lines beginning with # and blank lines are ignored.
Format is KEY=VALUE. The key is trimmed; the value is preserved as-is except for trimming surrounding spaces and stripping matching quotes if the entire value is quoted.
Duplicate keys: last one wins.
Keys and values are treated as case-sensitive.

Referencing secrets in YAML

Two equivalent forms are supported inside any of Headers, Query, or Body maps:

Mapping form (pure value replacement):

Authorization: { $secrets: AUTH_TOKEN }

Inline string form (template-like inside a larger string):

authorization: "Bearer { $secrets: AUTH_TOKEN }"

Both forms are valid YAML. The inline form is often convenient for tokens with required prefixes.

Example using anchors:

common_headers: &common_headers
  Content-Type: application/json
  Accept: application/json
  authorization: "Bearer { $secrets: AUTH_TOKEN }"

StashConfig:
  Name: ExampleWithSecrets
  Defaults:
    URLRoot: https://api.example.com
    Headers: *common_headers
  Sequences:
    - Name: Only
      Type: Sequential
      Requests:
        - Ping:
            Method: GET
            URLPath: /health

Request-time deferral with secrets

Although secrets can be referenced inside dynamic templates and technically support when: request just like timestamps, secrets are static values. Deferring their resolution provides no practical benefit because the secret will be identical at resolve-time and at request-time. For clarity and simplicity, prefer resolving secrets at the default time (resolve-time). They will still be redacted in the resolved config and in run logs.

CLI usage with secrets

Validate and ensure secrets resolve:

payloadstash validate config.yml --secrets ./config/my-secrets.env

Run with secrets (outputs kept redacted):

payloadstash run config.yml --out ./out --secrets ./config/my-secrets.env

Notes:

If a secret is referenced and the --secrets flag is omitted, validation/run will fail with a clear error.
If an unknown secret key is referenced, validation/run will fail.
The resolved config written to the run folder will contain ***REDACTED*** in place of any secret value.
The run log also redacts any occurrences of loaded secret values.

Docker usage

When using the provided Docker scripts, mount your secrets file and reference its in-container path with --secrets:

If you place your secrets under the repo’s ./config directory (mounted to /app/config in the container):

./x-docker-run-payloadstash.sh run config-example.yml --secrets /app/config/my-secrets.env

Troubleshooting

Error: "Secret '' requested but no --secrets file was provided"
- Pass a secrets file path with --secrets.
Error: "Unknown secret requested: ''"
- Ensure the key exists in your secrets file (case-sensitive).
Secrets not redacted in output
- The written resolved config and run log produced by payloadstash run will redact automatically. If you dump structures yourself in custom scripts, avoid logging raw secret values.

Merge & Precedence Rules

PayloadStash computes each request’s effective sections in this order:

Start with empty {Headers, Body, Query}.
If the request defines a section, copy it in.
If the request omits a section, copy from Defaults.
Forced is merged last and overrides.
URLRoot comes from Defaults only. It is not allowed inside a Request.

Example: If Defaults.Body.team = "blue", Request.Body.team omitted, and Forced.Body.team = "green", then team == "green".

Anchors, Aliases & Header Merging

YAML anchors are resolved before merging Defaults/Forced.

common_headers: &common_headers
  Content-Type: application/json
  Accept: application/json

player_headers: &player_headers
  X-App-Client: PayloadStash/1.0
  X-Player-API: v2

Headers:
  <<: [*common_headers, *player_headers]
  X-Request-Scope: state

If the same key appears in multiple merged maps, the last one wins. After anchor resolution, PayloadStash writes *-resolved.yml so you can audit.

Sequences & Concurrency

Sequences are executed in the order listed.
Each sequence has a Type:
- Sequential: requests execute one-at-a-time.
- Concurrent: requests execute in parallel (async/await). ConcurrencyLimit caps fan-out.
A failed request does not stop the run. Its response, HTTP status, and timing are written; execution continues.

Flow Control

Control pacing and client timeouts via a FlowControl block.

Location:

Required at Defaults: Defaults.FlowControl with both fields present.
Optional per-request override at Request.FlowControl (either field may be provided to override that aspect for the request).

Fields:

DelaySeconds: Non-negative integer. Delay applied between requests and when advancing to the next sequence.
TimeoutSeconds: Non-negative integer. Client-side request timeout.

Example (Defaults and per-request override):

StashConfig:
  Name: WithDelayAndTimeout
  Defaults:
    URLRoot: https://api.example.com
    FlowControl:
      DelaySeconds: 1
      TimeoutSeconds: 5
  Sequences:
    - Name: A
      Type: Sequential
      Requests:
        - First:
            Method: GET
            URLPath: /a
        - Second:
            Method: GET
            URLPath: /b
            FlowControl:
              TimeoutSeconds: 1   # override only timeout for this request
              # DelaySeconds omitted -> uses Defaults.FlowControl.DelaySeconds
    - Name: B
      Type: Concurrent
      ConcurrencyLimit: 3
      Requests:
        - One:
            Method: GET
            URLPath: /x
        - Two:
            Method: GET
            URLPath: /y

Retry Explained

The Retry block defines how PayloadStash retries failed HTTP requests.

Location

Can be defined under Defaults to apply globally.
Can be overridden or disabled (Retry: null) at the per-request level.

Fields

Attempts – total tries including the first. Attempts: 3 = first try + up to 2 retries.
BackoffStrategy – either fixed or exponential.
- fixed: each retry waits the same BackoffSeconds.
- exponential: waits grow by a Multiplier each retry (e.g., 0.5s, 1s, 2s, 4s…).
BackoffSeconds – base wait time before applying strategy.
Multiplier – growth factor for exponential backoff.
MaxBackoffSeconds – maximum wait allowed for a single retry.
MaxElapsedSeconds – maximum total time spent across all retries.
Jitter – controls randomness in the wait (boolean or one of "min"/"max"). For precise semantics, see the Formal Specification document. In brief: false or omitted = no jitter; true = enable jitter with default behavior; strings can refine behavior as "min" or "max".
RetryOnStatus – list of HTTP status codes to retry (e.g., 429, 500, 502, 503, 504).
RetryOnNetworkErrors – retry on DNS/connect/reset errors (default: true).
RetryOnTimeouts – retry when client timeout occurs (default: true).

Disabling Retry

Globally: set Defaults.Retry: null or omit it entirely.
Per request: set Retry: null under that request.

Example

Defaults:
  Retry:
    Attempts: 4
    BackoffStrategy: exponential
    BackoffSeconds: 0.5
    Multiplier: 2.0
    MaxBackoffSeconds: 10
    Jitter: true
    RetryOnStatus: [429, 500, 502, 503, 504]

Sequences:
  - Name: ExampleSeq
    Type: Sequential
    Requests:
      - GetState:
          Method: GET
          URLPath: /state
          Retry: null   # disable retry here
      - GetConfig:
          Method: GET
          URLPath: /config
          # inherits Defaults.Retry with jitter enabled

Output Files & Extensions

Response formatting controls (Defaults or per request)

You may control how response bodies are written either globally under Defaults or per request. Per-request settings override Defaults.

At Defaults level:

StashConfig:
  Defaults:
    Response:
      PrettyPrint: true
      Sort: true

Per-request override:

Response:
  PrettyPrint: true    # pretty print JSON or XML responses
  Sort: true           # sort output (JSON object keys; XML element children/attributes); implies PrettyPrint

Supported types for PrettyPrint/Sort:

JSON (application/json, */json): PrettyPrint uses rich to format; Sort sorts object keys.
XML (application/xml, text/xml, +xml): PrettyPrint uses xml.dom.minidom to format; Sort sorts child elements by tag name and attributes alphabetically.
Others: ignored; body written as-is.

Each request writes one file per request, named as reqNNN-<RequestKey>-response.<ext>, where NNN is the 1-based index within its sequence. The extension is derived from the response Content‑Type.

Content-Type	Extension
application/json	.json
text/plain	.txt
text/csv	.csv
application/xml or text/xml	.xml
application/pdf	.pdf
image/*	.png/.jpg
unknown/missing	.txt

Path construction

<out>/<StashConfig.Name>/<RunTimestamp>/seqNNN-<Sequence.Name>/reqNNN-<RequestKey>-response.<ext>

Run Results CSV

Each run produces a <original-config>-results.csv file in the run’s timestamped directory. This file logs metadata for every request executed.

File path:

<out>/<StashConfig.Name>/<RunTimestamp>/<original-config>-results.csv

Columns:

sequence – the sequence name.
request – the request key.
timestamp – UTC timestamp when executed.
status – HTTP status code (or -1 if none).
duration_ms – request time in ms.
attempts – number of attempts made.

Example:

sequence,request,timestamp,status,duration_ms,attempts
seq001-GetGeneralData,req001-GetConfig,2025-09-17T15:42:11Z,200,123,1
seq001-GetGeneralData,req002-GetCatalog,2025-09-17T15:42:11Z,500,87,1
seq002-GetPlayer01,req001-GetState,2025-09-17T15:42:12Z,200,212,1
seq002-GetPlayer01,req002-GrantItem,2025-09-17T15:42:13Z,200,145,1

Run Log

Every run produces a detailed human-readable log file to aid observability and troubleshooting.

File name: -log.txt
Location: alongside the run’s resolved config and results CSV in the timestamped run directory
Created by: payloadstash run
Purpose: records high-detail, chronological information about the run, including start/end markers, configuration resolution notices, per-request progress markers, retry decisions, HTTP status summaries, and any non-fatal errors encountered. This log is intended to complement the structured -results.csv file.

File path:

<out>/<StashConfig.Name>/<RunTimestamp>/<original-config>-log.txt

Example:

out/PXXX-Tester-01/2025-09-17T15-42-10Z/PXXX-Tester-01-log.txt

CLI Usage

payloadstash run CONFIG.yml --out ./out [--dry-run] [--yes]

payloadstash validate CONFIG.yml

payloadstash resolve CONFIG.yml --out ./out

Flags:

--dry-run: Resolve and log actions without making HTTP requests.
--yes: Automatically continue without the interactive "Continue? [y/N]" prompt.

Exit codes:

0 = run success, no validation errors and all requests were considered successful.
1 = run success, but at least one http request was not successful.
9 = run not successful due to a validation error, or output write error. Output might be partial.

Note: Individual request errors will not cause a premature exit.

Validation Rules

StashConfig.Name required.
StashConfig.Defaults.URLRoot required.
StashConfig.Defaults.FlowControl required (must include DelaySeconds and TimeoutSeconds).
At least one sequence.
Each sequence must have Name, Type, and at least one Request.
Each request must have one key, Method, and URLPath.
URLRoot is not allowed inside a Request.
Headers, Body, Query must be maps.
ConcurrencyLimit is only allowed for Type=Concurrent and must be >0 if present.

Examples

Minimal

StashConfig:
  Name: MiniRun
  Defaults:
    URLRoot: https://api.example.com
  Sequences:
    - Name: OnlySeq
      Type: Sequential
      Requests:
        - Ping:
            Method: GET
            URLPath: /health

With Defaults & Forced

common: &common
  Accept: application/json

StashConfig:
  Name: WithDefaultsForced
  Defaults:
    URLRoot: https://api.example.com/v1
    Headers:
      <<: *common
      User-Agent: PayloadStash/1.0
  Forced:
    Headers:
      X-Env: prod
    Query:
      lang: en-US
  Sequences:
    - Name: SeqA
      Type: Concurrent
      ConcurrencyLimit: 3
      Requests:
        - A:
            Method: GET
            URLPath: /a
        - B:
            Method: GET
            URLPath: /b
            Headers:
              Authorization: Bearer TOKEN

Implementation Notes

Async runtime: asyncio with httpx/aiohttp.
Anchor resolution: resolve << merges, write *-resolved.yml inside timestamp folder.
URL concat: URLRoot.rstrip('/') + '/' + URLPath.lstrip('/').
Error handling: record status/body; do not abort run; log in -results.csv.
Case handling: headers case-insensitive.
Timing: capture duration_ms and timestamp per request.
Extensibility: TimeoutSeconds and Retry possible.

FAQ

Failed requests are recorded, not fatal.
Resolved config written inside timestamped folder.

Happy stashing!!️

Name		Name	Last commit message	Last commit date
Latest commit History 31 Commits
.github		.github
.idea		.idea
config		config
output		output
packaged-docker		packaged-docker
packaged-python		packaged-python
payload_stash		payload_stash
.gitignore		.gitignore
Dockerfile		Dockerfile
LICENSE		LICENSE
README - Config File Formal Specification.md		README - Config File Formal Specification.md
README.md		README.md
RELEASE-NOTES-v1.0.0.md		RELEASE-NOTES-v1.0.0.md
bootstrap.py		bootstrap.py
compose.yml		compose.yml
payloadstash		payloadstash
requirements.txt		requirements.txt
setup.py		setup.py
x-docker-build-payloadstash.sh		x-docker-build-payloadstash.sh
x-docker-load-payloadstash.sh		x-docker-load-payloadstash.sh
x-docker-package-payloadstash.sh		x-docker-package-payloadstash.sh
x-docker-run-payloadstash.sh		x-docker-run-payloadstash.sh
x-python-package-payloadstash.sh		x-python-package-payloadstash.sh

Uh oh!

License

ericwastaken/PayloadStash

Folders and files

Latest commit

History

Repository files navigation

PayloadStash – README YAML Spec

Concept

Quick Start - Native (recommended for most users)

Quick Start - Docker Execution

Exporting to an Air-Gapped server using a Docker Image

Quick Start - Developer Install

Directory Layout

Configuration Overview

Full YAML Schema (informal)

Functions & Dynamics inside configs

Secrets ($secrets) and the --secrets argument

Secrets file format

Referencing secrets in YAML

Request-time deferral with secrets

CLI usage with secrets

Docker usage

Troubleshooting

Merge & Precedence Rules

Anchors, Aliases & Header Merging

Sequences & Concurrency

Flow Control

Retry Explained

Location

Fields

Disabling Retry

Example

Output Files & Extensions

Response formatting controls (Defaults or per request)

Run Results CSV

Run Log

CLI Usage

Validation Rules

Examples

Minimal

With Defaults & Forced

Implementation Notes

FAQ

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases 2

Sponsor this project

Uh oh!

Languages