Skip to content

01_cheatsheet.md

Latest commit

 

History

History
200 lines (158 loc) · 12.8 KB

01_cheatsheet.md

File metadata and controls

200 lines (158 loc) · 12.8 KB

Spikee CLI Cheatsheet

This is a quick CLI reference guide for using Spikee, containing commands, arguments and links to more detailed documentation.

Jump to Links:

spikee init

Initializes a new Spikee project in the current directory.

mkdir workspace && cd workspace
spikee init
Argument Description Docs
--force Overwrite existing directories
--include-builtin Copy built-in modules to local workspace
--include-viewer Include the built-in web viewer in the local workspace Link

spikee list

Lists all local and built-in modules available. Use --description to include module descriptions.

spikee list seeds
spikee list datasets
spikee list targets
spikee list plugins
spikee list attacks
spikee list judges
spikee list providers --description

spikee generate

Generates a dataset from a seed folder.

spikee generate --seed-folder my_seeds
Source Arguments Description Docs
--seed-folder Required Path of seed folder
--include-standalone-inputs Include standalone_user_inputs.jsonl Link
--include-system-message Includes system_messages.toml Link
--tag Include an identifying tag in the dataset filename
Transformation Arguments Description Docs
--plugins Space-separated list of plugins Link
--plugin-options Apply options as "plugin1:option1=value1,option2=value2;plugin2:option2=value2"
--plugin-only Only output plugin entries
--include-fixes Comma-separated list (e.g., 'adv_prefixes', 'adv_suffixes', prefixes=, suffixes=) Link
Formatting Arguments Description Docs
--format Output format: user-input (default / apps), full-prompt (LLMs), or burp
--languages Comma-separated list of languages to filter jailbreaks and instructions (e.g., en) Link
--match-languages Only combine jailbreaks and instructions with matching languages (default: True) Link
--positions Position to insert jailbreaks (start, middle, end). ignored if <PLACEHOLDER> is present Link
--injection-delimiters Delimiters to use when injecting jailbreaks (default: \nINJECTION_PAYLOAD\n) Link
--spotlighting-data-markers Comma-separated list of data markers (placeholder: "DOCUMENT") Link
--instruction-filter Comma-separated list of instruction types to include Link
--jailbreak-filter Comma-separated list of jailbreak types to include Link

spikee test

Tests a dataset against a target.

spikee test --dataset my_dataset.jsonl --target my_target
Dataset Arguments Description Docs
--dataset Path to a dataset file Link
--dataset-folder Path to a dataset folder containing multiple dataset files

(NB, These args can be used multiple times to specify multiple datasets, at least one of --dataset or --dataset-folder is required)

Module Arguments Description Docs
--target Required Name of the target to test (local or built-in) Link
--target-options Options to pass to the target
--judge-options Options to pass to the judge Link
Testing Arguments Description Default Docs
--threads Number of threads for parallel processing 4 Link
--attempts Number of attempts per entry 1 Link
--max-retries Number of retries for rate-limiting controls / 429 errors 3 Link
--throttle Time in seconds to wait between entries per thread 0 Link
--sample Only test a sample of a dataset (e.g., 0.15 for 15%) 1 Link
--sample-seed Seed for random sampling 42 Link
--tag Include a tag at the end of the results filename
Attack Arguments Description Docs
--attack Name of the attack module Link
--attack-iterations Number of attack iterations / maximum number of turns per dataset entry Link
--attack-options Options to pass to the attack module Link
--attack-only Only run the attack module without standard attempts Link

(NB, attacks are only run if the core test flow fails, or --attack-only is set)

Resume Arguments Description Docs
--resume-file Path to a results JSONL file to resume from. Only works with a single dataset Link
--auto-resume Silently pick the latest matching results file if present Link
--no-auto-resume Create new results file, do not attempt to resume Link

spikee results

analyze

Analyzes results files to produce aggregate statistics and visualizations - Link

spikee results analyze --results-file results.jsonl
Argument Description
--results-file Path to a results JSONL file
--results-folder Path to a folder containing multiple results JSONL files
--false-positive-checks Path to a JSONL file with benign prompts for false positive analysis. Only works with a single dataset.
--output-format Output format: console (default) or html
--overview Only output the general statistics of results files
--combine Combine results from multiple files into a single analysis.

(NB, Result file/folder args can be used multiple times to specify multiple datasets, but at least one is required)

rejudge

Rejudges a results file - Link

spikee results rejudge --results-file results.jsonl
Argument Description
--results-file Path to a results JSONL file
--results-folder Path to a folder containing multiple results JSONL files
--judge-options Options to pass to the judge
--resume Attempt to resume the most recent re-judge file

(NB, Result file/folder args can be used multiple times to specify multiple datasets, but at least one is required)

extract

Searches for result entries by category or search term and extracts them to another JSONL file - Link

spikee results extract --results-file results.jsonl --category "success"
Argument Description
--results-file Path to a results JSONL file
--results-folder Path to a folder containing multiple results JSONL files
--category Extracts prompts by category: success (default), failure, error, guardrail, no-guardrail, custom
--custom-search Custom search string to filter prompts when --category=custom. Formats: 'search_string', 'field:search_string' or '!search_string' to invert match
--tag Include a tag at the end of the results filename

(NB, Result file/folder args can be used multiple times to specify multiple datasets, but at least one is required)

spikee viewer

Launches the Spikee web viewer.

Viewer is a WIP, currently only the results viewer is available.

Common Arguments

Argument Description
-h, --host Host address for the web viewer (default: 127.0.0.1)
-p, --port Port number for the web viewer (default: 8080)
-d, --debug Enable debug mode for the web viewer (default: False)
--truncate Truncate long fields in the web viewer for better readability (default: 500 characters, disable with 0)

Usage

spikee viewer <common args> results <results-specific args>

results Specific Arguments - Link

Argument Description
--result-file Path to a results JSONL file, generated using the dataset
--result-folder Path to a results folder containing multiple JSONL files, generated using the dataset
--allow-ast Allow AST parsing in the web viewer (use with caution)

(NB, Result file/folder args can be used multiple times to specify multiple datasets, but at least one is required)

spikee viewer -p 8081 results --result-folder .\results\