diff --git a/README.md b/README.md index 23ab0c38..1244aed5 100644 --- a/README.md +++ b/README.md @@ -10,13 +10,19 @@ Virtual bash interpreter for multi-tenant environments. Written in Rust. ## Features +- **Secure by default** - No process spawning, no filesystem access, no network access unless explicitly enabled. [60+ threats](specs/006-threat-model.md) analyzed and mitigated - **POSIX compliant** - Substantial IEEE 1003.1-2024 Shell Command Language compliance -- **Sandboxed, in-process execution** - No real filesystem access by default -- **Virtual filesystem** - InMemoryFs, OverlayFs, MountableFs -- **Resource limits** - Command count, loop iterations, function depth -- **Network allowlist** - Control HTTP access per-domain +- **Sandboxed, in-process execution** - All 150 commands reimplemented in Rust, no `fork`/`exec` +- **Virtual filesystem** - InMemoryFs, OverlayFs, MountableFs with optional RealFs backend (`realfs` feature) +- **Resource limits** - Command count, loop iterations, function depth, output size, filesystem size, parser fuel +- **Network allowlist** - HTTP access denied by default, per-domain control +- **Multi-tenant isolation** - Each interpreter instance is fully independent - **Custom builtins** - Extend with domain-specific commands +- **LLM tool contract** - `BashTool` with discovery metadata, streaming output, and system prompts +- **Scripted tool orchestration** - Compose ToolDef+callback pairs into multi-tool bash scripts (`scripted_tool` feature) +- **MCP server** - Model Context Protocol endpoint via `bashkit mcp` - **Async-first** - Built on tokio +- **Language bindings** - Python (PyO3) and JavaScript/TypeScript (NAPI-RS) for Node.js, Bun, and Deno - **Experimental: Git support** - Virtual git operations on the virtual filesystem (`git` feature) - **Experimental: Python support** - Embedded Python interpreter via [Monty](https://github.com/pydantic/monty) (`python` feature) @@ -36,7 +42,10 @@ bashkit = "0.1" Optional features: ```bash -cargo add bashkit --features git # Virtual git operations +cargo add bashkit --features git # Virtual git operations +cargo add bashkit --features python # Embedded Python interpreter +cargo add bashkit --features realfs # Real filesystem backend +cargo add bashkit --features scripted_tool # Tool orchestration framework ``` ## Quick Start @@ -102,34 +111,48 @@ assert_eq!(output.result["stdout"], "hello\nworld\n"); | Category | Commands | |----------|----------| -| Core | `echo`, `printf`, `cat`, `nl`, `read` | -| Navigation | `cd`, `pwd`, `ls`, `find`, `pushd`, `popd`, `dirs` | +| Core | `echo`, `printf`, `cat`, `nl`, `read`, `mapfile`, `readarray` | +| Navigation | `cd`, `pwd`, `ls`, `tree`, `find`, `pushd`, `popd`, `dirs` | | Flow control | `true`, `false`, `exit`, `return`, `break`, `continue`, `test`, `[` | -| Variables | `export`, `set`, `unset`, `local`, `shift`, `source`, `.`, `eval`, `readonly`, `times`, `declare`, `typeset`, `let` | -| Shell | `bash`, `sh` (virtual re-invocation), `:`, `trap`, `caller`, `getopts`, `shopt` | -| Text processing | `grep`, `sed`, `awk`, `jq`, `head`, `tail`, `sort`, `uniq`, `cut`, `tr`, `wc`, `paste`, `column`, `diff`, `comm`, `strings`, `tac`, `rev`, `seq`, `expr` | -| File operations | `mkdir`, `mktemp`, `rm`, `cp`, `mv`, `touch`, `chmod`, `chown`, `ln`, `rmdir`, `realpath` | +| Variables | `export`, `set`, `unset`, `local`, `shift`, `source`, `.`, `eval`, `readonly`, `times`, `declare`, `typeset`, `let`, `alias`, `unalias` | +| Shell | `bash`, `sh` (virtual re-invocation), `:`, `trap`, `caller`, `getopts`, `shopt`, `command`, `type`, `which`, `hash`, `compgen`, `fc`, `help` | +| Text processing | `grep`, `rg`, `sed`, `awk`, `jq`, `head`, `tail`, `sort`, `uniq`, `cut`, `tr`, `wc`, `paste`, `column`, `diff`, `comm`, `strings`, `tac`, `rev`, `seq`, `expr`, `fold`, `expand`, `unexpand`, `join`, `iconv` | +| File operations | `mkdir`, `mktemp`, `mkfifo`, `rm`, `cp`, `mv`, `touch`, `chmod`, `chown`, `ln`, `rmdir`, `realpath`, `readlink`, `split` | | File inspection | `file`, `stat`, `less` | -| Archives | `tar`, `gzip`, `gunzip` | -| Byte tools | `od`, `xxd`, `hexdump` | -| Utilities | `sleep`, `date`, `basename`, `dirname`, `timeout`, `wait`, `watch`, `yes`, `kill` | +| Archives | `tar`, `gzip`, `gunzip`, `zip`, `unzip` | +| Byte tools | `od`, `xxd`, `hexdump`, `base64` | +| Checksums | `md5sum`, `sha1sum`, `sha256sum` | +| Utilities | `sleep`, `date`, `basename`, `dirname`, `timeout`, `wait`, `watch`, `yes`, `kill`, `bc`, `clear` | | Disk | `df`, `du` | | Pipeline | `xargs`, `tee` | | System info | `whoami`, `hostname`, `uname`, `id`, `env`, `printenv`, `history` | -| Network | `curl`, `wget` (requires allowlist) | +| Data formats | `csv`, `json`, `yaml`, `tomlq`, `template`, `envsubst` | +| Network | `curl`, `wget` (requires allowlist), `http` | +| DevOps | `assert`, `dotenv`, `glob`, `log`, `retry`, `semver`, `verify`, `parallel`, `patch` | | Experimental | `python`, `python3` (requires `python` feature), `git` (requires `git` feature) | ## Shell Features -- Variables and parameter expansion (`$VAR`, `${VAR:-default}`, `${#VAR}`) -- Command substitution (`$(cmd)`) -- Arithmetic expansion (`$((1 + 2))`) -- Pipelines and redirections (`|`, `>`, `>>`, `<`, `<<<`, `2>&1`) -- Control flow (`if`/`elif`/`else`, `for`, `while`, `case`) -- Functions (POSIX and bash-style) -- Arrays (`arr=(a b c)`, `${arr[@]}`, `${#arr[@]}`) -- Glob expansion (`*`, `?`) -- Here documents (`<`, `>>`, `<`, `<<<`, `2>&1`, `&>`) +- Control flow (`if`/`elif`/`else`, `for`, `while`, `until`, `case` with `;;`/`;&`/`;;&`, `select`) +- Functions (POSIX and bash-style) with dynamic scoping, FUNCNAME stack, `caller` +- Indexed arrays (`arr=(a b c)`, `${arr[@]}`, `${#arr[@]}`, slicing, `+=`) +- Associative arrays (`declare -A map=([key]=val)`) +- Nameref variables (`declare -n`) +- Brace expansion (`{a,b,c}`, `{1..10}`, `{01..05}`) +- Glob expansion (`*`, `?`) and extended globs (`@()`, `?()`, `*()`, `+()`, `!()`) +- Glob options (`dotglob`, `nullglob`, `failglob`, `nocaseglob`, `globstar`) +- Here documents (`<(cmd)`) +- Coprocesses (`coproc`) +- Background execution (`&`) with `wait` +- Shell options (`set -euxo pipefail`, `shopt`) +- Alias expansion +- Trap handling (`trap cmd EXIT`, `trap cmd ERR`) +- `[[ ]]` conditionals with regex matching (`=~`, BASH_REMATCH) ## Configuration @@ -244,10 +267,17 @@ mountable.mount("/data", Arc::new(InMemoryFs::new())); ```bash # Run a script -bashkit-cli run script.sh +bashkit run script.sh # Interactive REPL -bashkit-cli repl +bashkit repl + +# MCP server (Model Context Protocol) +bashkit mcp + +# Mount real filesystem (read-only or read-write) +bashkit run script.sh --mount-ro /data +bashkit run script.sh --mount-rw /workspace ``` ## Development @@ -291,7 +321,9 @@ just bench-list # List all benchmarks See [crates/bashkit-bench/README.md](crates/bashkit-bench/README.md) for methodology and assumptions. -## Python Bindings +## Language Bindings + +### Python Python bindings with LangChain integration are available in [crates/bashkit-python](crates/bashkit-python/README.md). @@ -305,9 +337,49 @@ result = await tool.execute("echo 'Hello, World!'") print(result.stdout) ``` +### JavaScript / TypeScript + +NAPI-RS bindings for Node.js, Bun, and Deno. Available as `@everruns/bashkit` on npm. + +```typescript +import { BashTool } from '@everruns/bashkit'; + +const tool = new BashTool({ username: 'agent', hostname: 'sandbox' }); +const result = await tool.execute("echo 'Hello, World!'"); +console.log(result.stdout); + +// Direct VFS access +await tool.writeFile('/tmp/data.txt', 'hello'); +const content = await tool.readFile('/tmp/data.txt'); +``` + +Platform matrix: macOS (x86_64, aarch64), Linux (x86_64, aarch64), Windows (x86_64), WASM. +See [crates/bashkit-js](crates/bashkit-js/) for details. + ## Security -Bashkit is designed as a virtual interpreter with sandboxed execution for untrusted scripts. See the [security policy](SECURITY.md) for reporting vulnerabilities and the [threat model](specs/006-threat-model.md) for detailed analysis of 60+ identified threats. +Bashkit is built for running untrusted scripts from AI agents and users. Security is a core design goal, not an afterthought. + +### Defense in Depth + +| Layer | Protection | +|-------|------------| +| **No process spawning** | All 150 commands are reimplemented in Rust — no `fork`, `exec`, or shell escape | +| **Virtual filesystem** | Scripts see an in-memory FS by default; no host filesystem access unless explicitly mounted | +| **Network allowlist** | HTTP access is denied by default; each domain must be explicitly allowed | +| **Resource limits** | Configurable caps on commands (10K), loop iterations (100K), function depth (100), output (10MB), input (10MB) | +| **Filesystem limits** | Max total bytes (100MB), max file size (10MB), max file count (10K) — prevents zip bombs, tar bombs, and append floods | +| **Parser limits** | Timeout (5s), fuel budget (100K ops), AST depth (100) — prevents pathological input from hanging the interpreter | +| **Multi-tenant isolation** | Each `Bash` instance is fully isolated — no shared state between tenants | +| **Panic recovery** | All builtins wrapped in `catch_unwind` — a panic in one command doesn't crash the host | +| **Path traversal prevention** | RealFs backend canonicalizes paths to prevent `../../etc/passwd` escapes | +| **Unicode security** | 68 byte-boundary tests across builtins; zero-width character rejection in VFS paths | + +### Threat Model + +60+ identified threats across 11 categories (DoS, sandbox escape, info disclosure, injection, network, isolation, internal errors, git, logging, Python, Unicode) — each with a stable ID, mitigation status, and test coverage. + +See the [threat model](specs/006-threat-model.md) for the full analysis and [security policy](SECURITY.md) for reporting vulnerabilities. ## Other Virtual Bash Implementations diff --git a/crates/bashkit-js/README.md b/crates/bashkit-js/README.md index 864e0b22..1948c985 100644 --- a/crates/bashkit-js/README.md +++ b/crates/bashkit-js/README.md @@ -1,23 +1,41 @@ # @everruns/bashkit -Sandboxed bash interpreter for JavaScript/TypeScript. Native NAPI-RS bindings to the [bashkit](https://github.com/everruns/bashkit) Rust core. +Sandboxed bash interpreter for JavaScript/TypeScript. Native NAPI-RS bindings to the [bashkit](https://github.com/everruns/bashkit) Rust core. Works with Node.js, Bun, and Deno. ## Install ```bash -npm install @everruns/bashkit +npm install @everruns/bashkit # Node.js +bun add @everruns/bashkit # Bun +deno add npm:@everruns/bashkit # Deno ``` +## Features + +- **Sandboxed execution** — all commands run in-process with a virtual filesystem, no containers needed +- **150 built-in commands** — echo, cat, grep, sed, awk, jq, curl, find, and more +- **Full bash syntax** — variables, pipelines, redirects, loops, functions, arrays +- **Resource limits** — protect against infinite loops and runaway scripts +- **Sync and async APIs** — `executeSync()` and `execute()` (Promise-based) +- **Virtual filesystem access** — read, write, mkdir, glob directly from JS +- **Cancellation** — `cancel()` and `AbortSignal` support +- **Scripted tool orchestration** — compose JS callbacks as bash builtins via `ScriptedTool` +- **LLM tool contract** — `BashTool` with discovery metadata, schemas, and system prompts + ## Usage ```typescript -import { Bash, BashTool, getVersion } from '@everruns/bashkit'; +import { Bash, BashTool, ScriptedTool, getVersion } from '@everruns/bashkit'; // Basic usage const bash = new Bash(); const result = bash.executeSync('echo "Hello, World!"'); console.log(result.stdout); // Hello, World!\n +// Async +const r = await bash.execute('echo "async!"'); +console.log(r.stdout); // async!\n + // State persists between calls bash.executeSync('X=42'); bash.executeSync('echo $X'); // stdout: 42\n @@ -30,8 +48,90 @@ console.log(tool.description()); // Token-efficient tool description console.log(tool.help()); // Markdown help document console.log(tool.systemPrompt()); // Compact system prompt -const r = tool.executeSync('echo hello'); -console.log(r.stdout); // hello\n +const tr = tool.executeSync('echo hello'); +console.log(tr.stdout); // hello\n +``` + +### Virtual Filesystem + +Read, write, and inspect files directly without executing bash commands: + +```typescript +const bash = new Bash(); +bash.writeFile('/data/config.json', '{"key": "value"}'); +const content = bash.readFile('/data/config.json'); +bash.mkdir('/data/subdir', true); // recursive +bash.exists('/data/config.json'); // true +bash.remove('/data/subdir', true); // recursive +bash.ls('/data'); // string[] +bash.glob('**/*.json'); // string[] +``` + +### File Mounts + +Mount files at construction time with strings, sync functions, or async functions: + +```typescript +const bash = new Bash({ + files: { + '/config.json': '{"key": "value"}', + '/lazy.txt': () => 'computed on first read', + '/async.txt': async () => fetchContent(), + }, +}); + +// For async file providers, use the static factory +const bash2 = await Bash.create({ + files: { '/data.txt': async () => loadData() }, +}); +``` + +### Cancellation + +```typescript +const bash = new Bash(); + +// Cancel method +const promise = bash.execute('sleep 60'); +bash.cancel(); + +// AbortSignal +const controller = new AbortController(); +const promise2 = bash.execute('sleep 60', { signal: controller.signal }); +controller.abort(); +``` + +### Error Handling + +```typescript +import { BashError } from '@everruns/bashkit'; + +// Throws BashError on non-zero exit +try { + bash.executeSyncOrThrow('exit 1'); +} catch (e) { + if (e instanceof BashError) { + console.log(e.exitCode); // 1 + console.log(e.stderr); + } +} + +// Async variant +await bash.executeOrThrow('false'); +``` + +### ScriptedTool + +Compose JS callbacks as bash builtins — an LLM writes a single bash script that pipes, loops, and branches across all registered tools: + +```typescript +const tool = new ScriptedTool({ name: 'api' }); +tool.addTool('get_user', 'Fetch user by ID', (params) => { + return JSON.stringify({ id: params.id, name: 'Alice' }); +}); + +const result = tool.executeSync("get_user --id 1 | jq -r '.name'"); +console.log(result.stdout); // Alice ``` ## API @@ -41,15 +141,25 @@ console.log(r.stdout); // hello\n Core interpreter with virtual filesystem. - `new Bash(options?)` — create instance +- `Bash.create(options?)` — async factory for async file providers - `executeSync(commands)` — run bash commands, returns `ExecResult` -- `executeSyncOrThrow(commands)` — run bash commands, throws `BashError` on non-zero exit +- `executeSyncOrThrow(commands)` — run, throws `BashError` on non-zero exit +- `execute(commands)` — async execution, returns `Promise` +- `executeOrThrow(commands)` — async, throws `BashError` on non-zero exit +- `cancel()` — cancel running execution - `reset()` — clear state, preserve config +- `readFile(path)` — read file as string +- `writeFile(path, content)` — write/overwrite file +- `mkdir(path, recursive?)` — create directory +- `exists(path)` — check path exists +- `remove(path, recursive?)` — delete file/directory +- `ls(path?)` — list directory contents +- `glob(pattern)` — find files by pattern ### `BashTool` -Interpreter + tool-contract metadata. +Interpreter + tool-contract metadata. All `Bash` methods, plus: -- All `Bash` methods, plus: - `name` — tool name (`"bashkit"`) - `version` — version string - `shortDescription` — one-liner @@ -59,6 +169,18 @@ Interpreter + tool-contract metadata. - `inputSchema()` — JSON input schema - `outputSchema()` — JSON output schema +### `ScriptedTool` + +Multi-tool orchestration — register JS callbacks as bash builtins. + +- `new ScriptedTool(options)` — create with name, shortDescription, limits +- `addTool(name, description, callback, schema?)` — register a tool +- `executeSync(script)` / `execute(script)` — run script +- `executeSyncOrThrow(script)` / `executeOrThrow(script)` — run, throw on error +- `env(key, value)` — set environment variable +- `toolCount()` — number of registered tools +- Tool metadata: `name`, `shortDescription`, `version`, `description()`, `help()`, `systemPrompt()`, `inputSchema()`, `outputSchema()` + ### `BashOptions` ```typescript @@ -67,6 +189,9 @@ interface BashOptions { hostname?: string; maxCommands?: number; maxLoopIterations?: number; + files?: Record string) | (() => Promise)>; + python?: boolean; + externalFunctions?: string[]; } ``` @@ -76,11 +201,28 @@ interface BashOptions { interface ExecResult { stdout: string; stderr: string; - exit_code: number; + exitCode: number; + success: boolean; error?: string; + stdoutTruncated?: boolean; + stderrTruncated?: boolean; } ``` +### `BashError` + +```typescript +class BashError extends Error { + exitCode: number; + stderr: string; + display(): string; +} +``` + +### `getVersion()` + +Returns the bashkit version string. + ## Platform Support | OS | Architecture | diff --git a/crates/bashkit-python/README.md b/crates/bashkit-python/README.md index 9cfffc21..115d7e30 100644 --- a/crates/bashkit-python/README.md +++ b/crates/bashkit-python/README.md @@ -88,6 +88,15 @@ fs = bash.fs() fs.mkdir("/data", recursive=True) fs.write_file("/data/blob.bin", b"\x00\x01hello") assert fs.read_file("/data/blob.bin") == b"\x00\x01hello" + +# Additional filesystem operations +fs.append_file("/data/blob.bin", b" world") +fs.copy("/data/blob.bin", "/data/backup.bin") +fs.rename("/data/backup.bin", "/data/copy.bin") +info = fs.stat("/data/blob.bin") # dict with size, type, etc. +entries = fs.read_dir("/data") # detailed directory listing +fs.symlink("/data/link", "/data/blob.bin") +fs.chmod("/data/blob.bin", 0o644) ``` ### Live Mounts @@ -159,6 +168,13 @@ bash_tool = create_bash_tool() # Use with any PydanticAI agent ``` +### Deep Agents + +```python +from bashkit.deepagents import BashkitBackend, BashkitMiddleware +# Use with Deep Agents framework +``` + ## ScriptedTool — Multi-Tool Orchestration Compose Python callbacks as bash builtins. An LLM writes a single bash script that pipes, loops, and branches across all registered tools. @@ -191,7 +207,10 @@ print(result.stdout) # Alice - `execute(commands: str) -> ExecResult` — execute commands asynchronously - `execute_sync(commands: str) -> ExecResult` — execute commands synchronously +- `execute_or_throw(commands: str) -> ExecResult` — async, raises on non-zero exit +- `execute_sync_or_throw(commands: str) -> ExecResult` — sync, raises on non-zero exit - `reset()` — reset interpreter state +- `fs() -> FileSystem` — direct filesystem access ### BashTool @@ -212,12 +231,31 @@ Convenience wrapper for AI agents. Inherits all execution methods from `Bash`, p - `success: bool` — True if exit_code == 0 - `to_dict() -> dict` — convert to dictionary +### FileSystem + +- `mkdir(path, recursive=False)` — create directory +- `write_file(path, content)` — write bytes to file +- `read_file(path) -> bytes` — read file contents +- `append_file(path, content)` — append bytes to file +- `exists(path) -> bool` — check if path exists +- `remove(path, recursive=False)` — delete file/directory +- `stat(path) -> dict` — file metadata (size, type, etc.) +- `read_dir(path) -> list` — detailed directory listing +- `rename(src, dst)` — rename file/directory +- `copy(src, dst)` — copy file +- `symlink(link, target)` — create symlink +- `chmod(path, mode)` — change file permissions +- `read_link(path) -> str` — read symlink target + ### ScriptedTool - `add_tool(name, description, callback, schema=None)` — register a tool - `execute(script: str) -> ExecResult` — execute script asynchronously - `execute_sync(script: str) -> ExecResult` — execute script synchronously +- `execute_or_throw(script: str) -> ExecResult` — async, raises on non-zero exit +- `execute_sync_or_throw(script: str) -> ExecResult` — sync, raises on non-zero exit - `env(key: str, value: str)` — set environment variable +- `tool_count() -> int` — number of registered tools ## How it works