diff --git a/CHANGELOG.md b/CHANGELOG.md index fa5d6541..3720c424 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,66 @@ # Changelog +## [Unreleased] + +### Highlights + +- 20+ new builtins: `declare`/`typeset`, `let`, `getopts`, `trap`, `caller`, `shopt`, `pushd`/`popd`/`dirs`, `seq`, `tac`, `rev`, `yes`, `expr`, `mktemp`, `realpath`, and more +- Glob options: `dotglob`, `nocaseglob`, `failglob`, `noglob`, `globstar` +- Shell flags: `bash -e/-x/-u/-f/-o`, `set -x` xtrace debugging +- `select` construct, `case ;;&` fallthrough, `FUNCNAME` variable +- Nameref variables (`declare -n`), case conversion (`declare -l/-u`) +- 10+ bug fixes for quoting, arrays, globs, and redirections + +### What's Changed + +* feat(interpreter): implement bash/sh -e/-x/-u/-f/-o flags ([#270](https://github.com/everruns/bashkit/pull/270)) +* chore(eval): run 2026-02-25 evals across 4 models ([#271](https://github.com/everruns/bashkit/pull/271)) +* feat(interpreter): implement glob options (dotglob, nocaseglob, failglob, noglob, globstar) ([#269](https://github.com/everruns/bashkit/pull/269)) +* feat(builtins): implement pushd, popd, dirs ([#268](https://github.com/everruns/bashkit/pull/268)) +* feat(builtins): implement file comparison test operators ([#267](https://github.com/everruns/bashkit/pull/267)) +* feat(builtins): implement expr builtin ([#266](https://github.com/everruns/bashkit/pull/266)) +* feat(builtins): implement yes and realpath builtins ([#265](https://github.com/everruns/bashkit/pull/265)) +* feat(interpreter): implement caller builtin ([#264](https://github.com/everruns/bashkit/pull/264)) +* feat(builtins): implement printf %q shell quoting ([#263](https://github.com/everruns/bashkit/pull/263)) +* feat(builtins): implement tac and rev builtins ([#262](https://github.com/everruns/bashkit/pull/262)) +* feat(builtins): implement seq builtin ([#261](https://github.com/everruns/bashkit/pull/261)) +* chore(deps): bump pyo3 to 0.28.2 and pyo3-async-runtimes to 0.28 ([#260](https://github.com/everruns/bashkit/pull/260)) +* feat(builtins): implement mktemp builtin ([#259](https://github.com/everruns/bashkit/pull/259)) +* feat(interpreter): implement trap -p flag and sorted trap listing ([#258](https://github.com/everruns/bashkit/pull/258)) +* feat(builtins): implement set -o / set +o option display ([#257](https://github.com/everruns/bashkit/pull/257)) +* feat(interpreter): implement declare -l/-u case conversion attributes ([#256](https://github.com/everruns/bashkit/pull/256)) +* feat(interpreter): implement declare -n nameref variables ([#255](https://github.com/everruns/bashkit/pull/255)) +* feat(builtins): implement shopt builtin with nullglob enforcement ([#254](https://github.com/everruns/bashkit/pull/254)) +* feat(interpreter): implement set -x xtrace debugging ([#253](https://github.com/everruns/bashkit/pull/253)) +* feat(bash): auto-populate shell variables (PWD, HOME, USER, etc.) ([#252](https://github.com/everruns/bashkit/pull/252)) +* feat(bash): implement select construct ([#251](https://github.com/everruns/bashkit/pull/251)) +* feat(bash): implement let builtin and fix declare -i arithmetic ([#250](https://github.com/everruns/bashkit/pull/250)) +* feat(bash): case ;& and ;;& fallthrough/continue-matching ([#249](https://github.com/everruns/bashkit/pull/249)) +* feat(bash): implement FUNCNAME special variable ([#248](https://github.com/everruns/bashkit/pull/248)) +* fix(bash): backslash-newline line continuation in double quotes ([#247](https://github.com/everruns/bashkit/pull/247)) +* fix(bash): nested double quotes inside $() in double-quoted strings ([#246](https://github.com/everruns/bashkit/pull/246)) +* fix(bash): input redirections on compound commands ([#245](https://github.com/everruns/bashkit/pull/245)) +* fix(bash): glob pattern matching in [[ == ]] and [[ != ]] ([#244](https://github.com/everruns/bashkit/pull/244)) +* fix(bash): negative array indexing ${arr[-1]} ([#243](https://github.com/everruns/bashkit/pull/243)) +* fix(bash): BASH_REMATCH not populated when regex starts with parens ([#242](https://github.com/everruns/bashkit/pull/242)) +* feat(bash): arithmetic exponentiation, base literals, mapfile ([#241](https://github.com/everruns/bashkit/pull/241)) +* feat: grep binary detection, awk %.6g and sorted for-in ([#240](https://github.com/everruns/bashkit/pull/240)) +* feat: bash compatibility — compound arrays, grep -f, awk getline, jq env/input ([#238](https://github.com/everruns/bashkit/pull/238)) +* feat: string ops, read -r, heredoc tests ([#237](https://github.com/everruns/bashkit/pull/237)) +* feat: associative arrays, chown/kill builtins, array slicing tests ([#236](https://github.com/everruns/bashkit/pull/236)) +* feat: cat -v, sort -m, brace/date/lexer fixes ([#234](https://github.com/everruns/bashkit/pull/234)) +* feat: type/which/declare/ln builtins, errexit, nounset fix, sort -z, cut -z ([#233](https://github.com/everruns/bashkit/pull/233)) +* feat: paste, command, getopts, nounset, [[ =~ ]], glob **, backtick subst ([#232](https://github.com/everruns/bashkit/pull/232)) +* feat(date): add -R, -I flags and %N format ([#231](https://github.com/everruns/bashkit/pull/231)) +* fix(lexer): handle backslash-escaped metacharacters ([#230](https://github.com/everruns/bashkit/pull/230)) +* feat(grep): add --include/--exclude glob patterns ([#229](https://github.com/everruns/bashkit/pull/229)) +* feat(sort,uniq,cut,tr): add sort/uniq/cut/tr missing options ([#228](https://github.com/everruns/bashkit/pull/228)) +* feat(sed): grouped commands, branching, Q quit, step/zero addresses ([#227](https://github.com/everruns/bashkit/pull/227)) +* chore(deps): upgrade monty to latest main (87f8f31) ([#226](https://github.com/everruns/bashkit/pull/226)) +* fix(ci): repair nightly CI and add fuzz compile guard ([#225](https://github.com/everruns/bashkit/pull/225)) + +**Full Changelog**: https://github.com/everruns/bashkit/compare/v0.1.6...HEAD + ## [0.1.6] - 2026-02-20 ### Highlights diff --git a/README.md b/README.md index 31b4667d..6deec782 100644 --- a/README.md +++ b/README.md @@ -43,23 +43,23 @@ async fn main() -> anyhow::Result<()> { -## Built-in Commands (85) +## Built-in Commands (100+) | Category | Commands | |----------|----------| | Core | `echo`, `printf`, `cat`, `nl`, `read` | -| Navigation | `cd`, `pwd`, `ls`, `find` | +| Navigation | `cd`, `pwd`, `ls`, `find`, `pushd`, `popd`, `dirs` | | Flow control | `true`, `false`, `exit`, `return`, `break`, `continue`, `test`, `[` | -| Variables | `export`, `set`, `unset`, `local`, `shift`, `source`, `.`, `eval`, `readonly`, `times` | -| Text processing | `grep`, `sed`, `awk`, `jq`, `head`, `tail`, `sort`, `uniq`, `cut`, `tr`, `wc`, `paste`, `column`, `diff`, `comm`, `strings` | -| File operations | `mkdir`, `rm`, `cp`, `mv`, `touch`, `chmod`, `rmdir` | +| Variables | `export`, `set`, `unset`, `local`, `shift`, `source`, `.`, `eval`, `readonly`, `times`, `declare`, `typeset`, `let` | +| Shell | `bash`, `sh` (virtual re-invocation), `:`, `trap`, `caller`, `getopts`, `shopt` | +| Text processing | `grep`, `sed`, `awk`, `jq`, `head`, `tail`, `sort`, `uniq`, `cut`, `tr`, `wc`, `paste`, `column`, `diff`, `comm`, `strings`, `tac`, `rev`, `seq`, `expr` | +| File operations | `mkdir`, `mktemp`, `rm`, `cp`, `mv`, `touch`, `chmod`, `chown`, `ln`, `rmdir`, `realpath` | | File inspection | `file`, `stat`, `less` | | Archives | `tar`, `gzip`, `gunzip` | | Byte tools | `od`, `xxd`, `hexdump` | -| Utilities | `sleep`, `date`, `basename`, `dirname`, `timeout`, `wait`, `watch` | +| Utilities | `sleep`, `date`, `basename`, `dirname`, `timeout`, `wait`, `watch`, `yes`, `kill` | | Disk | `df`, `du` | | Pipeline | `xargs`, `tee` | -| Shell | `bash`, `sh` (virtual re-invocation), `:` | | System info | `whoami`, `hostname`, `uname`, `id`, `env`, `printenv`, `history` | | Network | `curl`, `wget` (requires allowlist) | | Experimental | `python`, `python3` (requires `python` feature), `git` (requires `git` feature) | diff --git a/crates/bashkit/docs/compatibility.md b/crates/bashkit/docs/compatibility.md index d8894460..fb807b72 100644 --- a/crates/bashkit/docs/compatibility.md +++ b/crates/bashkit/docs/compatibility.md @@ -125,22 +125,37 @@ for sandbox security reasons. See the compliance spec for details. | `xxd` | `-l`, `-s`, `-c`, `-g`, `-p` | Hex dump | | `hexdump` | `-C`, `-n`, `-s` | Display file in hex+ASCII | +### Recently Added + +| Builtin | Flags / Arguments | Notes | +|---------|-------------------|-------| +| `ln` | `-s`, `-f` | Create links | +| `chown` | `OWNER[:GROUP] FILE` | Change ownership (virtual) | +| `kill` | `-SIGNAL PID` | Send signals (virtual) | +| `trap` | `COMMAND SIGNAL...`, `-p`, `-l` | Signal/event handlers | +| `type` | `NAME...` | Describe command type | +| `which` | `NAME...` | Locate a command | +| `command` | `-v`, `NAME...` | Run or identify commands | +| `hash` | (none) | No-op in sandboxed env | +| `declare`/`typeset` | `-i`, `-r`, `-x`, `-a`, `-p`, `-n`, `-l`, `-u` | Variable attributes | +| `let` | `EXPR...` | Evaluate arithmetic | +| `getopts` | `OPTSTRING NAME` | Parse positional parameters | +| `caller` | `[FRAME]` | Display call stack frame | +| `shopt` | `-s`, `-u`, `-q` | Shell options | +| `seq` | `[FIRST [INCR]] LAST` | Print number sequence | +| `tac` | (none) | Reverse file lines | +| `rev` | (none) | Reverse characters per line | +| `yes` | `[STRING]` | Output repeated string | +| `expr` | `EXPRESSION` | Evaluate expressions | +| `mktemp` | `-d`, `-p`, `-t` | Create temporary files | +| `realpath` | `PATH` | Resolve path | +| `pushd`/`popd`/`dirs` | standard flags | Directory stack | + ### Not Implemented | Builtin | Priority | Status | |---------|----------|--------| -| `ln` | Low | - | -| `chown` | Low | - | -| `kill` | Low | - | | `exec` | N/A | Security: intentionally excluded | -| `trap` | N/A | Security: intentionally excluded | -| `type` | Low | - | -| `which` | Low | - | -| `command` | Medium | POSIX utility | -| `hash` | Low | - | -| `declare` | Low | Bash extension | -| `typeset` | Low | Bash extension | -| `getopts` | Medium | POSIX utility | --- diff --git a/crates/bashkit/docs/threat-model.md b/crates/bashkit/docs/threat-model.md index 27efe7a1..bca5db6d 100644 --- a/crates/bashkit/docs/threat-model.md +++ b/crates/bashkit/docs/threat-model.md @@ -32,7 +32,7 @@ through configurable limits. | Recursion (TM-DOS-020) | `f() { f; }; f` | `max_function_depth` | [`limits.rs`][limits] | | Parser depth (TM-DOS-022) | `(((((...))))))` nesting | `max_ast_depth` + hard cap (100) | [`parser/mod.rs`][parser] | | Command sub depth (TM-DOS-021) | `$($($($())))` nesting | Inherited depth/fuel from parent | [`parser/mod.rs`][parser] | -| Arithmetic depth (TM-DOS-026) | `$(((((...))))))` | `MAX_ARITHMETIC_DEPTH` (200) | [`interpreter/mod.rs`][interp] | +| Arithmetic depth (TM-DOS-026) | `$(((((...))))))` | `MAX_ARITHMETIC_DEPTH` (50) | [`interpreter/mod.rs`][interp] | | Parser attack (TM-DOS-024) | Malformed input | `parser_timeout` | [`limits.rs`][limits] | | Filesystem bomb (TM-DOS-007) | Zip bomb extraction | `FsLimits` | [`fs/limits.rs`][fslimits] | | Many files (TM-DOS-006) | Create 1M files | `max_file_count` | [`fs/limits.rs`][fslimits] | @@ -308,7 +308,7 @@ attacks: This prevents attackers from bypassing depth limits through nested substitutions. 4. **Arithmetic depth limit** (TM-DOS-026): The arithmetic evaluator (`$((expr))`) - has its own depth limit (`MAX_ARITHMETIC_DEPTH = 200`) to prevent stack overflow + has its own depth limit (`MAX_ARITHMETIC_DEPTH = 50`) to prevent stack overflow from deeply nested parenthesized expressions. 5. **Parser fuel** (`max_parser_operations`, default 100K): Independent of depth, diff --git a/crates/bashkit/src/interpreter/mod.rs b/crates/bashkit/src/interpreter/mod.rs index e6e3b9d3..043c007d 100644 --- a/crates/bashkit/src/interpreter/mod.rs +++ b/crates/bashkit/src/interpreter/mod.rs @@ -5421,9 +5421,9 @@ impl Interpreter { } /// Maximum recursion depth for arithmetic expression evaluation. - /// THREAT[TM-DOS-025]: Prevents stack overflow via deeply nested arithmetic like + /// THREAT[TM-DOS-026]: Prevents stack overflow via deeply nested arithmetic like /// $(((((((...))))))) - const MAX_ARITHMETIC_DEPTH: usize = 200; + const MAX_ARITHMETIC_DEPTH: usize = 50; /// Evaluate arithmetic with assignment support (e.g. `X = X + 1`). /// Assignment must be handled before variable expansion so the LHS @@ -5578,7 +5578,7 @@ impl Interpreter { // First expand any variables in the expression let expanded = self.expand_arithmetic_vars(expr); - // Parse and evaluate with depth tracking (TM-DOS-025) + // Parse and evaluate with depth tracking (TM-DOS-026) self.parse_arithmetic_impl(&expanded, 0) } @@ -5658,7 +5658,7 @@ impl Interpreter { } /// Parse and evaluate a simple arithmetic expression with depth tracking. - /// THREAT[TM-DOS-025]: `arith_depth` prevents stack overflow from deeply nested expressions. + /// THREAT[TM-DOS-026]: `arith_depth` prevents stack overflow from deeply nested expressions. fn parse_arithmetic_impl(&self, expr: &str, arith_depth: usize) -> i64 { let expr = expr.trim(); @@ -5671,7 +5671,7 @@ impl Interpreter { return 0; } - // THREAT[TM-DOS-025]: Bail out if arithmetic nesting is too deep + // THREAT[TM-DOS-026]: Bail out if arithmetic nesting is too deep if arith_depth >= Self::MAX_ARITHMETIC_DEPTH { return 0; } diff --git a/crates/bashkit/tests/threat_model_tests.rs b/crates/bashkit/tests/threat_model_tests.rs index 3bd6788c..32a8c83f 100644 --- a/crates/bashkit/tests/threat_model_tests.rs +++ b/crates/bashkit/tests/threat_model_tests.rs @@ -157,8 +157,9 @@ mod resource_exhaustion { .await; let elapsed = start.elapsed(); - // Should complete quickly due to either timeout or loop limit - assert!(elapsed < Duration::from_secs(5)); + // Should complete quickly due to either timeout or loop limit. + // Under ASan the overhead can be ~10x, so use a generous bound. + assert!(elapsed < Duration::from_secs(15)); } } diff --git a/specs/006-threat-model.md b/specs/006-threat-model.md index eb80ef7d..338df6b9 100644 --- a/specs/006-threat-model.md +++ b/specs/006-threat-model.md @@ -191,7 +191,7 @@ runaway scripts without permanently breaking the session. | TM-DOS-020 | Function recursion | `f() { f; }; f` | Depth limit (100) | **MITIGATED** | | TM-DOS-021 | Command sub nesting | `$($($($())))` | Child parsers inherit remaining depth budget + fuel from parent | **MITIGATED** | | TM-DOS-022 | Parser recursion | Deeply nested `(((())))` | `max_ast_depth` limit (100) + `HARD_MAX_AST_DEPTH` cap (100) | **MITIGATED** | -| TM-DOS-026 | Arithmetic recursion | `$(((((((...)))))))` deeply nested parens | `MAX_ARITHMETIC_DEPTH` limit (200) | **MITIGATED** | +| TM-DOS-026 | Arithmetic recursion | `$(((((((...)))))))` deeply nested parens | `MAX_ARITHMETIC_DEPTH` limit (50) | **MITIGATED** | **Current Risk**: LOW - Both execution and parser protected @@ -201,7 +201,7 @@ max_function_depth: 100, // Runtime recursion (TM-DOS-020, TM-DOS-021) max_ast_depth: 100, // Parser recursion (TM-DOS-022) // TM-DOS-021: Child parsers in command/process substitution inherit remaining // depth budget and fuel from parent parser (parser/mod.rs lines 1553, 1670) -// TM-DOS-026: Arithmetic evaluator tracks recursion depth, capped at 200 +// TM-DOS-026: Arithmetic evaluator tracks recursion depth, capped at 50 // (interpreter/mod.rs MAX_ARITHMETIC_DEPTH) ``` @@ -850,7 +850,7 @@ This section maps former vulnerability IDs to the new threat ID scheme and track | V4 | TM-DOS-022 | Parser recursion | **MITIGATED** via `max_ast_depth` | | V5 | TM-DOS-018 | Nested loop multiplication | **MITIGATED** via `max_total_loop_iterations` (1M) | | V6 | TM-DOS-021 | Command sub parser limit bypass | **MITIGATED** via inherited depth/fuel | -| V7 | TM-DOS-026 | Arithmetic recursion overflow | **MITIGATED** via `MAX_ARITHMETIC_DEPTH` (200) | +| V7 | TM-DOS-026 | Arithmetic recursion overflow | **MITIGATED** via `MAX_ARITHMETIC_DEPTH` (50) | ### Open (Medium Priority) @@ -881,7 +881,7 @@ This section maps former vulnerability IDs to the new threat ID scheme and track | Parser fuel (100K ops) | TM-DOS-024 | `limits.rs` | Yes | | AST depth limit (100) | TM-DOS-022 | `limits.rs` | Yes | | Child parser limit propagation | TM-DOS-021 | `parser/mod.rs` | Yes | -| Arithmetic depth limit (200) | TM-DOS-026 | `interpreter/mod.rs` | Yes | +| Arithmetic depth limit (50) | TM-DOS-026 | `interpreter/mod.rs` | Yes | | Builtin parser depth limit (100) | TM-DOS-027 | `builtins/awk.rs`, `builtins/jq.rs` | Yes | | Execution timeout (30s) | TM-DOS-023 | `limits.rs` | Yes | | Virtual filesystem | TM-ESC-001, TM-ESC-003 | `fs/memory.rs` | Yes | @@ -927,7 +927,7 @@ ExecutionLimits::new() .max_input_bytes(10_000_000) // TM-DOS-001 (10MB) .max_ast_depth(100) // TM-DOS-022 (also inherited by child parsers: TM-DOS-021) .max_parser_operations(100_000) // TM-DOS-024 (also inherited by child parsers: TM-DOS-021) -// Note: MAX_ARITHMETIC_DEPTH (200) is a compile-time constant in interpreter (TM-DOS-026) +// Note: MAX_ARITHMETIC_DEPTH (50) is a compile-time constant in interpreter (TM-DOS-026) // Note: MAX_AWK_PARSER_DEPTH (100) is a compile-time constant in builtins/awk.rs (TM-DOS-027) // Note: MAX_JQ_JSON_DEPTH (100) is a compile-time constant in builtins/jq.rs (TM-DOS-027) diff --git a/specs/009-implementation-status.md b/specs/009-implementation-status.md index c02a3df6..bc69814d 100644 --- a/specs/009-implementation-status.md +++ b/specs/009-implementation-status.md @@ -184,15 +184,6 @@ Features that may be added in the future (not intentionally excluded): | Feature | Priority | Notes | |---------|----------|-------| | Coprocesses `coproc` | Low | Rarely used | -| ~~Extended globs `@()` `!()` `?()` `*()` `+()`~~ | ~~Medium~~ | Implemented: all five extglob operators | -| ~~Associative arrays `declare -A`~~ | ~~Medium~~ | Implemented: key-value access, iteration, unset, `${!m[@]}` | -| ~~`[[ =~ ]]` regex matching~~ | ~~Medium~~ | Implemented: `[[ ]]` conditionals with `=~` and BASH_REMATCH | -| ~~`getopts`~~ | ~~Medium~~ | Implemented: POSIX option parsing | -| ~~`command` builtin~~ | ~~Medium~~ | Implemented: `-v`, `-V`, bypass functions | -| ~~`type`/`which` builtins~~ | ~~Medium~~ | Implemented: `-t`, `-a`, `-p` flags | -| ~~`declare` builtin~~ | ~~Medium~~ | Implemented: `-i`, `-r`, `-x`, `-a`, `-p`, `-n`, `-l`, `-u` | -| ~~`ln` builtin~~ | ~~Medium~~ | Implemented: symbolic links (`-s`, `-f`) | -| ~~Directory stack `pushd`/`popd`/`dirs`~~ | ~~Low-Medium~~ | Implemented: push, pop, swap, clear, display | | `alias` | Low | Interactive feature | | History expansion | Out of scope | Interactive only | @@ -215,15 +206,16 @@ Features that may be added in the future (not intentionally excluded): ### Implemented -**97 core builtins + 3 feature-gated = 100 total** +**106 core builtins + 3 feature-gated = 109 total** `echo`, `printf`, `cat`, `nl`, `cd`, `pwd`, `true`, `false`, `exit`, `test`, `[`, `export`, `set`, `unset`, `local`, `source`, `.`, `read`, `shift`, `break`, `continue`, `return`, `grep`, `sed`, `awk`, `jq`, `sleep`, `head`, `tail`, -`basename`, `dirname`, `mkdir`, `rm`, `cp`, `mv`, `touch`, `chmod`, `chown`, `ln`, `wc`, +`basename`, `dirname`, `realpath`, `mkdir`, `mktemp`, `rm`, `cp`, `mv`, `touch`, `chmod`, `chown`, `ln`, `wc`, `sort`, `uniq`, `cut`, `tr`, `paste`, `column`, `diff`, `comm`, `date`, `wait`, `curl`, `wget`, `timeout`, `command`, `getopts`, `type`, `which`, `hash`, `declare`, `typeset`, `let`, `kill`, `shopt`, +`trap`, `caller`, `seq`, `tac`, `rev`, `yes`, `expr`, `time` (keyword), `whoami`, `hostname`, `uname`, `id`, `ls`, `rmdir`, `find`, `xargs`, `tee`, `:` (colon), `eval`, `readonly`, `times`, `bash`, `sh`, `od`, `xxd`, `hexdump`, `strings`,