Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,9 @@ Files in `doc/build` are build artifacts. Do not edit these.
There is a main base template at `doc/base.html`, which has most of the general purpose CSS and styles for code.
You should rebuild the docs when you make edits.

For markdown files, prefer to have sentences on their own line.
Only wrap really long lines, and try to wrap on a comma or other punctuation.

## VS Code

Code for the VS Code extension is at `code/`.
Expand Down
3 changes: 3 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,9 @@ Files in `doc/build` are build artifacts. Do not edit these.
There is a main base template at `doc/base.html`, which has most of the general purpose CSS and styles for code.
You should rebuild the docs when you make edits.

For markdown files, prefer to have sentences on their own line.
Only wrap really long lines, and try to wrap on a comma or other punctuation.

## VS Code

Code for the VS Code extension is at `code/`.
Expand Down
2 changes: 1 addition & 1 deletion doc/base.html
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
color: #0000FF;
}

.mshellIF, .mshellELSE, .mshellELSESTAR, .mshellSTARIF, .mshellEND {
.mshellIF, .mshellELSE, .mshellELSESTAR, .mshellSTARIF, .mshellEND, .mshellDEF {
color: #0F4C81;
font-weight: bold;
}
Expand Down
2 changes: 1 addition & 1 deletion doc/control-flow.inc.html
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,7 @@ <h1 id="control-flow-prefix-quote">Prefix Quote Syntax <a class="section-link" h

<p>
Prefix quotes provide an alternative syntax for applying functions to quotations.
The syntax is <code>functionName. ... end</code>, which is equivalent to <code>(...) functionName</code>.
The syntax is <code>functionName. ... end</code> (that is, the function name has a period '.' appended to the end), which is equivalent to <code>(...) functionName</code>.
</p>

<pre>
Expand Down
210 changes: 198 additions & 12 deletions doc/mshell.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,11 @@ History search is prefix-based and case-insensitive. The prefix is whatever is c
- Ctrl-N: search forward through history by prefix
- Ctrl-Y: accept the inline history completion
- Ctrl-Space: insert a literal space without expanding aliases
- Alt-D: insert the current date as YYYY-MM-DD
- Alt-.: insert the last argument from history; repeat to cycle older entries
- Tab: complete the current token; press Tab again to cycle matches and fill the input
- Shift-Tab: cycle completion backward when matches are active
- Ctrl-N/Ctrl-P: when cycling completions, move forward/backward through matches

Completion results render in one or two columns based on available rows. If the list still doesn't fit, the final line shows `[n] more items..`.

### Definition-based completions

The CLI can use definition metadata to provide argument completions for binaries. Add a `complete` key in the metadata dictionary of a `def` to register it for one or more command names. The definition is invoked with a clean stack containing a single list of argument tokens (excluding the binary name and the current prefix), and it should return a list of strings.
Expand Down Expand Up @@ -454,26 +451,207 @@ You can store to several variables in one go by separating the store tokens with
@my_var

# Storing multiple values at once
1 2 3 a!, b!, c!
1 2 3 a!, b!, c! # a is 1, b is 2, c is 3.
@a @b @c

# A trailing comma after the last store is ignored
4 5 a!, b!,
```

## Command Substitution
## Execution

Execution of external commands or binaries is different in `mshell`.
Instead of it being the main syntactical construct, in `mshell` you build up a list of arguments, and then at a time of your choosing, you execute it.

```mshell
['my-program' 'arg1' 'arg2'];
```

Often there are different things you want out of your execution, or you want different behavior depending on the exit code.
`mshell` gives you full flexibility to decide with concise syntax.

To initiate execution, you use one of 3 operators:

- `;`: Execute command, always continue. Don't provide any information on the exit code.
- `!`: Execute command, but stop the running script execution on any non-zero exit code.
- `?`: Execute command, leaving the exit code integer on the stack.

```mshell
['false']; # mshell will continue past this point
['true']! # Will execute and continue because of 0 exit code
['my-command']? exitCode!
$"Exit code was {@exitCode}" wl
```

The other choice you often have when executing commands is what to do with the standard output. Sometimes you will want to redirect it to a file, other times you will want to leave the contents on the stack to process further. For that, you use the `>`, `>>`, `*`, and `*b` operators.

```mshell
[yourCommand] `fileToRedirectTo` > ! # Redirects stdout to the file, truncating it.
[yourCommand] `fileToRedirectTo` >> ! # Redirects stdout to the file, appending.
[yourCommand] * ! # Puts all of stdout on the stack as a string.
[yourCommand] *b ! # Puts all of stdout on the stack as a binary.
```

You can do similar things with standard error. To redirect standard error to a file, use `2>`, and `^` instead of `*`.

To redirect both standard output and standard error to the same file, use `&>` (truncate) or `&>>` (append). This ensures both streams share the same file descriptor, preserving the order of output.

```mshell
[yourCommand] `output.log` &> ! # Redirects both stdout and stderr to the file, truncating it.
[yourCommand] `output.log` &>> ! # Redirects both stdout and stderr to the file, appending.
```

If you manually use `>` and `2>` with exactly the same path string, mshell will automatically use a single file descriptor for both streams, avoiding race conditions. However, if the append modes differ (e.g., `>` and `2>>` to the same file), an error will be raised.

```mshell
[yourCommand] `errors.log` 2> ! # Redirects stderr to the file, truncating it.
[yourCommand] `errors.log` 2>> ! # Redirects stderr to the file, appending.
[yourCommand] ^ ! # Puts all of stderr on the stack as a string.
[yourCommand] ^b ! # Puts all of stderr on the stack as a binary.
```

If you want to put both standard output and standard error onto the stack, you can do that.
Standard output will always be pushed first, and then standard error.

So the following are equivalent:

```mshell
[yourCommand] *b ^b ! # Order here does not matter
[yourCommand] ^b *b !
```

Summary of external command operators:

Operator | Effect on external commands | Notes
---------|--------------------------------------------|-----------------------------------------------------
`;` | Execute and continue. | No exit code on the stack.
`!` | Execute and stop on non-zero exit. | Uses the command exit code.
`?` | Execute and push the exit code. | Integer is left on the stack.
`>` | Redirect stdout to a file. | Truncates the file.
`>>` | Redirect stdout to a file. | Appends to the file.
`*` | Capture stdout to the stack. | As a string.
`*b` | Capture stdout to the stack. | As binary.
`2>` | Redirect stderr to a file. | Truncates the file.
`2>>` | Redirect stderr to a file. | Appends to the file.
`&>` | Redirect both stdout and stderr to a file. | Truncates the file.
`&>>` | Redirect both stdout and stderr to a file. | Appends to the file.
`^` | Capture stderr to the stack. | As a string.
`^b` | Capture stderr to the stack. | As binary.
`<` | Feed stdin from a value. | String, path, or binary.
`<>` | In-place file modification. | Reads file to stdin, writes stdout back on success.

### Redirection on quotations

All of the redirection operators above also work on quotations. This is useful when you want to redirect the output of mshell code that uses `wl`, `wle`, or other built-in functions that write to stdout or stderr. It is also useful when you have many commands that you want to run while appending all the outputs to a single file, without having to put the redirection on each command invocation.

```mshell
(
"Hello from stdout" wl
"Hello from stderr" wle
) `output.log` &> x # Redirects both stdout and stderr from the quotation to output.log
```

```mshell
(
[echo "Running step 1"]!
[echo "Running step 2"]!
[echo "Running step 3"]!
) `build.log` >> x # All command outputs appended to build.log
```

### Input Redirection

Use `<` to feed data into stdin. The type of the value on top of the stack determines how the input is provided.

`String` values are encoded as UTF-8 and streamed as text.

```mshell
[wc -l] "line 1\nline 2\n" < ; # Counts the lines from the provided string
```

`Path` values open the referenced file and stream its contents.

```mshell
[wc -l] `myfile.txt` < ; # Equivalent to shell input redirection from a file
```

`Binary` values are written directly without any string conversion.

```mshell
[md5sum] `binary_stdin.bin` readFileBytes < ; # Streams raw bytes into the command
```

### In-place file modification

The `<>` operator enables in-place file modification. It reads a file's contents, passes them to the command's stdin, and on successful completion (exit code 0), writes the command's stdout back to the same file. This is similar to the `sponge` command from moreutils.

```mshell
[sort -u] `file.txt` <> !
```

With a list on the stack, the following operators will leave output content on the stack after process execution:
This is equivalent to, but safer than:

```mshell
o: [str], Standard output, split by lines
oc: str, Standard output, complete untouched
os: str: Standard output, stripped
e: [str], Standard error, split by lines
ec: str, Standard error, complete untouched
es: str, Standard error, stripped
[sort -u] `file.txt` < `file.txt.tmp` > !
[mv file.txt.tmp file.txt] !
```

#### Semantics

- The file is read when `<>` is evaluated, capturing its contents at that moment.
- If the command exits with code 0 (success), stdout replaces the file contents.
- If the command exits with a non-zero code (failure), the original file is preserved unchanged.
- File permissions are preserved.

#### Requirements

- Only works with lists (external commands), not quotations.
- Requires a `Path` type (backtick-quoted), not a string.
- The file must exist at the time `<>` is evaluated.

#### Examples

```mshell
# Sort a file in place, removing duplicates
[sort -u] `data.txt` <> !

# Format JSON in place
[jq .] `config.json` <> !

# Filter lines in place (keep only lines containing "error")
[grep error] `log.txt` <> !

# Using with ? to check exit code without failing
[sort -u] `file.txt` <> ?
0 = if
"File sorted successfully" wl
else
"Sort failed, file unchanged" wl
end
```

### How mshell finds binaries

When executing an external command, `mshell` resolves the binary name using a two-step process. First it checks the bin map file for an override, and if none is found it falls back to `PATH`.

The bin map file lives alongside the history files (for example, `~/.local/share/msh/msh_bins.txt` on Linux/macOS or `%LOCALAPPDATA%\\mshell\\msh_bins.txt` on Windows). Each non-empty line is a tab-separated pair of fields: the binary name and the absolute path to the binary. Both fields are trimmed, the name must not contain path separators, and the line must contain exactly two fields.

```
mytool /usr/local/bin/mytool
another /home/me/bin/another
```

To manage the file, use the `msh bin` subcommands:

- `msh bin add <path>`: add/replace using the file basename and absolute path
- `msh bin add <name> <path>`: add/replace using an explicit name
- `msh bin remove <name>`: remove an entry by name
- `msh bin list`: print the bin map file contents
- `msh bin path`: print the bin map file path
- `msh bin edit`: edit the file in `$EDITOR`
- `msh bin audit`: report invalid or missing entries
- `msh bin debug <name>`: show lookup details for a binary

## Process Substitution

Process substitution is done using the `psub` operator.
Expand Down Expand Up @@ -542,3 +720,11 @@ The current object types supported by `mshell` are:
9. Date/Times
10. Dictionary
11. Maybe

## LLM Notes

- `mshell` is concatenative and stack-based: words run left-to-right, consuming values from the stack and pushing results.
- External commands are lists of strings; execute them with `;` (always continue), `!` (stop on non-zero), or `?` (push exit code).
- Comments use `#` and run to the end of the line.
- Paths are backticked literals with no escaping; use string interpolation with `toPath` when you need escapes.
- Quotations are `(...)` blocks; execute a quotation with `x`.
27 changes: 25 additions & 2 deletions doc/variables.inc.html
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,32 @@ <h1 id="variables-mshell">mshell Variables <a class="section-link" href="#variab
<span class="mshellVARRETRIEVE">@total</span> <span class="mshellLITERAL">wl</span></code>
</pre>

<p>
You can store multiple values at once by separating the store tokens with commas. A trailing comma after the last store is ignored.
</p>

<pre>
<code><span class="mshellLINECOMMENT"># Storing multiple values at once</span>
<span class="mshellINTEGER">1</span> <span class="mshellINTEGER">2</span> <span class="mshellINTEGER">3</span> <span class="mshellVARSTORE">a!</span>, <span class="mshellVARSTORE">b!</span>, <span class="mshellVARSTORE">c!</span> <span class="mshellLINECOMMENT"># a is 1, b is 2, c is 3.</span>
<span class="mshellVARRETRIEVE">@a</span> <span class="mshellVARRETRIEVE">@b</span> <span class="mshellVARRETRIEVE">@c</span>

<span class="mshellLINECOMMENT"># A trailing comma after the last store is ignored</span>
<span class="mshellINTEGER">4</span> <span class="mshellINTEGER">5</span> <span class="mshellVARSTORE">a!</span>, <span class="mshellVARSTORE">b!</span>,</code>
</pre>

<p>
Storing values at once is useful in definitions, so that the variables appear in order of the stack, instead of revserse.
</p>

<pre><code><span class="mshellDEF">def</span> <span class="mshellLITERAL">myDef</span> <span class="mshellLEFT_PAREN">(</span><span class="mshellTYPEINT">int</span> <span class="mshellSTR">str</span> <span class="mshellLITERAL">path</span> <span class="mshellDOUBLEDASH">--</span> <span class="mshellRIGHT_PAREN">)</span>
<span class="mshellVARSTORE">myInt!</span><span class="mshellCOMMA">,</span> <span class="mshellVARSTORE">myStr!</span><span class="mshellCOMMA">,</span> <span class="mshellVARSTORE">myPath!</span>
<span class="mshellLINECOMMENT"># Do things..</span>
<span class="mshellEND">end</span>
<span class="mshellEOF"></span></code></pre>

<p>
Retrieving an unknown variable results in an error.
Mshell variables do not consult the environment, so <code>@name</code> only looks in the mshell variable map.
<code>mshell</code> variables do not consult the environment, so <code><span class="mshellVARRETRIEVE">@name</span></code> only looks in the mshell variable map.
</p>

<h1 id="variables-environment">Environment Variables <a class="section-link" href="#variables-environment" aria-label="Permalink">§</a> <a class="back-to-top" href="#variables-top">Back to top</a></h1>
Expand All @@ -46,7 +69,7 @@ <h1 id="variables-environment">Environment Variables <a class="section-link" hre
</pre>

<p>
You can print all environment variables (sorted by key) using the built-in <code>.env</code> function.
You can print all environment variables (sorted by key) using the built-in <code>env</code> function.
</p>

<pre>
Expand Down