⚡️ LAO: Local AI Workflow Orchestrator

Chain. Build. Run. All offline. LAO is how developers bend AI to their will—no cloud, no compromise.

🧠 What is LAO?

LAO is a cross-platform desktop tool for chaining local AI models and plugins into powerful, agentic workflows. It supports prompt-driven orchestration, visual DAG editing, and full offline execution.

The LAO UI showing a workflow with hierarchical layout, execution levels, and color-coded connections. Notice:

Level bands (alternating blue/purple) showing execution levels
Level labels ("Level 0", "Level 1", "Level 2") on the left
Green edges for primary inputs (input_from)
Purple edges for parallel dependencies (depends_on)
L# labels on nodes showing execution levels
Auto-layout arranging nodes hierarchically

✨ Features

LAO fully supports processing images, audio, video, and mixed media with automatic modality detection:

7 Supported Modalities

🎵 Audio - .mp3, .wav, .ogg (transcription, speech analysis)
🖼️ Image - .png, .jpg, .gif (OCR, object detection, classification)
🎬 Video - .mp4, .avi, .mov (frame extraction, scene analysis)
📄 Text - .txt, .md, .json (NLP, summarization)
📦 Binary - .pdf, .docx (document parsing)
🗂️ Structured - .json, .yaml (data processing)
❓ Unknown - Fallback handling

Example: Batch Image Processing

workflow: "batch_images"
steps:
  - id: process_photos
    run: MultimodalPlugin
    input:
      for_each:
        items: ["photo1.jpg", "photo2.png", "photo3.gif"]
        var: "photo"
        max_parallel: 3  # Process 3 images at once
      file: "${photo}"
    input_modality: Image
    output_modality: Structured

Example: Video Frame Analysis

workflow: "video_analysis"
steps:
  - id: extract_frames
    run: MultimodalPlugin
    input:
      file: "video.mp4"
      task: "extract_frames"
      fps: 1  # 1 frame per second
    input_modality: Video
    output_modality: Image
    
  - id: analyze_frames
    run: MultimodalPlugin
    input:
      for_each:
        items: "${extract_frames.frames}"
        var: "frame"
        max_parallel: 4
      file: "${frame}"
    input_modality: Image
    output_modality: Structured

UI Features

Cmd+O: Toggle modality flow visualization
Automatic detection from file extensions and MIME types
Drag-drop file attachments with modality indicators
Visual modality flow showing data transformations

See docs/MULTIMODAL_GUIDE.md for complete examples and API reference.

] Plugin marketplace/discovery

Live workflow status/logs in UI

🚀 Quickstart

GUI (Recommended)

# Run the native GUI with visual workflow builder
cargo run --bin lao-ui

CLI

# Run workflows from command line
cargo run --bin lao-cli run workflows/test.yaml

# Generate workflows from natural language
cargo run --bin lao-cli prompt "Summarize this audio and tag action items"

# Validate prompt library
cargo run --bin lao-cli validate-prompts

Build Plugins

# Build all plugins for your platform
bash scripts/build-plugins.sh

# Or build plugins manually
cd plugins/EchoPlugin && cargo build --release
cd ../WhisperPlugin && cargo build --release
# ... etc for each plugin

# Then copy the built libraries to plugins/ directory
# On macOS: cp target/release/*.dylib ../../plugins/
# On Linux: cp target/release/*.so ../../plugins/
# On Windows: cp target/release/*.dll ../../plugins/

Note: Plugins must be built before running the UI. If you see "plugins not found", run the build script above.

📖 Tutorial: Using the LAO UI

Getting Started

Launch the UI:
```
cargo run --bin lao-ui
```
Understanding the Interface:
- Workflow Management (top): Select and manage workflows
- Visual Flow Builder (middle): Build workflows visually with nodes and connections
- Execution Logs (bottom): View real-time execution status and logs

Creating Your First Workflow

Step 1: Start Fresh

Click "🆕 New" to clear the canvas and start a new workflow

Step 2: Add Nodes

In the Visual Flow Builder, find the "Add Node" section
Enter a node name (e.g., "process_data")
Select a plugin from the dropdown (e.g., "EchoPlugin")
Click "➕ Add" to create the node
Repeat to add more nodes

Step 3: Connect Nodes

Click the "🔗 Connect" button to enter connection mode
First click: Select the source node (where data comes from)
Second click: Select the target node (where data goes to)
After connecting, click "📐 Auto-Layout" to automatically arrange nodes hierarchically by execution level
The connection is created automatically
Click "🔗 Connect" again to exit conn

Keyboard Shortcuts

LAO includes comprehensive keyboard shortcuts for efficient workflow editing:

Shortcut	Action	Feature
Cmd+Z	Undo last action	Undo/Redo System
Cmd+Shift+Z	Redo action	Undo/Redo System
Cmd+M	Toggle metrics dashboard	Real-time Metrics
Cmd+T	Toggle timeline view	Gantt Chart
Cmd+O	Toggle modality flow	Multimodal Visualization

Advanced Features

Loop/Iteration Support

Process multiple items in parallel with built-in loop support:

steps:
  - id: process_files
    run: EchoPlugin
    input:
      for_each:
        items: ["file1.txt", "file2.txt", "file3.txt"]
        var: "filename"
        collect_results: true
        max_parallel: 2  # Process 2 files concurrently
      message: "Processing ${filename}"

Metrics Dashboard

Press Cmd+M to view real-time execution metrics:

Execution time per step
Memory usage tracking
Throughput (items/second)
Error rates and retry statistics

Timeline Visualization

Press Cmd+T to view execution timeline:

Gantt chart showing step durations
Parallel execution visualization
Color-coded step status
Zoom and pan controlsection mode

Step 4: Set Primary Input (for nodes with multiple inputs)

Click on a node that has multiple incoming connections
A dialog appears showing all input sources
Select which connection provides the primary input (the main data flow)
Other connections become parallel dependencies (must complete but don't provide input)
Click "✓ Done" to confirm

Step 5: Run Your Workflow

Click "▶️ Run" to execute
The button automatically detects if your workflow can run in parallel
Watch the execution logs for real-time progress
Node status indicators show: pending → running → success/error

Loading Existing Workflows

Select from Dropdown:
- Use the workflow dropdown at the top
- Select a workflow file (e.g., "test.yaml")
- The workflow automatically loads into the Visual Flow Builder
Workflow Location:
- Workflows are stored in the workflows/ directory
- The full path is shown at the bottom of the Workflow Management section

Understanding Visual Indicators

Node Colors & Status

Gray dot: Pending
Blue dot: Running
Green dot: Success
Red dot: Error
Purple dot: Cached (skipped execution)

Edge Colors

Green (thick): Primary input (input_from) - main data flow
Purple (thin): Parallel dependency (depends_on) - must complete but doesn't provide input

Execution Levels

L# labels: Show which execution level a node belongs to
Nodes at the same level can run in parallel
Levels execute sequentially (Level 0 → Level 1 → Level 2...)

Node Indicators

Purple dot (top-left): Fan-in node (receives multiple inputs)
Blue dot (bottom-right): Fan-out node (sends to multiple nodes)

Parallel Execution

LAO automatically detects when workflows can run in parallel:

Automatic Detection:
- The "Run" button shows "▶️ Run (Parallel)" in purple when parallel execution is possible
- Otherwise, it shows "▶️ Run" in blue for sequential execution
How It Works:
- Independent nodes (no dependencies) run simultaneously
- Nodes at the same execution level run concurrently
- Dependencies are respected automatically
Debug Mode:
- Check "🐛 Debug" to force sequential execution
- Useful for debugging or step-by-step execution
- Overrides automatic parallel detection

Workflow Patterns

Sequential Chain

Node A → Node B → Node C

All nodes run one after another.

Parallel Branches

Node A ──┐
         ├──→ Node D
Node B ──┤
         │
Node C ──┘

Nodes A, B, C run simultaneously, then Node D runs.

Fan-Out

Node A ──→ Node B
    └──→ Node C
    └──→ Node D

Node A feeds multiple parallel nodes.

Fan-In

Node A ──┐
         ├──→ Node D (uses A as primary input)
Node B ──┤
         │
Node C ──┘

Multiple nodes feed into one merge node.

Managing Workflows

Saving Workflows

Workflows are automatically saved to the workflows/ directory
Use the "💾 Save" button in the Visual Flow Builder
Enter a filename (e.g., "my_workflow.yaml")

Exporting Workflows

Click "📤 Export" to copy the workflow YAML to clipboard
Useful for sharing or version control

Deleting Nodes/Edges

Delete node: Select a node and press Delete key
Delete edge: Right-click on an edge to remove it

Tips & Best Practices

Start Simple: Begin with 2-3 nodes to understand the flow
Use Meaningful Names: Name nodes descriptively (e.g., "process_data" not "node1")
Check Execution Levels: Use L# labels to understand parallelism
Monitor Logs: Watch the execution logs for errors and debugging info
Use Debug Mode: Enable debug mode when troubleshooting
Save Frequently: Save your workflows regularly

Example: Building a Simple Echo Chain

Click "🆕 New" to start fresh
Add first node: Name="start", Plugin="EchoPlugin", click "➕ Add"
Add second node: Name="middle", Plugin="EchoPlugin", click "➕ Add"
Add third node: Name="end", Plugin="EchoPlugin", click "➕ Add"
Click "🔗 Connect", then click "start" → "middle" → "end"
Click "▶️ Run" to execute
Watch the logs to see each step execute sequentially

Troubleshooting

"No workflows found": Ensure workflow YAML files exist in the workflows/ directory
Workflow won't load: Check that the YAML file is valid and plugins are available
Nodes not connecting: Make sure you're in Connect mode (button highlighted)
Execution fails: Check the execution logs for error messages

🍎 Apple Silicon Optimization (M1/M2/M3/M4)

LAO includes comprehensive optimizations for Apple Silicon Macs, delivering 3-4x faster inference and 10x better battery life.

Quick Start on Apple Silicon

# Build with Apple Silicon optimizations
./scripts/build-apple-silicon.sh

# Or manually:
cd core
../target/release/lao-cli plugin-list  # Shows ANE plugin

What's Optimized?

Feature	Benefit	Status
Metal GPU	2-3x faster inference	✅ Automatic
Neural Engine (ANE)	200%+ faster, 0.5W power	✅ Available
Accelerate Framework	Hardware BLAS acceleration	✅ Automatic
Power Management	Battery mode auto-config	✅ Available
Core Scheduling	P-cores for inference, E-cores for background	✅ Available
Unified Memory	Zero-copy GPU buffers	✅ Available
Metal Shaders	Fused operations on GPU	✅ Framework ready

System Detection & Auto-Optimization

LAO automatically detects your Apple Silicon chip and optimizes configuration:

# See system info with optimizations
cargo run --bin lao-cli -- plugin-list

# Shows:
# 🧠 Neural Engine: Available (M3 Max)
# 💾 Unified Memory: 96 GB
# 🔌 Core Config: 8 P-cores, 4 E-cores
# ⚡ Recommended: GPU inference, full offload

Power-Aware Inference

LAO optimizes based on power state:

Battery Mode:

Uses Neural Engine (0.5W)
Smaller batch sizes
Quantized models (int8)
Expected: 6-8 hours on M3 MacBook

Plugged In:

Uses GPU (Metal)
Full precision
Large batch sizes
Expected: 25 tokens/sec on M3 Pro

Thermal Throttle:

Falls back to CPU
Reduces thread count
Prevents fan noise

Performance Benchmarks

On M3 Pro MacBook Pro (12-core CPU, 18-core GPU):

Model: llama2-7b (GGUF)
─────────────────────────────
CPU Only:           4 tokens/sec
GPU (Metal):        18 tokens/sec (+350%)
Neural Engine*:     22 tokens/sec (+450%)
With Unified Mem:   +5% faster (zero-copy)
*For quantized int8 models

Model: mistral-7b
─────────────────────────────
CPU Only:           3 tokens/sec
GPU + MPS:          12 tokens/sec (+300%)
Battery Mode (ANE): ~1 token/sec, 0.5W

Building Optimized Binaries

# Standard release build (with optimizations)
cargo build --release

# Apple Silicon optimized build (recommended)
./scripts/build-apple-silicon.sh

# What it does:
# - Native ARM64 compilation (-C target-cpu=native)
# - Link-time optimization (LTO)
# - Metal GPU support
# - Accelerate framework
# - ANE plugin included

Using Neural Engine Plugin

The ANE plugin provides ultra-low power inference:

# Check availability
cd core
../target/release/lao-cli plugin-list | grep ANE

# In workflows, use it like:
# ane_task:
#   plugin: ANEInferencePlugin
#   config:
#     quantization: int8
#     model: model.gguf

Advanced: Custom Optimization Modes

// In your workflow code or plugin
use lao_orchestrator_core::power_management;
use lao_orchestrator_core::core_scheduler;

let power_state = power_management::get_power_state();
let config = power_management::get_optimized_config(power_state, thermal);
// config now has device, threads, batch_size optimized for power state

// P-core/E-core scheduling
let (inference_threads, background_threads) = core_scheduler::get_thread_pool_sizes();
// Use inference_threads for model inference
// Use background_threads for I/O and preprocessing

Full Optimization Roadmap

For complete details on all optimization phases:

Phase 1: Power management, P-core/E-core scheduling
Phase 2: Unified memory, Metal Performance Shaders
Phase 3: Neural Engine, macOS UI/UX (menu bar, Spotlight, Quick Look, notifications)

See: docs/SILICON_IMPROVEMENTS.md and docs/PHASE_3_MACOS_UI.md

macOS Native Features (Phase 3)

LAO includes native macOS UI/UX integration:

Menu Bar:

Standard File, Edit, View, Help menus
Native keyboard shortcuts (Cmd+N, Cmd+O, Cmd+R, etc.)
Preferences and app menu

Spotlight Search:

Index workflows and plugins
Full-text search across metadata
Recent items tracking

lao dev macos spotlight "audio"
# 🔍 Spotlight Results for 'audio': 3 items

Quick Look Preview:

Preview YAML workflows
Preview markdown documentation
Native file preview integration

lao dev macos quicklook --param ./workflows/test.yaml

Notification Center:

Workflow completion notifications
Error notifications
Background task updates
```
lao dev macos notify
```

Keyboard Shortcuts:

lao dev macos shortcuts
# 📋 LAO Keyboard Shortcuts:
#   New Workflow:        Cmd+N
#   Open Workflow:       Cmd+O
#   Run Workflow:        Cmd+R

Initialize All Features:

# Set up menu bar, Spotlight indexing, notifications
lao dev macos init --param ./workflows

Troubleshooting on Apple Silicon

🧪 Testing

LAO includes comprehensive test coverage across all features:

Test Statistics

99 total tests ✅ (all passing)
21 feature unit tests - Loops, modality detection, workflows
7 integration tests - Multi-feature combinations
71 existing tests - Core library, DAG, plugins

Run Tests

# Run all tests
cargo test -p lao-orchestrator-core

# Run specific test suites
cargo test -p lao-orchestrator-core --test features_test
cargo test -p lao-orchestrator-core --test integration_test

# Validate workflows
cargo run --bin lao-cli validate workflows/image_analysis.yaml

Example Workflows

LAO includes ready-to-run example workflows:

Multimodal Processing:

workflows/image_analysis.yaml - OCR + object detection + AI descriptions
workflows/image_batch_processing.yaml - Batch process 5 images in parallel
workflows/video_to_images.yaml - Extract video frames and analyze scenes
workflows/mixed_media_processing.yaml - Process multiple media types together

Standard Workflows:

workflows/test.yaml - Simple echo test
workflows/test_loop.yaml - Loop execution example
workflows/multimodal_analysis.yaml - Audio transcription with modality detection
workflows/parallel_data_processing.yaml - Parallel execution pattern

Run Examples:

cd core

# Run simple test
../target/release/lao-cli run ../workflows/test.yaml

# Run loop workflow
../target/release/lao-cli run ../workflows/test_loop.yaml

# Validate image workflow
../target/release/lao-cli validate ../workflows/image_analysis.yaml

See TEST_REPORT.md for detailed test coverage and results. "Rosetta 2 Warning": You're using an x86_64 binary on Apple Silicon.

# Use native build instead
./scripts/build-apple-silicon.sh
``**Architecture**: `docs/architecture.md` - Core system design
- **Plugins**: `docs/plugins.md` - Plugin development guide
- **Workflows**: `docs/workflows.md` - Workflow syntax reference
- **CLI**: `docs/cli.md` - Command-line interface
- **Observability**: `docs/observability.md` - Logging and monitoring
- **Multimodal Guide**: `docs/MULTIMODAL_GUIDE.md` - Image/video/audio processing
- **Test Report**: `TEST_REPORT.md` - Comprehensive test coverage (99 tests)
- **Integration Status**: `INTEGRATION_STATUS.md` - Feature integration details
# Should show M1, M2, M3, or M4

Slow GPU performance: Check Metal availability:

system_profiler SPDisplaysDataType | grep Metal

🧩 Prompt-Driven Workflows

LAO can generate and execute workflows from natural language prompts using a local LLM (Ollama). The system prompt is editable at core/prompt_dispatcher/prompt/system_prompt.txt.

Example:

lao prompt "Refactor this Python file and add comments"

📚 Prompt Library & Validation

Prompts and expected workflows: core/prompt_dispatcher/prompt/prompt_library.md and .json
Validate with: cargo run --bin lao-cli validate-prompts
Add new prompts to improve LLM output and test new plugins

🛠️ Contributing Plugins & Prompts

Add new plugins by implementing the LaoPlugin trait, building as a cdylib, and placing the resulting library in the plugins/ directory
Expose a C ABI function named plugin_entry_point that returns a Box<dyn LaoPlugin>
Add prompt/workflow pairs to the prompt library for validation and LLM tuning
See docs/plugins.md and docs/workflows.md for details

📄 Documentation

Architecture: docs/architecture.md
Plugins: docs/plugins.md
Workflows: docs/workflows.md
CLI: docs/cli.md
Observability: docs/observability.md

🌌 Manifesto

Cloud is optional. Intelligence is modular. Agents are composable.
LAO is how devs build AI workflows with total control.
No tokens. No latency. No lock-in.

Let’s define the category—one plugin at a time.

Name		Name	Last commit message	Last commit date
Latest commit History 56 Commits
.github		.github
assets		assets
cache		cache
cli		cli
core		core
docs		docs
lao_inference		lao_inference
lao_plugin_api		lao_plugin_api
plugins		plugins
scripts		scripts
tools		tools
ui/lao-ui		ui/lao-ui
workflow_states		workflow_states
workflows		workflows
.gitignore		.gitignore
COMPLETION.md		COMPLETION.md
Cargo.toml		Cargo.toml
LICENSE		LICENSE
README.md		README.md
lao.toml		lao.toml
main.rs		main.rs
packaging.toml		packaging.toml
temp_cache.yaml		temp_cache.yaml

License

abendrothj/LAO

Folders and files

Latest commit

History

Repository files navigation