ARO-0073: Plugin SDK & Developer Experience — Feedback Welcome

[KrisSimon]

TL;DR: Writing ARO plugins today requires 70–95% boilerplate (C ABI exports, JSON parsing, memory management). This proposal introduces language-native SDKs for Swift, Rust, C, and Python that let you write natural, idiomatic plugin code while the SDK handles the ceremony.

The Problem

An analysis of all nine plugin examples reveals a striking ceremony-to-logic imbalance:

Language (plugin type)	Business Logic	Boilerplate
C (qualifiers)	~5%	~95%
Swift (actions)	~20%	~80%
C (actions)	~30%	~70%
Swift (services)	~45%	~55%
Rust (actions)	~65%	~35%
Python (actions)	~80%	~20%

A C qualifier plugin needs ~150 lines of pointer arithmetic just to parse a JSON array. Swift plugins require NSDictionary workarounds for cross-dylib Foundation bridging. Rust needs six unsafe blocks for CString FFI. Three incompatible ABIs (actions, services, qualifiers) exist with no guidance on when to use which.

What This Proposal Introduces

1. Unified ABI v2

Collapse the three ABIs into one contract. Services route through aro_plugin_execute("service:<method>", ...). Qualifier-only plugins no longer need stub exports. Full backward compatibility via "abi": 2 detection.

2. Language-Native SDKs

Swift — @AROPlugin macro generates all @_cdecl exports:

import AROPluginSDK

@AROPlugin(handle: "Greeting", version: "1.0.0")
struct GreetingPlugin {
    @Action(verbs: ["Greet"], role: .own, prepositions: [.with])
    func greet(input: ActionInput) -> ActionOutput {
        let name = input.string("name") ?? "World"
        return .success(["greeting": "Hello, \(name)!"])
    }
}

Rust — #[aro_plugin] proc macro eliminates all unsafe:

use aro_plugin_sdk::prelude::*;

#[aro_plugin(handle = "CSV", version = "1.0.0")]
mod csv_plugin {
    #[action(verbs = ["ParseCSV"], role = "request", prepositions = ["from"])]
    fn parse_csv(input: &Input) -> Result<Output> {
        let data = input.string("data")?;
        let rows: Vec<Vec<String>> = data.lines()
            .map(|line| line.split(',').map(String::from).collect())
            .collect();
        Ok(Output::new().set("rows", rows).set("count", rows.len()))
    }
}

C — Single-header SDK with macros, bundled JSON parser, arena allocator:

#include "aro_plugin_sdk.h"

ARO_PLUGIN("Hash", "1.0.0");

ARO_ACTION("ComputeHash", ROLE_OWN, "from") {
    const char* data = aro_input_string(ctx, "data");
    uint32_t hash = djb2(data);
    aro_output_int(ctx, "hash", hash);
    return aro_ok(ctx);
}

Python — Decorators with typed helpers:

from aro_plugin_sdk import plugin, action, qualifier

@plugin(handle="Stats", version="1.0.0")
class StatsPlugin:
    @qualifier(input_types=["List"])
    def average(self, values: list) -> float:
        return sum(values) / len(values)

3. Plugin Scaffolding CLI

aro new plugin --name my-csv --lang rust --actions --qualifiers
# Generates a complete, ready-to-build project with SDK dependency

4. Event System Integration

Plugins can emit and subscribe to domain events:

@OnEvent("UserCreated")
func handleUser(event: EventData) {
    // React to domain events
}

5. Python Persistent Mode

Long-lived subprocess instead of per-call spawning — enables stateful plugins and eliminates startup overhead.

6. Per-SDK Testing Utilities

Test plugins without loading them through the ARO runtime.

Estimated Ceremony Reduction

Plugin Type	Language	Before	After	Reduction
Qualifier-only	C	~95%	~20%	−75pp
Action-only	Swift	~80%	~15%	−65pp
Action-only	C	~70%	~25%	−45pp
Service	Swift	~55%	~15%	−40pp
Action-only	Rust	~35%	~10%	−25pp

Implementation Plan

1. Phase 1: Unified ABI v2 (runtime changes, backward compat)
2. Phase 2: Python SDK + persistent mode (lowest effort, highest leverage)
3. Phase 3: C SDK single-header (biggest ceremony reduction)
4. Phase 4: Rust SDK proc macro crate
5. Phase 5: Swift SDK with Swift macros
6. Phase 6: aro new plugin CLI, docs generation, migration guide

---

📄 Full proposal: Proposals/ARO-0073-plugin-sdk.md (once merged)

We'd Love Your Feedback

• Does the SDK API feel natural for your language of choice?
• Are there pain points we missed?
• Which phase/language should be prioritized?
• Should we support additional languages (Go, Zig, ...)?
• Any concerns about the unified ABI v2 approach?
• Thoughts on the Python persistent subprocess mode vs embedding?

Please share your thoughts — this is a significant DX investment and we want to get it right.

[KrisSimon]

Gaps Identified After Cross-Referencing with The Plugin Guide

After reading the entire Plugin Guide book cover-to-cover and comparing it against this proposal, here is a comprehensive list of topics, features, and concepts documented in the book that ARO-0073 does not address. Feedback welcome on which of these should be folded into the proposal vs. left for follow-up work.

---

Major Feature Gaps (entire capabilities missing from the proposal)

1. C++ Plugin SDK — The book dedicates Chapter 9 to C++ plugins (extern "C" bridge, RAII, exception safety, nlohmann/json, OpenCV integration). The proposal provides SDKs for Swift, Rust, C, and Python only. C++ is absent from the SDK plan, the scaffolding CLI, and the comparison matrix.

2. System Objects — Chapter 13 documents aro_object_read, aro_object_write, aro_object_list, the Source/Sink/Bidirectional model, and capability declarations (readable, writable, enumerable, watchable). The proposal's unified ABI v2 does not include system object functions at all. This is a substantial plugin feature with no SDK support proposed.

3. Hybrid Plugins — Chapter 14 documents plugins that combine native code with .aro files (e.g., Argon2 hashing in Swift + authentication workflows in ARO). The proposal does not address how the SDK interacts with the aro-files provider type or the hybrid loading sequence.

4. Pure ARO Plugins — The book documents plugins that are aro-files only (no native code). The proposal's scaffolding does not generate pure ARO plugins.

5. Lifecycle Hooks — Appendix B documents optional aro_plugin_init() (one-time initialization) and aro_plugin_shutdown() (cleanup on unload). The proposal's unified ABI v2 lists only aro_plugin_info, aro_plugin_free, aro_plugin_execute, aro_plugin_qualifier, and aro_plugin_on_event. Init/shutdown lifecycle hooks are missing.

6. Plugin Unload/Reload at Runtime — Language Guide Ch. 26 documents UnifiedPluginLoader.shared.unload/reload. Not mentioned in the proposal.

---

Qualifier-Specific Gaps

7. Qualifier Chaining — There is no syntax to chain two plugin qualifiers in a single reference (e.g., <value: collections.reverse stats.sort>). The proposal does not address qualifier composition or pipelining.

8. Parameterized Qualifiers — Built-in qualifiers like clip and take accept parameters via the with clause, but plugin qualifiers receive only the bare value (no parameters). The SDK proposal does not address this asymmetry.

9. The Two Roles of Qualifiers — Language Guide Ch. 10 documents that qualifiers serve as both operation selectors and field navigation. The proposal treats them only as transformation functions.

10. Ambiguity Resolution — When a data field has the same name as a built-in operation (e.g., a field called length), the book documents resolution rules. Not addressed in the proposal.

11. Qualifier Precedence — If two plugins register the same namespaced qualifier, the second silently overwrites the first. No conflict detection or resolution mechanism is proposed.

12. Built-in Qualifiers Not in QualifierRegistry — Built-in qualifiers (hash, length, uppercase, etc.) are hardcoded in ComputeAction, not registered in QualifierRegistry. There is no single source of truth for "all available qualifiers." The SDK does not address this.

---

Architecture and Runtime Gaps

13. Context Propagation — Appendix B documents the _context_ prefix for passing context information (requestId, etc.) to plugins. The proposal's SDK helpers do not expose this.

14. The AROService Protocol — Language Guide Ch. 25 documents the Swift-native AROService protocol with init(), call(), and shutdown() methods. The proposal routes services through aro_plugin_execute("service:<method>") but does not address the native Swift protocol.

15. Service Shutdown — The AROService protocol has a shutdown() lifecycle method. The proposal does not address service cleanup.

16. ResultDescriptor and ObjectDescriptor — Chapter 5 documents these types that plugin actions receive. The proposal's simplified ActionInput type does not expose the full descriptor model (base, qualifiers, specifiers).

17. Preposition-Based Behavior Dispatch — The book shows how different prepositions can trigger different logic within an action. The proposal does not address this pattern.

18. Async Operations in Plugins — Chapter 6 documents the semaphore/Task pattern for bridging async Swift into the synchronous C ABI. No SDK addresses async support.

---

Build, Compilation, and Distribution Gaps

19. aro build Binary Embedding — Chapter 7 documents that aro build compiles plugins and base64-embeds them into the application binary. The proposal does not mention how SDK-based plugins interact with this build process.

20. Cross-Compilation — The book covers cross-compilation targets. Not in the proposal.

21. Platform-Specific Manifest Configuration — Appendix A documents the platforms section with per-platform min-version, architectures, and build overrides. Not in the proposal.

22. requirements.system Manifest Field — The FFmpeg chapter documents system dependency declarations. Not in the proposal.

23. Single-File Swift Plugins — The book documents that simple Swift plugins can be a single .swift file (no Package.swift needed). The proposal generates only package-based projects.

24. Multi-File C Plugins — The FFmpeg chapter shows header + multiple .c file structure. The proposal scaffolds only a single plugin.c.

---

Testing Gaps

25. Component Testing with .aro Files — Chapter 15 documents testing plugins by writing ARO test files and running them with aro run. The proposal provides SDK-level unit test helpers but not component or integration testing.

26. Memory Safety Testing — AddressSanitizer and Valgrind guidance. Not in the proposal.

27. CI/CD Templates — GitHub Actions cross-platform matrix workflows. Not in the proposal.

28. Test Coverage Tools — cargo-tarpaulin, Xcode coverage, Python coverage. Not in the proposal.

29. Boundary Value Testing — Unicode handling, embedded nulls, numeric boundaries. Not in the proposal.

---

Publishing and Ecosystem Gaps

30. Repository Structure Standard — Chapter 16 documents the expected layout (README, LICENSE, CHANGELOG, CONTRIBUTING, examples/, docs/). The proposal's scaffolding does not generate this full structure.

31. Deprecation Strategy — The book documents a deprecations manifest field and deprecation workflow. Not in the proposal.

32. Security Policy — SECURITY.md, CVE process, responsible disclosure. Not in the proposal.

33. API Documentation Standards — The book specifies per-action argument tables, error code tables, and example sections. The proposal mentions aro plugins docs but does not define the output format.

---

Error Handling Gaps

34. Standard Error Codes — Appendix C defines numeric error codes 0-10 (SUCCESS through RATE_LIMITED). The proposal does not reference or integrate these.

35. Domain-Specific Error Categories — Validation, I/O, Authentication, Rate Limiting error code tables. Not in the proposal.

36. Error Code Naming Convention — {CATEGORY}_{SPECIFIC_ERROR} pattern. Not in the proposal.

37. Multiple Errors Format — Validation responses with arrays of error objects. Not in the proposal.

---

Performance Gaps

38. Optimization Techniques — Compile-time regex, zero-copy string handling, SIMD, profile-guided optimization. Not in the proposal.

39. Rust Release Profile Settings — LTO, codegen-units, panic = "abort". Not in the proposal.

40. catch_unwind for Panic Safety — Preventing Rust panics from crossing the FFI boundary. The SDK hides unsafe blocks but does not mention panic safety.

41. JSON Overhead Batching — The book suggests batching multiple operations to reduce serialization overhead. Not in the proposal.

42. GPU Acceleration — Python-specific CUDA detection, quantization, and OOM handling. Not in the proposal.

---

CLI Command Gaps Not Mentioned in Proposal

43. aro plugins list --verbose
44. aro plugins validate
45. aro plugins export / aro plugins restore
46. aro plugins rebuild
47. aro actions list (debugging action registration)
48. aro check (manifest validation)

---

Questions for Discussion

• Which of these gaps are must-have for the SDK proposal vs. follow-up work?
• Should C++ get a full SDK, or is the C SDK header sufficient for C++ plugins?
• Should system objects (aro_object_read/write/list) be part of the unified ABI v2?
• Should aro_plugin_init/aro_plugin_shutdown lifecycle hooks be in ABI v2?
• Is qualifier chaining/composition worth addressing, or are sequential Compute statements sufficient?
• Should parameterized qualifiers be supported for plugins (not just built-ins)?

[KrisSimon]

Supplementary Findings After Deep-Reading The Plugin Guide

After reading every chapter and appendix of The Plugin Developer's Guide in full, here are concrete, specific findings that go beyond the high-level gap list above.

---

Two Coexisting C ABI Signatures — Migration Path Unclear

The book documents two completely different function signatures that coexist today:

Service ABI (Chapter 2 — the older pattern):

int32_t service_call(
    const char* method,        // method name
    const char* args_json,     // input JSON
    char**      result_json    // out-pointer for result
);
// Returns 0 for success, non-zero for error

Action ABI (Chapter 5 — the newer pattern):

char* aro_plugin_execute(
    const char* action,        // action name
    const char* input_json     // input JSON
);
// Returns heap-allocated JSON string (caller frees via aro_plugin_free)

The proposal's unified ABI v2 keeps only the action-style (2-param, return value) and routes services through aro_plugin_execute("service:<method>", ...). But the book's SQLite and Zip examples (Chapter 11) use the 3-param out-pointer service ABI exclusively. The SDK needs to clearly document the migration path for plugins currently using the service ABI — or the SDKs need to handle both patterns.

---

Production Plugin Patterns the SDK Should Enable

The book contains four substantial production-grade examples that test whether the SDK is expressive enough:

1. FFmpeg Plugin (Chapter 12) — Multi-file C plugin with:

• 5 actions (transcode, extract-audio, thumbnail, info, clip)
• System library detection via pkg-config
• Reverse-order resource cleanup (avcodec_free_context before avformat_close_input)
• Timeout handling for long-running transcodes
• The requirements.system: [ffmpeg, libavcodec, libavformat] manifest field

Question: Can the C SDK's ARO_ACTION macro handle the resource lifecycle pattern where cleanup order matters? The arena allocator frees in bulk — is that sufficient for plugins managing external library resources like FFmpeg contexts?

2. Auth Hybrid Plugin (Chapter 14) — Swift native + ARO files:

• Argon2id password hashing + JWT token generation in Swift (via JWTKit, Argon2Swift packages)
• Authentication workflows defined in .aro feature sets
• The Invoke action for calling ARO feature sets from within the plugin
• Loading sequence: native code first, then ARO files

Question: How does the @AROPlugin Swift macro interact with the aro-files provider? If a hybrid plugin uses both @Action methods and .aro feature sets, how are they coordinated?

3. LLM Inference Plugin (Chapter 10) — Python with:

• Model caching across calls (requires persistent state)
• Batch processing (EmbedBatch action for bulk embeddings)
• GPU detection and fallback (torch.cuda.is_available())
• 4-bit quantization via BitsAndBytesConfig
• Lazy imports for heavy dependencies (transformers, torch)
• Memory management (torch.cuda.empty_cache())

The proposal's Python persistent mode addresses model caching, but doesn't address GPU lifecycle management, quantization config, or the lazy import pattern. These are critical for ML plugins.

4. Redis System Object Plugin (Chapter 13) — C plugin providing:

• aro_object_read(identifier, qualifier, output) — read string/hash/list/set by type qualifier
• aro_object_write(identifier, qualifier, value) — write with TTL support
• aro_object_list(pattern, output) — list keys by glob pattern
• Capability declarations: readable, writable, enumerable, watchable

The system objects ABI is entirely separate from both the action and service ABIs. It needs its own SDK support, or at minimum, acknowledgment in the unified ABI v2.

---

The aro-templates Provider Type

Appendix A lists aro-templates as a valid provider type alongside aro-files, but the proposal does not mention it. Templates are embedded during aro build and available via the Render action. If the SDK scaffolding generates plugin.yaml files, it should support this provider type.

---

The Invoke Action for Cross-Boundary Calls

Chapter 14 documents the Invoke action that allows calling ARO feature sets from within plugin workflows. This is the bridge between native plugin code and ARO-defined business logic in hybrid plugins. The proposal's event system (@OnEvent) is one-directional (runtime → plugin). Invoke enables the reverse direction (plugin → runtime). Neither the SDK nor the unified ABI addresses this.

---

Specific Testing Patterns the SDK Testing Helpers Should Support

Chapter 15 defines a testing pyramid with concrete examples:

Unit tests (per-language, already in proposal):

• Rust: #[test] with direct function calls
• Swift: XCTest with @_cdecl function invocation
• C: Custom test harness with assert macros
• Python: pytest

Component tests (NOT in proposal — critical gap):

(* Test file: tests/crypto-tests.aro *)
(Application-Start: Crypto Tests) {
    (* Test 1: Hash produces consistent results *)
    Hash the <hash1: sha256> from "hello".
    Hash the <hash2: sha256> from "hello".
    Compare the <hash1> against the <hash2>.
    When <comparison: not-equal> {
        Log "FAIL: Hash not deterministic" to the <console>.
        Return an <Error: status> for the <test>.
    }
    Log "PASS: Hash deterministic" to the <console>.
}

Run with aro run ./tests/crypto-tests.aro. This tests the plugin through the ARO runtime, catching issues that unit tests miss (JSON serialization bugs, registration errors, qualifier resolution failures). The SDK's testing helpers should include a way to run these component tests programmatically.

Memory safety tests (NOT in proposal):

• C/Rust: Compile with -fsanitize=address and run tests
• Valgrind on Linux: valgrind --leak-check=full aro run ./tests/
• The book recommends adding these to CI

CI/CD template (NOT in proposal):
The book provides a complete GitHub Actions workflow with cross-platform matrix (macOS, Ubuntu), installing ARO, building plugins, running unit + component + integration tests.

---

The "Code is the Error Message" Pattern in Plugin Context

Chapter 3 and Appendix C describe how plugin errors are presented to the user:

Cannot Hash the <digest> from the <input>.
Plugin error: Unsupported algorithm 'sha999'

The runtime constructs this message by combining the ARO statement that failed with the plugin's error message. This means plugin error messages should be concise and specific — they'll be appended to the statement context. The SDK's ActionOutput::error() / aro_error() / Result::Err() helpers should document this convention, and perhaps enforce a max length or formatting guideline.

---

Book-Recommended Best Practices the SDK Should Enforce or Encourage

Across all chapters, these recurring recommendations could be built into the SDK:

1. Validate input early (Ch 6, 7, 8) — SDK input accessors could return Result/Optional types that force handling
2. Include context in errors (Appendix C) — SDK error builders could require a message + details structure
3. Don't hold references across calls (Ch 6) — SDK lifecycle documentation should warn about this
4. Use snprintf not sprintf (Ch 8) — C SDK should use safe formatting internally
5. Document system requirements (Ch 11, 12) — Scaffolding should generate a README section for system deps
6. Pin dependency versions (Ch 11) — Scaffolding should use exact: in SPM, specific versions in Cargo.toml
7. Test with boundary values (Ch 15) — SDK test helpers could include common boundary test fixtures (empty string, empty array, unicode, very large input, null-like values)

[KrisSimon]

Proposal Updated — Major Revision Based on Feedback

The proposal has been substantially rewritten to address the gaps identified from the Plugin Guide book review and the feedback discussion. Here is a summary of what changed and what was decided.

Key Design Decision: No Backward Compatibility

ARO is pre-1.0. Clean code over backward compatibility. The old service ABI (aro_plugin_init + _call with 3-parameter out-pointer signature) is removed entirely. The AROService protocol and ServiceRegistry are deprecated. One clean path replaces three overlapping mechanisms. All example plugins will be rewritten. The Plugin Guide book will be updated.

What Was Added

C++ Plugin SDK — Full SDK with aro_plugin_sdk.hpp header providing RAII wrappers, aro:: namespace with type-safe templates, automatic try/catch at the extern "C" boundary, and std::vector/std::string/std::map serialization. Scaffolding via aro new plugin --lang cpp.

System Objects — aro_object_read, aro_object_write, aro_object_list added to the ABI. SDK support via @SystemObject (Swift), #[system_object] (Rust), ARO_SYSTEM_OBJECT (C/C++), @system_object (Python). Capability declarations: readable, writable, enumerable, watchable.

Lifecycle Hooks — aro_plugin_init() (one-time setup: DB connections, model loading) and aro_plugin_shutdown() (cleanup: close connections, flush buffers) added as optional ABI functions. SDK support via @OnInit/@OnShutdown (Swift), #[init]/#[shutdown] (Rust), ARO_INIT/ARO_SHUTDOWN (C/C++), on_init/on_shutdown (Python).

Hybrid Plugins & ARO Files — Documented aro-files provider type (event handlers, reusable feature sets, pure ARO plugins). Added aro-templates provider type. Documented hybrid loading sequence (native first, then ARO files, then templates). Scaffolding supports --lang aro for pure ARO plugins and --hybrid flag.

Plugin-to-Runtime Invocation — aro_plugin_invoke callback mechanism enables plugins to call ARO feature sets. SDK wrappers for all languages. This bridges the gap for hybrid plugins where native code needs to trigger ARO business logic.

Plugin Unload/Reload — Documented UnifiedPluginLoader.shared.unload/reload for hot-reloading during development.

Qualifier Improvements:

• Parameterized qualifiers — Plugin qualifiers now accept with clause parameters, just like built-in clip/take. Input JSON includes _with key.
• Qualifier chaining — Pipe syntax: <result: stats.sort | list.take>. Evaluated left-to-right.
• Conflict resolution — Load-time error on duplicate qualifiers. Application manifest can alias plugin handles.
• Ambiguity resolution — Documented resolution order: plugin qualifiers → built-in operations → data field access. Recommendation: always use namespaced qualifiers.
• Unified registry — Built-in qualifiers registered in QualifierRegistry alongside plugin qualifiers. Single source of truth.

Full Descriptor and Context Access — Input JSON now includes result (base, qualifiers, specifiers), source (base, specifiers), preposition, and _context (requestId, featureSet, businessActivity). SDK helpers expose all of these.

Preposition-Based Dispatch — Documented pattern where different prepositions trigger different behavior. SDK exposes input.preposition.

Async Support — Swift SDK bridges async functions via Task + semaphore automatically. Documented as a macro feature.

aro build Binary Embedding — Explained why base64 embedding is needed: the binary must be self-contained, runnable on any machine without the Plugins/ directory.

Platform-Specific Manifest Configuration — platforms section with per-platform min-version, architectures, and build overrides.

requirements.system Manifest Field — System dependency declarations with per-platform install commands.

Deprecation Strategy — deprecations array in aro_plugin_info with feature, message, since, remove_in. Runtime emits warnings.

Standard Error Codes — Codes 0-10 (SUCCESS through RATE_LIMITED) included in all SDKs as constants/enums.

Domain-Specific Error Categories — VALIDATION, IO, AUTH, RATE_LIMIT categories with naming convention {CATEGORY}_{SPECIFIC_ERROR}.

Performance Section — Compile-time regex, zero-copy strings, SIMD, PGO, Rust release profiles, Python GPU acceleration (CUDA detection, quantization, OOM handling).

CLI Commands — Added: aro plugins list --verbose, aro plugins validate, aro plugins export/restore, aro plugins rebuild, aro actions list, aro check.

Component Testing — .aro test files that test plugins through the runtime. Run with aro run ./tests/. Memory safety testing with AddressSanitizer/Valgrind.

What Was Removed/Changed

• No ABI v2 versioning — Clean break, no "abi": 2 field, no v1 detection
• Old service ABI removed — aro_plugin_init (service discovery), 3-parameter _call, AROService protocol, ServiceRegistry
• Cross-compilation removed — Out of scope for pre-1.0. ARO compiles on the target it runs.
• Backward compatibility section removed — Replaced with "affected code" appendix listing everything that needs rewriting

What Was NOT Added (Explicitly Out of Scope)

• Cross-compilation
• Go/Zig/other language SDKs
• Central plugin registry
• Virtual environment management (beyond documenting it)
• CI/CD templates (left to plugin authors)
• Repository structure standard (left to publishing guide)

Impact: Code and Books to Update

The proposal now includes Appendix C listing every file that must be modified, every example to rewrite (9 examples), and every book chapter to update. This is a significant effort but results in a much cleaner, more consistent plugin system.

---

📄 Updated proposal: Proposals/ARO-0073-plugin-sdk.md (once merged)

[KrisSimon]

📄 The updated proposal is now on the feature branch and ready for review:

• Full proposal: Proposals/ARO-0073-plugin-sdk.md