How best to extend upon primitives with Automations or Templates

[jacobsmith928]

We expect that users (and agents) will need to customize their infrastructure, tap into partner solutions, and move from simple to robust. We envision a catalog of automations, some of which are maintained by Datum, some by the community / partners, and others configured by users for their own organizations.

On the infrastructure side, a good example of this is Railway's template system. On the more consumer side, Zapier's "Zaps" concept (Zapier Apps?) is a widely used and compelling example.

Below is a proposal from @scotwells for how to think about this on the backend "milo" side. We need to consider this, but also how to expose it and brand it for users of Datum Cloud.

Summary

• A template system Milo to define, share, and manage reusable blueprints for resources (proxies, secrets, export policies, workflows, etc). A template describes a consistent package — such as an environment, product, or tenant — that can be parameterized, versioned, and distributed through a catalog.
• The system simplifies delivery, ensures consistency, and provides governance across the full lifecycle of template creation, testing, release, and upgrades.

---

Suggested Features

• Template Definition
  • Schema / DSL for resources, workflows, parameters, and conditions
  • Composition, inheritance, and conditional logic
  • Hooks for pre/post instantiation and migration actions
• Authoring & Development
  • CLI / SDK for authoring, validation, linting, and dry-run previews
  • Local simulator and testing framework
  • Git integration for version-controlled development
• Catalog & Distribution
  • Official and community-supported template catalogs
  • Metadata: name, description, tags, version, owner, and preview
  • Search, discovery, and access control (global, org, project)
  • Import / export and sharing across environments
• Versioning & Lifecycle
  • Semantic versioning (major / minor / patch)
  • Deprecation and retirement of older versions
  • Upgrade and migration paths with roll forward / rollback
  • Canary / phased rollout and staged environments (dev, test, prod)
• Instance Management
  • Provenance tracking: template version, parameters, owner
  • Lifecycle operations: update, patch, delete / teardown
  • Drift detection and reconciliation against template definitions
• Governance & Policy
  • Guardrails and policy enforcement at definition and instantiation
  • Governance workflows for review and approval
  • Audit trails of edits, publishes, and instantiations
  • Certification and quality badges for trusted templates
• Observability & Operations
  • Metrics, logs, and dashboards for template usage and health
  • Alerts for failed instantiations, drift events, and upgrade issues
  • Support for scaling catalogs and concurrent instantiations
  • Dependency graph resolution and parallel provisioning

[drewr]

The three buckets I've been thinking about:

1. Templates. Terraform or CloudFormation or docker-compose-style declarative syntax that can deploy a stack of resources on your behalf. The benefit is that you don't have to write custom code to call all the APIs yourself.
2. Workflows. macOS Automator or Zapier-style bits of logic that you run occasionally or regularly or in response to a trigger from the environment. This can be to notify you about something or take some other action on your behalf, but it's something you likely want repeatedly.
3. Marketplace. Vendor-supplied recipes (maybe templates underneath) that stand up canned infrastructure that may not be a part of core Datum Cloud. For example, this could be a CNF that someone wants Datum to run at our edge.

3 is likely an extension of 1. 2 is somewhat orthogonal to the others. We should map out a quick design for 1 and 2 and figure out how we would sequence those.

[jacobsmith928]

Assuming that the most obvious starting point is templates, @drewr what are a handful that scream "I wish I had that around?" If we can't suggest 5-10 I think we might be at risk of premature optimization.

From other conversations, it seems that neither WAF (closely tied to the Proxy) or Metrics Export (suggested to move up to be on par with runtime, connections, assets, etc) fits the bill.

[drewr]

Right. I think I'd like to surface them from real automation and abstractions that we're building internally. Our product suite isn't quite large enough to need exactly this yet, but it's coming soon.