Add secrets loading from external providers (HashiCorp Vault) #18
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This introduces a secrets management system that allows loading secrets
from external providers and injecting them into template expansions.
New features:
- SecretProvider abstract class with caching support
- SecretManager for provider registry and template source creation
- VaultProvider for HashiCorp Vault KV v2 secrets engine
- Supports token, AppRole, and Kubernetes authentication
- VaultPlugin for configuration-based Vault integration
- Async template sources: ${vault:path/to/secret#key} syntax
- TemplateExpander refactored to support async source functions
The implementation follows a hybrid plugin + async template source pattern
where plugins handle connection setup and authentication, while template
sources provide flexible lazy-loading with caching.
Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
- Add global-setup.ts to start/stop Vault dev container automatically - Add integration-secrets workspace to vitest configuration - Remove skipIf conditions from vault integration tests - Fix TemplateExpander regex to support vault paths with slashes and hyphens while preserving the :- fallback delimiter parsing - Update todo plan with implementation status Tests now fail properly if Vault container fails to start, and GitHub CI can run vault tests without manual setup. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering>
- Add test:integration:secrets npm script - Include secrets tests in test:integration (and thus in npm test) - Add secrets integration test step to GitHub Actions CI workflow Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering>
- Add createTestContext() helper for simple context overrides - Add createTestSetup() helper for full test setup with temp directory - Update GetDeploymentPodsOperation and ComposeDownOperation tests as examples - Removes ~30 lines of boilerplate per test file Other test files can be migrated incrementally to use these helpers. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering>
Migrate remaining test files from manual setContext() calls to the new createTestSetup() helper, reducing boilerplate and improving consistency across the test suite. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering>
llambeau
force-pushed
the
secrets-sources
branch
from
26178e7 to
cd36e2e
Compare
Implements two new authentication methods for HashiCorp Vault: - OIDC: Interactive browser-based login for developer workstations - JWT: Non-interactive token-based auth for CI/CD pipelines Features: - VaultOidcHelper for browser-based OIDC flow with local callback server - Environment variable detection (VAULT_JWT, VAULT_JWT_ROLE, VAULT_OIDC_ROLE) - Integration tests with real Keycloak container - Docker network for cross-platform container communication Integration test infrastructure: - Keycloak container with test realm, client, and user - Docker network approach (works on both Linux and macOS) - Mocked browser opening to prevent interactive prompts in CI Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Update package-lock.json with the 'open' dependency - Remove pnpm-lock.yaml (project uses npm, not pnpm) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- New advanced/secrets.md page covering:
- Vault plugin configuration
- All authentication methods (token, AppRole, JWT, OIDC, Kubernetes)
- Secret reference syntax
- Environment variables reference
- Usage examples for dev, CI/CD, and production
- Vault server setup instructions
- Troubleshooting guide
- Updated configuration reference:
- Added vault plugin to built-in plugins list
- Added ${vault:path#key} syntax to variable expansion section
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Replace hardcoded RSA private key with runtime key generation using Node.js crypto.generateKeyPairSync(). This addresses GitGuardian security alert about stored private keys in the codebase. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
OIDC State Mismatch Fix: - Vault generates its own state parameter (prefixed with 'st_') regardless of what the client sends. The previous code compared the client-generated state with Vault's state, which always failed. - Now we pass the returned state/nonce from the callback to Vault's callback endpoint, letting Vault handle state validation internally. 1Password Connect Support: - The path normalization was incorrectly inserting '/data/' for all paths, which broke 1Password Connect engine paths (op/vaults/.../items/...). - Now detects 1Password paths and skips KV v2 normalization. Improved Error Messages: - Secret read errors now include the path and namespace being accessed. - Missing key errors now list available keys in the secret. Plugin Initialization Fix: - SecretManager is now created before Monorepo.init() so that plugins (like VaultPlugin) can register their providers during initialization. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Implements token caching for OIDC authentication to improve developer experience. Previously, every `emb up` command required opening the browser for OIDC authentication. Now tokens are cached and reused. Features: - Tokens cached in ~/.emb/vault-tokens/ with restricted permissions (0600) - Automatic expiry checking with 5-minute buffer before TTL - Graceful fallback to full auth if cache is corrupted/missing - Cache key based on vault address and namespace combination Changes: - New VaultTokenCache module for cache read/write operations - VaultProvider checks cache before initiating OIDC flow - VaultOidcHelper now returns TTL along with token - All auth methods updated to return AuthResult with TTL - 17 new unit tests for cache functionality Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Tokens are now encrypted at rest using AES-256-GCM: - Key derived from machine-specific data (hostname + username) via PBKDF2 with 100,000 iterations - Random 32-byte salt and 16-byte IV for each encryption - GCM authentication tag prevents tampering - Cached tokens cannot be used if copied to another machine or by another user The encrypted file format includes version, salt, iv, authTag, and encrypted data fields - all hex-encoded for safe storage. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Move 08-secrets-improvement-phase1.md to done/ - Add 09-secrets-dry-run-mode.md with implementation plan for `emb secrets list/validate/providers` commands Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Implements `emb secrets` subcommands for inspecting and validating
secret references without exposing actual values:
- `emb secrets` (or `emb secrets list`) - List all secret references
found in the configuration with provider, path, key, and component info
- `emb secrets validate` - Validate all secrets can be resolved,
with --fail-fast and --json options for CI/CD integration
- `emb secrets providers` - Show configured secret providers and status
New SecretDiscovery utility scans configuration to find all ${provider:path#key}
references. Supports vault, aws, azure, 1password, and op providers.
15 unit tests for the discovery logic.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
llambeau
force-pushed
the
secrets-sources
branch
from
3b18609 to
6e190ea
Compare
- Remove hardcoded SECRET_PROVIDERS set from SecretDiscovery.ts - Pass registered providers from SecretManager.getProviderNames() - Scan defaults and flavors sections for secret references - Add test for unregistered providers being ignored Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add CLI Commands section with emb secrets, validate, providers - Document OIDC token caching behavior - Add secrets topic to CLI reference Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
llambeau
force-pushed
the
secrets-sources
branch
from
85b82cf to
801074d
Compare
Previously, plugin configs used `"config": {}` in the schema which
allowed any object without validation. This caused incorrect plugin
configurations to pass schema validation silently.
Changes:
- Add PluginConfigItem with conditional validation based on plugin name
- Add schema definitions for all built-in plugins:
- VaultPluginConfig (address, namespace, auth)
- AutoDockerPluginConfig (glob, ignore)
- DotEnvPluginConfig (array of strings)
- EmbfilesPluginConfig (glob)
- Add missing Vault auth methods to schema (jwt, oidc)
- Add comprehensive unit tests for plugin config validation
Unknown plugins still pass schema validation (rejected at runtime)
to allow for third-party plugin extensibility.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This introduces a secrets management system that allows loading secrets from external providers and injecting them into template expansions.
New features:
The implementation follows a hybrid plugin + async template source pattern where plugins handle connection setup and authentication, while template sources provide flexible lazy-loading with caching.
Generated with Claude Code via Happy