Bloom v2 #158
Merged
Bloom v2 #158
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Planning docs for issue #609 (Bloom V2 - P0) Design decisions: - Embedded Ansible for deployment execution - Maintain V1 bloom.yaml format (identical) - Fail-fast linear playbook execution - Single binary with subcommands (deploy, webui, config) - Web UI generates bloom.yaml only (no deployment in browser) Documents: - BLOOM_V2_PLAN.md: 7-week implementation plan with 5 phases - BLOOM_V2_ARCHITECTURE.md: Technical architecture and design - BLOOM_YAML_SPEC_V1.md: V1 config spec for compatibility
Test coverage: - API endpoints: schema, validate, generate (17 tests) - Web UI interactions: form rendering, validation, YAML generation (10 tests) - Shared resources: server management, session handling Test infrastructure: - Makefile targets: test, test-api, test-webui - Test runner script with dependency checks - Comprehensive test documentation All V1 fields covered with both happy path and error cases.
Allows running tests without local pip installation using marketsquare/robotframework-browser Docker image. Usage: make test-docker
Run container with host user ID to avoid permission errors reading test files
Tests were incomplete and blocked on dependency issues
TLS_CERT and TLS_KEY were showing for additional nodes. Added FIRST_NODE=true dependency to ensure certificate configuration is only available for first node. Added comprehensive Robot Framework test suite covering API endpoints, browser UI interactions, and conditional field visibility.
Implemented comprehensive field format validation in validator.go: - Domain validation with regex pattern checks - IP address validation rejecting loopback/unspecified - URL validation requiring http/https schemes - Email validation with regex pattern - File path validation preventing empty/whitespace Added OIDC authentication fields to schema: - OIDC_ISSUER_URL for authentication provider - OIDC_ADMIN_EMAIL for admin user email - Both fields only visible for first node Added comprehensive Robot Framework field validation test suite covering all format validators and edge cases.
- Add missing arguments from V1 to match schema parity: - ADDITIONAL_TLS_SAN_URLS (SSL/TLS Configuration) - ROCM_DEB_PACKAGE (Advanced Configuration) - RKE2_INSTALLATION_URL, RKE2_VERSION, RKE2_EXTRA_CONFIG (Advanced Configuration) - Update defaults to match V1: - ROCM_BASE_URL: 7.0.2 (was 6.3.2) - CLUSTERFORGE_RELEASE: v1.5.2 specific release URL - PRELOAD_IMAGES: populated with rocm/pytorch and rocm/vllm images - Remove OIDC fields not present in V1: - OIDC_ISSUER_URL - OIDC_ADMIN_EMAIL - Implement V1-style section headers: - Green background (#4CAF50) matching V1 - White text with proper padding - Section containers with proper styling - Add section visibility logic: - Hide sections when all fields underneath are hidden - Works on initial render and dynamic field changes - Improves UX by removing empty sections - Add comprehensive test suite: - Browser tests for new V2 arguments (ADDITIONAL_TLS_SAN_URLS, ROCM_DEB_PACKAGE, RKE2_*) - Tests for default value validation - Section visibility behavior tests - Section header styling tests - All 31 tests passing (100%)
- Change default port from 8080 to 62078 (matches V1 behavior) - Add PortSpecified flag to track explicit port configuration - Implement findAvailablePort() to auto-discover available port (tries 100 ports) - Implement isPortAvailable() helper for port availability checks - Add conditional logic: fail if explicit port taken, auto-discover if default - Update test configuration to use fixed port 62080 with health checks - Update all Robot Framework test files to use new port - All 31 Robot Framework tests passing Relates to #609
- Add HTML5 validation patterns for DOMAIN, IP, URL, EMAIL, and file path fields - Implement blur-triggered validation with custom error messages - Add required attribute handling for visible/hidden fields dynamically - Add visual feedback with red/green borders for invalid/valid states - Use setCustomValidity() to show descriptive error messages - Match V1 domain pattern: lowercase alphanumeric with dots/hyphens - Validate on blur (focus leave) to avoid interrupting user input - Clear validation errors on focus/input for better UX Relates to #609
…tibility - Add Pattern and PatternTitle fields to Argument struct for HTML5 validation - Implement validation patterns matching V1 for all fields: - DOMAIN: lowercase domains with proper label structure - SERVER_IP: IPv4 address validation (0-255 per octet) - CLUSTER_DISKS: comma-separated device paths - CLUSTER_PREMOUNTED_DISKS: comma-separated disk names - ADDITIONAL_TLS_SAN_URLS: changed from array to string, comma-separated domains - TLS_CERT/TLS_KEY: absolute file paths with extension validation - RKE2_VERSION: semantic version format with optional rke2 suffix - ROCM_BASE_URL: HTTP/HTTPS URL validation - Remove validate button from UI (validation now on blur) - Reorganize Robot Framework tests: - Delete 5 verbose test files (886 lines) - Add 3 focused test suites (180 lines): api.robot, ui.robot, validation.robot - Reduce from 31 tests to 10 essential tests - Update test to fill CERT_OPTION required field
- Replace download with save to server's current working directory - Add filename input field (defaults to bloom.yaml) - Create /api/save endpoint that writes file to cwd - Filter YAML output to only non-default values - Always include FIRST_NODE and GPU_NODE regardless of value - Show success message with absolute file path Backend changes: - Add SaveRequest type with config and filename fields - Add handleSave() function for /api/save endpoint - Modify GenerateYAML() to skip default values - Add isDefaultValue() helper for default comparison Frontend changes: - Replace downloadYAML() with saveYAML() - Add filename input in preview section - Call /api/save instead of creating blob download - Show saved file path in success message
… execution Add pkg/ansible/runtime package enabling Ansible playbook execution without external dependencies. Eliminates need for Docker, Python, or Ansible installation by using go-containerregistry for image management and Linux namespaces for container isolation. Core components: - container.go: Image pull/cache using go-containerregistry (~500MB) - executor_linux.go: Linux namespace execution with /host mount - executor_other.go: Non-Linux platform stub - playbook.go: RunPlaybook() orchestration with embedded playbooks Embedded playbooks (813 lines): - cluster-bloom.yaml: Full RKE2 cluster deployment - hello.yml: Test playbook - Updated to UPPERCASE variable convention (FIRST_NODE, GPU_NODE, etc.) Dependencies added: - github.com/google/go-containerregistry v0.20.7 - golang.org/x/sys v0.39.0 Architecture documented in tmp/ANSIBLE_ARCHITECTURE.md, tmp/BLOOM_V2_PRD.md. Enables: sudo bloom ansible bloom.yaml Relates to #609
Wire up ansible subcommand to execute playbooks with config loading and namespace isolation. Users can now run "sudo bloom ansible hello.yml" for direct playbook execution or "sudo bloom ansible bloom.yaml" for full deployment from config. Command flow: - Load and validate bloom.yaml config files - Check root permissions (required for namespace operations) - Handle namespace re-execution via __child__ arg - Execute playbooks in isolated environment - Cache Ansible runtime image in .bloom/rootfs/ Changes: - Add config loader (internal/config/loader.go) - Implement ansible subcommand in main.go with runAnsible() - Move cache location from /var/lib/bloom to .bloom/ in working directory - Fix hello.yml playbook with default DOMAIN value - Update .gitignore for .bloom/ directory Completes Phase 4 of Bloom V2 implementation. Tested end-to-end with hello.yml playbook successfully creating /tmp/bloom-hello.txt. Relates to #609
Create YAML-native schema for bloom.yaml configuration with reusable type definitions and comprehensive test coverage. - Add schema/bloom.yaml.schema.yaml with 9 custom types - domain, domainList, ipv4, url, devicePath, diskList - certFilePath, keyFilePath, rke2Version - Include valid/invalid examples for each type in schema - Implement pattern tests loading examples from schema file - Add storage validation constraint (exactly one option required) - Fix missing URL pattern for RKE2_INSTALLATION_URL in schema.go Tests verify all 9 type patterns against their examples. Schema serves as single source of truth for validation. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
… of truth Eliminated 270 lines of hardcoded Go schema definitions by loading directly from YAML schema file. Schema now drives both backend validation and frontend test generation, ensuring consistency across system. Removed 245 lines of duplicate Robot Framework tests - validation now schema-driven with examples extracted directly from YAML. All 9 pattern types tested via UI using schema examples. Fixed frontend bug where fields with "URL" in name received incorrect input type. Fixed GPU_NODE checkbox visibility handling. BREAKING CHANGE: Schema loading now requires schema/bloom.yaml.schema.yaml at runtime
- Move hyphens to start of character classes in all patterns - Browser regex doesn't support \- escape inside character classes - Affects domain, domainList, diskList, certFilePath, keyFilePath patterns - Changed diskList to require absolute paths (/mnt/disk1) instead of names (disk1) - All Go unit tests pass with updated patterns Fixes browser validation errors in web UI
Add 8 Robot Framework test cases: - Generate valid first node config - Generate valid additional node config - Generate config with TLS certificates - Generate config with advanced ROCm/RKE2 options - Test /api/generate endpoint directly - Test API rejection of invalid configs - Verify minimal YAML output (only non-defaults) - Test field visibility affects generated config All tests validate proper form behavior and YAML generation
…ctoring - Mark Phase 4 (Testing) as complete - Add Phase 4.5 documenting schema refactoring work - Update architecture to show YAML schema as single source of truth - Document schema-driven testing approach
- Escape hyphens as \- inside character classes for browser regex - Escape forward slashes as \/ in diskList pattern - Fixes JavaScript regex syntax errors in browser validation - All Go unit tests still pass with escaped patterns
- Add Check No Console Errors keyword to detect browser JavaScript errors - Check console before/after each field and after each example - Catches regex syntax errors and other JavaScript issues during test execution - Provides context about which field/example caused the error
- Rename UI title from "Bloom" to "Cluster-Bloom" Configuration Generator - Update description for clarity - Add Python __pycache__ to gitignore - Update UI tests to match new branding
Replaces complex boolean condition strings with simple field lists for constraint definitions. Implements matching validation on backend (Go) and frontend (JavaScript). Schema changes: - Simplify one-of constraints from conditions to field lists - Makes schema more readable and maintainable Backend changes: - Add field-based constraint validation functions - Integrate constraint checks into main validation pipeline - Add dynamic unit tests that adapt to schema changes - Expose constraints via /api/schema endpoint Frontend changes: - Implement client-side constraint validation matching backend - Move error display below submit button for better visibility - Remove auto-hide timeout so errors persist Test improvements: - Add shared keywords.resource with form debugging utilities - Add TRY/EXCEPT blocks to capture form state on failures - Fix constraint violations in existing tests - Add dynamic Robot Framework tests for all constraints
…keywords Reduce duplication across test suite by extracting common patterns into reusable keywords resource file. Removes redundant test files whose coverage is now provided by dynamic constraint tests. Changes: - Create keywords.resource with shared setup, validation, and error handling - Delete constraint_validation.robot (covered by dynamic tests) - Delete validation.robot (merged into ui.robot) - Simplify config_generation.robot (293 → 131 lines) - Simplify ui.robot (79 → 49 lines) - Update all .robot files to use centralized BASE_URL - Remove dead code from ConstraintTestGenerator.py - Fix rke2_test.go function call signature Net reduction: 380 lines while maintaining test coverage.
Remove legacy Bloom v1 imperative Go implementation in favor of clean v2 architecture with Ansible playbooks and config generation tools. Deleted v1 components: - cmd/*.go (7 files) - v1 CLI commands and wizard - main.go - v1 entry point - pkg/ v1 files (28 files) - imperative deployment code - steps.go, rke2.go, rocm.go, disks.go, os-setup.go, packages.go - view.go, webhandlers.go, webmonitor.go (v1 web interface) - args/, binaries/, manifests/, scripts/, system/, templates/ - web/ (6 files) - v1 web interface (replaced by cmd/bloom/web/) - tests/e2e/ (26 files) - v1 end-to-end tests - tests/integration/ (60 files) - v1 integration tests - tests/ui/ (21 files) - old UI tests (replaced by Robot Framework) Retained v2 components: - cmd/bloom/ - v2 clean binary (webui + ansible commands) - internal/config/ - v2 schema-driven validation - pkg/ansible/runtime/ - v2 Ansible executor - pkg/webui/ - v2 web server - tests/robot/ - v2 Robot Framework tests - schema/ - v2 YAML schema Impact: 150 files deleted, ~25K lines removed
Running 'bloom' without arguments now starts the web UI by default, simplifying the most common use case. Before: bloom webui After: bloom The 'bloom webui' command still works for explicit invocation. Updated help text to show default behavior.
Add convenience wrapper in root that delegates to cmd/bloom, allowing developers to use 'go run .' instead of 'go run ./cmd/bloom'.
- Moved all files from cmd/bloom/ to cmd/ - Migrated CLI from manual arg parsing to Cobra framework - Added table-formatted config field documentation to help text - Config fields shown at bottom of help in same order as webui - Added bloom binary to .gitignore Benefits: - Simpler directory structure - Auto-generated help and usage messages - Proper flag handling - Shell completion support - Embedded config reference in --help output
Add diagnostic playbook to verify config loading pipeline and validate that config keys must exist in schema. - Add print-config.yml playbook that displays all loaded config values - Add --playbook flag to ansible command for playbook override - Add unknown key validation in Validate() to catch typos - Add TestValidate_UnknownKeyRejected test - Update BLOOM_V2_PLAN.md to reflect completed Phase 1b Verified: Config successfully loaded from YAML, validated, and passed to Ansible as extra vars. All 8 test values correctly received by playbook.
Remove 402 lines of redundant test code. Schema-driven integration tests provide comprehensive coverage, making static tests redundant. Removed from validate_test.go: - TestValidate_RequiredFields (covered by TestValidate_RequiredFieldsFromSchema) - TestValidate_DomainPattern (covered by TestValidate_AllTypesWithSchemaExamples) - TestValidate_IPAddress (covered by TestValidate_AllTypesWithSchemaExamples) - TestValidate_Constraints (covered by TestValidate_ConstraintsFromSchema) - TestValidate_EnumValidation (covered by TestValidate_AllTypesWithSchemaExamples) Removed from validate_integration_test.go: - TestValidate_AllValidExamplesPass (duplicate of valid portion in AllTypesWithSchemaExamples) - TestValidate_AllInvalidExamplesFail (duplicate of invalid portion in AllTypesWithSchemaExamples) Kept: - TestValidate_ValidConfigs (quick sanity checks with complete configs) - All schema-driven integration tests (comprehensive, adapt to schema changes) All tests passing. Coverage improved through consolidation.
Replaced local Ansible execution with SSH connection from Alpine container
to Ubuntu host, enabling full systemd/apt/kernel access.
BREAKING CHANGE: Ansible now requires SSH keys mounted from host user's
~/.ssh directory. Container connects to host via SSH (127.0.0.1) instead
of local connection.
Key architectural changes:
- executor: SSH connection with mounted user SSH keys from /home/{username}/.ssh
- executor: detect actual user via SUDO_USER environment variable
- executor: use --become for privilege escalation instead of --connection=local
- cli: add --dry-run flag (translates to Ansible --check mode)
- playbook: fix boolean type preservation using JSON format for extra vars
- playbook: convert CLUSTER_DISKS from comma-separated string to list
- playbook: fix kubeconfig paths to use ansible_user instead of SUDO_USER
- playbook: fix ConfigMap name from cluster-DOMAIN to cluster-domain
- playbook: fix additional node command path to use /tmp/
Tested on QEMU Ubuntu 24.04 VM with 52 tasks completing successfully.
RKE2 cluster deployed with proper disk formatting, mounting, and kubeconfig
creation.
Achieve complete feature parity with Bloom V1 by implementing all missing deployment steps as Ansible playbook tasks. This completes the migration to V2's SSH-based architecture while maintaining all V1 functionality. Pre-K8s Phase: - Add system requirements validation (CPU, memory, disk, kernel modules) - Add /var/lib/rancher partition size check for GPU nodes - Add automatic cleanup on deployment (Longhorn mounts, RKE2 uninstall, disk wiping) - Add ROCm installation and GPU verification for AMD GPUs Post-K8s Phase: - Add container image preloading support - Add Longhorn preflight check using embedded longhornctl script - Add Longhorn deployment with manifests and node annotator - Add PVC validation to verify Longhorn operational status - Add Bloom ConfigMap creation with version tracking - Add ClusterForge installation from tarball releases Implementation: - Embed Longhorn manifests (YAML) and scripts using go:embed - Extract embedded resources to playbook directory at runtime - Always run cleanup tasks to ensure fresh deployment - Use default storage class for PVC validation tests All tasks tested successfully on QEMU Ubuntu 24.04 VM. Ref: /home/node/.claude/plans/melodic-gliding-cloud.md
…de networking Fixes three critical bugs in the deployment pipeline: - Schema defaults not applied: Added applyDefaults() to config loader to ensure RKE2_VERSION and other schema defaults are set when missing from bloom.yaml, preventing undefined variable errors in Ansible playbooks - Missing Ansible logs: Implemented automatic logging to bloom.log by passing working directory from parent to child process, opening log file via /host mount, and using io.MultiWriter to tee output to both stdout and file - Multi-node network failures: Fixed RKE2 deployment to use socket network (10.0.3.x) instead of user network (10.0.2.x) by detecting correct node IP, setting node-ip in RKE2 config, and updating kubeconfig with detected IP. Also fixed UUID retrieval for disk mounting using blkid instead of ansible_facts
…failures Longhorn manifests weren't applying due to being in subdirectory - RKE2 only auto-applies files directly in /var/lib/rancher/rke2/server/manifests/. Moved to flat structure. Fixed missing bloom.disk node labels required for Longhorn disk annotation. Node-annotator cronjob converts these labels to Longhorn annotations - V2 was missing this step entirely. Now dynamically generates labels for each mounted disk. Corrected ClusterForge installation to execute bootstrap.sh script (matching V1) instead of incorrectly installing Helm charts. Added DOMAIN parameter and optional CF_VALUES support. Added BLOOM_DIR variable to pass .bloom working directory from Go to Ansible, ensuring ClusterForge files download to correct location. Enabled Ansible verbose mode (-v flag) for better debugging output. These fixes resolve issues where Longhorn never deployed (namespace existed but no pods), ClusterForge bootstrap timed out, and OpenBao deployment failed due to missing Longhorn volumes.
Moved bloom.yaml.schema.yaml from schema/ to pkg/config/ and embedded it using go:embed directive. This eliminates runtime filesystem dependencies and simplifies schema loading by removing fallback path logic. Benefits: - Schema is always available (compiled into binary) - No need to bundle schema file separately - Simpler deployment (single binary) - Faster loading (no file I/O) Removed 24 lines of path-searching code.
Implement --tags flag support for selective Ansible task execution, enabling targeted deployment phases. Tasks reordered to match Bloom V1 execution flow with cleanup steps moved early in the playbook. Changes: - Add --tags CLI flag to execute specific task groups or phases - Implement bloom.log backup with timestamped archival (bloom-YYYYMMDD-HHMMSS.log) - Organize tasks into 6 composite phase tags: * validate_node: system requirement checks * cleanup: Longhorn, RKE2, and disk cleanup * prep_node: dependencies, storage, GPU setup * deploy_cluster: RKE2 installation and bootstrap * deploy_k8s_apps: Kubernetes applications * deploy_clusterforge: ClusterForge components - Reorder playbook tasks to execute cleanup before preparation - Reduce Ansible verbosity from -vvvv to -v for cleaner output - Remove redundant || true from commands with ignore_errors: yes Tested on pci production system with full disk lifecycle verification (mount/unmount + fstab management).
Add comprehensive validation as the first task in cluster-bloom.yaml playbook to detect existing RKE2 installations before deployment. - Add --cleanup flag to 'bloom ansible' command for optional cleanup - Check for running RKE2 services (rke2-server, rke2-agent) - Validate RKE2 data directory (/var/lib/rancher/rke2) existence and content - Validate RKE2 config directory (/etc/rancher/rke2) existence and content - Provide clear error messages suggesting --cleanup flag when conflicts found Prevents deployment failures by detecting existing cluster installations and guides users through proper cleanup process before proceeding.
Add comprehensive validation as the first task in cluster-bloom.yaml playbook to detect existing RKE2 installations before deployment. - Add --cleanup flag to 'bloom ansible' command for optional cleanup - Check for running RKE2 services (rke2-server, rke2-agent) - Validate RKE2 data directory (/var/lib/rancher/rke2) existence and content - Validate RKE2 config directory (/etc/rancher/rke2) existence and content - Provide clear error messages suggesting --cleanup flag when conflicts found Prevents deployment failures by detecting existing cluster installations and guides users through proper cleanup process before proceeding.
Add CLUSTER_SIZE field with options [small, medium, large] and default 'small' to support cluster deployment planning and size categorization. - Add CLUSTER_SIZE to bloom.yaml schema with enum validation - Update configuration reference documentation - Add Ansible playbook integration for future size-based logic - Extend Robot test suite with comprehensive CLUSTER_SIZE validation - Fix schema field count test (26 → 27 fields)
Replace separate --cleanup and --clean-disks flags with single --destroy-data flag to simplify the CLI interface. Consolidate RKE2 and disk validation into unified "Pre-deployment Data Safety Validation" task. Change validation approach from fail-fast to comprehensive reporting that collects all issues before failing. This significantly improves user experience by providing a simpler CLI interface, better organized validation, and more efficient problem resolution workflow. BREAKING CHANGE: --cleanup and --clean-disks flags removed, use --destroy-data instead. SKIP_MOUNT_VALIDATION logic removed entirely. Affects: - cmd/main.go: CLI flag consolidation and validation skip removal - pkg/ansible/runtime/playbooks/cluster-bloom.yaml: validation consolidation
…nections - Add complete ephemeral SSH key lifecycle management (generate, install, cleanup) - Replace host SSH key mounting with self-contained key generation - Implement robust user home directory detection supporting sudo scenarios - Add automatic authorized_keys backup/restore for safe key installation - Include comprehensive cleanup mechanisms (defer, signal handlers, exit cleanup) - Use ED25519 keys for enhanced security and performance - Support single-node deployment SSH connections without external dependencies Security improvements: - Eliminates dependency on persistent user SSH keys - Generates fresh ephemeral keys per deployment - Automatic cleanup prevents key accumulation - Proper user detection ensures keys placed in correct home directory
- Add signal handling for immediate SSH cleanup on process interruption - Implement multi-layer cleanup strategy (backup restore → manual removal → file deletion) - Add cleanup verification to confirm ephemeral key removal - Handle SIGINT, SIGTERM, SIGHUP, SIGQUIT signals for graceful shutdown - Provide manual intervention guidance if automated cleanup fails - Ensure SSH keys cannot persist in authorized_keys after unexpected termination Security improvements: - Critical security fix: prevents ephemeral keys from remaining in authorized_keys - Handles playbook failures, process kills, and other interruption scenarios - Multi-layered failsafe approach with verification and user notification - Maintains existing SSH keys while removing only bloom-ephemeral-key entries
- Add explicit private key specification (-i /root/.ssh/id_ephemeral) - Include IdentitiesOnly=yes to prevent fallback to other keys - Ensures Ansible uses only our ephemeral key, not system SSH keys - Improves security and reliability of SSH authentication
punasusi
approved these changes
Jan 15, 2026
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.