Skip to content

Conversation

@ephur
Copy link
Owner

@ephur ephur commented Oct 11, 2025

Summary

  • Add native Go-backed HCL parsing with auto-detect and Python fallback.
  • Batch-parse .tf files to reduce process overhead and improve compatibility/perf.
  • Integrate Go build/tests into Makefile and strengthen docs and tests.

Rationale

  • python-hcl2 has compatibility gaps for complex HCL2/Terraform constructs (and even some simple ones)
  • HashiCorp’s native parser (via Go) offers better correctness.
  • Spawning a process per file is slow; batching reduces overhead on large repos.

What Changed

  • HCL abstraction:
    • Added tfworker/util/hcl_parser.py:
    • parse_string, parse_file, and new parse_files (batch).
    • Auto-detects engine: uses Go helper if found; otherwise python-hcl2.
    • Forced override: TFWORKER_HCL_ENGINE=go|python. For go without helper, raises error.
  • Refactors:
    • tfworker/commands/config.py: use parse_hcl_string (no direct python-hcl2 import).
    • tfworker/util/terraform_helpers.py:
    • Collect .tf files and call parse_files once.
    • Logs and skips per-file errors; safe per-file fallback if batch fails.
  • Go helper:
    • tools/hcl2json:
    • go.mod and main.go; uses github.com/tmccombs/hcl2json/convert.
    • Supports --stdin (single) and --multi (batch) modes.
    • Unit tests, including repo fixtures (versions.tf).
  • Tests:
    • tests/util/test_hcl_parser.py: python vs go paths, multi-file, error surfaces.
    • tests/util/test_hcl_engine_parity.py: engine parity integration test on fixture; skips if helper missing.
    • Go tests updated with fixture coverage.
  • Makefile:
    • New go-build and go-test targets; integrated into test and ci-test.
    • Build from the tool subdir to avoid module root errors; place binary at repo root (./tfworker-hcl2json).
    • If tools/hcl2json/go.sum is missing, print guidance to run (cd tools/hcl2json && go mod tidy) and skip Go steps.
    • clean removes tfworker-hcl2json.
  • Docs (README):
    • Engine selection: auto vs forced modes; TFWORKER_HCL_BIN respected.
    • Batch parsing behavior for required provider discovery.
    • Go development guidelines (commit go.mod/go.sum, tidy instructions, vendoring optional).

Backwards Compatibility

  • Default behavior remains unchanged (python-hcl2) unless the Go helper is present or the engine is forced.
  • Error handling/logging during required provider discovery mirrors previous behavior.

Performance

  • Batch parsing sends all .tf files to a single Go process (--multi) vs spawning per file.
  • Python fallback continues per-file but only used when Go helper is unavailable or forced off.

How To Enable Go Parser

  • Build: make go-build (requires Go 1.20+; run go mod tidy once if go.sum missing).
  • Default auto-detect: ensure tfworker-hcl2json is on PATH or set TFWORKER_HCL_BIN.
  • Force: TFWORKER_HCL_ENGINE=go (errors if helper missing) or TFWORKER_HCL_ENGINE=python.

Testing

  • Python unit tests include go-path stubs and multi-file cases.
  • Engine parity integration test runs if helper is available (found on PATH or at ./tfworker-hcl2json).
  • Go unit tests validate simple and real Terraform fixtures.

ephur added 3 commits October 10, 2025 21:50
…grate build/tests/docs

- Add tfworker/util/hcl_parser.py with auto engine selection (Go if helper on PATH; fallback to python-hcl2),
forced override via TFWORKER_HCL_ENGINE/TFWORKER_HCL_BIN, and new parse_files().
- Add tools/hcl2json Go helper (go.mod, main.go) with --stdin and --multi (batch) support; unit tests including
repo fixture.
- Refactor tfworker/commands/config.py and tfworker/util/terraform_helpers.py to use hcl_parser; batch-parse .tf
files for required_providers with graceful per-file fallback and consistent logging.
- Add tests: tests/util/test_hcl_parser.py, tests/util/test_hcl_engine_parity.py (skip if helper missing), and
strengthened Go tests.
- Update Makefile: add go-build/go-test, build from subdir, integrate into test/ci-test, emit guidance if tools/
hcl2json/go.sum missing, clean helper binary.
- Update README: engine auto-detect and forced modes, batch parsing note, Go development guidelines (commit
go.mod/go.sum, tidy instructions).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants