Skip to content

Add Go-backed HCL parser, auto-detect engine, batch .tf parsing #145

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ephur wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Add Go-backed HCL parser, auto-detect engine, batch .tf parsing #145

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
2cb9530
Add Go-backed HCL parser, auto-detect engine, batch .tf parsing; inte…
ephur Oct 11, 2025
1ecb89d
Improve calling the helper for multi vs single file parsing
ephur Oct 11, 2025
fbde93d
use a per definition provider cache
ephur Oct 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,3 +13,6 @@ registry.terraform.io/
coverage.xml
reports/*
plans*

# go tool related build files
tfworker-hcl2json
2 changes: 1 addition & 1 deletion .isort.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[settings]
known_third_party = atlassian,boto3,botocore,click,google,hcl2,jinja2,lark,mergedeep,moto,openai,packaging,pydantic,pydantic_core,pydantic_settings,pytest,yaml
known_third_party = atlassian,boto3,botocore,click,google,jinja2,mergedeep,moto,openai,packaging,pydantic,pydantic_core,pydantic_settings,pytest,yaml
profile = black
skip = ["*/__init__.py"]
37 changes: 35 additions & 2 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,11 @@ format: init-dev
@poetry run seed-isort-config || echo "known_third_party setting changed. Please commit pyproject.toml"
poetry run isort tfworker tests

test: init-dev
test: init-dev go-build go-test
poetry run pytest -p no:warnings --disable-socket
poetry run coverage report --fail-under=60 -m --skip-empty

ci-test: init-dev
ci-test: init-dev go-test
poetry run pytest --disable-socket --junitxml=reports/junit.xml
poetry run coverage xml -o reports/coverage.xml

Expand All @@ -37,3 +37,36 @@ clean:
@rm -rf build dist .eggs terraform_worker.egg-info
@find . -name *.pyc -exec rm {} \;
@find . -name __pycache__ -type d -exec rmdir {} \;
@rm -f tfworker-hcl2json

# --- Go integration ---

.PHONY: go-build go-test

go-build:
@if which go >/dev/null 2>&1; then \
if [ ! -f tools/hcl2json/go.sum ]; then \
echo "[go-build] Missing tools/hcl2json/go.sum"; \
echo " Run: (cd tools/hcl2json && go mod tidy) and commit go.sum"; \
echo " Skipping Go build"; \
else \
echo "Building Go HCL helper..."; \
(cd tools/hcl2json && go build -o ../../tfworker-hcl2json); \
fi; \
else \
echo "Go not installed, skipping go-build"; \
fi

go-test:
@if which go >/dev/null 2>&1; then \
if [ ! -f tools/hcl2json/go.sum ]; then \
echo "[go-test] Missing tools/hcl2json/go.sum"; \
echo " Run: (cd tools/hcl2json && go mod tidy) and commit go.sum"; \
echo " Skipping Go tests"; \
else \
echo "Running Go tests..."; \
(cd tools/hcl2json && go test ./...); \
fi; \
else \
echo "Go not installed, skipping go-test"; \
fi
31 changes: 31 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -221,4 +221,35 @@ To build HTML documentation:
% make clean && make html
```

### Optional: Native HCL (Go) Parser

For improved compatibility with HCL2/Terraform syntax, tfworker can use a small Go helper that leverages HashiCorp's native HCL parser.

- Build the helper (requires Go 1.20+):
- `go build -o tfworker-hcl2json ./tools/hcl2json`
- Put `tfworker-hcl2json` on your PATH or set `TFWORKER_HCL_BIN` to its full path.

- Engine selection:
- Default (auto): If the helper is found via `TFWORKER_HCL_BIN` or your `PATH`, tfworker uses the Go parser; otherwise it uses `python-hcl2`.
- Forced: Set `TFWORKER_HCL_ENGINE=go` or `TFWORKER_HCL_ENGINE=python` to force a specific engine. If set to `go` but the helper is missing, tfworker errors out.

- Batch parsing: tfworker batches `.tf` files when discovering required providers and sends them to the Go helper in one process (`--multi`), improving performance on large repos. Python fallback parses files individually.

### Go Development

For the small Go helper in `tools/hcl2json/`:

- Commit module files: Always commit both `go.mod` and `go.sum`.
- First-time setup: If `go.sum` is missing, run:
- `(cd tools/hcl2json && go mod tidy)`
- Commit the generated `go.sum`.
- Make targets:
- `make go-build` builds the helper into `./tfworker-hcl2json`.
- `make go-test` runs `go test` under `tools/hcl2json`.
- If Go isn’t installed or `go.sum` is missing, the Makefile prints guidance and skips Go steps.
- One system Go: Use a single up-to-date Go (1.22+) on your machine.
- Optional vendoring (air-gapped builds):
- `cd tools/hcl2json && go mod vendor`
- Build with `-mod=vendor` if needed; otherwise vendoring is not required.

The documentation can be viewed locally by open `./docs/build/index.html` in a browser.
28 changes: 18 additions & 10 deletions tests/definitions/test_definitions_prepare_extra.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,12 +137,14 @@ def test_create_local_vars(mocker, def_prepare, definition):

def test_create_worker_tf_calls_helpers(mocker, def_prepare):
m1 = mocker.patch.object(def_prepare, "_get_remotes", return_value=["r"])
m2 = mocker.patch.object(def_prepare, "_get_provider_content", return_value="PCONT")
m3 = mocker.patch.object(def_prepare, "_write_worker_tf")
m2 = mocker.patch.object(def_prepare, "_get_used_providers", return_value=["p"])
m3 = mocker.patch.object(def_prepare, "_get_provider_content", return_value="PCONT")
m4 = mocker.patch.object(def_prepare, "_write_worker_tf")
def_prepare.create_worker_tf("def1")
m1.assert_called_once_with("def1")
m2.assert_called_once_with("def1")
m3.assert_called_once_with("def1", ["r"], "PCONT")
m3.assert_called_once_with("def1", ["p"])
m4.assert_called_once_with("def1", ["r"], "PCONT", ["p"])


def test_create_terraform_vars(mocker, def_prepare, definition):
Expand Down Expand Up @@ -222,14 +224,21 @@ def test_download_modules_failure(mocker, def_prepare, definition):
def_prepare.download_modules("def1")


def test_get_provider_content(def_prepare, mocker, definition):
mocker.patch.object(Definition, "get_used_providers", return_value=None)
def test_get_provider_content(def_prepare):
def_prepare._app_state.providers.required_hcl.return_value = "REQ"
assert def_prepare._get_provider_content("def1") == "REQ"
assert def_prepare._get_provider_content("def1", None) == "REQ"
def_prepare._app_state.providers.required_hcl.assert_called_once_with(None)

mocker.patch.object(Definition, "get_used_providers", return_value=["p"])
assert def_prepare._get_provider_content("def1") == ""
assert def_prepare._get_provider_content("def1", ["p"]) == ""


def test_get_used_providers_caches(def_prepare, mocker):
mock_method = mocker.patch.object(
Definition, "get_used_providers", side_effect=[["p"], ["other"]]
)
assert def_prepare._get_used_providers("def1") == ["p"]
assert def_prepare._get_used_providers("def1") == ["p"]
assert mock_method.call_count == 1


def test_get_remotes(def_prepare, mocker, definition):
Expand Down Expand Up @@ -271,8 +280,7 @@ def test_write_worker_tf(def_prepare, definition, mocker):
def_prepare._app_state.providers.provider_hcl.return_value = "PROV"
def_prepare._app_state.backend.hcl.return_value = "BACKEND"
def_prepare._app_state.backend.data_hcl.return_value = "DATA"
mocker.patch.object(Definition, "get_used_providers", return_value=["p1"])
def_prepare._write_worker_tf("def1", ["r"], "REQ")
def_prepare._write_worker_tf("def1", ["r"], "REQ", ["p1"])
tf_path = (
Path(definition.get_target_path(def_prepare._app_state.working_dir))
/ WORKER_TF_FILENAME
Expand Down
49 changes: 49 additions & 0 deletions tests/util/test_hcl_engine_parity.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
import os
import shutil
from pathlib import Path

import pytest

from tfworker.util.terraform_helpers import _find_required_providers


def _helper_on_path() -> str | None:
# Respect explicit env if provided
explicit = os.getenv("TFWORKER_HCL_BIN")
if explicit and Path(explicit).exists():
return explicit
# Check PATH
found = shutil.which("tfworker-hcl2json")
if found:
return found
# Check repo root build location (make go-build outputs here)
repo_root = Path(__file__).parents[2]
candidate = repo_root / "tfworker-hcl2json"
if candidate.exists():
return str(candidate)
return None


@pytest.mark.integration
def test_required_providers_engine_parity(monkeypatch):
"""
Ensure python and go engines return identical provider requirements
for a known fixture directory.
"""
fixtures_dir = Path(__file__).parents[1] / "fixtures" / "definitions" / "test_a"

# Baseline using Python engine
monkeypatch.setenv("TFWORKER_HCL_ENGINE", "python")
providers_py = _find_required_providers(str(fixtures_dir))

# If the Go helper isn't available, skip parity check gracefully
helper = _helper_on_path()
if not helper:
pytest.skip("Go HCL helper not found; skipping engine parity test")

# Compare with Go engine
monkeypatch.setenv("TFWORKER_HCL_ENGINE", "go")
monkeypatch.setenv("TFWORKER_HCL_BIN", helper)
providers_go = _find_required_providers(str(fixtures_dir))

assert providers_go == providers_py
68 changes: 68 additions & 0 deletions tests/util/test_hcl_parser.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
import json
import stat
from pathlib import Path

import pytest

from tfworker.util import hcl_parser


class TestHCLParser:
def test_parse_string_python_engine(self, monkeypatch):
# Force python engine
monkeypatch.setenv("TFWORKER_HCL_ENGINE", "python")
hcl = """
terraform {
worker_options { x = 1 }
}
"""
d = hcl_parser.parse_string(hcl)
assert "terraform" in d

def test_parse_with_go_binary(self, tmp_path: Path, monkeypatch):
# Create a fake go helper that just prints a known JSON
payload = {"terraform": [{"worker_options": [{"x": 1}]}]}
fake = tmp_path / "tfworker-hcl2json"
fake.write_text("#!/bin/sh\n" + f"echo '{json.dumps(payload)}'\n")
fake.chmod(fake.stat().st_mode | stat.S_IEXEC)

monkeypatch.setenv("TFWORKER_HCL_ENGINE", "go")
monkeypatch.setenv("TFWORKER_HCL_BIN", str(fake))

d = hcl_parser.parse_string("terraform {}")
assert d == payload

def test_go_binary_error_surface(self, tmp_path: Path, monkeypatch):
# Fake helper exits non-zero
fake = tmp_path / "tfworker-hcl2json"
fake.write_text("#!/bin/sh\nexit 3\n")
fake.chmod(fake.stat().st_mode | stat.S_IEXEC)

monkeypatch.setenv("TFWORKER_HCL_ENGINE", "go")
monkeypatch.setenv("TFWORKER_HCL_BIN", str(fake))

with pytest.raises(ValueError):
hcl_parser.parse_string("terraform {}")

def test_parse_files_multi_go_and_errors(self, tmp_path: Path, monkeypatch):
# Create two files: one valid, one invalid
good = tmp_path / "good.hcl"
bad = tmp_path / "bad.hcl"
good.write_text("terraform { required_providers {} }")
bad.write_text("terraform { izgreat! }")

# Fake helper: returns ok for good and error for bad
payload = {
"ok": {str(good): {"terraform": [{"required_providers": []}]}},
"errors": {str(bad): "parse error"},
}
fake = tmp_path / "tfworker-hcl2json"
fake.write_text("#!/bin/sh\n" + f"echo '{json.dumps(payload)}'\n")
fake.chmod(fake.stat().st_mode | stat.S_IEXEC)

monkeypatch.setenv("TFWORKER_HCL_ENGINE", "go")
monkeypatch.setenv("TFWORKER_HCL_BIN", str(fake))

ok, errors = hcl_parser.parse_files([str(good), str(bad)])
assert str(good) in ok
assert str(bad) in errors
4 changes: 2 additions & 2 deletions tfworker/commands/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@
from typing import Any, Dict, List, Type, Union

import click
import hcl2
import jinja2
import yaml
from jinja2.runtime import StrictUndefined
Expand All @@ -17,6 +16,7 @@
from tfworker.app_state import AppState
from tfworker.custom_types.config_file import ConfigFile
from tfworker.util.cli import handle_config_error
from tfworker.util.hcl_parser import parse_string as parse_hcl_string

from .. import cli_options

Expand All @@ -39,7 +39,7 @@ def load_config(
log.safe_trace(f"rendered config: {rendered}")

if cf.endswith(".hcl"):
loaded: Dict[str, Any] = hcl2.loads(rendered)["terraform"]
loaded: Dict[str, Any] = parse_hcl_string(rendered)["terraform"]
else:
loaded = yaml.safe_load(rendered)["terraform"]

Expand Down
39 changes: 27 additions & 12 deletions tfworker/definitions/prepare.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import json
from os import environ
from typing import TYPE_CHECKING, Dict, Union
from typing import TYPE_CHECKING, Dict, List, Union

import jinja2

Expand Down Expand Up @@ -38,6 +38,7 @@ class DefinitionPrepare:

def __init__(self, app_state: "AppState"):
self._app_state: "AppState" = app_state
self._used_providers_cache: Dict[tuple[str, str], Union[List[str], None]] = {}

def copy_files(self, name: str) -> None:
"""
Expand Down Expand Up @@ -102,8 +103,9 @@ def create_worker_tf(self, name: str) -> None:
"""Create remote data sources, and required providers"""
log.trace(f"creating remote data sources for definition {name}")
remotes = self._get_remotes(name)
provider_content = self._get_provider_content(name)
self._write_worker_tf(name, remotes, provider_content)
provider_names = self._get_used_providers(name)
provider_content = self._get_provider_content(name, provider_names)
self._write_worker_tf(name, remotes, provider_content, provider_names)

def create_terraform_vars(self, name: str) -> None:
"""Create the variable definitions"""
Expand Down Expand Up @@ -133,9 +135,7 @@ def create_terraform_lockfile(self, name: str) -> None:
log.trace(f"creating terraform lockfile for definition {name}")
result = generate_terraform_lockfile(
providers=self._app_state.providers,
included_providers=definition.get_used_providers(
self._app_state.working_dir
),
included_providers=self._get_used_providers(name),
cache_dir=self._app_state.terraform_options.provider_cache,
)

Expand Down Expand Up @@ -167,11 +167,10 @@ def download_modules(self, name: str, stream_output: bool = True) -> None:
f"could not download modules for definition {name}: {strip_ansi(result.stderr.decode())}"
)

def _get_provider_content(self, name: str) -> str:
def _get_provider_content(
self, _name: str, provider_names: Union[List[str], None]
) -> str:
"""Get the provider content"""
definition = self._app_state.definitions[name]
provider_names = definition.get_used_providers(self._app_state.working_dir)

if provider_names is not None:
return ""
return self._app_state.providers.required_hcl(provider_names)
Expand All @@ -195,7 +194,13 @@ def _get_remotes(self, name: str) -> list:
log.trace(f"using remotes {remotes} for definition {name}")
return remotes

def _write_worker_tf(self, name: str, remotes: list, provider_content: str) -> None:
def _write_worker_tf(
self,
name: str,
remotes: list,
provider_content: str,
provider_names: Union[List[str], None],
) -> None:
"""Write the worker.tf file"""
definition = self._app_state.definitions[name]

Expand All @@ -205,7 +210,7 @@ def _write_worker_tf(self, name: str, remotes: list, provider_content: str) -> N
) as tffile:
# Write out the provider configurations for each provider
tffile.write(
f"{self._app_state.providers.provider_hcl(includes=definition.get_used_providers(self._app_state.working_dir))}\n\n"
f"{self._app_state.providers.provider_hcl(includes=provider_names)}\n\n"
)
tffile.write(
TERRAFORM_TPL.format(
Expand All @@ -217,6 +222,16 @@ def _write_worker_tf(self, name: str, remotes: list, provider_content: str) -> N
)
tffile.write(self._app_state.backend.data_hcl(remotes))

def _get_used_providers(self, name: str) -> Union[List[str], None]:
"""Return cached list of providers used by a definition"""
cache_key = (name, self._app_state.working_dir)
if cache_key not in self._used_providers_cache:
definition = self._app_state.definitions[name]
self._used_providers_cache[cache_key] = definition.get_used_providers(
self._app_state.working_dir
)
return self._used_providers_cache[cache_key]

def _get_template_vars(self, name: str) -> Dict[str, str]:
"""
Prepares the vars for rendering in a jinja template
Expand Down
Loading