Investigate VM-based testing tooling for dotfiles validation

[kergoth]

Context

Several testing gaps require VM-based solutions where containers aren't sufficient. This issue tracks research into VM tooling that can serve multiple testing needs:

• Non-Linux platform testing (Investigate non-Linux platform testing (macOS, Windows, FreeBSD) #57) — macOS, Windows, FreeBSD require VMs
• OS installation script testing (Test os-install scripts (Arch, Chimera) via VM-based automation #59) — script/{distro}/os-install runs from live ISOs, requiring bootable VM environments

Tooling to Evaluate

Local VM Management

• QEMU — Cross-platform, scriptable, supports many architectures and OS types
• UTM — macOS-native frontend for QEMU with Apple Virtualization framework support
• Lima — Linux VMs on macOS, lightweight, good CLI; defaults to QEMU but supports vmType: vz for near-native performance via Apple Virtualization.framework
• Vagrant — VM orchestration layer over VirtualBox/Hyper-V/libvirt, declarative config
• VirtualBox — Cross-platform, free, well-known but heavier
• Tart — macOS-native, designed for CI, Apple Virtualization framework

CI/CD Options

• GitHub Actions — macOS and Windows runners available; no FreeBSD or ISO boot
• Cirrus CI — FreeBSD support, broader OS matrix
• Self-hosted runners — Full control but maintenance burden

Questions to Answer

• Which tool best supports scriptable, headless VM creation and teardown?
• Can we boot ISOs and run os-install scripts non-interactively?
• What's the minimum viable automation: full CI pipeline vs local-only scripts?
• Cost/complexity trade-offs for each approach
• Can a single tool cover both use cases (platform testing + ISO install testing)?

Related Issues

• Investigate non-Linux platform testing (macOS, Windows, FreeBSD) #57 — Non-Linux platform testing (macOS, Windows, FreeBSD)
• Test os-install scripts (Arch, Chimera) via VM-based automation #59 — OS installation script testing (Arch, Chimera)

[kergoth]

Issue #58: VM-Based Testing Tooling Evaluation (by Claude)

Context

This evaluation supports three related dotfiles testing issues:

• Investigate non-Linux platform testing (macOS, Windows, FreeBSD) #57 — Non-Linux platform testing (macOS, Windows, FreeBSD)
• Investigate VM-based testing tooling for dotfiles validation #58 — Investigate VM-based testing tooling (this issue)
• Test os-install scripts (Arch, Chimera) via VM-based automation #59 — OS installation script testing (Arch, Chimera live ISO → installed rootfs)

The two use cases have distinct requirements: #57 needs to run dotfiles setup scripts on non-Linux OSes, while #59 needs to boot live ISOs, partition virtual disks, and verify that an installed system is bootable. A single tool or combination that covers both would be ideal.

---

Use Case Requirements Matrix

Requirement	#57 (Platform Testing)	#59 (OS Install Scripts)
macOS guest	✅ Required	—
Windows guest	✅ Required	—
FreeBSD guest	✅ Required	—
Linux guest	Nice-to-have	✅ Required
Boot from ISO	—	✅ Required
Virtual disk partitioning	—	✅ Required
Non-interactive execution	✅ Required	✅ Required
Snapshot/restore	Nice-to-have	✅ Required
CI integration	✅ Required	✅ Required
Runs on Apple Silicon host	✅ Required	✅ Required

---

Local VM Tools Evaluated

QEMU / Quickemu

What it is: QEMU is the industry-standard open-source machine emulator and virtualizer. Quickemu is a shell-based wrapper that automates QEMU configuration for ~1000 OS editions.

Strengths:

• Supports virtually every guest OS (Linux, macOS, Windows, FreeBSD, and many others)
• Full ISO booting and disk management — the gold standard for Test os-install scripts (Arch, Chimera) via VM-based automation #59's requirements
• Completely scriptable via CLI; no GUI needed
• Quickemu adds automatic ISO download, checksum verification, and optimized VM configs
• Quicktest (companion project) provides automated testing of OS installs inside Quickemu VMs
• Snapshot support via QEMU snapshots and qcow2 backing files
• Available on Linux and macOS (via Homebrew)
• Free and open source

Weaknesses:

• On Apple Silicon, QEMU runs x86 guests via emulation (slow) or arm64 guests via HVF (fast). Cross-architecture testing is possible but performance-limited.
• Quickemu is Linux-focused; macOS host support is secondary
• Configuration complexity when used raw (mitigated by Quickemu)
• No native macOS guest support on non-Apple hardware (licensing)

Verdict: Best single tool for #59 (ISO boot testing). Strong for #57's Linux/FreeBSD testing. macOS/Windows guest testing on Apple Silicon requires arm64 images or accepts emulation overhead.

---

Tart

What it is: A virtualization toolset by Cirrus Labs for macOS and Linux VMs on Apple Silicon, using Apple's native Virtualization.framework. 30,000+ installations.

Strengths:

• Near-native performance on Apple Silicon (Virtualization.framework)
• Excellent macOS guest support — directly addresses Investigate non-Linux platform testing (macOS, Windows, FreeBSD) #57's primary gap
• OCI registry integration (push/pull VM images like containers)
• CLI-first design (tart create, tart run, tart clone)
• Powers Cirrus Runners (drop-in GitHub Actions replacement, M4 Pro chips)
• Linux arm64 guests also supported
• Actively maintained, strong CI/CD pedigree

Weaknesses:

• Apple Silicon only (no Intel, no Linux hosts)
• No Windows guest support
• No FreeBSD guest support
• No ISO boot capability — starts from IPSW (macOS) or disk images (Linux)
• Cannot address Test os-install scripts (Arch, Chimera) via VM-based automation #59 at all (no ISO booting, no arbitrary disk setup)

Verdict: Best-in-class for macOS guest testing on Apple Silicon. Addresses the most difficult part of #57 (macOS). Not applicable to #59.

---

Lima

What it is: A CNCF Incubating project that launches Linux VMs on macOS with automatic file sharing and port forwarding. Now at v2.0.

Strengths:

• Excellent macOS host experience (similar to WSL2's UX)
• Supports both QEMU and Virtualization.framework backends
• v2.0 adds plugin infrastructure for custom VM drivers and URL schemes
• Automatic file sharing between host and guest
• Pre-built templates for common distros
• Can run on Linux hosts too

Weaknesses:

• Linux guests only — no macOS, Windows, or FreeBSD guests
• Not designed for ISO booting or bare-metal install testing
• Primarily a "run Linux on Mac" tool, not a general VM testing platform

Verdict: Good for running Linux-based dotfiles tests on a macOS host, but doesn't address the unique requirements of either #57 (non-Linux platforms) or #59 (ISO boot testing). Overlaps with existing container-based testing.

---

UTM

What it is: A full-featured virtualization app for macOS, built on QEMU (for emulation) and Apple Virtualization.framework (for native arm64).

Strengths:

• Supports both emulation (any architecture) and native virtualization
• AppleScript + utmctl CLI for automation
• UTM Kit provides full automation suite for VM creation/management
• ISO-to-template automation now possible (as of 2025)
• Supports macOS, Windows, Linux, and FreeBSD guests

Weaknesses:

• macOS-only host
• CLI/scripting is secondary to the GUI — automation was retrofitted
• AppleScript-based VM creation is less portable than shell scripts
• No CI integration story (designed for desktop use)
• Community-driven automation tooling (UTM Kit) is less mature

Verdict: Viable for local multi-OS testing on macOS, but the AppleScript-based automation is awkward for CI pipelines. Better as a manual testing supplement than a CI backbone.

---

Vagrant

What it is: HashiCorp's tool for building and managing portable development environments. Uses VirtualBox, VMware, Hyper-V, or libvirt as backends.

Strengths:

• Mature, well-documented, large box ecosystem
• Vagrantfile is declarative and version-controllable
• Multi-OS support: Linux, FreeBSD, Windows, (limited) macOS
• Proven for dotfiles testing (ULHPC/dotfiles uses this exact pattern)
• Provisioning via shell scripts, Ansible, Chef, Puppet
• Snapshot and multi-machine support

Weaknesses:

• VirtualBox (primary backend) does not support Apple Silicon natively
• VMware Fusion backend requires commercial license
• libvirt backend is Linux-only
• Slow startup compared to containers
• HashiCorp's licensing changes (BSL) have created uncertainty
• No ISO boot support — starts from pre-built box images
• Cannot address Test os-install scripts (Arch, Chimera) via VM-based automation #59 (no raw ISO booting)

Verdict: Would have been the natural choice 5 years ago, but Apple Silicon support is a blocker. The VirtualBox backend is not viable on M-series Macs. Could still work for x86 CI runners.

---

VirtualBox

What it is: Oracle's free, cross-platform hypervisor.

Strengths:

• Free, mature, supports many guest OSes
• Snapshot support, CLI (VBoxManage) is fully scriptable
• ISO boot support

Weaknesses:

• No Apple Silicon support — this is a hard blocker for local development
• Performance is mediocre compared to native hypervisors
• Oracle's stewardship concerns

Verdict: Not viable for Apple Silicon workflows. Only relevant if running on x86 Linux CI runners.

---

CI/CD Options Evaluated

GitHub Actions (Standard Runners)

Strengths:

• Free for public repos (generous limits)
• macOS 15 and Windows 2025 images now GA
• Direct dotfiles testing possible (proven pattern: checkout + run setup.sh)
• Windows on Arm runners available for public repos
• Zero infrastructure to manage

Weaknesses:

• No FreeBSD runners
• Cannot nest VMs inside runners (no KVM on standard runners)
• macOS runners are expensive (10x Linux minute cost for private repos)
• Limited customization of runner images
• Cannot test ISO boot scenarios (Test os-install scripts (Arch, Chimera) via VM-based automation #59)

Best for: Direct dotfiles testing on macOS and Windows (#57). Not suitable for #59.

---

Cirrus CI

Strengths:

• Native FreeBSD VM support — used by FreeBSD project itself and Neovim
• macOS VMs on Apple Silicon (M4 Pro as of 2025)
• Linux containers and VMs
• Good GitHub integration
• Free tier for open source
• Can run custom VM images

Weaknesses:

• Smaller ecosystem than GitHub Actions
• Another CI system to manage alongside GitHub Actions
• ISO boot testing would still require custom images

Best for: FreeBSD testing (#57) and as a macOS alternative to GitHub Actions. Partially helps #59 if custom images can be prepared.

---

Cirrus Runners (GitHub Actions Compatible)

Strengths:

• Drop-in replacement for GitHub-hosted runners
• 2-3x better macOS performance at lower cost
• Powered by Tart (Apple Silicon native)
• Uses standard GitHub Actions workflow syntax

Weaknesses:

• Commercial service (cost)
• macOS and Linux only (no FreeBSD, no Windows)

Best for: Accelerating macOS CI if GitHub Actions macOS runners are too slow/expensive.

---

Self-Hosted Runners

Strengths:

• Full control over environment (can run QEMU, boot ISOs, etc.)
• Can address Test os-install scripts (Arch, Chimera) via VM-based automation #59 completely
• Any OS, any architecture

Weaknesses:

• Infrastructure burden (hardware, maintenance, security)
• Network/availability concerns
• Not practical for an individual dotfiles project unless hardware is already available

Best for: #59 if no other CI solution works. Only justified if you already have suitable hardware.

---

Recommendations

For #57 (Non-Linux Platform Testing)

Recommended approach — layered strategy:

1. GitHub Actions for macOS and Windows: Use standard macos-15 and windows-2025 runners to run dotfiles setup scripts directly. This is the lowest-effort, highest-value option. A proven pattern exists (checkout → run setup → verify).

2. Cirrus CI for FreeBSD: Native FreeBSD VM support fills the gap that GitHub Actions cannot. Used by major projects (FreeBSD project, Neovim). Add a .cirrus.yml alongside existing GitHub Actions workflows.

3. Tart for local macOS VM testing: When you need to iterate on macOS-specific setup locally before pushing to CI. Near-native performance, OCI image management.

For #59 (OS Installation Script Testing)

Recommended approach:

1. QEMU + Quickemu/Quicktest for local development and testing: Boot Arch/Chimera live ISOs, attach virtual disks, run install scripts non-interactively, verify bootability. Quicktest already provides a framework for this exact workflow.

2. Self-hosted runner or Cirrus CI with custom images for CI: If CI automation of Test os-install scripts (Arch, Chimera) via VM-based automation #59 is desired, a self-hosted runner with QEMU/KVM is the most flexible option. Alternatively, prepare pre-built VM images and test them in Cirrus CI.

Does One Tool Cover Both?

No single tool covers both use cases well. The requirements are fundamentally different: #57 needs pre-built OS environments to run scripts in, while #59 needs raw ISO booting and disk management. The recommended combination is:

Use Case	Local	CI
macOS dotfiles (#57)	Tart	GitHub Actions
Windows dotfiles (#57)	— (CI only)	GitHub Actions
FreeBSD dotfiles (#57)	QEMU	Cirrus CI
OS install scripts (#59)	QEMU/Quickemu	Self-hosted runner

Suggested Next Steps

1. Quick win: Add GitHub Actions workflows for macOS and Windows dotfiles testing. This requires no new tooling and covers the highest-value platforms.
2. Add Cirrus CI for FreeBSD testing with a .cirrus.yml configuration.
3. Prototype QEMU/Quickemu locally for Test os-install scripts (Arch, Chimera) via VM-based automation #59's ISO boot testing. Start with Arch Linux as it likely has better automation tooling (archinstall).
4. Evaluate Tart for local macOS iteration if the GitHub Actions feedback loop is too slow.
5. Defer self-hosted runners unless Test os-install scripts (Arch, Chimera) via VM-based automation #59's CI automation becomes a priority.