Skip to content

feat: add CoreOS-based immutable image for remote test labs #213

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
aparcar wants to merge 7 commits into
base: main
Choose a base branch
from
Open

feat: add CoreOS-based immutable image for remote test labs #213

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
3d2ee6c
feat: add CoreOS-based immutable image for remote test labs
claude Jan 24, 2026
bad28c2
feat: add Raspberry Pi setup with Docker and cloud-init
claude Jan 24, 2026
60d21a9
feat: add Fedora CoreOS flash script for Raspberry Pi
claude Jan 24, 2026
f9505b3
refactor: simplify Raspberry Pi CoreOS flashing
claude Jan 24, 2026
2bf65c7
refactor: use podman container for coreos-installer
claude Jan 24, 2026
3f59117
refactor: separate image build from flashing
claude Jan 24, 2026
847f0f4
refactor: remove privileged container, user controls all sudo
claude Jan 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
82 changes: 82 additions & 0 deletions .github/workflows/containers.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
name: Build Lab Containers

on:
push:
branches:
- main
paths:
- 'coreos/containers/**'
- '.github/workflows/containers.yml'
pull_request:
paths:
- 'coreos/containers/**'
- '.github/workflows/containers.yml'
workflow_dispatch:
schedule:
# Rebuild weekly to pick up base image updates
- cron: '0 2 * * 0'

env:
REGISTRY: ghcr.io
IMAGE_PREFIX: ${{ github.repository }}

jobs:
build:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write

strategy:
fail-fast: false
matrix:
image:
- name: labgrid
file: Containerfile.labgrid
- name: pdudaemon
file: Containerfile.pdudaemon
- name: dnsmasq
file: Containerfile.dnsmasq
- name: ser2net
file: Containerfile.ser2net

steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Set up QEMU
uses: docker/setup-qemu-action@v3

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3

- name: Log in to Container Registry
if: github.event_name != 'pull_request'
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Extract metadata
id: meta
uses: docker/metadata-action@v5
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_PREFIX }}/${{ matrix.image.name }}
tags: |
type=ref,event=branch
type=ref,event=pr
type=sha,prefix=
type=raw,value=latest,enable={{is_default_branch}}

- name: Build and push
uses: docker/build-push-action@v5
with:
context: coreos/containers
file: coreos/containers/${{ matrix.image.file }}
platforms: linux/amd64,linux/arm64
push: ${{ github.event_name != 'pull_request' }}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max
262 changes: 262 additions & 0 deletions coreos/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,262 @@
# OpenWrt Test Lab - Containerized Infrastructure

Self-contained, auto-updating lab node setup for remote OpenWrt test labs with containerized services.

## Quick Start

```bash
# 1. Create your lab config
cp lab-config.yaml.example lab-config.yaml
nano lab-config.yaml # Add SSH keys, devices

# 2. Generate ignition
./scripts/build-ignition.sh lab-config.yaml -o config.ign

# 3. Download CoreOS + UEFI (no root, no containers)
./raspberry-pi/build-image.sh config.ign

# 4. Flash (inspect flash.sh first if you want)
cd coreos-rpi && sudo ./flash.sh /dev/sdX
```

No privileged containers. You control all root operations.

See [raspberry-pi/README.md](raspberry-pi/README.md) for details.

## x86 Servers

```bash
# Use coreos-installer via container
podman run --rm --privileged -v /dev:/dev -v .:/data:ro \
quay.io/coreos/coreos-installer:release \
install /dev/sdX --ignition-file /data/config.ign
```

## Architecture

```
┌─────────────────────────────────────────────────────────────┐
│ Fedora CoreOS Host │
│ │
│ ┌──────────────────┐ ┌──────────────────┐ │
│ │ labgrid-coord │ │ labgrid-exporter │ │
│ │ (container) │◄─┤ (container) │ │
│ │ :20408 │ │ host network │ │
│ └──────────────────┘ └────────┬─────────┘ │
│ │ │
│ ┌──────────────────┐ ┌───────┴──────────┐ │
│ │ pdudaemon │ │ dnsmasq │ │
│ │ (container) │ │ (container) │ │
│ │ :16421 │ │ DHCP/TFTP │ │
│ └────────┬─────────┘ └───────┬──────────┘ │
│ │ │ │
│ ┌────────┴────────────────────┴──────────┐ │
│ │ Host Network / VLANs │ │
│ └────────────────────┬───────────────────┘ │
└───────────────────────┼─────────────────────────────────────┘
│
┌───────────────┼───────────────┐
│ │ │
┌────┴────┐ ┌────┴────┐ ┌────┴────┐
│ Device 1│ │ Device 2│ │ Device N│
│ (VLAN) │ │ (VLAN) │ │ (VLAN) │
└─────────┘ └─────────┘ └─────────┘
```

## Configuration

### Lab Configuration File

Create `lab-config.yaml` from the example:

```yaml
# Lab identification
lab:
name: my-lab
hostname: labgrid-mylab

# SSH access - add your public keys
ssh_keys:
- ssh-ed25519 AAAAC3Nz... admin@example.com

# Network VLANs for device isolation
network:
interface: eth0
vlans:
- id: 101
address: 192.168.101.1/24
dhcp_start: 192.168.101.100
dhcp_end: 192.168.101.200

# Power distribution units
pdus:
- address: 192.168.128.2
driver: ubus

# Test devices
devices:
- name: openwrt-one
serial:
id_path: pci-0000:00:14.0-usb-0:1:1.0 # from: udevadm info /dev/ttyUSB0
speed: 115200
network:
vlan: 101
power:
pdu: 192.168.128.2
outlet: 1
```

### Finding Serial Port ID_PATH

```bash
# List USB serial devices
ls /dev/ttyUSB* /dev/ttyACM*

# Get ID_PATH for a device
udevadm info /dev/ttyUSB0 | grep ID_PATH
```

## Directory Structure

```
coreos/
├── lab-config.yaml.example # Template - copy and customize
├── containers/ # Container definitions
│ ├── Containerfile.labgrid
│ ├── Containerfile.pdudaemon
│ ├── Containerfile.dnsmasq
│ └── entrypoint-labgrid.sh
├── scripts/
│ ├── build-ignition.sh # Main build script
│ ├── generate-butane.py # Config → Butane converter
│ └── build-containers.sh # Container image builder
├── ignition/ # Advanced: raw Butane configs
└── quadlet/ # Advanced: container unit files
```

## Container Images

Pre-built images from GitHub Container Registry:

| Image | Description |
|-------|-------------|
| `ghcr.io/openwrt/openwrt-tests/labgrid` | Coordinator and exporter |
| `ghcr.io/openwrt/openwrt-tests/pdudaemon` | Power control daemon |
| `ghcr.io/openwrt/openwrt-tests/dnsmasq` | DHCP/TFTP server |

Images are automatically rebuilt weekly and on changes.

### Building Locally

```bash
./scripts/build-containers.sh # Build all
./scripts/build-containers.sh --push # Build and push
```

## Auto-Updates

### OS Updates

Fedora CoreOS updates automatically via Zincati:
- **Default schedule**: Sundays 03:00 UTC
- **Automatic rollback** on boot failure

```bash
rpm-ostree status # Check current/pending updates
systemctl status zincati # Update service status
```

### Container Updates

Containers update daily via Podman auto-update:
- **Default schedule**: Daily 04:00 UTC
- Pulls new `:latest` images automatically

```bash
sudo podman auto-update --dry-run # Check for updates
sudo podman auto-update # Update now
```

## Post-Installation

### Verify Services

```bash
# Check containers
sudo podman ps

# Check services
systemctl status labgrid-coordinator labgrid-exporter pdudaemon dnsmasq

# View logs
journalctl -u labgrid-exporter -f
```

### Configure VLANs (if not using ignition network config)

```bash
# Create VLAN interface
sudo nmcli con add type vlan con-name vlan101 dev eth0 id 101 \
ipv4.addresses 192.168.101.1/24 ipv4.method manual
sudo nmcli con up vlan101
```

## Troubleshooting

### Container Logs

```bash
sudo podman logs labgrid-coordinator
sudo podman logs labgrid-exporter
sudo podman logs pdudaemon
```

### Restart Services

```bash
sudo systemctl restart labgrid-exporter
sudo systemctl daemon-reload # After config changes
```

### Serial Devices Not Found

```bash
# Check devices exist
ls -la /dev/ttyUSB* /dev/ttyACM*

# Check container has access
sudo podman exec labgrid-exporter ls /dev/ttyUSB0
```

## Hardware Requirements

- **CPU**: x86_64 or aarch64
- **RAM**: 2GB minimum
- **Storage**: 16GB minimum
- **Network**: Gigabit Ethernet
- **USB**: Ports for serial consoles

### Tested Platforms

- Raspberry Pi 5
- Intel NUC
- Generic x86_64 servers

## Advanced: Manual Butane Configuration

For complex setups, you can edit the Butane files directly:

```bash
# Edit the standalone Butane file
vim ignition/labnode-standalone.bu

# Generate Ignition
./scripts/generate-ignition.sh ignition/labnode-standalone.bu
```

## Contributing

1. Edit config template: `lab-config.yaml.example`
2. Edit containers: `containers/Containerfile.*`
3. Edit generator: `scripts/generate-butane.py`
4. Container images auto-build on push to main
15 changes: 15 additions & 0 deletions coreos/containers/Containerfile.dnsmasq
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# dnsmasq container for DHCP and TFTP services
FROM alpine:3.19

LABEL org.opencontainers.image.source="https://github.com/openwrt/openwrt-tests"
LABEL org.opencontainers.image.description="dnsmasq DHCP/TFTP server for OpenWrt testing"
LABEL org.opencontainers.image.licenses="MIT"

RUN apk add --no-cache dnsmasq

# Create tftp directory
RUN mkdir -p /srv/tftp

# DNS disabled by default (port=0), only DHCP and TFTP
ENTRYPOINT ["dnsmasq", "-k", "--log-facility=-"]
CMD ["-C", "/etc/dnsmasq.conf"]
Loading
Loading