SwarmCracker — Unit Test Plan

Comprehensive testing strategy for all packages, prioritized by risk and coverage gaps.

---

Current State

Coverage Overview

Package	Coverage	Status
apiversion	100.0%	✅ Excellent
types	100.0%	✅ Excellent
config	97.6%	✅ Excellent
logging	94.7%	✅ Excellent
translator	94.9%	✅ Excellent
jailer	92.6%	✅ Excellent
executor	90.7%	✅ Excellent
cni	90.3%	✅ Excellent
health	89.5%	✅ Good
snapshot	87.9%	✅ Good
storage	87.3%	✅ Good
swarmkit	87.0%	✅ Good
network	86.7%	✅ Good
runtime	86.0%	✅ Good
image	85.8%	✅ Good
console	85.0%	✅ Good
metrics	84.4%	🟡 Fair
lifecycle	82.3%	🟡 Fair
discovery	80.6%	🟡 Fair

Overall target: 85% — measured 87.6% on a development host with Firecracker/jailer installed. On the CI runner (no Firecracker/jailer, Go 1.26) the same tree measures lower, because tests that require those binaries are skipped.

CI enforces a no-regression gate (.github/workflows/ci.yml): a pull request must not lower ./pkg/... coverage relative to its base branch, measured on the same runner. An absolute threshold is not stable — the runner lacks Firecracker/jailer and main gains code between branches — so the gate protects the invariant that actually matters. The 85% figure remains the target for the full suite.

Measured on 2026-09-29 with go test -short -race -coverprofile=coverage.out -covermode=atomic ./pkg/.... Caveat: on a non-root host the pkg/swarmkit tests TestVMMManagerConfigDefaults and TestVMMManagerConfigDefaultsUnit/default_jailer_UID/GID fail with mkdir /var/lib/swarmcracker: permission denied. This is a pre-existing environment failure; the package’s other tests (including all new ones) still run and its executed-code coverage is 87.0%.

Priority Classification

Priority	Packages	Reason
P0 — Critical	network/vxlan, swarmkit/vmm	Cross-node networking, core orchestration — lowest coverage, highest impact
P1 — High	network, snapshot, storage, image	Infrastructure — moderate coverage, needs improvement
P2 — Medium	storage/driver, storage/volume_meta, storage/volume_quota	Storage subsystem — partial coverage
P3 — Low	translator, executor, lifecycle, config, discovery, jailer, metrics, runtime	Already well-tested — add edge cases, error paths, fuzz targets

---

Phase 1: Critical (P0) — Core Orchestration

1.1 pkg/swarmkit/vmm — VMM Manager (831 LOC, 2 exported funcs, ~68% coverage)

Why critical: Manages Firecracker VM processes. Start/stop/configure are the most impactful operations. If VMMManager fails, nothing runs.

Test file: pkg/swarmkit/vmm_test.go — needs expanded coverage

Test Cases:
├── NewVMMManager
│   ├── Default config — valid paths → no error
│   ├── Invalid firecracker path → error
│   └── Empty socket dir → creates it
│
├── NewVMMManagerWithConfig
│   ├── Full config → fields populated correctly
│   ├── Jailer enabled → UseJailer = true
│   └── CgroupVersion "v2" → accepted
│
├── StartVM
│   ├── Valid config → VM starts, process tracked
│   ├── Invalid kernel path → error, no process created
│   ├── Socket already in use → error
│   ├── Jailer enabled → wraps with jailer
│   └── Context cancelled → VM not started
│
├── StopVM
│   ├── Running VM → stops cleanly, process removed
│   ├── Non-existent VM → error
│   ├── Already stopped VM → idempotent
│   └── Force stop → SIGKILL after timeout
│
├── GetVMStatus
│   ├── Running VM → returns RUNNING
│   ├── Stopped VM → returns STOPPED
│   └── Unknown VM → returns NOT_FOUND
│
├── ListVMs
│   ├── Empty → empty list
│   ├── 3 running VMs → all listed
│   └── Mixed running/stopped → only running
│
├── ConfigureVM (Firecracker PUT API)
│   ├── Valid boot-source config → 200 OK
│   ├── Invalid JSON → error
│   └── Socket not responding → connection refused error
│
├── Cleanup (deferred teardown)
│   ├── Remove socket files
│   ├── Remove jailer chroot dirs
│   └── Clean cgroup entries
│
└── Concurrent Operations
    ├── Parallel StartVM → no race conditions
    ├── Parallel StopVM → no race conditions
    └── Start + Stop same VM → deterministic result

Mocking strategy:

• Mock os/exec.Command via CommandExecutor interface (already exists in pkg/network/mocks.go)
• Mock HTTP calls to Firecracker API socket
• Use testing/fstest for filesystem operations
• Use temp directories for socket paths

---

1.2 pkg/swarmkit/translator — Task Translator (163 LOC, 1 exported func, ~97% coverage)

Why critical: Converts SwarmKit tasks to Firecracker VM configs. Wrong translation = broken VMs.

Test file: pkg/swarmkit/translator_test.go — maintain excellent coverage

Test Cases:
├── NewTaskTranslator
│   ├── Valid kernel + bridge IP → no error
│   ├── Empty kernel path → error
│   └── Invalid bridge IP → error
│
├── Translate (task → VM config)
│   ├── Basic task → valid VM config with defaults
│   ├── Task with env vars → env passed to rootfs
│   ├── Task with port bindings → network config correct
│   ├── Task with memory limit → machine config MemSizeMB set
│   ├── Task with CPU limit → machine config VcpuCount set
│   ├── Task with volume mounts → drive config includes mount
│   ├── Task with labels → propagated to VM metadata
│   └── Task with restart policy → reflected in config
│
├── Edge Cases
│   ├── Task with 0 replicas → empty config list
│   ├── Task with very large memory → capped at host limit
│   ├── Task with invalid image → error returned
│   └── Task with special chars in name → sanitized
│
└── Error Paths
    ├── Nil task → panic or error
    └── Missing required fields → descriptive error

---

1.3 pkg/runtime/state — State Manager (273 LOC, 1 exported func, ~89% coverage)

Why critical: Tracks VM lifecycle state across the system. State bugs cause ghost VMs or lost tasks.

Test file: pkg/runtime/state_test.go — maintain good coverage

Test Cases:
├── NewStateManager
│   ├── Valid data dir → created, no error
│   ├── Invalid permissions → error
│   └── Already exists → no error (idempotent)
│
├── VM State Transitions
│   ├── Created → Running → OK
│   ├── Running → Stopped → OK
│   ├── Stopped → Running → OK (restart)
│   ├── Created → Stopped → OK (pre-start cancel)
│   ├── Running → Failed → OK
│   └── Failed → Running → OK (retry)
│
├── Invalid Transitions
│   ├── Stopped → Created → error (no going back)
│   └── Same state → idempotent, no error
│
├── Persistence
│   ├── Save state → file written to disk
│   ├── Load state after restart → correct state restored
│   └── Corrupt state file → recovers gracefully
│
├── Concurrency
│   ├── Parallel state updates → no data race
│   └── Read during write → consistent snapshot
│
└── Cleanup
    ├── Remove dead entries (GC)
    └── Remove entries older than TTL

---

Phase 2: High Priority (P1) — Networking & Storage

2.1 pkg/network/vxlan — VXLAN Overlay (522 LOC, 2 exported funcs, 0 tests)

Why critical: Cross-node VM communication depends on VXLAN. Broken overlay = isolated VMs.

Test file: pkg/network/vxlan_test.go (needs creation)

Test Cases:
├── StaticPeerStore
│   ├── New → empty peers
│   ├── AddPeer → appears in GetPeers
│   ├── AddPeer duplicate → no duplicate entries
│   ├── RemovePeer → removed from list
│   ├── RemovePeer nonexistent → no error
│   ├── GetPeers concurrent → safe
│   └── Initial peers → populated on creation
│
├── VXLANManager (mock netlink)
│   ├── Create VXLAN interface → link created
│   ├── Create with existing name → error
│   ├── Add FDB entry → peer reachable
│   ├── Remove FDB entry → peer removed
│   └── List peers → correct set
│
├── VXLAN Configuration
│   ├── Valid VXLAN ID (1-16777215) → OK
│   ├── VXLAN ID 0 → error
│   ├── Valid overlay IP → configured
│   ├── Invalid overlay IP → error
│   └── Custom port → configured
│
├── Peer Discovery
│   ├── Add remote node → FDB entry added
│   ├── Remove remote node → FDB entry removed
│   ├── Node rejoins → entry updated
│   └── Multiple nodes → all entries present
│
└── Error Handling
    ├── netlink operation fails → wrapped error
    ├── Permission denied → clear error message
    └── Interface exists with different config → error

Mocking strategy:

• Mock netlink via interface wrapper (already has CommandExecutor)
• Or use netlink.Handle with fake netns (needs interface extraction)

---

2.2 pkg/storage/volume_block — Block Storage Driver (410 LOC, 1 exported func, 0 tests)

Test file: pkg/storage/volume_block_test.go (needs creation)

Test Cases:
├── NewBlockDriver
│   ├── Valid base dir → driver created
│   ├── Nonexistent base dir → auto-created
│   └── Read-only base dir → error
│
├── Create Volume
│   ├── Valid name + size → file created
│   ├── Size rounded to block boundary
│   ├── Duplicate name → error
│   ├── Invalid name (spaces, special chars) → error
│   └── Max size exceeded → error
│
├── Delete Volume
│   ├── Existing volume → removed
│   ├── Non-existent volume → error
│   ├── Volume in use → error
│   └── Force delete → removes even if in use
│
├── Attach Volume
│   ├── Available volume → attached to VM
│   ├── Already attached → error
│   └── VM not found → error
│
├── Detach Volume
│   ├── Attached volume → detached
│   ├── Not attached → error
│   └── VM shutdown during detach → force cleanup
│
├── List Volumes
│   ├── Empty → empty list
│   ├── Multiple volumes → all listed with status
│   └── Filter by status → correct subset
│
└── Persistence
    ├── Metadata saved after create
    ├── Metadata saved after attach/detach
    └── Corrupt metadata → recovery attempt

Mocking strategy:

• Use testing/fstest for filesystem
• Temp directories for real I/O tests

---

2.3 pkg/storage/credential_store — Secrets Manager (227 LOC, 1 exported func, 0 tests)

Test file: pkg/storage/credential_store_test.go (needs creation)

Test Cases:
├── NewSecretManager
│   ├── Valid dirs → manager created
│   ├── Nonexistent dirs → auto-created
│   └── Readonly dirs → error
│
├── Store Secret
│   ├── String secret → stored as file
│   ├── Binary secret → stored correctly
│   ├── Large secret (>1MB) → handled
│   ├── Secret with special chars → encoded
│   └── Nil value → error
│
├── Retrieve Secret
│   ├── Existing secret → returned
│   ├── Non-existent secret → error
│   └── Corrupt secret file → error with details
│
├── List Secrets
│   ├── Empty → empty list
│   ├── Multiple secrets → all listed
│   └── Filter by prefix → correct subset
│
├── Delete Secret
│   ├── Existing → removed
│   ├── Non-existent → error
│   └── Directory cleanup when empty
│
├── Config Management
│   ├── Store config file → file written
│   ├── Retrieve config → content matches
│   └── Delete config → removed
│
└── Permissions
    ├── Secret file mode → 0600
    ├── Dir mode → 0700
    └── Ownership preserved

---

Phase 3: Medium Priority (P2) — Infrastructure

3.1 pkg/storage/driver — Storage Interface (101 LOC, 0 tests)

Test Cases:
├── Interface Compliance
│   ├── BlockDriver implements StorageDriver
│   └── All methods have correct signatures
│
├── Driver Registry
│   ├── Register driver → available
│   ├── Duplicate name → error
│   └── Get driver → correct instance

3.2 pkg/storage/volume_meta — Volume Metadata (already has tests, expand)

Additional Test Cases:
├── JSON serialization roundtrip
├── Invalid JSON → error
├── Version mismatch → migration needed
├── Concurrent read/write → safe
└── Empty metadata → defaults

3.3 pkg/storage/volume_quota — Quota Management (partial coverage)

Additional Test Cases:
├── Quota update on running volume
├── Quota exceeded → write denied
├── Negative quota → error
└── Quota persistence across restart

---

Phase 4: Low Priority (P3) — Existing Packages (Edge Cases)

4.1 pkg/config — Add tests for

├── Env var overrides (SWARMCRACKER_*)
├── Config file + env var merge (env wins)
├── Invalid YAML → descriptive errors
├── Missing required fields → validation errors
└── Config migration (old → new format)

4.2 pkg/jailer — Add tests for

├── Jailer + cgroup integration
├── Resource limit edge cases (0, negative, very large)
├── Chroot escape prevention
└── Cleanup on crash (orphaned chroots)

4.3 pkg/network/manager — Add tests for

├── Bridge + VXLAN interaction
├── Concurrent VM network setup
├── Network namespace isolation
└── IPAM exhaustion handling

4.4 pkg/snapshot — Add tests for

├── Snapshot of running vs stopped VM
├── Concurrent snapshot + restore
├── Disk full during snapshot → partial cleanup
├── Snapshot metadata integrity
└── Large VM snapshot performance

4.5 Fuzz Targets (new)

Create fuzz targets in test/fuzz/:

├── config_fuzz.go        — Fuzz YAML config parsing
├── translator_fuzz.go    — Fuzz task → VM config translation
├── snapshot_meta_fuzz.go — Fuzz snapshot metadata JSON
└── vxlan_config_fuzz.go  — Fuzz VXLAN configuration

---

Implementation Order

Sprint 1 (P0 — ~3-4 days)

Day	Task	Files
1	network/vxlan_test.go — StaticPeerStore	New file
2	network/vxlan_test.go — VXLANManager (mocked)	New tests
3	swarmkit/vmm_test.go — expand coverage to 85%+	Extend tests
4	swarmkit/translator_test.go — maintain 97%+ coverage	Extend tests

Sprint 2 (P1 — ~3-4 days)

Day	Task	Files
1	network/vxlan_test.go — StaticPeerStore	New file
2	network/vxlan_test.go — VXLANManager (mocked)	New tests
3	storage/volume_block_test.go — CRUD operations	New file
3	storage/credential_store_test.go — secrets CRUD	New file
4	storage/volume_block_test.go — attach/detach/persistence	New tests

Sprint 3 (P2 + P3 — ~2-3 days)

Day	Task	Files
1	storage/driver_test.go + expand meta/quota tests	New + extend
2	Expand config, jailer, network/manager tests	Extend
3	Expand snapshot tests + create fuzz targets	Extend + new

---

Testing Conventions

File Naming

pkg/<package>/<name>_test.go           # Main test file
pkg/<package>/<name>_integration_test.go # Integration tests (build tag)
pkg/<package>/<name>_mock_test.go       # Tests using mocks

Build Tags

//go:build unit           // Fast, no external deps
//go:build integration    // May need Docker, network
//go:build e2e           // Full cluster required

Test Structure

func TestFunctionName_Scenario_ExpectedBehavior(t *testing.T) {
    // Arrange
    cfg := NewDefaultConfig()

    // Act
    result, err := DoThing(cfg)

    // Assert
    if err != nil {
        t.Fatalf("unexpected error: %v", err)
    }
    if result != expected {
        t.Errorf("got %v, want %v", result, expected)
    }
}

Table-Driven Tests

func TestParseSize(t *testing.T) {
    tests := []struct {
        name    string
        input   string
        want    int64
        wantErr bool
    }{
        {"gigabytes", "1G", 1073741824, false},
        {"megabytes", "512M", 536870912, false},
        {"invalid", "abc", 0, true},
    }
    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            got, err := ParseSize(tt.input)
            // assertions...
        })
    }
}

---

Mocking Strategy

Interfaces to Extract/Mock

Package	What to Mock	How
swarmkit/vmm	exec.Command, HTTP client	CommandExecutor interface
network/vxlan	netlink operations	NetlinkHandle interface
storage/volume_block	Filesystem I/O	testing/fstest + temp dirs
runtime/state	File persistence	testing/fstest

Mock Generation

# Using go generate
go generate ./pkg/...

# Or manually with mockgen
mockgen -source=pkg/swarmkit/vmm.go -destination=pkg/swarmkit/vmm_mock.go

Test Helpers

Create test/helpers/helpers.go:
├── NewTempDir(t)           — temp directory, cleaned up after test
├── NewTestConfig(t)        — valid config with temp paths
├── MockFirecrackerSocket(t) — fake Firecracker API socket
├── MockNetlinkHandle(t)    — fake netlink operations
└── AssertEqualJSON(t, a, b) — compare JSON structs

---

Coverage Targets

Metric	Current (dev host, 2026-09-29)	Target
Overall ./pkg/... coverage	87.6%	85%+ (project target, see gate below)
Newly covered packages (apiversion, logging, types, cni, console, health)	85.0–100%	85%+
Core orchestration (translator, executor, config, jailer)	90.7–97.6%	85%+
Infrastructure (network, image, storage, snapshot, swarmkit)	85.8–87.9%	85%+
Remaining below the line (metrics, lifecycle, discovery)	80.6–84.4%	85%+

The 85% figure is the project target. CI does not enforce an absolute number — it is unstable across runners and shifts whenever main gains code. Instead the “Coverage gate (no regression vs base)” step in .github/workflows/ci.yml fails a pull request when its ./pkg/... coverage is lower than the base branch’s, measured on the same runner. Change the target only with an explicit decision recorded here.

Measuring Coverage

# All unit tests with coverage
go test -short -coverprofile=coverage.out ./pkg/...
go tool cover -func=coverage.out | grep -v "_test.go"

# HTML report
go tool cover -html=coverage.out -o coverage.html

# Per-package coverage
go test -short -cover ./pkg/config/
go test -short -cover ./pkg/swarmkit/
go test -short -cover ./pkg/runtime/

Reproducing CI coverage locally

A developer machine reads higher coverage than the CI runner for two environmental reasons, not code differences:

1. Go toolchain. CI pins Go 1.26 (go.mod, .github/workflows/ci.yml). Measuring the same source with Go 1.27.x reports several points higher, because coverage instrumentation differs between release lines (e.g. pkg/metrics is 78.6% on 1.26 but 84.4% on 1.27).
2. Available binaries. Tests guarded by exec.LookPath("firecracker") / hasJailerBinary() — and by mkfs.ext4, debugfs, docker, podman — are skipped when those binaries are missing. CI has none of them; a host that ran setup install (or a full dev image) usually has them.

To measure what CI will measure, hide the extra binaries and use the CI toolchain:

PATH=$(echo "$PATH" | tr ':' '\n' | grep -vx /usr/local/bin | paste -sd:) \
GOTOOLCHAIN=go1.26.0 \
go test -short -race -count=1 -covermode=atomic -coverprofile=coverage.out ./pkg/... && \
go tool cover -func=coverage.out | tail -1

Reference points (2026-09-29): main measured 76.3% on CI and the test/coverage-improvement branch 82.7%, while that same branch read 87.6% locally on Go 1.27 with Firecracker/jailer installed.

---

CI Integration

GitHub Actions (update existing .github/workflows/ci.yml)

test-unit:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4
    - uses: actions/setup-go@v5
      with:
        go-version: '1.26'
    - run: make test-quick
    - run: make lint

test-unit-coverage:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4
    - uses: actions/setup-go@v5
      with:
        go-version: '1.26'
    - run: go test -short -race -coverprofile=coverage.out -covermode=atomic ./pkg/...
    # Enforce "no regression vs base" rather than an absolute threshold; see the
    # "Coverage gate (no regression vs base)" step in .github/workflows/ci.yml.

---

Summary

Priority	Packages	New Tests (est.)	LOC (est.)	Days
P0 Critical	vmm, translator, state	~45	~1800	3-4
P1 High	vxlan, volume_block, credential_store	~35	~1400	3-4
P2 Medium	driver, meta, quota	~15	~600	1-2
P3 Low	config, jailer, network, snapshot, fuzz	~20	~800	2-3
Total		~115	~4600	9-13

---

This plan is a living document. Update as tests are implemented and coverage gaps shift.