An end-to-end install and first boot, not a smoke test.
The orchestrator downloads and signature-verifies the release ISO, then drives the stock Calamares installer through an "erase disk" installation with LUKS full disk encryption, exactly as a user would. The goal state is R3: the install completes and the LUKS volume unlocks at the next boot.
After the install the ISO is detached and the installed disk is booted on its own. The suite confirms the installed system reaches its first-boot desktop (R4), then can run the release-critical checks inside it.
The live installer has no automation agent for its GUI, so the suite drives it from the host the way a person would: it reads the screen and moves the mouse and keyboard.
Each installer page is found by optical character recognition (OCR) of a screenshot, then confirmed by its own content text before the suite acts. Because targets are located by text rather than fixed pixels, the same drive works across display resolutions and firmware modes.
The suite waits for the GRUB menu, selects the unrestricted install session (the default session refuses to install), and launches the installer through VirtualBox guestcontrol rather than a blind-typed terminal line, so the launch has a real success signal.
Welcome, Location, Keyboard, then Partitions, where the suite selects erase disk, which turns on system encryption. It enters the LUKS passphrase twice, never placing it on a command line. On the Summary page it triggers Install by the keyboard accelerator, a deterministic action rather than a click on a button caption that reads unreliably, and waits for completion.
On reboot the suite enters the passphrase at the LUKS prompt and waits for the installed system to come up, closing the R3 and R4 goals.
One shared table of numbered, release-critical checks, run inside the installed system. The same table drives the upgrade-regression gate, so the two can never disagree about what a check is. Every check is cross-checked visually before its functional probe counts, so a green result needs both to agree.
The install is exercised under legacy BIOS, UEFI, and UEFI with Secure Boot. A reinstall-over-encryption chain covers installing onto a disk that already holds an encrypted system.
An opt-in path runs the installer's Secure Boot key enrollment, reads the per-machine certificate off the encrypted disk read-only from the host, enrolls it, and reboots so a signed out-of-tree kernel module loads under Secure Boot, making that check green for the right reason rather than merely tolerated.
VirtualBox state lives per home directory, so the Linux user account is the isolation boundary. A candidate that is not yet the blessed stable version runs in a disposable, version-named account that is removed afterwards; the blessed stable version runs in a long-lived golden account for longitudinal comparison. Test accounts are forced unprivileged, with no passwordless administration, checked and failing closed.
A run reports pass, fail, or inconclusive. Inconclusive marks a provisioning or infrastructure gap rather than a product defect, so a setup problem is never recorded as a passing or failing release.
Each run captures a screenshot and a machine-readable result and publishes them to a read-only results area. Screenshots make the automated verdict checkable by a person, which is the point: the machine runs the test, a human still decides what to trust.
This suite is tooling in
dist-ai and
follows the boundary described on the
org-ai-assisted page: it runs on disposable systems, never ships, and
is not a substitute for human review. See also the
Whonix test suite.