Firmware Update State Machine

This diagram mirrors the firmware update sequence in src/kerrigan/services/fleet_manager/firmware_update.py. The colored groups show where Kerrigan can restart a failed or interrupted update without replaying every previous state.

Recovery Groups

Group	States	Retry entry point	Why this is safe
Prepare netboot	pending, preparing_uboot, rebooting_to_netboot	preparing_uboot	Retry first checks /proc/cmdline. A running RAM netboot advances to verifying_netboot; otherwise U-Boot preparation is repeated. Late TFTP evidence moves a paused preparation checkpoint to waiting_for_netboot_ssh, followed by netboot verification.
Await netboot	netboot_started, waiting_for_netboot_ssh	waiting_for_netboot_ssh	Netboot has already been requested or observed, so retry waits for the netboot OS to become reachable.
Restage and flash image	verifying_netboot, netboot_completed, transferring_troot, transferring_emmc, image_ready, updating_troot, flashing_emmc	verifying_netboot	Kerrigan revalidates that the board is netbooted, then stages images again before flashing. If the board came back from eMMC during retry, recovery redoes U-Boot setup and netboots before transfer.
Await updated OS	updated_rebooting, waiting_for_updated_emmc_ssh	waiting_for_updated_emmc_ssh	Flashing has already completed or reboot has begun, so retry waits for the updated eMMC OS to reconnect.
Post-flash configuration	configuring_passwordless_sudo, configuring_dns, configuring_time, installing_sima_cli, remounting_nvme, installing_device_agent	same state	These states are active update states but are not grouped into an earlier recovery checkpoint; retry resumes from the current configuration step.

Terminal states are completed, failed, and cancelled. In the current manager, ordinary runtime exceptions pause the job for retry instead of marking it failed. A cancelled or failed job can still be retried if Kerrigan recorded a resumable last_active_state in job metadata.

U-Boot preparation on Platform 2.1 and 3.0

Recovery preserves the original /boot mount mode, temporarily remounting it writable when necessary. It saves the environment files before changes, detects the bootloader's single-file or redundant FAT environment format, and validates CRC data before updating boot settings. Legacy 2.1 images without u-boot.bin require the known redundant layout in /etc/fw_env.config.

The temporary configuration is created outside /boot. Failed preparation restores the saved environment and prevents reboot; a failed read-only remount also prevents reboot. Backups remain under /boot/kerrigan-netboot-backup.*. Kerrigan continues to use DHCP for recovery netboot.

eMMC preparation before recovery flashing

Before writing a whole-device image, Kerrigan identifies mounted filesystems by block-device identity, including eLxr 3.x LVM volumes such as /data and legacy 2.1.x partition mounts. It unmounts nested mounts first, deactivates the eMMC's LVM volumes, and verifies that no mounts or device mappings remain. Unrelated storage such as NVMe is left alone.

Recovery stops before writing if the eMMC hosts the root filesystem, has active swap, or any preparation step fails. Busy mounts must be released before retrying; Kerrigan does not force or lazily unmount them. Both dd and bmaptool use these checks. Decompression and write errors fail the job, and dd flushes writes before completion. This preparation does not apply to A/B OTA updates.