Skip to main content

Reset DevKits for Reuse

Kerrigan can clean one DevKit or every current member of a pool before the hardware is assigned to another user. Start cleanup from the device or pool settings in the local dashboard, the REST API, or the devices.cleanup MQTT operation.

Cleanup permanently removes unprotected top-level content from /data and, when it is a mounted filesystem, /media/nvme. It preserves Kerrigan-managed device bundles and agent files, sima-cli, pyneat, device identity and boot configuration, and GitHub Actions runner installation and configuration. The runner work directory configured in its .runner metadata is cleared, including the managed runner’s _work directory. Unsafe workspace paths or nested mounts fail cleanup before workload deletion. Runners using alternate migrated registration metadata are currently rejected before cleanup begins.

Admission and lifecycle​

The device must be online, healthy, and idle, with no active SDK/session lease, P2P lease, GitHub Actions job, firmware operation, or other agent-maintenance reservation. An active maintenance window is allowed. For a pool request, Kerrigan snapshots the current members and reserves all of them before changing any device; if one member cannot be admitted, it releases every reservation and does not run cleanup.

Each request creates a durable cleanup job with one result per target. Kerrigan resolves the requested target ID to its single normalized hardware MAC and uses that immutable MAC for every later phase, retry, and restart recovery. Target IDs may be renamed without retargeting an existing cleanup job. Immediately before any destructive command, Kerrigan also verifies that the SSH endpoint reports the expected MAC and fails closed on a mismatch.

Kerrigan records the original runner state, disables and stops runner services, and temporarily moves their registration metadata aside so they cannot accept work even if an overlay reset restores enabled units. It validates protected paths and filesystem boundaries before deleting workload data. After a fresh healthy device-agent heartbeat and successful tool verification, it restores runner registration and the original enabled and active states. An enabled but previously inactive runner remains inactive.

A failed read-only preflight releases its reservation without changing the device. The job remains available with its error and preflight_failed phase. Kerrigan persists mutation_started before sending the first command that can change the device. After that boundary, only verified completion releases the reservation. A timeout, validation error, lost SSH connection, or Fleet Manager restart leaves affected devices in recovery_required; they remain unavailable until an administrator inspects and retries the same cleanup job. The dashboard displays “Cleanup needs recovery” and shows the error and a retry action in the device settings. Older jobs without a mutation marker retain their reservations conservatively. A retry of a legacy missing-command failure now uses the built-in 3.0 reset. A retry is idempotent and revalidates the result before releasing the device. Device-side journals preserve original runner state through interrupted attempts. A completed deletion receipt is reused on retry, avoiding repeated deletion or overlay reset. The journal and runner registration backups must remain intact until recovery completes.

Platform 3.0 overlay reset​

Platform 3.0.x only includes a built-in overlay reset. Earlier platforms keep workload cleanup without clearing the root filesystem or rebooting for this step. Kerrigan checks the reported version against /etc/buildinfo, the active overlay mount options, and the expected /data/overlay/upper and /data/overlay/work directories. Version/filesystem mismatches, symlinks, and nested mounts fail closed. No administrator-supplied reset command is needed.

Before deleting data, Kerrigan saves a durable management backup outside the overlay. It preserves the active managed bundle, agent and Sentinel configuration, management identity and network/SSH configuration, installed package files and package metadata, and the installed pyneat environment. Package-owned directories are not copied recursively, so unrelated workload files beside packaged files are removed. Unknown bundle installations or missing preservation metadata stop cleanup before deletion.

The reset removes both overlay directories, recreates them, and restores only protected management/package state into the new upper directory. The running cleanup process schedules reboot itself because a new SSH session may no longer have access to management executables after the active upper is removed. The persistent deletion receipt and reset journal distinguish an interrupted reset from an already cleared overlay. Retrying a completed reset does not erase the newly booted filesystem again.

Success requires a changed kernel boot ID, a fresh healthy device-agent heartbeat, and successful tool and runner verification. The backup and journal remain on persistent storage as recovery evidence. Clearing the overlay removes workload changes; it does not roll back installed packages or erase protected management credentials. Firmware replacement is a separate operation.

REST API​

The start and retry operations require a system administrator:

POST /v1/device-cleanups/targets/{target_id}
POST /v1/device-cleanups/pools/{pool_id}
POST /v1/device-cleanups/{job_id}/retry
GET  /v1/device-cleanups/{job_id}
GET  /v1/device-cleanups

Start bodies accept an optional reason:

{"reason":"Prepare for the next user"}

The returned job state is pending, running, completed, failed, or recovery_required. Inspect each item for its phase, error, deleted paths, and protected paths. The result includes deleted_count; deleted lists at most the first 256 top-level paths and sets deleted_truncated when more were removed.