Device bundle policies and upgrades
Expand System → Kerrigan → Device bundle (collapsed by default) to select the software installed during bootstrap, post-flash setup, and device upgrades. Only system administrators can save policies and control upgrades. Firmware selection remains separate.
Components and inheritance
Device bundles always use production artifacts. The API rejects staging sources; there is no deployment-environment selector.
- Device agent: published branch or tag, optional exact build ID.
- sima-cli: an exact PyPI version, the latest PyPI release, or the latest
production artifact from a branch such as
develop. Branch selections can optionally pin an exact published build ID. - Sentinel + playbook: published branch or tag, optional exact build ID. The published Sentinel installer identifies the associated immutable playbook commit.
- P2P forwarder: published branch or tag, optional exact build ID. Kerrigan selects the verified Linux ARM64 payload for the DevKit.
- Developer Portal authorization: disabled by default, or Kerrigan-managed. The managed mode projects short-lived authorization to sima-cli without distributing Fleet Manager's refresh token.
Open a pool's pencil/edit control and its Device bundle section to override any component independently. Unchecked inheritance fixes that component's selection for the pool; inherited components continue following the system policy. Upgrade mode also inherits independently. Reset all to system removes the override.
The system policy also owns the cluster's catalog of common NFS shares. A pool inherits the system mount selection or selects its own subset; an explicit empty selection mounts none. Resolved share definitions are frozen into the bundle, so later catalog edits take effect through another rollout. Read-only is the default and must also be enforced by the NFS server.
Manual is the default. Saving a manual policy changes future bootstrap and post-flash setup without updating existing devices. Until a system or applicable pool policy is saved, legacy bootstrap/install command settings remain active.
Example system policy:
{
"device_agent": {"environment": "production", "ref": "develop", "build": null},
"sima_cli": {"version": "latest", "branch": "develop", "build": null},
"sentinel": {"environment": "production", "ref": "main", "build": null},
"p2p_forwarder": {"environment": "production", "ref": "develop", "build": null},
"developer_portal_auth": "disabled",
"update_mode": "manual"
}
A pool override can contain just {"sentinel": {"ref": "release-1"}}.
Omitted or null components inherit. A component override replaces that component's
whole selection, so specify its environment and build pin when needed.
CLI commands
Bundle activation exposes sima-cli and the Hugging Face hf command under
/usr/local/bin, pointing into the same bundled CLI environment. Installation
checks that hf runs, including through managed runner and login-shell paths.
The command links are restored with the rest of the bundle during rollback.
Existing devices receive installer changes through bootstrap or a bundle rollout;
updating Fleet Manager alone does not repair a device's command links.
Upgrade existing devices
- Save the pool policy and select Resolve desired bundle.
- Review the exact resolved component versions. Optionally select Pin these versions in policy and save again.
- Select devices and choose Upgrade N selected devices.
- Inspect per-device and per-component progress in the same section.
Up to four devices upgrade concurrently across Fleet Manager, shared across all
pools. Set KERRIGAN_BUNDLE_UPDATE_CONCURRENCY to a positive integer and restart
Fleet Manager to change this limit; 1 restores serial updates. Bootstrap jobs
have a separate concurrency limit. Offline, unhealthy, busy, or
leased devices wait. Bundle rollout is allowed after an applicable maintenance
window reaches active, but waits while maintenance is entering, exiting, or in
error. Devices remain reserved against conflicting tasks and other maintenance work.
A pool can have one unfinished rollout. Pause stops new admissions and lets all active devices finish;
cancel stops remaining work after active work/recovery completes. Changing policy
cancels stale queued work; an in-progress installation finishes its frozen snapshot.
Automatic-when-idle mode resolves publications with a five-minute cache and queues updates for differing bundle receipts. Cancelling holds that policy/build until a new build, a policy save, or a manual rollout. Unknown component versions are not reported as current. The table shows last verified component versions; it is not continuous detection of software modified outside Kerrigan.
Developer Portal authorization
Set Credential management to Project Kerrigan authorization only for pools whose DevKits should use the account authorized under System → Developer Portal. Enabling or disabling this setting changes the bundle ID, so existing devices follow the pool's normal manual or automatic rollout process. Updating Fleet Manager alone does not change a DevKit.
Managed projection currently requires production Developer Portal authorization. Kerrigan rejects a staging authorization rather than placing a staging token in a DevKit whose sima-cli uses the production Developer Portal contract.
Kerrigan refreshes the long-lived authorization centrally. It installs an atomic,
owner-only sima-cli cache under the sima account containing a short-lived access
token but no refresh token, browser cookies, or CSRF state. A private marker makes
the file explicitly Kerrigan-owned. Disabling the option removes a managed cache
but preserves a login created independently on the DevKit.
After a successful rollout, Fleet Manager renews the projection only while the DevKit is online, healthy, idle, unleased, and free of another maintenance operation. Projection is allowed during an active maintenance window, but waits while maintenance is entering, exiting, or in error. Busy and offline devices retain their current projection and reconcile when they next become eligible. If authorization expires while Fleet Manager is unavailable, sima-cli fails closed and the operator must restore Fleet Manager connectivity or authorize the DevKit independently.
Logging out of Developer Portal on Fleet Manager stops future renewal; it does not revoke already issued access tokens. To remove managed caches, first select Do not manage DevKit authorization and complete that bundle rollout, then log out from Fleet Manager. Token contents and identity claims are never included in bundle snapshots, receipts, API responses, or rollout logs.
Validation and recovery
Bundles accept aarch64 Platform 2.0.0 and newer, including 2.1.1, with system
Python 3.11 or newer. This runtime requirement is independent of the firmware
images supported for recovery. The supported_platforms snapshot field expresses
this range as [">=2.0.0"]. Individual components must still pass their installation
and activation checks; the version floor does not guarantee every future platform
or component combination. Git and
wireguard-tools are installed without recommended packages as standard
prerequisites during bootstrap and bundle installation; their versions are
selected by the platform APT repositories and are not policy fields. Python
venv support is installed if missing; system Python is not upgraded. Artifacts are checked
against their recorded SHA256 before upload. Fresh environments and binaries are
staged at permanent paths under /opt/kerrigan/bundles/releases before activation.
When Kerrigan adds a required prerequisite, it increments the bundle prerequisite
revision. This changes the resolved bundle ID so already-provisioned DevKits are
included in their next bundle rollout.
After sima-cli is installed or the selected bundle is activated, bootstrap inspects
the NVMe namespace. Existing partitions or recognized disk/filesystem signatures are
never automatically reformatted; bootstrap runs sima-cli nvme remount for those
devices. It runs sima-cli nvme format only when the namespace has neither child
partitions nor recognized signatures. Inspection, formatting, remount, and final mount
validation failures stop bootstrap instead of leaving later installers on unusable or
stale NVMe storage.
Existing configuration is retained during upgrades. Bootstrap/post-flash writes
the Fleet Manager connection configuration.
The installer validates Python dependencies, the CLI version, the requested
Developer Portal credential state, Sentinel binary and
API, the P2P forwarder checksum and version, and a fresh healthy agent heartbeat. Bundle receipts and pip installation
reports are saved on the device in /etc/kerrigan/device-bundle.json. Fleet Manager
records a completed receipt only after its heartbeat validation succeeds.
The device agent passively reports the current state of Kerrigan-managed common
mounts on registration and heartbeat. Fleet Manager displays old observations
as stale. A mounted observation comes from local mount metadata and is not an
I/O health check against the server. The status format reserves user-scoped
attachment IDs and generations for a later allocation-specific mount lifecycle.
For a sima-cli branch selection, Kerrigan resolves the production branch's
latest publication once, verifies the package ZIP, extracts its universal
wheel, and freezes the resulting build in the bundle snapshot. This has the same
source selection as sima-cli selfupdate --prod --branch <branch> while ensuring
every device in a rollout receives the same build.
Top-level artifacts and the Sentinel playbook are frozen. Python transitive
dependencies are currently resolved by pip for the device's Python version;
installation reports record them, but they are not yet locked across devices.
The native P2P forwarder is activated atomically with the bundle and linked at
/usr/local/bin/kerrigan-p2p-forwarder. Updating only its policy selection
creates a new immutable bundle while retaining the other resolved component
versions. Host clients install their matching local payload separately. Optional
WireGuard/NFS SDK-connectivity packages remain managed by SDK connection setup,
outside this bundle policy.
Failure pauses the rollout and attempts to restore previous executable links and services. If recovery cannot be verified, the device stays reserved. Retry bundle recovery explicitly retries the same frozen bundle, including the playbook. An unknown previous playbook cannot be claimed as restored. Host restarts pause interrupted rollouts and attempt rollback rather than silently repeating activation. Previous release directories are retained for running processes and recovery; no automatic pruning is performed yet.
Concurrent updates keep separate device reservations, transaction journals, and health checks. A failure pauses new admissions for that rollout immediately; other in-flight devices finish or recover independently. A slot remains occupied through rollback and cleanup. Cancellation waits until all active devices finish and any retained maintenance reservations are recovered. After an appliance restart, interrupted devices are recovered within the same concurrency limit; the rollout stays paused until an operator resumes it.
API
All modifying routes require a signed-in system administrator.
| Route | Purpose |
|---|---|
GET /v1/device-bundles/policy | System configured/effective policy |
PUT /v1/device-bundles/policy | Save the system policy |
GET /v1/device-bundles/pools/{pool_id}/policy | Policy, devices, receipts, rollout progress |
PUT /v1/device-bundles/pools/{pool_id}/policy | Save component overrides |
DELETE /v1/device-bundles/pools/{pool_id}/policy | Restore full inheritance |
POST /v1/device-bundles/pools/{pool_id}/resolve | Resolve an immutable bundle snapshot |
POST /v1/device-bundles/pools/{pool_id}/rollouts | Upgrade optional target_ids, with optional expected_bundle_id guard |
GET /v1/device-bundles/rollouts | Rollout history |
POST /v1/device-bundles/rollouts/{rollout_id}/{action} | pause, resume, recover, cancel |