Backoffice Control over IoT Core

For the complete caller contract, including topic names, JSON schemas, operation arguments, event fields, and retry behavior, see the MQTT Control API.

Kerrigan accepts asynchronous control commands through its cluster-scoped AWS IoT Core MQTT commands topic. It publishes acknowledgements, progress, and terminal results to publish_topics.events. If events is absent from an older profile, Kerrigan derives it by replacing the final status topic segment with events.

MQTT is an at-least-once transport. Backoffice must assign a globally unique, stable command_id and may safely redeliver the same command. Kerrigan persists the last event for every command and returns command.duplicate without running the operation again.

Command envelope

{
  "schema_version": 1,
  "command_id": "cmd-01J6...",
  "correlation_id": "ui-request-01J6...",
  "requester_id": "backoffice:operator@example.com",
  "operation": "devices.ota_update",
  "issued_at": "2026-08-31T19:00:00Z",
  "arguments": {
    "target_ids": ["modalix-01", "modalix-02"],
    "release_channel": "ga"
  }
}

Device operations accept either target_ids or pool_id. Supported operations are:

Operation	Arguments	Behavior
`kerrigan.ping`	none	Read-only end-to-end connectivity test.
`kerrigan.reboot`	`system_id`	Schedules controller-host reboot after one minute; completion means scheduled, not restarted.
`kerrigan.power.status`	`system_id`	Reads persistent controller power status without changing host power.
`kerrigan.update`	`system_id`, `channel`, optional `version` and `force`	Validates the immutable MQTT target, starts a host update, and streams updater state and logs through completion.
`devices.ota_update`	`target_ids` or `pool_id`, `release_channel` (`ga` or `pre-release`)	Starts an OTA job per target and streams job state/progress.
`devices.cleanup`	`target_ids` or `pool_id`, optional `reason`	Atomically reserves the selection, removes workload data, and reports the durable cleanup job through completion.
`devices.powerctl`	`target_ids` or `pool_id`, `action` (`on`, `off`, `cycle`)	Controls assigned USB relay channels; cycle interrupts power for 5 seconds.
`devices.power.status`	`target_ids` or `pool_id`	Reads relay availability and output state without changing power.
`devices.boot`	`target_ids` or `pool_id`	Issues the existing device reboot/boot action.
`devices.sdk.connect`	Exactly one `target_ids` entry, `connection_id`, public WireGuard endpoint configuration, and NFS configuration	Generates the DevKit key locally, publishes its public key for endpoint authorization, confirms a handshake, and mounts the SDK workspace.
`devices.sdk.disconnect`	Exactly one `target_ids` entry plus the active `connection_id`	Unmounts NFS and removes the DevKit key and tunnel. Endpoint cleanup remains the controller's responsibility; the cloud CLI handles it.

Event envelope

Every response includes the original command_id, correlation_id, requester_id, and operation:

{
  "schema_version": 1,
  "event_id": "evt-cmd-01J6...-...",
  "event_type": "command.progress",
  "command_id": "cmd-01J6...",
  "correlation_id": "ui-request-01J6...",
  "requester_id": "backoffice:operator@example.com",
  "operation": "devices.ota_update",
  "state": "running",
  "progress_percent": 45,
  "occurred_at": "2026-08-31T19:01:00Z",
  "detail": {}
}

Normal lifecycle:

command.received -> command.accepted -> command.progress (zero or more)
                 -> command.completed | command.failed

Invalid envelopes, unsupported operations, and target selectors that do not exist in the selected Kerrigan system produce command.rejected before the command is accepted. Human-readable CLI output includes the rejection reason. Duplicate delivery produces command.duplicate containing the command's last known state.

Control CLI

The Cloud Control CLI is an optional validation and troubleshooting tool. It exercises the same cloud contracts that a production cloud application can implement directly; the CLI is not required for cloud integration.

Install the packaged utility with:

sima-cli neat install --type cloud-control-cli kerrigan@develop:latest

With AWS IAM authentication, the CLI can discover paired systems directly from the environment's AWS IoT Thing Group. It joins the registry inventory with the latest cluster state stored in DynamoDB:

export AWS_PROFILE=vulcan-staging
kerrigan-cloud-control-cli list-systems
kerrigan-cloud-control-cli describe-system ll2
kerrigan-cloud-control-cli describe-system ll2 --json
kerrigan-cloud-control-cli describe-devkit modalix-123
kerrigan-cloud-control-cli --system ll2 describe-devkit modalix-123
kerrigan-cloud-control-cli --system ll2 ping

--system accepts either the human-readable ClusterName or the immutable cluster ID. If multiple systems have the same human-readable name, the CLI requires the immutable ID. The environment is inferred from AWS_PROFILE, or can be selected explicitly with --environment staging.

For target-specific controls, --system is optional. The CLI joins the DevKit ID against the latest discovered system status and routes the command to its owning Kerrigan system. Duplicate or stale ownership is rejected and requires an explicit --system; targets from different systems must be sent as separate commands.

describe-devkit <id> uses the same inventory lookup and shows the DevKit details, including connectivity status and operational state, together with its owning Kerrigan system. If the same DevKit ID appears under more than one system, pass --system with the system name or immutable cluster ID to disambiguate it.

The discovered routing profile exists only in memory. Its commands, events, status, and command-event topics are derived from the environment and immutable AWS IoT Thing name. A fresh MQTT client ID is generated for each invocation so concurrent operators do not disconnect one another.

Manual profiles remain supported for compatibility and for explicit mTLS testing. Place one under ~/.config/kerrigan/, or pass --profile. IAM mode only needs endpoint, region, client ID, and topics; certificate files are required only by --auth mtls. Available control commands include:

kerrigan-cloud-control-cli ping
kerrigan-cloud-control-cli status cmd-previous
kerrigan-cloud-control-cli update --channel develop --version latest
kerrigan-cloud-control-cli ota --target modalix-01 --channel ga
kerrigan-cloud-control-cli boot --target modalix-01
kerrigan-cloud-control-cli --system ll2 boot --target modalix-01
kerrigan-cloud-control-cli send kerrigan.ping --arguments '{}'

Normal IAM-authenticated control does not read a local YAML profile. Use --profile only for legacy/manual routing or explicit mTLS testing.

Human-readable discovery output uses color-coded tables when stdout is an interactive terminal. Use --color always to preserve color through a pipe or --color never to disable it; the standard NO_COLOR environment variable is also honored in automatic mode. Use --json with list-systems or describe-system for automation. JSON output never includes color codes or surrounding display text.

Human-readable command output prints the generated command ID immediately before publishing the command. Save that ID to query the persisted lifecycle later with status. JSON output remains newline-delimited event objects; every event includes the same command_id for machine correlation.

Override the conventional discovery resources with --thing-group or --state-table when testing a nonstandard deployment. list-systems --no-status queries only the IoT registry and does not require DynamoDB read access.

Discovery requires iot:DescribeEndpoint, iot:ListThingsInThingGroup, and iot:DescribeThing. Including latest status also requires dynamodb:BatchGetItem on the environment's cluster-state table. Status reads are batched by discovered cluster ID rather than scanning the table.

The CLI defaults to AWS IAM authentication over MQTT WebSockets. Run saml2aws login, select the matching AWS profile if necessary, and ensure the temporary credentials are exported or visible through the standard AWS credential chain. Use --auth mtls only for explicit certificate testing.

For example:

saml2aws login --profile vulcan-staging
export AWS_PROFILE=vulcan-staging
kerrigan-cloud-control-cli --profile examples/cloud-control-cli-staging.yaml ping

The assumed IAM role needs AWS IoT data-plane permissions to connect, publish to the selected cluster command topic, and subscribe/receive on that cluster's command-event topics. The example routing profile is examples/cloud-control-cli-staging.yaml.

The CLI subscribes before publishing, prints each sequenced event as it arrives, and exits nonzero on rejection, failure, or timeout.

Adding controls

New controls register an operation name and handler with ControlCommandDispatcher. Transport parsing, lifecycle events, correlation, idempotency, and error conversion remain shared. Keep handlers thin: invoke a durable domain job, then report its state rather than embedding operation logic in the MQTT callback.

The IoT policy for each Kerrigan Thing should allow subscription only to its own cluster command topic and publication only to its own event topic. Backoffice is responsible for authenticating and authorizing the requester before publishing; Kerrigan records requester_id as the operation actor.

The kerrigan-cloud-control-cli CLI must use a separate test-client certificate and client ID. Never copy the Kerrigan server certificate: connecting twice with the same MQTT identity can disconnect the server. The CLI YAML profile uses the same certificate and topic keys, but represents the requester identity.

Command envelope​

Event envelope​

Control CLI​

Adding controls​

Command envelope

Event envelope

Control CLI

Adding controls