Cloud Control CLI (Validation Tool)

kerrigan-cloud-control-cli discovers paired Kerrigan systems in AWS and sends cluster-scoped control commands through AWS IoT Core. It is an optional validation and troubleshooting tool, not a required component of a cloud integration. Production cloud applications can implement the discovery, MQTT control, and state-reporting contracts directly.

For the overall control model and cloud-pairing boundary, see the Cloud API overview. For appliance-side command handling, see Backoffice control.

Install

Install the package published by the desired Kerrigan branch:

sima-cli neat install --type cloud-control-cli kerrigan@develop:latest
kerrigan-cloud-control-cli --help

Authenticate

The default mode uses AWS IAM credentials to sign an MQTT WebSocket connection. Authenticate and select the AWS profile for the target Neat CICD Infrastructure (Vulcan) environment:

saml2aws login --profile vulcan-staging
export AWS_PROFILE=vulcan-staging
export AWS_DEFAULT_REGION=us-west-2

The environment is inferred from AWS_PROFILE. Use --environment when the profile name does not contain dev, stg/staging, or prod/production:

kerrigan-cloud-control-cli --environment staging list-systems

Discovery data sources

Data	Source	Default resource
Paired systems and display names	AWS IoT registry	vulcan-{environment}-kerrigan-clusters Thing Group
IoT WebSocket endpoint	AWS IoT	iot:Data-ATS endpoint hostname
Latest system and DevKit state	DynamoDB	vulcan-{environment}-kerrigan-cluster-state
Commands and streamed results	AWS IoT MQTT	Topics derived from environment and immutable Thing ID

Status reads use batched keys for only the discovered systems. They do not scan the state table. Registry-only discovery is available when DynamoDB access is not needed:

kerrigan-cloud-control-cli list-systems --no-status

Global options

Global options normally appear before the command:

--system NAME_OR_CLUSTER_ID
--environment ENVIRONMENT
--region AWS_REGION
--thing-group THING_GROUP
--state-table TABLE_NAME
--profile YAML_PATH
--auth aws|mtls
--timeout SECONDS
--json
--color auto|always|never

--thing-group and --state-table override the conventional resource names for nonstandard deployments. --profile selects legacy/manual routing and cannot be combined with discovery-based --system selection.

Query systems and DevKits

List systems

List every Kerrigan Thing registered in the environment and join its latest status:

kerrigan-cloud-control-cli list-systems
kerrigan-cloud-control-cli list-systems --json

Human-readable output includes system name, aggregate status, last report time, and immutable cluster ID.

Describe a system

Select a system by its human-readable ClusterName or immutable cluster ID:

kerrigan-cloud-control-cli describe-system ll2
kerrigan-cloud-control-cli describe-system ll2 --json

The human-readable view contains color-coded tables for identity, MQTT routing, latest status, controller utilization, and DevKits. Latest status includes the installed software channel, Git commit, and whether an update is available. The controller-utilization table shows rolling-average CPU, RAM, root-filesystem, and uplink receive/transmit usage over the current reporting window (60 seconds by default). An available update is highlighted in yellow. If a display name matches multiple systems, use the immutable cluster ID shown by list-systems.

Describe a DevKit

Find a DevKit and its owning Kerrigan system from the latest reported fleet state:

kerrigan-cloud-control-cli describe-devkit bootstrap-68e154065848
kerrigan-cloud-control-cli --system ll2 describe-devkit bootstrap-68e154065848
kerrigan-cloud-control-cli describe-devkit bootstrap-68e154065848 --json

Use --system to disambiguate a DevKit ID reported by multiple systems. The human-readable view includes connectivity status, operational state (for example ota_updating), the detected DevKit type, average CPU, RAM, root-filesystem and NVMe utilization, and Sentinel's default-route uplink receive/transmit rates and directional link utilization. Kerrigan reports the arithmetic mean of fresh device-agent samples collected during its current 60-second cloud-reporting window. The same numeric values are available under devkit.metrics with --json.

Select a system for control

Use --system for explicit routing:

kerrigan-cloud-control-cli --system ll2 ping

The selector can be a display name or cluster ID. Without --system, the CLI:

1. selects the only discovered system when exactly one exists;
2. resolves the owner from --target DevKit IDs for target-scoped commands;
3. stops with an ambiguity error instead of guessing when multiple systems remain possible.

Pool-scoped commands and system-wide commands should specify --system when an environment contains multiple Kerrigan appliances.

Control commands

Command	Purpose
ping	Run a read-only end-to-end transport and dispatcher check.
status COMMAND_ID	Retrieve persisted events for an earlier command.
boot	Request a reboot for DevKits selected by --target or --pool.
ota	Run firmware OTA for DevKits selected by --target or --pool.
update	Ask the Kerrigan appliance to update itself.
sdk connect	Generate a key on one DevKit, register its public key, connect WireGuard, and mount NFS.
sdk disconnect	Unmount NFS and remove both sides of the SDK WireGuard pairing.
send OPERATION	Submit a raw supported operation with JSON arguments.

Ping

kerrigan-cloud-control-cli --system ll2 ping

Query an earlier command

The CLI prints the generated command ID before sending a human-readable control request. Use that ID to query the persisted terminal state later:

kerrigan-cloud-control-cli --system ll2 status cmd-0123456789abcdef

Boot DevKits

Repeat --target to address multiple DevKits in the same Kerrigan system:

kerrigan-cloud-control-cli boot \
  --target bootstrap-68e154065848 \
  --target bootstrap-68e154065849

kerrigan-cloud-control-cli --system ll2 boot --pool validation

When --system is omitted, all target IDs must resolve to exactly one appliance.

Update DevKit firmware

kerrigan-cloud-control-cli ota \
  --target bootstrap-68e154065848 \
  --channel ga

kerrigan-cloud-control-cli --system ll2 ota \
  --pool validation \
  --channel pre-release

Update Kerrigan

kerrigan-cloud-control-cli --system ll2 update \
  --channel develop \
  --version latest

The CLI resolves --system to its immutable cluster ID, includes that ID in the MQTT request, and requires --channel. It remains subscribed while the update runs and prints each host-updater progress/log event, including command output, until Kerrigan reports success or failure. Update commands default to a 30-minute timeout; override it with global --timeout when needed.

Use --force only when an appliance update must proceed while firmware jobs are active.

Pair a DevKit with the remote SDK workspace

Use the higher-level SDK command for the secure pairing workflow. The command prints the generated connection ID in its MQTT arguments/events; retain that ID for disconnect:

AWS_PROFILE=vulcan-stg \
  kerrigan-cloud-control-cli \
  --environment staging \
  --system ll2 \
  sdk connect \
  --target bootstrap-68e154065848 \
  --connection-id sdk-manual-test-1 \
  --install-tools

The CLI reads only the endpoint public configuration, submits devices.sdk.connect, waits for Kerrigan to publish the DevKit public key, and then authorizes that public key on EC2 through Systems Manager. No DevKit private key reaches the CLI or cloud controller. Kerrigan completes only after both the WireGuard handshake and NFSv4 mount succeed.

Disconnect using the same ID:

AWS_PROFILE=vulcan-stg \
  kerrigan-cloud-control-cli \
  --environment staging \
  --system ll2 \
  sdk disconnect \
  --target bootstrap-68e154065848 \
  --connection-id sdk-manual-test-1

Endpoint cleanup occurs only after Kerrigan confirms that the DevKit unmounted the workspace and removed its local tunnel. Disconnect is authoritative: if /workspace is busy, Kerrigan kills processes using that NFS filesystem before retrying the unmount. If forced unmount still fails, the command preserves both peers and reports failure instead of removing the tunnel under a live mount.

Send a raw operation

kerrigan-cloud-control-cli --system ll2 send kerrigan.ping --arguments '{}'

--arguments must be a JSON object accepted by the named operation.

Output formats

Interactive query commands use color when stdout is a terminal. Control it with:

kerrigan-cloud-control-cli describe-system ll2 --color always
kerrigan-cloud-control-cli describe-system ll2 --color never
NO_COLOR=1 kerrigan-cloud-control-cli describe-system ll2

Use --json for automation. Query commands emit one JSON document. Control commands emit one JSON object per received lifecycle event, making their output JSON Lines rather than a single JSON array:

kerrigan-cloud-control-cli describe-system ll2 --json | jq .
kerrigan-cloud-control-cli --json --system ll2 ping | jq -c .

JSON output never contains ANSI color codes or display headings.

The CLI derives each system's cloud_status from the server-side vulcan_updated_at_ms receipt timestamp. A persisted snapshot is shown as stale after 180 seconds by default, even if the appliance last reported itself as healthy. This prevents a replaced pairing or disconnected appliance from appearing healthy indefinitely. Use --stale-after SECONDS or KERRIGAN_CLOUD_STATUS_STALE_AFTER_SEC when an integration uses a reporting interval longer than the default 60 seconds.

Validate cloud pairing

The CLI can optionally validate the cloud-authorized half of the staging pairing protocol. Select the AWS environment through AWS_PROFILE, authenticate with an IAM identity that may invoke the claim route, and enter the code displayed by Kerrigan. The CLI maps the environment to its pairing service URL:

AWS_PROFILE=vulcan-stg \
  kerrigan-cloud-control-cli claim-pairing AAAQE-AYEAU-DAOCA-JBIFQ-YDIOB-4

--environment or KERRIGAN_ENV can select the environment when the AWS profile name does not contain stg, staging, prod, or production. --pairing-url (or KERRIGAN_PAIRING_URL) remains available as an override for custom and development deployments. Environments without a built-in pairing endpoint fail with an explicit error instead of using another environment's service.

Production integrations may implement the same signed HTTPS claim contract directly; this command is not part of the runtime control path.

Delete a cloud-registered system

Delete an obsolete Kerrigan identity by its immutable cluster ID:

AWS_PROFILE=vulcan-stg \
  kerrigan-cloud-control-cli delete-system \
  poc-1-5b9a4519-5757-444e-9636-9baa5b61f7c8

The CLI requires you to type the immutable cluster ID before it removes the AWS IoT Thing, Thing Group membership, dedicated certificate and policy attachment, named host shadow, and latest DynamoDB state. A principal shared with another Thing is detached but not deleted. For automation, pass --yes. When --yes is used, the selector must be the immutable cluster ID rather than a display name.

Deletion is allowed without --force only when the server-side status receipt is older than --stale-after. This protects an actively reporting system from accidental deletion. Use --force only when intentionally retiring a current or not-yet-reporting identity.

Manual profile and mTLS mode

Manual YAML profiles remain supported:

kerrigan-cloud-control-cli \
  --profile ~/.config/kerrigan/cloud-control-cli-staging.yaml \
  ping

Use --auth mtls only with a file profile containing cert_file, key_file, and root_ca_file. Never reuse the Kerrigan appliance certificate or MQTT client ID for the CLI; a second connection with the same identity can disconnect the appliance.

Required IAM permissions

Discovery requires:

iot:DescribeEndpoint
iot:ListThingsInThingGroup
iot:DescribeThing

Joining latest status requires dynamodb:BatchGetItem on the environment's cluster-state table. Control commands additionally require iot:Connect, iot:Publish on the selected commands topic, and iot:Subscribe plus iot:Receive on its command-event topic.

sdk connect and sdk disconnect additionally require ec2:DescribeInstances, ssm:SendCommand, and ssm:GetCommandInvocation for the dedicated endpoint instance.

The optional claim-pairing validation command requires execute-api:Invoke for the environment's POST /v1/pairings/claim route.

Deleting systems additionally requires iot:ListThingPrincipals, iot:ListAttachedPolicies, iot:ListPrincipalThings, iot:ListThingGroupsForThing, iot:DetachPolicy, iot:DetachThingPrincipal, iot:UpdateCertificate, iot:DeleteCertificate, iot:RemoveThingFromThingGroup, iot:DeleteThing, iot:DeleteThingShadow, and dynamodb:DeleteItem on the selected environment's Kerrigan resources.

Troubleshooting

Symptom	Likely cause	Action
environment is required	AWS profile name does not identify an environment.	Pass --environment or set KERRIGAN_ENV.
Kerrigan system not found	Wrong environment, region, Thing Group, or selector.	Run list-systems and use its cluster ID.
System or DevKit is ambiguous	A display name or DevKit ID appears in multiple systems.	Pass the immutable --system cluster ID.
AWS_IO_SOCKET_CLOSED	Credentials belong to the wrong AWS account, are expired, or the endpoint is unsuitable for WebSockets.	Verify aws sts get-caller-identity; discovery uses the raw ATS endpoint.
Command times out after publish	Appliance is offline, lacks cloud-control support, or cannot publish command events.	Check describe-system, appliance connector state, and installed Kerrigan version.
Status is stale or unavailable	The appliance has not recently published status for persistence in the cluster-state table.	Check appliance connectivity or use list-systems --no-status for registry-only discovery.
WireGuard endpoint not found	The test instance is absent, stopped, in another environment, or its tags changed.	Start or create the tagged endpoint, or pass --endpoint-instance-id.
WireGuard handshake times out	UDP 51820 is blocked, the DevKit lacks outbound Internet access, or the endpoint peer was replaced.	Check the endpoint security group, site egress policy, and whether another connection used the single-peer endpoint.

Device power control

Configure unique relay channels and NC/NO wiring in Kerrigan first. The CLI uses devices.powerctl with arguments.action for mutations and devices.power.status for readback:

kerrigan-cloud-control-cli power status --target modalix-123
kerrigan-cloud-control-cli power on --target modalix-123
kerrigan-cloud-control-cli power off --target modalix-123
kerrigan-cloud-control-cli power cycle --target modalix-123 --target modalix-456
kerrigan-cloud-control-cli --system ll2 power cycle --pool validation-lab

Use either repeated --target arguments or --pool. Target discovery can infer the owning controller; pool selection requires an explicit system. Devices must belong to the same controller. Power commands default to a 300-second CLI wait; override with --timeout before power for large batches. A timeout does not cancel an accepted command. Use status COMMAND_ID to inspect its progress before issuing another mutation.

A powered-off DevKit can be switched on without SSH. The controller must be connected, with an assigned and accessible USB relay. Readback indicates relay output, not successful device boot. For full result and failure semantics, see devices.powerctl.