MQTT State Reporting API

Kerrigan reports appliance, fleet, and liveness state to AWS IoT Core over MQTT. Cloud systems can consume these reports directly from MQTT, route them through AWS IoT Rules, or read the latest fleet state maintained by Neat CICD Infrastructure.

This reference describes the reporting contract produced by Kerrigan. For profile installation and connector operations, see Neat CICD Infrastructure reporting.

Connection and Identity

The appliance connects to the AWS IoT Core data endpoint from its imported profile using mutual TLS on port 8883 and MQTT 3.1.1.

The following profile values must contain the same immutable identity:

cluster_id = thing_name = MQTT client_id

The human-readable cluster_name is metadata and does not replace the immutable cluster ID. A cloud integration should use cluster_id as its primary key.

All reports are:

• UTF-8 JSON objects with schema_version: 1;
• published at MQTT QoS 1;
• published with MQTT retain disabled.

QoS 1 is an at-least-once delivery contract. Consumers must tolerate duplicate messages. Do not give a cloud consumer the appliance certificate or private key; use an AWS IAM identity or an AWS IoT Rule for cloud-side access.

Pairing profiles can also contain publish_topics.claims and subscribe_topics.offers entries reserved for compatibility with the cloud pairing protocol. The current Fleet Manager does not publish claims or process offers over MQTT; they are not part of the state-reporting or control APIs.

Report Types and Frequency

Report	Destination	Default frequency	Additional triggers	Maximum size
Fleet snapshot	publish_topics.status	Every 60 seconds	MQTT connection and material fleet-state changes	120 KiB
Heartbeat	publish_topics.heartbeats	Every 30 seconds	None	120 KiB
Host state	Named Device Shadow host	Every 60 seconds	MQTT connection	8 KiB

Kerrigan checks for material fleet-state changes approximately once per second. Changes to reported_at, sequence, or only a DevKit's last_seen_at do not trigger an extra snapshot. The next periodic snapshot still includes the latest values.

The reporting intervals can be changed on the appliance:

KERRIGAN_VULCAN_REPORTING_INTERVAL_SEC=60
KERRIGAN_VULCAN_HEARTBEAT_INTERVAL_SEC=30

Kerrigan enforces a minimum effective interval of five seconds for both values. If the MQTT connection is unavailable, local fleet operation continues. After reconnection, Kerrigan immediately publishes a fresh fleet snapshot and host state.

Fleet Snapshot

Topic

The topic is supplied by publish_topics.status. The standard topic is:

kerrigan/{environment}/clusters/{cluster_id}/status

Example:

kerrigan/staging/clusters/ll2-b887d76b-8989-461d-b42b-8d5b1e93de9a/status

Example payload

{
  "schema_version": 1,
  "cluster_id": "ll2-b887d76b-8989-461d-b42b-8d5b1e93de9a",
  "cluster_name": "ll2",
  "reported_at": "2026-09-03T00:40:00.123456+00:00",
  "sequence": 42,
  "kerrigan": {
    "version": "0.1.0",
    "hostname": "jim-NucBox-K10",
    "status": "degraded",
    "channel": "develop",
    "commit": "abcdef1234567890",
    "release_status": "update_available",
    "update_available": true,
    "latest_commit": "fedcba6543210987",
    "maintenance_control": {"available": true},
    "cleanup_control": {"available": true},
    "metrics_window_seconds": 60.0,
    "metrics_sample_count": 60,
    "network": {
      "uplink_interface": "enp4s0",
      "link_speed_mbps": 1000.0
    },
    "metrics": {
      "cpu_used_percent": 18.4,
      "memory_used_percent": 42.1,
      "root_fs_used_percent": 31.7,
      "network_rx_bytes_per_second": 1250000.0,
      "network_tx_bytes_per_second": 625000.0,
      "network_rx_utilization_percent": 1.0,
      "network_tx_utilization_percent": 0.5
    }
  },
  "summary": {
    "devkit_count": 2,
    "online_count": 1,
    "offline_count": 1,
    "busy_count": 0,
    "available_count": 1
  },
  "devkits": [
    {
      "id": "modalix-01",
      "name": "DevKit 01",
      "type": "modalix",
      "status": "online",
      "state": "idle",
      "platform_version": "2.1.2",
      "neat_version": "0.4.0+release-3.0.0-prep.b728dc3a138b",
      "ip_address": "10.77.77.20",
      "last_seen_at": "2026-09-03T00:39:55.000000+00:00",
      "architecture": "aarch64",
      "capabilities": ["nvme"],
      "metrics": {
        "cpu_used_percent": 12.5,
        "memory_used_percent": 37.2,
        "root_fs_used_percent": 48.0,
        "nvme_fs_used_percent": 21.4,
        "temperature_celsius": 72.9,
        "power_watts": 13.3
      }
    },
    {
      "id": "modalix-02",
      "status": "offline",
      "state": "idle",
      "last_seen_at": "2026-09-03T00:34:20.000000+00:00"
    }
  ]
}

Top-level fields

Field	Type	Description
schema_version	integer	Payload schema version. Currently 1.
cluster_id	string	Immutable cluster identity and AWS IoT Thing name.
cluster_name	string	Human-readable appliance or cluster name. It is not guaranteed to be unique.
reported_at	RFC 3339 string	UTC time at which Kerrigan assembled the snapshot.
sequence	integer	Process-local sequence shared with heartbeat reports. It can contain gaps and resets after a Fleet Manager restart.
kerrigan	object	Kerrigan software and aggregate status.
summary	object	Counts derived from the devkits array.
devkits	array	Complete known DevKit inventory and current state. This is a snapshot, not a patch.

kerrigan fields

Field	Type	Description
version	string	Running Kerrigan software version.
hostname	string	Hostname reported by the Kerrigan server.
status	string	Aggregate state: healthy, degraded, or error.
channel	string	Installed Ubuntu host software channel.
commit	string	Git commit recorded by the installed Ubuntu host package.
release_status	string	Channel comparison result: latest, update_available, or unknown.
update_available	boolean or null	Whether the channel catalog identifies a newer commit; null when no reliable comparison is possible.
latest_commit	string	Latest Git commit advertised for the installed channel, when available.
metrics_window_seconds	number	Rolling host-utilization window, 60 seconds by default.
metrics_sample_count	integer	Number of controller samples represented by the current rolling average.
network	object	Selected uplink interface and its advertised link speed, when available. Kerrigan selects the active IPv4 default-route interface with the lowest route metric.
metrics	object	Rolling controller and uplink utilization averages. Unavailable measurements are omitted.

Controller metric keys are:

Metric	Unit	Description
cpu_used_percent	percent	Host CPU time that was not idle.
memory_used_percent	percent	Host memory in use, calculated using MemAvailable.
root_fs_used_percent	percent	Used space on the root filesystem visible to Kerrigan.
network_rx_bytes_per_second	bytes/second	Average traffic received through the selected uplink.
network_tx_bytes_per_second	bytes/second	Average traffic transmitted through the selected uplink.
network_rx_utilization_percent	percent	Receive throughput divided by the interface link speed. Omitted when link speed is unavailable.
network_tx_utilization_percent	percent	Transmit throughput divided by the interface link speed. Omitted when link speed is unavailable.

Aggregate status is derived in this order:

1. error when any DevKit has status error;
2. degraded when no DevKit has an error but one or more are offline;
3. healthy otherwise.

summary fields

Field	Type	Description
devkit_count	integer	Total number of DevKits in the snapshot.
online_count	integer	DevKits whose status is not offline, including busy and error states.
offline_count	integer	DevKits whose status is offline.
busy_count	integer	DevKits whose status is busy.
available_count	integer	DevKits with fresh online connectivity, healthy underlying health, idle effective state, and no blocking maintenance. This is basic readiness, not allocation authorization.

DevKit fields

Field	Type	Required	Description
id	string	yes	Stable DevKit target ID.
type	string	yes	Detected DevKit type, for example modalix.
status	string	yes	online, busy, offline, or error.
state	string	yes	Current operational state. During firmware work this is the detailed update phase, such as ota_updating; otherwise it is the target state, such as idle, running, peer-pairing, or peer-paired.
firmware_update_state	string	no	Firmware-job phase or result, when retained on the target. Retained labels are history only. Detailed state comes from a matching active durable firmware job; terminal and orphaned labels do not override it.
name	string	no	Operator-assigned display name.
platform_version	string	no	Last reported platform or image version.
neat_version	string	no	Installed Neat core version reported by neat --json. Omission means the device agent could not read a non-empty version; it does not assert that Neat is absent.
ip_address	string	no	Last reported DevKit IP address.
last_seen_at	RFC 3339 string	no	Time of the most recent device-agent heartbeat known to Kerrigan.
architecture	string	no	Reported machine architecture, for example aarch64.
capabilities	array of strings	no	Enabled capability labels with the capability_ prefix removed.
peripherals	array of strings	no	Explicit peripheral inventory from the device agent. Omission means unknown, not an empty peripheral set.
metrics	object	no	Arithmetic mean of device-agent samples in the current reporting window (60 seconds by default). Supported keys are cpu_used_percent, memory_used_percent, root_fs_used_percent, nvme_fs_used_percent, network_rx_bytes_per_second, network_tx_bytes_per_second, network_rx_utilization_percent, network_tx_utilization_percent, temperature_celsius, and power_watts; unavailable measurements are omitted. temperature_celsius is the hottest finite thermal sensor in a fresh Sentinel sample, and power_watts is Sentinel's current total board-rail power. Other DevKit metrics come from a fresh Sentinel sample when available, with local CPU, memory, and filesystem collection as a fallback.
peer_pairing	object	no	Active Peer Pairing state reported by the device agent. Contains only session_id, lifecycle state, selected transport (p2p, routed, or staging-only ice), path (direct or relay), health, expiry, handshake timestamp, and a fixed diagnostic message on error. Unknown keys and device error text are omitted; no arbitrary nested data is forwarded.

A DevKit is classified as:

• offline when it has no heartbeat, its last heartbeat is more than 30 seconds old, or its internal state is explicitly offline;
• error when its health is unhealthy or degraded, or it is quarantined;
• busy while booting, bootstrapping, leased, running, cleaning up, recovering, updating, pairing a peer, or paired with a peer;
• online otherwise.

Heartbeat

Topic

The topic is supplied by publish_topics.heartbeats. The standard topic is:

kerrigan/{environment}/clusters/{cluster_id}/heartbeats

Payload

{
  "schema_version": 1,
  "cluster_id": "ll2-b887d76b-8989-461d-b42b-8d5b1e93de9a",
  "reported_at": "2026-09-03T00:40:30.123456+00:00",
  "sequence": 43,
  "status": "online",
  "uptime_seconds": 86400
}

The heartbeat confirms that Fleet Manager is running and connected to AWS IoT Core. uptime_seconds is measured from the start of the current Fleet Manager process. A heartbeat does not assert that every DevKit is healthy; use the fleet snapshot for cluster health.

Because the topic is not retained, cloud consumers should determine offline or stale state from their own message-receipt time. reported_at is useful for diagnostics but depends on the appliance clock.

Host Named Device Shadow

Kerrigan publishes host information to the AWS IoT named Device Shadow host.

Update topic

$aws/things/{thing_name}/shadow/name/host/update

The corresponding cloud API is the standard AWS IoT Data Plane named-shadow API. Cloud consumers normally read the current document with GetThingShadow(thingName, shadowName="host") instead of subscribing to the appliance's update topic.

Update payload

{
  "state": {
    "reported": {
      "schema_version": 1,
      "observed_at": "2026-09-03T00:40:00.123456+00:00",
      "os": {
        "name": "Ubuntu",
        "version": "24.04",
        "kernel": "6.8.0-79-generic",
        "architecture": "x86_64"
      },
      "cpu": {
        "logical_cores": 16,
        "model": "Intel(R) Core(TM)"
      },
      "memory": {
        "total_bytes": 33554432000,
        "available_bytes": 21474836480
      },
      "disks": [
        {
          "mount": "/",
          "total_bytes": 1000204886016,
          "available_bytes": 750153664512
        }
      ],
      "network": {
        "hostname": "jim-NucBox-K10",
        "uplink_interface": "enp4s0",
        "link_speed_mbps": 1000.0
      },
      "metrics_window_seconds": 60.0,
      "metrics_sample_count": 60,
      "metrics": {
        "cpu_used_percent": 18.4,
        "memory_used_percent": 42.1,
        "root_fs_used_percent": 31.7,
        "network_rx_bytes_per_second": 1250000.0,
        "network_tx_bytes_per_second": 625000.0,
        "network_rx_utilization_percent": 1.0,
        "network_tx_utilization_percent": 0.5
      },
      "kerrigan": {
        "version": "0.1.0",
        "uptime_seconds": 86400,
        "status": "degraded"
      },
      "fleet_summary": {
        "devkits": 2,
        "online": 1,
        "offline": 1,
        "busy": 0,
        "available": 1
      }
    }
  },
  "clientToken": "3d4ce37bde514501bdf38cf8f95cd91b"
}

CPU model and memory byte values can be absent when the host operating system does not expose them. The shadow service adds its standard metadata, version, and timestamp fields to the stored response document.

The shadow and fleet snapshot carry the same rolling controller metrics. Network rates are derived from Linux interface byte-counter deltas. Link-utilization percentages are included only when the selected uplink exposes a positive link speed through sysfs; throughput rates remain available otherwise.

The host shadow's kerrigan.status is degraded when at least one DevKit is offline and healthy otherwise. Use the fleet snapshot when an integration must distinguish DevKit error state from offline state.

Kerrigan subscribes to the named shadow's accepted and rejected response topics and correlates responses with clientToken. The device policy must allow the exact named-shadow update and response topics.

Sequence and Time Semantics

Fleet snapshots and heartbeats share one increasing, process-local sequence. Consequently, a consumer can observe gaps in status-topic sequence values when heartbeats were published between snapshots. A Fleet Manager restart resets the sequence, so it must not be treated as a permanent event ID.

For freshness and ordering across reconnects or restarts, cloud systems should record a server-side ingestion timestamp. The Neat CICD Infrastructure AWS IoT Rule adds vulcan_updated_at_ms when it stores a fleet snapshot.

Neat CICD Infrastructure Persistence

The standard Neat CICD Infrastructure deployment processes reports as follows:

Input	Cloud destination	Behavior
kerrigan/{environment}/clusters/+/status	DynamoDB vulcan-{environment}-kerrigan-cluster-state	Replaces the previous item for that cluster.
kerrigan/{environment}/clusters/+/+	SQS inbound event queue	Preserves reports and other appliance events for asynchronous processing.
Named Device Shadow host	AWS IoT Device Shadow service	Maintains the current host-reported state.

The DynamoDB table uses cluster_id as its string partition key and has no sort key. The JSON snapshot is stored in the state attribute. The IoT Rule also adds:

• vulcan_cluster_id, derived from the authenticated MQTT topic;
• vulcan_updated_at_ms, generated by AWS IoT Core at ingestion time.

Use vulcan_updated_at_ms to evaluate cloud-side freshness. Kerrigan certificates publish through AWS IoT Core and do not receive direct DynamoDB access.

Consumer Requirements

A cloud integration should:

1. authorize MQTT subscription or IoT Rule access only to the required environment and cluster topics;
2. validate schema_version before processing a report;
3. verify that the payload cluster_id matches the cluster ID in the topic;
4. treat a fleet snapshot as a complete replacement rather than a partial update;
5. tolerate duplicate QoS 1 deliveries and process-local sequence resets;
6. use cloud receipt time to detect stale heartbeats and snapshots;
7. avoid using cluster_name as a unique identifier.

If a fleet snapshot exceeds 120 KiB or a host shadow update exceeds 8 KiB, Kerrigan first omits optional fleet detail and sets details_omitted: true. Every device, its identity, operational state, health, maintenance blocker and control capabilities remain present. If mandatory inventory still exceeds the limit, or a host shadow exceeds its limit, the complete payload is rejected. Reporting stages are failure-isolated and repeated failure logs are rate-limited.

Power-control availability

The full cluster state includes kerrigan.power_control.available and devkits[].power_control.available booleans. For example:

{
  "kerrigan": {"power_control": {"available": true}},
  "devkits": [
    {"id": "board-with-relay", "power_control": {"available": true}},
    {"id": "board-without-relay", "power_control": {"available": false}}
  ]
}

This example shows only the added fields. Appliance availability means a USB relay was discovered without an access error. Device availability additionally requires an assigned channel within that connected relay's capacity. Availability is independent of DevKit network reachability and does not indicate whether power is on. Discovery does not initialize or change relay outputs.

Relay disconnection, reconnection, and assignment changes update availability in subsequent cluster snapshots. Consumers must treat absent fields from older controllers as unavailable and must not enable controls using stale appliance state. NeatOps shows the Power tab only for a fresh appliance report with both availability flags true. The controller revalidates availability when executing devices.powerctl; reported metadata is not an authorization grant.

Local admin portal

kerrigan.network.admin_url advertises the local dashboard URL, for example http://192.168.90.222:8001/admin. Kerrigan discovers the IPv4 address of its lowest-metric default-route interface and uses dashboard port 8001. If discovery fails, the field is omitted. Deployments must expose that port on the host.

NeatOps uses this URL for the administrator-only Open admin link. The browser must have network access to the appliance (LAN or VPN); this link does not proxy the portal through the cloud. Older appliances without this field require an update before they can advertise a local portal URL.

Maintenance state

The appliance capability is described under Maintenance capability.

DevKit entries include a separate maintenance object with enabled, blocked, phase, scopes, services, error, and updated_at. While blocked, effective health and status are in-maintenance. underlying_health and connectivity_status retain health/connectivity independently; offline counts continue to reflect actual connectivity and maintained devices are not available for new work. Intent survives controller restarts and is inherited from appliance, pool, and device scopes.

Cleanup state

kerrigan.cleanup_control.available: true advertises support for the devices.cleanup operation. A DevKit's latest cleanup is included in its fleet snapshot:

{
  "cleanup": {
    "job_id": "cleanup-0123456789ab",
    "state": "running",
    "phase": "deleting_workloads",
    "last_progress_at": "2026-10-16T12:00:00+00:00"
  }
}

last_progress_at is the controller's UTC observation time of the latest state or phase transition. It is not a device heartbeat, percentage, or proof that a long-running phase is still making progress. Repeated snapshots and receipt-only updates do not advance it. Older jobs without progress records may expose only job_id and state until their next transition.

The latest cleanup remains in the snapshot after completion or failure, even after the device reservation is released. A later cleanup replaces it. Device availability is determined by the device state and reservations, not merely by the presence of the cleanup object. Recovery-required devices remain reserved.

Each state or phase transition also produces a device_cleanup_progress event on the configured aggregate events topic (the status-topic sibling events when no events topic is configured). Events include event_id, job_id, target_id, target_mac, state, phase, and last_progress_at. They are persisted with the job update and published with MQTT QoS 1, without retain. Unacknowledged events are retried after reconnection or controller restart; consumers must deduplicate by event_id. Events are not command acknowledgements and have no command_id. Use the local cleanup job API for per-path results, errors, and retry.

Pool membership

devkits[].pool_id reports membership in a local Kerrigan pool. Pools divide an appliance's devices into local groups for allocation, policy, and administration. A pool ID is scoped to its Kerrigan instance: the same ID on two appliances does not identify a shared cloud pool. Consumers should use (cluster_id, pool_id) when identifying or grouping pools across appliances.

An unassigned device reports pool_id: "default". Pool membership does not indicate maintenance support or whether a device is currently in maintenance. Local pool maintenance policy can affect member devices, but that effective state is reported separately in devkits[].maintenance.

The wire format remains the flat devkits array in schema version 1. Consumers may group it by local pool for display without changing that contract. Older appliances may omit pool_id; display those devices without assuming a reported pool membership.

Maintenance capability

kerrigan.maintenance_control.available: true advertises that this Kerrigan instance supports the devices.maintenance control operation. It is independent of pool membership and is not a per-device maintenance state, readiness signal, or authorization grant.

Consumers should offer maintenance controls only when a fresh appliance report advertises this capability. Treat an absent capability on older appliances, or available: false, as unsupported. The field is additive in schema version 1.

After submitting a command, follow each device's maintenance state. See the maintenance window lifecycle for entry, reconciliation, and explicit exit.

Projection and retry semantics

connectivity_status is online only with a heartbeat no older than 30 seconds, no more than five seconds in the future, and a target state other than offline. Busy and maintenance states do not imply a lost connection. Fleet and host-shadow health use the same aggregate rule; their independent observation times may differ.

available_count describes basic readiness. Allocation still checks reservations, active firmware/agent maintenance, session leases, capability, cloud inventory freshness, and cloud locks at admission. A snapshot does not authorize a lease.

Firmware detail is resolved from active durable jobs by target identity or MAC. The reporter never repairs lifecycle state or releases operation ownership.

Publication signatures are committed only after MQTT accepts the message into its local queue. This is not a broker acknowledgment. Failed reports retry after five seconds without requiring a state change; reconnect publishes fresh state. Sequence and fleet-signature bookkeeping are serialized across reconnect and periodic reporting. The connector's reporting_errors reports failing stages and clears each stage after recovery. Historical last_error remains diagnostic evidence.

Optional strings, lists, maintenance detail and pairing fields are bounded before publication. Maintenance and pairing error text is replaced with fixed diagnostic messages; inspect local diagnostics for details. Pairing timestamps must be bounded nonnegative integers; session identifiers must match the 32-character lowercase hexadecimal transport format.