Evaluate an Appliance with the Test Harness

Use the appliance test harness to qualify the thermal behavior, board-rail power, video throughput, metadata throughput, and detection consistency of a Kerrigan appliance and its attached Modalix DevKits. The harness runs natively on a macOS or Linux test computer connected to the appliance's DevKit switch.

Start with one DevKit. After that run is nominal, repeat the test with every DevKit that belongs to the appliance. This separates a pipeline or model problem from a network or appliance-scale problem.

What the Harness Tests

For each selected DevKit, the harness starts 48 copies of the same RTSP source and runs them through a shared YOLO26 detector. Each channel returns:

the original encoded H.264 stream, without re-encoding it;
JSON detection metadata from the frames processed by the detector.

The host receives the returned video without continuously decoding it. It uses RTP headers to calculate video FPS, bitrate, and packet loss, and parses the metadata to calculate metadata rate and golden-detection accuracy. The host also reads temperature and board-rail power from Kerrigan's metric history.

The Platform 2.1.3 profile publishes 9 FPS per channel, offering 432 aggregate decoder FPS for a 400 aggregate detector-FPS target (approximately 8.3 FPS per channel). The Platform 3.0 profile retains the validated 30 FPS source and targets 600 aggregate detector FPS, or 12.5 FPS per channel. Detector FPS and source FPS are intentionally different measurements.

The harness does not measure AC wall power. The displayed power value is the arithmetic mean of Sentinel board-rail samples in Fleet Manager's latest one-minute metric-history bucket. It is not an instantaneous reading or a peak. Use raw Sentinel sampling or an external power meter in parallel when the qualification requires transient or peak power.

Architecture

                         read-only REST requests
                  +-------------------------------+
                  |                               v
+-----------------+----------------+       +------------------+
| macOS or Linux test computer     |       | Kerrigan         |
|                                  |       | Fleet Manager    |
| appliance-test-harness           |       |                  |
|  - Fleet discovery and selection |<------| inventory        |
|  - MediaMTX RTSP server          |       | eligibility      |
|  - FFmpeg stream-copy publisher  |       | metric history   |
|  - UDP/RTP + metadata receivers  |       +------------------+
|  - local monitoring dashboard    |
+---------+-------------------^-----+
          |                   |
          | RTSP/H.264        | H.264 RTP + JSON metadata
          | 48 reads/DevKit   | 96 UDP ports/DevKit
          v                   |
       appliance DevKit switch
          |                   ^
          v                   |
  +----------------+  +----------------+       +----------------+
  | Modalix        |  | Modalix        |  ...  | Modalix        |
  | 48-stream      |  | 48-stream      |       | 48-stream      |
  | pyneat + YOLO  |  | pyneat + YOLO  |       | pyneat + YOLO  |
  +----------------+  +----------------+       +----------------+

The control and data paths are intentionally separate:

Kerrigan is the authority for inventory, current eligibility, management addresses, temperature, and power.
The test computer serves the reference media and receives all returned data.
The harness stages and controls its workload over SSH only after confirming that Fleet Manager reports the DevKit as healthy, online, idle, and outside maintenance.
RTSP and UDP traffic stays on the DevKit switch. It does not need to cross the appliance uplink.

caution

The standalone harness does not acquire an atomic Kerrigan execution lease. Run it only during a controlled appliance qualification window. Pause other work sources, select idle DevKits, and do not start another scheduler or manual workload until the harness has stopped and cleanup has completed.

Host and Network Requirements

Use a test computer with:

macOS on Apple Silicon or Intel, or Linux on amd64 or arm64;
a 2.5 GbE interface connected to the same switch as the DevKits;
sima-cli on PATH;
access to the artifact service used by sima-cli;
reachability to the Fleet Manager REST API and each selected DevKit's SSH management address;
enough firewall access for the flows below.

Direction	Protocol and port	Purpose
Test computer → Fleet Manager	HTTP or HTTPS on the configured API port	Inventory, eligibility, and metric history
Test computer → DevKit	SSH	Bootstrap, workload launch, logs, and cleanup
DevKit → test computer	RTSP over TCP, default port 18554	Reference H.264 input
DevKit → test computer	UDP, 96 ports per DevKit	48 returned video streams and 48 metadata streams

At the default 1 Mbps source rate, eight DevKits produce about 384 Mbps from the host to the DevKits and about 384 Mbps back to the host, before protocol overhead. A 2.5 GbE host link provides useful headroom for the combined traffic and for monitoring.

On a computer with both Wi-Fi and wired Ethernet, keep corporate or artifact access on either route, but make sure the DevKit RTSP and UDP addresses resolve to the wired interface. During a full run, the harness selects the interface that can reach the chosen DevKit and prints the resolved RTSP URL before launch. Source-only mode has no selected DevKit from which to infer a nonstandard subnet; configure its wired host address explicitly as described below.

Prepare the Appliance

Before running the test:

Finish Kerrigan installation and DevKit enrollment.
Confirm that every test DevKit has a recent heartbeat and a reachable management address in Fleet Manager.
End maintenance mode for the devices you intend to test.
Stop or drain CI runners, SDK sessions, and other workloads on those DevKits.
Confirm that the active CVU firmware matches the Core release you intend to test. The harness checks this again before starting the pipeline.
Record the appliance build, Core reference, DevKit firmware, switch configuration, ambient conditions, and selected device IDs with the test result.

The harness selects a compatibility profile from the platform version reported by each selected DevKit. All selected DevKits must use one supported release. Platform 2.1.3 uses core@v0.5.0, a real 9 FPS RTSP cadence, and a 400 FPS aggregate target. Platform 3.0 uses core@develop, the original 30 FPS stream-copy source, and a 600 FPS aggregate target. If Fleet Manager does not report the version, the interactive harness asks the operator, while unattended runs must set platform_version. Leave source.fps at 0 for automatic selection. An explicitly configured incompatible Core target or source cadence is rejected before the harness changes a DevKit.

The harness installs the compatible Core package when PyNeat is absent or does not match the release marker. It deliberately does not activate new firmware. Firmware activation is a separate maintenance action because it can disrupt other device state.

Install the Harness

Install the harness package from the Kerrigan branch or release being qualified. Set KERRIGAN_REF to that Git reference:

export KERRIGAN_REF=develop

Choose the package type that matches the test computer:

# Apple Silicon Mac
sima-cli neat install --type darwin-arm64 \
  "kerrigan/appliance-test-harness@${KERRIGAN_REF}:latest"

# Intel Mac
sima-cli neat install --type darwin-amd64 \
  "kerrigan/appliance-test-harness@${KERRIGAN_REF}:latest"

# Linux x86-64
sima-cli neat install --type linux-amd64 \
  "kerrigan/appliance-test-harness@${KERRIGAN_REF}:latest"

# Linux ARM64
sima-cli neat install --type linux-arm64 \
  "kerrigan/appliance-test-harness@${KERRIGAN_REF}:latest"

The installer places the command in ${XDG_BIN_HOME:-$HOME/.local/bin} and the versioned configuration, reference media, workload, and scripts under ${XDG_DATA_HOME:-$HOME/.local/share}/sima-neat/appliance-test-harness. It also installs the packaged, version-pinned MediaMTX binary. When FFmpeg is not already available, the installer installs it with Homebrew on macOS or the host package manager on Linux. A Linux host may prompt for sudo; on macOS, Homebrew must already be available when FFmpeg is missing. Reinstalling the harness reuses a working FFmpeg installation.

Verify the installation:

appliance-test-harness --version
ffmpeg -version

Validate the Reference Source

Test the source before involving a DevKit:

appliance-test-harness --source-only

The command starts MediaMTX, loops the packaged reference clip with FFmpeg stream copy, verifies that the RTSP path is readable, and prints both the local and DevKit-facing URLs. Leave it running and validate it from another terminal:

ffprobe -v error -rtsp_transport tcp \
  -show_entries stream=codec_name,width,height,avg_frame_rate \
  -of default=noprint_wrappers=1 \
  rtsp://127.0.0.1:18554/reference

For a visual check:

ffplay -rtsp_transport tcp -fflags nobuffer -flags low_delay \
  rtsp://127.0.0.1:18554/reference

Expect H.264, 1280×720, and 30 FPS. Stop source-only mode with Ctrl+C before starting the full harness; only one harness instance runs for each host user.

On the conventional appliance DevKit LAN, source-only mode resolves {host} from the known subnet. On any other subnet, copy the installed example configuration and replace both host templates with the test computer's wired DevKit-LAN address:

{
  "source": {
    "devkit_url": "rtsp://HOST_LAN_ADDRESS:18554/reference"
  },
  "devkit_control": {
    "receiver_host": "HOST_LAN_ADDRESS"
  }
}

These are excerpts; retain the other fields from the complete example. Then pass the edited file:

appliance-test-harness \
  --config ./appliance-test-harness.json \
  --source-only

Replace HOST_LAN_ADDRESS with the wired address before running the command. Both fields must be explicit because the default receiver setting also contains {host}, even though source-only mode does not start a receiver. Passing --kerrigan-url does not select this address because source-only mode does not connect to Fleet Manager.

Run an Interactive Qualification

Start the installed command:

appliance-test-harness

The interactive flow:

probes the conventional Fleet Manager location and sends discovery requests over every active multicast-capable IPv4 interface;
tries the inventory and metric-history endpoints with anonymous read-only access;
prompts for Fleet Manager credentials only when read access is protected;
lists healthy, online, idle, non-maintenance DevKits;
asks which DevKits to test;
verifies that the selected DevKits use one supported platform release and asks for the release when Fleet Manager does not report it;
selects the compatible Core package and performance target;
binds all UDP receivers before starting any DevKit workload;
starts and verifies the RTSP source;
revalidates DevKit eligibility;
downloads or reuses the validated YOLO26 model on the host;
bootstraps and starts all selected DevKits concurrently;
waits for every pipeline to report Graph ready;
opens the monitoring service.

The first run can take several minutes while the model is downloaded, Core is installed, and each 48-channel graph is created. The monitoring URL is printed only after every selected graph is ready. Do not interpret this startup period as a web-server failure.

Open the printed monitoring URL in a browser. With the default configuration it is:

http://localhost:8088

Run one DevKit first. When all 48 channels are stable, stop the run and repeat with all appliance DevKits selected.

Run Without Prompts

The installed example is a complete JSON configuration. Copy it instead of constructing a partial file:

export HARNESS_HOME="${XDG_DATA_HOME:-$HOME/.local/share}/sima-neat/appliance-test-harness/current"
cp "${HARNESS_HOME}/examples/config.json" ./appliance-test-harness.json

Edit the copy:

set kerrigan_url or export KERRIGAN_SERVER_URL;
set platform_version when Fleet Manager does not report current_version for every selected DevKit;
list every exact Fleet Manager target ID under devices;
give each device a unique, non-overlapping video and metadata port range;
do not override core_target or core_version_marker with a build that is incompatible with platform_version;
adjust duration_seconds for the planned soak;
change golden boxes only when using a different approved reference image or detector configuration.

The device list uses Fleet Manager target IDs, not display names:

"devices": [
  {
    "id": "TARGET_ID_1",
    "video_port_base": 10000,
    "metadata_port_base": 10048
  },
  {
    "id": "TARGET_ID_2",
    "video_port_base": 10100,
    "metadata_port_base": 10148
  }
]

For 48 channels, reserve 100 ports per device. The first 48 ports carry video, the next 48 carry metadata, and the four-port gap separates adjacent device blocks. For example, use video base 10000 and metadata base 10048 for the first device, then bases 10100 and 10148 for the second. These are UDP port numbers, not network addresses.

Provide credentials through environment variables, never in the JSON file:

export KERRIGAN_SERVER_URL="http://FLEET_MANAGER_HOST:API_PORT"
export KERRIGAN_SESSION_TOKEN="SESSION_TOKEN_IF_REQUIRED"
export KERRIGAN_DEVKIT_PASSWORD="DEVKIT_SSH_PASSWORD"

appliance-test-harness \
  --config ./appliance-test-harness.json \
  --non-interactive

The session token is necessary only when Fleet Manager protects read access. Remove the variables from the shell after the run.

Read the Dashboard

Each DevKit is one compact card. Its five rows contain one cell per channel:

Row	Green	Yellow	Red
Video FPS	At least 96% of the configured per-channel target	At least 5 FPS but below green	Below 5 FPS, or metadata arrives without returned video
Mbps	75%–105% of the configured source bitrate	At least 20% but outside the green range	Below 20%
Metadata	At least 96% of the configured per-channel target	At least 5 records/s but below green	Below 5 records/s
Accuracy	At least 90%	70% to below 90%	Below 70%
Loss	At most 0.1%	Above 0.1% and at most 1%	Above 1%

A gray cell is waiting for its first sample. Hover over or keyboard-focus a cell to see its numeric value. If any metric is red, the device card reports the number of critical channels. Otherwise it reports degraded, waiting, or nominal.

Hovering a channel also opens a preview. The image is decoded on demand from an independently decodable H.264 keyframe actually received on that channel; it is not a copy of the source image. Current detection metadata is drawn over that frame. No preview image is shown until returned RTP contains a complete keyframe.

The temperature and power values are averages from the latest one-minute bucket in Kerrigan's metric history. A missing value means that the metric was unavailable; it does not mean zero. Do not treat the displayed power as a transient peak.

Decide Whether the Appliance Passes

Allow the graph and five-second measurement windows to stabilize before making a decision. A useful qualification record includes:

harness, Kerrigan, Core, firmware, and model versions;
reference-video and model checksums;
selected DevKit IDs and run duration;
ambient and airflow conditions;
the highest displayed one-minute temperature and board-rail power averages;
separate raw Sentinel or external-meter peaks when peak power is required;
minimum and aggregate video FPS;
minimum and aggregate metadata rate;
detection accuracy and RTP loss for all channels;
any interval in which a channel was critical, waiting, or starved;
the harness log and retained DevKit workload logs.

Treat the run as invalid if the host NIC, switch path, or receiver is saturated. Do not attribute host-side packet drops to the DevKit. A full appliance passes only when every intended DevKit participates and all channels remain within the quality limits selected for that appliance build.

Stop and Verify Cleanup

Stop the harness with Ctrl+C. It signals only the processes whose command lines match the run-specific staged workload, removes each transferred model, stops the local source and receivers, and logs out a Fleet Manager session that it created.

Confirm after the run that:

the harness reports no cleanup error;
no test workload remains on a selected DevKit;
receiver and RTSP ports are free;
Fleet Manager again reports every device as eligible and idle;
CI runners or other drained work sources are restored only after those checks pass.

Workload logs remain under the configured remote root on each DevKit so a failed run has recoverable evidence.

Troubleshooting

Fleet Manager is not discovered

Confirm that the wired adapter has an IPv4 address and is active.
Check the logged interface list; USB Ethernet must appear there.
Pass --kerrigan-url or set KERRIGAN_SERVER_URL when discovery is blocked or the appliance uses a nonstandard topology.
Verify the API in a browser or with an approved read-only request before retrying.

The RTSP connection is refused

Run source-only mode and wait for its readiness message.
Reinstall the harness if its packaged MediaMTX binary is missing. The installer also verifies that FFmpeg and FFprobe are available.
Use the DevKit-facing URL printed by the harness, not the loopback URL, from a DevKit.
Check host firewall rules for the RTSP listener.

Startup reports a Core or firmware mismatch

Do not bypass the check. Activate the firmware supplied by the selected Core package during an idle maintenance window, reboot if the platform procedure requires it, then rerun the harness. The error message prints the applicable activation command.

Some channels have metadata but no video

This is a returned-video failure, even if a source preview looks correct. Check the red Video FPS cells, RTP loss, host UDP firewall, assigned port ranges, and the DevKit workload log. The browser preview intentionally remains empty when that channel has not returned a complete keyframe.

Metadata arrives but accuracy is low

Verify that the configured model is the raw-output YOLO26 package expected by the workload.
Verify the reference clip and golden-box configuration.
Hover a channel and compare the actual returned frame with the metadata overlay.
Confirm that Core, active firmware, model, and workload belong to the same qualified release combination.

A second run reports ports already in use

Wait for the first harness to finish cleanup. Starting a new instance requests shutdown of the previous instance, but DevKit and local listener cleanup must complete before the replacement can bind its ports. If the previous run crashed, inspect processes and workload logs before removing only the stale harness-owned process.

What the Harness Tests​

Architecture​

Host and Network Requirements​

Prepare the Appliance​

Install the Harness​

Validate the Reference Source​

Run an Interactive Qualification​

Run Without Prompts​

Read the Dashboard​

Decide Whether the Appliance Passes​

Stop and Verify Cleanup​

Troubleshooting​

Fleet Manager is not discovered​

The RTSP connection is refused​

Startup reports a Core or firmware mismatch​

Some channels have metadata but no video​

Metadata arrives but accuracy is low​

A second run reports ports already in use​

Related Information​