Skip to main content

Kerrigan Test Manifest Guide

Kerrigan test runners use a YAML manifest to describe what should run on the device fleet. In GitHub Actions, the workflow usually points the runner bridge at a checked-in manifest:

jobs:
  kerrigan:
    runs-on: [self-hosted, kerrigan]
    env:
      KERRIGAN_MANIFEST: .kerrigan/ctest-job.yaml
    steps:
      - uses: actions/checkout@v4
      - run: kerrigan-runner-bridge --timeout-sec 1800

Prefer static manifests for normal CI. Dynamic manifest generation is useful for ad hoc experiments, but checked-in YAML is easier to review, reproduce, and reuse across branches.

For label-based resource selection details, see resource selection. This guide describes the current manifest schema, including labels and tags. For an exhaustive field-by-field schema, see the Workflow spec reference.

File Shape​

metadata:
  name: core-ctest-smoke
  labels:
    suite: core
    mode: ctest

source:
  repository: sima-neat/core
  artifacts:
    - name: ctest-runtime
      url: https://example.com/artifacts/ctest-runtime.tar.gz
      sha256: "<sha256>"
      extract: true
      destination: .

setup:
  per_devkit:
    - id: prepare-runtime
      commands:
        - echo "preparing ${KERRIGAN_TARGET_ID}"
        - ./install-runtime.sh
      timeout_sec: 300

scheduling:
  strategy: spread
  min_devkits: 0
  max_devkits: 4
  max_parallel: 4
  fail_fast: false
  pools:
    - sim-pool
  capabilities:
    board: modalix
    os_type: elxr
    firmware: 2.1.0

tasks:
  - id: ctest-core
    type: ctest
    test_dir: ctest-runtime/build/tests
    selectors:
      - ctest_shard_a
      - ctest_shard_b
      - ctest_shard_c
    timeout_sec: 120

Metadata​

metadata.name gives the session a readable name. metadata.labels are for classification and reporting; they do not select devices.

metadata:
  name: apps-regression
  labels:
    suite: apps
    priority: nightly

Source​

source tells device agents how to materialize the task workspace.

Use source.artifacts for realistic DevKit testing where agents download a prebuilt runtime or test package:

source:
  artifacts:
    - name: ctest-runtime
      url: https://shanguangzai.github.io/kerrigan-dispatcher-test/artifacts/ctest-runtime.tar.gz
      sha256: 8cb4cf03f3c7bd7052ed12067c36a9c98a7ed6587ecba5f1db2952bf4aeccdca
      extract: true
      destination: .

Fields:

  • name: local artifact name used in logs and temporary files.
  • url: downloadable URL. file:// is supported by the simulation stack.
  • sha256: optional but recommended. Agents verify it before execution.
  • extract: when true, extract the tarball into destination.
  • destination: workspace-relative destination. Absolute paths and .. are rejected.

Use source.bundle for small repository files, such as Python test fixtures:

source:
  bundle:
    include:
      - tests/**/*.py
    exclude:
      - tests/experimental/**

Bundles are intended for small files. Large runtimes, binaries, SDKs, and CTest trees should be hosted as artifacts.

Use source.vulcan_artifacts when the runtime is published through Neat CICD Infrastructure (Vulcan) and should be installed uniformly on every selected devkit:

source:
  vulcan_artifacts:
    - env: stg
      name: core
      branch: develop
      tag: latest
      type: minimal
      install_dir: /tmp/kerrigan-vulcan-core-develop
      skip_devkit_sync: true
      timeout_sec: 300

Fields:

  • env: Neat CICD Infrastructure environment, such as dev, stg, or prd.
  • name: artifact name, such as core.
  • branch: artifact branch, such as main.
  • tag: artifact tag, such as latest or a git hash.
  • type: optional Neat CICD Infrastructure install type passed as sima-cli neat install --type. Use minimal for smoke tests when the full artifact has optional resource prompts.
  • install_dir: optional install directory passed as --install-dir.
  • base_url: optional explicit artifact base URL passed as --base-url.
  • force: optional boolean that passes --force.
  • skip_devkit_sync: optional boolean that sets NEAT_INSTALLER_SKIP_DEVKIT_SYNC=ON for this install. This is useful in SDK or simulation containers where a paired physical DevKit may be visible but the test should only validate artifact retrieval/install in the agent container.
  • timeout_sec: optional install timeout in seconds. The default is 900.

The device agent installs each Neat CICD Infrastructure artifact before running setup or test commands:

sima-cli neat install --type minimal \
  --install-dir /tmp/kerrigan-vulcan-core-develop core@develop:latest

This is useful for real DevKit workflows where the test package or runtime should be installed through the same path a developer would use manually. Kerrigan caches the install per devkit agent process, so repeated tasks on the same devkit do not reinstall the same env/name/branch/tag tuple.

Setup​

setup.per_devkit runs once on every selected devkit before normal tasks start. Use it to prepare runtime state, mount paths, start background services, or validate device readiness.

setup:
  per_devkit:
    - id: prepare-runtime
      commands:
        - export PATH="/opt/runtime/bin:$PATH"
        - ./scripts/start-helper.sh &
        - ./scripts/check-device-ready.sh
      timeout_sec: 300
      output_idle_timeout_sec: 60

The command list is executed by the agent as one shell command joined with &&. Shell syntax such as &&, ;, and & is available. If setup fails on any selected devkit, Kerrigan fails the session and skips remaining test tasks. Use output_idle_timeout_sec for setup steps that should continuously report progress, such as package downloads or model retrieval. This catches accidental interactive prompts, stalled network downloads, or tools waiting for login. Device-agent commands run with stdin closed and CI=true, so workflows should not depend on interactive input.

Teardown​

teardown.per_devkit runs once on every selected devkit after normal tasks have reached a terminal state. Use it for cleanup that must happen on every device, such as removing temporary MPK extraction directories, stopping helper services, or collecting final disk usage. Unlike a normal task, teardown is not spread to one available devkit; Kerrigan fans it out across the devkits selected for the session.

teardown:
  per_devkit:
    - id: cleanup-runtime
      shell: |
        rm -rf /tmp/sima-mpk-kerrigan-core-ci*
        find /tmp -maxdepth 1 -name 'sima_gst_registry_*.bin' -delete
      timeout_sec: 120

Teardown still runs after test failures, so the session can preserve failure evidence in the task summary while returning devices to a clean state. A teardown failure is reported as its own task failure in the final session summary.

Scheduling​

scheduling controls admission, fanout, and device matching.

scheduling:
  strategy: spread
  min_devkits: 2
  max_devkits: 4
  max_parallel: 4
  fail_fast: false
  pools:
    - sim-pool
  capabilities:
    board: modalix
    os_type: elxr
  labels:
    kerrigan.pool_id: sdk-2-2-som
    sdk: 2.2.0
  tags:
    - 2.2.0-som-devkits

Fields:

  • strategy: spread, pack, or serial.
  • min_devkits: minimum eligible devkits required before dispatch starts.
  • max_devkits: maximum distinct devkits the session may use.
  • max_parallel: maximum concurrent task assignments.
  • fail_fast: when true, queued work is cancelled after a test failure.
  • pools: eligible device pools.
  • capabilities: device capability requirements.
  • labels: exact-match resource labels. Kerrigan provides implicit labels such as kerrigan.pool_id, kerrigan.target_id, and kerrigan.board.
  • tags: shorthand selectors equivalent to tag.<name>: true.

min_devkits: 0 means "do not require a minimum before starting." Kerrigan will run with whatever eligible capacity is available, capped by max_devkits. It still cannot run tasks until at least one eligible devkit is available.

Examples:

# Fallback-friendly: use up to four, but start with one if that is all we have.
scheduling:
  min_devkits: 0
  max_devkits: 4
# Strict fanout: wait until at least two eligible devkits are available.
scheduling:
  min_devkits: 2
  max_devkits: 4
# Use at most four devkits, but only run two tasks at a time.
scheduling:
  max_devkits: 4
  max_parallel: 2

Scheduling Strategy​

strategy controls how Kerrigan chooses among eligible devkits when assigning ready tasks. It does not change which tasks exist; selectors and task definitions control that. It also does not override min_devkits, max_devkits, or max_parallel.

spread​

spread balances work across the selected devkits. It prefers the eligible devkit with the fewest assignments so far.

Use spread for normal parallel test execution:

scheduling:
  strategy: spread
  min_devkits: 0
  max_devkits: 4
  max_parallel: 4

With six ready shards and four available devkits, the first wave uses four devkits. As devkits finish, remaining shards are assigned to the next available devkit while keeping assignment counts as balanced as possible.

Use this for:

  • CTest shards.
  • Pytest shards.
  • Independent smoke/regression tests.
  • Any workload where shorter wall-clock time is the goal.

pack​

pack prefers reusing already-selected devkits before expanding to more devkits. It keeps the session footprint smaller when full fanout is not needed.

scheduling:
  strategy: pack
  min_devkits: 0
  max_devkits: 4
  max_parallel: 2

With two concurrent slots, pack tries to keep work concentrated on the same small set of devkits instead of spreading assignments across all four. This can reduce setup cost, cache misses, device churn, or contention with other users.

Use this for:

  • Tests with expensive per-device setup.
  • Workloads that benefit from warm caches or reused state.
  • Low-priority jobs where minimizing fleet footprint matters more than perfect distribution.

pack still respects max_parallel. If max_parallel: 2, it will not run more than two tasks at the same time.

serial​

serial runs at most one task at a time. It is equivalent to a conservative single-lane schedule even if multiple devkits are available.

scheduling:
  strategy: serial
  max_devkits: 1
  max_parallel: 1

Use this for:

  • Tests that mutate shared lab state.
  • Firmware update or recovery flows.
  • Debug workflows where ordered logs matter.
  • Any suite that is not safe to parallelize yet.

For clarity, pair serial with max_parallel: 1. If you need strict use of a specific kind of device, express that with pools and capabilities.

Tasks​

Selectors define the partial test scope assigned to one devkit. Good selectors are the main difference between "Kerrigan can distribute this suite" and "one devkit has to run everything."

Guidelines:

  • Use selectors that are independently runnable.
  • Keep selector runtimes reasonably balanced.
  • Prefer many medium-sized selectors over one giant selector.
  • Avoid selectors that depend on execution order unless dependencies are explicitly modeled.
  • Make selectors stable names, not line numbers or temporary generated IDs.

CTest​

Use type: ctest when the artifact contains a CTest tree. selectors are expanded into independently schedulable tasks:

tasks:
  - id: ctest-core
    type: ctest
    test_dir: ctest-runtime/build/tests
    selectors:
      - ctest_shard_a
      - ctest_shard_b
    env:
      KERRIGAN_CTEST_SHARD_DURATION_SEC: "10"
    timeout_sec: 120

Kerrigan runs each selector as:

ctest --test-dir ctest-runtime/build/tests --output-on-failure -R '^ctest_shard_a$'

Each selector is treated as an exact CTest test name. Kerrigan wraps it as ^name$ for ctest -R, so ctest_shard_a matches only ctest_shard_a, not ctest_shard_a_extra.

You can inspect the names available in a CTest package with:

ctest --test-dir ctest-runtime/build/tests -N

If that prints:

Test #1: ctest_shard_a
Test #2: ctest_shard_b
Test #3: ctest_shard_c

then the manifest selectors should be:

tasks:
  - id: ctest-core
    type: ctest
    test_dir: ctest-runtime/build/tests
    selectors:
      - ctest_shard_a
      - ctest_shard_b
      - ctest_shard_c

For real CTest suites, structure the package so partial execution is natural. For example, define separate CTest tests by shard, suite, model, board mode, or test binary:

add_test(NAME core_graph_smoke COMMAND core_tests --suite graph_smoke)
add_test(NAME core_tensor_smoke COMMAND core_tests --suite tensor_smoke)
add_test(NAME core_runtime_smoke COMMAND core_tests --suite runtime_smoke)

Then select those names:

selectors:
  - core_graph_smoke
  - core_tensor_smoke
  - core_runtime_smoke

A single monolithic CTest target cannot be evenly distributed across devkits. If ctest -N shows only one long test, Kerrigan can only assign that one test to one devkit. Split the CTest package first, then list the split names in selectors.

Use selector instead of selectors when there is only one CTest name:

tasks:
  - id: ctest-one-suite
    type: ctest
    test_dir: ctest-runtime/build/tests
    selector: core_graph_smoke

Pytest​

Use type: pytest for bundled Python tests:

tasks:
  - id: smoke-a
    type: pytest
    selector: tests/test_slow_suite.py::test_manual_shard_a
    timeout_sec: 60

The runner bridge normalizes this to a command similar to:

pytest -q -s tests/test_slow_suite.py::test_manual_shard_a

Pytest selectors are normal pytest node IDs. Common forms are:

# Entire file
selector: tests/test_runtime.py

# One test function
selector: tests/test_runtime.py::test_boot

# One class
selector: tests/test_runtime.py::TestRuntime

# One test method
selector: tests/test_runtime.py::TestRuntime::test_boot

For sharding, create one task per selector:

tasks:
  - id: pytest-runtime-a
    type: pytest
    selector: tests/test_runtime.py::test_runtime_shard_a
    timeout_sec: 120

  - id: pytest-runtime-b
    type: pytest
    selector: tests/test_runtime.py::test_runtime_shard_b
    timeout_sec: 120

Use pytest selectors when the tests are small enough to bundle from the repo. For larger test assets or compiled binaries, prefer artifact-backed CTest or an explicit command task that downloads and runs a package.

Command​

Use type: command for explicit commands:

tasks:
  - id: custom-smoke
    type: command
    command:
      argv:
        - ./run-smoke.sh
        - --target
        - "${KERRIGAN_TARGET_ID}"
      env:
        LOG_LEVEL: debug
      cwd: ctest-runtime
    timeout_sec: 300

The device agent injects KERRIGAN_TARGET_ID for the assigned devkit.

Requirements Per Task​

Tasks can override or refine global scheduling requirements:

tasks:
  - id: firmware-upgrade
    type: command
    command:
      argv: ["./upgrade-fw.sh"]
    requirements:
      pools:
        - lab-a
      capabilities:
        board: modalix
        role: controller
      labels:
        fixture: smoke
      tags:
        - modalix

Use this when only some tasks need special hardware. Keep common requirements in scheduling.capabilities or scheduling.labels.

Retries​

Retries can distinguish test failures from infrastructure failures:

tasks:
  - id: flaky-link-test
    type: command
    command:
      argv: ["./run-link-test.sh"]
    retries:
      test_failure: 0
      infra_failure: 1

Use infrastructure retries for device disconnects, artifact fetch failures, or other non-test failures. Avoid retrying deterministic test failures by default.

GitHub Log Expectations​

The runner bridge prints session progress into GitHub Actions logs. Artifact retrieval is reported by device agents through the control plane:

runner-bridge: progress #... ctest-core-ctest-shard-a target=sim-devkit-01: downloading artifact ctest-runtime from https://...
runner-bridge: progress #... ctest-core-ctest-shard-a target=sim-devkit-01: downloaded artifact ctest-runtime (... bytes)
runner-bridge: progress #... ctest-core-ctest-shard-a target=sim-devkit-01: verified artifact ctest-runtime sha256 ...
runner-bridge: progress #... ctest-core-ctest-shard-a target=sim-devkit-01: extracted artifact ctest-runtime to .

Task output includes timestamp and devkit:

runner-bridge: [2026-05-27T10:11:12.345-07:00] [sim-devkit-01] | Test project /tmp/...

Recommendations​

  • Put stable manifests under .kerrigan/.
  • Prefer artifact-backed CTest packages for realistic DevKit testing.
  • Keep test tasks independently selectable so Kerrigan can shard them.
  • Use min_devkits: 0 for smoke tests that should run with whatever is free.
  • Use min_devkits >= 2 when the test result is only meaningful with fanout.
  • Always specify max_devkits to bound fleet usage.
  • Always set task timeouts.
  • Include sha256 for downloadable artifacts.
  • Keep setup idempotent; it may run once per selected devkit.