cargo-ktstr

cargo ktstr is a cargo plugin for kernel build, cache, and test workflow. Subcommands in --help order: test (alias: nextest), coverage, llvm-cov, stats, kernel, model, verifier, funify (alias: costume), completions, show-host, show-thresholds, export, locks, shell.

test

Build the kernel (if needed) and run tests via cargo nextest run. Also available as cargo ktstr nextest — a visible clap alias that expands to the same subcommand, so the two forms are interchangeable.

cargo ktstr test                                               # auto-discover kernel
cargo ktstr test --kernel ../linux                             # local source tree
cargo ktstr test --kernel 6.14.2                               # version (auto-downloads on miss)
cargo ktstr test --kernel 6.14.2-tarball-x86_64-kc...          # cache key (from kernel list)
cargo ktstr test --kernel 6.12..6.14                           # range: every stable+longterm release in [6.12, 6.14]
cargo ktstr test --kernel git+https://example.com/r.git#v6.14  # git URL + ref (tag/branch)
cargo ktstr test --kernel git+https://example.com/r.git#deadbeef1234  # specific commit
cargo ktstr test --kernel 6.14.2 --kernel 6.15.0               # multi-kernel: repeatable
cargo ktstr test --release                                     # release profile (stricter assertions)

--kernel is repeatable and accepts a path, version string, cache key, version range (START..END), or git source (git+URL#REF). When absent, the test framework discovers a kernel from KTSTR_TEST_KERNEL, then KTSTR_KERNEL, then falls back to cache and filesystem lookup. When --kernel is a path, cargo-ktstr configures and builds the kernel before running tests. Version strings auto-download and build on cache miss (both explicit patch versions like 6.14.2 and major.minor prefixes like 6.14). Cache keys resolve from the cache only — they error if not cached (run cargo ktstr kernel list to see available keys).

Ranges (START..END) expand against kernel.org’s releases.json to every stable and longterm release whose version sits inside [START, END] inclusive (mainline / linux-next rows are dropped). The endpoints themselves do NOT need to appear in releases.json — 6.10..6.16 brackets the surviving releases even if 6.10 and 6.16 have aged out.

Git sources (git+URL#REF) clone the repo shallow at the given ref, build, and cache the result. A repeat invocation against an unchanged branch tip lands a cache hit; a moved tip rebuilds.

Multi-kernel: kernel as a gauntlet dimension

When --kernel resolves to two or more kernels (multiple --kernel flags, or a single --kernel START..END range that expands to several releases), cargo-ktstr resolves all kernels upfront and exports the resolved set to cargo nextest via the KTSTR_KERNEL_LIST env var. The test binary’s gauntlet expansion adds the kernel as an additional dimension to the gauntlet cross-product, so each (test × scenario × topology × kernel) tuple becomes a distinct nextest test case. Two name shapes carry the kernel suffix:

• Base tests: ktstr/{name}/{kernel_label} — one variant per registered #[ktstr_test] per kernel.
• Gauntlet variants: gauntlet/{name}/{preset}/{kernel_label} — one variant per (test × topology preset × kernel).

Single-kernel runs (zero or one resolved kernel) keep the historical name shapes ktstr/{name} and gauntlet/{name}/{preset} with no kernel suffix, so existing CI baselines and per-test config overrides keep matching.

Kernel labels are semantic, operator-readable identifiers sanitized to kernel_[a-z0-9_]+:

• Version / range expansion → kernel_6_14_2, kernel_6_15_rc3
• Cache key → version prefix only (kernel_6_14_2 from 6.14.2-tarball-x86_64-kc<hash>)
• Git source → kernel_git_{owner}_{repo}_{ref} (e.g. kernel_git_tj_sched_ext_for_next from git+https://github.com/tj/sched_ext#for-next)
• Path → kernel_path_{basename}_{hash6} (e.g. kernel_path_linux_a3f2b1); the 6-char crc32 of the canonical path disambiguates two linux directories under different parents. Dirty-tree builds (uncommitted source changes, mid-build worktree mutations, or non-git trees) append _dirty to the label — e.g. kernel_path_linux_a3f2b1_dirty — so the test report distinguishes the non-reproducible run from a subsequent clean rebuild of the same path.
• Local cache entry → kernel_local_{hash6} (first 6 chars of the source tree’s git short_hash, captured at cache-store time) or kernel_local_unknown for non-git trees. The hash6 keeps two distinct local trees from collapsing to the same label; the unknown literal is the shared bucket for every non-git tree (no discriminator exists at the cache layer to spread them apart).

Filter with nextest’s -E 'test(kernel_6_14)' to pick a single kernel from a multi-kernel matrix; nextest’s parallelism, retries, and --ignored flag all apply natively. Sidecars partition per kernel: each kernel runs in its own target/ktstr/{kernel}-{project_commit}/ directory keyed on the resolved kernel’s identity and the project tree’s HEAD short hex (with -dirty suffix when the worktree differs). Coverage profraw does NOT partition per kernel — __llvm_profile_write_buffer writes flat into target/llvm-cov-target/ with PID-keyed filenames (ktstr-test-{pid}-{counter}.profraw), and cargo-llvm-cov merges every variant’s profraw automatically into the single output report.

Build / download / clone failures abort BEFORE any test runs — a missing kernel can’t be tested, and continuing would mask which kernel was requested-but-unavailable in the operator-visible error stream. Test failures within a kernel are nextest-handled normally.

host_only tests under multi-kernel: tests marked host_only (those that run on the host without booting a VM) skip the kernel suffix and list / run once regardless of KTSTR_KERNEL_LIST cardinality. The dispatch sites (list_tests, list_tests_budget, and --exact’s run_host_only_test in src/test_support/dispatch.rs) all gate on entry.host_only before consulting the resolved kernel set, so a host-side test never observes the kernel directory and multiplying it across kernels would just run N copies of identical work for no signal.

What it does (path mode only)

These steps run only when --kernel is a source directory path. Cached version and cache-key identifiers skip straight to test execution (step 6); uncached version identifiers run through download + configure + build + cache-store first. Ranges fan out to per-version resolution (every release downloads + builds + caches independently if not already present); git sources clone shallow at the ref, build, and cache. Multi-kernel resolution finishes for every requested kernel BEFORE step 6 — the cargo-nextest invocation in step 6 sees the complete kernel set as a single KTSTR_KERNEL_LIST export, so nextest fans the gauntlet across kernels in a single run.

For path mode, the source tree is gix-discovered and classified as either clean (HEAD reachable, index matches HEAD, worktree matches index) or dirty or non-git (any tracked-file diff, or the directory is not a git repo at all). The cache is keyed in one of three shapes:

• local-{hash7}-{arch}-kc{suffix} — clean git tree, no user .config file in the source tree yet (build will run make defconfig). {hash7} is the source tree’s HEAD short hash; {suffix} distinguishes ktstr framework kconfig fragments.
• local-{hash7}-{arch}-cfg{user_config}-kc{suffix} — clean git tree with a user .config whose CRC32 hash discriminates distinct configurations against the same commit, so iterative .config edits at a fixed commit populate distinct cache entries instead of colliding.
• local-unknown-{path_hash}-{arch}-kc{suffix} — dirty / non-git tree (HEAD does not describe the source). {path_hash} is the full 8-char (32-bit) CRC32 of the canonical source path so two parallel cargo ktstr test --kernel ./linux-a and --kernel ./linux-b runs do not collide on the same local-unknown-... slot.

Dirty / non-git trees never cache — the build pipeline runs in the source directory, the kernel label gets a _dirty suffix, and a subsequent run of the same path that goes clean produces a distinct cache entry under the clean shape.

1. Source-tree validation — verifies <kernel>/Makefile and <kernel>/Kconfig both exist. If either is missing, bails with not a kernel source tree.
2. Cache lookup (clean trees only) — looks up the local-{hash7}-{arch}[-cfg{user_config}]-kc{suffix} key (the cfg segment present iff a user .config exists in the source tree). Cache hit short-circuits to step 6: cargo-ktstr exports the cache entry directory via KTSTR_KERNEL and emits a cargo ktstr: cache hit for {input_path} ({cache_key}, built {age} ago) line on stderr (the , built {age} ago suffix is omitted when the timestamp is unparseable or future-dated). Cache miss continues to step 3.
3. Auto-configure — if <kernel>/.config lacks the CONFIG_SCHED_CLASS_EXT=y sentinel, runs make defconfig (when no .config exists), appends ktstr.kconfig to .config, then runs make olddefconfig.
4. Kernel build — runs make -j$(nproc) KCFLAGS=-Wno-error, then runs validate_kernel_config to verify critical config options (CONFIG_SCHED_CLASS_EXT, CONFIG_DEBUG_INFO_BTF, CONFIG_BPF_SYSCALL, CONFIG_FTRACE, CONFIG_KPROBE_EVENTS, CONFIG_BPF_EVENTS) survived the build — the kernel build system silently disables options whose dependencies are not met, and the validator surfaces those failures with a per- option remediation hint. make handles the no-op case when the kernel is already built. For dirty / non-git trees this is the unconditional path; for clean trees, only reached on cache miss.
5. compile_commands.json + cache store — runs make compile_commands.json (skipped only for transient temp directories like extracted tarballs) so LSP / clangd work against the local tree. Then for clean trees, the kernel image + stripped vmlinux are persisted under the resolved local-{hash7}-{arch}[-cfg{user_config}]-kc{suffix} key with metadata.json recording the source tree path. A post-build re-check of the dirty state catches mid-build mutations (worktree edits or commits that happened during make) and skips the cache store on either signal so a racing-write build can not land under a stale identity. Dirty / non-git trees skip the cache store unconditionally (no stable HEAD identity for the cache key) but still get compile_commands.json.
6. Test execution — execs cargo nextest run once with KTSTR_KERNEL set in the environment (single-kernel) or with both KTSTR_KERNEL and KTSTR_KERNEL_LIST (multi-kernel; the latter encodes the resolved kernel set as label1=path1;label2=path2;…). For clean Path-spec resolution KTSTR_KERNEL points at the cache entry directory; for dirty or non-git trees it points at the source tree directly. The test binary’s gauntlet expansion adds the kernel as a fifth dimension when the list carries 2+ entries; nextest’s parallelism, retries, and -E filtering apply natively to every (test × kernel) variant.

Implicit vs explicit kernel discovery diverge: cargo ktstr test --kernel ../linux (explicit Path spec) routes through the cache pipeline above — the source tree is gix-classified, the local-{hash7}-{arch}[-cfg{user_config}]-kc{suffix} cache key is computed, the kernel is built (or short-circuited on cache hit), and the cache entry directory is exported via KTSTR_KERNEL. cargo ktstr test (no --kernel flag) does NOT run the build pipeline or produce a new cache entry. The test binary’s find_kernel chain reads existing cache entries (most-recent-valid first; entries built with a different kconfig fragment are skipped) and falls back to local build trees (./linux, ../linux) and host paths. Whatever pre-built image it finds is returned as-is — no cache key is computed for source trees discovered on the filesystem, no make is invoked, and the result does not land in the kernel cache for a future cache_key-keyed lookup. The KTSTR_KERNEL env var with a path value follows this same direct-image flow — the cache write path is reached only via the cargo ktstr --kernel argument (or via cargo ktstr kernel build --source ../linux as an explicit cache-populate step). Pass --kernel ../linux to opt into the cache pipeline so a clean tree’s build is stored once and reused on subsequent runs.

Passing nextest arguments

Arguments after test are passed through to cargo nextest run:

cargo ktstr test -- -E 'test(my_test)'        # nextest filter
cargo ktstr test -- --workspace               # all workspace tests
cargo ktstr test -- --retries 2               # nextest retries

coverage

Build the kernel (if needed) and run tests with coverage via cargo llvm-cov nextest. Same kernel resolution and multi-kernel semantics as test: --kernel is repeatable; multi-kernel runs add the kernel suffix to every test name and partition the sidecar tree per kernel via target/ktstr/{kernel}-{project_commit}/, where {project_commit} is the project HEAD short hex (with -dirty when the worktree differs). Coverage profraw lands flat in target/llvm-cov-target/ with PID-keyed filenames — it does NOT partition per kernel — and cargo-llvm-cov merges every variant’s profraw automatically into the single output report.

cargo ktstr coverage                                               # auto-discover kernel
cargo ktstr coverage --kernel ../linux                             # local source tree
cargo ktstr coverage --kernel 6.14.2                               # version (auto-downloads on miss)
cargo ktstr coverage --kernel 6.14.2 --kernel 6.15.0               # multi-kernel coverage matrix
cargo ktstr coverage --release                                     # release profile (stricter assertions)
cargo ktstr coverage -- --workspace --lcov --output-path lcov.info # lcov output

Requires cargo-llvm-cov and the llvm-tools-preview rustup component:

cargo install cargo-llvm-cov
rustup component add llvm-tools-preview

Passing arguments

Arguments after coverage are passed through to cargo llvm-cov nextest:

cargo ktstr coverage -- --workspace --profile ci --lcov --output-path lcov.info
cargo ktstr coverage -- --features integration

profraw layout

Three populations of *.profraw files arise from cargo ktstr runs. They land in different directories and are not all collected by the same workflow:

cargo ktstr test injects LLVM_PROFILE_FILE (added to prevent default.profraw leaking into a kernel source tree when the shell cwd was the kernel dir; see Stale vmlinux.btf or default.profraw). The resulting host-side default-{pid}-{binary_hash}.profraw files do NOT land in the target/llvm-cov-target/ directory that cargo ktstr coverage (cargo-llvm-cov) reads; they are NOT picked up by a later cargo ktstr coverage run unless you explicitly include them in a cargo llvm-cov report invocation pointed at the cargo-ktstr binary’s llvm-cov-target/ directory.

To clean accumulated profraw between runs:

# Remove ONLY *.profraw under target/llvm-cov-target/ (top-level glob, non-recursive):
cargo ktstr llvm-cov clean --profraw-only

# Drop host-side test-path profraw next to the cargo-ktstr binary.
# Run only the line(s) matching how cargo-ktstr was launched —
# the brace-list form is bash-only, so each path is its own command
# for portable POSIX shells (sh / dash):
rm -f target/debug/llvm-cov-target/default-*.profraw
rm -f target/release/llvm-cov-target/default-*.profraw

# If ktstr was installed via `cargo install`:
rm -f ~/.cargo/bin/llvm-cov-target/default-*.profraw

--profraw-only is the safe default: it removes only *.profraw files at the top level of target/llvm-cov-target/ (the cargo- llvm-cov-managed dir) and leaves coverage reports, profdata, and build artifacts intact. It does NOT touch the default-*.profraw files next to the cargo-ktstr binary (under target/{profile}/llvm-cov-target/ for cargo run / cargo build, or ~/.cargo/bin/llvm-cov-target/ for cargo install-deployed binaries) produced by the host-side injection — remove those with the explicit rm -f lines above for whichever launch mode you use. Avoid cargo ktstr llvm-cov clean without arguments (recursively wipes all of target/llvm-cov-target/, including reports) and --workspace (additionally runs cargo clean on workspace packages, removing build artifacts); both are destructive beyond profraw.

To opt out of the host-side LLVM_PROFILE_FILE injection entirely, export LLVM_PROFILE_FILE yourself before running cargo ktstr test — the injector only fires when the env is absent, so an explicit operator setting takes precedence.

llvm-cov

Raw passthrough to cargo llvm-cov with arbitrary arguments. Use this for llvm-cov subcommands that don’t fit the coverage flow — report, clean, show-env, etc. When you want cargo llvm-cov nextest, prefer cargo ktstr coverage; this subcommand carries the same kernel-resolution and --no-perf-mode plumbing but hands every remaining argument to cargo llvm-cov unchanged.

cargo ktstr llvm-cov report --lcov --output-path lcov.info    # generate report from prior run
cargo ktstr llvm-cov clean --workspace                         # wipe accumulated coverage data
cargo ktstr llvm-cov show-env                                  # print env cargo-llvm-cov would set
cargo ktstr llvm-cov --kernel ../linux report                  # pin kernel + passthrough

Note: a bare cargo ktstr llvm-cov (no trailing subcommand) dispatches to cargo llvm-cov, which runs cargo test — ktstr tests rely on the nextest harness for gauntlet expansion (topology-preset variants), verifier cell emission, and VM dispatch. Under bare cargo test, only the #[test] stubs run and gauntlet variants + verifier cells are silently skipped. Always pass a subcommand after llvm-cov (most often nextest, for which cargo ktstr coverage is the shorter route).

kernel

Manage cached kernel images. Three subcommands: list, build, clean. The standalone ktstr kernel subcommands are identical.

kernel list

List cached kernel images, sorted newest first. With --range, switches to PREVIEW MODE: prints the versions a START..END range expands to without performing any download or build.

cargo ktstr kernel list
cargo ktstr kernel list --json                    # JSON output for CI scripting
cargo ktstr kernel list --range 6.12..6.14        # preview range expansion
cargo ktstr kernel list --range 6.12..6.14 --json # preview as JSON

Default mode walks the local cache. Human-readable output shows key, version, source type, arch, and build timestamp. Entries built with a different ktstr.kconfig are marked (stale kconfig). Entries whose major.minor version is no longer in kernel.org’s active releases list are marked (EOL); prefix lookups for EOL series fall back to probing cdn.kernel.org for the latest patch release.

--range mode performs no cache reads: it fetches kernel.org’s releases.json once, expands the inclusive range against the stable and longterm releases (mainline / linux-next dropped), and prints one version per line on stdout. Use this to answer “what does --kernel 6.12..6.16 actually cover?” before paying the build cost — no kernel is downloaded or compiled. With --json, emits a JSON object carrying the literal range, the parsed start / end, and the expanded versions array.

kernel build

Download, build, and cache a kernel image. Three source modes: version (tarball download), --source (local tree), --git (clone).

cargo ktstr kernel build                               # latest stable from kernel.org
cargo ktstr kernel build 6.14.2                        # specific version
cargo ktstr kernel build 6.15-rc3                      # RC release
cargo ktstr kernel build 6.12                          # latest 6.12.x patch release
cargo ktstr kernel build --source ../linux             # local source tree
cargo ktstr kernel build --git URL --ref v6.14         # git clone (shallow, depth 1)
cargo ktstr kernel build --force 6.14.2                # rebuild even if cached

When no version or source is given, fetches the latest stable series that has had at least 8 maintenance releases — keeping CI off brand-new majors whose early builds are more likely to break — from kernel.org’s releases.json. A major.minor prefix (e.g. 6.12) resolves to the highest patch release in that series. For EOL series no longer in releases.json, probes cdn.kernel.org to find the latest available tarball. Skips building when a cached entry already exists (use --force to override). Stale entries (built with a different ktstr.kconfig) are rebuilt automatically. For --source, generates compile_commands.json for LSP support. Dirty local trees (uncommitted changes to tracked files) are built but not cached.

kernel clean

Remove cached kernel images.

cargo ktstr kernel clean                          # remove all (with confirmation prompt)
cargo ktstr kernel clean --keep 3                 # keep 3 most recent
cargo ktstr kernel clean --force                  # skip confirmation prompt
cargo ktstr kernel clean --corrupt-only --force   # remove only corrupt entries

model

Manage the LLM model cache used by OutputFormat::LlmExtract payloads. fetch downloads the default pinned model into the ktstr model cache; status reports whether a SHA-checked copy is already cached; clean deletes the cached artifact plus its warm-cache sidecar.

cargo ktstr model fetch                          # download + SHA-check (no-op if cached)
cargo ktstr model status                         # report cache path + verdict
cargo ktstr model clean                          # delete cached artifact + sidecar

fetch is a no-op when the cache already holds a SHA-checked copy. Respects KTSTR_MODEL_OFFLINE=1 — set to refuse network fetches. Cache root resolution: KTSTR_CACHE_DIR (if set), then $XDG_CACHE_HOME/ktstr/models/, then $HOME/.cache/ktstr/models/.

status prints four fields and adds a one-line annotation when the verdict is anything other than Matches (a clean hit gets no annotation):

The annotation distinguishes four verdicts: NotCached (no entry — emit a cargo ktstr model fetch hint plus the expected download size), CheckFailed (cached entry could not be SHA-checked due to an I/O error — re-fetch), Mismatches (cached entry hash does not match the pinned digest — re-fetch), Matches (silent — the all-clear path). Re-fetch is the shared remediation tail for every cached-but- not-Matches branch.

clean removes both the GGUF artifact at {cache_root}/models/{file_name} and its .mtime-size warm-cache sidecar (a small companion file the SHA fast-path uses to skip re-hashing on subsequent status calls). Per- file output names what was deleted with an IEC-prefixed size in parentheses (removed /path/to/Qwen3-4B-Q4_K_M.gguf (2.34 GiB)); a final freed N total line sums the artifact and sidecar bytes. A no-op clean (nothing cached) prints a single no cached model found at {path} line so an idempotent re-run produces a clear “nothing to do” outcome instead of two “(absent)” lines. Subsequent cargo ktstr model fetch re-downloads the pin from scratch.

verifier

Collect BPF verifier statistics for every scheduler declared via declare_scheduler! in the workspace’s test binaries. Spawns cargo nextest run -E 'test(/^verifier/)' and lets nextest fan out per (scheduler × kernel-list entry × accepted topology preset) cell — each cell boots its own VM, loads the scheduler’s BPF programs, and reports per-program verified instruction counts from host-side memory introspection.

cargo ktstr verifier                              # auto-discover kernel
cargo ktstr verifier --kernel ../linux            # pin to one kernel
cargo ktstr verifier --kernel 6.14 --kernel 7.0   # multi-kernel sweep
cargo ktstr verifier --raw                        # raw verifier log

There are no --scheduler / --scheduler-bin flags: the sweep discovers schedulers from the KTSTR_SCHEDULERS distributed slice populated by declare_scheduler!. To exclude a scheduler from the sweep, omit it from the test binary (or declare it with SchedulerSpec::Eevdf / SchedulerSpec::KernelBuiltin — both are skipped at cell-emission time because neither has a userspace binary to verify).

--kernel is repeatable; cargo-ktstr always exports KTSTR_KERNEL_LIST to the nextest invocation (synthesizing a single entry from auto-discovery when no --kernel is passed). Each scheduler’s kernels = [...] declaration acts as a per-scheduler filter on the operator-supplied set; an empty (or omitted) kernels field accepts every entry. See BPF Verifier: Matrix dimension + per-scheduler filter for the full filter contract.

--raw exports KTSTR_VERIFIER_RAW=1; the cell handler reads it via env::var_os and switches format_verifier_output from the cycle-collapsed default to the raw scheduler-log dump. See BPF Verifier: Cycle collapse algorithm for the rendering details.

See BPF Verifier for the cell-based dispatch design and output format, and Scheduler Definitions for the declare_scheduler! macro that registers a scheduler in KTSTR_SCHEDULERS.

shell

Shares the VM boot flow with ktstr shell and accepts the same flags. See ktstr shell for the full flag reference. The one behavior difference from ktstr shell is that cargo ktstr shell accepts raw image file paths for --kernel.

cargo ktstr shell
cargo ktstr shell --kernel 6.14.2
cargo ktstr shell --topology 1,2,4,1
cargo ktstr shell -i ./my-binary -i strace

completions

Generate shell completions for cargo-ktstr. See ktstr completions for the base subcommand.

cargo ktstr completions bash >> ~/.local/share/bash-completion/completions/cargo
cargo ktstr completions zsh > ~/.zfunc/_cargo-ktstr
cargo ktstr completions fish > ~/.config/fish/completions/cargo-ktstr.fish

stats

Sidecar analysis, per-record diagnostics, and run-to-run comparison. See Runs for the directory layout.

cargo ktstr stats                                             # print analysis of newest run
cargo ktstr stats list                                        # list runs
cargo ktstr stats list-metrics                                # list registered regression metrics
cargo ktstr stats compare --a-kernel 6.14 --b-kernel 6.15     # slice on kernel
cargo ktstr stats compare --a-scheduler scx_rusty --b-scheduler scx_alpha  # slice on scheduler
cargo ktstr stats compare --a-kernel 6.14 --b-kernel 6.15 --scheduler scx_rusty  # slice on kernel, pin scheduler on both sides
cargo ktstr stats compare --a-kernel 6.14 --b-kernel 6.15 -E cgroup_steady       # add substring filter
cargo ktstr stats compare --a-project-commit abcdef1 --b-project-commit fedcba2 --no-average  # opt out of trial averaging
cargo ktstr stats compare --a-kernel-commit abcdef1 --b-kernel-commit fedcba2    # slice on kernel source HEAD
cargo ktstr stats compare --a-run-source ci --b-run-source local                 # slice on run environment
cargo ktstr stats explain-sidecar --run RUN_ID                                   # diagnose Option-field absences

When invoked without a subcommand, prints gauntlet analysis from either the most recent run directory under {CARGO_TARGET_DIR or "target"}/ktstr/ (newest by mtime) or the explicit directory in KTSTR_SIDECAR_DIR when that variable is set. With KTSTR_SIDECAR_DIR set, that directory is the sidecar source directly – there is no newest-subdirectory walk under it:

• Gauntlet analysis – outlier detection, per-scenario/topology dimension summaries, stimulus cross-tab.
• BPF verifier stats – per-program verified instruction counts, warnings for programs near the 1M complexity limit.
• BPF callback profile – per-program invocation counts, total CPU time, and average nanoseconds per call.
• KVM stats – cross-VM averages for exits, halt polling, host preemptions.

list

Print a table of run directories under {CARGO_TARGET_DIR or "target"}/ktstr/ with four columns:

• RUN: the run-directory leaf name, formatted as {kernel}-{project_commit} per Runs. list does NOT consult KTSTR_SIDECAR_DIR — that override only affects where the test harness writes sidecars; list always enumerates the default runs-root.
• TESTS: number of sidecars in the directory (and one level of subdirectories — collect_sidecars walks per-job gauntlet layouts).
• DATE: the earliest sidecar timestamp present in the directory — under last-writer-wins this equals the most recent run’s first sidecar timestamp (the prior run’s sidecars were pre-cleared at the new run’s first write, so only the new run’s timestamps remain). See Runs for the full semantics.
• ARCH: the host.arch value from the run’s first sidecar (e.g. x86_64, aarch64). Renders as - when no sidecar in the directory carries a populated host context — pre-host- context archives and host-only test stubs that never populate the field land in this bucket.

Rows are sorted by directory mtime, most recent first, so the latest run lands at the top — the operator’s usual interest. Entries whose mtime cannot be read fall back to filename order as a deterministic tiebreaker and sort to the end of the listing.

list-metrics

List the registered regression metrics and their default thresholds. Enumerates the ktstr::stats::METRICS registry: metric name, polarity (higher/lower better), default absolute-delta gate, default relative-delta gate, and display unit. Use this to see which metric names ComparisonPolicy.per_metric_percent keys can reference, and what each default absolute and relative gate starts at before an override. Default output is a human-readable table; --json emits a JSON array with the same fields.

cargo ktstr stats list-metrics              # table
cargo ktstr stats list-metrics --json       # JSON array

list-values

List the distinct values present per filterable dimension in the sidecar pool. Walks every run directory under target/ktstr/ (or --dir), pools the sidecars, and reports per-dimension sets for all seven dimensions: kernel, commit, kernel_commit, source, scheduler, topology, and work_type. The commit and source keys map to the internal SidecarResult::project_commit / run_source fields; the JSON wire keys keep the shorter spellings.

Use this before crafting a cargo ktstr stats compare invocation to discover what --a-X / --b-X values the pool actually carries: --a-kernel 6.20 against an empty pool fails downstream with “no rows match filter A”, and list-values is the upstream answer to “what kernels do I have?”.

cargo ktstr stats list-values                       # text per-dim blocks
cargo ktstr stats list-values --json                # JSON object
cargo ktstr stats list-values --dir /tmp/archived   # archived sidecar tree

The text shape renders one block per dimension with values one per line. The JSON shape emits a single object keyed by dimension name with arrays of values:

{
  "kernel": [null, "6.14.2", "6.15.0"],
  "commit": [null, "abcdef1", "abcdef1-dirty"],
  "kernel_commit": [null, "kabcde7", "kabcde7-dirty"],
  "source": [null, "ci", "local"],
  "scheduler": ["eevdf", "scx_rusty"],
  "topology": ["1n2l4c1t", "1n4l2c1t"],
  "work_type": ["SpinWait", "PageFaultChurn"]
}

The JSON keys commit and source are the wire contract; internally the corresponding fields are SidecarResult::project_commit and SidecarResult::run_source, and the per-side filter flags spell as --project-commit / --run-source (see compare).

kernel, commit, kernel_commit, and source are optional on the source sidecar (SidecarResult::kernel_version / project_commit / kernel_commit / run_source are Option<String>); the textual sentinel unknown and JSON null both denote a sidecar that did not record a value for that dimension.

show-host

Print the archived HostContext for a specific run: CPU identity, memory/hugepage config, transparent-hugepage policy, NUMA node count, kernel uname triple, kernel cmdline, and every /proc/sys/kernel/sched_* tunable captured at archive time. Useful for inspecting the same fingerprint compare’s host-delta section uses, available on a single run.

The command scans sidecars in the run directory in iteration order and prints the FIRST sidecar that carries a populated host field — older pre-enrichment sidecars may have host: None, and the forward scan tolerates those. If no sidecar has a populated host field the command fails with an actionable error rather than returning empty output.

explain-sidecar

Diagnose Option-field absences across a run’s sidecars. Loads every *.ktstr.json under --run ID (or its subdirectories one level deep, mirroring compare’s gauntlet-job layout) and reports, per sidecar, which Option<T> fields landed as None plus the documented causes for each absence and a classification:

• expected — None is the steady-state shape; no operator action recovers it (e.g. payload for a scheduler-only test, scheduler_commit which no SchedulerSpec variant exposes today).
• actionable — None indicates a recoverable gap; re-running in a different environment (in-repo cwd, non-tarball kernel, non-host-only test) would populate the field.

Different gauntlet variants on the same run legitimately differ on which fields populate (host-only vs VM-backed, scheduler-only vs payload-bearing), so the report is per-sidecar rather than aggregate.

Sidecars are loaded verbatim — this command does NOT rewrite run_source to "archive" even when --dir is set. Diverges intentionally from compare / list-values; matches show-host. The override would erase the only signal that surfaces the pre-rename source-key drop case.

The output header reports walked N sidecar file(s), parsed M valid: N counts every .ktstr.json file the walker visited, M counts how many parsed against the current schema. walked > parsed signals a corrupt or pre-1.0-schema sidecar — re-run the test to regenerate under the current schema.

Per-None blocks in the text output also include a fix: line for fields whose None is recoverable by an operator action (e.g. kernel_commit recovers when KTSTR_KERNEL points at a local kernel git tree). Fields whose None is the steady-state shape (or a multi-cause set with no single remediation) emit no fix: line.

When the walk encounters parse failures, the text output appends a trailing corrupt sidecars (N): block listing each corrupt path on its own line followed by the serde error message indented as error: ..., optionally followed by an enriched: ... line with operator-facing remediation prose when the parse failure matches a known schema-drift case (currently the host missing-field case). When the walk encounters IO failures (file matched the predicate but read_to_string failed before parsing could begin — permission denied, mid-rotate truncation, broken symlink, EISDIR), the text output appends a parallel io errors (N): block, structured the same way (path on its own line, error: ... line below) but carrying std::io::Error::Display rather than serde-error text. IO errors do NOT carry enriched: lines — there is no schema-drift catalog for filesystem incidents; the raw std::io::Error Display is the remediation surface. Each block is suppressed independently when its source vec is empty.

All-corrupt and all-IO-failure runs (every predicate- matching file failed to parse, or every one failed to read) are NOT a hard error — text output renders the header (walked N sidecar file(s), parsed 0 valid) followed directly by the corrupt sidecars (N): and/or io errors (N): block(s), skipping the per-sidecar breakdown that has nothing to render. JSON output mirrors this with valid: 0, _walk.errors and/or _walk.io_errors populated, and per-field counts at zero. This preserves structured per-file visibility for dashboard consumers facing total-failure runs of either class.

All-corrupt and all-IO-failure runs exit 0 (not a hard error); CI scripts must inspect the JSON channel for failure detection rather than relying on exit code. Two common gating policies, each appropriate for different operational stances:

• Lenient (treat partial failures as warnings): _walk.valid > 0. Accepts any run with at least one successfully-parsed sidecar; per-file parse or IO failures surface in the JSON arrays for triage but do not fail the gate.
• Strict (fail on any sidecar failure): _walk.errors.len() == 0 && _walk.io_errors.len() == 0. Requires every predicate-matching file to parse cleanly. Both checks are required because the two arrays cover disjoint failure classes (parse vs read) — a run with zero parse errors but one IO error still has a missing sidecar.

The two policies are NOT equivalent: a run with one valid and one corrupt sidecar passes lenient (valid == 1 > 0) but fails strict (errors.len() == 1 > 0). Pick the policy that matches the operational tolerance for partial data.

--json emits a single object with three top-level keys: _schema_version (a string version stamp — currently "1" — that consumers can gate on for incompatible shape changes), _walk (an envelope carrying walked / valid counts — same numbers the text header reports under “walked N sidecar file(s), parsed M valid” — plus an errors array of {path, error, enriched_message} entries covering every parse failure (enriched_message is a human-facing remediation string when a known schema-drift case matches, JSON null otherwise) AND an io_errors array of {path, error} entries covering every IO failure (file matched the predicate but read_to_string failed; error carries the raw std::io::Error Display). Both arrays emit on every render — empty array when no failures of that class occurred — so dashboard consumers see a uniform shape without contains_key branching. With both arrays, walked == valid + errors.len() + io_errors.len() by construction in the steady state — every predicate-matching file lands in exactly one bucket. (Filesystem races between the count and load passes can perturb this; see the rustdoc on WalkStats for the full caveat.) Then fields. Each entry under fields carries none_count and some_count (counts across all valid sidecars in the run, summing to _walk.valid), classification, causes, and fix (string when a remediation applies, JSON null otherwise).

Output produced before the schema-version stamp landed has no _schema_version key; consumers should treat the key’s absence as pre-stamp output (compatible with shape "1" in practice but unstamped).

The version bumps on incompatible shape changes (key rename, key removal, semantic shift in an existing key) but NOT on additive changes (new optional top-level keys, new entries in fields, new optional sub-keys under existing entries). The stamp is emitted as a JSON string (e.g. "1", "2"); parse it by stripping the quotes and converting the inner digits to an integer, then gate on parsed >= 1 (integer comparison) — never use raw string comparison, since lexicographic order would put "10" ahead of "2". Pin loosely (e.g. accept any version >= 1) so dashboard code keeps working when the catalog grows; tighten only on the specific bumps a consumer cannot tolerate.

cargo ktstr stats explain-sidecar --run RUN_ID                       # text per-sidecar diagnostic
cargo ktstr stats explain-sidecar --run RUN_ID --json                 # aggregate JSON for dashboards
cargo ktstr stats explain-sidecar --run RUN_ID --dir /path/archive    # diagnose archived sidecars

compare

Pool every sidecar under target/ktstr/ (or --dir), partition the rows into A and B sides via per-side filter flags, average each side’s matching sidecars per pairing key (or pass through distinct sidecars under --no-average), and report regressions on the A→B delta. Exits non-zero on regression.

The dimensions on which the A and B filters DIFFER are the SLICING dimensions — the axes of the A/B contrast. Every other dimension is part of the dynamic PAIRING key the comparison joins on. Slicing dims are derived automatically from the filters:

# Slice on kernel: A is 6.14, B is 6.15. Pair on every other dim.
cargo ktstr stats compare --a-kernel 6.14 --b-kernel 6.15

# Slice on kernel AND scheduler simultaneously.
cargo ktstr stats compare \
    --a-kernel 6.14 --a-scheduler scx_rusty \
    --b-kernel 6.15 --b-scheduler scx_alpha

# Slice on project commit, narrow both sides to one scheduler+kernel.
cargo ktstr stats compare \
    --a-project-commit abcdef1 --b-project-commit fedcba2 \
    --kernel 6.14 --scheduler scx_rusty

# Slice on run environment: CI runs vs local developer runs.
cargo ktstr stats compare \
    --a-run-source ci --b-run-source local

Symmetric sugar. Shared --X flags (--kernel, --scheduler, --topology, --work-type, --project-commit, --kernel-commit, --run-source) pin BOTH sides to the same value(s). Per-side --a-X / --b-X flags REPLACE the corresponding shared --X value for that side only — “more-specific replaces” semantics. So --kernel 6.14 --a-kernel 6.13 puts A on 6.13 and B on 6.14. Together the seven slicing dimensions (kernel, scheduler, topology, work-type, project-commit, kernel-commit, run-source) cover every typed axis the comparison can contrast on.

Validation. The dispatch site rejects two cases up front:

• Empty slicing: no --a-X / --b-X at all, OR the per-side flags resolve to identical effective filters. Bails with “specify at least one per-side filter (e.g. --a-kernel 6.14 --b-kernel 6.15) to define what dimension separates the two sides.”
• Multi-dim slicing: slicing on more than one dimension prints a warning to stderr (“warning: slicing on N dimensions; results compress multiple axes into a single A/B contrast”) but continues — multi-dim contrasts are a deliberate feature for cohort sweeps.

Averaging. By default the comparison aggregates every matching sidecar within each side into a single arithmetic-mean row per pairing key, smoothing run-to-run jitter. Failing / skipped contributors are excluded from the metric mean; the aggregated row’s passed is the AND across every contributor. A header line above the comparison table reads averaged across N runs (A) and M runs (B) and a per-group passes_observed/total_observed block prints below the summary.

+mixed commit marker. When contributors to an averaged group disagree on the -dirty suffix for the same canonical hex (some clean, some -dirty), the rendered commit and kernel_commit columns show {hex}+mixed for that group. +mixed is a COHORT-level marker (distinct from -dirty, which is a per-record property of one sidecar): it indicates mixed working-tree state across the group’s contributors. Mixed-dirty tracking spans EVERY contributor (passing, failing, skipped) so WIP-vs-committed disagreement surfaces in the averaged row even when one of the two states only appears on a failing run. The marker is rendered against the canonical un-suffixed hex, so a abc1234 clean entry plus an abc1234-dirty entry render as abc1234+mixed regardless of which contributor was scanned first. Homogeneous cohorts (every contributor clean, every contributor dirty, or every contributor None) preserve the first-seen value verbatim and never get the +mixed marker.

--no-average keeps each sidecar distinct. If multiple sidecars on the same side share the same pairing key under --no-average, the comparison bails with “duplicate pairing keys” — pairing across A/B sides is ambiguous when one A-row could match many B-rows. Either drop --no-average to average them, or add another per-side filter to disambiguate.

Kernel match shape. A --kernel 6.12 filter (two-segment major.minor) PREFIX-matches every patch release in that series: 6.12, 6.12.0, 6.12.5 all match. A three-or-more-segment filter (--kernel 6.14.2, --kernel 6.15-rc3) is strict equality — 6.14.2 does NOT match 6.14.20. The same shape applies to --a-kernel / --b-kernel.

Discovering filter values. Run cargo ktstr stats list-values before crafting a compare invocation to see what kernel, commit, kernel_commit, source, scheduler, topology, and work_type values the sidecar pool actually carries; passing a --a-kernel 6.20 against an empty pool fails downstream with “no rows match filter A” and list-values is the upstream answer to “what have I got?”. list-values reports all seven filterable dimensions; the JSON keys commit and source map to the per-side filter flags --project-commit and --run-source.

When a side comes back as unknown for one of the optional dimensions (kernel, commit, kernel_commit, source), cargo ktstr stats explain-sidecar on the underlying run reports per-sidecar which optional fields are missing and what each absence means.

Prerequisites

Run tests first to generate sidecar JSON files:

cargo ktstr test                     # generates target/ktstr/{kernel}-{project_commit}/*.json
cargo ktstr stats                    # reads the newest run

Set KTSTR_SIDECAR_DIR to override the sidecar directory; otherwise the default is {CARGO_TARGET_DIR or "target"}/ktstr/{kernel}-{project_commit}/, where {project_commit} is the project HEAD short hex (with -dirty when the worktree differs).

show-host

Print the live host context used by the sidecar collector: CPU identity, memory/hugepage config, transparent-hugepage policy, NUMA node count, kernel uname triple (sysname / release / machine), kernel cmdline, and every /proc/sys/kernel/sched_* tunable. Useful for diagnosing cross-run regressions that trace back to host-context drift (sysctl change, THP policy flip, hugepage reservation) or for confirming what cargo ktstr stats compare would record on the next run produced here.

cargo ktstr show-host

This is a live snapshot (reads /proc, /sys, and uname() at invocation time). For the archived host context captured at sidecar-write time for a past run, use cargo ktstr stats show-host --run RUN_ID instead — same HostContext::format_human formatter so the two outputs are byte-for-byte comparable when the host is unchanged.

For historical drift between archived runs (host-side diff across two run partitions), use cargo ktstr stats compare — its host-delta section reports which host-context fields changed between side A and side B using the same HostContext::diff logic.

show-thresholds

Print the resolved assertion thresholds for the named test — the same merged Assert value run_ktstr_test_inner evaluates against worker reports, produced by the runtime merge chain Assert::default_checks().merge(entry.scheduler.assert()).merge(&entry.assert). Surfaces every threshold field (or none when inherited or unset) so an operator can see what the test will actually check against without reading source or guessing which layer contributed each bound.

cargo ktstr show-thresholds preempt_regression_fault_under_load

Fails with an actionable message when no registered test matches the given name; the diagnostic includes a Did you mean ...? Levenshtein suggestion when a near match exists.

locks

Enumerate every ktstr flock held on this host — read-only, does NOT attempt any flock acquire. Troubleshooting companion for --cpu-cap contention: when a build or test is stalled behind a peer’s reservation, cargo ktstr locks names the peer (PID + cmdline) without disturbing any of its flocks.

Scans four lock-file roots:

• /tmp/ktstr-llc-*.lock — per-LLC reservations held by perf-mode test runs and --cpu-cap-bounded builds.
• /tmp/ktstr-cpu-*.lock — per-CPU reservations from the same flow.
• {cache_root}/.locks/*.lock — cache-entry locks held during kernel build writes, and source-{path_hash}.lock files held for the duration of kernel build --source and cargo ktstr test --kernel <path> against the same source tree.
• {runs_root}/.locks/{kernel}-{project_commit}.lock — per-run-key sidecar-write locks held for the duration of the (pre-clear + write) cycle to serialize concurrent ktstr processes targeting the same run directory.

Each lock is cross-referenced against /proc/locks to name the holder PID and cmdline.

cargo ktstr locks                       # one-shot snapshot
cargo ktstr locks --json                # JSON snapshot
cargo ktstr locks --watch 1s            # redraw every second until SIGINT
cargo ktstr locks --watch 1s --json     # ndjson stream, one object per interval

The same subcommand is available as ktstr locks with identical flag semantics.

Install

cargo install --locked ktstr --bin ktstr --bin cargo-ktstr   # the two user-facing binaries

The explicit --bin flags scope the install to just ktstr and cargo-ktstr; without them, cargo install would also place the test-fixture binaries (ktstr-jemalloc-probe, ktstr-jemalloc-alloc-worker) on $PATH.

Or build from the workspace:

cargo build --bin cargo-ktstr