ktstr

ktstr is a test harness for Linux process schedulers, with a focus on sched_ext (BPF-extensible process scheduling). It boots Linux kernels in KVM virtual machines with controlled CPU topologies, runs workloads, and verifies scheduling correctness. Also tests under the kernel’s default EEVDF scheduler.

Quick taste

The simplest test calls a canned scenario:

use ktstr::prelude::*;

#[ktstr_test(llcs = 1, cores = 2, threads = 1)]
fn my_test(ctx: &Ctx) -> Result<AssertResult> {
    scenarios::steady(ctx)
}

cargo ktstr test --kernel ../linux

Without a scheduler attribute, tests run under EEVDF. See Getting Started for testing a sched_ext scheduler.

Library API

The ktstr::prelude module re-exports the types needed for writing tests. Declare cgroups and workloads as data with CgroupDef:

use ktstr::prelude::*;

#[ktstr_test(llcs = 1, cores = 2, threads = 1)]
fn my_test(ctx: &Ctx) -> Result<AssertResult> {
    execute_defs(ctx, vec![
        CgroupDef::named("cg_0").workers(2),
        CgroupDef::named("cg_1").workers(2),
    ])
}

The prelude also exports low-level types (CgroupGroup, WorkloadConfig, WorkloadHandle) for manual cgroup and worker management, Assert for composable assertion config, and WorkerReport for telemetry access.

For binary workloads (running schbench, fio, or any external executable as part of a test), see Payload Definitions. #[ktstr_test(payload = FIXTURE)] runs a Payload (binary workload) alongside the cgroup workers; the scheduler = slot takes a bare Scheduler reference — the const emitted by declare_scheduler!.

What it tests

• Fair scheduling – workers get CPU time without starvation or excessive scheduling gaps.
• Cpuset isolation – workers stay on assigned CPUs.
• Dynamic operations – cgroups created, destroyed, and resized mid-run.
• Affinity – scheduler respects thread affinity constraints.
• Stress – many cgroups, many workers, rapid topology changes.
• Stall detection – scheduler doesn’t drop tasks.

Design

Two principles drive ktstr’s architecture:

Fidelity without overhead – every test boots a real Linux kernel in a KVM VM with real cgroups and real BPF programs. No mocking, no containers, no shared state. The VMM is minimal and PCI-free: two 16550 serial ports (COM1 for kernel console, COM2 for application I/O), a shared-memory ring buffer, and three virtio-MMIO devices (virtio-console for guest console I/O, virtio-blk for file-backed block storage with optional btrfs templates, virtio-net for in-VMM L2 loopback used by network workload tests).

Direct access over tooling layers – the host-side monitor reads guest memory directly via BTF (BPF Type Format)-resolved struct offsets to observe scheduler state. The monitor runs entirely host-side — no BPF programs are injected into the guest to collect scheduler telemetry, so observations do not perturb scheduling decisions. (BPF programs loaded by the scheduler under test, the BPF verifier pipeline, and the auto-repro probe pipeline are separate concerns; those are the code under test, not the observation layer.) See Monitor for details on BTF resolution and guest memory introspection.

BPF verifier analysis

The verifier_pipeline tests boot a scheduler in a VM and capture per-program verifier output from the real kernel verifier. The default output applies cycle collapse to reduce repetitive loop unrolling. See BPF Verifier for details.

Auto-repro probe pipeline

When a scheduler crashes, ktstr can automatically rerun the failing scenario with BPF probes attached to the crash-path functions. See Auto-Repro for details.

Workspace structure

ktstr and cargo-ktstr are the two user-facing [[bin]] targets in the crate; install them with cargo install --locked ktstr --bin ktstr --bin cargo-ktstr. The crate also defines two test-fixture [[bin]] targets — ktstr-jemalloc-probe and ktstr-jemalloc-alloc-worker — used by the tests/jemalloc_probe_tests.rs integration tests. The explicit --bin flags scope the install to just the two operator-facing entry points; without them, cargo install would also place the test-fixture binaries on $PATH.

Kernel config

ktstr.kconfig in the repo root contains the kernel config fragment needed for scheduler testing (sched_ext, BPF, kprobes, cgroups). Copy it to your kernel source tree and run make olddefconfig.

Features

ktstr is a test framework for Linux process schedulers. See Overview for a quick introduction with code examples.

Supported kernels

ktstr’s runtime dispatches to per-kernel-version fallback paths for the watchdog timeout and event counters. CI explicitly exercises 6.14 and 7.0 on both x86_64 and aarch64. On 7.1+ kernels the watchdog override uses scx_sched.watchdog_timeout via BTF detection; older kernels use the static scx_watchdog_timeout symbol.

Event counters follow a different layout split: 6.18+ kernels (backported to 6.17.7+ stable) read via scx_sched.pcpu -> scx_sched_pcpu.event_stats; 6.16-6.17 kernels read scx_sched.event_stats_cpu directly. When neither path is available, event-counter sampling is silently disabled.

Testing

use ktstr::declare_scheduler;

declare_scheduler!(MITOSIS, {
    name = "mitosis",
    binary = "scx_mitosis",
    topology = (1, 2, 4, 1),
    sched_args = ["--exit-dump-len", "1048576"],
});

Observability

Debugging

Infrastructure

High test coverage — broad self-coverage of the framework itself

See Getting Started to set up your first test, or browse the Recipes for common workflows.

Getting Started

Prerequisites

Linux only (x86_64, aarch64). ktstr boots KVM virtual machines; it does not build or run on other platforms.

• Linux host with KVM access (/dev/kvm)
• Rust toolchain (stable, >= 1.94.1; pinned via rust-toolchain.toml)
• clang (BPF skeleton compilation)
• pkg-config, make, gcc
• autotools (autoconf, autopoint, flex, bison, gawk) – vendored libbpf/libelf/zlib build
• BTF (/sys/kernel/btf/vmlinux – present by default on most distros; set KTSTR_KERNEL if missing)
• Internet access on first build (downloads busybox source)
• Linux kernel 6.12+ for sched_ext tests (check with uname -r). The host kernel has no version requirement beyond KVM; the test kernel is whichever you build or cache via cargo ktstr kernel build. See Supported kernels for per-feature version boundaries.

Ubuntu/Debian:

sudo apt install clang pkg-config make gcc autoconf autopoint flex bison gawk

Fedora:

sudo dnf install clang pkgconf make gcc autoconf gettext-devel flex bison gawk

Install tools

cargo install cargo-nextest           # required
cargo install --locked ktstr --bin ktstr --bin cargo-ktstr   # both user-facing binaries (optional)

cargo-nextest is required. cargo ktstr test delegates to nextest internally; without it, cargo ktstr test will fail.

cargo install --locked ktstr --bin ktstr --bin cargo-ktstr installs the two user-facing binaries (ktstr host-side CLI and cargo-ktstr dev workflow plugin); the --bin flags scope the install away from the two test-fixture binaries (ktstr-jemalloc-probe, ktstr-jemalloc-alloc-worker) that the crate’s integration tests spawn.

Add the dependency

Add ktstr as a dev-dependency:

[dev-dependencies]
ktstr = { version = "0.4" }

Kernel discovery

Tests require a bootable Linux kernel. The test harness checks these locations in order:

1. KTSTR_TEST_KERNEL environment variable (direct image path).
2. KTSTR_KERNEL environment variable, parsed as one of three forms:
   • Path: search that directory for arch/<arch>/boot/<image>
   • Version (e.g. 6.14.2): look up the version in XDG cache
   • Cache key (from cargo ktstr kernel list): exact cache lookup
3. XDG cache: most recent cached image (newest first); entries built with a different kconfig fragment are skipped. When KTSTR_KERNEL named an explicit version or cache key that was not present in the cache, the cache scan is skipped entirely – discovery moves on to step 4 rather than substituting an unrelated cached kernel.
4. ./linux/arch/<arch>/boot/<image> (workspace-local build tree)
5. ../linux/arch/<arch>/boot/<image> (sibling directory)
6. /lib/modules/$(uname -r)/build/arch/<arch>/boot/<image> (installed kernel build tree)
7. /lib/modules/$(uname -r)/vmlinuz (installed kernel)
8. /boot/vmlinuz-$(uname -r)
9. /boot/vmlinuz (unversioned symlink)

On x86_64, the build-tree image is arch/x86/boot/bzImage; on aarch64, arch/arm64/boot/Image.

The host’s installed kernel works for basic testing. For sched_ext tests, build a kernel with the ktstr config fragment (below). See Troubleshooting for details.

Implicit discovery reads existing cache entries but does not run the build pipeline or produce a new cache entry. The chain reads existing cache entries on the read path (most-recent-valid first; entries built with a different kconfig fragment are skipped) and falls back to local build trees and host paths when nothing matches. It does NOT compute a local-{hash7}-{arch}-kc{suffix} cache key, run the build pipeline, or store a new cache entry from whatever source-tree image it ends up using. To opt into the build + cache-store pipeline so a source tree’s build is recorded and reused under a stable cache key, pass the path explicitly via cargo ktstr test --kernel ../linux; see cargo-ktstr — What it does for the full path-mode flow including the cache-hit short-circuit.

Build a kernel

cargo ktstr kernel build downloads a kernel tarball from kernel.org, configures it with the embedded ktstr.kconfig fragment, builds it, and caches the result:

cargo ktstr kernel build               # latest stable series with >= 8 maintenance releases
cargo ktstr kernel build 6.14.2        # specific version
cargo ktstr kernel build 6.12          # highest 6.12.x patch release
cargo ktstr kernel build 6             # highest 6.x.y release

The bare cargo ktstr kernel build skips series that have fewer than 8 maintenance releases to keep CI off brand-new majors whose early point releases tend to hit build issues on older toolchains; pass the specific version explicitly if you need a series that hasn’t reached .8 yet.

Subsequent runs of cargo ktstr test or cargo nextest run will find the cached kernel automatically (step 3 in the discovery chain above).

To build from a local source tree:

cargo ktstr kernel build --source ../linux

To list and manage cached kernels:

cargo ktstr kernel list
cargo ktstr kernel clean --keep 3

See cargo-ktstr for all options.

Manual

cd /path/to/linux
make defconfig
cat /path/to/ktstr/ktstr.kconfig >> .config
make olddefconfig
make -j$(nproc)

ktstr.kconfig in the repo root contains a kernel config fragment tuned for scheduler testing (sched_ext, BPF, kprobes, minimal boot).

Write a test

Create a file in your crate’s tests/ directory (e.g. tests/sched_test.rs) and write a #[ktstr_test] function. The prelude module re-exports the types you need.

The simplest test uses a canned scenario. AssertResult carries the pass/fail verdict, diagnostic messages, and per-cgroup statistics from the run.

use ktstr::prelude::*;

#[ktstr_test(llcs = 1, cores = 2, threads = 1)]  // llcs = last-level caches
fn my_test(ctx: &Ctx) -> Result<AssertResult> {
    // `scenarios::steady` is a canned scenario: two cgroups of equal
    // CPU-spin workers, no cpuset restrictions, run for the default
    // duration.
    scenarios::steady(ctx)
}

For custom cgroup topology, declare cgroups with CgroupDef and run them with execute_defs. A CgroupDef bundles the cgroup name, optional cpuset, and workload specification into a single declaration. This is the most common custom test pattern:

use std::time::Duration;
use ktstr::prelude::*;

#[ktstr_test(llcs = 1, cores = 2, threads = 1)]
fn my_test(ctx: &Ctx) -> Result<AssertResult> {
    execute_defs(ctx, vec![
        CgroupDef::named("cg_0").workers(4),
        CgroupDef::named("cg_1")
            .workers(2)
            // CPU burst for 50 ms, sleep for 100 ms, repeat.
            .work_type(WorkType::bursty(
                Duration::from_millis(50),
                Duration::from_millis(100),
            )),
    ])
}

execute_defs is a convenience wrapper that creates a single step holding for the full duration – use it when all cgroups run concurrently for one phase. Use execute_steps when you need multiple phases (e.g., adding cgroups mid-test or changing cpusets between phases).

Step::with_defs pairs a list of CgroupDefs with a HoldSpec that controls how long the step runs. This example starts two cgroups, then adds a third mid-test:

use ktstr::prelude::*;

#[ktstr_test(llcs = 1, cores = 4, threads = 1)]
fn my_test(ctx: &Ctx) -> Result<AssertResult> {
    let steps = vec![
        // Phase 1: two cgroups for the first half.
        Step::with_defs(
            vec![
                CgroupDef::named("cg_0").workers(2),
                CgroupDef::named("cg_1").workers(2),
            ],
            HoldSpec::Frac(0.5),
        ),
        // Phase 2: add a third cgroup for the remaining half.
        Step::with_defs(
            vec![CgroupDef::named("cg_2").workers(2)],
            HoldSpec::Frac(0.5),
        ),
    ];
    execute_steps(ctx, steps)
}

How it runs

The framework boots a KVM VM with the requested topology and runs your test binary as the guest’s init process. Your test function executes inside the VM – execute_defs and execute_steps immediately create cgroups, spawn workers, run the workload, and return assertion results. Ctx provides the guest topology (ctx.topo) and cgroup management (ctx.cgroups).

What gets checked

Every test automatically checks for worker starvation, scheduling fairness, scheduling gaps, and host-side runqueue health (including imbalance, stalls, dispatch queue depth). These defaults come from Assert::default_checks() and can be overridden per-scheduler or per-test. See Checking for the full list of checks and thresholds.

Run

The recommended way to run #[ktstr_test] tests is cargo ktstr test, which handles kernel resolution and wraps cargo nextest:

cargo ktstr test --kernel ../linux

The ktstr ctor automatically intercepts nextest protocol args (--list, --exact) for gauntlet expansion and budget-driven test selection.

Fallbacks:

• cargo nextest run: ctor intercepts, runs gauntlet-expanded tests (you must supply your own kernel via KTSTR_KERNEL / KTSTR_TEST_KERNEL or the discovery chain).
• cargo test: standard harness runs the #[test] wrappers (base topology only, no gauntlet expansion).

Requires /dev/kvm. See Troubleshooting if KVM is unavailable.

Passing tests:

    PASS [  11.34s] my_crate::my_sched_tests ktstr/my_test

A failing test prints assertion details:

    FAIL [  12.05s] my_crate::my_sched_tests ktstr/my_test

--- STDERR ---
ktstr_test 'my_test' [topo=1n1l2c1t] failed:
  stuck 3500ms on cpu1 at +1200ms

--- stats ---
4 workers, 2 cpus, 8 migrations, worst_spread=12.3%, worst_gap=3500ms
  cg0: workers=2 cpus=2 spread=5.1% gap=3500ms migrations=4 iter=15230
  cg1: workers=2 cpus=2 spread=12.3% gap=890ms migrations=4 iter=14870

Each test invocation writes results into {CARGO_TARGET_DIR or "target"}/ktstr/{kernel}-{project_commit}/ as one *.ktstr.json sidecar per #[ktstr_test] variant. Run cargo ktstr stats list to see runs (RUN, TESTS, DATE, ARCH columns). See Runs for the full layout and analysis workflow.

Using cargo-ktstr

cargo ktstr test handles kernel resolution and test execution in one command:

cargo ktstr test                                              # auto-discover kernel
cargo ktstr test --kernel ../linux                            # local source tree (builds + caches; subsequent runs hit cache)
cargo ktstr test --kernel 6.14.2                              # version (auto-downloads on miss)
cargo ktstr test -- -E 'test(my_test)'                        # pass nextest args

See cargo-ktstr for details.

Interactive shell

cargo ktstr shell boots a VM with busybox for manual exploration:

cargo ktstr shell                              # default 1,1,1,1 topology
cargo ktstr shell --topology 1,2,4,1           # 1 NUMA node, 2 LLCs, 4 cores/LLC, 1 thread/core
cargo ktstr shell -i ./my-scheduler            # include a file in the guest
cargo ktstr shell -i ./test-data/              # include a directory recursively

Included ELF binaries get automatic shared library resolution. Directories are walked recursively; their contents appear under /include-files/<dirname>/ preserving the original structure. Individual files are available at /include-files/<name> inside the guest. See cargo-ktstr shell for details.

Next steps

To understand scenarios, flags, and checking: Core Concepts.

To write new tests: Writing Tests.

To test your own scheduler: Test a New Scheduler.

Zero to ktstr

This tutorial walks through writing a complete #[ktstr_test] from scratch. By the end you’ll have a working scheduler test that runs two cgroups with different lifecycle patterns across a multi-LLC topology, tunes test duration and the watchdog, and asserts fairness, throughput parity, and cpuset isolation.

What you’ll build

A test named mixed_workloads that:

• Runs two cgroups on separate LLCs:
  • background_spinner – a persistent CPU-bound load that runs for the entire test duration.
  • phased_worker – a worker that loops through explicit Spin -> Yield -> Spin -> Yield ... phases via WorkType::Sequence.
• Targets a 2-LLC, 4-core topology so the scheduler has a real cache boundary to respect.
• Sets explicit test duration and scx watchdog timeout via #[ktstr_test] attributes.
• Asserts fairness (per-cgroup spread), throughput parity (CV across workers + minimum rate), and cpuset isolation (workers stay on their assigned CPUs). Scheduling gaps and host-side runqueue health are checked automatically.

The complete test is at the end of this page.

Prerequisites

Set up the host and a kernel before continuing:

• Getting Started covers KVM access, the toolchain, and the dev-dependency.
• A bootable Linux kernel image is required. Build one with cargo ktstr kernel build or point at a source tree with cargo ktstr test --kernel ../linux. See Getting Started: Build a kernel for the full kernel-management workflow.

Once the dependency is in place, create a file under your crate’s tests/ directory (e.g. tests/mixed_workloads.rs) and follow along.

Step 1: The skeleton

Every #[ktstr_test] is a Rust function that takes &Ctx and returns Result<AssertResult>. Start with an empty body that passes unconditionally:

use ktstr::prelude::*;

#[ktstr_test(llcs = 1, cores = 2, threads = 1)]
fn mixed_workloads(ctx: &Ctx) -> Result<AssertResult> {
    let _ = ctx;
    Ok(AssertResult::pass())
}

use ktstr::prelude::*; brings in every type the test body needs – Ctx, AssertResult, CgroupDef, WorkType, CpusetSpec, execute_defs, and the Result alias from anyhow. The #[ktstr_test] attribute registers the function so cargo ktstr test discovers it and boots a VM with the requested topology.

A test without a scheduler = ... attribute runs under the kernel’s default EEVDF scheduler — useful as a baseline. Step 2 swaps in a sched_ext scheduler so the rest of the tutorial exercises that scheduler instead.

For the full attribute reference, see The #[ktstr_test] Macro.

Step 2: Define your scheduler

To target a sched_ext scheduler, declare it with declare_scheduler! and reference the generated const from #[ktstr_test(scheduler = …)]. The example uses scx-ktstr, the test-fixture scheduler shipped in this workspace; substitute your own binary name to target a different scheduler.

use ktstr::declare_scheduler;
use ktstr::prelude::*;

declare_scheduler!(KTSTR_SCHED, {
    name = "ktstr_sched",
    binary = "scx-ktstr",
});

#[ktstr_test(scheduler = KTSTR_SCHED, llcs = 1, cores = 2, threads = 1)]
fn mixed_workloads(ctx: &Ctx) -> Result<AssertResult> {
    let _ = ctx;
    Ok(AssertResult::pass())
}

declare_scheduler! emits a pub static KTSTR_SCHED: Scheduler holding the declared fields and registers a private static in the KTSTR_SCHEDULERS distributed slice via linkme so cargo ktstr verifier discovers it automatically. The scheduler = slot on #[ktstr_test] expects an &'static Scheduler — pass the bare KTSTR_SCHED ident.

The macro fields:

• name — scheduler name for display and sidecar keys.
• binary — binary name for auto-discovery in target/{debug,release}/, the directory containing the test binary, or a KTSTR_SCHEDULER override path. When the scheduler is a [[bin]] target in the same workspace, cargo build already places it where discovery looks. The resolved binary is packed into the VM’s initramfs.
• topology = (numa, llcs, cores, threads) — optional default VM topology. Tests can override individual dimensions via #[ktstr_test(llcs = ...)]. Omitted here; the per-test attributes in Step 4 set every dimension explicitly.
• sched_args = ["--flag", "--another"] — optional CLI args prepended to every test that uses this scheduler. Useful when a scheduler needs the same --enable-llc-style switches in every run; for one-off variations, use #[ktstr_test(extra_sched_args = [...])] on the test instead.
• kernels = ["6.14", "6.15..=7.0"] — optional set of kernel specs the verifier sweep should exercise this scheduler against. See BPF Verifier for the cell emission contract.

For the full attribute surface (sysctls, kargs, config_file, gauntlet constraints, scheduler-level assertion overrides), see Scheduler Definitions.

When the macro doesn’t fit — the most common case being inline JSON config supplied per-test or programmatic composition — define the Scheduler const through the manual builder instead. Step 12 below walks through that path with scx_layered.

Step 3: Add workloads

A CgroupDef declares a cgroup along with the workers that will run inside it. The builder methods configure worker count, the work each worker performs, scheduling policy, and cpuset assignment.

Add two cgroups – both running tight CPU spinners for now. Step 5 will swap one of them for a phased workload:

use ktstr::prelude::*;

#[ktstr_test(scheduler = KTSTR_SCHED, llcs = 1, cores = 2, threads = 1)]
fn mixed_workloads(ctx: &Ctx) -> Result<AssertResult> {
    execute_defs(ctx, vec![
        CgroupDef::named("background_spinner")
            .workers(2)
            .work_type(WorkType::SpinWait),
        CgroupDef::named("phased_worker")
            .workers(2)
            .work_type(WorkType::SpinWait),
    ])
}

Without .with_cpuset(...), a cgroup’s workers run on every CPU in the test’s topology — they share the VM’s full CPU set with all other cgroups. .with_cpuset(CpusetSpec::Llc(idx)) (introduced in Step 4) restricts a cgroup to one LLC’s CPUs, and the other CpusetSpec variants narrow further.

WorkType::SpinWait runs a tight CPU spin loop; it is one of many work primitives – see WorkType for the full enum (Bursty, FutexPingPong, CachePressure, IoSyncWrite, PageFaultChurn, MutexContention, Sequence, etc.) and the work-type-to-scheduler-behavior mapping table.

execute_defs is a convenience wrapper that runs each cgroup concurrently for the test’s full duration. Both cgroups are persistent – they hold for the entire scenario. Use execute_steps when you need to add cgroups mid-run or swap cpusets between phases; see Ops and Steps for the multi-step API.

Step 4: Set topology

The #[ktstr_test] attribute carries the VM’s CPU topology. Topology dimensions are big-to-little: numa_nodes (default 1), llcs (total across all NUMA nodes), cores per LLC, and threads per core. Total CPU count is llcs * cores * threads.

LLC count matters because the last-level cache is the primary scheduling boundary – tasks sharing an LLC benefit from shared cache lines, while cross-LLC migration carries a cold-cache penalty. A scheduler that ignores LLC topology will look fine on llcs = 1 and start failing as soon as there is a real cache boundary to respect.

Bump the topology to two LLCs with two cores each (4 CPUs total) so each cgroup can own its own LLC:

#[ktstr_test(scheduler = KTSTR_SCHED, llcs = 2, cores = 2, threads = 1)]
fn mixed_workloads(ctx: &Ctx) -> Result<AssertResult> {
    execute_defs(ctx, vec![
        CgroupDef::named("background_spinner")
            .workers(2)
            .work_type(WorkType::SpinWait)
            .with_cpuset(CpusetSpec::Llc(0)),
        CgroupDef::named("phased_worker")
            .workers(2)
            .work_type(WorkType::SpinWait)
            .with_cpuset(CpusetSpec::Llc(1)),
    ])
}

CpusetSpec::Llc(idx) confines a cgroup to the CPUs that belong to LLC idx. Other variants (Numa, Range, Disjoint, Overlap, Exact) cover NUMA-node binding, fractional partitioning, and hand-built CPU sets.

For the full topology surface (NUMA accessors, per-LLC info, cpuset generation helpers), see TestTopology.

Step 5: Compose phased work inside a cgroup

So far both cgroups run identical CPU spinners. The point of this test is to exercise a scheduler against different lifecycle patterns at once, so swap phased_worker for a worker that loops through explicit phases.

WorkType::Sequence { first: Phase, rest: Vec<Phase> } runs each phase for its specified duration and then advances to the next; when the last phase ends the loop restarts from first. Phases: Phase::Spin(Duration), Phase::Sleep(Duration), Phase::Yield(Duration), Phase::Io(Duration). Use the WorkType::sequence(first, rest) constructor.

Phase, WorkType, and CpusetSpec are all in ktstr::prelude::*; only std::time::Duration needs an extra use line — added on the first line of the example below:

use std::time::Duration;
use ktstr::prelude::*;

#[ktstr_test(scheduler = KTSTR_SCHED, llcs = 2, cores = 2, threads = 1)]
fn mixed_workloads(ctx: &Ctx) -> Result<AssertResult> {
    execute_defs(ctx, vec![
        // Persistent CPU pressure on LLC 0 for the whole run.
        CgroupDef::named("background_spinner")
            .workers(2)
            .work_type(WorkType::SpinWait)
            .with_cpuset(CpusetSpec::Llc(0)),
        // Phased worker on LLC 1: spin 100 ms, yield for 20 ms,
        // then loop. Stresses the scheduler's wake-after-yield
        // placement repeatedly while the LLC-0 spinner keeps
        // host runqueue pressure constant.
        CgroupDef::named("phased_worker")
            .workers(2)
            .work_type(WorkType::sequence(
                Phase::Spin(Duration::from_millis(100)),
                [Phase::Yield(Duration::from_millis(20))],
            ))
            .with_cpuset(CpusetSpec::Llc(1)),
    ])
}

The two cgroups now exercise distinct paths concurrently:

• background_spinner keeps two CPUs continuously busy on LLC 0.
• phased_worker alternates between burning CPU and yielding on LLC 1, exercising the scheduler’s voluntary-preemption + wakeup placement code paths.

Both cgroups still run for the entire scenario duration: the phasing happens within each phased_worker worker’s loop, while execute_defs holds both cgroups across the whole run via HoldSpec::FULL. To express phasing across cgroups (e.g. add phased_worker only for the second half of the run), use execute_steps with multiple Step entries – see Ops and Steps. Step 9 below adds an Op::snapshot capture into a step’s op list.

Step 6: Tune execution

Several #[ktstr_test] attributes control how the VM runs the scenario. The defaults are tuned for fast iteration; tune up for longer / heavier runs:

watchdog_timeout_s is sched_ext’s per-task stall threshold — if a runnable task is not picked for watchdog_timeout_s seconds, the scheduler exits with SCX_EXIT_ERROR_STALL. The scenario duration and watchdog are independent; a 12 s scenario with a 5 s watchdog is normal. Tune the watchdog only when the scheduler under test is expected to legitimately leave a runnable task parked longer than the default 5 s.

For the run we’re building, set the duration to 20 s (so each phase iteration repeats many times):

#[ktstr_test(
    scheduler = KTSTR_SCHED,
    llcs = 2,
    cores = 2,
    threads = 1,
    duration_s = 20,
)]
fn mixed_workloads(ctx: &Ctx) -> Result<AssertResult> {
    // body unchanged from Step 5 -- two cgroups via execute_defs
}

For the full attribute reference (auto-repro, performance mode, topology constraints, etc.), see The #[ktstr_test] Macro.

Step 7: Add assertions

Default checks already run with no configuration – not_starved is Some(true) in Assert::default_checks(), which enables:

• Starvation – any worker with zero work units fails the test.
• Fairness spread – per-cgroup max(off-CPU%) - min(off-CPU%) must stay under the spread threshold (release default 15%; debug default 35% — debug builds in small VMs show higher spread, so the threshold loosens automatically when cfg!(debug_assertions) is true).
• Scheduling gaps – the longest wall-clock gap observed at work-unit checkpoints must stay under the gap threshold (release default 2000 ms; debug default 3000 ms — same cfg!(debug_assertions) gate as spread).

Host-side monitor checks (imbalance ratio, DSQ depth, stall detection, fallback / keep-last event rates) are also enabled by default with thresholds from MonitorThresholds::DEFAULT.

Cpuset isolation is opt-in – enable it with isolation = true. Override the spread threshold and add throughput-parity gates:

use std::time::Duration;
use ktstr::prelude::*;

#[ktstr_test(
    scheduler = KTSTR_SCHED,
    llcs = 2,
    cores = 2,
    threads = 1,
    duration_s = 20,
    isolation = true,
    max_spread_pct = 20.0,
    max_throughput_cv = 0.5,
    min_work_rate = 1.0,
)]
fn mixed_workloads(ctx: &Ctx) -> Result<AssertResult> {
    execute_defs(ctx, vec![
        CgroupDef::named("background_spinner")
            .workers(2)
            .work_type(WorkType::SpinWait)
            .with_cpuset(CpusetSpec::Llc(0)),
        CgroupDef::named("phased_worker")
            .workers(2)
            .work_type(WorkType::sequence(
                Phase::Spin(Duration::from_millis(100)),
                [Phase::Yield(Duration::from_millis(20))],
            ))
            .with_cpuset(CpusetSpec::Llc(1)),
    ])
}

What each new attribute gates:

• isolation = true – workers must only run on CPUs in their assigned cpuset; any execution on an unexpected CPU fails the test.
• max_spread_pct = 20.0 – per-cgroup fairness override (the release default is 15.0; this loosens it slightly to absorb noise from the phased worker’s yield-driven re-placement). Bare max_spread_pct = 15.0 would silently match the default and have no observable effect.
• max_throughput_cv = 0.5 – coefficient of variation of work_units / cpu_time across workers. Catches a scheduler that gives some workers disproportionately less effective CPU.
• min_work_rate = 1.0 – minimum work units per CPU-second per worker. Catches the case where every worker is equally slow (CV passes but absolute throughput is too low).

#[ktstr_test] exposes the full Assert surface (scheduling gaps, monitor thresholds, NUMA locality, wake-latency benchmarks). See Checking for the merge chain (default_checks() -> Scheduler.assert -> per-test) and the complete threshold list.

Step 8: Run it

Run the test with cargo ktstr test, scoped to this one test name:

cargo ktstr test --kernel ../linux -- -E 'test(mixed_workloads)'

If cargo ktstr test reports “kernel not found”, the --kernel path either points at a directory without a built vmlinux or at a kernel the cache cannot locate. Run cargo ktstr kernel build to populate the cache, or pass an explicit path to a built kernel source tree — see Getting Started: Build a kernel for the resolution order.

If a probe-related error surfaces (“probe skeleton load failed”, “trigger attach failed”), re-run with RUST_LOG=ktstr=debug to see the underlying libbpf reason. Common causes: missing tp_btf target on older kernels (handled automatically by the two-phase fallback), CONFIG_DEBUG_INFO_BTF=n in the guest kernel (rebuild with BTF enabled), or a verifier rejection on a non-optional program (the retry surfaces both the original and retry errors so the verifier output is preserved).

cargo ktstr test resolves the kernel image, boots a VM with the declared topology, runs the test as the guest’s init, and reports the result. A passing run looks like:

    PASS [  11.34s] my_crate::mixed_workloads ktstr/mixed_workloads

A failure prints the violated threshold along with per-cgroup stats:

    FAIL [  12.05s] my_crate::mixed_workloads ktstr/mixed_workloads

--- STDERR ---
ktstr_test 'mixed_workloads' [sched=scx-ktstr] [topo=1n2l2c1t] failed:
  unfair cgroup: spread=22% (10-32%) 2 workers on 2 cpus (threshold 20%)

--- stats ---
4 workers, 4 cpus, 12 migrations, worst_spread=22.4%, worst_gap=180ms
  cg0: workers=2 cpus=2 spread=22.4% gap=180ms migrations=8 iter=15230
  cg1: workers=2 cpus=2 spread=4.1% gap=120ms migrations=4 iter=14870

The detail line unfair cgroup: spread=N% (min-max%) N workers on N cpus (threshold N%) is the exact format produced by assert::assert_not_starved. Other detail-line shapes the same producer emits:

• tid {N} starved (0 work units) — when a worker made no progress. Example:

  ktstr_test 'mixed_workloads' [topo=1n2l2c1t] failed:
    tid 2 starved (0 work units)
  

• tid {N} stuck {N}ms on cpu{N} at +{N}ms (threshold {N}ms) — when a worker’s longest off-CPU gap crossed Assert::max_gap_ms. Example:

  ktstr_test 'mixed_workloads' [topo=1n2l2c1t] failed:
    tid 7 stuck 2500ms on cpu3 at +4200ms (threshold 2000ms)
  

• unfair cgroup: spread={pct}% ({lo}-{hi}%) — when per-cgroup fairness exceeded max_spread_pct. Example:

  ktstr_test 'mixed_workloads' [topo=1n2l2c1t] failed:
    unfair cgroup: spread=22% (10-32%) 2 workers on 2 cpus (threshold 20%)
  

The reporting layer does NOT include the cgroup name — cg{i} is the positional index in the stats roll-up (cg0, cg1, …) matching the cg{i}: workers=... cpus=... spread=... per-cgroup stats line emitted by test_support::eval::evaluate_vm_result.

For the full run lifecycle, sidecar layout, and analysis workflow, see Running Tests.

Step 9: Capture a snapshot

Threshold-based assertions tell you something is off; snapshots tell you what the scheduler’s state actually was. Op::snapshot(name) asks the host to freeze every vCPU long enough to read the BPF (in-kernel program) map state, vCPU registers, and per-CPU counters into a FailureDumpReport keyed by name, then resumes the guest.

execute_defs (used so far) takes a flat list of cgroups and runs them concurrently. To inject a snapshot mid-run, switch to execute_steps, which takes a list of Steps — each step has setup cgroups, an ops list (where Op::snapshot(...) lives), and a hold duration:

use ktstr::prelude::*;

#[ktstr_test(scheduler = KTSTR_SCHED, llcs = 2, cores = 2, threads = 1, duration_s = 20)]
fn mixed_workloads(ctx: &Ctx) -> Result<AssertResult> {
    execute_steps(ctx, vec![
        Step {
            setup: Setup::Defs(vec![
                CgroupDef::named("background_spinner")
                    .workers(2)
                    .work_type(WorkType::SpinWait)
                    .with_cpuset(CpusetSpec::Llc(0)),
                CgroupDef::named("phased_worker")
                    .workers(2)
                    .work_type(WorkType::SpinWait)
                    .with_cpuset(CpusetSpec::Llc(1)),
            ]),
            ops: vec![Op::snapshot("after_workload")],
            hold: HoldSpec::FULL,
        },
    ])
}

After the scenario completes, the captured report is keyed by name on the active SnapshotBridge — the host-side store that owns the captured FailureDumpReport map for the run. Downstream test code drains it and walks scalar variables with the dotted-path accessor — e.g. snap.var("nr_cpus_onln").as_u64()? reads a scheduler global (any .bss/.data/.rodata symbol; Snapshot::var walks all three) as a u64.

For the bridge wiring, the full traversal API (Snapshot::map, SnapshotEntry::get, per-CPU narrowing, error variants), and the symbol-driven Op::watch_snapshot variant that fires whenever the guest writes a kernel symbol, see Snapshots.

Step 10: Gauntlet expansion

The #[ktstr_test] macro doesn’t just emit a single test – it also generates a gauntlet of variants that run the same body across every accepted topology preset (single-LLC, multi-LLC, multi-NUMA, with/without SMT).

Gauntlet variants are nextest-discovered and run with cargo ktstr test -- --run-ignored ignored-only -E 'test(gauntlet/)'. Constrain coverage with min_llcs / max_llcs, min_cpus / max_cpus, and requires_smt on the attribute. See Gauntlet Tests for the full filtering and worked examples.

Step 11: Name and prioritize workers

Per-cgroup defaults travel through CgroupDef’s builder methods so schedulers that key on task->comm or task_struct->static_prio can be exercised with realistic, distinguishable workers:

CgroupDef::named("background_spinner")
    .workers(2)
    .comm("bg_spinner")           // prctl(PR_SET_NAME, "bg_spinner")
    .nice(10)                     // setpriority(PRIO_PROCESS, 0, 10)
    .work_type(WorkType::SpinWait)

• .comm("name") — every worker calls prctl(PR_SET_NAME, name) at startup. The kernel truncates task->comm to 15 bytes inside __set_task_comm. Distinguishes workers in top / ps output and in scheduler tracepoints.
• .nice(n) — every worker calls setpriority(PRIO_PROCESS, 0, n). Values below the calling task’s current nice require CAP_SYS_NICE; ktstr always runs as root in-VM so the full -20..=19 range is available. Skips the syscall entirely when .nice(...) is not chained (workers inherit the parent’s nice).
• .pcomm("name") — set the thread-group leader’s task->comm. Triggers ktstr’s fork-then-thread spawn path: workers sharing a pcomm value coalesce into ONE forked leader process whose task->group_leader->comm is name, with worker threads inside it. Models real applications like chrome (pcomm) hosting ThreadPoolForeg (per-thread comm) and java (pcomm) hosting GC Thread / C2 CompilerThre.

pcomm is a WorkSpec field, NOT a CloneMode variant. The two real CloneMode variants are Fork (default; each worker is its own thread group) and Thread (workers share the harness’s tgid as std::thread::spawn threads). pcomm triggers an in-process fork-then-thread shape that combines per-process leader visibility schedulers expect with the in-process thread-spawn dispatch the worker bodies use. PipeIo and CachePipe workers placed in a .pcomm(...) cgroup run as threads inside the pcomm container; their pipe-pair partner indices are computed within the container’s thread group, not across forked siblings. SignalStorm uses tkill (per-task signal delivery, PIDTYPE_PID) rather than kill (PIDTYPE_TGID), so the partner-vs-self addressing is correct uniformly across Fork and Thread modes — including inside pcomm-coalesced thread groups.

Per-WorkSpec overrides win over cgroup-level defaults — write .work(WorkSpec::default().nice(0).comm("hot_spinner")) to opt a specific worker out of the cgroup-level defaults.

Step 12: Inline scheduler config

Schedulers like scx_layered and scx_lavd accept a JSON config via --config /path/to/file.json. Declare the arg template + guest path on a Scheduler const built via the manual builder, then supply the inline content from the test attribute:

const LAYERED_SCHED: Scheduler = Scheduler::new("layered")
    .binary(SchedulerSpec::Discover("scx_layered"))
    .topology(1, 2, 4, 1)
    .config_file_def("--config {file}", "/include-files/layered.json");

const LAYERED_CONFIG: &str = r#"{ "layers": [{ "name": "default" }] }"#;

#[ktstr_test(scheduler = LAYERED_SCHED, config = LAYERED_CONFIG)]
fn layered_default(ctx: &Ctx) -> Result<AssertResult> {
    Ok(AssertResult::pass())
}

The framework writes LAYERED_CONFIG to the guest at the path declared on the scheduler (/include-files/layered.json) and substitutes {file} in the arg template with that path before launching the scheduler binary. A scheduler that declares config_file_def REQUIRES every test to supply config = … (compile-time + runtime gate); a scheduler that doesn’t declare it REJECTS config = … (the content would be silently dropped). See The #[ktstr_test] Macro for the full pairing rules.

For schedulers whose config lives on disk on the host (no inline content), use Scheduler::config_file(host_path) instead — the host file is packed into the initramfs and --config is injected into scheduler args automatically; no config = … on the test is needed in that flavor.

Step 13: Decouple virtual topology from host hardware

By default, ktstr pins vCPUs to host cores in a layout that mirrors the declared virtual topology. A test declaring numa_nodes = 2, llcs = 8 cannot run on a 1-NUMA-node host — the gauntlet preset filter rejects it. Set no_perf_mode = true to drop the host mirroring and run the declared virtual topology unchanged:

#[ktstr_test(
    numa_nodes = 2,
    llcs = 8,             // 8 % 2 == 0; the macro requires divisibility
    cores = 4,
    no_perf_mode = true,  // VM built as declared, even on 1-NUMA hosts
)]
fn two_node_test(ctx: &Ctx) -> Result<AssertResult> { /* ... */ }

In no_perf_mode:

• The VM’s virtual topology is built as declared via KVM vCPU layout, ACPI SRAT/SLIT (x86_64), or FDT cpu nodes (aarch64) — the guest sees the full requested NUMA / LLC structure.
• vCPU-to-host-core pinning, 2 MB hugepages, NUMA mbind, RT scheduling, and KVM exit suppression are skipped.
• Host topology constraints (min_numa_nodes, min_llcs, requires_smt, per-LLC CPU widths) are NOT compared against host hardware. The only host check that survives is “total host CPUs >= total vCPUs”.

no_perf_mode = true is mutually exclusive with performance_mode = true (KtstrTestEntry::validate rejects the combination at runtime). Equivalent to setting KTSTR_NO_PERF_MODE=1 per-test — either source forces the no-perf path. See Performance Mode for the full lifecycle.

Step 14: Periodic capture and temporal assertions

On-demand Op::snapshot (Step 9) captures the scheduler’s BPF state at a point you choose. Periodic capture fires automatically at evenly-spaced points across the workload window, producing a time-ordered SampleSeries (the host-side container of drained snapshots, in capture order; .periodic_only() filters to periodic-tagged samples) for temporal assertions. Periodic capture is only useful when paired with a post_vm callback that drains the bridge and asserts something about the sequence — the two attributes belong together.

Enable periodic capture with num_snapshots = N and register the host-side callback with post_vm = function_name. The callback drains the bridge and runs assertions over the time-ordered series:

use ktstr::prelude::*;

fn check_dispatch_advances(result: &VmResult) -> Result<()> {
    let series = SampleSeries::from_drained(
        result.snapshot_bridge.drain_ordered_with_stats(),
    )
    .periodic_only();

    let mut v = Verdict::new();

    let nr_dispatched: SeriesField<u64> = series.bpf(
        "nr_dispatched",
        |snap| snap.var("nr_dispatched").as_u64(),
    );
    nr_dispatched.nondecreasing(&mut v);

    let r = v.into_result();
    anyhow::ensure!(r.passed, "temporal assertions failed: {:?}", r.details);
    Ok(())
}

#[ktstr_test(
    scheduler = KTSTR_SCHED,
    llcs = 2,
    cores = 2,
    threads = 1,
    duration_s = 20,
    num_snapshots = 5,
    post_vm = check_dispatch_advances,
)]
fn dispatch_advances(ctx: &Ctx) -> Result<AssertResult> {
    execute_defs(ctx, vec![
        CgroupDef::named("workers").workers(2).work_type(WorkType::SpinWait),
    ])
}

num_snapshots = 5 fires 5 freeze-and-capture boundaries inside the 10%-90% window of the 20 s workload — at roughly +5 s, +7 s, +10 s, +13 s, +15 s. Each capture freezes every vCPU, reads BPF map state, and resumes. The host watchdog deadline is extended by each freeze duration so captures do not eat into the workload budget. The captures are stored under periodic_000…periodic_004 on the SnapshotBridge.

Verdict is the assertion accumulator: every pattern call records its outcome on the same Verdict, and v.into_result() consumes it into a pass/fail AssertResult.

The seven temporal patterns on SeriesField:

Every pattern method takes &mut Verdict as its first argument and returns it, so calls chain into the same accumulator.

SeriesField::each provides per-sample scalar bounds: field.each(&mut v).at_least(1u64), field.each(&mut v).between(0.0, 100.0).

When a temporal pattern fails, the AssertDetail entries identify the offending sample by tag and elapsed-ms timestamp. Example for nondecreasing flagging a regression on nr_dispatched:

nr_dispatched (nondecreasing): regression at sample periodic_002 (+10000ms): \
value 41 after prior value 42 at sample periodic_001 (+7000ms)

The rate, steady, converges, ratio, and always-true variants emit parallel shapes — every detail names the pattern, the specific sample(s) involved, and the violating value, so a failing test points at the data without re-running.

For boundary timing, spacing rules, and the bridge cap, see Periodic Capture. For the full projection API (bpf, stats, auto-projectors) and failure rendering, see Temporal Assertions.

Step 15: After the run — test statistics

cargo ktstr stats aggregates the sidecar JSON files that each test variant writes — useful for tracking gauntlet coverage, BPF verifier complexity, and scheduling behavior across commits. This is a post-run CLI workflow, not part of the test definition:

cargo ktstr stats                                 # summary: gauntlet coverage, verifier, KVM stats
cargo ktstr stats list                            # list runs with date, test count, arch
cargo ktstr stats compare --a-kernel 6.14 \       # diff sidecar partitions defined by
    --b-kernel 6.15                               #   per-side --a-X / --b-X filter flags

Statistics are collected even on test failure (if: !cancelled() in CI). For the full subcommand surface, see cargo-ktstr stats.

The complete test

The shape exercised by every step above, in one file. sched_args = ["--slow"] always-applies scx-ktstr’s --slow mode (Step 2); watchdog_timeout_s = 10 overrides the sched_ext stall threshold (Step 6); num_snapshots + post_vm enable periodic capture and a temporal assertion (Step 14):

use std::time::Duration;
use ktstr::declare_scheduler;
use ktstr::prelude::*;

declare_scheduler!(KTSTR_SCHED, {
    name = "ktstr_sched",
    binary = "scx-ktstr",
    sched_args = ["--slow"],
});

fn check_dispatch_advances(result: &VmResult) -> Result<()> {
    let series = SampleSeries::from_drained(
        result.snapshot_bridge.drain_ordered_with_stats(),
    )
    .periodic_only();

    let mut v = Verdict::new();

    let nr_dispatched: SeriesField<u64> = series.bpf(
        "nr_dispatched",
        |snap| snap.var("nr_dispatched").as_u64(),
    );
    nr_dispatched.nondecreasing(&mut v);

    let r = v.into_result();
    anyhow::ensure!(r.passed, "temporal assertions failed: {:?}", r.details);
    Ok(())
}

#[ktstr_test(
    scheduler = KTSTR_SCHED,
    llcs = 2,
    cores = 2,
    threads = 1,
    duration_s = 20,
    watchdog_timeout_s = 10,
    isolation = true,
    max_spread_pct = 20.0,
    max_throughput_cv = 0.5,
    min_work_rate = 1.0,
    num_snapshots = 5,
    post_vm = check_dispatch_advances,
)]
fn mixed_workloads(ctx: &Ctx) -> Result<AssertResult> {
    execute_defs(ctx, vec![
        CgroupDef::named("background_spinner")
            .workers(2)
            .work_type(WorkType::SpinWait)
            .with_cpuset(CpusetSpec::Llc(0)),
        CgroupDef::named("phased_worker")
            .workers(2)
            .work_type(WorkType::sequence(
                Phase::Spin(Duration::from_millis(100)),
                [Phase::Yield(Duration::from_millis(20))],
            ))
            .with_cpuset(CpusetSpec::Llc(1)),
    ])
}

Run it:

cargo ktstr test --kernel ../linux -- -E 'test(mixed_workloads)'

What you’ll see when things break

The output examples below are the shapes ktstr emits in real runs. They’re worth skimming before you ship a test so a future failure is recognisable.

Auto-repro probe chain

When the scheduler crashes, ktstr re-runs the scenario with BPF probes attached and dumps the path leading to the exit. Decoded struct fields appear inline, with → between fentry-captured entry values and fexit-captured exit values:

ktstr_test 'demo_host_crash_auto_repro' [sched=scx-ktstr] [topo=1n1l2c1t] failed:
  scheduler died

--- auto-repro ---
=== AUTO-PROBE: scx_exit fired ===

  ktstr_enqueue                                                   main.bpf.c:21
    task_struct *p
      pid         97
      cpus_ptr    0xf(0-3)
      dsq_id      SCX_DSQ_INVALID
      enq_flags   NONE
      slice       0
      vtime       0
      weight      100
      sticky_cpu  -1
      scx_flags   QUEUED|ENABLED
  do_enqueue_task                                               kernel/sched/ext.c:1344
    rq *rq
      cpu         1
    task_struct *p
      pid         97
      cpus_ptr    0xf(0-3)
      dsq_id      SCX_DSQ_INVALID          →  SCX_DSQ_LOCAL
      enq_flags   NONE
      slice       20000000
      vtime       0
      weight      100
      sticky_cpu  -1
      scx_flags   QUEUED|DEQD_FOR_SLEEP    →  QUEUED

For the probe pipeline architecture, the BTF resolution path, event-stitching rules, and the demo_host_crash_auto_repro fixture, see Auto-Repro.

Failure dumps with cast-recovered pointers

The freeze coordinator builds a FailureDumpReport on every snapshot, periodic capture, and post-failure dump. Each captured map prints as a map <name> (type=..., value_size=..., max_entries=...) header followed by the rendered value (single-entry global sections like .bss/.data) or entry: key=... blocks (multi-entry maps). u64 fields the cast analyzer flagged as typed pointers chase to the recovered struct and print with a (cast→arena) or (cast→kernel) annotation distinguishing them from BTF-typed pointers; an (sdt_alloc) suffix is added when the sdt_alloc bridge recovered the real payload struct from a forward-declared pointee. A separate cross-BTF Fwd resolution path also recovers a forward-declared pointee whose body lives in a sibling embedded BPF object’s BTF — that path adds no annotation, the body is rendered transparently:

map scx_lavd.bss (type=array, value_size=4096, max_entries=1)
.bss:
  nr_cpus_onln=4
  task_ctx_root 0xffff888103a01000 (cast→arena) → task_ctx{cpu_id=2, last_runtime_ns=12345678, nice=0}
  current_task 0xffff90124f80c000 (cast→kernel) → task_struct:
    pid=4321   weight=100
    cpus_ptr 0xffff888103b40000 → cpus={0-3}
  taskc_data 0x7f0000080000 (cast→arena (sdt_alloc)) → task_data{slice_ns=20000000, vtime=12345678}

A field that the analyzer cannot prove is a pointer falls back to its raw u64 shape, which is the prior behavior — no test-author configuration is required either way.

Verifier output

cargo ktstr verifier runs the BPF verifier against every declare_scheduler!-registered scheduler’s struct_ops programs inside a real kernel and prints per-program verified-instruction counts. The dispatcher hands off to cargo nextest run -E 'test(/^verifier/)'; nextest fans out across (scheduler × declared kernel × accepted topology preset) cells, each cell booting its own VM. Per-cell output starts with a banner identifying the axis values:

=== ktstr_sched | kernel kernel_6_14_2 | topology tiny-1llc ===

verifier
  enqueue                                  verified_insns=42

verifier --- verifier stats ---
  processed=42  states=8/10

verifier --- scheduler log ---
func#0 @0
0: R1=ctx() R10=fp0
processed 42 insns (limit 1000000) max_states_per_insn 1 total_states 10 peak_states 8 mark_read 5

When the scheduler did not capture a log, the output is just the per-program table:

=== ktstr_sched | kernel kernel_6_14_2 | topology tiny-1llc ===

verifier
  enqueue                                  verified_insns=500
  dispatch                                 verified_insns=1200
  init                                     verified_insns=300

--raw disables cycle collapsing in the scheduler-log section. --kernel A --kernel B runs the sweep against multiple kernels; the cell handler walks KTSTR_KERNEL_LIST to match each cell’s sanitized kernel label against the resolved set. For the full verifier-sweep model, cycle-collapse rules, and the cell-name → kernel matching contract, see Verifier.

What’s next

• Custom Scenarios – when the declarative ops API is not enough and the scenario needs arbitrary Rust logic between phases.
• Ops and Steps – multi-phase scenarios: add/remove cgroups, swap cpusets, freeze, resume.
• Watch Snapshots – Op::watch_snapshot("symbol") registers a hardware data-write watchpoint (up to 3 per scenario; slot 0 is reserved for the error-class exit_kind trigger).
• MemPolicy – NUMA-aware tests that bind memory allocations to specific nodes and check page locality.
• Performance Mode – pinned vCPUs, hugepages, and LLC-exclusivity validation for benchmark-grade runs.
• Auto-Repro – on a scheduler crash, ktstr can boot a second VM with probes attached and dump the failing state automatically.
• Recipes – task-specific guides (test a new scheduler, A/B compare branches, customize checking, benchmarking, host-state diff, ctprof).

Running Tests

Tests run via cargo ktstr test --kernel ../linux, which resolves the kernel and wraps cargo nextest run to boot KVM virtual machines for each #[ktstr_test] entry. Raw cargo nextest run remains available as a fallback once a kernel is in place via the discovery chain.

Quick reference

# Run all tests
cargo ktstr test --kernel ../linux

# Run a specific test
cargo ktstr test --kernel ../linux -- -E 'test(sched_basic_proportional)'

# Run ignored gauntlet tests
cargo ktstr test --kernel ../linux -- --run-ignored ignored-only -E 'test(gauntlet/)'

Run analysis

Each test invocation writes a *.ktstr.json sidecar per variant into {CARGO_TARGET_DIR or "target"}/ktstr/{kernel}-{project_commit}/. cargo ktstr stats list enumerates runs; cargo ktstr stats compare, list-values, list-metrics, and show-host operate on those sidecars. See Runs for the directory layout, last-writer-wins semantics, and the comparison workflow.

Budget-based test selection

Set KTSTR_BUDGET_SECS to select the subset of tests that maximizes feature coverage within a time budget. Useful for CI pipelines or quick smoke tests.

# Run the best 5 minutes of tests
KTSTR_BUDGET_SECS=300 cargo ktstr test --kernel ../linux

# Budget applies to gauntlet variants too
KTSTR_BUDGET_SECS=600 cargo ktstr test --kernel ../linux -- --run-ignored all

The selector encodes each test as a bitset of properties (scheduler, topology class, SMT, workload characteristics) and greedily picks tests with the highest marginal coverage per estimated second. Duration estimates account for VM boot overhead based on vCPU count.

A summary is printed to stderr during --list:

ktstr budget: 42/1200 tests, 295/300s used, 38/38 configurations covered

When KTSTR_BUDGET_SECS is not set, all tests are listed as usual.

Custom scheduler

Declare a scheduler with declare_scheduler! and reference the bare const from #[ktstr_test(scheduler = ...)]:

use ktstr::declare_scheduler;
use ktstr::prelude::*;

declare_scheduler!(MY_SCHED, {
    name = "my_sched",
    binary = "scx_my_sched",
});

#[ktstr_test(scheduler = MY_SCHED)]
fn my_sched_test(ctx: &Ctx) -> Result<AssertResult> {
    Ok(AssertResult::pass())
}

The binary is injected into the VM’s initramfs and started before scenarios run. See Test a New Scheduler for the full end-to-end workflow, and Payload Definitions for the #[derive(Payload)] macro that handles binary-kind workloads (schbench, fio, etc.) — distinct from the scheduler-under-test surface.

Single Scenario

Running a specific test

cargo ktstr test --kernel ../linux -- -E 'test(sched_basic_proportional)'

Running with verbose output

RUST_BACKTRACE=1 cargo ktstr test --kernel ../linux -- -E 'test(sched_basic_proportional)'

Investigating failures

Run one test with verbose output to see scheduler logs and kernel console:

RUST_BACKTRACE=1 cargo ktstr test --kernel ../linux -- -E 'test(cover_cgroup_cpuset_cross_llc_race)'

VM topology

Each #[ktstr_test] declares its topology via macro attributes:

#[ktstr_test(llcs = 2, cores = 4, threads = 2)]

The test framework boots a VM with the specified topology automatically.

See Investigate a Crash for interpreting failure output and Troubleshooting for common error messages.

Gauntlet

The gauntlet runs every test across 24 topology presets (14 on aarch64). Gauntlet variants are prefixed with gauntlet/ and ignored by default.

# Run only base tests (default)
cargo ktstr test --kernel ../linux

# Run only gauntlet variants
cargo ktstr test --kernel ../linux -- --run-ignored ignored-only -E 'test(gauntlet/)'

# Run everything
cargo ktstr test --kernel ../linux -- --run-ignored all

Entries with host_only = true never produce gauntlet variants (topology variation is meaningless without a VM). See host_only for how that flag is set.

Variant naming

Single-kernel runs name each gauntlet variant gauntlet/{test_name}/{preset}:

• {test_name} – the #[ktstr_test] function name
• {preset} – one of the topology preset names below

When --kernel resolves to two or more kernels (multiple --kernel flags or a START..END range that expands to several releases), cargo ktstr test / coverage / llvm-cov add the kernel as a third dimension and append a {kernel_label} suffix: gauntlet/{test_name}/{preset}/{kernel_label}. See Multi-kernel: kernel as a gauntlet dimension for how the kernel labels are derived (sanitized from the resolved version, range expansion, cache key, git source, or path basename).

To run a single variant:

cargo ktstr test --kernel ../linux -- --run-ignored ignored-only \
  -E 'test(=gauntlet/my_test/smt-2llc)'

Topology presets

Topology format: {numa_nodes}n{llcs}l{cores_per_llc}c{threads_per_core}t (e.g. 1n2l4c2t = 1 NUMA node, 2 LLCs, 4 cores per LLC, 2 threads per core = 16 CPUs). Presets are defined in gauntlet_presets(). Multi-NUMA presets are excluded by default (max_numa_nodes: Some(1) in TopologyConstraints::DEFAULT), so tests opt in to NUMA testing by raising max_numa_nodes.

aarch64: ARM64 CPUs do not have SMT. Presets with threads_per_core > 1 are excluded on aarch64, leaving 14 presets (the 5 small presets, 6 -nosmt variants, and 3 non-SMT NUMA presets).

Constraint filtering

#[ktstr_test] topology constraints filter which presets a test runs on. A preset is skipped when any constraint is not met:

• num_numa_nodes() < min_numa_nodes
• max_numa_nodes is set and num_numa_nodes() > max_numa_nodes
• num_llcs() < min_llcs
• max_llcs is set and num_llcs() > max_llcs
• requires_smt and threads_per_core < 2
• total_cpus() < min_cpus
• max_cpus is set and total_cpus() > max_cpus

See Topology Constraints for the full attribute table and Gauntlet Tests for a worked example showing which presets survive a given constraint set.

Budget interaction

When KTSTR_BUDGET_SECS is set, greedy coverage maximization selects the most diverse set of test configurations within the time budget. Each candidate test is represented as a feature bitset (CPU count bucket, LLC count, SMT vs non-SMT, etc.). The selector greedily picks tests that cover the most uncovered feature bits per estimated second. The result is a mix of base tests and gauntlet variants that maximizes configuration diversity within the budget.

See Budget-based test selection.

Memory allocation

Each gauntlet VM gets max(topology_mb, initramfs_floor) MB of RAM, where topology_mb = max(cpus * 64, 256, entry.memory_mb) is the topology-requested minimum and initramfs_floor is computed from the actual initramfs size after build. For max-cpu (252 CPUs) the topology minimum is at least 16128 MB.

Runs

Each cargo ktstr test --kernel ../linux invocation writes per-test result sidecars into a run directory under {CARGO_TARGET_DIR or "target"}/ktstr/. The directory is the record of the latest test run for that (kernel, project commit) pair – there is no separate “baselines” cache.

Warning: Re-running the suite at the same kernel and project commit reuses the same directory and deletes prior sidecars at the first sidecar write of the new run. To preserve a previous run’s outputs, archive the directory elsewhere first (e.g. mv target/ktstr/6.14-abc1234 target/ktstr/6.14-abc1234.archived-{date}) or commit your changes (or amend to drop a -dirty suffix) so the next run lands in a separate snapshot directory.

Layout

target/
└── ktstr/
    ├── 6.14-abc1234/        # one run: kernel 6.14, project commit abc1234 (clean)
    │   ├── test_a.ktstr.json
    │   └── test_b.ktstr.json
    └── 7.0-def5678-dirty/   # another run: kernel 7.0, project commit def5678 with uncommitted changes
        ├── test_a.ktstr.json
        └── test_b.ktstr.json

Each subdirectory is keyed {kernel}-{project_commit}, where {kernel} is the kernel version resolved from the directory KTSTR_KERNEL points at — first the version field in its metadata.json, else the content of include/config/kernel.release, else unknown (when KTSTR_KERNEL is unset or neither file yields a version) — and {project_commit} is the project tree’s HEAD short hex (7 chars), suffixed -dirty when the worktree differs from HEAD, or the literal unknown when the test process is not running inside a git repository.

The commit is discovered by walking parents of the test process’s working directory until a .git marker is found — for a scheduler crate using ktstr as a dev-dependency, this is the scheduler crate’s commit, not ktstr’s. The function that performs the probe (detect_project_commit) is called from the test process’s cwd, so running tests from inside the scheduler crate’s clone yields that crate’s HEAD. Run from inside ktstr’s clone if you want to record ktstr’s HEAD instead.

Two runs sharing the same kernel and project commit (the typical “re-run the suite without committing changes” loop) reuse the same directory: the second run pre-clears any prior *.ktstr.json files in the directory at first sidecar write so the directory is a last-writer-wins snapshot of (kernel, project commit), not an append-only archive of every invocation. Re-run the suite to regenerate the sidecars; commit your changes (or amend to drop the -dirty suffix) to land a separate snapshot directory.

Pre-clear is shallow — only *.ktstr.json files in the immediate run directory are removed. Subdirectories created by external orchestrators (per-job gauntlet layouts, cluster shards) are left untouched, but cargo ktstr stats walks one level of subdirectories when collecting sidecars, so stale sidecar files left in subdirectories from a prior run will still appear in stats output. Operators driving subdirectory layouts must clean those subdirectories themselves; pre-clear’s contract covers the top-level only.

Filesystem requirement

The runs root must reside on a local filesystem (ext4, xfs, btrfs, tmpfs). NFS and other remote filesystems are rejected by the advisory lock used for cross-process sidecar-write serialization.

Unknown-commit collisions

When the test process is not inside a git repository (so detect_project_commit returns None), the on-disk dirname uses the literal sentinel unknown in the commit slot — every such run lands in {kernel}-unknown. Concurrent or successive non-git runs collide on this single directory, with the latest run pre-clearing the previous one’s sidecars. To disambiguate non-git runs, set KTSTR_SIDECAR_DIR to a per-run path or place the project tree under git so each run carries its own commit hash.

ktstr emits a one-shot stderr warning on first sidecar write under this configuration; setting KTSTR_SIDECAR_DIR both disambiguates the run and suppresses the warning (the override branch returns from sidecar_dir before the warning site is reached).

The unknown sentinel applies to the dirname only. The in-memory SidecarResult.project_commit field stays None (serialized as JSON null) for these runs — the dirname uses a filesystem-safe sentinel, while the JSON field preserves the original probe outcome. As a consequence, cargo ktstr stats compare --project-commit unknown will not match a sidecar whose project_commit is None; omit the --project-commit filter entirely to include None-commit rows in the comparison.

KTSTR_SIDECAR_DIR overrides the sidecar directory itself (used as-is, no key suffix), not the parent. The override only affects where new sidecars are written and what bare cargo ktstr stats reads. When the override is set, pre-clear is skipped — the operator chose that directory and owns its contents, so any pre-existing sidecars there are preserved. cargo ktstr stats list, cargo ktstr stats compare, cargo ktstr stats list-values, and cargo ktstr stats show-host all walk {CARGO_TARGET_DIR or "target"}/ktstr/ by default — pass --dir DIR on compare / list-values / show-host to point them at an alternate run root (e.g. an archived sidecar tree copied off a CI host). They do NOT consult KTSTR_SIDECAR_DIR.

Workflow

1. Run tests for kernel A:

   cargo ktstr test --kernel 6.14
   

2. Run again for kernel B:

   cargo ktstr test --kernel 7.0
   

3. List runs:

   cargo ktstr stats list
   

   Each row carries RUN, TESTS, DATE, and ARCH columns. DATE is the earliest sidecar timestamp present in the directory — under the last-writer-wins semantics, this equals the most recent run’s first sidecar timestamp (the prior run’s sidecars were pre-cleared at the new run’s first write, so only the new run’s timestamps remain). ARCH is the host.arch value (x86_64, aarch64, …) from the run’s first sidecar, or - when no sidecar carries a populated host context. Rows are ordered by directory mtime, most recent first.

4. Compare across dimensions:

   cargo ktstr stats compare --a-kernel 6.14 --b-kernel 7.0
   cargo ktstr stats compare --a-kernel 6.14 --b-kernel 7.0 -E cgroup_steady
   cargo ktstr stats compare --a-scheduler scx_rusty --b-scheduler scx_lavd --kernel 6.14
   cargo ktstr stats compare --a-project-commit abcdef1 --b-project-commit fedcba2
   cargo ktstr stats compare --a-project-commit abc1234 --b-project-commit abc1234-dirty
   cargo ktstr stats compare --a-kernel-commit abcdef1 --b-kernel-commit fedcba2
   cargo ktstr stats compare --a-run-source ci --b-run-source local
   

   The abc1234 vs abc1234-dirty row is the canonical WIP-vs-baseline pattern: run the suite once at a clean commit to capture the baseline directory {kernel}-abc1234, edit the tree without committing, run the suite again to capture {kernel}-abc1234-dirty, then diff the two. Both sidecar pools coexist under target/ktstr/ because the -dirty suffix makes them distinct directories.

   Per-side filters (--a-* / --b-*) partition the sidecar pool into two sides; shared filters (--kernel, --scheduler, --project-commit, --kernel-commit, --run-source, etc.) pin both sides. The seven slicing dimensions are kernel, scheduler, topology, work-type, project-commit, kernel-commit, and run-source; differing on any subset of them defines the A/B contrast. Per-metric deltas are computed using the unified MetricDef registry (polarity, absolute and relative thresholds). Output is colored: red for regressions, green for improvements. The command exits non-zero when regressions are detected. Use cargo ktstr stats list-values to discover available dimension values before constructing a comparison.

5. Print analysis for the most recent run (no subcommand):

   cargo ktstr stats
   

   Picks the newest subdirectory under target/ktstr/ by mtime and prints gauntlet analysis, BPF verifier stats, callback profile, and KVM stats.

6. Inspect the archived host context for a specific run:

   cargo ktstr stats show-host --run 6.14-abc1234
   cargo ktstr stats show-host --run archive-2024-01-15 --dir /tmp/archived-runs
   

   Resolves --run against target/ktstr/ (or --dir when set), scans the run’s sidecars in order, and renders the first populated host-context field via HostContext::format_human: CPU model, memory config, transparent-hugepage policy, NUMA node count, uname triple, kernel cmdline, and every /proc/sys/kernel/sched_* tunable. Same fingerprint stats compare uses for its host-delta section, but available on a single run. Fails with an actionable error when no sidecar carries a host field (pre-enrichment run).

Metric registry discovery

Before configuring per-metric ComparisonPolicy overrides, enumerate the available metric names:

cargo ktstr stats list-metrics
cargo ktstr stats list-metrics --json

Prints the ktstr::stats::METRICS registry: metric name, polarity (higher / lower better), default_abs and default_rel gate thresholds, and display unit. Use the metric names from this list as keys in ComparisonPolicy.per_metric_percent; unknown names are rejected at --policy load time so typos surface loudly. --json emits the same data as a serde array — the row accessor function is omitted (#[serde(skip)]) so the wire surface carries only wire-stable fields.

Sidecar format

Each test writes a SidecarResult JSON file containing the test name, topology, scheduler, work type, pass/fail, per-cgroup stats, monitor summary, stimulus events, verifier stats, KVM stats, effective sysctls, kernel command-line args, kernel version, timestamp, and run ID. Files are named with a .ktstr. infix for discovery. cargo ktstr stats reads all sidecar files from a run directory (recursing one level for gauntlet per-job subdirectories).

See also: KTSTR_SIDECAR_DIR.

ktstr

ktstr is the standalone debugging companion to the #[ktstr_test] test harness. It owns kernel cache management, interactive VM shells, host-wide per-thread profiling, and lock introspection — the operations a scheduler author reaches for when investigating a test failure.

To reproduce a test scenario as a self-contained shell script without a VM, use cargo ktstr export. To run the test suite, use cargo ktstr test.

See also cargo ktstr for the cargo-integrated companion that also covers test execution, coverage, BPF verifier stats, and gauntlet statistics.

Build from the workspace:

cargo build --bin ktstr

Subcommands

topo

Show the host CPU topology (CPUs, LLCs, NUMA nodes):

ktstr topo

kernel

The kernel subcommand manages cached kernel images. Subcommands: list, build, clean. See cargo-ktstr kernel for full documentation – the kernel subcommands are identical in both binaries.

shell

Boot an interactive shell in a KVM virtual machine. Launches a VM with busybox and drops into a shell.

ktstr shell
ktstr shell --kernel ../linux
ktstr shell --kernel 6.14.2
ktstr shell --topology 1,2,4,1
ktstr shell -i /path/to/binary
ktstr shell -i my_tool -i another_tool

Files and directories passed via -i are available at /include-files/<name> inside the guest. Directories are walked recursively, preserving structure (e.g. -i ./release includes all files under release/ at /include-files/release/...). Bare names (without path separators) are resolved via PATH lookup. Dynamically-linked ELF binaries get automatic shared library resolution via ELF DT_NEEDED parsing. Non-ELF files are copied as-is.

Stdin is a terminal requirement. The host terminal enters raw mode for bidirectional stdin/stdout forwarding. Terminal state is restored on all exit paths.

cargo ktstr shell runs the same VM boot flow and differs in one respect: it accepts raw image file paths for --kernel (e.g. bzImage, Image). Source-tree directories auto-build and no-kernel invocations auto-download — same as ktstr shell.

ctprof

Capture or compare a host-wide per-thread state snapshot. Useful for diagnosing “the scheduler looks fine but something on the host is still behaving oddly” by producing a baseline/candidate diff of every live thread’s scheduling, memory, and I/O counters — a superset of what any single test’s sidecar captures.

ktstr ctprof capture --output baseline.ctprof.zst
# ... run workload of interest ...
ktstr ctprof capture --output candidate.ctprof.zst
ktstr ctprof compare baseline.ctprof.zst candidate.ctprof.zst

capture walks /proc at capture time and writes every visible thread’s metric values (cumulative counters from schedstat / sched / status CSW / page faults / I/O bytes / taskstats; lifetime peaks from schedstat *_max and hiwater_*; instantaneous gauges sampled at capture time including nr_threads, fair_slice_ns, state; categorical / ordinal scalars including policy, nice, cpu_affinity, identity strings) as zstd-compressed JSON (conventional extension .ctprof.zst). Cumulative counters and lifetime peaks are probe-timing-invariant — sampled twice, the value either monotonically increased or stayed at its high-water mark — so a diff between two snapshots measures exactly the activity over the window. Instantaneous gauges and categorical scalars are point-in-time readings that can legitimately differ between two probes of the same thread. Per-cgroup aggregates (cpu.stat, memory.current) are captured once per distinct path. Capture is read-only; nothing is attached, no kprobes, no tracing.

compare joins two snapshots on the selected grouping axis (pcomm by default) and renders a per-metric baseline/candidate/delta table. The join key survives across captures taken on different hosts or after process restarts, so deltas reflect the behavior of the named workload rather than a specific pid. Metrics with cumulative semantics (CPU time, page faults, wait time) show the candidate-minus-baseline delta; instantaneous metrics (affinity, cgroup path) show the value at candidate capture time. See the ctprof reference for the full metric registry, aggregation rules, derived-metric formulas, and taskstats kconfig gating.

show renders a single snapshot’s per-(group, metric) values without diff math. Same flag vocabulary as compare, minus the baseline/candidate/delta/pct columns:

ktstr ctprof show snapshot.ctprof.zst --group-by cgroup
ktstr ctprof show snapshot.ctprof.zst --sections taskstats-delay

metric-list prints every registered metric (primary + derived) with its description, unit, kconfig gate, and sched_class scope. Use this to discover the vocabulary --sort-by and --metrics accept.

ktstr ctprof metric-list

locks

Enumerate every ktstr flock held on this host. Read-only — does NOT attempt any flock acquire. Useful as a troubleshooting companion for --cpu-cap contention: when a build or test is stalled behind a peer’s reservation, ktstr locks names the peer (PID + cmdline) without disturbing any of its flocks.

Scans four lock-file roots:

• /tmp/ktstr-llc-*.lock — per-LLC reservations held by perf-mode test runs and --cpu-cap-bounded builds.
• /tmp/ktstr-cpu-*.lock — per-CPU reservations from the same flow.
• {cache_root}/.locks/*.lock — cache-entry locks held during kernel build writes, and source-{path_hash}.lock files held for the duration of kernel build --source and cargo ktstr test --kernel <path> against the same source tree.
• {runs_root}/.locks/{kernel}-{project_commit}.lock — per-run-key sidecar-write locks held for the duration of the (pre-clear + write) cycle to serialize concurrent ktstr processes targeting the same run directory.

Each lock is cross-referenced against /proc/locks to name the holder PID and cmdline.

ktstr locks                       # one-shot snapshot
ktstr locks --json                # JSON snapshot
ktstr locks --watch 1s            # redraw every second until SIGINT
ktstr locks --watch 1s --json     # ndjson stream, one object per interval

The same subcommand is available as cargo ktstr locks with identical flag semantics.

completions

Generate shell completions for ktstr.

ktstr completions bash
ktstr completions zsh
ktstr completions fish

The same subcommand is available as cargo ktstr completions with identical flag semantics (--binary accepted on both; defaults to the respective binary name).

cargo-ktstr

cargo ktstr is a cargo plugin for kernel build, cache, and test workflow. Subcommands in --help order: test (alias: nextest), coverage, llvm-cov, stats, kernel, model, verifier, funify (alias: costume), completions, show-host, show-thresholds, export, locks, shell.

test

Build the kernel (if needed) and run tests via cargo nextest run. Also available as cargo ktstr nextest — a visible clap alias that expands to the same subcommand, so the two forms are interchangeable.

cargo ktstr test                                               # auto-discover kernel
cargo ktstr test --kernel ../linux                             # local source tree
cargo ktstr test --kernel 6.14.2                               # version (auto-downloads on miss)
cargo ktstr test --kernel 6.14.2-tarball-x86_64-kc...          # cache key (from kernel list)
cargo ktstr test --kernel 6.12..6.14                           # range: every stable+longterm release in [6.12, 6.14]
cargo ktstr test --kernel git+https://example.com/r.git#v6.14  # git URL + ref (tag/branch)
cargo ktstr test --kernel git+https://example.com/r.git#deadbeef1234  # specific commit
cargo ktstr test --kernel 6.14.2 --kernel 6.15.0               # multi-kernel: repeatable
cargo ktstr test --release                                     # release profile (stricter assertions)

--kernel is repeatable and accepts a path, version string, cache key, version range (START..END), or git source (git+URL#REF). When absent, the test framework discovers a kernel from KTSTR_TEST_KERNEL, then KTSTR_KERNEL, then falls back to cache and filesystem lookup. When --kernel is a path, cargo-ktstr configures and builds the kernel before running tests. Version strings auto-download and build on cache miss (both explicit patch versions like 6.14.2 and major.minor prefixes like 6.14). Cache keys resolve from the cache only — they error if not cached (run cargo ktstr kernel list to see available keys).

Ranges (START..END) expand against kernel.org’s releases.json to every stable and longterm release whose version sits inside [START, END] inclusive (mainline / linux-next rows are dropped). The endpoints themselves do NOT need to appear in releases.json — 6.10..6.16 brackets the surviving releases even if 6.10 and 6.16 have aged out.

Git sources (git+URL#REF) clone the repo shallow at the given ref, build, and cache the result. A repeat invocation against an unchanged branch tip lands a cache hit; a moved tip rebuilds.

Multi-kernel: kernel as a gauntlet dimension

When --kernel resolves to two or more kernels (multiple --kernel flags, or a single --kernel START..END range that expands to several releases), cargo-ktstr resolves all kernels upfront and exports the resolved set to cargo nextest via the KTSTR_KERNEL_LIST env var. The test binary’s gauntlet expansion adds the kernel as an additional dimension to the gauntlet cross-product, so each (test × scenario × topology × kernel) tuple becomes a distinct nextest test case. Two name shapes carry the kernel suffix:

• Base tests: ktstr/{name}/{kernel_label} — one variant per registered #[ktstr_test] per kernel.
• Gauntlet variants: gauntlet/{name}/{preset}/{kernel_label} — one variant per (test × topology preset × kernel).

Single-kernel runs (zero or one resolved kernel) keep the historical name shapes ktstr/{name} and gauntlet/{name}/{preset} with no kernel suffix, so existing CI baselines and per-test config overrides keep matching.

Kernel labels are semantic, operator-readable identifiers sanitized to kernel_[a-z0-9_]+:

• Version / range expansion → kernel_6_14_2, kernel_6_15_rc3
• Cache key → version prefix only (kernel_6_14_2 from 6.14.2-tarball-x86_64-kc<hash>)
• Git source → kernel_git_{owner}_{repo}_{ref} (e.g. kernel_git_tj_sched_ext_for_next from git+https://github.com/tj/sched_ext#for-next)
• Path → kernel_path_{basename}_{hash6} (e.g. kernel_path_linux_a3f2b1); the 6-char crc32 of the canonical path disambiguates two linux directories under different parents. Dirty-tree builds (uncommitted source changes, mid-build worktree mutations, or non-git trees) append _dirty to the label — e.g. kernel_path_linux_a3f2b1_dirty — so the test report distinguishes the non-reproducible run from a subsequent clean rebuild of the same path.
• Local cache entry → kernel_local_{hash6} (first 6 chars of the source tree’s git short_hash, captured at cache-store time) or kernel_local_unknown for non-git trees. The hash6 keeps two distinct local trees from collapsing to the same label; the unknown literal is the shared bucket for every non-git tree (no discriminator exists at the cache layer to spread them apart).

Filter with nextest’s -E 'test(kernel_6_14)' to pick a single kernel from a multi-kernel matrix; nextest’s parallelism, retries, and --ignored flag all apply natively. Sidecars partition per kernel: each kernel runs in its own target/ktstr/{kernel}-{project_commit}/ directory keyed on the resolved kernel’s identity and the project tree’s HEAD short hex (with -dirty suffix when the worktree differs). Coverage profraw does NOT partition per kernel — __llvm_profile_write_buffer writes flat into target/llvm-cov-target/ with PID-keyed filenames (ktstr-test-{pid}-{counter}.profraw), and cargo-llvm-cov merges every variant’s profraw automatically into the single output report.

Build / download / clone failures abort BEFORE any test runs — a missing kernel can’t be tested, and continuing would mask which kernel was requested-but-unavailable in the operator-visible error stream. Test failures within a kernel are nextest-handled normally.

host_only tests under multi-kernel: tests marked host_only (those that run on the host without booting a VM) skip the kernel suffix and list / run once regardless of KTSTR_KERNEL_LIST cardinality. The dispatch sites (list_tests, list_tests_budget, and --exact’s run_host_only_test in src/test_support/dispatch.rs) all gate on entry.host_only before consulting the resolved kernel set, so a host-side test never observes the kernel directory and multiplying it across kernels would just run N copies of identical work for no signal.

What it does (path mode only)

These steps run only when --kernel is a source directory path. Cached version and cache-key identifiers skip straight to test execution (step 6); uncached version identifiers run through download + configure + build + cache-store first. Ranges fan out to per-version resolution (every release downloads + builds + caches independently if not already present); git sources clone shallow at the ref, build, and cache. Multi-kernel resolution finishes for every requested kernel BEFORE step 6 — the cargo-nextest invocation in step 6 sees the complete kernel set as a single KTSTR_KERNEL_LIST export, so nextest fans the gauntlet across kernels in a single run.

For path mode, the source tree is gix-discovered and classified as either clean (HEAD reachable, index matches HEAD, worktree matches index) or dirty or non-git (any tracked-file diff, or the directory is not a git repo at all). The cache is keyed in one of three shapes:

• local-{hash7}-{arch}-kc{suffix} — clean git tree, no user .config file in the source tree yet (build will run make defconfig). {hash7} is the source tree’s HEAD short hash; {suffix} distinguishes ktstr framework kconfig fragments.
• local-{hash7}-{arch}-cfg{user_config}-kc{suffix} — clean git tree with a user .config whose CRC32 hash discriminates distinct configurations against the same commit, so iterative .config edits at a fixed commit populate distinct cache entries instead of colliding.
• local-unknown-{path_hash}-{arch}-kc{suffix} — dirty / non-git tree (HEAD does not describe the source). {path_hash} is the full 8-char (32-bit) CRC32 of the canonical source path so two parallel cargo ktstr test --kernel ./linux-a and --kernel ./linux-b runs do not collide on the same local-unknown-... slot.

Dirty / non-git trees never cache — the build pipeline runs in the source directory, the kernel label gets a _dirty suffix, and a subsequent run of the same path that goes clean produces a distinct cache entry under the clean shape.

1. Source-tree validation — verifies <kernel>/Makefile and <kernel>/Kconfig both exist. If either is missing, bails with not a kernel source tree.
2. Cache lookup (clean trees only) — looks up the local-{hash7}-{arch}[-cfg{user_config}]-kc{suffix} key (the cfg segment present iff a user .config exists in the source tree). Cache hit short-circuits to step 6: cargo-ktstr exports the cache entry directory via KTSTR_KERNEL and emits a cargo ktstr: cache hit for {input_path} ({cache_key}, built {age} ago) line on stderr (the , built {age} ago suffix is omitted when the timestamp is unparseable or future-dated). Cache miss continues to step 3.
3. Auto-configure — if <kernel>/.config lacks the CONFIG_SCHED_CLASS_EXT=y sentinel, runs make defconfig (when no .config exists), appends ktstr.kconfig to .config, then runs make olddefconfig.
4. Kernel build — runs make -j$(nproc) KCFLAGS=-Wno-error, then runs validate_kernel_config to verify critical config options (CONFIG_SCHED_CLASS_EXT, CONFIG_DEBUG_INFO_BTF, CONFIG_BPF_SYSCALL, CONFIG_FTRACE, CONFIG_KPROBE_EVENTS, CONFIG_BPF_EVENTS) survived the build — the kernel build system silently disables options whose dependencies are not met, and the validator surfaces those failures with a per- option remediation hint. make handles the no-op case when the kernel is already built. For dirty / non-git trees this is the unconditional path; for clean trees, only reached on cache miss.
5. compile_commands.json + cache store — runs make compile_commands.json (skipped only for transient temp directories like extracted tarballs) so LSP / clangd work against the local tree. Then for clean trees, the kernel image + stripped vmlinux are persisted under the resolved local-{hash7}-{arch}[-cfg{user_config}]-kc{suffix} key with metadata.json recording the source tree path. A post-build re-check of the dirty state catches mid-build mutations (worktree edits or commits that happened during make) and skips the cache store on either signal so a racing-write build can not land under a stale identity. Dirty / non-git trees skip the cache store unconditionally (no stable HEAD identity for the cache key) but still get compile_commands.json.
6. Test execution — execs cargo nextest run once with KTSTR_KERNEL set in the environment (single-kernel) or with both KTSTR_KERNEL and KTSTR_KERNEL_LIST (multi-kernel; the latter encodes the resolved kernel set as label1=path1;label2=path2;…). For clean Path-spec resolution KTSTR_KERNEL points at the cache entry directory; for dirty or non-git trees it points at the source tree directly. The test binary’s gauntlet expansion adds the kernel as a fifth dimension when the list carries 2+ entries; nextest’s parallelism, retries, and -E filtering apply natively to every (test × kernel) variant.

Implicit vs explicit kernel discovery diverge: cargo ktstr test --kernel ../linux (explicit Path spec) routes through the cache pipeline above — the source tree is gix-classified, the local-{hash7}-{arch}[-cfg{user_config}]-kc{suffix} cache key is computed, the kernel is built (or short-circuited on cache hit), and the cache entry directory is exported via KTSTR_KERNEL. cargo ktstr test (no --kernel flag) does NOT run the build pipeline or produce a new cache entry. The test binary’s find_kernel chain reads existing cache entries (most-recent-valid first; entries built with a different kconfig fragment are skipped) and falls back to local build trees (./linux, ../linux) and host paths. Whatever pre-built image it finds is returned as-is — no cache key is computed for source trees discovered on the filesystem, no make is invoked, and the result does not land in the kernel cache for a future cache_key-keyed lookup. The KTSTR_KERNEL env var with a path value follows this same direct-image flow — the cache write path is reached only via the cargo ktstr --kernel argument (or via cargo ktstr kernel build --source ../linux as an explicit cache-populate step). Pass --kernel ../linux to opt into the cache pipeline so a clean tree’s build is stored once and reused on subsequent runs.

Passing nextest arguments

Arguments after test are passed through to cargo nextest run:

cargo ktstr test -- -E 'test(my_test)'        # nextest filter
cargo ktstr test -- --workspace               # all workspace tests
cargo ktstr test -- --retries 2               # nextest retries

coverage

Build the kernel (if needed) and run tests with coverage via cargo llvm-cov nextest. Same kernel resolution and multi-kernel semantics as test: --kernel is repeatable; multi-kernel runs add the kernel suffix to every test name and partition the sidecar tree per kernel via target/ktstr/{kernel}-{project_commit}/, where {project_commit} is the project HEAD short hex (with -dirty when the worktree differs). Coverage profraw lands flat in target/llvm-cov-target/ with PID-keyed filenames — it does NOT partition per kernel — and cargo-llvm-cov merges every variant’s profraw automatically into the single output report.

cargo ktstr coverage                                               # auto-discover kernel
cargo ktstr coverage --kernel ../linux                             # local source tree
cargo ktstr coverage --kernel 6.14.2                               # version (auto-downloads on miss)
cargo ktstr coverage --kernel 6.14.2 --kernel 6.15.0               # multi-kernel coverage matrix
cargo ktstr coverage --release                                     # release profile (stricter assertions)
cargo ktstr coverage -- --workspace --lcov --output-path lcov.info # lcov output

Requires cargo-llvm-cov and the llvm-tools-preview rustup component:

cargo install cargo-llvm-cov
rustup component add llvm-tools-preview

Passing arguments

Arguments after coverage are passed through to cargo llvm-cov nextest:

cargo ktstr coverage -- --workspace --profile ci --lcov --output-path lcov.info
cargo ktstr coverage -- --features integration

profraw layout

Three populations of *.profraw files arise from cargo ktstr runs. They land in different directories and are not all collected by the same workflow:

cargo ktstr test injects LLVM_PROFILE_FILE (added to prevent default.profraw leaking into a kernel source tree when the shell cwd was the kernel dir; see Stale vmlinux.btf or default.profraw). The resulting host-side default-{pid}-{binary_hash}.profraw files do NOT land in the target/llvm-cov-target/ directory that cargo ktstr coverage (cargo-llvm-cov) reads; they are NOT picked up by a later cargo ktstr coverage run unless you explicitly include them in a cargo llvm-cov report invocation pointed at the cargo-ktstr binary’s llvm-cov-target/ directory.

To clean accumulated profraw between runs:

# Remove ONLY *.profraw under target/llvm-cov-target/ (top-level glob, non-recursive):
cargo ktstr llvm-cov clean --profraw-only

# Drop host-side test-path profraw next to the cargo-ktstr binary.
# Run only the line(s) matching how cargo-ktstr was launched —
# the brace-list form is bash-only, so each path is its own command
# for portable POSIX shells (sh / dash):
rm -f target/debug/llvm-cov-target/default-*.profraw
rm -f target/release/llvm-cov-target/default-*.profraw

# If ktstr was installed via `cargo install`:
rm -f ~/.cargo/bin/llvm-cov-target/default-*.profraw

--profraw-only is the safe default: it removes only *.profraw files at the top level of target/llvm-cov-target/ (the cargo- llvm-cov-managed dir) and leaves coverage reports, profdata, and build artifacts intact. It does NOT touch the default-*.profraw files next to the cargo-ktstr binary (under target/{profile}/llvm-cov-target/ for cargo run / cargo build, or ~/.cargo/bin/llvm-cov-target/ for cargo install-deployed binaries) produced by the host-side injection — remove those with the explicit rm -f lines above for whichever launch mode you use. Avoid cargo ktstr llvm-cov clean without arguments (recursively wipes all of target/llvm-cov-target/, including reports) and --workspace (additionally runs cargo clean on workspace packages, removing build artifacts); both are destructive beyond profraw.

To opt out of the host-side LLVM_PROFILE_FILE injection entirely, export LLVM_PROFILE_FILE yourself before running cargo ktstr test — the injector only fires when the env is absent, so an explicit operator setting takes precedence.

llvm-cov

Raw passthrough to cargo llvm-cov with arbitrary arguments. Use this for llvm-cov subcommands that don’t fit the coverage flow — report, clean, show-env, etc. When you want cargo llvm-cov nextest, prefer cargo ktstr coverage; this subcommand carries the same kernel-resolution and --no-perf-mode plumbing but hands every remaining argument to cargo llvm-cov unchanged.

cargo ktstr llvm-cov report --lcov --output-path lcov.info    # generate report from prior run
cargo ktstr llvm-cov clean --workspace                         # wipe accumulated coverage data
cargo ktstr llvm-cov show-env                                  # print env cargo-llvm-cov would set
cargo ktstr llvm-cov --kernel ../linux report                  # pin kernel + passthrough

Note: a bare cargo ktstr llvm-cov (no trailing subcommand) dispatches to cargo llvm-cov, which runs cargo test — ktstr tests rely on the nextest harness for gauntlet expansion (topology-preset variants), verifier cell emission, and VM dispatch. Under bare cargo test, only the #[test] stubs run and gauntlet variants + verifier cells are silently skipped. Always pass a subcommand after llvm-cov (most often nextest, for which cargo ktstr coverage is the shorter route).

kernel

Manage cached kernel images. Three subcommands: list, build, clean. The standalone ktstr kernel subcommands are identical.

kernel list

List cached kernel images, sorted newest first. With --range, switches to PREVIEW MODE: prints the versions a START..END range expands to without performing any download or build.

cargo ktstr kernel list
cargo ktstr kernel list --json                    # JSON output for CI scripting
cargo ktstr kernel list --range 6.12..6.14        # preview range expansion
cargo ktstr kernel list --range 6.12..6.14 --json # preview as JSON

Default mode walks the local cache. Human-readable output shows key, version, source type, arch, and build timestamp. Entries built with a different ktstr.kconfig are marked (stale kconfig). Entries whose major.minor version is no longer in kernel.org’s active releases list are marked (EOL); prefix lookups for EOL series fall back to probing cdn.kernel.org for the latest patch release.

--range mode performs no cache reads: it fetches kernel.org’s releases.json once, expands the inclusive range against the stable and longterm releases (mainline / linux-next dropped), and prints one version per line on stdout. Use this to answer “what does --kernel 6.12..6.16 actually cover?” before paying the build cost — no kernel is downloaded or compiled. With --json, emits a JSON object carrying the literal range, the parsed start / end, and the expanded versions array.

kernel build

Download, build, and cache a kernel image. Three source modes: version (tarball download), --source (local tree), --git (clone).

cargo ktstr kernel build                               # latest stable from kernel.org
cargo ktstr kernel build 6.14.2                        # specific version
cargo ktstr kernel build 6.15-rc3                      # RC release
cargo ktstr kernel build 6.12                          # latest 6.12.x patch release
cargo ktstr kernel build --source ../linux             # local source tree
cargo ktstr kernel build --git URL --ref v6.14         # git clone (shallow, depth 1)
cargo ktstr kernel build --force 6.14.2                # rebuild even if cached

When no version or source is given, fetches the latest stable series that has had at least 8 maintenance releases — keeping CI off brand-new majors whose early builds are more likely to break — from kernel.org’s releases.json. A major.minor prefix (e.g. 6.12) resolves to the highest patch release in that series. For EOL series no longer in releases.json, probes cdn.kernel.org to find the latest available tarball. Skips building when a cached entry already exists (use --force to override). Stale entries (built with a different ktstr.kconfig) are rebuilt automatically. For --source, generates compile_commands.json for LSP support. Dirty local trees (uncommitted changes to tracked files) are built but not cached.

kernel clean

Remove cached kernel images.

cargo ktstr kernel clean                          # remove all (with confirmation prompt)
cargo ktstr kernel clean --keep 3                 # keep 3 most recent
cargo ktstr kernel clean --force                  # skip confirmation prompt
cargo ktstr kernel clean --corrupt-only --force   # remove only corrupt entries

model

Manage the LLM model cache used by OutputFormat::LlmExtract payloads. fetch downloads the default pinned model into the ktstr model cache; status reports whether a SHA-checked copy is already cached; clean deletes the cached artifact plus its warm-cache sidecar.

cargo ktstr model fetch                          # download + SHA-check (no-op if cached)
cargo ktstr model status                         # report cache path + verdict
cargo ktstr model clean                          # delete cached artifact + sidecar

fetch is a no-op when the cache already holds a SHA-checked copy. Respects KTSTR_MODEL_OFFLINE=1 — set to refuse network fetches. Cache root resolution: KTSTR_CACHE_DIR (if set), then $XDG_CACHE_HOME/ktstr/models/, then $HOME/.cache/ktstr/models/.

status prints four fields and adds a one-line annotation when the verdict is anything other than Matches (a clean hit gets no annotation):

The annotation distinguishes four verdicts: NotCached (no entry — emit a cargo ktstr model fetch hint plus the expected download size), CheckFailed (cached entry could not be SHA-checked due to an I/O error — re-fetch), Mismatches (cached entry hash does not match the pinned digest — re-fetch), Matches (silent — the all-clear path). Re-fetch is the shared remediation tail for every cached-but- not-Matches branch.

clean removes both the GGUF artifact at {cache_root}/models/{file_name} and its .mtime-size warm-cache sidecar (a small companion file the SHA fast-path uses to skip re-hashing on subsequent status calls). Per- file output names what was deleted with an IEC-prefixed size in parentheses (removed /path/to/Qwen3-4B-Q4_K_M.gguf (2.34 GiB)); a final freed N total line sums the artifact and sidecar bytes. A no-op clean (nothing cached) prints a single no cached model found at {path} line so an idempotent re-run produces a clear “nothing to do” outcome instead of two “(absent)” lines. Subsequent cargo ktstr model fetch re-downloads the pin from scratch.

verifier

Collect BPF verifier statistics for every scheduler declared via declare_scheduler! in the workspace’s test binaries. Spawns cargo nextest run -E 'test(/^verifier/)' and lets nextest fan out per (scheduler × kernel-list entry × accepted topology preset) cell — each cell boots its own VM, loads the scheduler’s BPF programs, and reports per-program verified instruction counts from host-side memory introspection.

cargo ktstr verifier                              # auto-discover kernel
cargo ktstr verifier --kernel ../linux            # pin to one kernel
cargo ktstr verifier --kernel 6.14 --kernel 7.0   # multi-kernel sweep
cargo ktstr verifier --raw                        # raw verifier log

There are no --scheduler / --scheduler-bin flags: the sweep discovers schedulers from the KTSTR_SCHEDULERS distributed slice populated by declare_scheduler!. To exclude a scheduler from the sweep, omit it from the test binary (or declare it with SchedulerSpec::Eevdf / SchedulerSpec::KernelBuiltin — both are skipped at cell-emission time because neither has a userspace binary to verify).

--kernel is repeatable; cargo-ktstr always exports KTSTR_KERNEL_LIST to the nextest invocation (synthesizing a single entry from auto-discovery when no --kernel is passed). Each scheduler’s kernels = [...] declaration acts as a per-scheduler filter on the operator-supplied set; an empty (or omitted) kernels field accepts every entry. See BPF Verifier: Matrix dimension + per-scheduler filter for the full filter contract.

--raw exports KTSTR_VERIFIER_RAW=1; the cell handler reads it via env::var_os and switches format_verifier_output from the cycle-collapsed default to the raw scheduler-log dump. See BPF Verifier: Cycle collapse algorithm for the rendering details.

See BPF Verifier for the cell-based dispatch design and output format, and Scheduler Definitions for the declare_scheduler! macro that registers a scheduler in KTSTR_SCHEDULERS.

shell

Shares the VM boot flow with ktstr shell and accepts the same flags. See ktstr shell for the full flag reference. The one behavior difference from ktstr shell is that cargo ktstr shell accepts raw image file paths for --kernel.

cargo ktstr shell
cargo ktstr shell --kernel 6.14.2
cargo ktstr shell --topology 1,2,4,1
cargo ktstr shell -i ./my-binary -i strace

completions

Generate shell completions for cargo-ktstr. See ktstr completions for the base subcommand.

cargo ktstr completions bash >> ~/.local/share/bash-completion/completions/cargo
cargo ktstr completions zsh > ~/.zfunc/_cargo-ktstr
cargo ktstr completions fish > ~/.config/fish/completions/cargo-ktstr.fish

stats

Sidecar analysis, per-record diagnostics, and run-to-run comparison. See Runs for the directory layout.

cargo ktstr stats                                             # print analysis of newest run
cargo ktstr stats list                                        # list runs
cargo ktstr stats list-metrics                                # list registered regression metrics
cargo ktstr stats compare --a-kernel 6.14 --b-kernel 6.15     # slice on kernel
cargo ktstr stats compare --a-scheduler scx_rusty --b-scheduler scx_alpha  # slice on scheduler
cargo ktstr stats compare --a-kernel 6.14 --b-kernel 6.15 --scheduler scx_rusty  # slice on kernel, pin scheduler on both sides
cargo ktstr stats compare --a-kernel 6.14 --b-kernel 6.15 -E cgroup_steady       # add substring filter
cargo ktstr stats compare --a-project-commit abcdef1 --b-project-commit fedcba2 --no-average  # opt out of trial averaging
cargo ktstr stats compare --a-kernel-commit abcdef1 --b-kernel-commit fedcba2    # slice on kernel source HEAD
cargo ktstr stats compare --a-run-source ci --b-run-source local                 # slice on run environment
cargo ktstr stats explain-sidecar --run RUN_ID                                   # diagnose Option-field absences

When invoked without a subcommand, prints gauntlet analysis from either the most recent run directory under {CARGO_TARGET_DIR or "target"}/ktstr/ (newest by mtime) or the explicit directory in KTSTR_SIDECAR_DIR when that variable is set. With KTSTR_SIDECAR_DIR set, that directory is the sidecar source directly – there is no newest-subdirectory walk under it:

• Gauntlet analysis – outlier detection, per-scenario/topology dimension summaries, stimulus cross-tab.
• BPF verifier stats – per-program verified instruction counts, warnings for programs near the 1M complexity limit.
• BPF callback profile – per-program invocation counts, total CPU time, and average nanoseconds per call.
• KVM stats – cross-VM averages for exits, halt polling, host preemptions.

list

Print a table of run directories under {CARGO_TARGET_DIR or "target"}/ktstr/ with four columns:

• RUN: the run-directory leaf name, formatted as {kernel}-{project_commit} per Runs. list does NOT consult KTSTR_SIDECAR_DIR — that override only affects where the test harness writes sidecars; list always enumerates the default runs-root.
• TESTS: number of sidecars in the directory (and one level of subdirectories — collect_sidecars walks per-job gauntlet layouts).
• DATE: the earliest sidecar timestamp present in the directory — under last-writer-wins this equals the most recent run’s first sidecar timestamp (the prior run’s sidecars were pre-cleared at the new run’s first write, so only the new run’s timestamps remain). See Runs for the full semantics.
• ARCH: the host.arch value from the run’s first sidecar (e.g. x86_64, aarch64). Renders as - when no sidecar in the directory carries a populated host context — pre-host- context archives and host-only test stubs that never populate the field land in this bucket.

Rows are sorted by directory mtime, most recent first, so the latest run lands at the top — the operator’s usual interest. Entries whose mtime cannot be read fall back to filename order as a deterministic tiebreaker and sort to the end of the listing.

list-metrics

List the registered regression metrics and their default thresholds. Enumerates the ktstr::stats::METRICS registry: metric name, polarity (higher/lower better), default absolute-delta gate, default relative-delta gate, and display unit. Use this to see which metric names ComparisonPolicy.per_metric_percent keys can reference, and what each default absolute and relative gate starts at before an override. Default output is a human-readable table; --json emits a JSON array with the same fields.

cargo ktstr stats list-metrics              # table
cargo ktstr stats list-metrics --json       # JSON array

list-values

List the distinct values present per filterable dimension in the sidecar pool. Walks every run directory under target/ktstr/ (or --dir), pools the sidecars, and reports per-dimension sets for all seven dimensions: kernel, commit, kernel_commit, source, scheduler, topology, and work_type. The commit and source keys map to the internal SidecarResult::project_commit / run_source fields; the JSON wire keys keep the shorter spellings.

Use this before crafting a cargo ktstr stats compare invocation to discover what --a-X / --b-X values the pool actually carries: --a-kernel 6.20 against an empty pool fails downstream with “no rows match filter A”, and list-values is the upstream answer to “what kernels do I have?”.

cargo ktstr stats list-values                       # text per-dim blocks
cargo ktstr stats list-values --json                # JSON object
cargo ktstr stats list-values --dir /tmp/archived   # archived sidecar tree

The text shape renders one block per dimension with values one per line. The JSON shape emits a single object keyed by dimension name with arrays of values:

{
  "kernel": [null, "6.14.2", "6.15.0"],
  "commit": [null, "abcdef1", "abcdef1-dirty"],
  "kernel_commit": [null, "kabcde7", "kabcde7-dirty"],
  "source": [null, "ci", "local"],
  "scheduler": ["eevdf", "scx_rusty"],
  "topology": ["1n2l4c1t", "1n4l2c1t"],
  "work_type": ["SpinWait", "PageFaultChurn"]
}

The JSON keys commit and source are the wire contract; internally the corresponding fields are SidecarResult::project_commit and SidecarResult::run_source, and the per-side filter flags spell as --project-commit / --run-source (see compare).

kernel, commit, kernel_commit, and source are optional on the source sidecar (SidecarResult::kernel_version / project_commit / kernel_commit / run_source are Option<String>); the textual sentinel unknown and JSON null both denote a sidecar that did not record a value for that dimension.

show-host

Print the archived HostContext for a specific run: CPU identity, memory/hugepage config, transparent-hugepage policy, NUMA node count, kernel uname triple, kernel cmdline, and every /proc/sys/kernel/sched_* tunable captured at archive time. Useful for inspecting the same fingerprint compare’s host-delta section uses, available on a single run.

The command scans sidecars in the run directory in iteration order and prints the FIRST sidecar that carries a populated host field — older pre-enrichment sidecars may have host: None, and the forward scan tolerates those. If no sidecar has a populated host field the command fails with an actionable error rather than returning empty output.

explain-sidecar

Diagnose Option-field absences across a run’s sidecars. Loads every *.ktstr.json under --run ID (or its subdirectories one level deep, mirroring compare’s gauntlet-job layout) and reports, per sidecar, which Option<T> fields landed as None plus the documented causes for each absence and a classification:

• expected — None is the steady-state shape; no operator action recovers it (e.g. payload for a scheduler-only test, scheduler_commit which no SchedulerSpec variant exposes today).
• actionable — None indicates a recoverable gap; re-running in a different environment (in-repo cwd, non-tarball kernel, non-host-only test) would populate the field.

Different gauntlet variants on the same run legitimately differ on which fields populate (host-only vs VM-backed, scheduler-only vs payload-bearing), so the report is per-sidecar rather than aggregate.

Sidecars are loaded verbatim — this command does NOT rewrite run_source to "archive" even when --dir is set. Diverges intentionally from compare / list-values; matches show-host. The override would erase the only signal that surfaces the pre-rename source-key drop case.

The output header reports walked N sidecar file(s), parsed M valid: N counts every .ktstr.json file the walker visited, M counts how many parsed against the current schema. walked > parsed signals a corrupt or pre-1.0-schema sidecar — re-run the test to regenerate under the current schema.

Per-None blocks in the text output also include a fix: line for fields whose None is recoverable by an operator action (e.g. kernel_commit recovers when KTSTR_KERNEL points at a local kernel git tree). Fields whose None is the steady-state shape (or a multi-cause set with no single remediation) emit no fix: line.

When the walk encounters parse failures, the text output appends a trailing corrupt sidecars (N): block listing each corrupt path on its own line followed by the serde error message indented as error: ..., optionally followed by an enriched: ... line with operator-facing remediation prose when the parse failure matches a known schema-drift case (currently the host missing-field case). When the walk encounters IO failures (file matched the predicate but read_to_string failed before parsing could begin — permission denied, mid-rotate truncation, broken symlink, EISDIR), the text output appends a parallel io errors (N): block, structured the same way (path on its own line, error: ... line below) but carrying std::io::Error::Display rather than serde-error text. IO errors do NOT carry enriched: lines — there is no schema-drift catalog for filesystem incidents; the raw std::io::Error Display is the remediation surface. Each block is suppressed independently when its source vec is empty.

All-corrupt and all-IO-failure runs (every predicate- matching file failed to parse, or every one failed to read) are NOT a hard error — text output renders the header (walked N sidecar file(s), parsed 0 valid) followed directly by the corrupt sidecars (N): and/or io errors (N): block(s), skipping the per-sidecar breakdown that has nothing to render. JSON output mirrors this with valid: 0, _walk.errors and/or _walk.io_errors populated, and per-field counts at zero. This preserves structured per-file visibility for dashboard consumers facing total-failure runs of either class.

All-corrupt and all-IO-failure runs exit 0 (not a hard error); CI scripts must inspect the JSON channel for failure detection rather than relying on exit code. Two common gating policies, each appropriate for different operational stances:

• Lenient (treat partial failures as warnings): _walk.valid > 0. Accepts any run with at least one successfully-parsed sidecar; per-file parse or IO failures surface in the JSON arrays for triage but do not fail the gate.
• Strict (fail on any sidecar failure): _walk.errors.len() == 0 && _walk.io_errors.len() == 0. Requires every predicate-matching file to parse cleanly. Both checks are required because the two arrays cover disjoint failure classes (parse vs read) — a run with zero parse errors but one IO error still has a missing sidecar.

The two policies are NOT equivalent: a run with one valid and one corrupt sidecar passes lenient (valid == 1 > 0) but fails strict (errors.len() == 1 > 0). Pick the policy that matches the operational tolerance for partial data.

--json emits a single object with three top-level keys: _schema_version (a string version stamp — currently "1" — that consumers can gate on for incompatible shape changes), _walk (an envelope carrying walked / valid counts — same numbers the text header reports under “walked N sidecar file(s), parsed M valid” — plus an errors array of {path, error, enriched_message} entries covering every parse failure (enriched_message is a human-facing remediation string when a known schema-drift case matches, JSON null otherwise) AND an io_errors array of {path, error} entries covering every IO failure (file matched the predicate but read_to_string failed; error carries the raw std::io::Error Display). Both arrays emit on every render — empty array when no failures of that class occurred — so dashboard consumers see a uniform shape without contains_key branching. With both arrays, walked == valid + errors.len() + io_errors.len() by construction in the steady state — every predicate-matching file lands in exactly one bucket. (Filesystem races between the count and load passes can perturb this; see the rustdoc on WalkStats for the full caveat.) Then fields. Each entry under fields carries none_count and some_count (counts across all valid sidecars in the run, summing to _walk.valid), classification, causes, and fix (string when a remediation applies, JSON null otherwise).

Output produced before the schema-version stamp landed has no _schema_version key; consumers should treat the key’s absence as pre-stamp output (compatible with shape "1" in practice but unstamped).

The version bumps on incompatible shape changes (key rename, key removal, semantic shift in an existing key) but NOT on additive changes (new optional top-level keys, new entries in fields, new optional sub-keys under existing entries). The stamp is emitted as a JSON string (e.g. "1", "2"); parse it by stripping the quotes and converting the inner digits to an integer, then gate on parsed >= 1 (integer comparison) — never use raw string comparison, since lexicographic order would put "10" ahead of "2". Pin loosely (e.g. accept any version >= 1) so dashboard code keeps working when the catalog grows; tighten only on the specific bumps a consumer cannot tolerate.

cargo ktstr stats explain-sidecar --run RUN_ID                       # text per-sidecar diagnostic
cargo ktstr stats explain-sidecar --run RUN_ID --json                 # aggregate JSON for dashboards
cargo ktstr stats explain-sidecar --run RUN_ID --dir /path/archive    # diagnose archived sidecars

compare

Pool every sidecar under target/ktstr/ (or --dir), partition the rows into A and B sides via per-side filter flags, average each side’s matching sidecars per pairing key (or pass through distinct sidecars under --no-average), and report regressions on the A→B delta. Exits non-zero on regression.

The dimensions on which the A and B filters DIFFER are the SLICING dimensions — the axes of the A/B contrast. Every other dimension is part of the dynamic PAIRING key the comparison joins on. Slicing dims are derived automatically from the filters:

# Slice on kernel: A is 6.14, B is 6.15. Pair on every other dim.
cargo ktstr stats compare --a-kernel 6.14 --b-kernel 6.15

# Slice on kernel AND scheduler simultaneously.
cargo ktstr stats compare \
    --a-kernel 6.14 --a-scheduler scx_rusty \
    --b-kernel 6.15 --b-scheduler scx_alpha

# Slice on project commit, narrow both sides to one scheduler+kernel.
cargo ktstr stats compare \
    --a-project-commit abcdef1 --b-project-commit fedcba2 \
    --kernel 6.14 --scheduler scx_rusty

# Slice on run environment: CI runs vs local developer runs.
cargo ktstr stats compare \
    --a-run-source ci --b-run-source local

Symmetric sugar. Shared --X flags (--kernel, --scheduler, --topology, --work-type, --project-commit, --kernel-commit, --run-source) pin BOTH sides to the same value(s). Per-side --a-X / --b-X flags REPLACE the corresponding shared --X value for that side only — “more-specific replaces” semantics. So --kernel 6.14 --a-kernel 6.13 puts A on 6.13 and B on 6.14. Together the seven slicing dimensions (kernel, scheduler, topology, work-type, project-commit, kernel-commit, run-source) cover every typed axis the comparison can contrast on.

Validation. The dispatch site rejects two cases up front:

• Empty slicing: no --a-X / --b-X at all, OR the per-side flags resolve to identical effective filters. Bails with “specify at least one per-side filter (e.g. --a-kernel 6.14 --b-kernel 6.15) to define what dimension separates the two sides.”
• Multi-dim slicing: slicing on more than one dimension prints a warning to stderr (“warning: slicing on N dimensions; results compress multiple axes into a single A/B contrast”) but continues — multi-dim contrasts are a deliberate feature for cohort sweeps.

Averaging. By default the comparison aggregates every matching sidecar within each side into a single arithmetic-mean row per pairing key, smoothing run-to-run jitter. Failing / skipped contributors are excluded from the metric mean; the aggregated row’s passed is the AND across every contributor. A header line above the comparison table reads averaged across N runs (A) and M runs (B) and a per-group passes_observed/total_observed block prints below the summary.

+mixed commit marker. When contributors to an averaged group disagree on the -dirty suffix for the same canonical hex (some clean, some -dirty), the rendered commit and kernel_commit columns show {hex}+mixed for that group. +mixed is a COHORT-level marker (distinct from -dirty, which is a per-record property of one sidecar): it indicates mixed working-tree state across the group’s contributors. Mixed-dirty tracking spans EVERY contributor (passing, failing, skipped) so WIP-vs-committed disagreement surfaces in the averaged row even when one of the two states only appears on a failing run. The marker is rendered against the canonical un-suffixed hex, so a abc1234 clean entry plus an abc1234-dirty entry render as abc1234+mixed regardless of which contributor was scanned first. Homogeneous cohorts (every contributor clean, every contributor dirty, or every contributor None) preserve the first-seen value verbatim and never get the +mixed marker.

--no-average keeps each sidecar distinct. If multiple sidecars on the same side share the same pairing key under --no-average, the comparison bails with “duplicate pairing keys” — pairing across A/B sides is ambiguous when one A-row could match many B-rows. Either drop --no-average to average them, or add another per-side filter to disambiguate.

Kernel match shape. A --kernel 6.12 filter (two-segment major.minor) PREFIX-matches every patch release in that series: 6.12, 6.12.0, 6.12.5 all match. A three-or-more-segment filter (--kernel 6.14.2, --kernel 6.15-rc3) is strict equality — 6.14.2 does NOT match 6.14.20. The same shape applies to --a-kernel / --b-kernel.

Discovering filter values. Run cargo ktstr stats list-values before crafting a compare invocation to see what kernel, commit, kernel_commit, source, scheduler, topology, and work_type values the sidecar pool actually carries; passing a --a-kernel 6.20 against an empty pool fails downstream with “no rows match filter A” and list-values is the upstream answer to “what have I got?”. list-values reports all seven filterable dimensions; the JSON keys commit and source map to the per-side filter flags --project-commit and --run-source.

When a side comes back as unknown for one of the optional dimensions (kernel, commit, kernel_commit, source), cargo ktstr stats explain-sidecar on the underlying run reports per-sidecar which optional fields are missing and what each absence means.

Prerequisites

Run tests first to generate sidecar JSON files:

cargo ktstr test                     # generates target/ktstr/{kernel}-{project_commit}/*.json
cargo ktstr stats                    # reads the newest run

Set KTSTR_SIDECAR_DIR to override the sidecar directory; otherwise the default is {CARGO_TARGET_DIR or "target"}/ktstr/{kernel}-{project_commit}/, where {project_commit} is the project HEAD short hex (with -dirty when the worktree differs).

show-host

Print the live host context used by the sidecar collector: CPU identity, memory/hugepage config, transparent-hugepage policy, NUMA node count, kernel uname triple (sysname / release / machine), kernel cmdline, and every /proc/sys/kernel/sched_* tunable. Useful for diagnosing cross-run regressions that trace back to host-context drift (sysctl change, THP policy flip, hugepage reservation) or for confirming what cargo ktstr stats compare would record on the next run produced here.

cargo ktstr show-host

This is a live snapshot (reads /proc, /sys, and uname() at invocation time). For the archived host context captured at sidecar-write time for a past run, use cargo ktstr stats show-host --run RUN_ID instead — same HostContext::format_human formatter so the two outputs are byte-for-byte comparable when the host is unchanged.

For historical drift between archived runs (host-side diff across two run partitions), use cargo ktstr stats compare — its host-delta section reports which host-context fields changed between side A and side B using the same HostContext::diff logic.

show-thresholds

Print the resolved assertion thresholds for the named test — the same merged Assert value run_ktstr_test_inner evaluates against worker reports, produced by the runtime merge chain Assert::default_checks().merge(entry.scheduler.assert()).merge(&entry.assert). Surfaces every threshold field (or none when inherited or unset) so an operator can see what the test will actually check against without reading source or guessing which layer contributed each bound.

cargo ktstr show-thresholds preempt_regression_fault_under_load

Fails with an actionable message when no registered test matches the given name; the diagnostic includes a Did you mean ...? Levenshtein suggestion when a near match exists.

locks

Enumerate every ktstr flock held on this host — read-only, does NOT attempt any flock acquire. Troubleshooting companion for --cpu-cap contention: when a build or test is stalled behind a peer’s reservation, cargo ktstr locks names the peer (PID + cmdline) without disturbing any of its flocks.

Scans four lock-file roots:

• /tmp/ktstr-llc-*.lock — per-LLC reservations held by perf-mode test runs and --cpu-cap-bounded builds.
• /tmp/ktstr-cpu-*.lock — per-CPU reservations from the same flow.
• {cache_root}/.locks/*.lock — cache-entry locks held during kernel build writes, and source-{path_hash}.lock files held for the duration of kernel build --source and cargo ktstr test --kernel <path> against the same source tree.
• {runs_root}/.locks/{kernel}-{project_commit}.lock — per-run-key sidecar-write locks held for the duration of the (pre-clear + write) cycle to serialize concurrent ktstr processes targeting the same run directory.

Each lock is cross-referenced against /proc/locks to name the holder PID and cmdline.

cargo ktstr locks                       # one-shot snapshot
cargo ktstr locks --json                # JSON snapshot
cargo ktstr locks --watch 1s            # redraw every second until SIGINT
cargo ktstr locks --watch 1s --json     # ndjson stream, one object per interval

The same subcommand is available as ktstr locks with identical flag semantics.

Install

cargo install --locked ktstr --bin ktstr --bin cargo-ktstr   # the two user-facing binaries

The explicit --bin flags scope the install to just ktstr and cargo-ktstr; without them, cargo install would also place the test-fixture binaries (ktstr-jemalloc-probe, ktstr-jemalloc-alloc-worker) on $PATH.

Or build from the workspace:

cargo build --bin cargo-ktstr

Auto-Repro

When a test fails because the scheduler crashes or exits, auto-repro boots a second VM with BPF probes attached to capture function arguments and struct fields along the scheduling path. Stack functions extracted from the crash output seed the probe list; when no crash stack is available (e.g. a BPF text error or verifier failure with no backtrace), auto-repro falls back to dynamic BPF program discovery in the repro VM.

How it works

1. First VM – the test runs normally. If the scheduler crashes or exits (BPF error, verifier failure, stall), ktstr captures any stack trace from the scheduler log (COM2) or kernel console (COM1).

2. Stack extraction – function names are parsed from the crash trace when available. BPF program symbols (bpf_prog_*) are recognized and their short names extracted. Generic functions (scheduler entry points, spinlocks, syscall handlers, sched_ext exit machinery, BPF trampolines, stack dump helpers) are filtered out. When no stack functions are found, the pipeline continues with an empty probe list.

3. BPF discovery – in the repro VM, ktstr discovers loaded struct_ops programs via libbpf-rs and adds them to the probe list alongside any stack-extracted functions. Their kernel-side callers are added (e.g. enqueue -> do_enqueue_task) for bridge kprobes. This step ensures probes can capture variable states across the scheduler exit call chain even when the crash produced no extractable stack.

4. BTF resolution – function signatures are resolved from vmlinux BTF (kernel functions) and program BTF (BPF callbacks). Known struct types (task_struct, rq, scx_dispatch_q, etc.) have curated fields resolved to byte offsets. Other struct pointer params have scalar, enum, and cpumask pointer fields auto-discovered from vmlinux or BPF program BTF.

5. Second VM – ktstr boots a new VM and reruns the scenario with BPF probes:

   • Kprobe skeleton for kernel function entry (uses bpf_get_func_ip)
   • Fentry/fexit skeleton for BPF callbacks and kernel function exit (batched in groups of 4, shares maps via reuse_fd). Fexit re-reads struct fields after the function executes, capturing post-mutation state alongside the entry snapshot.
   • Tracepoint trigger (tp_btf/sched_ext_exit) fires inside scx_claim_exit() when the scheduler exits, in the context of the current task at exit time

6. Stitching – the task_struct pointer is read from the trigger event’s bpf_get_current_task() value. Events with a task_struct parameter are filtered to that pointer; events without a task_struct parameter are retained if their task_ptr (from bpf_get_current_task() at probe time) matches the triggering task. Events are sorted by timestamp and formatted with decoded field values (cpumask ranges, DSQ names, enqueue flags, etc.) and source locations (DWARF for kernel, line_info for BPF).

7. Diagnostic tails – the last 40 lines of the repro VM’s scheduler log (COM2, cycle-collapsed), sched_ext dump (COM1), and kernel console (COM1) are appended after the probe output when non-empty. A duration line reports total repro VM wall time. When probe data is absent, a crash reproduction status line indicates whether the crash reproduced.

Requirements

Auto-repro requires a kernel with the sched_ext_exit tracepoint (used as the probe trigger). Kernels built with CONFIG_SCHED_CLASS_EXT and tracepoint support include this. If the tracepoint is unavailable, auto-repro is skipped and the pipeline diagnostics report the cause.

Enabling auto-repro

In #[ktstr_test]:

#[ktstr_test(auto_repro = true)]
fn my_test(ctx: &Ctx) -> Result<AssertResult> { ... }

auto_repro defaults to true in #[ktstr_test].

Repro mode

During the second VM run, ktstr sets “repro mode” which disables the work-conservation watchdog. Workers normally send SIGUSR2 to the scheduler when stuck > 2 seconds. In repro mode, the scheduler stays alive so BPF assertion probes can fire.

Example output

The demo_host_crash_auto_repro test triggers a host-initiated crash via BPF map write and captures the scheduling path. Probe output shows each function with decoded struct fields and source locations. When fexit captures post-mutation state, changed fields show an arrow (→) between entry and exit values:

ktstr_test 'demo_host_crash_auto_repro' [sched=scx-ktstr] failed:
  scheduler died

--- auto-repro ---
=== AUTO-PROBE: scx_exit fired ===

  ktstr_enqueue                                                   main.bpf.c:21
    task_struct *p
      pid         97
      cpus_ptr    0xf(0-3)
      dsq_id      SCX_DSQ_INVALID
      enq_flags   NONE
      slice       0
      vtime       0
      weight      100
      sticky_cpu  -1
      scx_flags   QUEUED|ENABLED
  do_enqueue_task                                               kernel/sched/ext.c:1344
    rq *rq
      cpu         1
    task_struct *p
      pid         97
      cpus_ptr    0xf(0-3)
      dsq_id      SCX_DSQ_INVALID          →  SCX_DSQ_LOCAL
      enq_flags   NONE
      slice       20000000
      vtime       0
      weight      100
      sticky_cpu  -1
      scx_flags   QUEUED|DEQD_FOR_SLEEP    →  QUEUED

After the probe data, the auto-repro section includes the repro VM duration and the last 40 lines of the repro VM’s scheduler log, sched_ext dump, and dmesg (each only when non-empty).

Demo test

A demo test in this shape (reduced from demo_host_crash_auto_repro in tests/scenario_coverage.rs):

use ktstr::prelude::*;
use ktstr::test_support::{BpfMapWrite, KtstrTestEntry, run_ktstr_test};

fn scenario_yield_heavy(ctx: &Ctx) -> Result<AssertResult> {
    let steps = vec![Step::with_defs(
        vec![
            CgroupDef::named("demo_workers")
                .work_type(WorkType::YieldHeavy)
                .workers(4),
        ],
        HoldSpec::Fixed(Duration::from_secs(8)),
    )];
    execute_steps(ctx, steps)
}

Run manually to see full output:

cargo ktstr test --kernel ../linux -- --run-ignored ignored-only -E 'test(demo_host_crash_auto_repro)'

BPF Verifier

The verifier sweep boots every declared scheduler in a KVM VM and captures per-program verifier statistics from the real kernel verifier.

Design

The verifier sweep follows ktstr’s two core principles.

Fidelity without overhead. Each scheduler binary runs inside a VM on the same kernel the scheduler targets in production. The verifier that runs is the real verifier in the real kernel — no host-side BPF loading, no version skew between the host kernel’s verifier and the target kernel’s verifier.

Direct access over tooling layers. No subprocess to bpftool or veristat. The host reads per-program verified_insns directly from guest memory via bpf_prog_aux introspection and applies cycle collapse to verifier logs instead of truncating.

Quick start

# Run every declared scheduler against the kernel discovered via
# KTSTR_KERNEL (or the cache).
cargo ktstr verifier

# Run against a specific kernel build.
cargo ktstr verifier --kernel ../linux

# Sweep across multiple kernels (each cell runs against its own).
cargo ktstr verifier --kernel 6.14.2 --kernel 6.15.0

# Print the raw verifier log without cycle collapse.
cargo ktstr verifier --raw

See cargo-ktstr verifier for the full flag list.

How it works

cargo ktstr verifier is a thin dispatcher around cargo nextest run -E 'test(/^verifier/)'. The matrix that nextest runs is generated by the test binaries themselves.

1. Cell emission — every test binary that links the ktstr test support and contains at least one declare_scheduler! declaration emits a verifier/<sched>/<kernel>/<preset>: test listing for each (declared scheduler × kernel-list entry × accepted gauntlet topology preset) triple. Cells whose topology preset would exceed the host’s CPU / LLC / per-LLC capacity are filtered at emission time (mirroring the gauntlet variant filter).

2. Per-cell dispatch — nextest invokes the test binary once per cell with --exact verifier/<sched>/<kernel>/<preset>. The binary’s #[ctor] intercepts the prefix, parses the cell name into its three components, resolves the scheduler binary, kernel directory, and topology, and boots a single VM dedicated to that cell.

3. Verifier collection — inside the VM, the scheduler loads its BPF programs via scx_ops_load!; the real kernel verifier runs against them. The host reads per-program verified_insns from bpf_prog_aux via guest physical memory introspection. On load failure, libbpf prints the verifier log to stderr; the VM forwards it to the host via the bulk SHM port between ===SCHED_OUTPUT_START=== / ===SCHED_OUTPUT_END=== markers.

4. Rendering — per-program summary lines, then the verifier log with cycle collapse applied (or raw, with --raw).

Eevdf + KernelBuiltin scheduler variants have no userspace binary to load BPF programs from, so they are skipped at cell-emission time. Direct invocation outside nextest (--exact verifier/<eevdf-sched>/...) prints a SKIP banner and exits 0.

Matrix dimension + per-scheduler filter

The verifier sweep matrix is driven by the operator’s --kernel set, not by per-scheduler declare_scheduler! declarations. The dispatcher always exports KTSTR_KERNEL_LIST (label1=path1;label2=path2;...) to the nextest invocation — even with no --kernel flag it synthesizes a single entry from the auto-discovered kernel. The test binary’s lister walks that list as the matrix dimension and emits one cell per (declared scheduler × kernel-list entry × accepted preset).

Each scheduler’s declare_scheduler! kernels = [...] declaration acts as a per-scheduler filter on the matrix:

• Empty (kernels = []) accepts every kernel-list entry — the scheduler verifies against everything the operator passes.
• Version specs ("6.14.2") match entries whose raw label equals the version (or whose sanitized label equals the sanitized form of the version).
• Range specs ("6.14..6.16", "6.14..=6.16") match entries whose raw version falls in the inclusive range, parsed via the same decompose_version_for_compare helper the operator-side range expansion uses.
• Path / CacheKey / Git specs match by sanitized-label equality.

# Scheduler declares kernels = ["6.14..6.16"]
# Operator runs --kernel 6.14.2 --kernel 6.15.0 --kernel 6.17.0
# Dispatcher's KTSTR_KERNEL_LIST: kernel_6_14_2, kernel_6_15_0,
#                                 kernel_6_17_0
# Scheduler filter: 6.14.2 ∈ [6.14, 6.16] ✓
#                   6.15.0 ∈ [6.14, 6.16] ✓
#                   6.17.0 ∈ [6.14, 6.16] ✗ — rejected
# Cells emitted: verifier/<sched>/kernel_6_14_2/<preset>
#                verifier/<sched>/kernel_6_15_0/<preset>
cargo ktstr verifier --kernel 6.14.2 --kernel 6.15.0 --kernel 6.17.0

# No --kernel: dispatcher auto-discovers one kernel via the
# cache + filesystem chain and synthesizes a single-entry
# KTSTR_KERNEL_LIST. The auto-discovered entry's label is
# derived from the resolved path (`kernel_path_<basename>_<hash6>`).
# Schedulers with non-empty `kernels = [...]` may filter the
# entry out — operators wanting deterministic coverage should
# always pass --kernel.
cargo ktstr verifier

The verifier cell handler resolves the per-cell kernel directory by looking up the cell’s sanitized label in KTSTR_KERNEL_LIST — there is no single-kernel fallback that would silently run a cell against an unrelated kernel. A label that doesn’t appear in the list errors out with an actionable diagnostic naming the present labels and pointing at both fix paths (add --kernel <SPEC> or drop the matching entry from declare_scheduler!).

Output

Brief (default)

Per-program summary line:

  ktstr_enqueue                              verified_insns=500

verified_insns is the number of instructions the kernel verifier processed, read from bpf_prog_aux via host-side memory introspection.

On load failure, the scheduler-log section shows libbpf’s verifier output with cycle collapse applied — repeating loop iterations are reduced to the first iteration, an omission marker, and the last iteration:

--- 8x of the following 10 lines ---
100: (bf) r0 = r1 ; frame1: R0_w=scalar(id=0,umin=0)
101: (bf) r1 = r2 ; frame1: R1_w=scalar(id=1,umin=1)
...
--- 6 identical iterations omitted ---
100: (bf) r0 = r1 ; frame1: R0_w=scalar(id=70,umin=700)
101: (bf) r1 = r2 ; frame1: R1_w=scalar(id=71,umin=701)
...
--- end repeat ---

Raw (--raw)

Full raw verifier log without cycle collapse. Use for debugging verification failures where the exact register state at each iteration matters. The flag exports KTSTR_VERIFIER_RAW=1 for the nextest invocation; the in-binary cell handler reads it via env::var_os and switches the format_verifier_output rendering branch.

Cycle collapse algorithm

The kernel verifier unrolls loops by re-verifying each instruction with updated register states. A bounded loop of 8 instructions verified 100 times produces 800 near-identical lines — differing only in register-state annotations. Naive truncation loses context. Cycle collapse preserves structure: the first iteration shows what the loop does, the last shows the final state, and a count tells you how many iterations were elided.

The algorithm normalizes lines by stripping variable annotations, then detects repeating blocks:

1. Normalize — strip ; frame1: R0_w=... annotations, standalone register dumps (3041: R0=scalar()), and inline branch-target state after goto pc+N. Source comments (; for (int j = 0; ...)) are preserved as cycle anchors.

2. Detect — find the most frequent normalized line (the “anchor”), compute gaps between anchor occurrences to determine the cycle period, then verify consecutive blocks match after normalization. Minimum period: 5 lines. Minimum repetitions: 3.

3. Collapse — replace the cycle with the first iteration, an omission count, and the last iteration. Run iteratively (up to 5 passes) to handle nested loops.

scx-ktstr test flags

scx-ktstr supports these flags to exercise the verifier pipeline:

--fail-verify — sets a .rodata variable before scx_ops_load!, enabling a code path the BPF verifier rejects. On failure, libbpf prints the verifier log to stderr.

--verify-loop — sets a .rodata variable that enables an unrolled loop followed by while(1) in ktstr_dispatch. The verifier rejects the infinite loop and libbpf prints the full instruction trace to stderr, exercising cycle collapse.

Core Concepts

ktstr tests compose from four layers:

1. Scenarios – what to test: cgroup layout, CPU partitioning, workloads, custom logic.

2. Flags – which scheduler features to enable for each run.

3. WorkSpec types – what each worker process does: CPU spin, yield, I/O, bursty patterns, pipe-based IPC.

4. Checking – how to evaluate results: starvation, fairness, isolation, scheduling gaps, monitor thresholds.

These compose orthogonally. A scenario runs with every valid flag combination, and checks apply uniformly across all runs.

Four supporting concepts complete the picture: Ops and Steps is the primary API for defining scenarios – most tests use CgroupDef and execute_defs from this module. TestTopology provides CPU and LLC layout for cpuset partitioning. Performance Mode applies host-side isolation for noise-sensitive measurements. Resource Budget describes the --cpu-cap tier that coordinates concurrent no-perf-mode VMs and kernel builds via LLC flocks and cgroup v2 cpuset sandboxes.

Scenarios

Scenarios define the scheduling conditions a test creates. Each scenario sets up cgroups, workers, and cpusets to produce a specific condition, then verifies the scheduler handles it correctly.

Canned scenarios (scenarios::*)

ktstr::scenario::scenarios provides curated scenario functions that can be called directly from #[ktstr_test]:

use ktstr::prelude::*;

#[ktstr_test(llcs = 1, cores = 2, threads = 1)]
fn my_test(ctx: &Ctx) -> Result<AssertResult> {
    scenarios::steady(ctx)
}

Additional custom_* functions are available in ktstr::scenario::{affinity, basic, cpuset, dynamic, interaction, nested, performance, stress}. See the API docs for the full list.

Most tests use these canned functions or build custom scenarios with CgroupDef and execute_defs / execute_steps (see Ops and Steps). Custom scenarios receive a Ctx reference and use the same building blocks; see Custom Scenarios for the Ctx struct and helper functions.

WorkType

WorkType controls what each worker process does during a scenario.

The WorkType enum in ktstr::workload is the source of truth. The variants below are grouped by intent; each one-line summary is the leading sentence of the variant’s rustdoc. Run cargo doc --open for full per-variant semantics, parameter ranges, and kernel-path citations — this page reproduces only the high-level shape.

pub enum WorkType {
    // CPU primitives
    SpinWait,                            // Tight CPU spin loop (1024 iterations per cycle).
    YieldHeavy,                          // Repeated sched_yield with minimal CPU work.
    Mixed,                               // CPU spin burst followed by sched_yield.
    AluHot { width: AluWidth },          // Dependent integer multiply chain at high IPC (>= 2.0); optional SIMD width.
    SmtSiblingSpin,                      // Tight PAUSE-spin from a paired worker pinned to two SMT siblings.
    IpcVariance {                        // Alternating high-IPC (multiplies) / low-IPC (random cache touches) phases.
        hot_iters: u64,
        cold_iters: u64,
        period_iters: u64,
    },

    // Block-device I/O (operates on /dev/vda; falls back to per-worker tempfile when absent)
    IoSyncWrite,                         // 16 x 4 KB pwrites + fdatasync per iteration (O_SYNC).
    IoRandRead,                          // Single 4 KB pread at a random sector-aligned offset (O_DIRECT).
    IoConvoy,                            // Interleaved sequential pwrite + random pread with periodic fdatasync (O_DIRECT).

    // Burst-and-sleep
    Bursty {                             // CPU burst for `burst_duration`, sleep for `sleep_duration`, repeat.
        burst_duration: Duration,
        sleep_duration: Duration,
    },
    IdleChurn {                          // CPU burst then `nanosleep` (exercises hrtimer + idle-class path).
        burst_duration: Duration,
        sleep_duration: Duration,
        precise_timing: bool,
    },

    // Cache pressure
    CachePressure { size_kb: usize, stride: usize },    // Strided RMW sized to pressure L1.
    CacheYield { size_kb: usize, stride: usize },       // Cache pressure burst then sched_yield().

    // Wake-placement / cross-CPU paths
    PipeIo { burst_iters: u64 },                        // CPU burst then 1-byte pipe exchange with a partner worker.
    FutexPingPong { spin_iters: u64 },                  // Paired futex wait/wake between partner workers (non-WF_SYNC).
    CachePipe { size_kb: usize, burst_iters: u64 },     // Cache-hot working set + pipe wake.
    FutexFanOut { fan_out: usize, spin_iters: u64 },    // 1:N fan-out wake (one messenger, N receivers).
    FanOutCompute {                                     // Messenger/worker fan-out with matrix-multiply compute per receiver.
        fan_out: usize,
        cache_footprint_kb: usize,
        operations: usize,
        sleep_usec: u64,
    },
    AsymmetricWaker {                                   // Paired workers in mismatched scheduling classes share one futex word.
        waker_class: SchedClass,
        wakee_class: SchedClass,
        burst_iters: u64,
    },
    WakeChain {                                         // Ring of waker-wakee hops via Pipe (WF_SYNC) or Futex wake.
        depth: usize,
        wake: WakeMechanism,
        work_per_hop: Duration,
    },
    EpollStorm {                                        // eventfd producers + epoll_wait consumers (exclusive autoremove wake).
        producers: usize,
        consumers: usize,
        events_per_burst: u64,
    },
    ThunderingHerd {                                    // N waiters on ONE global futex word; broadcast FUTEX_WAKE rouses the herd.
        waiters: usize,
        batches: u64,
        inter_batch_ms: u64,
    },

    // Compound / sequence
    Sequence { first: Phase, rest: Vec<Phase> },        // Loop through ordered phases (Spin / Sleep / Yield / Io).

    // Lifecycle / scheduling-class churn
    ForkExit,                                           // Rapid fork+_exit cycling; parent waitpid's then repeats.
    NiceSweep,                                          // Cycle nice level from -20 to 19 across iterations.
    AffinityChurn { spin_iters: u64 },                  // Rapid self-directed sched_setaffinity to random CPUs.
    PolicyChurn { spin_iters: u64 },                    // Cycle SCHED_OTHER -> BATCH -> IDLE (-> FIFO/RR if CAP_SYS_NICE).
    NumaMigrationChurn { period_ms: u64 },              // Rotate sched_setaffinity across NUMA nodes.
    CgroupChurn { groups: usize, cycle_ms: u64 },       // Cycle cgroup membership between sibling cgroups.

    // Memory pressure / NUMA
    PageFaultChurn {                                    // mmap NOHUGEPAGE -> touch random pages -> MADV_DONTNEED, repeat.
        region_kb: usize,
        touches_per_cycle: usize,
        spin_iters: u64,
    },
    NumaWorkingSetSweep {                               // Rotate the working-set memory across NUMA nodes via mbind.
        region_kb: usize,
        sweep_period_ms: u64,
        target_nodes: Vec<usize>,
    },

    // Lock contention
    MutexContention {                                   // N-way futex mutex contention (CAS acquire / FUTEX_WAIT on failure).
        contenders: usize,
        hold_iters: u64,
        work_iters: u64,
    },
    PriorityInversion {                                 // Three priority tiers contending for one shared lock (Pi or Plain futex).
        high_count: usize,
        medium_count: usize,
        low_count: usize,
        hold_iters: u64,
        work_iters: u64,
        pi_mode: FutexLockMode,
    },

    // Producer/consumer + signal/preempt pressure
    ProducerConsumerImbalance {                         // Producer / consumer pipeline with deliberately-unbalanced rates.
        producers: usize,
        consumers: usize,
        produce_rate_hz: u64,
        consume_iters: u64,
        queue_depth_target: u64,
    },
    SignalStorm {                                       // Paired workers fire tkill(partner, SIGUSR1) between CPU bursts.
        signals_per_iter: u64,
        work_iters: u64,
    },
    PreemptStorm {                                      // One SCHED_FIFO worker preempts CFS spinners on the same CPU at ~kHz rate.
        cfs_workers: usize,
        rt_burst_iters: u64,
        rt_sleep_us: u64,
    },
    RtStarvation {                                      // SCHED_FIFO workers monopolise the CPU at 100%; CFS workers starve.
        rt_workers: usize,
        cfs_workers: usize,
        rt_priority: i32,
        burst_iters: u64,
    },

    // User-supplied
    Custom {                                            // User-supplied work function (name + fn pointer).
        name: String,
        run: fn(&AtomicBool) -> WorkerReport,
    },
}

Imports: WorkType, Phase, SchedPolicy, WorkSpec, and WorkloadConfig are in ktstr::prelude::*. The auxiliary enums FutexLockMode (used by PriorityInversion::pi_mode), WakeMechanism (used by WakeChain::wake), and SchedClass (used by AsymmetricWaker) live under ktstr::workload. Bring them into scope with use ktstr::workload::*; (or import each by name) before writing variant literals that reference them.

Parameterized variants have snake-case convenience constructors — e.g. WorkType::bursty(burst_duration, sleep_duration), WorkType::pipe_io(burst_iters), WorkType::cache_pressure(size_kb, stride), WorkType::page_fault_churn(region_kb, touches_per_cycle, spin_iters), WorkType::mutex_contention(contenders, hold_iters, work_iters), WorkType::priority_inversion(high_count, medium_count, low_count, hold_iters, work_iters, pi_mode), WorkType::wake_chain(depth, wake, work_per_hop), WorkType::custom(name, run). Every parameterised variant has one; see cargo doc --open on WorkType for the full constructor list and parameter validation rules.

Bursty, IdleChurn, and WakeChain take Duration parameters (humantime-serialised in captured configs) — pass Duration::from_millis(N) or Duration::from_micros(N) from std::time rather than raw integers. IpcVariance, ProducerConsumerImbalance, RtStarvation, PriorityInversion, EpollStorm, PreemptStorm, and ThunderingHerd reject zero-valued counters at spawn time (WorkTypeValidationError::*).

Choosing a work type

Variants

SpinWait – tight spin loop with spin_loop() hints. 1024 iterations per check. Pure CPU-bound workload.

YieldHeavy – thread::yield_now() on every iteration. Exercises scheduler wake/sleep paths.

Mixed – 1024 spin iterations then yield. Combines CPU and voluntary preemption.

IoSyncWrite – 16 × 4 KB pwrites totaling 64 KB at the worker’s stripe offset (per-worker striping prevents fdatasync from coalescing across writers), then fdatasync(). Drives fsync-heavy D-state cycles. Opens /dev/vda with O_SYNC; falls back to a per-worker tempfile when /dev/vda is absent (host-side unit tests).

IoRandRead – single 4 KB pread at a sector-aligned random offset within the device capacity. Opens /dev/vda with O_DIRECT (tempfile fallback); drives high-IOPS short-D-state cycles. Per-worker xorshift PRNG seeded from tid.

IoConvoy – alternates 4 KB pwrite at the worker’s monotonic sequential cursor with 4 KB pread at a random offset; fdatasync() runs every 16 iterations. /dev/vda opened O_DIRECT (tempfile fallback). Currently uses direct IO so the pathology surface is the synchronous flush + IO-mix latency rather than page-cache convoy build-up.

Bursty – CPU burst for burst_duration, sleep for sleep_duration, repeat. Both fields are Duration (humantime- serialised); pass Duration::from_millis(N) from std::time. Frees CPUs during sleep, exercising CPU borrowing.

PipeIo – CPU burst then 1-byte pipe exchange with a partner worker. Workers are paired: (0,1), (2,3), etc. Sleep duration depends on partner scheduling, exercising cross-CPU wake placement. Requires even num_workers.

FutexPingPong – paired futex wait/wake between partner workers. Each iteration does spin_iters of CPU work then wakes the partner and waits on a shared futex word. Exercises the non-WF_SYNC wake path. Requires even num_workers.

CachePressure – strided read-modify-write over a buffer sized to pressure the L1 cache. Each worker allocates its own buffer post-fork. size_kb controls buffer size, stride controls the byte step between accesses.

CacheYield – cache pressure followed by sched_yield(). Tests scheduler re-placement after voluntary yield with a cache-hot working set.

CachePipe – cache pressure burst then 1-byte pipe exchange with a partner worker. Combines cache-hot working set with cross-CPU wake placement. Requires even num_workers.

FutexFanOut – 1:N fan-out wake pattern without cache pressure. One messenger per group does spin_iters of CPU spin work then wakes fan_out receivers via FUTEX_WAKE. Receivers measure wake-to-run latency. For cache-aware fan-out with matrix multiply work, see FanOutCompute. Requires num_workers divisible by fan_out + 1.

FanOutCompute – messenger/worker fan-out with compute work. One messenger per group stamps a CLOCK_MONOTONIC timestamp then wakes fan_out workers via FUTEX_WAKE. Workers measure wake-to-run latency (time from messenger’s timestamp to worker getting the CPU), sleep for sleep_usec microseconds (simulating think time), then do operations iterations of naive matrix multiply over a cache_footprint_kb-sized working set (three square matrices of u64, O(n^3)). Requires num_workers divisible by fan_out + 1.

Sequence – compound work pattern: loop through phases in order, repeat. Each phase runs for its specified duration before the next starts. Phases are defined via the Phase enum:

• Phase::Spin(Duration) – CPU spin for the given duration.
• Phase::Sleep(Duration) – thread::sleep for the given duration.
• Phase::Yield(Duration) – repeated sched_yield for the given duration.
• Phase::Io(Duration) – simulated I/O (write 64 KB + 100 us sleep) for the given duration.

Sequence cannot be constructed via WorkType::from_name() because it requires explicit phase definitions. Build it directly:

WorkType::Sequence {
    first: Phase::Spin(Duration::from_millis(100)),
    rest: vec![
        Phase::Sleep(Duration::from_millis(50)),
        Phase::Yield(Duration::from_millis(20)),
    ],
}

ForkExit – rapid fork+_exit cycling. Each iteration forks a child that immediately calls _exit(0). The parent waitpids then repeats. Exercises wake_up_new_task, do_exit, and wait_task_zombie.

NiceSweep – cycles the worker’s nice level from -20 to 19 across iterations. Each iteration: 512-iteration spin burst, setpriority(PRIO_PROCESS, 0, nice_val), then sched_yield. Exercises reweight_task and dynamic priority reweighting. Skips negative nice values when CAP_SYS_NICE is absent. Resets nice to 0 before exit. Records per-yield wake latency.

AffinityChurn – rapid self-directed CPU affinity changes. Each iteration: spin_iters spin burst, sched_setaffinity to a random CPU from the effective cpuset, then sched_yield. Exercises affine_move_task and migration_cpu_stop. Records per-yield wake latency.

PolicyChurn – cycles through scheduling policies each iteration. Each iteration: spin_iters spin burst, sched_setscheduler to the next policy in the sequence, then sched_yield. Cycles through SCHED_OTHER, SCHED_BATCH, SCHED_IDLE (and SCHED_FIFO/SCHED_RR with priority 1 when CAP_SYS_NICE is available). Exercises __sched_setscheduler and scheduling class transitions. Resets to SCHED_OTHER before exit. Records per-yield wake latency.

PageFaultChurn – rapid page fault cycling. Workers mmap a region_kb KB region with MADV_NOHUGEPAGE (forcing 4 KB pages), touch touches_per_cycle random pages via write faults through do_anonymous_page, then MADV_DONTNEED to zap PTEs and repeat. spin_iters iterations of CPU work separate cycles. Exercises the page allocator, TLB pressure on migration, and rapid user/kernel transitions. Uses xorshift64 PRNG for random page selection (seeded from the process ID).

MutexContention – N-way futex mutex contention. contenders workers per group contend on a shared AtomicU32 via CAS acquire (FUTEX_WAIT on failure). Loop: spin_burst(work_iters) then CAS acquire, spin_burst(hold_iters) in the critical section, then store 0 + FUTEX_WAKE(1) to release. Exercises convoy effect, lock-holder preemption cascading stalls, and futex wait/wake contention paths. Requires num_workers divisible by contenders.

Custom – user-supplied work function. The run function pointer receives a reference to the stop flag (&AtomicBool, set by SIGUSR1) and returns a WorkerReport when the flag becomes true. The framework handles fork, cgroup placement, affinity, scheduling policy, and signal setup; the user function owns the work loop and all WorkerReport field population. Framework telemetry (migration tracking, gap detection, schedstat deltas, iteration counter updates) is not provided – the user function is responsible for any telemetry it needs.

Warning — pgid SIGKILL sweep on teardown. Every worker process calls setpgid(0, 0) immediately after fork, so the worker and any children a Custom closure spawns share a single process group. At teardown, stop_and_collect issues killpg(worker_pid, SIGKILL) on BOTH the graceful-exit and StillAlive-escalation paths, and WorkloadHandle::drop issues another killpg during handle destruction. Every descendant that inherits the worker’s pgid (a helper binary via execv, a subshell via sh -c, a test fixture the closure forks to drive the scheduler) will be SIGKILLed at teardown. Closures that need a child to outlive the worker must either detach it from the worker’s pgid (call setpgid(child_pid, 0) after fork) or wait on it explicitly before returning the WorkerReport.

Function pointers (fn(&AtomicBool) -> WorkerReport) are fork-safe because they carry no captured state across the fork boundary. Closures are not supported. Cannot be constructed via WorkType::from_name().

use std::sync::atomic::{AtomicBool, Ordering};
use ktstr::workload::{WorkType, WorkerReport};

fn my_workload(stop: &AtomicBool) -> WorkerReport {
    // `tid` in `WorkerReport` is an `i32` (libc::pid_t). Using
    // `std::process::id() as i32` avoids a direct `libc` dependency in
    // the consumer crate; inside ktstr the two produce the same value
    // because one worker = one process (no threads).
    let tid: i32 = std::process::id() as i32;
    let start = std::time::Instant::now();
    let mut work_units = 0u64;
    while !stop.load(Ordering::Relaxed) {
        // ... custom work ...
        work_units += 1;
    }
    let wall_time_ns = start.elapsed().as_nanos() as u64;
    // Start from `WorkerReport::default()` so the fields you don't
    // populate take their zero / empty values automatically and new
    // fields added to `WorkerReport` in the future do not require an
    // edit here. Only populate the telemetry your custom workload
    // actually produces.
    WorkerReport {
        tid,
        work_units,
        wall_time_ns,
        iterations: work_units,
        ..WorkerReport::default()
    }
}

let wt = WorkType::custom("my_workload", my_workload);

Grouped work types

PipeIo, FutexPingPong, and CachePipe require num_workers divisible by 2 (paired). FutexFanOut and FanOutCompute require num_workers divisible by fan_out + 1 (1 messenger + N receivers per group). MutexContention requires num_workers divisible by contenders. WorkType::worker_group_size() returns the group size for these variants, or None for ungrouped types. PipeIo and CachePipe use pipes; FutexPingPong, FutexFanOut, FanOutCompute, and MutexContention use shared mmap pages with futex wait/wake.

Clone-mode and pcomm interactions

CloneMode is a per-WorkloadConfig enum with two variants — Fork (the default; each worker is its own thread group, reaped via waitpid) and Thread (workers share the parent’s tgid, run as std::thread::spawn threads, reaped via JoinHandle).

pcomm is not a CloneMode variant — it is a WorkSpec field set via WorkSpec::pcomm(name) / CgroupDef::pcomm(name) in the tutorial. When a WorkSpec carries pcomm = Some(name), apply_setup routes it through the fork-then-thread spawn path: ONE forked thread-group leader whose task->comm is name hosts every matching worker as a pthread-style thread under that leader. Workers sharing a pcomm value coalesce into one container; this combines the per-process-leader visibility schedulers expect (a chrome parent, a java parent) with the in-process std::thread::spawn dispatch shape CloneMode::Thread already uses for the worker bodies themselves.

PipeIo and CachePipe work correctly inside a pcomm container. When workers run as threads inside one forked leader, the per-pair pipe-fd indices computed in the global pipe_pairs table are addressed by each worker’s position WITHIN the container’s thread group, so worker A reads its partner’s write end whether the pair lives in two forked processes (Fork mode) or in two threads of one pcomm container.

SignalStorm uses tkill(partner_tid, SIGUSR1) (per-task signal delivery, PIDTYPE_PID), NOT kill (per-tgid, PIDTYPE_TGID) and NOT tgkill(self_tgid, partner_tid, …) (would return ESRCH under Fork mode because each forked worker is its own tgid leader). tkill looks up the target via find_task_by_vpid(pid) and skips the tgid check, so the signal hits the partner thread’s per-task pending queue under Fork and Thread modes uniformly — including inside pcomm-coalesced thread groups. Sibling threads in a pcomm container do NOT dequeue each other’s SignalStorm signals because the PIDTYPE_PID queue is per-task, not per-tgid.

Default values

WorkType::from_name() uses these defaults:

• Bursty: burst_duration=50ms, sleep_duration=100ms
• PipeIo: burst_iters=1024
• FutexPingPong: spin_iters=1024
• CachePressure: size_kb=32, stride=64
• CacheYield: size_kb=32, stride=64
• CachePipe: size_kb=32, burst_iters=1024
• FutexFanOut: fan_out=4, spin_iters=1024
• FanOutCompute: fan_out=4, cache_footprint_kb=256, operations=5, sleep_usec=100
• AffinityChurn: spin_iters=1024
• PolicyChurn: spin_iters=1024
• PageFaultChurn: region_kb=4096, touches_per_cycle=256, spin_iters=64
• MutexContention: contenders=4, hold_iters=256, work_iters=1024

String lookup

WorkType::from_name() accepts PascalCase names matching the enum variants (e.g. "SpinWait", "FutexPingPong"). Sequence and Custom return None because they require explicit construction parameters. WorkType::ALL_NAMES lists every variant name. WorkType::name() returns the PascalCase name for a given value; for Custom, it returns the user-provided name field.

WorkloadConfig

WorkloadConfig is the low-level struct passed to WorkloadHandle::spawn(). CgroupDef builds one internally; use WorkloadConfig directly when calling setup_cgroups() or WorkloadHandle::spawn() in custom scenarios.

pub struct WorkloadConfig {
    pub num_workers: usize,           // Number of worker processes to fork
    pub affinity: AffinityIntent,     // Per-worker affinity intent (resolved at spawn time)
    pub work_type: WorkType,          // What each worker does
    pub sched_policy: SchedPolicy,    // Linux scheduling policy
    pub mem_policy: MemPolicy,        // NUMA memory placement policy
    pub mpol_flags: MpolFlags,        // Optional mode flags for set_mempolicy(2)
    pub nice: Option<i32>,            // Per-worker nice via setpriority(2); None inherits
    pub clone_mode: CloneMode,        // Fork (default) or Thread dispatch
    pub comm: Option<Cow<'static, str>>, // task->comm via prctl(PR_SET_NAME); kernel truncates to 15 bytes
    pub uid: Option<u32>,             // Effective UID via setresuid; None inherits
    pub gid: Option<u32>,             // Effective GID via setresgid; None inherits
    pub numa_node: Option<u32>,       // Restrict affinity to one NUMA node's CPU set
    pub composed: Vec<WorkSpec>,      // Secondary worker groups spawned alongside the primary
}

Default: 1 worker, AffinityIntent::Inherit, SpinWait, Normal policy, Default mem_policy, no mpol_flags, nice/comm/uid/gid/numa_node = None, clone_mode = Fork, composed = empty.

AffinityIntent is the type-unified affinity expression used at the top level and inside WorkSpec entries — Inherit, Exact(...), and RandomSubset(...) are accepted at WorkloadHandle::spawn; topology-aware variants (SingleCpu, LlcAligned, CrossCgroup, SmtSiblingPair) require scenario context and are rejected at the spawn gate with an actionable diagnostic. composed carries secondary WorkSpec groups that spawn alongside the primary; each composed entry can override work_type, num_workers, sched_policy, affinity, etc., and reports back via WorkerReport::group_idx (0 for the primary, 1..=N for composed entries in declaration order).

See MemPolicy for the NUMA memory placement API.

Scheduling policies

Workers can run under different Linux scheduling policies:

pub enum SchedPolicy {
    Normal,
    Batch,
    Idle,
    Fifo(u32),       // priority 1-99
    RoundRobin(u32), // priority 1-99
    Deadline {
        runtime: Duration,   // budget per period
        deadline: Duration,  // relative deadline from period start
        period: Duration,    // period; Duration::ZERO uses `deadline`
    },
}

Fifo, RoundRobin, and Deadline require CAP_SYS_NICE. The sched-deadline gate (runtime <= deadline <= period, all non-zero unless period == Duration::ZERO, which the kernel substitutes with deadline) is validated user-side in SchedPolicy::deadline() before sched_setattr so a malformed Deadline fails fast rather than tunneling EINVAL through the syscall.

Overriding work types

The work type override (configured via gauntlet or Ctx.work_type_override) replaces the default SpinWait work type for all scenarios that use it. Scenarios with non-SpinWait work types are not overridden.

Overrides to grouped work types (PipeIo, FutexPingPong, CachePipe, FutexFanOut, FanOutCompute, MutexContention) are skipped when num_workers is not divisible by the work type’s group size.

Ops-based scenarios have a separate override mechanism via CgroupDef.swappable. See Ops and Steps.

Checking

ktstr checks scheduler behavior through two channels: worker-side telemetry and host-side monitoring.

Worker checks

After each scenario, ktstr collects WorkerReport from every worker process. Several checks run against these reports:

Starvation – any worker with work_units == 0 fails the test.

Fairness – workers in the same cgroup should get similar CPU time. The “spread” (max off-CPU% - min off-CPU%) must be below a threshold (15% in release builds, 35% in debug). Violations report the spread and per-cgroup statistics.

Scheduling gaps – the longest wall-clock gap observed at work-unit checkpoints. Gaps above a threshold (2000ms release, 3000ms debug) indicate the scheduler dropped a task. Reports include the gap duration, CPU, and timing.

Cpuset isolation – workers must only run on CPUs in their assigned cpuset. Any execution on an unexpected CPU fails the test. Opt-in via isolation = true on the #[ktstr_test] attribute or via Assert::check_isolation(); Assert::default_checks() leaves this None, so the runtime merge resolves to false and the check is skipped unless explicitly enabled.

Throughput parity – assert_throughput_parity() checks that workers produce similar throughput (work_units per CPU-second). Two thresholds:

• max_throughput_cv: coefficient of variation across workers. High CV means the scheduler gives some workers disproportionately less effective CPU. Requires at least 2 workers with nonzero CPU time.
• min_work_rate: minimum work_units per CPU-second per worker. Catches cases where all workers are equally slow (CV passes but absolute throughput is too low).

Neither threshold is set by default; enable via Assert setters or #[ktstr_test] attributes.

Benchmarking – assert_benchmarks() checks per-wakeup latency and iteration throughput. Three thresholds:

• max_p99_wake_latency_ns: p99 of all resume_latencies_ns samples across workers in a cgroup. Populated only for work types that record wake-to-run latency: IoSyncWrite, IoRandRead, IoConvoy, Bursty, PipeIo, FutexPingPong, CacheYield, CachePipe, FutexFanOut (receivers), Sequence (Sleep / Yield / Io phases), ForkExit, NiceSweep, AffinityChurn, PolicyChurn, FanOutCompute, MutexContention. Pure-CPU work types (SpinWait, Mixed, CachePressure, PageFaultChurn) do not record samples.
• max_wake_latency_cv: coefficient of variation of wake latency samples. High CV means inconsistent scheduling latency.
• min_iteration_rate: minimum outer-loop iterations per wall-clock second per worker.

None are set by default. Set via Assert setters or #[ktstr_test] attributes.

Monitor checks

The host-side monitor reads guest VM memory (per-CPU runqueue structs via BTF offsets) and evaluates:

• Imbalance ratio: max(nr_running) / max(1, min(nr_running)) across CPUs. The denominator is clamped to 1 so an all-idle sample does not divide by zero.
• Local DSQ depth: per-CPU dispatch queue depth.
• Stall detection: rq_clock not advancing on a CPU with runnable tasks. Idle CPUs and preempted vCPUs are exempt. See Monitor: Stall detection for exemption details.
• Event rates: scx fallback and keep-last event counters.

Monitor thresholds use a sustained sample window (default: 5 samples). A violation must persist for N consecutive samples before failing.

NUMA checks

When workers use a MemPolicy, ktstr collects NUMA page placement data and checks it against thresholds:

Page locality – assert_page_locality() checks the fraction of pages residing on the expected NUMA node(s). Expected nodes are derived from the worker’s MemPolicy::node_set() at evaluation time. Page counts come from WorkerReport::numa_pages (parsed from /proc/self/numa_maps). Returns 1.0 (vacuously local) when no pages are observed. Fails if the observed fraction falls below min_page_locality.

Cross-node migration – assert_cross_node_migration() checks the ratio of migrated pages to total allocated pages. WorkerReport::vmstat_numa_pages_migrated provides the delta of the numa_pages_migrated counter from /proc/vmstat over the work loop. Fails if the ratio exceeds max_cross_node_migration_ratio.

Slow-tier ratio – max_slow_tier_ratio checks the fraction of pages on memory-only NUMA nodes (CXL tiers). Fails if more than the specified fraction of pages land on memory-only nodes.

None of these thresholds are set by default. Set via Assert setters or #[ktstr_test] attributes.

Assert struct

Assert is a composable configuration that carries both worker checks and monitor thresholds:

pub struct Assert {
    // Worker checks
    pub not_starved: Option<bool>,
    pub isolation: Option<bool>,
    pub max_gap_ms: Option<u64>,
    pub max_spread_pct: Option<f64>,

    // Throughput checks
    pub max_throughput_cv: Option<f64>,
    pub min_work_rate: Option<f64>,

    // Benchmarking checks
    pub max_p99_wake_latency_ns: Option<u64>,
    pub max_wake_latency_cv: Option<f64>,
    pub min_iteration_rate: Option<f64>,
    pub max_migration_ratio: Option<f64>,

    // Monitor checks
    pub max_imbalance_ratio: Option<f64>,
    pub max_local_dsq_depth: Option<u32>,
    pub fail_on_stall: Option<bool>,
    pub sustained_samples: Option<usize>,
    pub max_fallback_rate: Option<f64>,
    pub max_keep_last_rate: Option<f64>,

    // NUMA checks
    pub min_page_locality: Option<f64>,
    pub max_cross_node_migration_ratio: Option<f64>,
    pub max_slow_tier_ratio: Option<f64>,
}

Every field is Option. None means “inherit from parent layer.”

Merge layers

Checking uses a three-layer merge:

1. Assert::default_checks() – baseline: not_starved enabled, monitor thresholds from MonitorThresholds::DEFAULT.
2. Scheduler.assert – scheduler-level overrides.
3. Per-test assert – test-specific overrides via #[ktstr_test] attributes.

All fields use last-Some-wins semantics. A Some(false) in a higher layer can disable a check that a lower layer enabled.

let final_assert = Assert::default_checks()
    .merge(&scheduler.assert)
    .merge(&test_assert);

Default thresholds

Worker checks

Debug builds run in small VMs with higher scheduling overhead, so thresholds are relaxed. Coverage-instrumented builds collect profraw data for code coverage analysis; all assertion and monitor threshold checks run normally.

Monitor checks

All monitor thresholds use the sustained_samples window – a violation must persist for N consecutive samples before failing.

Worker checks via Assert

Assert provides assert_cgroup() for running worker-side checks directly against collected reports:

let a = Assert::default_checks().max_gap_ms(5000);
let result = a.assert_cgroup(&reports, Some(&cpuset));

Use Assert for both the merge chain (#[ktstr_test] attributes, Scheduler.assert, execute_steps_with) and direct report checking.

Constants

• Assert::NO_OVERRIDES – identity for merge; every field is None, so it overrides nothing. This is not “no checks” – when used as a per-test or per-scheduler assert, the runtime chain still applies defaults because it merges default_checks() -> scheduler -> test.
• Assert::default_checks() – not_starved enabled, monitor thresholds populated from MonitorThresholds::DEFAULT.

AssertResult

AssertResult carries pass/fail status, diagnostic messages, and aggregated statistics from a scenario run.

Construction

• AssertResult::pass() – creates a passing result with empty details and default stats.
• AssertResult::skip(reason) – creates a passing result with a skip reason in details and skipped = true. Used when a scenario cannot run under the current topology or flag combination but is not a failure.
• AssertResult::fail(detail) – failing result carrying a single AssertDetail. Mirrors pass / skip for the failure axis.
• AssertResult::fail_msg(msg) – shortcut for the common case where the failure is a plain diagnostic message tagged DetailKind::Other.

Mutation and inspection

• result.note(msg) – append an informational annotation tagged DetailKind::Note. Does NOT flip passed or skipped — a note is context, not a verdict. Returns &mut Self so calls chain.
• result.with_note(msg) – builder-style sibling of note that consumes and returns self. Use at the return site to chain a context annotation onto a fresh result without an intermediate let mut.
• result.is_skipped() – convenience accessor returning skipped. Stats tooling uses this to subtract non-executions from pass counts.
• result.is_failed() – convenience accessor returning !passed. Mirrors is_skipped so branches reading “did this claim fail?” don’t negate .passed inline.

Fields

• passed: bool – whether all checks passed.
• skipped: bool – distinguishes a passing result that ran every check from one that skipped execution (topology / flag mismatch, prerequisite absent). AssertResult::skip sets this; pass / fail / fail_msg leave it false.
• details: Vec<AssertDetail> – structured diagnostic entries; each carries a kind: DetailKind (Other, Note, Skip, Temporal, …) plus a human-readable message: String. Consumers filter by kind for routing (failure vs informational note) and read message for display.
• stats: ScenarioStats – aggregated worker telemetry across all cgroups (spread, gaps, migrations, wake latency, iterations).
• measurements: BTreeMap<String, NoteValue> – structured per-test measurements keyed by name. Sidecar consumers and comparison tooling read this map directly without parsing details strings, so populate it (via Verdict::note_value during claim evaluation) for any value a downstream comparison needs to lift programmatically.

Merging

result.merge(other) combines two results. If other.passed is false, the merged result is also false. Details and stats are accumulated:

let mut combined = AssertResult::pass();
combined.merge(cgroup_0_result);
combined.merge(cgroup_1_result);
// combined.passed is false if either cgroup failed
// combined.details contains messages from both

Stats merging takes worst values across cgroups for spread, gap, wake latency, and migration ratio. Counters (total_workers, total_cpus, total_migrations, total_iterations) are summed.

For examples of overriding thresholds at the scheduler and per-test level, see Customize Checking.

Ops and Steps

The ops system is a composable way to express dynamic cgroup topology changes. It replaces hand-written Action::Custom functions for most dynamic scenarios.

Op

An Op is an atomic operation on the cgroup topology. The enum is #[non_exhaustive], so external pattern matches must end with .. to stay compatible across ktstr version bumps that add new variants:

Op constructors accept string literals directly (no .into() needed):

Op::add_cgroup("cg_0")
Op::set_cpuset("cg_0", CpusetSpec::disjoint(0, 2))
Op::stop_cgroup("cg_0")
Op::spawn("cg_0", WorkSpec::default().workers(4))
Op::set_affinity("cg_0", AffinityIntent::RandomSubset)
Op::spawn_host(WorkSpec::default().workers(4))
Op::freeze_cgroup("cg_0")
Op::unfreeze_cgroup("cg_0")
Op::snapshot("after_spawn")
Op::watch_snapshot("jiffies_64")

SpawnHost creates workers in the parent cgroup, not in a managed cgroup. Use this to simulate host-level CPU contention alongside managed cgroups.

OpKind

OpKind is a payload-free discriminant enum generated from Op via #[strum_discriminants]. It carries the same variant set as Op (AddCgroup, RemoveCgroup, …, RunPayload, WaitPayload, KillPayload, FreezeCgroup, UnfreezeCgroup, Snapshot, WatchSnapshot) with none of the inner fields, so it is cheap to copy and use as a map key. Framework code uses OpKind when it only cares WHICH operation ran (per-op statistics, stimulus-event tagging, verifier/monitor bookkeeping) without the payload. Test authors rarely spell OpKind directly — the strum::EnumIter derive also lets tooling enumerate every OpKind variant for coverage checks.

OpKind shares Op’s #[non_exhaustive] attribute: external pattern matches over OpKind must end with ...

CpusetSpec

CpusetSpec computes a cpuset from the topology at runtime. The enum is #[non_exhaustive], so external callers should construct via the associated constructor functions (see the list below this snippet) rather than naming variant literals — a future field addition (e.g. a stride on Range) can land behind a defaulted parameter without breaking call sites. Pattern matches over CpusetSpec must also end with ..:

pub enum CpusetSpec {
    Llc(usize),                          // All CPUs in an LLC
    Numa(usize),                         // All CPUs in a NUMA node
    Range { start_frac: f64, end_frac: f64 }, // Fraction of usable CPUs
    Disjoint { index: usize, of: usize },     // Equal disjoint partitions
    Overlap { index: usize, of: usize, frac: f64 }, // Overlapping partitions
    Exact(BTreeSet<usize>),              // Exact CPU set
}

Convenience constructors accept parameters directly: CpusetSpec::disjoint(0, 2), CpusetSpec::range(0.0, 0.5), CpusetSpec::exact([0, 1, 2]), CpusetSpec::llc(0), CpusetSpec::numa(0), CpusetSpec::overlap(0, 2, 0.5).

All fractional specs operate on usable_cpus().

CgroupDef

CgroupDef bundles three ops that always go together: create cgroup, set cpuset, spawn workers. It is the primary way to define cgroups in ops-based scenarios.

let def = CgroupDef::named("cg_0")
    .with_cpuset(CpusetSpec::disjoint(0, 2))
    .workers(4)
    .work_type(WorkType::SpinWait);

Builder methods

• .with_cpuset(CpusetSpec) – set the cpuset (CPU set the cgroup is pinned to).
• .with_cpuset_mems(BTreeSet<usize>) – explicit cpuset.mems override (default derives from the resolved cpuset’s NUMA nodes).
• .workers(n) – set worker count.
• .work_type(WorkType) – set work type (default: SpinWait).
• .sched_policy(SchedPolicy) – set Linux scheduling policy (default: Normal). See WorkSpec Types.
• .work(WorkSpec) – add a work group (multiple calls for concurrent groups).
• .workload(&'static Payload) – attach a binary workload payload to run alongside the worker group; the framework launches it as a child process inside the cgroup. Panics when called with a scheduler-kind Payload (PayloadKind::Scheduler(_)); the scheduler slot is #[ktstr_test(scheduler = ...)] at the test level, not the cgroup-level workload slot. Step-level Op::RunPayload rejects scheduler-kind payloads with an anyhow::Error instead of panicking; the build-time workload call panics because there is no scenario-level recovery path.
• .affinity(AffinityIntent) – set per-worker affinity (default: Inherit).
• .mem_policy(MemPolicy) – set NUMA memory placement policy (default: Default). See MemPolicy.
• .mpol_flags(MpolFlags) – set mode flags for set_mempolicy(2) (default: NONE). See MemPolicy.
• .nice(n) – cgroup-level default per-worker nice value, merged into every WorkSpec whose own nice is unset. See Tutorial: Step 11.
• .comm(name) – cgroup-level default per-worker task->comm via prctl(PR_SET_NAME). Merged into every WorkSpec whose own comm is unset.
• .pcomm(name) – thread-group-leader task->comm for the fork-then-thread spawn path (workers run as threads under one forked leader). Stamps every existing WorkSpec in-place; not order-independent with .work(...).
• .uid(uid) / .gid(gid) – cgroup-level default per-worker effective UID / GID via setresuid / setresgid. Merged into every WorkSpec whose own uid / gid is unset.
• .numa_node(node) – cgroup-level default NUMA-node affinity for every WorkSpec. Merged at apply-setup time.
• .swappable(bool) – opt into gauntlet work type override.

Cgroup controllers

The cgroup-v2 cpu / memory / io / pids controllers are exposed as typed setters (default: unconstrained):

• .cpu_quota_pct(pct) / .cpu_quota(quota, period) / .cpu_unlimited() – write cpu.max (pct is shorthand: 100 = one full CPU). cpu_unlimited resets to the kernel default.
• .cpu_weight(weight) – write cpu.weight (1..=10000, default 100).
• .memory_max(bytes) / .memory_high(bytes) / .memory_low(bytes) / .memory_unlimited() – write memory.max / memory.high / memory.low. memory_unlimited resets memory.max to max.
• .memory_swap_max(bytes) / .memory_swap_unlimited() – write memory.swap.max.
• .io_weight(weight) – write io.weight (1..=10000, default 100).
• .pids_max(n) / .pids_unlimited() – write pids.max.

MemPolicy-cpuset validation

When a cgroup has a cpuset, ktstr validates that the MemPolicy’s node set is covered by the NUMA nodes reachable from that cpuset. A MemPolicy::Bind([1]) on a cgroup whose cpuset covers only NUMA node 0 fails at setup time. Policies without a node set (Default, Local) skip validation.

WorkSpec type overrides and swappable

CgroupDef has a swappable flag (default: false). When true and a work type override is active (Ctx.work_type_override), the override replaces this def’s work type.

In contrast, the Scenario-level override (in run_scenario()) only replaces SpinWait work types. The two mechanisms serve different scopes:

• Scenario-level: replaces SpinWait in WorkSpec.work_type
• CgroupDef-level: replaces the work type when swappable = true

Both skip overrides to grouped work types when num_workers is not divisible by the work type’s group size.

WorkSpec type overrides apply only to CgroupDef setup, not to raw Op::Spawn. Op::Spawn always uses the work type as given. Use CgroupDef with .swappable(true) when the work type should participate in gauntlet overrides.

Step

A Step is a sequence of ops with a hold period:

pub struct Step {
    pub setup: Setup,   // CgroupDefs to create after ops
    pub ops: Vec<Op>,   // Operations to apply
    pub hold: HoldSpec, // How long to wait after
}

Setup is either Defs(Vec<CgroupDef>) or Factory(fn(&Ctx) -> Vec<CgroupDef>). Vec<CgroupDef> implements Into<Setup>, so you can write setup: vec![...].into() instead of setup: Setup::Defs(vec![...]).

Constructors

Step::new(ops, hold) – creates a step with ops only (no CgroupDef setup). Use when the step only applies dynamic operations to an existing topology.

Step::with_defs(defs, hold) – creates a step with CgroupDef setup and a hold period. The primary constructor for steps that create cgroups with workers.

Step::set_ops(self, ops) – REPLACES the ops on a step (builder method). Chain after with_defs to add dynamic operations to a step that also creates cgroups.

Naming asymmetry: Step::set_ops REPLACES; the sibling Backdrop::with_ops APPENDS. The two methods deliberately use different verbs to signal the different semantics. A Step::new(ops).set_ops(more) chain produces a step whose ops vec is exactly more (the original ops is dropped); a Backdrop::new().with_ops(ops_a).with_ops(ops_b) chain produces a backdrop whose ops vec is ops_a + ops_b. If you need to extend a step’s ops vec, build the combined Vec<Op> at the call site and pass it to set_ops, or compose at the Backdrop layer instead.

HoldSpec

How long to hold after a step completes:

HoldSpec::FULL is a constant for Frac(1.0) (hold for the full scenario duration).

execute_defs

execute_defs(ctx, defs) is a convenience wrapper for the common pattern of creating cgroups and running them for the full duration:

execute_defs(ctx, vec![
    CgroupDef::named("cg_0").workers(4),
    CgroupDef::named("cg_1").workers(4),
])

Equivalent to execute_steps(ctx, vec![Step::with_defs(defs, HoldSpec::FULL)]).

execute_steps

execute_steps(ctx, steps) runs a step sequence:

1. For each step: apply ops, then apply setup (create cgroups from CgroupDefs), hold for the specified duration. Ops run first so parent cgroups can be created before children are spawned. Loop steps reverse this: setup runs once before the loop, then ops repeat at the specified interval.
2. Check scheduler liveness between steps.
3. After all steps: collect worker reports and run checks.
4. Writes stimulus events to the SHM ring buffer for timeline analysis.

execute_steps_with

execute_steps_with(ctx, steps, assertions) is the same as execute_steps but accepts an explicit Assert for worker checks. execute_steps is a convenience wrapper that passes None.

use ktstr::prelude::*;

fn my_scenario(ctx: &Ctx) -> Result<AssertResult> {
    let assertions = Assert::NO_OVERRIDES
        .check_not_starved()
        .max_gap_ms(3000);

    let steps = vec![/* ... */];
    execute_steps_with(ctx, steps, Some(&assertions))
}

When assertions is Some, the provided Assert overrides ctx.assert for worker checks. When None, uses ctx.assert (the merged three-layer config: default_checks -> scheduler -> per-test).

TestTopology

TestTopology provides CPU topology information for test configuration. It discovers CPUs, last-level caches (LLCs), and NUMA nodes, and generates cpuset partitions for scenarios.

CPU topology hierarchy

ktstr models four levels of CPU topology, from largest to smallest:

• NUMA node – a memory-proximity domain. Each node is a group of CPUs with fast access to a local memory bank. Cross-node memory access is slower.
• LLC (last-level cache) – the largest cache shared by a group of cores. LLCs are the key scheduling boundary: tasks sharing an LLC benefit from shared cache lines.
• Core – a physical execution unit with its own pipeline and L1/L2 caches.
• Thread – an SMT (simultaneous multithreading) sibling. Multiple threads share a single core’s execution resources.

Containment: threads belong to a core, cores belong to an LLC, LLCs belong to a NUMA node. For example, 2n4l4c2t describes 2 NUMA nodes, each with 2 LLCs, each LLC with 4 cores, each core with 2 threads = 32 CPUs total.

Most tests use a single NUMA node (the default). NUMA matters when a scheduler makes placement decisions based on memory locality. Single-NUMA topologies (numa_nodes = 1) test scheduling without memory-locality effects. Multi-NUMA topologies test whether a scheduler keeps tasks close to their memory. See the gauntlet NUMA presets for multi-NUMA configurations.

use ktstr::prelude::*;

pub struct TestTopology {
    // private fields — use the accessors below
}

Construction

from_system() -> Result<Self> – reads sysfs (/sys/devices/system/cpu/) to discover the live topology. Reads LLC IDs, NUMA node IDs, core IDs, and cache sizes for each online CPU. Also scans /sys/devices/system/node/ to discover memory-only nodes (CXL), reads per-node meminfo and inter-node distances.

from_vm_topology(topo: &Topology) -> Self – builds a topology from a VM spec. Topology fields are big-to-little: NUMA nodes, last-level caches, cores per LLC, threads per core. Multiple LLCs can share a NUMA node when numa_nodes < llcs; llcs must be an exact multiple of numa_nodes so LLCs partition evenly across nodes (the declare_scheduler! macro rejects violations at compile time and runtime callers inside ktstr hold the same invariant). CPUs numbered sequentially. Used as a fallback when sysfs is incomplete inside a guest VM. For the memory-aware variant, see from_vm_topology_with_memory.

synthetic(num_cpus, num_llcs) -> Self (test-only) – creates a topology with evenly distributed CPUs across LLCs. Used in unit tests.

Topology queries

total_cpus() – total number of CPUs.

num_llcs() – number of last-level caches.

num_numa_nodes() – number of NUMA nodes.

all_cpus() -> &[usize] – all CPU IDs, sorted.

all_cpuset() -> BTreeSet<usize> – all CPU IDs as a set.

usable_cpus() -> &[usize] – CPUs available for workload placement. Reserves the last CPU for the root cgroup (cgroup 0) when the topology has more than 2 CPUs. On 8 CPUs: usable = 0-6, CPU 7 reserved. Most built-in scenarios and CgroupDef cpuset specs operate on usable_cpus() automatically; test authors rarely need to query it directly.

usable_cpuset() -> BTreeSet<usize> – usable CPUs as a set.

llcs() -> &[LlcInfo] – all LLC domains with their CPUs, NUMA node, cache size, and core map.

cpus_in_llc(idx) -> &[usize] – CPUs belonging to LLC at index.

llc_aligned_cpuset(idx) -> BTreeSet<usize> – CPUs in LLC as a set.

numa_aligned_cpuset(node) -> BTreeSet<usize> – CPUs in all LLCs belonging to NUMA node node. Filters LLCs by numa_node() == node and collects their CPUs.

numa_node_ids() -> &BTreeSet<usize> – NUMA node IDs as a BTreeSet.

numa_nodes_for_cpuset(cpus) -> BTreeSet<usize> – NUMA nodes covered by the given CPU set. Returns the set of NUMA nodes that contain at least one LLC with a CPU in the given set.

node_meminfo(node_id) -> Option<&NodeMemInfo> – per-node memory info (total and free KiB). Returns None when the node ID is not present or meminfo is unavailable. NodeMemInfo has total_kb, free_kb, and used_kb() (saturating subtraction).

numa_distance(from, to) -> u8 – inter-node NUMA distance. Returns 255 when either node ID is not present (matches the kernel’s unreachable distance). For from_vm_topology() topologies without explicit distances, returns 10 for local and 20 for remote.

is_memory_only(node_id) -> bool – whether the node is memory-only (has RAM but no CPUs). Typical for CXL-attached memory tiers.

Construction from VM topology

from_vm_topology(topo) -> Self – build a TestTopology from a Topology (the VMM’s topology spec). Populates LLCs, NUMA nodes, distances, per-node memory info, and memory-only node flags.

from_vm_topology_with_memory(topo, total_memory_mb) -> Self – same as from_vm_topology but accepts an optional total memory size for uniform topologies. When Some, divides memory evenly across nodes to populate NodeMemInfo. When None, memory info is omitted.

Cpuset generation

split_by_llc() -> Vec<BTreeSet<usize>> – one set of CPUs per LLC.

overlapping_cpusets(n, overlap_frac) -> Vec<BTreeSet<usize>> – generates n cpusets with overlap_frac overlap between adjacent sets. Available to scenarios that want to hand-build overlapping cpusets (e.g. via CpusetSpec::Exact); CpusetSpec::Overlap computes its slice inline rather than calling this helper.

cpuset_string(cpus) -> String – formats a CPU set as a compact range string (e.g. "0-3,5,7-9"). Used when writing cpuset.cpus.

LlcInfo

Each LLC domain is represented by an LlcInfo:

pub struct LlcInfo {
    cpus: Vec<usize>,
    numa_node: usize,
    cache_size_kb: Option<u64>,
    cores: BTreeMap<usize, Vec<usize>>, // core_id -> SMT siblings
}

Accessors: cpus(), numa_node(), cache_size_kb(), cores(), num_cores().

num_cores() returns the number of physical cores (from the core map), or falls back to cpus.len() if no core map is populated (synthetic topologies).

How scenarios use topology

TestTopology is available to scenarios via Ctx.topo. The CpusetSpec variants use topology methods to resolve a cgroup’s cpuset:

Llc confines a cgroup to a single LLC’s CPUs; Numa spans all LLCs in a NUMA node. The fraction- and partition-style variants operate on the usable-CPUs pool the host reservation has granted.

See Ops and Steps for the full CpusetSpec enum.

CPU list parsing

Two standalone functions parse CPU list strings:

parse_cpu_list(s) -> Result<Vec<usize>> – strict parsing of "0-3,5,7-9" format. Returns an error on invalid entries.

parse_cpu_list_lenient(s) -> Vec<usize> – lenient parsing that silently skips invalid entries.

See also: CgroupManager for set_cpuset() which consumes cpuset strings, CgroupGroup for RAII cgroup management, WorkloadHandle for worker lifecycle, Scenarios for how CpusetSpec drives cpuset partitioning.

Host-side reservation

TestTopology::numa_distance is also consumed at host-side --cpu-cap plan time: acquire_llc_plan uses it to order the spill from the seed NUMA node to nearest-by-distance neighbors when the CPU budget cannot fit within a single node. The resulting plan is a LOCK_SH reservation on every selected LLC (flock granularity stays per-LLC even when the last LLC is partial-taken for the CPU budget) with a cpuset.mems union written to a cgroup v2 sandbox. See Resource Budget for the full pipeline.

MemPolicy

MemPolicy controls NUMA memory placement for worker processes. It wraps set_mempolicy(2) and is applied after fork, before the work loop starts.

pub enum MemPolicy {
    Default,
    Bind(BTreeSet<usize>),
    Preferred(usize),
    Interleave(BTreeSet<usize>),
    Local,
    PreferredMany(BTreeSet<usize>),
    WeightedInterleave(BTreeSet<usize>),
}

Variants

Default – inherit the parent process’s memory policy. No set_mempolicy syscall is made.

Bind(nodes) – allocate only from the specified NUMA nodes (MPOL_BIND). Allocation fails with ENOMEM if all specified nodes are exhausted.

Preferred(node) – prefer allocations from the specified node, falling back to others when the preferred node is full (MPOL_PREFERRED).

Interleave(nodes) – interleave allocations round-robin across the specified nodes (MPOL_INTERLEAVE).

Local – prefer the nearest node to the CPU where the allocation occurs (MPOL_LOCAL). No nodemask.

PreferredMany(nodes) – prefer allocations from any of the specified nodes, falling back to others when all preferred nodes are full (MPOL_PREFERRED_MANY, kernel 5.15+).

WeightedInterleave(nodes) – weighted interleave across the specified nodes. Page distribution is proportional to per-node weights set via /sys/kernel/mm/mempolicy/weighted_interleave/nodeN (MPOL_WEIGHTED_INTERLEAVE, kernel 6.9+).

Convenience constructors

MemPolicy::bind([0, 1])
MemPolicy::preferred(0)
MemPolicy::interleave([0, 1])
MemPolicy::preferred_many([0, 1])
MemPolicy::weighted_interleave([0, 1])

Node-set constructors (bind, interleave, preferred_many, weighted_interleave) accept any IntoIterator<Item = usize> – arrays, ranges, Vec, BTreeSet. preferred takes a single usize node ID.

MpolFlags

MpolFlags provides optional mode flags OR’d into the set_mempolicy(2) mode argument:

Flags combine with | or MpolFlags::union():

let flags = MpolFlags::STATIC_NODES | MpolFlags::NUMA_BALANCING;

Usage in WorkSpec and CgroupDef

WorkSpec and CgroupDef both expose .mem_policy() and .mpol_flags() builder methods:

use ktstr::prelude::*;

let w = WorkSpec::default()
    .workers(4)
    .mem_policy(MemPolicy::bind([0]))
    .mpol_flags(MpolFlags::STATIC_NODES);

let def = CgroupDef::named("cg_0")
    .with_cpuset(CpusetSpec::numa(0))
    .workers(4)
    .mem_policy(MemPolicy::bind([0]));

Cpuset validation

When a cgroup has a cpuset, ktstr validates that the MemPolicy’s node set is covered by the NUMA nodes reachable from that cpuset. A MemPolicy::Bind([1]) on a cgroup whose cpuset covers only NUMA node 0 will fail with an error at setup time.

Policies without a node set (Default, Local) skip validation.

node_set()

MemPolicy::node_set() returns the NUMA node IDs referenced by the policy. Returns the node set for Bind, Interleave, PreferredMany, and WeightedInterleave; a single-element set for Preferred; and an empty set for Default/Local.

NUMA checking

Page locality and migration results from workers using MemPolicy are checked by the NUMA checking assertions. The expected node set for locality checks is derived from the worker’s MemPolicy at evaluation time.

Example: NUMA-aware test

A complete test that checks page locality across two NUMA nodes:

use ktstr::prelude::*;

#[ktstr_test(
    numa_nodes = 2, llcs = 4, cores = 4, threads = 1,
    min_numa_nodes = 2,
    min_page_locality = 0.8,
)]
fn numa_locality(ctx: &Ctx) -> Result<AssertResult> {
    execute_defs(ctx, vec![
        CgroupDef::named("node0")
            .with_cpuset(CpusetSpec::numa(0))
            .workers(4)
            .mem_policy(MemPolicy::bind([0])),
        CgroupDef::named("node1")
            .with_cpuset(CpusetSpec::numa(1))
            .workers(4)
            .mem_policy(MemPolicy::bind([1])),
    ])
}

Each cgroup’s workers are pinned to a single NUMA node’s CPUs via CpusetSpec::numa() and their memory allocations are bound to the same node via MemPolicy::bind(). The min_page_locality threshold fails the test if less than 80% of pages land on the expected node.

Performance Mode

Performance mode reduces noise during VM execution by applying host-side isolation (vCPU pinning, hugepages, NUMA mbind, RT scheduling). On x86_64, additionally: a guest-visible CPUID hint (KVM_HINTS_REALTIME) and KVM exit suppression (PAUSE and HLT VM exits disabled). On aarch64, the four host-side optimizations apply (vCPU pinning, hugepages, NUMA mbind, RT scheduling); KVM exit suppression and CPUID hints are not available.

What it does

On x86_64, seven optimizations are applied when performance_mode is enabled (six host-side, one guest-visible via CPUID). On aarch64, four of these apply (vCPU pinning, hugepages, NUMA mbind, RT scheduling); the x86-specific items (PAUSE/HLT exit disabling, KVM_HINTS_REALTIME CPUID, halt poll) are not available.

Host-side KVM_CAP_HALT_POLL is explicitly skipped on x86_64 — the guest haltpoll cpuidle driver disables it via MSR_KVM_POLL_CONTROL (see below):

vCPU pinning – each virtual LLC is mapped to a physical LLC group on the host. vCPU threads are pinned to cores within their assigned LLC via sched_setaffinity. This prevents the host scheduler from migrating vCPU threads across LLCs, which would add cache thrashing noise to measurements.

Hugepages – guest memory is allocated with 2MB hugepages (MAP_HUGETLB) when sufficient free hugepages exist. This eliminates TLB pressure from host-side page walks during guest execution.

NUMA mbind – guest memory is bound to the NUMA node(s) of the pinned vCPUs via mbind(MPOL_BIND). This ensures memory allocations are local to the CPUs executing vCPU threads, avoiding cross-node memory access latency.

RT scheduling – vCPU threads are set to SCHED_FIFO priority 1. The watchdog and monitor threads run at priority 2 on a dedicated host CPU not assigned to any vCPU, so they can preempt for timeout/sampling without competing for vCPU cores. The serial console mutex uses PTHREAD_PRIO_INHERIT to avoid priority inversion between RT vCPU threads and service threads.

Disable PAUSE VM exits (x86_64 only) – KVM_CAP_X86_DISABLE_EXITS with KVM_X86_DISABLE_EXITS_PAUSE suppresses VM exits on PAUSE instructions. Guest spinlocks execute PAUSE in tight loops; each PAUSE normally causes a vmexit so the hypervisor can schedule other vCPUs. With dedicated cores (vCPU pinning), this reschedule is unnecessary overhead. The capability is optional – if unsupported, a warning is logged and the VM proceeds without it.

Disable HLT VM exits (x86_64 only) – KVM_X86_DISABLE_EXITS_HLT suppresses VM exits on HLT instructions, the most frequent exit type during boot and idle. BSP shutdown detection uses I8042 reset (port 0x64, value 0xFE via reboot=k) and VcpuExit::Shutdown instead of VcpuExit::Hlt. KVM blocks HLT disable when mitigate_smt_rsb is active (host has X86_BUG_SMT_RSB and cpu_smt_possible()); in that case, only PAUSE exits are disabled.

KVM_HINTS_REALTIME CPUID (x86_64 only) – sets bit 0 of CPUID leaf 0x40000001 EDX, telling the guest kernel that vCPUs are pinned to dedicated host cores. The guest disables PV spinlocks, PV TLB flush, and PV sched_yield (all add hypercall overhead unnecessary on dedicated cores), and enables haltpoll cpuidle (polls briefly before halting, reducing wakeup latency). PV spinlocks require CONFIG_PARAVIRT_SPINLOCKS, which is not in ktstr.kconfig, so that disable is a no-op for ktstr guests.

Skip host-side halt poll (x86_64 only) – when a guest vCPU halts (executes HLT with nothing to do), KVM can busy-wait briefly on the host before putting the vCPU thread to sleep, reducing wakeup latency at the cost of host CPU time. KVM_CAP_HALT_POLL controls this per-VM ceiling. In performance mode it is not set because the guest haltpoll cpuidle driver (enabled by KVM_HINTS_REALTIME above) handles polling inside the guest and writes MSR_KVM_POLL_CONTROL=0 to disable host-side polling via kvm_arch_no_poll(). Non-performance-mode VMs set KVM_CAP_HALT_POLL to 200µs (matching the x86 kernel default), or 0 when vCPUs exceed host CPUs.

Prerequisites

Sufficient host CPUs – the host must have at least (llcs * cores_per_llc * threads_per_core) + 1 online CPUs. The extra CPU is reserved for service threads (monitor, watchdog) so they do not share a core with any RT vCPU. The host must also have at least as many LLC groups as virtual LLCs.

2MB hugepages (optional) – the host must have free 2MB hugepages (check /sys/kernel/mm/hugepages/hugepages-2048kB/free_hugepages). Without them, guest memory uses regular pages. A warning is printed.

CAP_SYS_NICE or rtprio limit (optional) – SCHED_FIFO requires either CAP_SYS_NICE (root) or an RLIMIT_RTPRIO >= the requested priority. Set an rtprio limit for non-root use:

# /etc/security/limits.conf
username  -  rtprio  99

Log out and back in for the limit to take effect. Without either capability, RT scheduling is skipped with a warning and vCPU threads run at normal priority (results may be noisy).

Validation

validate_performance_mode() runs during VM build and applies two levels of checks:

Errors (fatal):

• Total vCPUs + 1 service CPU exceed available host CPUs.
• Virtual LLCs exceed available LLC groups.
• Pinning plan cannot be satisfied (an LLC group has fewer available CPUs than the virtual LLC requires).
• No free host CPU for service threads after vCPU assignment.

Warnings (non-fatal):

• Insufficient free hugepages – regular page allocation is used.
• Host load is high – procs_running from /proc/stat exceeds half the vCPU count, results may be noisy.
• TSC not stable (x86_64 only, checked at VM creation time) – KVM_CLOCK_TSC_STABLE not set after KVM_GET_CLOCK, kvmclock falls back to per-vCPU timekeeping. Timing measurements may have higher variance. Common in nested virtualization.

Usage

In #[ktstr_test]:

#[ktstr_test(
    llcs = 2,
    cores = 4,
    threads = 2,
    performance_mode = true,
)]
fn my_perf_test(ctx: &Ctx) -> Result<AssertResult> {
    // vCPUs are pinned, hugepage-backed
    Ok(AssertResult::pass())
}

Via the builder API:

let vm = vmm::KtstrVm::builder()
    .kernel(&kernel_path)
    .init_binary(&ktstr_binary)
    .topology(1, 2, 4, 2)
    .memory_mb(4096)
    .performance_mode(true)
    .build()?
    .run()?;

When to use

Performance mode is for tests where host-side scheduling noise affects results – fairness spread measurements, scheduling gap detection, imbalance ratio checks. It is not needed for correctness tests (cpuset isolation, starvation detection) where pass/fail is binary.

The gauntlet runs many VMs in parallel. Performance mode on parallel VMs can oversubscribe the host if scheduled naively. Avoid performance_mode unless the host has enough CPUs for the topology matrix.

Two dimensions

Performance mode serves two purposes:

Noise reduction – pinning, hugepages, NUMA mbind, and RT scheduling reduce measurement variance on both architectures. On x86_64, PAUSE and HLT VM exit disabling, the KVM_HINTS_REALTIME CPUID hint, and skipping host-side halt poll further reduce noise. Scheduling gaps, spread, and throughput checks become meaningful because host jitter is controlled. Without performance mode, a 50ms gap could be host noise; with it, the same gap indicates a scheduler problem.

Performance assertions – with stable measurements, tests can set tight thresholds (max_gap_ms, min_iteration_rate, max_p99_wake_latency_ns) to detect scheduling regressions. A test using execute_steps_with can pass custom Assert checks that are evaluated inside the guest against worker telemetry. These thresholds are only meaningful under performance mode’s controlled environment.

Nextest parallelism

Performance-mode tests each consume one LLC group on the host. The vm-perf test group in .config/nextest.toml sets a static max-threads limit. The flock-based LLC slot reservation (acquire_resource_locks) handles runtime contention: if all LLC slots are busy, the test returns ResourceContention.

On contention, the test returns exit code 0 (skip) – it never ran. The SKIP: prefix in stderr distinguishes skips from real passes.

LLC exclusivity validation

When performance_mode is enabled, the build step validates LLC exclusivity: each virtual LLC must reserve the entire physical LLC group it maps to. The validation sums the actual CPU count of each LLC group and checks the total (plus service CPU) fits within the host’s online CPUs. If validation fails, the build returns an error (tests skip with ResourceContention).

Three-way mode tier

ktstr’s host-side resource coordination has three effective tiers, selected by the combination of performance_mode, --no-perf-mode/KTSTR_NO_PERF_MODE, and --cpu-cap/KTSTR_CPU_CAP:

Tier 1: performance mode (full isolation)

Enabled when performance_mode=true is set on the VM builder (or via #[ktstr_test(performance_mode = true)]). Acquires LOCK_EX on each selected LLC’s /tmp/ktstr-llc-{N}.lock — the LLC-level exclusive lock already covers every CPU in the group, so per-CPU /tmp/ktstr-cpu-{C}.lock files are NOT touched (try_acquire_all in vmm/host_topology.rs short-circuits the per-CPU loop when LlcLockMode == Exclusive). Applies every isolation feature listed under “What it does”: vCPU pinning via sched_setaffinity, 2 MB hugepages, NUMA mbind, RT SCHED_FIFO scheduling, and (x86_64) PAUSE/HLT exit suppression + KVM_HINTS_REALTIME CPUID.

Tier 2: no-perf-mode with CPU-cap reservation

Enabled by --no-perf-mode / KTSTR_NO_PERF_MODE=1. Every no-perf-mode VM goes through acquire_llc_plan: the reservation is LOCK_SH across a NUMA-aware, consolidation-aware set of LLCs, sized to meet the CPU budget — either --cpu-cap N (or KTSTR_CPU_CAP=N) if set, or 30% of the calling process’s sched_getaffinity cpuset (minimum 1) if not. The flock granularity stays per-LLC; plan.cpus holds EXACTLY the budget (partial-take on the last LLC when the budget falls mid-LLC). Multiple no-perf-mode VMs coexist on the same LLCs because shared locks are reentrant; a concurrent perf-mode VM attempting LOCK_EX blocks until every no-perf-mode peer has released.

Enforcement under --cpu-cap:

• cgroup v2 cpuset sandbox — the reserved CPUs and derived NUMA nodes are written to a child cgroup’s cpuset.cpus and cpuset.mems, and the build pid is migrated into that cgroup, so make -jN gcc children inherit the binding. Under --cpu-cap, narrowing by a parent cgroup is a fatal error; without the flag (but with acquire_llc_plan still running on the 30% default) the sandbox warns and proceeds.
• Soft-mask affinity — vCPU threads receive a sched_setaffinity mask covering only the reserved CPUs, so the guest’s CPU placement respects the budget even though no pinning is applied.
• No RT scheduling, no hugepages, no mbind, no KVM exit suppression — these remain off; --cpu-cap is not a partial performance mode.
• make -jN hint — kernel-build pipelines pass plan.cpus.len() to make so gcc’s fan-out matches the reserved capacity rather than nproc.

This tier is mutually exclusive with performance_mode=true (on the CLI, clap requires = "no_perf_mode" rejects --cpu-cap without --no-perf-mode at parse time) and with KTSTR_BYPASS_LLC_LOCKS=1 (rejected at every entry point because the contract and the bypass escape hatch are contradictory). Library consumers that set performance_mode=true on KtstrVmBuilder directly bypass the CLI parse — KTSTR_CPU_CAP is silently ignored in that path because the builder’s perf-mode branch never consults CpuCap::resolve.

See Resource Budget for the CpuCap, LlcPlan, and ktstr locks surfaces in detail.

Tier 3: default (per-CPU window + LLC LOCK_SH)

Selected when neither performance_mode=true nor --no-perf-mode/KTSTR_NO_PERF_MODE is set — the default path for #[ktstr_test] entries that don’t declare performance_mode (entry.rs KtstrTestEntry::DEFAULT sets performance_mode: false). acquire_cpu_locks (in vmm/host_topology.rs) walks a contiguous CPU window, takes LOCK_EX on each window CPU’s /tmp/ktstr-cpu-{C}.lock, then additionally takes LOCK_SH on the LLC lockfiles covering those CPUs so a perf-mode (tier 1) VM cannot grab LOCK_EX on an LLC that this path is using. No pinning, no isolation, no cgroup sandbox — the per-CPU reservation is purely for host-scheduling-noise avoidance between concurrent VMs.

This is the ONLY tier that actually flocks per-CPU lockfiles. Tier 1 skips them (LLC EX already covers all CPUs in the group); tier 2 skips them (capped LLC SH is enforced via the cgroup cpuset and the flock is sufficient per-LLC coordination).

Disabling performance mode

--no-perf-mode (or KTSTR_NO_PERF_MODE=1) forces performance_mode=false. The result is tier 2 above — a CPU-capped LOCK_SH reservation (either explicit --cpu-cap N or the 30%-of-allowed default). The feature differences relative to tier 1 are:

• LLC flock mode — tier 1 holds LOCK_EX on each reserved LLC; tier 2 holds LOCK_SH. Multiple shared holders coexist; an exclusive holder blocks every shared acquirer and vice-versa.
• Per-CPU flocks — tier 1 relies on LLC-level LOCK_EX for exclusivity; per-CPU /tmp/ktstr-cpu-{C}.lock files are skipped (try_acquire_all in vmm/host_topology.rs short-circuits the per-CPU loop when LlcLockMode == Exclusive because the LLC lock already covers every CPU in the group). Tier 2 also skips them — the cgroup cpuset is the enforcement layer.
• vCPU pinning — tier 1 pins via sched_setaffinity to the reserved LLC’s CPUs. Tier 2 applies soft-mask affinity (budget-scoped but no 1:1 vCPU-to-CPU binding).
• RT scheduling — tier 1 only; tier 2 runs vCPU threads at normal priority.
• Hugepages — tier 1 only; tier 2 uses regular pages.
• NUMA mbind — tier 1 only; tier 2 instead writes cpuset.mems on its child cgroup to achieve NUMA locality at the cgroup layer.
• KVM exit suppression (x86_64) — tier 1 only; tier 2 leaves PAUSE and HLT exits enabled.
• KVM_HINTS_REALTIME CPUID (x86_64) — tier 1 only; tier 2 leaves the guest on PV spinlocks and standard cpuidle.

Use tier 2 on multi-tenant hosts where you want bounded concurrency (at most N concurrent builds or no-perf-mode VMs per host) but cannot afford the full perf-mode contract. Use tier 1 for regression measurement where host jitter must be controlled.

Available via:

• ktstr shell --no-perf-mode
• cargo ktstr test --no-perf-mode
• cargo ktstr coverage --no-perf-mode
• cargo ktstr llvm-cov --no-perf-mode
• cargo ktstr shell --no-perf-mode
• KTSTR_NO_PERF_MODE=1 (any value; presence is sufficient)

--cpu-cap N layers on top of any of the above when present; when absent, the 30%-of-allowed default applies automatically. The env var is read by every VM builder call site (test harness, auto-repro, verifier, shell). The CLI flags set the env var before test execution so library consumers inherit it.

Resource Budget

--cpu-cap N adds a third tier between full performance-mode isolation and unreserved no-perf-mode execution. Instead of “lock each reserved LLC exclusively” (perf-mode), it reserves a NUMA-aware, consolidation-aware set of host CPUs under LOCK_SH, enforces the reservation via a cgroup v2 cpuset sandbox, and scales make -jN fan-out to the reserved capacity. The flock granularity stays per-LLC: every selected LLC is flocked whole, but plan.cpus holds EXACTLY N CPUs (the last LLC is partial-taken when the budget falls mid-LLC). See Performance Mode for the comparison against the other two tiers.

Every no-perf-mode VM and kernel build runs through this pipeline — there is no “no cap” path. When --cpu-cap is absent, the planner applies a 30% default of the calling process’s sched_getaffinity cpuset (minimum 1 CPU). This keeps sched_setaffinity safe under cgroup-restricted CI runners (CI hosts, systemd slices, sudo-under-a-limited-cpuset) where the process cannot run on every online CPU even if sysfs lists them.

When to use it

• Multi-tenant CI hosts where unbounded parallelism starves concurrent builds but the full performance-mode contract (SCHED_FIFO, hugepages, NUMA mbind, KVM exit suppression) is too heavy.
• Kernel builds run alongside perf-mode VM tests — the shared LOCK_SH coordinates with the perf-mode LOCK_EX so make never stomps a measurement in progress.
• Concurrent no-perf-mode VMs on a shared host — a cap of N CPUs bounds how much capacity each run reserves; peers that would exceed the host’s flock availability wait rather than racing for CPU.

CpuCap — parsed and resolved

CpuCap::new(N: usize) -> Result<CpuCap> constructs a cap from a CLI integer. N is a CPU count. N == 0 is rejected with --cpu-cap must be ≥ 1 CPU (got 0) — zero is a scripting sentinel, not a silent “no cap” fallback.

CpuCap::resolve(cli_flag: Option<usize>) -> Result<Option<CpuCap>> is the three-tier precedence:

1. CLI flag (--cpu-cap N) wins over env var.
2. KTSTR_CPU_CAP=N env var applies when the CLI flag is absent. Empty string is treated as unset; 0 or non-numeric values produce the same rejection as the CLI path.
3. Neither set → Ok(None). The planner expands this into the 30%-of-allowed default at acquire time.

CpuCap::effective_count(allowed_cpus: usize) -> Result<usize> clamps at acquire time, not construction time. N > allowed_cpus returns a ResourceContention error naming both numbers — operators reading the error see immediately that the cap exceeds the process’s sched_getaffinity cpuset, not the host’s total online CPU count. Fixing the cap requires either lowering N or releasing the cgroup restriction on the calling process.

host_allowed_cpus — the reference set

host_allowed_cpus() reads the calling process’s allowed CPUs via sched_getaffinity(0) with a /proc/self/status Cpus_allowed_list: fallback. Every consumer of the --cpu-cap pipeline plans against this set instead of HostTopology::online_cpus, so sched_setaffinity on the plan’s CPU list never produces an empty effective mask under a cgroup-restricted runner.

An empty allowed set is a bail condition, not a fallback to “every CPU” — guessing on a misconfigured host is worse than failing visibly. A host topology that has no LLC overlapping the allowed set (sysfs and sched_getaffinity disagree — e.g. stale sysfs after hot-plug, cgroup cpuset pinned to CPUs the kernel no longer reports in LLC groups) also bails with an actionable diagnostic.

LlcPlan — the ACQUIRE result

acquire_llc_plan(topo, test_topo, cpu_cap) runs three phases:

1. DISCOVER — for every LLC, stat the canonical /tmp/ktstr-llc-{N}.lock, read /proc/locks once, and build a snapshot of holders per LLC. No flocks are taken.
2. PLAN — rank LLCs (eligible = at least one allowed CPU): consolidation (prefer LLCs with existing holders) first, then fresh LLCs, all tiebroken by ascending index. Seed on the highest-scored LLC’s NUMA node; greedily fill that node before spilling to nearest-by-distance nodes via TestTopology::numa_distance. Accumulate allowed-CPU contribution per LLC until the accumulated count meets target_cpus. Final acquire order is ascending LLC index for livelock safety.
3. ACQUIRE — non-blocking LOCK_SH on every selected LLC. A single EWOULDBLOCK drops every held fd and retries once (one TOCTOU retry — the second DISCOVER’s /proc/locks read IS the backoff; more retries would amplify livelock risk without adding coordination signal).

Partial-take on the last LLC

Post-ACQUIRE, the materialization layer walks each selected LLC’s CPUs in ascending order, intersects with the allowed set, and STOPS at exactly target_cpus total. The last selected LLC typically contributes only a prefix of its allowed CPUs — the flock is still held at LLC granularity (coordination with concurrent ktstr peers is always per-LLC), but plan.cpus reflects the exact CPU budget. sched_setaffinity masks and cgroup cpuset.cpus writes narrow to that exact set.

The returned LlcPlan carries:

• locked_llcs: Vec<usize> — selected host LLC indices, ASC.
• cpus: Vec<usize> — flat list of reserved CPUs, sized exactly target_cpus (a subset of every selected LLC’s allowed CPUs, with the last LLC possibly contributing only a prefix).
• mems: BTreeSet<usize> — NUMA nodes actually hosting plan.cpus (an LLC that contributes a partial slice only registers the nodes of its used CPUs).
• snapshot: Vec<LlcSnapshot> — per-LLC discovery trail.
• locks: Vec<OwnedFd> — RAII flock handles; Drop releases.

When mems spans more than one node (warn_if_cross_node_spill fires), stderr gets a ktstr: reserving LLCs […] across N NUMA nodes warning so the operator knows to expect cross-node memory latency. Single-node plans are silent.

Cgroup v2 cpuset sandbox

BuildSandbox::try_create(plan_cpus, plan_mems, hard_error_on_degrade) writes the plan into a child cgroup under the caller’s own cgroup, in the kernel-required order: cpuset.cpus → cpuset.mems → cgroup.procs. A task in a cgroup with empty cpuset.mems may be killed by the cpuset allocator, so migration into cgroup.procs MUST happen after both cpuset fields are populated.

After each cpuset write, .effective is read back. Narrowing by a parent cgroup (e.g. systemd slice restriction) is a fatal error under --cpu-cap (hard_error_on_degrade = true) and a warn- only degrade without the flag.

Drop migrates the build pid back to root, tolerates transient EBUSY on cgroup.rmdir (5 × 10 ms retries), and orphans the directory with a tag=resource_budget.cgroup_orphan_left warn- log if the rmdir still refuses. Orphans older than 24 h are swept on the next sandbox creation.

make -jN hint

make_jobs_for_plan(plan) returns plan.cpus.len().max(1). The kernel-build pipeline threads this as make -jN. Without the hint, make -j$(nproc) fans gcc children across every online CPU, defeating the cpuset reservation in scheduling terms — the kernel still enforces cpuset membership at the fs layer, but gcc’s parallel width silently violates the budget. The .max(1) floor guards against make -j0 (unbounded on GNU make).

ktstr locks — observational surface

ktstr locks (or cargo ktstr locks) prints every ktstr flock currently held on the host, cross-referenced against /proc/locks to name each holder by PID + truncated cmdline. Read-only — takes no flocks. Four categories:

1. LLC locks under /tmp/ktstr-llc-*.lock
2. Per-CPU locks under /tmp/ktstr-cpu-*.lock
3. Cache-entry locks under {cache_root}/.locks/*.lock
4. Run-dir locks under {runs_root}/.locks/{kernel}-{project_commit}.lock — held for the duration of the (pre-clear + write) cycle by serialize_and_write_sidecar so two concurrent ktstr processes targeting the same run-dir key serialize on the sidecar write rather than tearing each other’s mid-write files.

Flags:

• --json — emit a structured snapshot. One-shot uses to_string_pretty for readability; under --watch each frame is compact on its own line (ndjson-style) for streaming consumers. Top-level keys: llcs, cpus, cache, run_dirs. Each row names its lockfile path and a holders array; every holder has pid + cmdline.
• --watch <interval> — redraw on the given interval until SIGINT. Interval uses humantime syntax (100ms, 1s, 5m, 1h).

Use ktstr locks when --cpu-cap acquires fail with ResourceContention: the error already names busy LLCs, but the live snapshot shows every contending peer at once.

KTSTR_BYPASS_LLC_LOCKS — escape hatch

Setting KTSTR_BYPASS_LLC_LOCKS=1 (any non-empty value) skips acquire_llc_plan entirely. The VM boots or the kernel builds immediately without coordinating against any concurrent perf-mode run. Use only when the operator explicitly accepts measurement noise:

• A shell session doing unrelated work alongside tests.
• An isolated developer workstation.
• A CI queue that already serializes jobs at a higher layer.

Mutually exclusive with --cpu-cap / KTSTR_CPU_CAP at every entry point (CLI parse for shell + kernel build on both ktstr and cargo ktstr, the kernel_build_pipeline reservation phase, and the library-layer KtstrVmBuilder::build no-perf-mode branch). The error wording always contains "resource contract" so operators can grep for it; the contract and the bypass cannot coexist at any of those six sites.

Note: the performance_mode=true vs --cpu-cap exclusion is weaker. It is enforced at CLI parse (shell --cpu-cap requires --no-perf-mode via clap requires), but library consumers that set performance_mode=true on KtstrVmBuilder directly see KTSTR_CPU_CAP silently ignored — the builder’s perf-mode branch never calls CpuCap::resolve, it goes through validate_performance_mode + acquire_resource_locks (LOCK_EX) instead.

Filesystem requirement

Every ktstr lockfile (/tmp/ktstr-llc-*.lock, /tmp/ktstr-cpu-*.lock, {cache_root}/.locks/*.lock, {runs_root}/.locks/*.lock) must live on a local filesystem — tmpfs, ext4, xfs, btrfs, f2fs, or bcachefs are the explicitly-accepted set. flock(2) behavior on NFS, CIFS, SMB2, CephFS, AFS, and FUSE is unreliable: NFSv3 is advisory-only without an NLM peer and NFSv4 byte-range locking does not cover flock(2); SMB does not emit /proc/locks entries so ktstr cannot enumerate peer holders; Ceph MDS does not participate in flock serialization across nodes; AFS does not support flock(2) at all; FUSE flock semantics depend on whether the userspace server implements the op. try_flock statfs-checks every lockfile path at open time via reject_remote_fs in src/flock.rs — hitting any deny-listed filesystem produces an actionable runtime error naming the filesystem plus the remediation “Move the lockfile path to a local filesystem (tmpfs, ext4, xfs, btrfs, f2fs, bcachefs).” Unknown local filesystems (zfs, erofs, etc.) are not on the deny-list and pass through, on the basis that rejecting unknown-but-local is more disruptive than accepting a potentially-unreliable flock.

Related

• Performance Mode — the full-isolation tier; the tier comparison lives there.
• Environment Variables — KTSTR_CPU_CAP, KTSTR_BYPASS_LLC_LOCKS, and every other ktstr-controlled env var.

Writing Tests

Tests are Rust functions annotated with #[ktstr_test]. Each test boots a KVM VM, runs the scenario inside it, and evaluates results on the host.

use ktstr::prelude::*;

#[ktstr_test(llcs = 1, cores = 2, threads = 1)]
fn my_test(ctx: &Ctx) -> Result<AssertResult> {
    execute_defs(ctx, vec![
        CgroupDef::named("cg_0").workers(2),
        CgroupDef::named("cg_1").workers(2),
    ])
}

Run with cargo ktstr test --kernel ../linux. See Getting Started for setup and The #[ktstr_test] Macro for all available attributes. Each test also generates gauntlet variants across topology presets and flag profiles. See Gauntlet Tests. For scenarios that need logic beyond what the ops system can express, see Custom Scenarios.

The #[ktstr_test] Macro

#[ktstr_test] registers a function as an integration test that runs inside a VM.

Basic usage

use ktstr::prelude::*;

#[ktstr_test(llcs = 2, cores = 4, threads = 2)]
fn my_test(ctx: &Ctx) -> Result<AssertResult> {
    // ctx provides cgroup manager, topology, duration, etc.
    Ok(AssertResult::pass())
}

When a scheduler with a default topology is specified, the topology can be omitted:

use ktstr::declare_scheduler;

declare_scheduler!(MY_SCHED, {
    name = "my_sched",
    binary = "scx_my_sched",
    //          numa, llcs, cores/llc, threads/core
    topology = (1,    2,    4,         1),
});

#[ktstr_test(scheduler = MY_SCHED)]
fn inherited_topo(ctx: &Ctx) -> Result<AssertResult> {
    // Inherits 1n2l4c1t from MY_SCHED
    Ok(AssertResult::pass())
}

declare_scheduler! emits a pub static MY_SCHED: Scheduler and registers a private linkme static in the KTSTR_SCHEDULERS distributed slice. The scheduler = slot expects &'static Scheduler — pass the bare MY_SCHED ident; the macro takes a reference internally.

The function must have signature fn(&ktstr::scenario::Ctx) -> anyhow::Result<ktstr::assert::AssertResult>.

What the macro generates

1. Renames the function to __ktstr_inner_{name}.
2. Registers it in the KTSTR_TESTS distributed slice via linkme.
3. Emits a #[test] wrapper that calls run_ktstr_test().

The #[test] wrapper boots a VM with the specified topology and runs the function inside it.

Attributes

All attributes are optional with defaults.

Topology

Each dimension independently inherits from Scheduler.topology when a scheduler is specified and that dimension is not explicitly set. Without a scheduler, unset dimensions use macro defaults (numa_nodes=1, llcs=1, cores=2, threads=1). The default is a single-NUMA topology, so most tests do not need to set numa_nodes. See Default topology.

Scheduler

Payloads

See Payload Definitions for authoring new Payload fixtures; tests/common/fixtures.rs carries reusable examples (SCHBENCH, SCHBENCH_HINTED, SCHBENCH_JSON).

Checking

not_starved = true enables three distinct checks: starvation (any worker with zero work units), fairness spread (max-min off-CPU% below max_spread_pct), and scheduling gaps (longest gap below max_gap_ms). Each threshold can be overridden independently. See Customize Checking for override examples and Checking for the merge chain.

Topology constraints

The gauntlet skips presets that do not satisfy these constraints. Multi-NUMA presets are excluded by default (max_numa_nodes = 1). See Gauntlet for filtering rules and Gauntlet Tests for a worked example.

Execution

See Performance Mode for details on what performance_mode enables, prerequisites, and validation behavior.

Inline scheduler config

Some schedulers (e.g. scx_layered, scx_lavd) accept a JSON config file via a CLI argument like --config /path/to/config.json. Two pieces wire this into a test:

1. Scheduler declaration — the Scheduler builder declares the arg template and the guest path via .config_file_def:

   const LAYERED_SCHED: Scheduler = Scheduler::new("layered")
       .binary(SchedulerSpec::Discover("scx_layered"))
       .config_file_def("--config {file}", "/include-files/layered.json");

   {file} in the arg template is replaced with the guest path. The framework mkdir -ps the parent and writes the config content to /include-files/layered.json inside the guest before the scheduler binary starts.

2. Test attribute — the test supplies the inline JSON via config = …:

   const LAYERED_CONFIG: &str = r#"{ "layers": [...] }"#;
   
   #[ktstr_test(scheduler = LAYERED_SCHED, config = LAYERED_CONFIG)]
   fn layered_test(ctx: &Ctx) -> Result<AssertResult> {
       Ok(AssertResult::pass())
   }

   config = "..." (string literal) and config = SOME_CONST (path to a const &'static str) are both accepted.

The pairing gate is bidirectional:

• A scheduler with config_file_def set requires config = … on every test (otherwise the scheduler binary would launch without --config).
• A scheduler without config_file_def rejects config = … on the test (the content would be silently dropped at dispatch).

Both halves are validated at compile time via a const assertion emitted by the macro AND at runtime by KtstrTestEntry::validate, so direct programmatic-entry construction sees the same gate.

For schedulers that take a config file from a host-side path instead of inline content, use Scheduler::config_file(host_path) instead of config_file_def. The framework packs the host file into the initramfs at /include-files/{filename} and prepends --config /include-files/{filename} to scheduler args; no config = … on the test is needed in that flavor.

Example with custom scheduler

Define the scheduler with declare_scheduler! (see Scheduler Definitions), then reference it in #[ktstr_test]:

use ktstr::declare_scheduler;
use ktstr::prelude::*;

declare_scheduler!(MY_SCHED, {
    name = "my_sched",
    binary = "scx_my_sched",
    topology = (1, 2, 4, 1),
    sched_args = ["--enable-llc", "--enable-stealing"],
});

#[ktstr_test(
    scheduler = MY_SCHED,
    not_starved = true,
    max_gap_ms = 5000,
)]
fn my_sched_basic(ctx: &Ctx) -> Result<AssertResult> {
    // Inherits 1n2l4c1t from MY_SCHED
    Ok(AssertResult::pass())
}

declare_scheduler! emits a pub static MY_SCHED: Scheduler and registers it in the KTSTR_SCHEDULERS distributed slice via a private linkme static so cargo ktstr verifier discovers it. The bare MY_SCHED ident is what #[ktstr_test(scheduler = ...)] expects. See Scheduler Definitions for the full macro grammar.

For the manual builder pattern (no distributed-slice registration), see Scheduler Definitions: Manual definition.

Custom Scenarios

For dynamic scenarios (cgroup creation/removal, cpuset changes), prefer the ops/steps system over raw Action::Custom.

Use Action::Custom only when you need logic that the ops system cannot express.

Writing a custom scenario

use ktstr::prelude::*;
use ktstr::scenario::*;

fn my_custom_scenario(ctx: &Ctx) -> Result<AssertResult> {
    let wl = dfl_wl(ctx);
    let (handles, _guard) = setup_cgroups(ctx, 2, &wl)?;

    // Custom logic: resize cpusets, move workers, etc.
    std::thread::sleep(ctx.duration);

    Ok(collect_all(handles, &ctx.assert))
}

Helper functions

setup_cgroups(ctx, n, wl) – creates N cgroups, spawns workers, returns Result<(Vec<WorkloadHandle>, CgroupGroup)>. Bind the CgroupGroup to a named variable (e.g. _guard) so it lives until end of scope. See CgroupGroup for drop semantics.

Imports: setup_cgroups and dfl_wl live in ktstr::scenario, not in the prelude. The use ktstr::scenario::*; line in the example above is required — use ktstr::prelude::*; alone does not bring them into scope.

collect_all(handles, checks) – stops all workers, collects reports, runs worker-level checks when configured, otherwise falls back to assert_not_starved(). Merges results: if any worker group fails, the overall result fails.

dfl_wl(ctx) – creates a WorkloadConfig with ctx.workers_per_cgroup workers and default settings.

spawn_diverse(ctx, cgroup_names) – spawns different work types across cgroups, rotating through (SpinWait, Bursty{50ms burst / 100ms sleep}, IoSyncWrite, Mixed, YieldHeavy). Each cgroup uses ctx.workers_per_cgroup workers except IoSyncWrite cgroups, which always use 2 workers so blocking IO does not drown the scenario.

The Ctx struct

Custom scenarios receive a Ctx reference:

pub struct Ctx<'a> {
    pub cgroups: &'a dyn CgroupOps,
    pub topo: &'a TestTopology,
    pub duration: Duration,
    pub workers_per_cgroup: usize,
    pub sched_pid: Option<libc::pid_t>,
    pub settle: Duration,
    pub work_type_override: Option<WorkType>,
    pub assert: Assert,
    pub wait_for_map_write: bool,
}

cgroups – create/remove cgroups, set cpusets, move tasks. The slot is a &dyn CgroupOps trait object, not a concrete CgroupManager, so tests can substitute a no-op double for host-only scenarios while production paths receive the real manager. Method signatures are defined on CgroupOps; see CgroupManager for the production implementation.

topo – query CPU topology (LLCs, NUMA nodes, memory info, distances). Provides CPU enumeration, LLC/NUMA partitioning, cpuset generation, and inter-node distance queries. See TestTopology for the full API reference.

sched_pid – scheduler process ID (Option<libc::pid_t>) for liveness checks. None when the test runs without an scx scheduler (the EEVDF default path has no userspace scheduler binary). Unwrap or is_some_and(...) before passing to process_alive or kill(Pid::from_raw(pid), None).

settle – time to wait after cgroup creation for the scheduler to stabilize.

Checking in custom scenarios

Use Assert for both direct report checking and ops-based scenarios. Call assert.assert_cgroup(reports, cpuset) for manual report collection, or use execute_steps_with() for ops-based scenarios. See Checking.

Scheduler Definitions

A Scheduler tells the test framework how to launch and configure the scheduler under test. Tests reference one via #[ktstr_test(scheduler = MY_SCHED)]; the verifier sweep reads every declared scheduler from the KTSTR_SCHEDULERS distributed slice automatically.

The Scheduler type

pub struct Scheduler {
    pub name: &'static str,
    pub binary: SchedulerSpec,
    pub sysctls: &'static [Sysctl],
    pub kargs: &'static [&'static str],
    pub assert: Assert,
    pub cgroup_parent: Option<CgroupPath>,
    pub sched_args: &'static [&'static str],
    pub topology: Topology,
    pub constraints: TopologyConstraints,
    pub config_file: Option<&'static str>,
    pub config_file_def: Option<(&'static str, &'static str)>,
    pub kernels: &'static [&'static str],
}

config_file packs a host-side file into the initramfs at /include-files/{filename} and prepends --config /include-files/{filename} to scheduler args automatically.

config_file_def declares an arg-template + guest-path pair for schedulers that take inline JSON content via the test attribute #[ktstr_test(config = …)]: the framework writes the test’s config_content to the declared guest path and substitutes {file} in the arg template before launching the scheduler. The two fields are alternatives — config_file is the host-file path, config_file_def is the inline-content path. See The #[ktstr_test] Macro for the inline pairing gate.

sysctls takes Sysctl values. Construct them with Sysctl::new("key", "value") in const context. Use the dot-separated form for the key (e.g. "kernel.foo", not "kernel/foo"); duplicate keys are applied in order and the last write wins.

kargs is the extra GUEST KERNEL command-line (not the scheduler binary’s CLI — use sched_args for that). Do not override the kargs ktstr injects itself (console=, loglevel=, init=): those break guest-side init and leave the VM unable to run tests.

kernels is the per-scheduler filter on the BPF Verifier sweep matrix. The matrix dimension itself is the operator’s cargo ktstr verifier --kernel <SPEC> set (which the dispatcher always populates into KTSTR_KERNEL_LIST, including a single auto-discovered entry when --kernel is omitted). For each scheduler, the lister emits one cell per (kernel-list entry that passes this filter × accepted topology preset).

Each entry is a string consumed by KernelId::parse — the same parser as the cargo ktstr verifier --kernel <SPEC> CLI flag. Match semantics per variant:

• Exact Version ("6.14.2") — matches an entry whose raw or sanitized label equals the version.
• Range ("6.14..7.0" or "6.14..=7.0" — both inclusive on both endpoints) — matches entries whose raw version falls inside [start, end] via decompose_version_for_compare.
• Path / CacheKey / Git ("git+URL#REF", "path/to/dir", "6.14.2-tarball-x86_64-kc...") — matches by sanitized-label equality.

An empty kernels = [] means no filter — the scheduler verifies against every kernel-list entry the operator supplied.

SchedulerSpec

How to find the scheduler binary:

pub enum SchedulerSpec {
    Eevdf,                   // No sched_ext binary -- use kernel EEVDF
    Discover(&'static str),  // Auto-discover by name
    Path(&'static str),      // Explicit path
    KernelBuiltin {          // Kernel-built scheduler (no binary)
        enable: &'static [&'static str],
        disable: &'static [&'static str],
    },
}

KernelBuiltin is for schedulers compiled into the kernel (e.g. BPF-less sched_ext or debugfs-tuned variants). The enable commands run in the guest to activate the scheduler; disable commands run to deactivate it. No binary is injected into the VM.

SchedulerSpec::has_active_scheduling() returns true for all variants except Eevdf. When true, the framework runs monitor threshold evaluation after the scenario and enables auto-repro on crash.

Eevdf and KernelBuiltin are excluded from the verifier sweep at cell-emission time — neither has a userspace binary to load BPF programs from, so the verifier has nothing to verify.

Built-in: EEVDF

Scheduler::EEVDF runs tests without a sched_ext scheduler, using the kernel’s default EEVDF scheduler. Its binary is SchedulerSpec::Eevdf. It is the default scheduler for #[ktstr_test] entries that do not pass scheduler = ....

Defining a scheduler

declare_scheduler! is the preferred entry point: it constructs a pub static Scheduler and registers it in the KTSTR_SCHEDULERS distributed slice in one step, so the verifier sweep picks it up automatically.

use ktstr::declare_scheduler;
use ktstr::prelude::*;

declare_scheduler!(MY_SCHED, {
    name = "my_sched",
    binary = "scx_my_sched",
    sched_args = ["--exit-dump-len", "1048576"],
    topology = (1, 2, 4, 1),
    kernels = ["6.14", "6.15..=7.0"],
});

#[ktstr_test(scheduler = MY_SCHED)]
fn basic(ctx: &Ctx) -> Result<AssertResult> {
    execute_defs(ctx, vec![
        CgroupDef::named("cg_0").workers(2),
        CgroupDef::named("cg_1").workers(2),
    ])
}

The macro emits:

• pub static MY_SCHED: Scheduler with the supplied fields.
• A private static __KTSTR_SCHED_REG_MY_SCHED: &'static Scheduler registered in the KTSTR_SCHEDULERS distributed slice via linkme so cargo ktstr verifier discovers it.

#[ktstr_test(scheduler = ...)] expects an &'static Scheduler — pass the bare ident (e.g. MY_SCHED). The macro takes a reference internally, so passing the bare const yields the correct &Scheduler.

Accepted fields

Every key=value pair after name and binary is optional. The key names match Scheduler struct fields:

• name = "..." — short human name (required).
• binary = "scx_name" — defaults to SchedulerSpec::Discover(name). Accepts SchedulerSpec::Path("/abs/path"), SchedulerSpec::Eevdf, or SchedulerSpec::KernelBuiltin { enable: &[...], disable: &[...] }.
• sched_args = ["--a", "--b"] — CLI args prepended to every test that uses this scheduler.
• kernels = ["6.14", "6.15..=7.0", "git+URL#REF", "/path", "cache-key"] — verifier sweep set; see the field doc above.
• cgroup_parent = "/path" — must begin with /, must not be "/" alone.
• kargs = ["nosmt"] — guest-kernel cmdline additions.
• sysctls = [Sysctl::new("kernel.foo", "1")] — applied before the scheduler starts.
• topology = (numa_nodes, llcs, cores, threads) — default VM topology for #[ktstr_test] entries.
• constraints = TopologyConstraints { ... } — gauntlet topology constraints inherited by #[ktstr_test] entries.
• config_file = "configs/my_sched.toml" — opaque host-side config to pack into the guest initramfs.
• config_file_def = ("--config={file}", "/include-files/my.json") — alternative inline-config seam (see The #[ktstr_test] Macro).
• assert = Assert::NO_OVERRIDES.method_chain() — scheduler-level assertion overrides merged on top of Assert::default_checks().

Visibility

The identifier can be pub or pub(crate):

declare_scheduler!(pub MY_SCHED, { name = "my_sched", binary = "scx_my_sched" });
declare_scheduler!(pub(crate) INTERNAL, { name = "internal", binary = "scx_internal" });

The macro emits #[allow(missing_docs)] on the generated static so crates with #![deny(missing_docs)] compile cleanly.

Manual definition

The const builder pattern still works when the macro doesn’t fit — e.g. when the scheduler is composed programmatically or when test-only fixtures need to avoid the distributed-slice registration:

use ktstr::prelude::*;

const MITOSIS: Scheduler = Scheduler::new("scx_mitosis")
    .binary(SchedulerSpec::Discover("scx_mitosis"))
    .topology(1, 2, 4, 1)
    .sched_args(&["--exit-dump-len", "1048576"])
    .cgroup_parent("/ktstr")
    .assert(Assert::NO_OVERRIDES.max_imbalance_ratio(2.0));

A manually-defined Scheduler is not registered in KTSTR_SCHEDULERS automatically; the verifier sweep does not see it. Use declare_scheduler! for any scheduler that should participate in cargo ktstr verifier.

Cgroup parent

Scheduler.cgroup_parent specifies a cgroup subtree under /sys/fs/cgroup for the scheduler to manage. When set, the VM init creates the directory before starting the scheduler, and --cell-parent-cgroup <path> is injected into the scheduler args. The field is Option<CgroupPath>. CgroupPath::new() is a const constructor that panics at compile time if the path does not begin with / or is "/" alone. The Scheduler::cgroup_parent() builder and the declare_scheduler! cgroup_parent = "..." field both accept &'static str and construct a CgroupPath internally.

declare_scheduler!(MITOSIS, {
    name = "scx_mitosis",
    binary = "scx_mitosis",
    topology = (1, 2, 4, 1),
    cgroup_parent = "/ktstr",
});

This creates /sys/fs/cgroup/ktstr in the guest and passes --cell-parent-cgroup /ktstr to the scheduler binary.

Config file

Scheduler.config_file specifies a host-side path to an opaque config file that the scheduler binary reads at startup. The framework packs the file into the guest initramfs at /include-files/{filename} and prepends --config /include-files/{filename} to the scheduler args. ktstr does not parse or validate the file — it is passed through as-is.

The --config flag name is not configurable. Schedulers that use config_file must accept --config <path>. For schedulers that use a different flag, use config_file to place the file in the guest and add the desired flag via sched_args — the scheduler will also receive --config and must not reject unknown flags.

declare_scheduler!(MY_SCHED, {
    name = "my_sched",
    binary = "scx_my_sched",
    topology = (1, 2, 4, 1),
    config_file = "configs/my_sched.toml",
});

This copies configs/my_sched.toml from the host into the guest at /include-files/my_sched.toml and passes --config /include-files/my_sched.toml to the scheduler binary.

Scheduler args

Scheduler.sched_args provides default CLI args that apply to every test using this scheduler. They are prepended before per-test extra_sched_args.

declare_scheduler!(MITOSIS, {
    name = "scx_mitosis",
    binary = "scx_mitosis",
    topology = (1, 2, 4, 1),
    cgroup_parent = "/ktstr",
    sched_args = ["--exit-dump-len", "1048576"],
});

Merge order: config_file injection, then cgroup_parent injection, then sched_args, then per-test extra_sched_args.

Default topology

Scheduler.topology sets the default VM topology for all tests using this scheduler. When #[ktstr_test] omits llcs, cores, and threads, the scheduler’s topology is used. Explicit attributes on #[ktstr_test] override the scheduler default.

// (numa_nodes, llcs, cores_per_llc, threads_per_core)
declare_scheduler!(MITOSIS, {
    name = "scx_mitosis",
    binary = "scx_mitosis",
    topology = (1, 2, 4, 1),
});

Arguments are (numa_nodes, llcs, cores_per_llc, threads_per_core). Most schedulers use numa_nodes = 1 (single NUMA node). Scheduler::new() defaults to (1, 1, 2, 1) — a minimal 2-CPU single-NUMA VM, sufficient for tests that don’t exercise topology-dependent scheduling.

Tests that need a different topology (e.g. SMT) override individual dimensions. Unset dimensions still inherit from the scheduler:

// Inherits llcs=2, cores=4 from MITOSIS; overrides threads to 2
#[ktstr_test(scheduler = MITOSIS, threads = 2)]
fn smt_test(ctx: &Ctx) -> Result<AssertResult> { /* ... */ }

Checking overrides

Scheduler.assert provides scheduler-level checking defaults. These sit between Assert::default_checks() and per-test overrides in the merge chain.

A scheduler that tolerates higher imbalance:

declare_scheduler!(RELAXED, {
    name = "relaxed",
    binary = "scx_relaxed",
    assert = Assert::NO_OVERRIDES.max_imbalance_ratio(5.0),
});

Kernel-built scheduler example

For schedulers compiled into the kernel (no userspace binary), use SchedulerSpec::KernelBuiltin with shell commands to activate/deactivate the scheduler:

use ktstr::declare_scheduler;
use ktstr::prelude::*;

declare_scheduler!(MINLAT, {
    name = "minlat",
    binary = SchedulerSpec::KernelBuiltin {
        enable: &["echo minlat > /sys/kernel/debug/sched/ext/root/ops"],
        disable: &["echo none > /sys/kernel/debug/sched/ext/root/ops"],
    },
});

The enable commands run in the guest before scenarios start. The disable commands run after scenarios complete.

KernelBuiltin schedulers do not participate in the verifier sweep (no userspace binary to load BPF programs from); the declaration is still useful for #[ktstr_test(scheduler = ...)] attribution and sidecar identification.

Payload Definitions

A Payload describes a binary workload that a test can run alongside its cgroup workers. The struct encodes PayloadKind::Binary (an external executable — schbench, fio, stress-ng) for the workload role. Tests reference a Payload via #[ktstr_test(payload = FIXTURE)] (primary slot) or #[ktstr_test(workloads = [FIXTURE_A, FIXTURE_B])] (additional slots); the test body then runs it via ctx.payload(&FIXTURE).

The scheduler slot is separate from the payload / workloads slots — #[ktstr_test(scheduler = MY_SCHED)] takes a bare Scheduler reference (the declare_scheduler! const), not a Payload.

#[non_exhaustive] and construction rules

Payload is #[non_exhaustive] (see crate::non_exhaustive). Downstream crates cannot use struct-literal construction — a future ktstr bump can add fields without breaking callers only if everyone constructs through the provided associated functions:

• Payload::binary(name, binary) — minimal binary-kind Payload with exit-code-only defaults (no declared args, checks, metrics, or include files). Fills name, sets kind = PayloadKind::Binary(binary).
• Payload::new(...) — full positional constructor; the #[derive(Payload)] macro emits a call to this internally.

For richer binary payloads (custom default args, declared MetricChecks, MetricHints, include_files), use #[derive(Payload)] on a marker struct — the derive generates the matching const via the same non-exhaustive-preserving construction path. tests/common/fixtures.rs has worked examples — SCHBENCH, SCHBENCH_HINTED, SCHBENCH_JSON — suitable as reference shapes to copy.

Quick reference: Payload fields

The fields are listed here for readers tracing the fixture files, not as a license to hand-roll literals. Each is populated by Payload::binary + the derive’s builder methods:

• name: &'static str — display name that appears in sidecar JSON, stats tables, and test filtering. Distinct from the binary name (kind) so e.g. SCHBENCH_HINTED can run the same schbench binary with a different label.
• kind: PayloadKind — either Binary(executable_name) (for test payloads like schbench) or Scheduler(&'static Scheduler) (the in-memory shape of Payload::KERNEL_DEFAULT and similar scheduler-wrapping payloads). Test authors normally do not construct PayloadKind::Scheduler directly — the #[ktstr_test(scheduler = MY_SCHED)] slot takes the bare Scheduler ref without a Payload wrapper.
• output: OutputFormat — how to interpret the payload’s stdout/stderr. ExitCode (status code only), Json (parse numeric leaves), or LlmExtract(Option<&'static str>) (route through a local LLM with an optional hint).
• default_args: &'static [&'static str] — CLI args prepended to every invocation. Per-test ctx.payload(...).args(...) appends after these.
• default_checks: &'static [MetricCheck] — static assertions applied to the payload’s output/exit (min / max / range / exists / exit_code_eq constructors on MetricCheck). Merged with per-test .checks(...).
• metrics: &'static [MetricHint] — declared metrics the payload emits (name, unit, polarity). Drives list-metrics and comparison thresholds.
• metric_bounds: Option<&'static MetricBounds> — optional per-metric host-side bounds applied AFTER the payload exits. Consumed by LlmExtract payloads (where extraction runs host-side post-VM-exit); Json and ExitCode payloads ignore this field and route assertions through default_checks instead.
• include_files: &'static [&'static str] — extra files packaged into the guest alongside the binary (config files, datasets).
• uses_parent_pgrp: bool — when true, the payload child inherits the test’s process group so the teardown SIGKILL sweep reaches it. Most binaries leave this false and are reaped explicitly.
• known_flags: Option<&'static [&'static str]> — optional allow-list of CLI flags the payload accepts; used by the gauntlet-style flag expansion.

For an end-to-end workflow from building a scheduler to running the gauntlet, see Test a New Scheduler.

Gauntlet Tests

The gauntlet expands each #[ktstr_test] into a matrix of test × topology_preset variants. The test definition controls which cells of the matrix it populates.

Controlling topology coverage

Topology constraints in #[ktstr_test] filter which gauntlet presets a test runs on. See Topology Constraints for the full attribute table and Topology Presets for the preset list.

Worked example

A test with min_llcs = 2, requires_smt = true, and default max_numa_nodes = 1 against the preset table:

• tiny-1llc (1 LLC): excluded — below min_llcs
• All non-SMT presets (tiny-2llc, odd-*, *-nosmt): excluded — requires_smt
• near-max-llc (15 LLCs): excluded — above default max_llcs = 12
• max-cpu (252 CPUs, 14 LLCs): excluded — above default max_cpus = 192 (also above default max_llcs = 12)
• All numa* presets: excluded — above default max_numa_nodes = 1

Result: 6 of 24 presets survive (smt-2llc, smt-3llc, medium-4llc, medium-8llc, large-4llc, large-8llc). On aarch64, none survive — all aarch64 presets lack SMT.

Total variant count

The total number of gauntlet variants for a test is:

valid_presets × resolved_kernels

A test with 8 valid presets produces 8 gauntlet variants under a single-kernel run; passing two kernels (--kernel A --kernel B) doubles that to 16. The kernel dimension is contributed by cargo ktstr test / coverage / llvm-cov at the CLI surface (zero or one resolved kernels keeps the historical 3-segment name shape gauntlet/{name}/{preset}; two or more expands the gauntlet across kernels with an extra {kernel_label} segment). See Multi-kernel: kernel as a gauntlet dimension.

Tests that skip gauntlet

• Entries with host_only = true never produce gauntlet variants (no VM to vary topology on). They also skip the kernel-dim multiplication under multi-kernel runs: a host_only test lists and runs once regardless of KTSTR_KERNEL_LIST cardinality, since a host-side test never observes the kernel directory and N copies of identical work would carry no signal. See host_only for how that flag is set, and Multi-kernel: kernel as a gauntlet dimension for the kernel-suffix dispatch contract.
• Tests whose names start with demo_ are ignored by default. Their gauntlet variants are also ignored (all gauntlet variants are ignored).

Cross-references

• Gauntlet (Running Tests) — how to run gauntlet variants, preset table, budget interaction
• The #[ktstr_test] Macro — full attribute reference

Snapshots

A snapshot is a frozen record of guest BPF map state and scheduler globals captured at a specific point in a scenario. The freeze coordinator pauses every vCPU long enough to walk the kernel’s BPF maps, BTF-render every captured value, and bundle the result into a FailureDumpReport keyed by a name you choose. Test code then reads it back via the Snapshot accessor for typed traversal.

Op::snapshot("name") is the on-demand capture trigger. Use it to ask “what does the scheduler look like right now?” at a precise point in the scenario. For automatic capture on a kernel write to a specific symbol, see Watch Snapshots. For cadenced capture across the workload window without invoking Op::snapshot from the scenario body, see Periodic Capture — it produces a time-ordered SampleSeries that flows into the temporal-assertion patterns (nondecreasing, rate_within, steady_within, converges_to, always_true, ratio_within).

Issuing a snapshot

Op::snapshot(name) is a single op in a Step’s op list. The executor invokes the active SnapshotBridge’s capture callback, which performs the freeze rendezvous and returns the report; the bridge stores the report under name.

use ktstr::prelude::*;

let steps = vec![Step {
    setup: vec![CgroupDef::named("workers").workers(2)].into(),
    ops: vec![
        Op::snapshot("after_spawn"),
        // ... other ops ...
        Op::snapshot("after_workload"),
    ],
    hold: HoldSpec::FULL,
}];
execute_steps(ctx, steps)?;

A scenario may issue any number of Op::snapshot ops with distinct names. Reusing a name overwrites the prior capture (and emits a tracing::warn!).

Wiring the bridge

The bridge is what turns an Op::snapshot into stored data. The host typically wires it before execute_steps runs, but a scenario can install one inline:

use ktstr::prelude::*;

let cb: CaptureCallback = std::sync::Arc::new(|_name: &str| {
    // Production: freeze the VM and build a real FailureDumpReport.
    // Tests: return a hand-crafted report so the executor + bridge
    // pipeline runs without booting a guest.
    Some(FailureDumpReport::default())
});
let bridge = SnapshotBridge::new(cb);
let bridge_handle = bridge.clone();
let _guard = bridge.set_thread_local();

execute_steps(ctx, steps)?;

let captured = bridge_handle.drain();
let report = captured.get("after_spawn").expect("snapshot recorded");

set_thread_local returns a BridgeGuard that restores the prior bridge on drop, so a nested scenario inside an outer one cannot leak its bridge into the outer scope. Bind the guard to an underscore-prefixed identifier such as _guard so the binding lives for the scope of the scenario — a bare let _ = bridge.set_thread_local() drops the guard immediately and clears the bridge before any op runs. must_use will warn if the return value is discarded entirely.

If no bridge is installed, Op::snapshot is a no-op with a tracing::warn! and the scenario continues. If the capture callback returns None (capture pipeline unavailable), the bridge stays empty and the scenario continues. Existing scenarios that never declare snapshot ops keep working unchanged.

Reading the captured report

Snapshot::new(report) builds a borrowed view over a FailureDumpReport. The view does not copy the report; accessor methods walk the report in place and return further borrowed views.

Map-name lookup

let snap = Snapshot::new(report);
let map = snap.map("scx_per_task")?;        // SnapshotMap

Snapshot::map(name) returns Result<SnapshotMap, SnapshotError>. A miss yields SnapshotError::MapNotFound { requested, available } — the available list enumerates every captured map name so a typo surfaces in test output.

Top-level globals (.bss / .data / .rodata)

let nr_cpus = snap.var("nr_cpus_onln").as_u64()?;

Snapshot::var(name) walks every *.bss, *.data, and *.rodata global-section map for a top-level member named name and returns the unique match as a SnapshotField. Multiple matches yield SnapshotError::AmbiguousVar { requested, found_in } — disambiguate via Snapshot::map(name). A miss yields SnapshotError::VarNotFound { requested, available } with the union of every section’s top-level member names.

Entries inside a map

let map = snap.map("scx_per_task")?;
let first = map.at(0);                          // by ordinal index
let busy = map.find(|e| e.get("tid").as_i64().unwrap_or(-1) == 1234);
let busiest = map.max_by(|e| e.get("runtime_ns").as_u64().unwrap_or(0));
let all_active = map.filter(|e| e.get("runtime_ns").as_u64().unwrap_or(0) > 0);

SnapshotMap exposes:

• at(n) — entry at ordinal index n. Out of range returns SnapshotEntry::Missing(SnapshotError::IndexOutOfRange).
• find(predicate) — first matching entry. No match returns SnapshotEntry::Missing(SnapshotError::NoMatch { op: "find", ... }).
• filter(predicate) — every matching entry collected into a Vec.
• max_by(key_fn) — entry whose key_fn produces the maximum u64. Empty map returns Missing with op: "max_by".

Per-CPU maps

BPF_MAP_TYPE_PERCPU_ARRAY / _PERCPU_HASH / _LRU_PERCPU_HASH maps require narrowing to a CPU before reading individual values:

let map = snap.map("scx_pcpu")?;
let entry = map.cpu(1).at(0);                    // CPU 1's slot
let value = entry.get("").as_u64()?;             // empty path = root

SnapshotMap::cpu(n) narrows subsequent at / find calls to a specific CPU’s slot. An out-of-range CPU returns Missing with SnapshotError::PerCpuSlot { unmapped: false, len, ... }; an unmapped slot (None in the per-CPU vec) returns the same error variant with unmapped: true.

Calling entry.get(path) on a per-CPU entry without narrowing first surfaces SnapshotError::PerCpuNotNarrowed { map } — call .cpu(N) first.

Field accessors and dotted paths

SnapshotEntry::get(path) and SnapshotField::get(path) walk the entry’s value side along a dotted path. Each component matches a struct member; pointer dereferences are followed transparently.

let weight = entry.get("ctx.weight").as_u64()?;
let policy = entry.get("ctx.policy").as_str()?;     // enum variant name
let pid    = entry.get("leader.pid").as_i64()?;     // pointer chase

The dotted-path walker:

1. Pointer chase. When a path step lands on RenderedValue::Ptr { deref: Some(...) }, the walker transparently follows the dereference (up to 16 hops) before matching the next component. The test author writes the path the BTF would suggest; pointer indirection is invisible.

2. Empty path. get("") returns the current value as a SnapshotField::Value — useful for terminal accessors on per-CPU slots that hold a scalar directly.

3. Composability. Two-segment paths are equivalent to chained get calls: entry.get("ctx.weight") ≡ entry.get("ctx").get("weight").

   Note that Snapshot::var does not split — it treats the full string as one global name. To walk into a struct, use snap.var("ctx").get("weight").

Terminal accessors

SnapshotField exposes typed terminal reads, all returning Result<T, SnapshotError>:

Type mismatches surface as SnapshotError::TypeMismatch { expected, actual, requested } — for example, as_str() on a Uint reports expected: "Enum", actual: "Uint".

Cast-recovered pointers

Schedulers stash kernel pointers (task_struct *, cgroup *, …) and arena pointers in BPF map fields whose BTF declares them as u64 because BTF cannot express a pointer to a per-allocation type. The host-side cast analyzer walks the scheduler’s .bpf.o instruction stream during load, recovers the target struct for each provable (source_struct, field_offset) → target_struct mapping, and feeds the result into the renderer.

When the renderer encounters a u64 slot the analyzer flagged, it emits a RenderedValue::Ptr with cast_annotation set and chases the dereference through the address-space-appropriate reader. The full set of cast_annotation values:

A parallel cross-BTF Fwd resolution path is consulted whenever a chase target survives the local same-BTF Fwd resolve as a BTF_KIND_FWD: when the body lives in a sibling embedded BPF object’s BTF (the multi-.bpf.objs shape), the renderer switches the recursion to that sibling BTF and renders the full body. Cross-BTF resolution does NOT add a new annotation — the body is recovered transparently and the rendered subtree carries whichever annotation ("cast→arena", "cast→kernel", or None for a BTF-typed Type::Ptr) it would have had if the same struct lived in the entry BTF.

From the test author’s perspective:

• as_u64() returns the raw pointer value (matching pre-analysis behavior, so existing tests do not need updating).
• entry.get("ctx.task") and similar dotted-path walks transparently follow the cast-recovered chase; nested struct fields appear under the same path the BTF would suggest for a natively-typed pointer.
• The cast_annotation is visible in failure-dump rendering and diagnostic output so an operator can distinguish cast-recovered pointers from BTF-typed ones; the test API does not require any extra calls to consume them.

Error handling

SnapshotError is the unified error type for every fallible accessor. Each variant carries the path or available alternatives needed to fix the call site without re-running the test:

• MapNotFound { requested, available } — Snapshot::map(name) miss.
• VarNotFound { requested, available } — Snapshot::var(name) miss.
• AmbiguousVar { requested, found_in } — more than one *.bss/*.data/*.rodata map exposes a top-level member with the requested name. found_in lists every map (in capture order) where the name was seen; disambiguate via Snapshot::map(name) + .at(0).get(...) against a specific map.
• FieldNotFound { requested, walked, component, available } — a path component did not match any struct member at that depth. walked is the prefix that resolved successfully; component is the failing segment; requested is the original user-supplied path.
• NotAStruct { requested, walked, component, kind } — a path component reached a non-struct value where a struct was expected (e.g. descending into a Uint leaf). kind names the actual variant.
• TypeMismatch { expected, actual, requested } — terminal accessor called on a rendered shape it cannot decode. expected names the scalar type the accessor requires; actual names the rendered variant; requested is the user-supplied lookup string (empty when the accessor was invoked on a leaf without a path walk).
• IndexOutOfRange { map, index, len } — SnapshotMap::at(n) past the entry list end.
• PerCpuSlot { map, cpu, len, unmapped } — out-of-range or unmapped per-CPU slot; unmapped: true distinguishes a None slot from an out-of-range CPU.
• NoMatch { map, op } — predicate-based lookup (find, max_by) found no match. op names the operation.
• EmptyPathComponent { requested } — a path string contained an empty component (e.g. "a..b").
• PerCpuNotNarrowed { map } — entry.get called on a per-CPU entry without cpu(N) first.
• NoRendered { map, side } — entry has no rendered key/value side (BTF type id missing at capture time, leaving hex bytes only).
• PlaceholderSample { tag, reason } — a periodic-capture sample’s underlying FailureDumpReport is a placeholder produced by the freeze-rendezvous timeout fallback. Surfaces when projecting via SampleSeries::bpf; temporal patterns route the variant through their skip path so a placeholder never falsely registers as zero progress against a monotonicity / rate / steady / ratio band. reason carries the rendezvous-timeout cause text.
• MissingStats { tag } — a SampleSeries::stats projection ran on a sample whose stats slot is None (stats client not wired or per-sample stats request failed). Distinct from in-JSON path misses (FieldNotFound / TypeMismatch) so the assertion site can branch on the cause without re-walking the source.

SnapshotError implements std::error::Error and Display, so it composes with ? and anyhow. The Display impl includes the path and any available alternatives so a failure message points the test author at the fix.

Worked example

Capture a snapshot, look up a map, walk into its first entry, and read a nested field:

use ktstr::prelude::*;

fn snapshot_then_inspect(ctx: &Ctx) -> Result<AssertResult> {
    // Wire a bridge for the duration of the scenario.
    let cb: CaptureCallback = std::sync::Arc::new(|_name| {
        // Production: freeze + build a real FailureDumpReport. The
        // host installs this callback in real runs.
        Some(FailureDumpReport::default())
    });
    let bridge = SnapshotBridge::new(cb);
    let handle = bridge.clone();
    let _guard = bridge.set_thread_local();

    // Run the scenario, capturing once after spawn.
    let steps = vec![Step {
        setup: vec![CgroupDef::named("workers").workers(2)].into(),
        ops: vec![Op::snapshot("after_spawn")],
        hold: HoldSpec::FULL,
    }];
    let mut result = execute_steps(ctx, steps)?;

    // Drain the bridge and inspect the captured report.
    let captured = handle.drain();
    let report = captured
        .get("after_spawn")
        .ok_or_else(|| anyhow::anyhow!("snapshot 'after_spawn' missing"))?;
    let snap = Snapshot::new(report);

    // Top-level scalar.
    if let Ok(nr_cpus) = snap.var("nr_cpus_onln").as_u64() {
        result.details.push(AssertDetail::new(
            DetailKind::Other,
            format!("captured nr_cpus_onln = {nr_cpus}"),
        ));
    }

    Ok(result)
}

For the executor + bridge wiring outside a VM, see the host-side smoke tests in tests/snapshot_e2e.rs — they exercise the same pipeline against a hand-crafted FailureDumpReport so the assertion shape is covered without booting a guest.

Composing reads with writes

Snapshots are the read half of the host↔guest interaction. The write half — pre-seeding a BPF map value before the scenario starts — is the #[ktstr_test] attribute bpf_map_write = CONST, which targets a BpfMapWrite constant:

use ktstr::prelude::*;

const TRIGGER_FAULT: BpfMapWrite = BpfMapWrite {
    map_name_suffix: ".bss",   // matched against discovered maps
    offset: 42,                // byte offset within the map's value
    value: 1,                  // u32 written by the host
};

#[ktstr_test(bpf_map_write = TRIGGER_FAULT, expect_err = true)]
fn fault_then_inspect(ctx: &Ctx) -> Result<AssertResult> {
    // The host has already written `1` at `.bss + 42` before
    // the scenario started. Capture and inspect the resulting
    // scheduler state mid-run.
    /* bridge wiring + Op::snapshot + Snapshot::new as above */
    Ok(AssertResult::pass())
}

The write is event-driven: the host polls for BPF map discoverability (scheduler loaded), polls the SHM ring for scenario start, then writes the configured u32 at the configured offset. Only BPF_MAP_TYPE_ARRAY maps are supported; the framework finds the map by map_name_suffix (e.g. ".bss") via BpfMapAccessor::find_map. See Monitor → BPF map writes for the prerequisites (vmlinux) and the full host-side contract.

Read+write workflows then compose naturally: the test pre-seeds guest state with bpf_map_write, lets the scheduler run, and asserts on the resulting state with Op::snapshot + the Snapshot accessor:

1. Write (pre-scenario) — bpf_map_write flips a .bss flag the scheduler reads.
2. Run — the scenario’s ops drive workload behavior; the scheduler reacts to the flag.
3. Read (mid-scenario) — Op::snapshot("after") captures the scheduler state at the chosen point.
4. Assert — Snapshot::var(...).as_u64() / Snapshot::map(...).find(...).get(...).as_*() verifies the reaction. Errors carry the available alternatives so a typo or stale field name surfaces before the test author hand-edits the case.

The write side is a single one-shot poke at scheduler-load time; there is no Op variant for runtime writes. Ergonomic mid-scenario state mutation is reserved for cases where the scheduler itself exports a writable interface (sysfs, debugfs, BPF map command interface) and the test invokes that interface from a workload process.

Watch Snapshots

A watch snapshot registers a hardware data-write watchpoint on a named kernel symbol. The host arms a watchpoint slot via the guest’s hardware debug facilities; the produced captures share the Snapshot accessor surface documented in Snapshots.

Watch snapshots are supported on x86_64 and aarch64 KVM hosts. The slot terminology below is arch-neutral — each architecture’s KVM plumbing maps the slots onto its native hardware-watchpoint facility (debug registers on x86_64, hardware watchpoints on aarch64).

Op::watch_snapshot("symbol") is the write-driven capture trigger. Use it when the question is “what does the scheduler look like whenever the kernel touches X?” rather than “what does it look like at this point in my scenario?”. For time-driven capture, use Op::snapshot instead.

How it works

The full pipeline is implemented and tested end-to-end:

1. Op::watch_snapshot(symbol) registers the symbol via the virtio-console port 1 MSG_TYPE_SNAPSHOT_REQUEST TLV frame.
2. The freeze coordinator resolves the KVA from the vmlinux ELF, validates 4-byte alignment, and arms a free user watchpoint slot via KVM_SET_GUEST_DEBUG.
3. When the guest writes to the watched address, the corresponding debug exit fires and the host identifies which slot tripped.
4. The coordinator captures via freeze_and_capture and stores the report in the SnapshotBridge under the symbol tag.
5. The report is also mirrored to a sidecar JSON file for post-hoc inspection.

The per-scenario cap of MAX_WATCH_SNAPSHOTS (= 3) is enforced (slot 0 is reserved for the error-class exit_kind trigger; the remaining 3 slots are available for user watches). A 4th Op::watch_snapshot fails the step with a “cap exceeded” message. Symbol-resolution failures bail immediately so a typo surfaces visibly.

Op::watch_snapshot covers the full pipeline: registration, cap enforcement, symbol resolution, hardware arming, and automatic capture on write.

Issuing a watch

use ktstr::prelude::*;

let steps = vec![Step {
    setup: vec![CgroupDef::named("workers").workers(2)].into(),
    ops: vec![
        Op::watch_snapshot("jiffies_64"),
        Op::watch_snapshot("scx_watchdog_timestamp"),
    ],
    hold: HoldSpec::FULL,
}];
execute_steps(ctx, steps)?;

Each Op::watch_snapshot invokes the active SnapshotBridge’s register_watch callback with the symbol string. On success, the callback is responsible for arming a hardware watchpoint that will fire whenever the guest writes to the symbol’s address. Each fire produces one capture, tagged with the symbol path itself.

Wiring the bridge

In #[ktstr_test] scenarios that boot a VM, the bridge is wired automatically. Use post_vm to read captures from VmResult::snapshot_bridge. Do not install a thread-local bridge inside the scenario function — the in-VM Op::watch_snapshot registers via the virtio-console port 1 MSG_TYPE_SNAPSHOT_REQUEST TLV frame, the host coordinator arms the watchpoint and stores captures on the bridge it owns, and the test reads them after the VM exits.

The set_thread_local pattern below is for host-side unit tests that exercise the executor in process without booting a guest.

A watch-capable bridge for host-side unit tests needs both a capture callback and a register_watch callback:

use ktstr::prelude::*;

let cb: CaptureCallback = std::sync::Arc::new(|_name| {
    Some(FailureDumpReport::default())
});
let reg: WatchRegisterCallback = std::sync::Arc::new(|symbol: &str| {
    // Host-side unit tests: record the symbol and return Ok. In a
    // booted VM, the host coordinator's pipeline runs instead —
    // see arm_user_watchpoint in src/vmm/freeze_coord.rs.
    println!("would arm watchpoint on {symbol}");
    Ok(())
});

let bridge = SnapshotBridge::new(cb).with_watch_register(reg);
let _guard = bridge.set_thread_local();

A bridge built only with SnapshotBridge::new(cb) (no with_watch_register) rejects every Op::watch_snapshot with an error pointing the operator at the missing wiring.

Symbol resolution

Production resolution is a verbatim match against the vmlinux ELF symbol table. The freeze coordinator walks Elf::syms and accepts the symbol whose strtab entry equals the requested string byte-for-byte — there is no prefix stripping, BTF lookup, kallsyms walk, or per-CPU offset arithmetic. Use the exact name nm vmlinux would print:

• "jiffies_64" — the kernel’s monotonic tick counter.
• "scx_watchdog_timestamp" — sched_ext’s watchdog timestamp.

Warning: high-frequency symbols soft-lock the guest. Watching a symbol that the kernel writes every jiffie (e.g. jiffies_64 at HZ=1000) fires 1000+ captures per second. Each capture freezes all vCPUs for the full dump pipeline. The guest spends almost all of its wall time paused, which is indistinguishable from a soft lock-up — schedulers stall, watchdogs fire, and the test wedges before any meaningful work runs. Pick symbols the kernel writes at scenario-relevant cadence (a state field, a per-event counter), not on every tick.

The string passed to Op::watch_snapshot must match a vmlinux ELF symtab entry exactly; otherwise the step fails with symbol '...' not found in vmlinux symtab. The register_watch callback on a host-side test bridge can accept any shape it wants — the e2e tests in tests/snapshot_e2e.rs use "kernel.a" / "kernel.b" / etc. for the cap-enforcement test and "exit_kind" for the in-VM test — but the Op::watch_snapshot ops that flow through the production pipeline (in-VM scenarios with no host-side bridge override) must use a verbatim ELF symbol.

Maximum of 3 watches per scenario

pub const MAX_WATCH_SNAPSHOTS: usize = 3;

The bridge enforces a per-scenario cap of 3 successfully-registered watches. The number is tied to the per-vCPU hardware-watchpoint slots KVM exposes via KVM_SET_GUEST_DEBUG: slot 0 is reserved for the existing *scx_root->exit_kind watchpoint that drives the error-class freeze trigger; the remaining three user watchpoint slots are available for on-demand watches.

A 4th Op::watch_snapshot in the same scenario fails the step with “cap exceeded” when the cap is exceeded: