Keyboard shortcuts

Press or to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

The #[ktstr_test] Macro

#[ktstr_test] registers a function as an integration test that runs inside a VM.

Basic usage

use ktstr::prelude::*;

#[ktstr_test(llcs = 2, cores = 4, threads = 2)]
fn my_test(ctx: &Ctx) -> Result<AssertResult> {
    // ctx provides cgroup manager, topology, duration, etc.
    Ok(AssertResult::pass())
}

When a scheduler with a default topology is specified, the topology can be omitted:

use ktstr::declare_scheduler;

declare_scheduler!(MY_SCHED, {
    name = "my_sched",
    binary = "scx_my_sched",
    //          numa, llcs, cores/llc, threads/core
    topology = (1,    2,    4,         1),
});

#[ktstr_test(scheduler = MY_SCHED)]
fn inherited_topo(ctx: &Ctx) -> Result<AssertResult> {
    // Inherits 1n2l4c1t from MY_SCHED
    Ok(AssertResult::pass())
}

declare_scheduler! emits a pub static MY_SCHED: Scheduler and registers a private linkme static in the KTSTR_SCHEDULERS distributed slice. The scheduler = slot expects &'static Scheduler — pass the bare MY_SCHED ident; the macro takes a reference internally.

The function must have signature fn(&ktstr::scenario::Ctx) -> anyhow::Result<ktstr::assert::AssertResult>.

What the macro generates

  1. Renames the function to __ktstr_inner_{name}.
  2. Registers it in the KTSTR_TESTS distributed slice via linkme.
  3. Emits a #[test] wrapper that calls run_ktstr_test().

The #[test] wrapper boots a VM with the specified topology and runs the function inside it.

Attributes

All attributes are optional with defaults.

Topology

AttributeDefaultDescription
llcsinheritedNumber of LLCs
numa_nodesinheritedNumber of NUMA nodes
coresinheritedCores per LLC
threadsinheritedThreads per core
memory_mb2048VM memory in MB

Each dimension independently inherits from Scheduler.topology when a scheduler is specified and that dimension is not explicitly set. Without a scheduler, unset dimensions use macro defaults (numa_nodes=1, llcs=1, cores=2, threads=1). The default is a single-NUMA topology, so most tests do not need to set numa_nodes. See Default topology.

Scheduler

AttributeDefaultDescription
scheduler = CONST&Scheduler::EEVDFRust const path to a &'static Scheduler. The bare const emitted by declare_scheduler! (e.g. MY_SCHED) is the expected form. The default Scheduler::EEVDF runs tests under the kernel’s default scheduler (EEVDF on Linux 6.6+) so tests without an explicit scheduler = run under the kernel default.
extra_sched_args = [...][]Extra CLI args for the scheduler, appended after Scheduler::sched_args.
watchdog_timeout_s5scx watchdog override (seconds). Applied via scx_sched.watchdog_timeout on 7.1+ kernels (BTF-detected) and via the static scx_watchdog_timeout symbol on pre-7.1 kernels. When neither path is available the override silently no-ops.

Payloads

AttributeDefaultDescription
payload = CONSTNoneRust const path to a binary-kind Payload (PayloadKind::Binary). Populates KtstrTestEntry::payload; the test body can run it via ctx.payload(&CONST). Scheduler-kind payloads are rejected at compile time — use the scheduler = … slot for those.
workloads = [CONST, …][]Array of binary-kind Payload const paths composed alongside the primary payload. Each entry is runnable from the test body via ctx.payload(&CONST); the include-file pipeline packages every referenced binary into the guest automatically.
extra_include_files = ["path", …][]Array of string-literal paths to extra host-side files (datasets, fixture configs, helper scripts) that the framework packages into the guest initramfs alongside the binaries declared by scheduler / payload / workloads. Maps onto KtstrTestEntry::extra_include_files (&'static [&'static str]); union with per-payload Payload::include_files is computed at run time via KtstrTestEntry::all_include_files. Use this slot for test-level dependencies that don’t belong on a specific Payload.

See Payload Definitions for authoring new Payload fixtures; tests/common/fixtures.rs carries reusable examples (SCHBENCH, SCHBENCH_HINTED, SCHBENCH_JSON).

Checking

AttributeDefaultDescription
not_starvedinheritedEnable starvation (zero work units), fairness spread, and scheduling gap checks
isolationinheritedEnable cpuset isolation check (workers must stay on assigned CPUs)
max_gap_msinheritedMax scheduling gap threshold
max_spread_pctinheritedMax fairness spread threshold
max_throughput_cvinheritedMax coefficient of variation for worker throughput
min_work_rateinheritedMinimum work_units per CPU-second per worker
max_imbalance_ratioinheritedMonitor imbalance ratio
max_local_dsq_depthinheritedMonitor DSQ depth
fail_on_stallinheritedFail on stall detection
sustained_samplesinheritedSample window for sustained violations
max_fallback_rateinheritedMax fallback event rate
max_keep_last_rateinheritedMax keep-last event rate
max_p99_wake_latency_nsinheritedMax p99 wake latency in nanoseconds
max_wake_latency_cvinheritedMax wake latency coefficient of variation
min_iteration_rateinheritedMinimum iterations per wall-clock second per worker
max_migration_ratioinheritedMax migration ratio (migrations/iterations) per cgroup
min_page_localityinheritedMin fraction of pages on expected NUMA nodes (0.0-1.0)
max_cross_node_migration_ratioinheritedMax ratio of NUMA-migrated pages to total pages (0.0-1.0)
max_slow_tier_ratioinheritedMax fraction of pages on memory-only (CXL) nodes (0.0-1.0)

not_starved = true enables three distinct checks: starvation (any worker with zero work units), fairness spread (max-min off-CPU% below max_spread_pct), and scheduling gaps (longest gap below max_gap_ms). Each threshold can be overridden independently. See Customize Checking for override examples and Checking for the merge chain.

Topology constraints

AttributeDefaultDescription
min_llcs1Minimum LLCs for gauntlet topology filtering
max_llcs12Maximum LLCs for gauntlet topology filtering
min_cpus1Minimum total CPU count for gauntlet topology filtering
max_cpus192Maximum total CPU count for gauntlet topology filtering
min_numa_nodes1Minimum NUMA nodes for gauntlet topology filtering
max_numa_nodes1Maximum NUMA nodes for gauntlet topology filtering
requires_smtfalseRequire SMT (threads > 1) topologies. On aarch64 the gauntlet ships only non-SMT presets, so any test with requires_smt = true is skipped entirely on that arch.

The gauntlet skips presets that do not satisfy these constraints. Multi-NUMA presets are excluded by default (max_numa_nodes = 1). See Gauntlet for filtering rules and Gauntlet Tests for a worked example.

Execution

AttributeDefaultDescription
auto_reprotrueOn scheduler crash, boot a second VM with probes attached. Set to false for fast iteration.
performance_modefalsePin vCPUs to host cores, hugepages, NUMA mbind, RT scheduling, LLC exclusivity validation
no_perf_modefalseDecouple the virtual topology from host hardware: build the VM with the declared numa_nodes / llcs / cores / threads even on smaller hosts; skip vCPU pinning, hugepages, NUMA mbind, RT scheduling, and KVM exit suppression; relax gauntlet preset filtering to the single “host has enough total CPUs” check. Mutually exclusive with performance_mode = true (validated at runtime by KtstrTestEntry::validate). Equivalent to setting KTSTR_NO_PERF_MODE=1 per-test — either source forces the no-perf path. See Performance Mode.
duration_s12Per-scenario duration in seconds
expect_errfalseTest expects run_ktstr_test to return Err; disables auto-repro
bpf_map_write = CONSTemptyRust const path to a BpfMapWrite; host writes this value to a BPF map after the scheduler loads. The entry field is a slice; the macro wraps the single path in a one-element slice.
host_onlyfalseRun the test function directly on the host instead of inside a VM. Use for tests that need host tools (e.g. cargo, nested VMs) unavailable in the guest initramfs.
num_snapshots = N0Fire N periodic freeze_and_capture(false) boundaries inside the workload’s 10 %–90 % window; each capture is stored on the host SnapshotBridge under periodic_NNN. 0 disables periodic capture entirely. Validated against MAX_STORED_SNAPSHOTS (= 64), host_only = true, and a 100 ms minimum-spacing rule. See Periodic Capture and Temporal Assertions.
cleanup_budget_ms = NNoneSub-watchdog cap on host-side VM teardown wall time. When the budget is exceeded the test’s AssertResult is folded with a failing AssertDetail. None disables the check.
post_vm = PATHNoneHost-side callback invoked after vm.run() returns. Signature: fn(&VmResult) -> anyhow::Result<()>. Use for assertions that need host-side state — e.g. draining VmResult.snapshot_bridge for periodic-capture analysis (see Periodic Capture).
config = EXPRNoneInline scheduler config content (string literal or path to a const &'static str). Written to the guest path declared by the scheduler’s config_file_def; the framework substitutes {file} in the scheduler’s arg template with the guest path. Required when the scheduler declares config_file_def; rejected when it doesn’t. The pairing is enforced at compile time via a const assertion against Payload::config_file_def, and again at runtime by KtstrTestEntry::validate. See Inline scheduler config.

See Performance Mode for details on what performance_mode enables, prerequisites, and validation behavior.

Inline scheduler config

Some schedulers (e.g. scx_layered, scx_lavd) accept a JSON config file via a CLI argument like --config /path/to/config.json. Two pieces wire this into a test:

  1. Scheduler declaration — the Scheduler builder declares the arg template and the guest path via .config_file_def:

    const LAYERED_SCHED: Scheduler = Scheduler::new("layered")
        .binary(SchedulerSpec::Discover("scx_layered"))
        .config_file_def("--config {file}", "/include-files/layered.json");

    {file} in the arg template is replaced with the guest path. The framework mkdir -ps the parent and writes the config content to /include-files/layered.json inside the guest before the scheduler binary starts.

  2. Test attribute — the test supplies the inline JSON via config = …:

    const LAYERED_CONFIG: &str = r#"{ "layers": [...] }"#;
    
    #[ktstr_test(scheduler = LAYERED_SCHED, config = LAYERED_CONFIG)]
    fn layered_test(ctx: &Ctx) -> Result<AssertResult> {
        Ok(AssertResult::pass())
    }

    config = "..." (string literal) and config = SOME_CONST (path to a const &'static str) are both accepted.

The pairing gate is bidirectional:

  • A scheduler with config_file_def set requires config = … on every test (otherwise the scheduler binary would launch without --config).
  • A scheduler without config_file_def rejects config = … on the test (the content would be silently dropped at dispatch).

Both halves are validated at compile time via a const assertion emitted by the macro AND at runtime by KtstrTestEntry::validate, so direct programmatic-entry construction sees the same gate.

For schedulers that take a config file from a host-side path instead of inline content, use Scheduler::config_file(host_path) instead of config_file_def. The framework packs the host file into the initramfs at /include-files/{filename} and prepends --config /include-files/{filename} to scheduler args; no config = … on the test is needed in that flavor.

Example with custom scheduler

Define the scheduler with declare_scheduler! (see Scheduler Definitions), then reference it in #[ktstr_test]:

use ktstr::declare_scheduler;
use ktstr::prelude::*;

declare_scheduler!(MY_SCHED, {
    name = "my_sched",
    binary = "scx_my_sched",
    topology = (1, 2, 4, 1),
    sched_args = ["--enable-llc", "--enable-stealing"],
});

#[ktstr_test(
    scheduler = MY_SCHED,
    not_starved = true,
    max_gap_ms = 5000,
)]
fn my_sched_basic(ctx: &Ctx) -> Result<AssertResult> {
    // Inherits 1n2l4c1t from MY_SCHED
    Ok(AssertResult::pass())
}

declare_scheduler! emits a pub static MY_SCHED: Scheduler and registers it in the KTSTR_SCHEDULERS distributed slice via a private linkme static so cargo ktstr verifier discovers it. The bare MY_SCHED ident is what #[ktstr_test(scheduler = ...)] expects. See Scheduler Definitions for the full macro grammar.

For the manual builder pattern (no distributed-slice registration), see Scheduler Definitions: Manual definition.