ktstr

ktstr is the standalone debugging companion to the #[ktstr_test] test harness. It owns kernel cache management, interactive VM shells, host-wide per-thread profiling, and lock introspection — the operations a scheduler author reaches for when investigating a test failure.

To reproduce a test scenario as a self-contained shell script without a VM, use cargo ktstr export. To run the test suite, use cargo ktstr test.

See also cargo ktstr for the cargo-integrated companion that also covers test execution, coverage, BPF verifier stats, and gauntlet statistics.

Build from the workspace:

cargo build --bin ktstr

Subcommands

topo

Show the host CPU topology (CPUs, LLCs, NUMA nodes):

ktstr topo

kernel

The kernel subcommand manages cached kernel images. Subcommands: list, build, clean. See cargo-ktstr kernel for full documentation – the kernel subcommands are identical in both binaries.

shell

Boot an interactive shell in a KVM virtual machine. Launches a VM with busybox and drops into a shell.

ktstr shell
ktstr shell --kernel ../linux
ktstr shell --kernel 6.14.2
ktstr shell --topology 1,2,4,1
ktstr shell -i /path/to/binary
ktstr shell -i my_tool -i another_tool

Files and directories passed via -i are available at /include-files/<name> inside the guest. Directories are walked recursively, preserving structure (e.g. -i ./release includes all files under release/ at /include-files/release/...). Bare names (without path separators) are resolved via PATH lookup. Dynamically-linked ELF binaries get automatic shared library resolution via ELF DT_NEEDED parsing. Non-ELF files are copied as-is.

Stdin is a terminal requirement. The host terminal enters raw mode for bidirectional stdin/stdout forwarding. Terminal state is restored on all exit paths.

cargo ktstr shell runs the same VM boot flow and differs in one respect: it accepts raw image file paths for --kernel (e.g. bzImage, Image). Source-tree directories auto-build and no-kernel invocations auto-download — same as ktstr shell.

ctprof

Capture or compare a host-wide per-thread state snapshot. Useful for diagnosing “the scheduler looks fine but something on the host is still behaving oddly” by producing a baseline/candidate diff of every live thread’s scheduling, memory, and I/O counters — a superset of what any single test’s sidecar captures.

ktstr ctprof capture --output baseline.ctprof.zst
# ... run workload of interest ...
ktstr ctprof capture --output candidate.ctprof.zst
ktstr ctprof compare baseline.ctprof.zst candidate.ctprof.zst

capture walks /proc at capture time and writes every visible thread’s metric values (cumulative counters from schedstat / sched / status CSW / page faults / I/O bytes / taskstats; lifetime peaks from schedstat *_max and hiwater_*; instantaneous gauges sampled at capture time including nr_threads, fair_slice_ns, state; categorical / ordinal scalars including policy, nice, cpu_affinity, identity strings) as zstd-compressed JSON (conventional extension .ctprof.zst). Cumulative counters and lifetime peaks are probe-timing-invariant — sampled twice, the value either monotonically increased or stayed at its high-water mark — so a diff between two snapshots measures exactly the activity over the window. Instantaneous gauges and categorical scalars are point-in-time readings that can legitimately differ between two probes of the same thread. Per-cgroup aggregates (cpu.stat, memory.current) are captured once per distinct path. Capture is read-only; nothing is attached, no kprobes, no tracing.

compare joins two snapshots on the selected grouping axis (pcomm by default) and renders a per-metric baseline/candidate/delta table. The join key survives across captures taken on different hosts or after process restarts, so deltas reflect the behavior of the named workload rather than a specific pid. Metrics with cumulative semantics (CPU time, page faults, wait time) show the candidate-minus-baseline delta; instantaneous metrics (affinity, cgroup path) show the value at candidate capture time. See the ctprof reference for the full metric registry, aggregation rules, derived-metric formulas, and taskstats kconfig gating.

show renders a single snapshot’s per-(group, metric) values without diff math. Same flag vocabulary as compare, minus the baseline/candidate/delta/pct columns:

ktstr ctprof show snapshot.ctprof.zst --group-by cgroup
ktstr ctprof show snapshot.ctprof.zst --sections taskstats-delay

metric-list prints every registered metric (primary + derived) with its description, unit, kconfig gate, and sched_class scope. Use this to discover the vocabulary --sort-by and --metrics accept.

ktstr ctprof metric-list

locks

Enumerate every ktstr flock held on this host. Read-only — does NOT attempt any flock acquire. Useful as a troubleshooting companion for --cpu-cap contention: when a build or test is stalled behind a peer’s reservation, ktstr locks names the peer (PID + cmdline) without disturbing any of its flocks.

Scans four lock-file roots:

• /tmp/ktstr-llc-*.lock — per-LLC reservations held by perf-mode test runs and --cpu-cap-bounded builds.
• /tmp/ktstr-cpu-*.lock — per-CPU reservations from the same flow.
• {cache_root}/.locks/*.lock — cache-entry locks held during kernel build writes, and source-{path_hash}.lock files held for the duration of kernel build --source and cargo ktstr test --kernel <path> against the same source tree.
• {runs_root}/.locks/{kernel}-{project_commit}.lock — per-run-key sidecar-write locks held for the duration of the (pre-clear + write) cycle to serialize concurrent ktstr processes targeting the same run directory.

Each lock is cross-referenced against /proc/locks to name the holder PID and cmdline.

ktstr locks                       # one-shot snapshot
ktstr locks --json                # JSON snapshot
ktstr locks --watch 1s            # redraw every second until SIGINT
ktstr locks --watch 1s --json     # ndjson stream, one object per interval

The same subcommand is available as cargo ktstr locks with identical flag semantics.

completions

Generate shell completions for ktstr.

ktstr completions bash
ktstr completions zsh
ktstr completions fish

The same subcommand is available as cargo ktstr completions with identical flag semantics (--binary accepted on both; defaults to the respective binary name).