SPSLR

Sanemaker Case Study

Sanemaking the Linux Kernel Boot Process

A reproducible Bootpatch-SLR investigation in Linux 7.2-rc3: from an endless RCU stall to the four instructions that first accessed the wrong randomized field.

By York Jasper Niebuhr About 10 minutes

Introduction

This article is a practical, reproducible Sanemaker walkthrough. It assumes familiarity with the validation model, terminology, userspace command chain, and interpretation rules described in the Sanemaker documentation.

The experiment uses the Linux kernel tree published with Bootpatch-SLR RFC v3 and the latest Sanemaker version at the time (marked with the tag rfc-v3 in both repositories). You can follow each step using the same sources and compare your own Sanemaker output with the addresses, disassembly, and mismatches shown here.

The RFC v3 implementation randomizes almost all of task_struct. One of the few exceptions is rcu_read_lock_nesting, which is kept at its original offset with the __spslr_field_fixed annotation specifically because of the issue investigated here.

To reproduce the original failure, remove the __spslr_field_fixed annotation from rcu_read_lock_nesting in include/linux/sched.h, then rebuild the kernel. This allows the field to participate in the randomized task_struct layout.

The resulting boot reaches an RCU warning during the first scheduling transition and then emits an endless stream of RCU stall reports. The warning shows that RCU state is wrong, but not where it first became wrong. Sanemaker reduces the investigation to four mismatching memory accesses and reveals that the corruption happens while Bootpatch-SLR is still patching the kernel image.

Bootpatch-SLR RFCs v1 and v2 were based on Linux 6.12 because that version did not trigger this issue. The relevant kernel implementation changed afterwards, making newer kernels enter RCU from the instruction-patching path.

Reproducing the experiment

Check out the Bootpatch-SLR RFC v3 kernel tree. Then remove the fixed-field annotation from rcu_read_lock_nesting in include/linux/sched.h.

The field currently appears with an annotation equivalent to:

int rcu_read_lock_nesting __spslr_field_fixed;

For this experiment, change it to:

int rcu_read_lock_nesting;

Start from the default x86 configuration, then enable SPSLR and Sanemaker:

make defconfig
scripts/config --enable SPSLR
scripts/config --enable SANEMAKER
make olddefconfig

Then build the kernel.

Only the kernel boot path is needed for this demonstration, so the initramfs can contain an /init that immediately powers the machine off.

Generate Sanemaker metadata from the unstripped vmlinux image, not from the compressed bzImage:

python3 binscan.py -o binscan.json Bootpatch-SLR/vmlinux

The bootable image is still arch/x86/boot/bzImage, but vmlinux is required for binscan because it retains the symbols used by Sanemaker.

Observed failure without Sanemaker

Without Sanemaker active, Bootpatch-SLR reports success and later reaches:

Successfully applied SPSLR
...
Voluntary context switch within RCU read-side critical section!
WARNING: kernel/rcu/tree_plugin.h:332 at rcu_note_context_switch+0x358/0x530
...
rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { P0 }

The machine then repeats RCU stall diagnostics indefinitely. In this build, rcu_note_context_switch starts at 0xffffffff813c7690, so the warning at offset 0x358 is reached at 0xffffffff813c79e8.

ffffffff813c79dc: e9 25 fd ff ff         jmp  ffffffff813c7706 <rcu_note_context_switch+0x76>
ffffffff813c79e1: 48 8d 3d 70 de 50 02   lea  rdi,[rip+0x250de70]
ffffffff813c79e8: e8 a3 f2 06 01         call ffffffff82436c90 <__SCT__WARN_trap>

The only branch to the warning tests task_struct.rcu_read_lock_nesting:

ffffffff813c76b6: 48 c7 c0 7c 04 00 00  mov  rax,0x47c
ffffffff813c76bd: 8b 34 03              mov  esi,DWORD PTR [rbx+rax]
ffffffff813c76c0: 85 f6                 test esi,esi
ffffffff813c76c2: 0f 8f 19 03 00 00     jg   ffffffff813c79e1 <rcu_note_context_switch+0x351>

The immediate pin sits at a 3 byte offset into the first instruction, so it begins at 0xffffffff813c76b9. Binscan identifies it as an access to field 38 of target 0:

{
  "addr": "0xffffffff813c76b9",
  "size": 4,
  "target_id": 0,
  "field_idx": 38,
  "unit": 316
}

According to the binscan file's "targets" array, target 0 is task_struct, and field 38 is rcu_read_lock_nesting, originally at offset 1148 (0x47c). This confirms that the warning-site instruction is correctly instrumented. The field must have been corrupted earlier.

Recording the kernel runs

Baseline run

The first Sanemaker run records a semantic reference execution with Selfpatch-SLR disabled. This uses the same kernel image that produced the initial observation, but now boots under QEMU with the Sanemaker plugin enabled and spslr=off.

qemu-system-x86_64 \
  -accel tcg \
  -plugin "sanemaker.so,focus=binscan.json,out=logs/baseline,spslr=off,entry=0xffffffff83da4ad0" \
  -m 1024 \
  -kernel Bootpatch-SLR/arch/x86/boot/bzImage \
  -initrd initramfs.cpio.gz \
  -append "nokaslr earlycon=uart,io,0x3f8,115200 console=ttyS0,115200 rdinit=/init panic=-1 oops=panic ignore_loglevel loglevel=8" \
  -nographic \
  -serial mon:stdio \
  -no-reboot

The command is identical to the later SPSLR run except that spslr=off disables runtime randomization and kill-at is omitted because the kernel shuts down normally.

The entry=0xffffffff83da4ad0 argument tells Sanemaker to begin tracing only once execution reaches start_kernel(). The value is simply the runtime address of start_kernel in this kernel build. Delaying instrumentation avoids tracing the decompressor and early boot loader code, substantially reducing the runtime of the baseline recording while still capturing the complete kernel initialization sequence. Note that the entire boot may still take up to a minute with this version of Sanemaker.

After QEMU exits, merge the generated logs into baseline.json using logmerge.py.

SPSLR run

The SPSLR boot never terminates normally, so this run stops at the known warning call:

qemu-system-x86_64 \
  -accel tcg \
  -plugin "sanemaker.so,focus=binscan.json,out=logs/spslr,spslr=on,entry=0xffffffff83da4ad0,kill-at=0xffffffff813c79e8" \
  -m 1024 \
  -kernel Bootpatch-SLR/arch/x86/boot/bzImage \
  -initrd initramfs.cpio.gz \
  -append "nokaslr earlycon=uart,io,0x3f8,115200 console=ttyS0,115200 rdinit=/init panic=-1 oops=panic ignore_loglevel loglevel=8" \
  -nographic \
  -serial mon:stdio \
  -no-reboot

QEMU and plugin arguments

  • -accel tcg forces QEMU's software translator. Sanemaker cannot observe execution performed by KVM.
  • focus=binscan.json supplies metadata for the kernel image.
  • out=logs/... selects the log output directory.
  • spslr=on enables runtime randomization. The baseline uses spslr=off.
  • entry=0xffffffff83da4ad0 delays instrumentation until start_kernel.
  • kill-at=0xffffffff813c79e8 terminates emulation at the warning call and flushes the logs.
  • nokaslr keeps runtime addresses aligned with binscan metadata, disassembly, entry, and kill-at.

After QEMU exits, merge the generated logs into spslr.json using logmerge.py. Then compare baseline.json and spslr.json with logdiff.py.

Reading the diff

Missing instructions can result from timing or control-flow differences. Mismatches are the primary correctness signal because they mean that the same instruction address accessed different logical fields.

An early-kill artifact

Post-patch mismatch at pc <main>:0xffffffff81363ffc
  common: [<no-fields>]
  baseline.json additional: [task_struct.se[24-40]]
  spslr.json additional: [task_struct.se[24-32]]

<main> names the image, not a C function. This instruction is in __enqueue_entity. The baseline accessed bytes 24-40 of task_struct.se, while the SPSLR run accessed only bytes 24-32.

struct sched_entity {
    struct load_weight load;
    struct rb_node run_node;
    /* ... */
};

struct rb_node {
    unsigned long __rb_parent_color;
    struct rb_node *rb_right;
    struct rb_node *rb_left;
};

The randomized execution did not touch a wrong field. It was terminated before the instruction that would access the final eight bytes. Applying the same early kill point to the baseline removes the difference, so this is an early-termination artifact rather than a Bootpatch-SLR bug.

The real mismatch

Pre-patch mismatch at pc <main>:0xffffffff813c0903
  common: [task_struct.rcu_read_lock_nesting[0-4]]
  spslr.json additional: [task_struct.ptrace[0-4]]
Pre-patch mismatch at pc <main>:0xffffffff813c0910
  common: [task_struct.rcu_read_lock_nesting[0-4]]
  spslr.json additional: [task_struct.ptrace[0-4]]
Pre-patch mismatch at pc <main>:0xffffffff813c2ee3
  common: [task_struct.rcu_read_lock_nesting[0-4]]
  spslr.json additional: [task_struct.ptrace[0-4]]
Pre-patch mismatch at pc <main>:0xffffffff813c2ef0
  common: [task_struct.rcu_read_lock_nesting[0-4]]
  spslr.json additional: [task_struct.ptrace[0-4]]

These four PCs are the read and write accesses in __rcu_read_lock and __rcu_read_unlock. The __rcu_read_lock disassembly is:

ffffffff813c08f0 <__rcu_read_lock>:
ffffffff813c08f0: f3 0f 1e fa             endbr64
ffffffff813c08f4: 65 48 8b 15 1c 57 ca 02 mov rdx,QWORD PTR gs:[rip+0x2ca571c]
ffffffff813c08fc: 48 c7 c0 7c 04 00 00    mov rax,0x47c
ffffffff813c0903: 8b 04 02                mov eax,DWORD PTR [rdx+rax]
ffffffff813c0906: 48 c7 c1 7c 04 00 00    mov rcx,0x47c
ffffffff813c090d: 83 c0 01                add eax,0x1
ffffffff813c0910: 89 04 0a                mov DWORD PTR [rdx+rcx],eax
ffffffff813c0913: e9 18 62 07 01          jmp ffffffff82436b30 <__pi___x86_return_thunk>

The instruction pins are present and correctly refer to rcu_read_lock_nesting, yet this function somewhy touches the ptrace member. The decisive clue is the label Pre-patch mismatch. It means the accesses occurred before selfpatch signaled that the entire image had finished patching. By then, SPSLR had already published the randomized layout and reordered init_task, while some instruction pins still contained their original offsets.

For Sanemaker to report task_struct.ptrace[0-4], all of the following had already happened:

  1. The randomized task_struct layout had been generated and published.
  2. ptrace[0-4] had moved into the original byte position of rcu_read_lock_nesting.
  3. init_task had already been reordered.
  4. An instruction still using the original rcu_read_lock_nesting offset executed.

The mismatch therefore occurred during instruction patching itself. The suspicious environment call in that phase was text_poke_early().

Why text_poke_early() touches task_struct

In Linux 7.2-rc3, text_poke_early() checks whether its destination belongs to module text. That reaches:

bool is_module_text_address(unsigned long addr)
{
    guard(rcu)();
    return __module_text_address(addr) != NULL;
}

The RCU guard eventually enters and exits preemptible RCU through these functions:

static void rcu_preempt_read_enter(void)
{
    WRITE_ONCE(current->rcu_read_lock_nesting,
               READ_ONCE(current->rcu_read_lock_nesting) + 1);
}

static int rcu_preempt_read_exit(void)
{
    int ret = READ_ONCE(current->rcu_read_lock_nesting) - 1;

    WRITE_ONCE(current->rcu_read_lock_nesting, ret);
    return ret;
}

That is enough to explain the Sanemaker report: every instruction-pin patch performed through text_poke_early() accesses the original location of current->rcu_read_lock_nesting until the accesses in the rcu logic itself have been patched.

Linux 6.12 used a different implementation:

bool is_module_text_address(unsigned long addr)
{
    bool ret;

    preempt_disable();
    ret = __module_text_address(addr) != NULL;
    preempt_enable();

    return ret;
}

That version does not enter an RCU read-side critical section and therefore does not access rcu_read_lock_nesting from this path. This is why the Bootpatch-SLR v1 and v2 RFCs could use Linux 6.12 without triggering the issue.

The patching hazard

One concrete failing interleaving is enough to show the design problem:

  1. init_task has already been reordered.
  2. The load instruction pin in rcu_preempt_read_enter() still contains the original offset.
  3. The store instruction pin has already been patched to the randomized offset.
  4. Another instruction pin is patched through text_poke_early().
  5. text_poke_early() enters is_module_text_address(), whose RCU guard executes the half-patched lock path.
current->rcu_read_lock_nesting =
    current->ptrace + 1;

The wrong field depends on the randomized layout and the order of instruction-pin updates. In this run, the original RCU offset referred to task_struct.ptrace. The general hazard is that randomized data becomes live before every instruction that can access it has been converted.

Current mitigation: Keep rcu_read_lock_nesting fixed with the SPSLR fixed-field attribute. A future BPSLR version needs an instruction-patching mechanism that does not execute code paths depending on structures already being randomized, or a staging strategy that guarantees such paths cannot observe a partially patched state.

Conclusion

The original warning identified invalid RCU state, and disassembly showed that the eventual warning-site pin was correct. Neither observation explained when the state was corrupted.

Sanemaker identified four concrete accesses whose field semantics diverged and showed that they occurred before the complete-image patch boundary. That narrowed the failure to code executed during patching and led directly to the RCU guard inside text_poke_early().

The resulting bug was not a missing final-image instruction pin. It was a bootstrapping failure caused by executing a correctly instrumented path while that path was only partially converted.

The comparison with Linux 6.12 also explains why the earlier RFC kernels worked: their module-text lookup disabled preemption instead of entering RCU, so text_poke_early() did not execute the randomized task field during patching.

This case study is one example of the validation model described in the Sanemaker documentation. The same baseline-versus-randomized comparison can validate future Bootpatch-SLR patching strategies and reject unsafe images before deployment.