Skip to content

Latest commit

 

History

History
249 lines (195 loc) · 9.73 KB

reproducing_crashes.md

File metadata and controls

249 lines (195 loc) · 9.73 KB

How to reproduce syzkaller crashes

Using a C reproducer

If the bug was reported by syzbot, you first need to build the kernel used by the tool. Syzbot provides the necessary information in its report:

Hello,

syzbot found the following issue on:

HEAD commit:    ae58226b89ac Add linux-next specific files for 20241118
git tree:       linux-next
console+strace: https://syzkaller.appspot.com/x/log.txt?x=14a67378580000
kernel config:  https://syzkaller.appspot.com/x/.config?x=45719eec4c74e6ba
dashboard link: https://syzkaller.appspot.com/bug?extid=2159cbb522b02847c053
compiler:       Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=137beac0580000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=177beac0580000

In this case, you would run:

$ git checkout ae58226b89ac
$ wget -O '.config' 'https://syzkaller.appspot.com/x/.config?x=45719eec4c74e6ba`
$ make CC=clang LD=ld.lld olddefconfig
$ make CC=clang LD=ld.lld -j$(nproc)

You also need a bootable disk image. Syzbot currently uses small Buildroot-based images that you can either build locally or download.

Download and build the reproducer:

$ wget -O 'repro.c' 'https://syzkaller.appspot.com/x/repro.c?x=177beac0580000'
$ gcc repro.c -lpthread -static -o repro

Run the VM:

$ export DISK_IMAGE='buildroot_amd64_2024.09'
$ qemu-system-x86_64 -m 2G -smp 2,sockets=2,cores=1 -drive file=$DISK_IMAGE,format=raw -net nic,model=e1000 -net user,host=10.0.2.10,hostfwd=tcp::10022-:22 -enable-kvm -nographic -snapshot -machine pc-q35-7.1

Run the reproducer:

$ scp -P 10022 -o UserKnownHostsFile=/dev/null  -o StrictHostKeyChecking=no -o IdentitiesOnly=yes ./repro root@127.0.0.1:/root/
$ ssh -p 10022 -o UserKnownHostsFile=/dev/null  -o StrictHostKeyChecking=no -o IdentitiesOnly=yes root@127.0.0.1 'chmod +x ./repro && ./repro'

Using a Syz reproducer

Syzkaller always generates a "Syz" reproducer first (in Syzkaller DSL). Afterwards, syzkaller attempts to convert the Syz reproducer into C code. The process does not always succeed due to the differences between the syz-executor environment and the environment emulated in the C reproducer. Therefore, in some cases, only the Syz version is available.

To run a Syz reproducer locally, the required actions are mostly similar to those in the previous section.

Download and build syzkaller. If you have Docker installed, the instructions are simpler:

$ git clone https://github.com/google/syzkaller.git
$ cd syzkaller
$ ./tools/syz-env make

Build the kernel and boot the VM as described in the section above.

Download the reproducer:

$ wget -O 'repro.syz' 'https://syzkaller.appspot.com/x/repro.syz?x=137beac0580000'

Copy the reproducer and the syzkaller binaries to the test machine:

$ export SYZKALLER_PATH="~/syzkaller"
$ scp -P 10022 -o UserKnownHostsFile=/dev/null  -o StrictHostKeyChecking=no -o IdentitiesOnly=yes $SYZKALLER_PATH/bin/linux_amd64/* ./repro.syz root@127.0.0.1:/root/

Now you can use the syz-execprog tool to actually execute the program.

$ ssh -p 10022 -o UserKnownHostsFile=/dev/null  -o StrictHostKeyChecking=no -o IdentitiesOnly=yes root@127.0.0.1 './syz-execprog -enable=all -repeat=0 -procs=6 ./repro.syz'

Several useful syz-execprog flags:

  -procs int
    	number of parallel processes to execute programs (default 1)
  -repeat int
    	repeat execution that many times (0 for infinite loop) (default 1)
  -sandbox string
    	sandbox for fuzzing (none/setuid/namespace) (default "setuid")
  -threaded
    	use threaded mode in executor (default true)

If you pass -threaded=0, all syscalls will be executed in the same thread. -threaded=1 forces execution of each syscall in a separate thread, so that execution can proceed over blocking syscalls.

Before 2021, syz-execprog also supported the following flag:

  -collide
    	collide syscalls to provoke data races (default true)

-collide=1 forced second round of execution of syscalls when pairs of syscalls are executed concurrently.

Starting from the revision fd8caa54, the behavior is controlled directly in syzlang. If you are running older reproducers, you might still need to set the -collide=1 flag.

If you are replaying a reproducer program that contains a header along the following lines:

# {Threaded:true Repeat:true RepeatTimes:0 Procs:8 Slowdown:1 Sandbox:none Leak:false NetInjection:true NetDevices:true NetReset:true Cgroups:true BinfmtMisc:true CloseFDs:true KCSAN:false DevlinkPCI:false USB:true VhciInjection:true Wifi:true IEEE802154:true Sysctl:true UseTmpDir:true HandleSegv:true Repro:false Trace:false LegacyOptions:{Collide:false Fault:false FaultCall:0 FaultNth:0}}

then you need to adjust syz-execprog flags based on the values in the header. Namely, Threaded/Procs/Sandbox directly relate to -threaded/-procs/-sandbox flags. If Repeat is set to true, add -repeat=0 flag to syz-execprog.

Using ktest

ktest is a collection of tests for Linux and an infrastructure that simplifies running them locally.

Ktest includes a special syzbot-repro.ktest test that automates building the kernel, booting the VM, fetching syzbot bug report details and running the reproducer.

Installation instructions:

$ git clone git://evilpiepirate.org/ktest.git
$ cd ktest
$ export KTEST_PATH=$(pwd)
$ sudo ./root_image init
$ sudo ./root_image create
$ cargo install --path $KTEST_PATH

Instructions to reproduce a syzbot bug:

$ cd ~/linux
$ git checkout <kernel-commit>
$ $KTEST_PATH/build-test-kernel run $KTEST_PATH/tests/syzbot-repro.ktest <bug-id>

<bug-id> can be taken from syzbot bug reports:

dashboard link: https://syzkaller.appspot.com/bug?extid=2159cbb522b02847c053

In this case, bug-id is 2159cbb522b02847c053.

Using downloadable assets

In each report, syzbot shares the exact disk image, kernel image and the vmlinux file that were used to find it.

See the corresponding documentation on how you can use those files to reproduce bugs locally.

From execution logs

The process of creating reproducer programs for syzkaller bugs is automated, but it's not perfect. In some cases, the tool cannot narrow down the kernel crash to a single program.

Obtaining execution logs

  • A local syzkaller instance
    Crash logs created in manager workdir/crashes dir contain programs executed just before a crash. In parallel execution mode (when procs parameter in manager config is set to value larger than 1), program that caused the crash does not necessary immediately precedes it; the guilty program can be somewhere before.

  • Syzbot shares execution logs in its reports:

console output: https://syzkaller.appspot.com/x/log.txt?x=148914c0580000

Crafting reproducers manually

There are two tools that can help you identify and minimize the program that causes a crash: syz-execprog and syz-prog2c. You can build them with make execprog and make prog2c, respectively.

syz-execprog executes a single syzkaller program or a set of programs in various modes (once or loop indefinitely; in threaded/collide mode (see below), with or without coverage collection).

You can start by running all programs in the crash log in a loop to check that at least one of them indeed crashes kernel:

./syz-execprog -executor=./syz-executor -repeat=0 -procs=8 -cover=0 crash-log-file.txt

Note: syz-execprog executes programs locally. So you need to copy syz-execprog and syz-executor into a VM with the test kernel and run it there. See the Using a Syz reproducer section.

To identify the single program that causes the crash, you can cut out individual programs from crash-log-file.txt and run syz-execprog separately.

Once you have a single program that causes the crash, you can try to minimize it by:

  • Removing individual syscalls from the program (you can comment out single lines with # at the beginning of line)
  • By removing unnecessary data (e.g. replacing &(0x7f0000001000)="73656c6600" syscall argument with &(0x7f0000001000)=nil).
  • You can also try to coalesce all mmap calls into a single mmap call that maps whole required area.

Don't forget to test minimization results with the syz-execprog tool.

Now that you have a minimized program, check if the crash still reproduces with ./syz-execprog -threaded=0 -collide=0 flags. If not, then you will need to do some additional work later.

Now, run the syz-prog2c tool on the program. It will give you an executable C source code. If the crash reproduces with -threaded/collide=0 flags, then this C program should cause the crash as well.

If the crash is not reproducible with -threaded/collide=0 flags, then you need this last step. You can think of threaded mode as if each syscall is executed in its own thread. To model such execution mode, move individual syscalls into separate threads. You can see an example here: https://groups.google.com/d/msg/syzkaller/fHZ42YrQM-Y/Z4Xf-BbUDgAJ.

This process is automated to some degree in the syz-repro utility. You need to give it your manager config and a crash report file. And you can refer to the example config file.

./syz-repro -config my.cfg crash-qemu-1-1455745459265726910

It will try to find the offending program and minimize it. But since there are lots of factors that can affect reproducibility, it does not always work.