Evaluation results

README.md

@@ -168,6 +168,9 @@ For convenience, `vbdb_challenge_peek.sh` enumerates every location given in the
 challenge and analyzes only one configuration using `vbdb_challenge.sh`. It
 takes no arguments.
 
+Evaluation has been done using `vbdb_evaluate.sh`. This generates CSV files and
+preprocessed C files with statistical information.
+
 #### BusyBox
 
 To analyze BusyBox, build PCLocator and `cd` into `scripts/busybox`. Use
@@ -188,9 +191,11 @@ analyze, any following arguments are propagated to PCLocator, e.g.
 
 The final binary can be executed with `busybox-1.18.5/busybox`. We have only
 tried random samples, so compilation might fail due to libraries missing. This
-can be resolved by installing the missing libraries. Also note that using SuperC
-or FeatureCoPP (see example) significantly speeds up the process (but may
-deliver not as accurate results).
+can be resolved by installing the missing libraries.
+
+Evaluation has been done using `busybox_evaluate.sh` and `busybox_sample.sh`.
+Like with VBDB, this generates statistical information. Some evaluation results
+can be found in the `evaluation` directory.
 
 The Kmax files have been generated using
 [kmax-vm](https://github.com/ekuiter/kmax-vm). The DIMACS file from the

evaluation/busybox_correctness_sample.txt

@@ -0,0 +1,100 @@
+busybox-1.18.5/archival/bbunzip.c:171
+busybox-1.18.5/archival/bbunzip.c:384
+busybox-1.18.5/archival/libarchive/data_extract_to_command.c:88
+busybox-1.18.5/archival/libarchive/get_header_cpio.c:157
+busybox-1.18.5/archival/libarchive/get_header_cpio.c:57
+busybox-1.18.5/archival/libarchive/get_header_tar.c:260
+busybox-1.18.5/archival/libarchive/get_header_tar_gz.c:27
+busybox-1.18.5/archival/rpm2cpio.c:27
+busybox-1.18.5/archival/tar.c:642
+busybox-1.18.5/console-tools/resize.c:1
+busybox-1.18.5/console-tools/setconsole.c:24
+busybox-1.18.5/console-tools/setconsole.c:27
+busybox-1.18.5/console-tools/setlogcons.c:21
+busybox-1.18.5/coreutils/chroot.c:29
+busybox-1.18.5/coreutils/cksum.c:62
+busybox-1.18.5/coreutils/comm.c:83
+busybox-1.18.5/coreutils/date.c:281
+busybox-1.18.5/coreutils/fsync.c:8
+busybox-1.18.5/coreutils/libcoreutils/getopt_mk_fifo_nod.c:47
+busybox-1.18.5/coreutils/nice.c:18
+busybox-1.18.5/coreutils/sleep.c:66
+busybox-1.18.5/coreutils/stty.c:357
+busybox-1.18.5/coreutils/stty.c:7
+busybox-1.18.5/coreutils/sum.c:58
+busybox-1.18.5/coreutils/sum.c:8
+busybox-1.18.5/coreutils/sync.c:4
+busybox-1.18.5/coreutils/tail.c:286
+busybox-1.18.5/coreutils/test_ptr_hack.c:9
+busybox-1.18.5/coreutils/tr.c:197
+busybox-1.18.5/coreutils/uniq.c:85
+busybox-1.18.5/coreutils/whoami.c:23
+busybox-1.18.5/e2fsprogs/old_e2fsprogs/ext2fs/swapfs.c:111
+busybox-1.18.5/e2fsprogs/old_e2fsprogs/ext2fs/unix_io.c:628
+busybox-1.18.5/editors/awk.c:2237
+busybox-1.18.5/editors/ed.c:223
+busybox-1.18.5/editors/patch.c:423
+busybox-1.18.5/init/halt.c:64
+busybox-1.18.5/init/mesg.c:20
+busybox-1.18.5/libbb/compare_string_array.c:82
+busybox-1.18.5/libbb/copy_file.c:37
+busybox-1.18.5/libbb/copyfd.c:61
+busybox-1.18.5/libbb/dump.c:288
+busybox-1.18.5/libbb/fgets_str.c:57
+busybox-1.18.5/libbb/find_pid_by_name.c:2
+busybox-1.18.5/libbb/obscure.c:138
+busybox-1.18.5/libbb/obscure.c:20
+busybox-1.18.5/libbb/perror_nomsg.c:21
+busybox-1.18.5/libbb/platform.c:42
+busybox-1.18.5/libbb/printable.c:12
+busybox-1.18.5/libbb/progress.c:141
+busybox-1.18.5/libbb/remove_file.c:51
+busybox-1.18.5/libbb/run_shell.c:75
+busybox-1.18.5/libbb/selinux_common.c:32
+busybox-1.18.5/libbb/str_tolower.c:13
+busybox-1.18.5/libbb/update_passwd.c:250
+busybox-1.18.5/libbb/xregcomp.c:31
+busybox-1.18.5/libpwdgrp/pwd_grp.c:696
+busybox-1.18.5/loginutils/cryptpw.c:75
+busybox-1.18.5/loginutils/login.c:61
+busybox-1.18.5/loginutils/su.c:28
+busybox-1.18.5/mailutils/sendmail.c:36
+busybox-1.18.5/miscutils/beep.c:10
+busybox-1.18.5/miscutils/chrt.c:113
+busybox-1.18.5/miscutils/crontab.c:86
+busybox-1.18.5/miscutils/flashcp.c:138
+busybox-1.18.5/miscutils/inotifyd.c:138
+busybox-1.18.5/miscutils/man.c:95
+busybox-1.18.5/miscutils/rfkill.c:107
+busybox-1.18.5/miscutils/taskset.c:71
+busybox-1.18.5/miscutils/volname.c:29
+busybox-1.18.5/miscutils/wall.c:24
+busybox-1.18.5/modutils/modprobe-small.c:542
+busybox-1.18.5/networking/libiproute/ip_parse_common_args.c:70
+busybox-1.18.5/networking/libiproute/ip_parse_common_args.c:74
+busybox-1.18.5/networking/libiproute/iptunnel.c:441
+busybox-1.18.5/networking/nameif.c:83
+busybox-1.18.5/networking/nbd-client.c:14
+busybox-1.18.5/networking/udhcp/dumpleases.c:1
+busybox-1.18.5/networking/udhcp/dumpleases.c:47
+busybox-1.18.5/networking/wget.c:709
+busybox-1.18.5/procps/sysctl.c:88
+busybox-1.18.5/procps/watch.c:30
+busybox-1.18.5/runit/runsvdir.c:79
+busybox-1.18.5/selinux/selinuxenabled.c:11
+busybox-1.18.5/shell/shell_common.c:92
+busybox-1.18.5/sysklogd/klogd.c:37
+busybox-1.18.5/sysklogd/syslogd_and_logger.c:34
+busybox-1.18.5/sysklogd/syslogd_and_logger.c:36
+busybox-1.18.5/util-linux/fdformat.c:69
+busybox-1.18.5/util-linux/fdisk_sgi.c:357
+busybox-1.18.5/util-linux/fsck_minix.c:549
+busybox-1.18.5/util-linux/fsck_minix.c:865
+busybox-1.18.5/util-linux/hexdump.c:6
+busybox-1.18.5/util-linux/mkfs_ext2.c:596
+busybox-1.18.5/util-linux/mkfs_reiser.c:270
+busybox-1.18.5/util-linux/mkswap.c:135
+busybox-1.18.5/util-linux/more.c:76
+busybox-1.18.5/util-linux/more.c:95
+busybox-1.18.5/util-linux/rdev.c:17
+busybox-1.18.5/util-linux/volume_id/linux_swap.c:2