Binary Analysis and Rewriting Tool Evaluation

See the wiki page(s) for detailed information.

• Methodology
• Tools
• Subjects

Linux binary corpus

Ubuntu 16.04 binaries

Some of the binaries were not built for Ubuntu 16:

• No icx binaries. It was difficult to run an old icc executable that would run on Ubuntu 16.
• "asterisk": requires libjansson >= 2.11.
• "bitcoin": requires C++17 support.
• "filezilla": requires C++17 support.
• "gnome-calculator": new dependencies required.
• "libreoffice": new gcc/clang required.
• "poppler": gcc/clang compile errors.
• "samba": python version >= 3.6.0 required.
• "sipwitch": gcc/clang compile errors.
• "unrealircd": libargon2 required.

Partially built:

• "mysql": gcc only
• "monero": gcc only and without -O0

Ubuntu 20.04 binaries

Only PIE versions:

• "libzmq": dynamic library
• "lighttpd"
• "memcached"
• "openssh"
• "poppler": dynamic library
• "redis"
• "libreoffice": dynamic library, only gcc without -O1

Partially built:

• "bind": only gcc/icx (clang compile error)
• "gnome-calculator" is gcc and PIE only, -Ofast is not supported
• "mysql": only gcc/clang without -Ofast (compile error)
• "bitcoin": no icx with -O1 and -O2 (link errors)

In some cases different optimization flags produce the same binary (e.g. -O3 and -Ofast for Intel compiler). Such duplicates were not included for: "dnsmasq", "hello", "nginx", "qmail", "openssh", "pks", "sendmail", "zip".

Ubuntu 20.04 32 bit binaries

• "gnome-calculator": Lib-gee rquires a number of dependecies
• "monero": cmake issues
• "poppler": cmake issues
• "libreoffice": jawt library
• "sipwitch"

"ollvm" binaries

Obfuscator-LLVM binaries are a reduced set of Ubuntu 20.04 and clang binaries.

Binaries not built due to building problems:

• "asterisk"
• "bitcoin"
• "filezilla"
• "libreoffice"
• "memcached"
• "monero"
• "poppler"
• "snort3"

For some binaries -mllvm -bcf -mllvm -bcf_prob=100 option hangs the compiler:

• "anope"
• "libzmq"
• "mosh"
• "mysql"
• "sipwitch"
• "snort3"
• "squid"

Building binaries

Different docker images are used for building different sets of binaries:

• Ubuntu 20 (x86_64, aarch64 and arm)
• Obfuscator-LLVM (Ubuntu 20, x86_64)
• Ubuntu 16 (x86_64)
• Intel compiler (Ubuntu 20, x86_64)

There are also pre-generated lists of all make targets to build each set:

• Ubuntu 20 (x86_64)
• Ubuntu 20 (x86)
• Ubuntu 20 (aarch64)
• Fortran (Ubuntu 20, x86_64)
• Obfuscator-LLVM (Ubuntu 20, x86_64)
• Ubuntu 16 (x86_64)
• Intel compiler (Ubuntu 20, x86_64)

Using "subjects/.py" script

Every subject program has associated "subjects/.py" script. You can build and test subject binaries using the associated script. For example for "hello" program you can run the script like this:

PYTHONPATH=src subjects/hello.py --help

Some subject programs might require a specific docker image for building/testing. You can build such docker image using "image" command:

PYTHONPATH=src subjects/proftpd.py image

Build a single binary:

PYTHONPATH=src subjects/hello.py build

Use "--help" to find out how to specify different isa/os/compler and other options.

Smoke-test a single binary:

PYTHONPATH=src subjects/hello.py smoke-test results/x86_64.ubuntu20.hello.gcc.O0.pie.nostrip.elf

Run functional tests for a single binary:

PYTHONPATH=src subjects/poppler.py evaluate results/x86_64.ubuntu20.poppler.gcc.O0.pie.nostrip.elf

Run AFL for a single binary:

PYTHONPATH=src subjects/hello.py afl-fuzz results/x86_64.ubuntu20.hello.gcc.O0.nopie.nostrip.ddisasm.afl

Using "alternate_make.py"

"bin/alternate_make.py" is a high-level driver over make. The script knows how to build every binary from the corpus and which docker image should be used. The script also reports binaries that cannot be built for a particular combination of os/compiler/optimization. The script also checks if the produced binary corresponds to the intended pie/isa options. Use "--dry-run" option to get an idea of what the script does:

bin/alternate_make.py --dry-run

You can specify a subset of binaries to build. To build "hello" subject only with "icx" compiler:

bin/alternate_make.py hello --compiler icx

You can also use different docker volumes for storing resulting binaries and build artifacts. See "--help" output for more details.

Adding a new subject/isa/os to the corpus

On adding a new subject update the target lists ("compiler/list-targets*.txt) using gen-list-targets.py. Also, update should_be_skipped function in "bin/alternate_make.py" if there are known failures for specific isa/compiler.

Collecting results

"bin/results.py" can be used for quick results assessment. The script knows the baseline (every binary from the corpus that has been successfully built) and can report if specified tools/transforms were successful. E.g. to see if all the corpus binaries for x64/arm64 are present in "results" subdirectory and have been successfully smoke-tested:

bin/results.py .elf .elf.errno --isa x86_64 aarch64

It should report "success rate: 100.00%" in the end.

To report results for particular subjects and transforms:

bin/results.py .ddisasm.S --isa x86_64 aarch64 --subject hello nginx

See "--help" output for more details.

Windows binaries

• "npp": Notepad++ -- https://github.com/notepad-plus-plus/notepad-plus-plus
• "vlc": VLC media player -- https://github.com/videolan/vlc
• "libreoffice": Libreoffice (soffice.exe) -- https://github.com/LibreOffice/core
• "openssh": https://github.com/openssh/openssh-portable

ONR programs that were not included:

• Calculator -- https://github.com/microsoft/calculator: C#
• "postfix", "zip", "vim": have no sources for Windows (at the provided links)
• "poppler": is a library used in "evince", cannot be build for windows with Visual Studio

Testing a generic lifter

A script has been provided for testing lifters that have not already been included under tools/. This script (tools/test-generic-lifter.py) requires the following resources to run:

• A directory containing target binaries (defaults to results/)
• A docker image for running the lifter
• A command to run the lifter. This command must accept the input format: lift_command target_in.elf target_out.rewritten. tools/libfilter/run_libfilter.sh has been provided as an example wrapper for Libfilter.

Acknowledgment

This material is based upon work supported by the Office of Naval Research (ONR) under Contract No. N00014-21-C-1032. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the ONR.