admin/lifting-bits-mcsema
1
0
Fork 0
mirror of https://github.com/lifting-bits/mcsema synced 2026-06-08 15:31:09 +00:00
Code Issues Packages Projects Releases Wiki Activity

Issue 281 update documentation (#327)

Browse Source 
* Docs: add content, fix typos, fix formatting

* Rename doc file: Mcsema -> McSema

* Add a standard Contributing Guidelines doc

* Add remarks on verifying McSema, and unit tests
This commit is contained in:
Mike Myers
2017-11-15 19:23:42 -08:00
committed by Peter Goodman
parent 11149d25e1
commit f5cce245f7
13 changed files with 850 additions and 677 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CONTRIBUTING.md
+58
View File
@@ -0,0 +1,58 @@
# Contributing to McSema
Creating an executable lifter like McSema is no easy task. Even after the initial design is complete, there is still a huge linear effort to implement instruction semantics for every instruction. "Semantics" are a C++ description of what the microcode is doing for a given native-code instruction: as each instruction is fed to the lifter, it translates it to LLVM bitcode using the appropriate semantic function.
The Intel x86-64 architecture alone has around 1,000 instruction mnemonics, most of which have multiple operand types and operand widths. We try to use as much [metaprogramming](https://en.wikipedia.org/wiki/Template_(C%2B%2B)) as possible to reduce the amount of work required to support new instructions, but there is still a lot of work to do. Contributions are greatly appreciated!
## Get in Touch
We do our best to respond to McSema users and potential contributors in our Slack instance, [Empire Hacking](https://empirehacking.slack.com/). Stop into `#binary-lifting` and say hello.
## Taking on a Task
We have many issues already defined. Currently, the majority of them have to do with instruction support. Adding support for a new instruction is a great place to begin if you are a new contributor, to get familiar with the McSema codebase.
Before you start, it would be best if you communicate that you are working on an issue, and if there is not already an issue for what you want to work on, to file an issue first. We use GitHub Issues to track the ongoing progress on the project, deconflict what everyone is doing, and record our accomplishments.
**Note:** If you find a **Closed** issue that seems like it is the same thing that you're experiencing, open a new issue and include a link to the original issue in the body of your new one.
To get started, use git to clone the `remill` repo, and start a new branch that you name according to the issue you are working on. If you are working on "Issue #1234: Missing FPU flag on ARM" then name your branch `issue_1234_missing_fpu_flag_on_arm`.
## Adding a New Instruction
To add support for lifting a new instruction, you will be extending the `remill` library. That is where the instruction semantics are implemented. We have created a whole guide to adding an instruction, which [you can find here](https://github.com/trailofbits/remill/docs/ADD_AN_INSTRUCTION.md).
## Creating Your GitHub Pull Request
If you submit a pull request we will do our best to review it and suggest changes in a timely manner. It helps if you constrain your pull request to just one issue fix or one enhancement. Pull requests that change tens of files and make thousands of lines of diff are much harder to approve and merge.
We have GitHub set up to run Travis continuous integration tests. Watch your pull request to see if all of the tests are passing. We will request that all tests pass before we review or merge your changes.
### Coding Style
Before you create your pull request, we ask that you conform your code to our preferred style, by running `clang-format` on your staged source files before you `git commit`.
### Git Commit Messages
* Use the present tense ("Add feature" not "Added feature")
* Use the imperative mood ("Move cursor to..." not "Moves cursor to...")
* Limit the first line to 72 characters or less
* Reference issues and pull requests liberally after the first line
### Documentation Styleguide
* Use [Markdown](https://daringfireball.net/projects/markdown).
* Consider using spellcheck and a Markdown linter, because you'd be surprised what you miss.
* In documentation, the preferred spelling is *McSema*, not *mcsema* or *Mcsema*.
## Advice and Guidance for New Contributors
Please take a look at our McSema documentation! We try to keep it updated with
[debugging tips](https://github.com/trailofbits/mcsema/docs/DebuggingTips.md), [common errors you might encounter](https://github.com/trailofbits/mcsema/docs/CommonErrrors.md), and [how to get acquainted with the codebase](https://github.com/trailofbits/mcsema/docs/NavigatingTheCode.md).
## Useful External Links
* [Intel architecture software developer manuals](http://www.intel.com/sdm)
* [Intel XED](https://software.intel.com/sites/landingpage/xed/ref-manual/html/index.html)
* [ARM AARCH64 specifications](https://developer.arm.com/products/architecture/a-profile/docs)
* [ARM AARCH64 specs in HTML (unofficial)](https://meriac.github.io/A64_v83A_ISA/)
README.md
+63 -52
View File
@@ -1,31 +1,37 @@
# McSema [![Slack Chat](http://empireslacking.herokuapp.com/badge.svg)](https://empireslacking.herokuapp.com/)
McSema lifts x86 and amd64 binaries to LLVM bitcode modules. McSema support both Linux and Windows binaries, and most x86 and amd64 instructions, including integer, X87, MMX, SSE and AVX operations.
McSema is an executable lifter. It translates ("lifts") executable binaries from native machine code to LLVM bitcode. LLVM bitcode is an [intermediate representation](https://en.wikipedia.org/wiki/Intermediate_representation) form of a program that was originally created for the [retargetable LLVM compiler](https://llvm.org), but which is also very useful for performing program analysis methods that would not be possible to perform on an executable binary directly.
McSema is separated into two conceptual parts: control flow recovery and instruction translation. Control flow recovery is performed using the `mcsema-disass` tool, which uses IDA Pro to disassemble a binary file and produces a control flow graph. Instruction translation is performed using the `mcsema-lift` tool, which converts the control flow graph into LLVM bitcode.
McSema enables analysts to find and retroactively harden binary programs against security bugs, independently validate vendor source code, and generate application tests with high code coverage. McSema isn’t just for static analysis. The lifted LLVM bitcode can also be [fuzzed with libFuzzer](https://github.com/trailofbits/mcsema/blob/master/docs/UsingLibFuzzer.md), an LLVM-based instrumented fuzzer that would otherwise require the target source code. The lifted bitcode can even be [compiled](https://github.com/trailofbits/mcsema/blob/master/docs/UsingLibFuzzer.md) back into a [runnable program](https://github.com/trailofbits/mcsema/blob/master/docs/McsemaWalkthrough.md)! This is a procedure known as static binary rewriting, binary translation, or binary recompilation.
McSema supports lifting both Linux (ELF) and Windows (PE) executables, and understands most x86 and amd64 instructions, including integer, X87, MMX, SSE and AVX operations. AARCH64 (ARMv8) instruction support is in active development. For a full list of which instructions are supported and which are not, you can use the `mcsema-lift --list-supported` command.
Using McSema is a two-step process: control flow recovery, and instruction translation. Control flow recovery is performed using the `mcsema-disass` tool, which relies on IDA Pro to disassemble a binary file and produce a control flow graph. Instruction translation is then performed using the `mcsema-lift` tool, which converts the control flow graph into LLVM bitcode. Under the hood, the instruction translation capability of `mcsema-lift` is implemented in the [`remill` library](https://github.com/trailofbits/remill). The development of `remill` was a result of refactoring and improvements to McSema, and was first introduced with McSema version 2.0.0. Read more about `remill` [below](#FAQ).
McSema and `remill` were developed and are maintained by Trail of Bits, funded by and used in research for DARPA and the US Department of Defense.
## Build status
| | master |
| ----- | ------ |
| | master |
| ----- | ---------------------------------------- |
| Linux | [![Build Status](https://travis-ci.org/trailofbits/mcsema.svg?branch=master)](https://travis-ci.org/trailofbits/mcsema) |
## Features
* Translates 32- and 64-bit Linux ELF and Windows PE binaries to bitcode, including executables and shared libraries for each platform.
* Lifts 32- and 64-bit Linux ELF and Windows PE binaries to bitcode, including executables and shared libraries for each platform.
* Supports a large subset of x86 and x86-64 instructions, including most integer, X87, MMX, SSE, and AVX operations.
* Runs on both Windows and Linux, and can translate Linux binaries on Windows and Windows binaries on Linux.
* McSema runs on Windows and Linux and has been tested on Windows 7, 10, Ubuntu (14.04, 16.04), and openSUSE.
* McSema can cross-lift: it can translate Linux binaries on Windows, or Windows binaries on Linux.
* Output bitcode is compatible with the LLVM toolchain (versions 3.5 and up).
* Translated bitcode can be analyzed or [recompiled as a new, working executable](docs/McsemaWalkthrough.md) with functionality identical to the original.
* McSema runs on Windows and Linux and has been tested on Windows 7, 10, Ubuntu (14.04, 16.04), and openSUSE.
## Using Mcsema
## Use-cases
Why would anyone translate binaries *back* to bitcode?
* **Binary Patching And Modification**. Lifting to LLVM IR lets you cleanly modify the target program. You can run obfuscation or hardening passes, add features, remove features, rewrite features, or even fix that pesky typo, grammatical error, or insane logic. When done, your new creation can be recompiled to a new binary sporting all those changes. In the [Cyber Grand Challenge](https://blog.trailofbits.com/2015/07/15/how-we-fared-in-the-cyber-grand-challenge/), we were able to use mcsema to translate challenge binaries to bitcode, insert memory safety checks, and then re-emit working binaries.
* **Binary Patching And Modification**. Lifting to LLVM IR lets you cleanly modify the target program. You can run obfuscation or hardening passes, add features, remove features, rewrite features, or even fix that pesky typo, grammatical error, or insane logic. When done, your new creation can be recompiled to a new binary sporting all those changes. In the [Cyber Grand Challenge](https://blog.trailofbits.com/2015/07/15/how-we-fared-in-the-cyber-grand-challenge/), we were able to use McSema to translate challenge binaries to bitcode, insert memory safety checks, and then re-emit working binaries.
* **Symbolic Execution with KLEE**. [KLEE](https://klee.github.io/) operates on LLVM bitcode, usually generated by providing source to the LLVM toolchain. Mcsema can lift a binary to LLVM bitcode, [permitting KLEE to operate on previously unavailable targets](https://blog.trailofbits.com/2014/12/04/close-encounters-with-symbolic-execution-part-2/).
* **Symbolic Execution with KLEE**. [KLEE](https://klee.github.io/) operates on LLVM bitcode, usually generated by providing source to the LLVM toolchain. McSema can lift a binary to LLVM bitcode, [permitting KLEE to operate on previously unavailable targets](https://blog.trailofbits.com/2014/12/04/close-encounters-with-symbolic-execution-part-2/).
* **Re-use existing LLVM-based tools**. KLEE is not the only tool that becomes available for use on bitcode. It is possible to run LLVM optimization passes and other LLVM-based tools like [libFuzzer](http://llvm.org/docs/LibFuzzer.html) on [lifted bitcode](docs/UsingLibFuzzer.md).
@@ -35,24 +41,24 @@ Why would anyone translate binaries *back* to bitcode?
## Dependencies
| Name | Version |
| ---- | ------- |
| [Git](https://git-scm.com/) | Latest |
| [CMake](https://cmake.org/) | 3.2+ |
| [Google Protobuf](https://github.com/google/protobuf) | 2.6.1 |
| [Google Flags](https://github.com/google/glog) | Latest |
| [Google Log](https://github.com/google/glog) | Latest |
| [Google Test](https://github.com/google/googletest) | Latest |
| [Intel XED](https://github.com/intelxed/xed) | Latest |
| [LLVM](http://llvm.org/) | 3.6+ |
| [Clang](http://clang.llvm.org/) | 3.6+ (3.9 if using Visual Studio 2015) |
| [Python](https://www.python.org/) | 2.7 |
| [Python Package Index](https://pypi.python.org/pypi) | Latest |
| [python-protobuf](https://pypi.python.org/pypi/protobuf) | 3.2.0 |
| [IDA Pro](https://www.hex-rays.com/products/ida) | 6.7+ |
| [Visual Studio](https://www.visualstudio.com/downloads/) | 2013+ (Windows Only) |
| Name | Version |
| ---------------------------------------- | -------------------------------------- |
| [Git](https://git-scm.com/) | Latest |
| [CMake](https://cmake.org/) | 3.2+ |
| [Google Protobuf](https://github.com/google/protobuf) | 2.6.1 |
| [Google Flags](https://github.com/google/glog) | Latest |
| [Google Log](https://github.com/google/glog) | Latest |
| [Google Test](https://github.com/google/googletest) | Latest |
| [Intel XED](https://github.com/intelxed/xed) | Latest |
| [LLVM](http://llvm.org/) | 3.6+ |
| [Clang](http://clang.llvm.org/) | 3.6+ (3.9 if using Visual Studio 2015) |
| [Python](https://www.python.org/) | 2.7 |
| [Python Package Index](https://pypi.python.org/pypi) | Latest |
| [python-protobuf](https://pypi.python.org/pypi/protobuf) | 3.2.0 |
| [IDA Pro](https://www.hex-rays.com/products/ida) | 6.7+ |
| [Visual Studio](https://www.visualstudio.com/downloads/) | 2013+ (Windows Only) |
## Getting and building the code
## Getting and building the McSema code
### On Linux
@@ -78,10 +84,10 @@ sudo pip install 'protobuf==3.2.0'
Note: If you are using IDA on 64 bit Ubuntu and your IDA install does not use the system Python, you can add the `protobuf` library manually to IDA's zip of modules.
```
```shell
# Python module dir is generally in /usr/lib or /usr/local/lib
touch /path/to/python2.7/dist-packages/google/__init__.py
cd /path/to/lib/python2.7/dist-packages/
cd /path/to/lib/python2.7/dist-packages/
sudo zip -rv /path/to/ida-6.X/python/lib/python27.zip google/
sudo chown your_user:your_user /home/your_user/ida-6.7/python/lib/python27.zip
```
@@ -90,7 +96,7 @@ sudo chown your_user:your_user /home/your_user/ida-6.7/python/lib/python27.zip
The next step is to clone the [Remill](https://github.com/trailofbits/remill) repository. We then clone the McSema repository into the `tools` subdirectory of Remill. This is kind of like how Clang and LLVM are distributed separately, and the Clang source code needs to be put into LLVM's tools directory.
**Notice that when building McSema you should always use a specific Remill commit hash (the one we test). This hash can be found in the .remill_commit_id file**.
**Notice that when building McSema, you should always use a specific Remill commit hash (the one we test). This hash can be found in the .remill_commit_id file**.
```shell
git clone --depth 1 https://github.com/trailofbits/mcsema.git
@@ -103,23 +109,23 @@ git checkout -b temp $REMILL_VERSION
mv ../mcsema tools
```
#### Step 3: Build and install the code
#### Step 3: Build McSema
McSema is a kind of sub-project of Remill, kind of like how Clang is a sub-project of LLVM. To that end, we invoke Remill's build script, and it will build both Remill and McSema. It will also download all remaining dependencies needed by Remill.
McSema is a kind of sub-project of Remill, similar to how Clang is a sub-project of LLVM. To that end, we invoke Remill's build script to build both Remill and McSema. It will also download all remaining dependencies needed by Remill.
The following script will build Remill and McSema into the `remill-build` directory, which will be placed in the current working directory.
```
```shell
./remill/scripts/build.sh
```
This script accepts several additional command line options:
- `--prefix PATH`: Install files to `PATH`. By default, `PATH` is `/usr/local`.
- `--llvm-version MAJOR.MINOR`: Download pre-built dependencies for LLVM version MAJOR.MINOR. The default is to use LLVM 4.0.
- `--build-dir PATH`: Produce all intermediate build files in `PATH`. By default, `PATH` is `$CWD/remill-build`.
- `--use-system-compiler`: Compile Remill+McSema using the system compiler toolchain (typically the GCC).
* `--prefix PATH`: Install files to `PATH`. By default, `PATH` is `/usr/local`.
* `--llvm-version MAJOR.MINOR`: Download pre-built dependencies for LLVM version MAJOR.MINOR. The default is to use LLVM 4.0.
* `--build-dir PATH`: Produce all intermediate build files in `PATH`. By default, `PATH` is `$CWD/remill-build`.
* `--use-system-compiler`: Compile Remill+McSema using the system compiler toolchain (typically the GCC).
#### Step 4: Install McSema
The next step is to build the code.
@@ -131,15 +137,20 @@ sudo make install
This will install McSema _globally_. Once installed, you may use `mcsema-disass` for disassembling binaries, and `mcsema-lift-4.0` for lifting the disassembled binaries. If you specified `--llvm-version 3.6` to the `build.sh` script, then you would use `mcsema-lift-3.6`.
#### Step 5: Verifying Your McSema Installation
In order to verify that McSema works correctly as built, head on over to [the documentation on integration tests](docs/AddingNewTests.md). Check that you can run the tests and that they pass.
## Additional Documentation
- [Common Errors](docs/CommonErrors.md) and [Debugging Tips](docs/DebuggingTips.md)
- [How to implement the semantics of an instruction](https://github.com/trailofbits/remill/blob/master/docs/ADD_AN_INSTRUCTION.md)
- [How to use mcsema: A walkthrough](docs/McsemaWalkthrough.md)
- [Life of an instruction](docs/LifeOfAnInstruction.md)
- [Limitations](docs/Limitations.md)
- [Navigating the source code](docs/NavigatingTheCode.md)
- [Using Mcsema with libFuzzer](docs/UsingLibFuzzer.md)
* [McSema command line reference](docs/CommandLineReference.md)
* [Common Errors](docs/CommonErrors.md) and [Debugging Tips](docs/DebuggingTips.md)
* [How to add support for a new instruction](https://github.com/trailofbits/remill/blob/master/docs/ADD_AN_INSTRUCTION.md)
* [How to use McSema: A walkthrough](docs/McSemaWalkthrough.md)
* [Life of an instruction](docs/LifeOfAnInstruction.md)
* [Limitations](docs/Limitations.md)
* [Navigating the source code](docs/NavigatingTheCode.md)
* [Using McSema with libFuzzer](docs/UsingLibFuzzer.md)
## Getting help
@@ -147,18 +158,18 @@ If you are experiencing problems with McSema or just want to learn more and cont
## FAQ
### How do you pronounce McSema and where did the name come from?
### How do you pronounce McSema and where did the name come from
This is a hotly contested issue. We must explore the etymology of the name to find an answer. The "Mc" in McSema was originally a contraction of the words "Machine Code." At that time, McSema used LLVM's instruction decoder to take machine code bytes, and turn them into `llvm::MCInst` data structures. It is possible that "MC" in that case is pronounced em-see. Alas, even those who understand the origin of the name pronounce it as if it were related to America's favorite fast food joint.
This is a hotly contested issue. We must explore the etymology of the name to find an answer. The "Mc" in McSema was originally a contraction of the words "Machine Code," and the "sema" is short for "semantics." At that time, McSema used LLVM's instruction decoder to take machine code bytes, and turn them into `llvm::MCInst` data structures. It is possible that "MC" in that case is pronounced em-see. Alas, even those who understand the origin of the name pronounce it as if it were related to America's favorite fast food joint.
### Why do I need IDA Pro to use McSema?
### Why do I need IDA Pro to use McSema
McSema's goal is binary to bitcode translation. Accurate disassembly and control flow recovery is a separate and difficult problem. IDA has already invested countless man-hours into getting disassembly right, and it only makes sense that we re-use existing work. We understand that not everyone can afford an IDA license. With the original release of McSema, we shipped our own tool recursive descent disassembler. It was never as good as IDA and it never would be. Maintaining the broken tool took away valuable development time from more important McSema work. We hope to eventually transition to more accessible control flow recovery frontends, such as Binary Ninja (we have a branch with [initial Binary Ninja support](https://github.com/trailofbits/mcsema/tree/getcfg_binja)). We very warmly welcome pull requests that implement new control flow recovery frontends.
McSema's goal is binary to bitcode translation. Accurate disassembly and control flow recovery is a separate and difficult problem. IDA has already invested countless hours of engineering into getting disassembly right, and it only makes sense that we re-use existing work. We understand that not everyone can afford an IDA license. With the original release of McSema, we shipped our own recursive-descent disassembler. It was never as good as IDA, and it never would be. Maintaining the broken tool took away valuable development time from more important McSema work. We hope to eventually transition to more accessible control flow recovery front-ends, such as Binary Ninja (we have a branch with [initial Binary Ninja support](https://github.com/trailofbits/mcsema/tree/getcfg_binja)). We very warmly welcome pull requests that implement support for new control flow recovery front-ends.
### What is Remill and why does McSema need it?
### What is Remill, and why does McSema need it
[Remill](https://github.com/trailofbits/remill) is a library that McSema uses to lift individual machine code instructions to LLVM IR. You can think of McSema being to Remill as Clang is to LLVM. Remill's scope is small: it focuses on instruction semantics only, and it provides semantics for x86, x86-64, and AArch64 instruction semantics. McSema's scope is much bigger: it focuses on lifting entire programs. To do so, McSema must lift the individual instructions, but there's a lot more to lifting programs than just the instructions; there are code and data cross-references, segments, etc.
[Remill](https://github.com/trailofbits/remill) is a library that McSema uses to lift individual machine code instructions to LLVM IR. You can think of McSema being to Remill as Clang is to LLVM. Remill's scope is small: it focuses on instruction semantics only, and it provides semantics for x86, x86-64, and AArch64 instruction semantics. McSema's scope is much bigger: it focuses on lifting entire programs. To do so, McSema must lift the individual instructions, but there's a lot more to lifting programs than just the instructions; there are code and data cross-references, segments, etc.
### I'm a student and I'd like to contribute to McSema. How can I help?
### I'm a student and I'd like to contribute to McSema: how can I help
We would love to take you on as an intern to help improve McSema. We have several project ideas labelled `intern_project` in [the issues tracker](https://github.com/trailofbits/mcsema/issues?q=is%3Aissue+is%3Aopen+label%3Aintern_project). You are not limited to those items: if you think of a great feature you want in McSema, let us know and we will sponsor it. Simply contact us on our [Slack channel](https://empireslacking.herokuapp.com/) or via mcsema@trailofbits.com and let us know what you'd want to work on and why.
docs/AddingNewTests.md
+199 -191
View File
@@ -1,250 +1,258 @@
# Adding New Tests
This document describes how to add new tests to mcsema's testing frameworks.
This document describes how to add new tests to McSema's testing frameworks.
# Integration Tests
## Integration Tests
The mcsema integration tests are designed to test that mcsema translates real programs correctly and in a reasonable amount of time.
The McSema integration tests are designed to test that McSema translates real programs correctly and in a reasonable amount of time.
Translation is a two step process: first there is CFG recovery, and then transcription to bitcode. As of mcsema 0.6, CFG recovery requires IDA Pro. To ensure integration tests can run without IDA Pro present, the recovered control flow (.cfg files) are checked into git.
Translation is a two step process: first there is CFG recovery, and then transcription to bitcode. As of McSema 0.6, CFG recovery requires IDA Pro. To ensure integration tests can run without IDA Pro present, the recovered control flow (.cfg files) are checked into git.
Translation is correct if the following hold:
* Mcsema exits cleanly while translating the binary (e.g. exit code 0).
* McSema exits cleanly while translating the binary (e.g. exit code 0).
* The translated bitcode is emitted to a file.
* The bitcode can be re-built to a translated executable.
* The translated executable produces identical output (both on stdin and stderr) as the original.
* The translated executable produces identical output (both on stdin and stderr) as the original.
* The translated executable exits cleanly (e.g. exit code 0).
Below is a step-by-step guide to adding a new integration test.
## Linux
### Linux
### Source or Binary?
#### Source or Binary
Tests can start with source code or with a precompiled binary.
Tests can start with source code or with a pre-compiled binary.
Source-based tests are automatically built to x86 and amd64 architectures and should be contained within a single source file, to avoid modifying the main Makefile.
Binary-based tests start with a precompiled binary that should be placed in its architecture specific directory (either [x86](https://github.com/trailofbits/mcsema/tree/master/tests/linux/x86) or [amd64](https://github.com/trailofbits/mcsema/tree/master/tests/linux/amd64)). The file should end in `.elf` to be detected by the test generation scripts.
Binary-based tests start with a pre-compiled binary that should be placed in its architecture specific directory (either [x86](https://github.com/trailofbits/mcsema/tree/master/tests/linux/x86) or [amd64](https://github.com/trailofbits/mcsema/tree/master/tests/linux/amd64)). The file should end in `.elf` to be detected by the test generation scripts.
For this example, we'll start with the following source file, to be named `switch.c`:
#include <stdio.h>
#include <stdlib.h>
```c
#include <stdio.h>
#include <stdlib.h>
int main(int argc, const char *argv[]) {
int main(int argc, const char *argv[]) {
if(argc < 2) {
return -1;
}
int input = atoi(argv[1]);
switch(input) {
case 0:
printf("Input was zero\n");
break;
case 1:
printf("Input was one\n");
break;
case 2:
printf("Input was two\n");
break;
case 4:
printf("Input was four\n");
break;
case 6:
printf("Input was six\n");
break;
case 12:
printf("Input was twelve\n");
break;
case 13:
printf("Input was thirteen\n");
break;
case 19:
printf("Input was nineteen\n");
break;
case 255:
printf("Input was two hundred fifty-five\n");
break;
case 0x12389:
printf("Really big input: 0x12389\n");
break;
case 0x1238A:
printf("Really big input: 0x1238A\n");
break;
case 0x1238B:
printf("Really big input: 0x1238B\n");
break;
case 0x1238C:
printf("Really big input: 0x1238C\n");
break;
case 0x1238D:
printf("Really big input: 0x1238D\n");
break;
case 0x1238F:
printf("Really big input: 0x1238F\n");
break;
case 0x12390:
printf("Really big input: 0x12390\n");
break;
case 0x12391:
printf("Really big input: 0x12391\n");
break;
case 0x12392:
printf("Really big input: 0x12392\n");
break;
case 0x12393:
printf("Really big input: 0x12393\n");
break;
default:
printf("Unknown input: %d\n", input);
}
return 0;
if(argc < 2) {
return -1;
}
### Generating Expected Output
int input = atoi(argv[1]);
switch(input) {
case 0:
printf("Input was zero\n");
break;
case 1:
printf("Input was one\n");
break;
case 2:
printf("Input was two\n");
break;
case 4:
printf("Input was four\n");
break;
case 6:
printf("Input was six\n");
break;
case 12:
printf("Input was twelve\n");
break;
case 13:
printf("Input was thirteen\n");
break;
case 19:
printf("Input was nineteen\n");
break;
case 255:
printf("Input was two hundred fifty-five\n");
break;
case 0x12389:
printf("Really big input: 0x12389\n");
break;
case 0x1238A:
printf("Really big input: 0x1238A\n");
break;
case 0x1238B:
printf("Really big input: 0x1238B\n");
break;
case 0x1238C:
printf("Really big input: 0x1238C\n");
break;
case 0x1238D:
printf("Really big input: 0x1238D\n");
break;
case 0x1238F:
printf("Really big input: 0x1238F\n");
break;
case 0x12390:
printf("Really big input: 0x12390\n");
break;
case 0x12391:
printf("Really big input: 0x12391\n");
break;
case 0x12392:
printf("Really big input: 0x12392\n");
break;
case 0x12393:
printf("Really big input: 0x12393\n");
break;
default:
printf("Unknown input: %d\n", input);
}
return 0;
}
```
#### Generating Expected Output
We need to tell the test generation scripts what output the program should have. To do that, edit [x86/config.json](https://github.com/trailofbits/mcsema/blob/master/tests/linux/x86/config.json) and [amd64/config.json](https://github.com/trailofbits/mcsema/blob/master/tests/linux/amd64/config.json) and add:
"switch": {
"_comment": "A test for switch statements, which hits mcsema's jump table handling",
"switch 1": {
"_comment": "Executes ./switch 1 [the most simple test case]",
"args": ["1"]
},
"switch 14": {
"_comment": "Executes ./switch 14 [hits the default case]",
"args": ["14"]
},
"switch 74642": {
"_comment": "Executes ./switch 74642 [hits the big number case]",
"args": ["74642"]
}
```json
"switch": {
"_comment": "A test for switch statements, which hits mcsema's jump table handling",
"switch 1": {
"_comment": "Executes ./switch 1 [the most simple test case]",
"args": ["1"]
},
"switch 14": {
"_comment": "Executes ./switch 14 [hits the default case]",
"args": ["14"]
},
"switch 74642": {
"_comment": "Executes ./switch 74642 [hits the big number case]",
"args": ["74642"]
}
}
```
The top level name, `"switch"`, has to correspond to the test name.
Any entries that begin with `_`, like `_comment`, are ignored by the test generation scripts, and serve as comments to human readers.
Each entry in the testing block for `"switch"` is a test name, like `"switch 14"`. Inside the test name entry the only required entry is `args`, which is a list specifying command line arguments to pass to the program under test. if no arguments are needed, the list should be the empty list `[]`.
Each entry in the testing block for `"switch"` is a test name, like `"switch 14"`. Inside the test name entry the only required entry is `args`, which is a list specifying command line arguments to pass to the program under test. if no arguments are needed, the list should be the empty list `[]`.
### Generate the tests
#### Generate the tests
The Makefile in [tests/linux](https://github.com/trailofbits/mcsema/blob/master/tests/linux/Makefile) will generate the CFG files and expected outputs.
IDA Pro is required to generate the pre-computed CFG files.
The Makefile will attempt to find IDA Pro and mcsema using the `IDA_PATH` and `MCSEMA_PATH` environment variables, respectively. If that fails, the Makefile will default to looking for each in their usual locations.
The Makefile will attempt to find IDA Pro and McSema using the `IDA_PATH` and `MCSEMA_PATH` environment variables, respectively. If that fails, the Makefile will default to looking for each in their usual locations.
In this example, we'll manually specify the IDA and mcsema paths. To build the new tests, simply type `make all`:
In this example, we'll manually specify the IDA and McSema paths. To build the new tests, simply type `make all`:
artem@nessie:/store/artem/git/mcsema/tests/linux$ IDA_PATH=/home/artem/ida-6.9 MCSEMA_PATH=/store/artem/git/mcsema/bin/ make all
clang-3.8 -m64 -o amd64/hello.elf hello.c
/store/artem/git/mcsema/bin//mcsema-disass --disassembler /home/artem/ida-6.9/idal64 --arch amd64 --os linux --output amd64/hello.cfg --binary amd64/hello.elf --entrypoint main
clang-3.8 -m64 -o amd64/stringpool.elf stringpool.c
/store/artem/git/mcsema/bin//mcsema-disass --disassembler /home/artem/ida-6.9/idal64 --arch amd64 --os linux --output amd64/stringpool.cfg --binary amd64/stringpool.elf --entrypoint main
clang-3.8 -m64 -o amd64/switch.elf switch.c
/store/artem/git/mcsema/bin//mcsema-disass --disassembler /home/artem/ida-6.9/idal64 --arch amd64 --os linux --output amd64/switch.cfg --binary amd64/switch.elf --entrypoint main
/store/artem/git/mcsema/tests/linux//generate_expected_output.py amd64/config.json amd64/expected_outputs.json
Using temporary directory: /tmp/tmp2KD6RP
Processing test: ls
Processing configuration: default
Executing: [u'/store/artem/git/mcsema/tests/linux/amd64/ls.elf', u'-d', u'/usr']
Processing configuration: width test
Executing: [u'/store/artem/git/mcsema/tests/linux/amd64/ls.elf', u'--width=100', u'-d', u'/usr']
Processing test: switch
Processing configuration: switch 74642
Executing: [u'/store/artem/git/mcsema/tests/linux/amd64/switch.elf', u'74642']
Processing configuration: switch 1
Executing: [u'/store/artem/git/mcsema/tests/linux/amd64/switch.elf', u'1']
Processing configuration: switch 14
Executing: [u'/store/artem/git/mcsema/tests/linux/amd64/switch.elf', u'14']
Processing test: hello
Processing configuration: default
Executing: [u'/store/artem/git/mcsema/tests/linux/amd64/hello.elf']
Processing test: stringpool
Processing configuration: default
Executing: [u'/store/artem/git/mcsema/tests/linux/amd64/stringpool.elf']
Saved ground truth to: amd64/expected_outputs.json
clang-3.8 -m32 -o x86/hello.elf hello.c
/store/artem/git/mcsema/bin//mcsema-disass --disassembler /home/artem/ida-6.9/idal --arch x86 --os linux --output x86/hello.cfg --binary x86/hello.elf --entrypoint main
clang-3.8 -m32 -o x86/stringpool.elf stringpool.c
/store/artem/git/mcsema/bin//mcsema-disass --disassembler /home/artem/ida-6.9/idal --arch x86 --os linux --output x86/stringpool.cfg --binary x86/stringpool.elf --entrypoint main
clang-3.8 -m32 -o x86/switch.elf switch.c
/store/artem/git/mcsema/bin//mcsema-disass --disassembler /home/artem/ida-6.9/idal --arch x86 --os linux --output x86/switch.cfg --binary x86/switch.elf --entrypoint main
/store/artem/git/mcsema/tests/linux//generate_expected_output.py x86/config.json x86/expected_outputs.json
Using temporary directory: /tmp/tmpdp8QDP
Processing test: switch
Processing configuration: switch 74642
Executing: [u'/store/artem/git/mcsema/tests/linux/x86/switch.elf', u'74642']
Processing configuration: switch 1
Executing: [u'/store/artem/git/mcsema/tests/linux/x86/switch.elf', u'1']
Processing configuration: switch 14
Executing: [u'/store/artem/git/mcsema/tests/linux/x86/switch.elf', u'14']
Processing test: hello
Processing configuration: default
Executing: [u'/store/artem/git/mcsema/tests/linux/x86/hello.elf']
Processing test: stringpool
Processing configuration: default
Executing: [u'/store/artem/git/mcsema/tests/linux/x86/stringpool.elf']
Saved ground truth to: x86/expected_outputs.json
```shell
artem@nessie:/store/artem/git/mcsema/tests/linux$ IDA_PATH=/home/artem/ida-6.9 MCSEMA_PATH=/store/artem/git/mcsema/bin/ make all
clang-3.8 -m64 -o amd64/hello.elf hello.c
/store/artem/git/mcsema/bin//mcsema-disass --disassembler /home/artem/ida-6.9/idal64 --arch amd64 --os linux --output amd64/hello.cfg --binary amd64/hello.elf --entrypoint main
clang-3.8 -m64 -o amd64/stringpool.elf stringpool.c
/store/artem/git/mcsema/bin//mcsema-disass --disassembler /home/artem/ida-6.9/idal64 --arch amd64 --os linux --output amd64/stringpool.cfg --binary amd64/stringpool.elf --entrypoint main
clang-3.8 -m64 -o amd64/switch.elf switch.c
/store/artem/git/mcsema/bin//mcsema-disass --disassembler /home/artem/ida-6.9/idal64 --arch amd64 --os linux --output amd64/switch.cfg --binary amd64/switch.elf --entrypoint main
/store/artem/git/mcsema/tests/linux//generate_expected_output.py amd64/config.json amd64/expected_outputs.json
Using temporary directory: /tmp/tmp2KD6RP
Processing test: ls
Processing configuration: default
Executing: [u'/store/artem/git/mcsema/tests/linux/amd64/ls.elf', u'-d', u'/usr']
Processing configuration: width test
Executing: [u'/store/artem/git/mcsema/tests/linux/amd64/ls.elf', u'--width=100', u'-d', u'/usr']
Processing test: switch
Processing configuration: switch 74642
Executing: [u'/store/artem/git/mcsema/tests/linux/amd64/switch.elf', u'74642']
Processing configuration: switch 1
Executing: [u'/store/artem/git/mcsema/tests/linux/amd64/switch.elf', u'1']
Processing configuration: switch 14
Executing: [u'/store/artem/git/mcsema/tests/linux/amd64/switch.elf', u'14']
Processing test: hello
Processing configuration: default
Executing: [u'/store/artem/git/mcsema/tests/linux/amd64/hello.elf']
Processing test: stringpool
Processing configuration: default
Executing: [u'/store/artem/git/mcsema/tests/linux/amd64/stringpool.elf']
Saved ground truth to: amd64/expected_outputs.json
clang-3.8 -m32 -o x86/hello.elf hello.c
/store/artem/git/mcsema/bin//mcsema-disass --disassembler /home/artem/ida-6.9/idal --arch x86 --os linux --output x86/hello.cfg --binary x86/hello.elf --entrypoint main
clang-3.8 -m32 -o x86/stringpool.elf stringpool.c
/store/artem/git/mcsema/bin//mcsema-disass --disassembler /home/artem/ida-6.9/idal --arch x86 --os linux --output x86/stringpool.cfg --binary x86/stringpool.elf --entrypoint main
clang-3.8 -m32 -o x86/switch.elf switch.c
/store/artem/git/mcsema/bin//mcsema-disass --disassembler /home/artem/ida-6.9/idal --arch x86 --os linux --output x86/switch.cfg --binary x86/switch.elf --entrypoint main
/store/artem/git/mcsema/tests/linux//generate_expected_output.py x86/config.json x86/expected_outputs.json
Using temporary directory: /tmp/tmpdp8QDP
Processing test: switch
Processing configuration: switch 74642
Executing: [u'/store/artem/git/mcsema/tests/linux/x86/switch.elf', u'74642']
Processing configuration: switch 1
Executing: [u'/store/artem/git/mcsema/tests/linux/x86/switch.elf', u'1']
Processing configuration: switch 14
Executing: [u'/store/artem/git/mcsema/tests/linux/x86/switch.elf', u'14']
Processing test: hello
Processing configuration: default
Executing: [u'/store/artem/git/mcsema/tests/linux/x86/hello.elf']
Processing test: stringpool
Processing configuration: default
Executing: [u'/store/artem/git/mcsema/tests/linux/x86/stringpool.elf']
Saved ground truth to: x86/expected_outputs.json
```
### Running the tests
#### Running the tests
There is one more step needed before these tests will be run by the integration test system: telling the test system to run them.
To do that, edit [tests/integration_test.py](https://github.com/trailofbits/mcsema/blob/master/tests/integration_test.py) and add a new function to the LinuxTest class. Since these tests use the Python `unittest` framework, the name of each function must start with `test`. We'll name this one `testswitch`:
def testswitch(self):
self._runX86Test("switch")
self._runAMD64Test("switch")
```python
def testswitch(self):
self._runX86Test("switch")
self._runAMD64Test("switch")
```
Now we can run the tests, by executing `integration_test.py`. These tests will automatically run in Travis-CI upon push to GitHub.
Now we can run the tests, by executing `integration_test.py`. These tests will automatically run in Travis-CI upon push to Github.
```shell
$ python integration_test.py
testHello (__main__.LinuxTest) ... ~
~
~
~
~
~
ok
testStringPool (__main__.LinuxTest) ... ~
~
~
~
~
~
ok
testls (__main__.LinuxTest) ... skipped 'Re-enable after we fix issue #108'
testswitch (__main__.LinuxTest) ... ~
~
~
~
~
~
~
~
~
~
ok
$ python integration_test.py
testHello (__main__.LinuxTest) ... ~
~
~
~
~
~
ok
testStringPool (__main__.LinuxTest) ... ~
~
~
~
~
~
ok
testls (__main__.LinuxTest) ... skipped 'Re-enable after we fix issue #108'
testswitch (__main__.LinuxTest) ... ~
~
~
~
~
~
~
~
~
~
ok
----------------------------------------------------------------------
Ran 4 tests in 22.087s
OK (skipped=1)
----------------------------------------------------------------------
Ran 4 tests in 22.087s
OK (skipped=1)
```
Success, it works! The output tells us the testswitch function ran, and all conditions for a successful test were met.
## Windows
### Windows
To be determined once the integration test system has been ported to Windows, but it will look a lot like Linux.
# Unit Tests
## Unit Tests
Currently in flux as we are re-doing the unit testing framework.
We write and execute unit tests for Remill using Google Test, which serves as the unit testing framework for McSema. Every instruction semantic added to Remill is tested with multiple test cases in native assembly, such that we compare the execution results of the native assembly with the results of the equivalent lifted version. See the [Remill repository](https://github.com/trailofbits/remill) for more information.
docs/CommandLineReference.md
+26
View File
@@ -0,0 +1,26 @@
# McSema Command Line Reference
## mcsema-disass
Usage: mcsema-disass --disassembler _path-to-IDA_ --os _operating-system_ --arch _architecture_ --output _cfg-path_ --binary _input-binary_ --entrypoint _function_ [--log_file _log-path_]
Where:
* `path-to-IDA` = the path to your IDA Pro disassembler executable, e.g., `~/ida-6.9/idal64`
* `operating-system` = the OS _of the binary being disassembled_: `linux`, or `windows`
* `architecture` = the instruction set architecture _of the binary being disassembled_: `amd64`, `amd64_avx`, `x86`, `x86_avx`, or `aarch64` (64-bit ARMv8)
* `cfg-path` = the path a .cfg file where you want the recovered control flow graph to be saved
* `input-binary` = the path to a binary executable to be disassembled
* `function` = the entry point function where the disassembler should start recovering control flow, e.g., `main`
* `log-path` = (optional) the path to a log file to save the logging output of McSema
## mcsema-lift
Usage: mcsema-lift --arch _architecture_ --os _platform_ --cfg _cfg-path_ [--output _output-path_]
Where:
* `architecture` = architecture to use for the instruction semantics during lifting: `amd64`, `amd64_avx`, `x86`, `x86_avx`, or `aarch64` (64-bit ARMv8)
* `platform` = the operating system _of the binary that was disassembled_ to generate this CFG. Currently the valid options are `linux` or `windows`. This option is required for certain aspects of translation, like ABI compatibility for external functions, etc.
* `cfg-path` = path to the control flow graph file emitted by `mcsema-disass` that you want to convert into bitcode
* `output-path` = path to a .bc file where you want the lifted code to be saved. If the `--output` option is not specified, the bitcode will be written to stdout
docs/CommonErrors.md
+138 -111
View File
@@ -2,68 +2,74 @@
This document describes common errors and fixes to problems faced when running `mcsema-lift` and `mcsema-disass`. If you are facing problems when running the compiled bitcode then check out the [Debugging Tips](DebuggingTips.md) document.
# Segfaults, segfaults everywhere!
## Segfaults, segfaults everywhere
So, you've got a case of the segfaults. Lets try to diagnose the issue.
Most likely, you've got a case of CFG recovery failure. Here are some common causes:
## Lifting position independent code without `--pie-mode`
### Lifting position independent code without `--pie-mode`
When mcsema-lift sees an instruction like `mov rax, 0x60008`, it needs to know whether that `0x60008` is a constant value, or whether it references code or data somewhere in the program at location `0x60008`. IDA is pretty good at figuring out the differnce, but not always. Sometimes mcsema just has to make an educated guess. On binaries built with position independent code (`-pie`, `-fPIC`), the default heuristic is wrong. You want to use `mcsema-disass --pie-mode` for more correct behavior.
When `mcsema-lift` sees an instruction like `mov rax, 0x60008`, it needs to know whether that `0x60008` is a constant value, or whether it references code or data somewhere in the program at location `0x60008`. IDA is pretty good at figuring out the difference, but not always. Sometimes McSema just has to make an educated guess. On binaries built with position independent code (`-pie`, `-fPIC`), the default heuristic is wrong. You want to use `mcsema-disass --pie-mode` for more correct behavior.
How can you tell which to use? Check the binary type.
$ file my-pie-binary
my-pie-binary: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, ...
```shell
$ file my-pie-binary
my-pie-binary: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, ...
```
If you see the words `LSB shared object`, you probably want to use `--pie-mode`. This isn't a 100% reliable heuristic; if you are having segfault issues try `--pie-mode` to see if it helps.
**Technical Details:** Using `--pie-mode` mcsema assumes values it encounters are constants. Normally, mcsema is biased towards immediate values that fall into the code or data section as being references.
**Technical Details:** Using `--pie-mode` McSema assumes values it encounters are constants. Normally, McSema is biased towards immediate values that fall into the code or data section as being references.
# Errors when using `mcsema-disass`
## Errors when using `mcsema-disass`
## Unknown External `<Function Name>`
### Unknown External `<Function Name>`
The error in the mcsema-disass output log looks something like this:
The error in the `mcsema-disass` output log looks something like this:
Unknown external: cs_open
Traceback (most recent call last):
File "/home/artem/.local/lib/python2.7/site-packages/mcsema_disass-0.0.1-py2.7.egg/mcsema_disass/ida/get_cfg.py", line 2287, in <module>
recoverCfg(eps, outf, args.exports_are_apis)
File "/home/artem/.local/lib/python2.7/site-packages/mcsema_disass-0.0.1-py2.7.egg/mcsema_disass/ida/get_cfg.py", line 1893, in recoverCfg
recoverFunction(M, F, fea, new_eas)
File "/home/artem/.local/lib/python2.7/site-packages/mcsema_disass-0.0.1-py2.7.egg/mcsema_disass/ida/get_cfg.py", line 1693, in recoverFunction
recoverFunctionFromSet(M, F, blockset, new_eas)
File "/home/artem/.local/lib/python2.7/site-packages/mcsema_disass-0.0.1-py2.7.egg/mcsema_disass/ida/get_cfg.py", line 1681, in recoverFunctionFromSet
I, endBlock = instructionHandler(M, B, head, new_eas)
File "/home/artem/.local/lib/python2.7/site-packages/mcsema_disass-0.0.1-py2.7.egg/mcsema_disass/ida/get_cfg.py", line 817, in instructionHandler
if doesNotReturn(fn):
File "/home/artem/.local/lib/python2.7/site-packages/mcsema_disass-0.0.1-py2.7.egg/mcsema_disass/ida/get_cfg.py", line 220, in doesNotReturn
raise Exception("Unknown external: " + fname)
Exception: Unknown external: cs_open
```shell
Unknown external: cs_open
Traceback (most recent call last):
File "/home/artem/.local/lib/python2.7/site-packages/mcsema_disass-0.0.1-py2.7.egg/mcsema_disass/ida/get_cfg.py", line 2287, in <module>
recoverCfg(eps, outf, args.exports_are_apis)
File "/home/artem/.local/lib/python2.7/site-packages/mcsema_disass-0.0.1-py2.7.egg/mcsema_disass/ida/get_cfg.py", line 1893, in recoverCfg
recoverFunction(M, F, fea, new_eas)
File "/home/artem/.local/lib/python2.7/site-packages/mcsema_disass-0.0.1-py2.7.egg/mcsema_disass/ida/get_cfg.py", line 1693, in recoverFunction
recoverFunctionFromSet(M, F, blockset, new_eas)
File "/home/artem/.local/lib/python2.7/site-packages/mcsema_disass-0.0.1-py2.7.egg/mcsema_disass/ida/get_cfg.py", line 1681, in recoverFunctionFromSet
I, endBlock = instructionHandler(M, B, head, new_eas)
File "/home/artem/.local/lib/python2.7/site-packages/mcsema_disass-0.0.1-py2.7.egg/mcsema_disass/ida/get_cfg.py", line 817, in instructionHandler
if doesNotReturn(fn):
File "/home/artem/.local/lib/python2.7/site-packages/mcsema_disass-0.0.1-py2.7.egg/mcsema_disass/ida/get_cfg.py", line 220, in doesNotReturn
raise Exception("Unknown external: " + fname)
Exception: Unknown external: cs_open
```
**Technical Background:** Mcsema needs to know how to call external functions, including the function calling convention and number of arguments. It knows how to call many common functions, but you hit one that it does not know about.
**Technical Background:** McSema needs to know how to call external functions, including the function calling convention and number of arguments. It knows how to call many common functions, but you hit one that it does not know about.
**Possible Fixes:** Add an entry to the [external function definitions file](https://github.com/trailofbits/mcsema/tree/master/tools/mcsema_disass/defs) describing the function's calling convention and number of arguments. Don't forget to re-build `mcsema-disass`. Submit a pull request so we can include it in future mcsema releases. Alternatively, you can specify a custom definitions file location by using the `--std-defs` argument (e.g. `mcsema-disass --std-defs /path/to/my/defs/file.txt ...`).
**Possible Fixes:** Add an entry to the [external function definitions file](https://github.com/trailofbits/mcsema/tree/master/tools/mcsema_disass/defs) describing the function's calling convention and number of arguments. Don't forget to re-build `mcsema-disass`. Submit a pull request so we can include it in future McSema releases. Alternatively, you can specify a custom definitions file location by using the `--std-defs` argument (e.g. `mcsema-disass --std-defs /path/to/my/defs/file.txt ...`).
## Could not parse function type
### Could not parse function type
You are trying to disassemble a binary and see something like the following in the output log:
Could not parse function type:__int64(void)
Traceback (most recent call last):
File "/home/artem/.local/lib/python2.7/site-packages/mcsema_disass-0.0.1-py2.7.egg/mcsema_disass/ida/get_cfg.py", line 2287, in <module>
recoverCfg(eps, outf, args.exports_are_apis)
File "/home/artem/.local/lib/python2.7/site-packages/mcsema_disass-0.0.1-py2.7.egg/mcsema_disass/ida/get_cfg.py", line 1890, in recoverCfg
F = entryPointHandler(M, fea, fname, exports_are_apis)
File "/home/artem/.local/lib/python2.7/site-packages/mcsema_disass-0.0.1-py2.7.egg/mcsema_disass/ida/get_cfg.py", line 265, in entryPointHandler
(argc, conv, ret) = getExportType(name, ep)
File "/home/artem/.local/lib/python2.7/site-packages/mcsema_disass-0.0.1-py2.7.egg/mcsema_disass/ida/get_cfg.py", line 2023, in getExportType
return parseTypeString(tp, ep)
File "/home/artem/.local/lib/python2.7/site-packages/mcsema_disass-0.0.1-py2.7.egg/mcsema_disass/ida/get_cfg.py", line 2001, in parseTypeString
raise Exception("Could not parse function type:"+typestr)
Exception: Could not parse function type:__int64(void)
```shell
Could not parse function type:__int64(void)
Traceback (most recent call last):
File "/home/artem/.local/lib/python2.7/site-packages/mcsema_disass-0.0.1-py2.7.egg/mcsema_disass/ida/get_cfg.py", line 2287, in <module>
recoverCfg(eps, outf, args.exports_are_apis)
File "/home/artem/.local/lib/python2.7/site-packages/mcsema_disass-0.0.1-py2.7.egg/mcsema_disass/ida/get_cfg.py", line 1890, in recoverCfg
F = entryPointHandler(M, fea, fname, exports_are_apis)
File "/home/artem/.local/lib/python2.7/site-packages/mcsema_disass-0.0.1-py2.7.egg/mcsema_disass/ida/get_cfg.py", line 265, in entryPointHandler
(argc, conv, ret) = getExportType(name, ep)
File "/home/artem/.local/lib/python2.7/site-packages/mcsema_disass-0.0.1-py2.7.egg/mcsema_disass/ida/get_cfg.py", line 2023, in getExportType
return parseTypeString(tp, ep)
File "/home/artem/.local/lib/python2.7/site-packages/mcsema_disass-0.0.1-py2.7.egg/mcsema_disass/ida/get_cfg.py", line 2001, in parseTypeString
raise Exception("Could not parse function type:"+typestr)
Exception: Could not parse function type:__int64(void)
```
**Technical Background:** Mcsema tries to parse IDA's type signatures for functions it doesn't know about. The type signature parsing is a last ditch effort, and it failed.
@@ -71,70 +77,83 @@ You are trying to disassemble a binary and see something like the following in t
**Debugging Hints:** Make sure you specify an entrypoint when using `mcsema-disass` (e.g. `--entrypoint main`).
# Errors when using `mcsema-lift`
## Errors when using `mcsema-lift`
## Error translating instruction
### Error translating instruction
The error message reads similar to:
Error translating instruction at 4007cf; unsupported opcode 176
```shell
Error translating instruction at 4007cf; unsupported opcode 176
```
**Technical Background:** The semantics of the instruction you are trying to translate are not present in mcsema.
**Technical Background:** The semantics of the instruction you are trying to translate are not yet present in McSema.
**Possible Fixes:** First, you could implement the instruction semantics and submit a pull request with the implementation. Second, you can try to use the `-ignore-unsupported` flag to `mcsema-lift` so mcsema will silenty ignore this unsupported instruction. Missing instructions may or may not matter, depending on what you want to do with the translated bitcode.
**Possible Fixes:** First, you could implement the instruction semantics, and submit a pull request with the implementation. Second, you can try to use the `-ignore-unsupported` flag to `mcsema-lift` so McSema will silently ignore this unsupported instruction. Missing instructions may or may not matter, depending on what you want to do with the translated bitcode.
**Debugging Hints:** The error message tells you the location of the instruction in the binary, and its LLVM MC-layer opcode. This information will help in implmenting the instruction. In the case of our example message, the instruction is `aeskeygenassist`:
$ objdump -x -d our_binary | grep 4007cf
4007cf: 66 0f 3a df d1 00 aeskeygenassist $0x0,%xmm1,%xmm2
**Debugging Hints:** The error message tells you the location of the instruction in the binary, and its LLVM MC-layer opcode. This information will help in implementing the instruction. In the case of our example message, the instruction is `aeskeygenassist`:
```shell
$ objdump -x -d our_binary | grep 4007cf
4007cf: 66 0f 3a df d1 00 aeskeygenassist $0x0,%xmm1,%xmm2
```
The `llvm-mc` tool can then tell you more about how the LLVM MC layer identifies the instruction:
$ echo 'aeskeygenassist $0x0,%xmm1,%xmm2' | llvm-mc-3.8 -assemble -show-inst
.text
aeskeygenassist $0, %xmm1, %xmm2 # <MCInst #176 AESKEYGENASSIST128rr
# <MCOperand Reg:128>
# <MCOperand Reg:127>
# <MCOperand Imm:0>>
```shell
$ echo 'aeskeygenassist $0x0,%xmm1,%xmm2' | llvm-mc-3.8 -assemble -show-inst
.text
aeskeygenassist $0, %xmm1, %xmm2 # <MCInst #176 AESKEYGENASSIST128rr
# <MCOperand Reg:128>
# <MCOperand Reg:127>
# <MCOperand Imm:0>>
```
In this case, we'd need to add an entry for `X86::AESKEYGENASSIST128rr` in the instruction dispatch map to implement `aeskeygenassist`.
## Basic Block does not have a terminator
### Basic Block does not have a terminator
You are translating a binary, and you see something that resembles the following output:
...
Adding entry point: main
main is implemented by sub_804e570
Basic Block in function 'strcoll' does not have terminator!
label %190
Could not verify module!
```shell
...
Adding entry point: main
main is implemented by sub_804e570
Basic Block in function 'strcoll' does not have terminator!
label %190
Could not verify module!
```
**Technical background:** LLVM requires that every basic block end in a terminating instruction (e.g. call, ret, branch, etc.). For some reason, the mcasema generated IR created a block that does not end in a terminator.
**Technical background:** LLVM requires that every basic block end in a terminating instruction (e.g. call, ret, branch, etc.). For some reason, the McSema-generated IR created a block that does not end in a terminator.
**Possible fixes:** There is likely a bug in an instruction implementation. Does your instruction create new blocks? If yes, did you update the "current block to add stuff to" (the `block` argument to the instruction implementation handler)? It is passed as a reference to a pointer, so `block = newBlock;` will update it.
**Debugging Hints:** Use the `--print-before-all` flag to `mcsema-lift` to dump generated IR to a file. Look through it backwards to identify the terminator-less block, and its corresponding x86 instruction.
## NIY Error
### NIY Error
You get an error that says "NIY" and has a line number, for example:
error:
/home/user/mcsema/mcsema/cfgToLLVM/x86Instrs_String.cpp:773
: NIY
```shell
error:
/home/user/mcsema/mcsema/cfgToLLVM/x86Instrs_String.cpp:773
: NIY
```
**Technical Background:** Congratulations! You hit a part of mcsema that we thought about, but didn't finish implementing.
**Technical Background:** Congratulations! You hit a part of McSema that we thought about, but didn't finish implementing.
**Possible Fixes:** The best option is to implement the missing functionality and submit a pull request :). Alternatively, file an issue on Github and provide us a binary so we can reproduce the problem.
**Possible Fixes:** The best option is to implement the missing functionality and submit a pull request :). Alternatively, file an issue on GitHub and provide us a binary so that we can reproduce the problem.
**Debugging Hints:** To know more about the problem, look at the line number in the error message. The line may be a macro, so you may need to find the macro definition to see the root cause of the problem.
## StoreInst assertion failure
### StoreInst assertion failure
When using `mcsema-lift` you see an assertion failure in StoreInst like the following:
mcsema-lift: /store/artem/git/mcsema/third_party/llvm/lib/IR/Instructions.cpp:1304: void llvm::StoreInst::AssertOK(): Assertion `getOperand(0)->getType() == cast<PointerType>(getOperand(1)->getType())->getElementType() && "Ptr must be a pointer to Val type!"' failed.
Aborted (core dumped)
```shell
mcsema-lift: /store/artem/git/mcsema/third_party/llvm/lib/IR/Instructions.cpp:1304: void llvm::StoreInst::AssertOK(): Assertion `getOperand(0)->getType() == cast<PointerType>(getOperand(1)->getType())->getElementType() && "Ptr must be a pointer to Val type!"' failed.
Aborted (core dumped)
```
**Technical Background:** There is an LLVM `store` instruction, and the type of what you're trying to store and where you are trying to put it are different.
@@ -142,45 +161,49 @@ When using `mcsema-lift` you see an assertion failure in StoreInst like the foll
**Debugging Hints:** Use a debugger to launch `mcsema-lift`, and identify the instruction from the backtrace. Here is an example:
Program received signal SIGABRT, Aborted.
0x00007ffff6418428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
54 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 0x00007ffff6418428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1 0x00007ffff641a02a in __GI_abort () at abort.c:89
#2 0x00007ffff6410bd7 in __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x1129956 "getOperand(0)->getType() == cast<PointerType>(getOperand(1)->getType())->getElementType() && \"Ptr must be a pointer to Val type!\"",
file=file@entry=0x112869c "/store/artem/git/mcsema/third_party/llvm/lib/IR/Instructions.cpp", line=line@entry=1304, function=function@entry=0x11298ec "void llvm::StoreInst::AssertOK()") at assert.c:92
#3 0x00007ffff6410c82 in __GI___assert_fail (assertion=0x1129956 "getOperand(0)->getType() == cast<PointerType>(getOperand(1)->getType())->getElementType() && \"Ptr must be a pointer to Val type!\"",
file=0x112869c "/store/artem/git/mcsema/third_party/llvm/lib/IR/Instructions.cpp", line=1304, function=0x11298ec "void llvm::StoreInst::AssertOK()") at assert.c:101
#4 0x0000000000c1961e in llvm::StoreInst::AssertOK (this=0x9a3fe40) at /store/artem/git/mcsema/third_party/llvm/lib/IR/Instructions.cpp:1302
#5 0x0000000000c19ad3 in llvm::StoreInst::StoreInst (this=0x9a3fe40, val=0x9a3fdc0, addr=0x9a3fc98, isVolatile=false, Align=0, Order=llvm::NotAtomic, SynchScope=llvm::CrossThread, InsertAtEnd=0x9a2b1e0)
at /store/artem/git/mcsema/third_party/llvm/lib/IR/Instructions.cpp:1362
#6 0x0000000000c19893 in llvm::StoreInst::StoreInst (this=0x9a3fe40, val=0x9a3fdc0, addr=0x9a3fc98, isVolatile=false, Align=0, InsertAtEnd=0x9a2b1e0) at /store/artem/git/mcsema/third_party/llvm/lib/IR/Instructions.cpp:1330
#7 0x0000000000c19799 in llvm::StoreInst::StoreInst (this=0x9a3fe40, val=0x9a3fdc0, addr=0x9a3fc98, isVolatile=false, InsertAtEnd=0x9a2b1e0) at /store/artem/git/mcsema/third_party/llvm/lib/IR/Instructions.cpp:1321
#8 0x00000000005a7d71 in INTERNAL_M_WRITE (width=32, addrspace=0, b=0x9a2b1e0, addr=0x9a3fbd8, data=0x9a3fdc0) at /store/artem/git/mcsema/mcsema/cfgToLLVM/raiseX86.cpp:132
#9 0x00000000006762e9 in M_WRITE<32> (ip=0x5873c20, b=0x9a2b1e0, addr=0x9a3fbd8, data=0x9a3fdc0) at /store/artem/git/mcsema/mcsema/cfgToLLVM/raiseX86.h:136
#10 0x0000000000676020 in doMIMovV<32> (ip=0x5873c20, b=@0x7fffffffd730: 0x9a2b1e0, dstAddr=0x9a3fbd8, src=0x9a3fdc0) at /store/artem/git/mcsema/mcsema/cfgToLLVM/x86Instrs_MOV.cpp:473
#11 0x000000000066d2b9 in translate_MOV32mi (ctx=..., block=@0x7fffffffd730: 0x9a2b1e0) at /store/artem/git/mcsema/mcsema/cfgToLLVM/x86Instrs_MOV.cpp:643
#12 0x00000000005b7db6 in LiftInstIntoBlockImpl (ctx=..., block=@0x7fffffffd730: 0x9a2b1e0) at /store/artem/git/mcsema/mcsema/cfgToLLVM/x86Instrs.cpp:138
#13 0x00000000005ad51a in LiftInstIntoBlock (ctx=..., block=@0x7fffffffd730: 0x9a2b1e0, doAnnotation=true) at /store/artem/git/mcsema/mcsema/cfgToLLVM/raiseX86.cpp:407
#14 0x00000000005ad25e in LiftBlockIntoFunction (ctx=...) at /store/artem/git/mcsema/mcsema/cfgToLLVM/raiseX86.cpp:440
#15 0x00000000005acec9 in InsertFunctionIntoModule (mod=0x72093b0, func=0x586e550, M=0x16d2060) at /store/artem/git/mcsema/mcsema/cfgToLLVM/raiseX86.cpp:509
#16 0x00000000005ac814 in LiftFunctionsIntoModule (natMod=0x72093b0, M=0x16d2060) at /store/artem/git/mcsema/mcsema/cfgToLLVM/raiseX86.cpp:937
#17 0x00000000005aaee4 in LiftCodeIntoModule (natMod=0x72093b0, M=0x16d2060) at /store/artem/git/mcsema/mcsema/cfgToLLVM/raiseX86.cpp:952
#18 0x00000000005544a0 in main (argc=11, argv=0x7fffffffdec8) at /store/artem/git/mcsema/mcsema/Lift.cpp:124
```shell
Program received signal SIGABRT, Aborted.
0x00007ffff6418428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
54 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 0x00007ffff6418428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1 0x00007ffff641a02a in __GI_abort () at abort.c:89
#2 0x00007ffff6410bd7 in __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x1129956 "getOperand(0)->getType() == cast<PointerType>(getOperand(1)->getType())->getElementType() && \"Ptr must be a pointer to Val type!\"",
file=file@entry=0x112869c "/store/artem/git/mcsema/third_party/llvm/lib/IR/Instructions.cpp", line=line@entry=1304, function=function@entry=0x11298ec "void llvm::StoreInst::AssertOK()") at assert.c:92
#3 0x00007ffff6410c82 in __GI___assert_fail (assertion=0x1129956 "getOperand(0)->getType() == cast<PointerType>(getOperand(1)->getType())->getElementType() && \"Ptr must be a pointer to Val type!\"",
file=0x112869c "/store/artem/git/mcsema/third_party/llvm/lib/IR/Instructions.cpp", line=1304, function=0x11298ec "void llvm::StoreInst::AssertOK()") at assert.c:101
#4 0x0000000000c1961e in llvm::StoreInst::AssertOK (this=0x9a3fe40) at /store/artem/git/mcsema/third_party/llvm/lib/IR/Instructions.cpp:1302
#5 0x0000000000c19ad3 in llvm::StoreInst::StoreInst (this=0x9a3fe40, val=0x9a3fdc0, addr=0x9a3fc98, isVolatile=false, Align=0, Order=llvm::NotAtomic, SynchScope=llvm::CrossThread, InsertAtEnd=0x9a2b1e0)
at /store/artem/git/mcsema/third_party/llvm/lib/IR/Instructions.cpp:1362
#6 0x0000000000c19893 in llvm::StoreInst::StoreInst (this=0x9a3fe40, val=0x9a3fdc0, addr=0x9a3fc98, isVolatile=false, Align=0, InsertAtEnd=0x9a2b1e0) at /store/artem/git/mcsema/third_party/llvm/lib/IR/Instructions.cpp:1330
#7 0x0000000000c19799 in llvm::StoreInst::StoreInst (this=0x9a3fe40, val=0x9a3fdc0, addr=0x9a3fc98, isVolatile=false, InsertAtEnd=0x9a2b1e0) at /store/artem/git/mcsema/third_party/llvm/lib/IR/Instructions.cpp:1321
#8 0x00000000005a7d71 in INTERNAL_M_WRITE (width=32, addrspace=0, b=0x9a2b1e0, addr=0x9a3fbd8, data=0x9a3fdc0) at /store/artem/git/mcsema/mcsema/cfgToLLVM/raiseX86.cpp:132
#9 0x00000000006762e9 in M_WRITE<32> (ip=0x5873c20, b=0x9a2b1e0, addr=0x9a3fbd8, data=0x9a3fdc0) at /store/artem/git/mcsema/mcsema/cfgToLLVM/raiseX86.h:136
#10 0x0000000000676020 in doMIMovV<32> (ip=0x5873c20, b=@0x7fffffffd730: 0x9a2b1e0, dstAddr=0x9a3fbd8, src=0x9a3fdc0) at /store/artem/git/mcsema/mcsema/cfgToLLVM/x86Instrs_MOV.cpp:473
#11 0x000000000066d2b9 in translate_MOV32mi (ctx=..., block=@0x7fffffffd730: 0x9a2b1e0) at /store/artem/git/mcsema/mcsema/cfgToLLVM/x86Instrs_MOV.cpp:643
#12 0x00000000005b7db6 in LiftInstIntoBlockImpl (ctx=..., block=@0x7fffffffd730: 0x9a2b1e0) at /store/artem/git/mcsema/mcsema/cfgToLLVM/x86Instrs.cpp:138
#13 0x00000000005ad51a in LiftInstIntoBlock (ctx=..., block=@0x7fffffffd730: 0x9a2b1e0, doAnnotation=true) at /store/artem/git/mcsema/mcsema/cfgToLLVM/raiseX86.cpp:407
#14 0x00000000005ad25e in LiftBlockIntoFunction (ctx=...) at /store/artem/git/mcsema/mcsema/cfgToLLVM/raiseX86.cpp:440
#15 0x00000000005acec9 in InsertFunctionIntoModule (mod=0x72093b0, func=0x586e550, M=0x16d2060) at /store/artem/git/mcsema/mcsema/cfgToLLVM/raiseX86.cpp:509
#16 0x00000000005ac814 in LiftFunctionsIntoModule (natMod=0x72093b0, M=0x16d2060) at /store/artem/git/mcsema/mcsema/cfgToLLVM/raiseX86.cpp:937
#17 0x00000000005aaee4 in LiftCodeIntoModule (natMod=0x72093b0, M=0x16d2060) at /store/artem/git/mcsema/mcsema/cfgToLLVM/raiseX86.cpp:952
#18 0x00000000005544a0 in main (argc=11, argv=0x7fffffffdec8) at /store/artem/git/mcsema/mcsema/Lift.cpp:124
```
This tells us the problem is in the `MOV32mi` instruction. That instruction writes a 32-bit value to memory, so the core issue is probably a 32-bit pointer and a 64-bit value mismatch, because the architecture for this specific program is `amd64`. Looking at the code at line 643 we see:
640 data_v = IMM_AS_DATA_REF(block, natM, ip);
641 }
642
643 doMIMovV<32>(ip, block, ADDR_NOREF(0), data_v);
```c++
640 data_v = IMM_AS_DATA_REF(block, natM, ip);
641 }
642
643 doMIMovV<32>(ip, block, ADDR_NOREF(0), data_v);
```
The `IMM_AS_DATA_REF` function returns an architecture sized pointer, which in this case would be 64-bit. The MOV itself will be to a 32-bit value. Problem found: we are trying to put an `i64` into an `i32*` pointer! The solution is to replace `IMM_AS_DATA_REF` with `IMM_AS_DATA_REF<32>`, which will return a 32-bit value.
# Errors rebuilding binaries from Bitcode
## Errors rebuilding binaries from Bitcode
## LLVM ERROR: expected relocatable expression
### LLVM ERROR: expected relocatable expression
You are trying to recompile bitcode into a new binary, but clang crashes with the following error:
@@ -188,21 +211,25 @@ You are trying to recompile bitcode into a new binary, but clang crashes with th
**Technical Background:** This is most likely a sign you mismatched the architecture between CFG recovery and translation.
If you are sure you didn't, this is a combination of CFG recovery problem and clang bug. Mcsema is emitting bitcode that takes the lower 32-bits of a 64-bit function pointer, and puts it in a data section. Clang does not want to do this. This may be a CFG recovery bug if somehow only the lower 32-bits were deteted as a function pointer. Unfortunately, some compilers emit just the lower 32-bits of a pointer into the data section. Mcsema has no choice but to deal witht it as best it can.
If you are sure you didn't, this is a combination of CFG recovery problem and clang bug. McSema is emitting bitcode that takes the lower 32-bits of a 64-bit function pointer, and puts it in a data section. Clang does not want to do this. This may be a CFG recovery bug if somehow only the lower 32-bits were detected as a function pointer. Unfortunately, some compilers emit just the lower 32-bits of a pointer into the data section. McSema has no choice but to deal with it as best it can.
**Possible Fixes:** Make sure you use the correct architecture (x86, amd64) for both the translation and CFG recovery.
If that fails, disassemble the bitcode with `llvm-dis`. Look for lines similar to the ones below. Specifically, you are looking for `ptrtoint` that converts a pointer to a 32-bit integer.
@data_600e00 = internal global %3 <{ i32 ptrtoint (void ()* @callback_sub_400790 to i32), [4 x i8] zeroinitializer }>, align 64
@data_600e08 = internal global %4 <{ i32 ptrtoint (void ()* @callback_sub_400770 to i32), [4 x i8] zeroinitializer }>, align 64
```llvm
@data_600e00 = internal global %3 <{ i32 ptrtoint (void ()* @callback_sub_400790 to i32), [4 x i8] zeroinitializer }>, align 64
@data_600e08 = internal global %4 <{ i32 ptrtoint (void ()* @callback_sub_400770 to i32), [4 x i8] zeroinitializer }>, align 64
```
If these data sections are not used, delete them and recompile the bitcode with `llvm-as`. Alternatively, remove the `zeroinitializer` padding and expand the `ptrtoint` to 64 bits. This will also require editing the structure types of the data sections:
%3 = type <{ i64 }>
%4 = type <{ i64 }>
...
@data_600e00 = internal global %3 <{ i64 ptrtoint (void ()* @callback_sub_400790 to i64) }>, align 64
@data_600e08 = internal global %4 <{ i64 ptrtoint (void ()* @callback_sub_400770 to i64) }>, align 64
```llvm
%3 = type <{ i64 }>
%4 = type <{ i64 }>
...
@data_600e00 = internal global %3 <{ i64 ptrtoint (void ()* @callback_sub_400790 to i64) }>, align 64
@data_600e08 = internal global %4 <{ i64 ptrtoint (void ()* @callback_sub_400770 to i64) }>, align 64
```
Neither of these fixes is guaranteed to work.
docs/DEFFileGeneration.md
+5 -5
View File
@@ -2,11 +2,11 @@
## Motivation
When performing CFG recovery, if mcsema encounters an external function whose calling convention or agrument count is unknown, it will fail. The workaround is to provide mcsema with this information externally via a defs file. With one or two external functions, this is a simple enough task, but with more it becomes tedious and error prone work.
When performing CFG recovery, if mcsema encounters an external function whose calling convention or argument count is unknown, it will fail. The workaround is to provide McSema with this information externally via a defs file. With one or two external functions, this is a simple enough task, but with more it becomes tedious and error prone work.
## Use case
This DEF file generation script is intended to be used when header files that declare the external functions are available. For instance, if you're attempting to lift a binary that relies upon a shared library, the external calls to the library would result in errors. However, you should have (or be able to acquire) the header files for that library, which wll permit the generation of the needed defs file.
This DEF file generation script is intended to be used when header files that declare the external functions are available. For instance, if you're attempting to lift a binary that relies upon a shared library, the external calls to the library would result in errors. However, you should have (or be able to acquire) the header files for that library, which will permit the generation of the needed defs file.
## Usage
@@ -20,12 +20,12 @@ Output is printed to stdout, so especially when processing many files at once, p
* python
* git
* access to github (for the first invocation)
* access to GitHub (for the first invocation)
## Example usage
```
user@host:tools/ $ python generate_def_file.py /usr/include/strings.h
```shell
user@host:tools/ $ python generate_def_file.py /usr/include/strings.h
bcmp 3 C N
bcopy 3 C N
bzero 2 C N
docs/DebuggingTips.md
+43 -43
View File
@@ -1,8 +1,8 @@
# Things have gone wrong, now what?
# Things have gone wrong, now what
This document describes some approaches to debugging the lifted bitcode produced by McSema on Linux. This document does not describe [how to add missing instructions](AddAnInstruction.md), or how to [resolve common errors](CommonErrors.md).
#### Terminology
## Terminology
This document uses the term "original binary" to mean the binary that was disassembled using `mcsema-disass`. This document uses "native code" to mean machine code in the original binary.
@@ -14,7 +14,7 @@ There are a few helpers for using GDB to debug lifted bitcode. Some setup before
First, open up `~/.gdbinit`, add the below code to that file, and save it.
```
```gdb
set history filename ~/.gdb_history
set history save
set history size 4096
@@ -36,15 +36,15 @@ Finally, `set language c++` tells GDB that the source code that you will be debu
There are opportunities and drawbacks to debugging lifted code.
**Drawbacks**
### Drawbacks
- Lifted code is more verbose than native code. An instruction from the original binary may be represented by tens of instructions in the lifted code.
- Lifted code does not contain the useful debug information that may allow one to view the source code associated with some bits of machine code.
* Lifted code is more verbose than native code. An instruction from the original binary may be represented by tens of instructions in the lifted code.
* Lifted code does not contain the useful debug information that may allow one to view the source code associated with some bits of machine code.
**Opportunities**
### Opportunities
- The [`RegState`](/mcsema/Arch/X86/Runtime/State.h) is stored in memory. The advantage to this is that one can set data breakpoints (hardware watchpoints) on individual registers in the state structure. This is incredibly useful if you have a [time-travelling debugger](http://undo.io/products/undodb/).
- Lifted bitcode can be instrumented using the LLVM toolchain. Some useful-for-debugging instrumentations come built-in to `mcsema-lift`.
* The [`RegState`](/mcsema/Arch/X86/Runtime/State.h) is stored in memory. The advantage to this is that one can set data breakpoints (hardware watchpoints) on individual registers in the state structure. This is incredibly useful if you have a [time-travelling debugger](http://undo.io/products/undodb/).
* Lifted bitcode can be instrumented using the LLVM toolchain. Some useful-for-debugging instrumentations come built-in to `mcsema-lift`.
#### Built-in instrumentation
@@ -56,9 +56,9 @@ One of the aforementioned drawbacks when trying to debug lifted code is that it
That is why there is the `-add-breakpoints` option. The idea is that, just as you can say `b *0x402a00` to set a breakpoint on an instruction in the original binary, you can also do `b breakpoint_402a00` to set a breakpoint on the location of an "original instruction", but in the lifted binary.
Native code | Lifted code with breakpoint functions
:----------:|:-------------------------:
![Native code](images/breakpoint_orig_code.png) | ![Lifted code](images/breakpoint.png)
| Native code | Lifted code with breakpoint functions |
| :--------------------------------------: | :-----------------------------------: |
| ![Native code](images/breakpoint_orig_code.png) | ![Lifted code](images/breakpoint.png) |
On the left we see two instructions from the native code. On the right, we see the lifted code associated with the first and part of the second native instruction. Interspersed between the two are the `breakpoint_` functions. These breakpoint functions are "serializing" instructions. We can be certain that the `RegState` structure is in a consistent state at each call to a `breakpoint_` function. That is, the contents of the `RegState` struct at `breakpoint_402a00` in the lifted code should mostly match the native register state at `0x402a00` in the original binary.
@@ -68,16 +68,16 @@ Here's an example of using the `print-reg-state-64` GDB command in conjunction w
First, we set a breakpoint at `breakpoint_402a00` in `/tmp/ls_lifted`. The lifted state at this point will correspond to the native state at `0x402a00` in `/bin/ls`.
```
```gdb
(gdb) b breakpoint_402a00
Breakpoint 1 at 0x4f9160
```
Second, run the program until the breakpoint is hit.
```
```gdb
(gdb) r
Starting program: /tmp/ls_lifted
Starting program: /tmp/ls_lifted
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Breakpoint 1, 0x00000000004f9160 in breakpoint_402a00 ()
@@ -85,7 +85,7 @@ Breakpoint 1, 0x00000000004f9160 in breakpoint_402a00 ()
We're now stopped at the `breakpoint_402a00` and can inspect the values of the `RegState` structure.
```
```gdb
(gdb) print-reg-state-64
emulated native
rip 0x0000000000402a00 0x00000000004f9160
@@ -105,16 +105,16 @@ r12 0x0000000000402670 0x0000000000402670
r13 0x00007fffffffdda0 0xde7accccde7acccc
r14 0x0000000000000000 0x0000000000000000
r15 0x0000000000000000 0x00007ffff7ebb810
(gdb)
(gdb)
```
Here, lets go see what things look like in the original `/bin/ls` program at the same place.
```
```gdb
(gdb) b *0x402a00
Breakpoint 1 at 0x402a00
(gdb) r
Starting program: /bin/ls
Starting program: /bin/ls
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
@@ -143,7 +143,7 @@ Things won't perfectly match up, especially near the beginning of the execution
This is a nifty way of visually seeing if things match up with your expectations. Though, not everything is printed out at this point. For example, if you're observing that lifted execution goes one way, while native execution goes the other, then you may want to inspect the `EFLAGS` register to see what's going on. There's a command for that too!
```
```gdb
(gdb) print-flags-64
eflags [PF AF ZF ]
```
@@ -156,21 +156,22 @@ The below example is based on using UndoDB. This example is contrived, but shows
Let's say we're in `sub_40eca0` and we want to know who initializes the value of first argument, stored in register `rdi`. We can see from the picture on the right that the function `sub_40eca0` is called from many places.
Function `sub_40eca0` using `rdi` | Callers of `sub_40eca0`
---------------------------------:|:------------------------
![Function using RDI](images/rdi_in_called_function.png) | ![Function using RDI](images/sources_of_rdi.png)
| Function `sub_40eca0` using `rdi` | Callers of `sub_40eca0` |
| ---------------------------------------: | :--------------------------------------- |
| ![Function using RDI](images/rdi_in_called_function.png) | ![Function using RDI](images/sources_of_rdi.png) |
Now we get to take advantage of one of our opportunities: we can set data breakpoints on registers in the `RegState` structure! Let's go find the source of `rdi` given a call to `sub_40eca0`.
First, we get the address of `rdi` within the `RegState` structure.
```
```gdb
(gdb) addr-of-rdi
&(RegState::rdi) = 0x00007ffff7ebb840
```
Then, we set a hardware watchpoint, and reverse-execute.
```
```gdb
(undodb-gdb) reverse-continue
Continuing.
Hardware watchpoint 1: *0x00007f0334483840
@@ -233,7 +234,7 @@ So we have a trace of the lifted program, and we'd like to discover where it div
#### Collecting native traces with PIN
One way to drill down on divergences is by comparing the lifted trace with a ground truth: a trace recorded from a native program execution. We can do this with the [`regtrace` PIN tool](/mcsema/tools/regtrace/README.md). The first step is to [download and install PIN](https://software.intel.com/en-us/articles/pintool-downloads) before we can use the PIN tool.
One way to drill down on divergences is by comparing the lifted trace with a ground truth: a trace recorded from a native program execution. We can do this with the [`regtrace` PIN tool](/mcsema/tools/regtrace/README.md). The first step is to [download and install PIN](https://software.intel.com/en-us/articles/pintool-downloads) before we can use the PIN tool.
The next step to using the PIN tool is to build it!
@@ -345,7 +346,7 @@ So, the value of `RDI` should be `0x25` but instead it is `0x0`. We can step bac
The value of `RDI` comes from `R15`, but unfortunately there's a lot of predecessors. We can deal with this by working backward through our trace, figuring out the location of the last assignment to `R15`. I'm going to cheat and use reverse execution.
```shell
```gdb
(undodb-gdb) addr-of-r15
&(RegState::r15) = 0x00007f8d4ff1d890
(undodb-gdb) watch *0x00007f8d4ff1d890
@@ -367,13 +368,13 @@ rip 0x000000000040b710 0x00000000004c4c26
0x7f8d4ff1d940: 0 0 0 0
```
This tells us that the value of `R15` was changed at instruction `0x40b710`, shown below.
This tells us that the value of `R15` was changed at instruction `0x40b710`, shown below.
![Source of r15](images/source_of_r15.png)
The `cvttss2si` instruction is pretty complicated. Let's see what should have happened in the native execution.
```shell
```gdb
(undodb-gdb) b *0x40b710
Breakpoint 2 at 0x40b710
(undodb-gdb) reverse-continue
@@ -390,7 +391,7 @@ $2 = {v4_float = {37.5, 0, 0, 0}, ...}
Looks like we can see a difference in the values of `XMM0` compared. In the lifted trace, the first float is a `0`, but in the native trace, the value is `37.5`, which converted to an integer is `0x25`. We're going to need to work backward again.
```shell
```gdb
(undodb-gdb) addr-of-xmm0-64
&(RegState::xmm0) = 0x00007f8d4ff1d940
(undodb-gdb) watch *0x00007f8d4ff1d940
@@ -412,7 +413,7 @@ rip 0x0000000000402190 0x0000000000514ea6
Oof. We've found ourselves in one of the attach/detach routines (`__mcsema_attach_ret`) for transitioning between lifted code and native code. These routines are tricky.
When we see an `__mcsema_attach_ret`, what it's really meaning is that at some earlier point, lifted code called into some native library code. This lifted-to-native transition happens via the `__mcsema_detach_call` or `__mcsema_detach_call_value` functions. Eventually that native code needs to transition back to lifted code. This is achieved by setting up a special return address on the stack, `__mcsema_attach_ret`. When native code returns, it returns to `__mcsema_attach_ret`, which marshals native register state back into the `RegState` structure, and continues on in lifted code.
When we see an `__mcsema_attach_ret`, what it's really meaning is that at some earlier point, lifted code called into some native library code. This lifted-to-native transition happens via the `__mcsema_detach_call` or `__mcsema_detach_call_value` functions. Eventually that native code needs to transition back to lifted code. This is achieved by setting up a special return address on the stack, `__mcsema_attach_ret`. When native code returns, it returns to `__mcsema_attach_ret`, which marshals native register state back into the `RegState` structure, and continues on in lifted code.
Alright, lets step back and attack this problem from the native side. Looking back at our control-flow graph, we can see that `XMM0` is modified by a `divss` instruction.
@@ -420,7 +421,7 @@ Alright, lets step back and attack this problem from the native side. Looking ba
Let's confirm that the `divss` instruction at `0x40b6d6` is really the source of the `37.5` number.
```shell
```gdb
(undodb-gdb) b *0x40b6d6
Breakpoint 3 at 0x40b6d6
(undodb-gdb) reverse-continue
@@ -437,7 +438,7 @@ $7 = 37.499999437500009
Yup, it is. Our reverse-execution based on a watchpoint on `&(RegState::XMM0)` led us astray. Lets go back to the breakpoint approach, and see what things look like at `breakpoint_40b6d6`.
```shell
```gdb
(undodb-gdb) c
Continuing.
@@ -468,7 +469,7 @@ So this is sort of encouraging. `XMM1` contains the correct value of `0.80000001
OK we're in business. The `pxor xmm0, xmm0` clears out the `XMM0` register, then the `cvtsi2ss` converts `R15` into a `float`, and stores it into `XMM0`. Lets step back to the `cvtsi2ss` and compare lifted to native.
```shell
```gdb
(undodb-gdb) b breakpoint_40b6d1
Breakpoint 6 at 0x50f6b0
(undodb-gdb) reverse-continue
@@ -483,7 +484,7 @@ r15 0x000000000000001e 0xde7accccde7acccc
And in the native code.
```shell
```gdb
(undodb-gdb) b *0x40b6d1
Breakpoint 4 at 0x40b6d1
(undodb-gdb) reverse-continue
@@ -506,7 +507,7 @@ So, it will take a signed integer in `R15`, convert it into a `float`, and store
What we're seeing here is that `R15` is converted into a `double`, then storing that to the low 8 bytes of `XMM0`, then clearing out the high 8 bytes of `XMM0`. This doesn't sound right! Let's confirm that this is indeed what is happening.
```shell
```gdb
(undodb-gdb) b *0x4c4879
Breakpoint 7 at 0x4c4879
(undodb-gdb) c
@@ -524,7 +525,7 @@ Alright, we have confirmation, the lifted code for `cvtsi2ss` is definitely doin
First, we'll figure out what LLVM opcode implements this instruction.
```shell
pag@sloth:~/Code/mcsema/build$ echo "cvtsi2ss xmm0, r15" | llvm-mc-3.8 -x86-asm-syntax=intel -show-inst
pag@sloth:~/Code/mcsema/build$ echo "cvtsi2ss xmm0, r15" | llvm-mc-3.8 -x86-asm-syntax=intel -show-inst
.text
cvtsi2ssq %r15, %xmm0 # <MCInst #670 CVTSI2SS64rr
# <MCOperand Reg:126>
@@ -549,14 +550,14 @@ We can also disassemble (using `llvm-dis-3.8`) the lifted bitcode file and look
What we are seeing is that the semantics function `translate_CVTSI2SS64rr` is using `64` for target floating point width when calling `doCVTSI2SrV`. The `doCVTSI2SrV` function
1. Calls the `INT_TO_FP_TO_INT<64>` helper function with the value of `R15`.
2. `INT_TO_FP_TO_INT<64>` converts `R15` to a `width`-sized floating point number (in our case, a 64-bit float, i.e. a `double`).
3. `INT_TO_FP_TO_INT<64>` reinterprets that value as an integer (without a format change).
4. `doCVTSI2SrV` writes that 64-bit integer (containing a float) to `XMM0`, which zero extends the 64-bit integer into a 128-bit integer.
1.Calls the `INT_TO_FP_TO_INT<64>` helper function with the value of `R15`.
2.`INT_TO_FP_TO_INT<64>` converts `R15` to a `width`-sized floating point number (in our case, a 64-bit float, i.e. a `double`).
3.`INT_TO_FP_TO_INT<64>` reinterprets that value as an integer (without a format change).
4.`doCVTSI2SrV` writes that 64-bit integer (containing a float) to `XMM0`, which zero extends the 64-bit integer into a 128-bit integer.
Clearly we should have called `doCVTSI2SrV<32>` instead of `doCVTSI2SrV<64>`, so that `INT_TO_FP_TO_INT<width=32>` would do a narrower conversion.
Let's make this change, rebuild and install mcsema, re-run `mcsema-lift`, compile the new bitcode, and hope for the best. Here's what the new bitcode looks like after the fix:
Let's make this change, rebuild and install McSema, re-run `mcsema-lift`, compile the new bitcode, and hope for the best. Here's what the new bitcode looks like after the fix:
```llvm
call void @breakpoint_40b6d1(%RegState* %0), !mcsema_real_eip !14434
@@ -568,7 +569,6 @@ Let's make this change, rebuild and install mcsema, re-run `mcsema-lift`, compil
br label %block_40b6d6, !mcsema_real_eip !14435
```
#### Closing comments
This document presented a few of the helpful commands provided for debugging, and then walked through the diagnosis and fixing of a specific bug that was present in McSema at commit [409abe3d31a7b0d09b1fee9c60e1d190de39cced](https://github.com/trailofbits/mcsema/commit/409abe3d31a7b0d09b1fee9c60e1d190de39cced).
docs/IDA.md
+15 -13
View File
@@ -1,24 +1,26 @@
IDA Pro Setup for McSema
========================
# IDA Pro Setup for McSema
## Linux Instructions
# Linux Instructions
These instructions describe the prerequisites for using IDA Pro for CFG Recovery under Linux.
## Install IDA Pro
### Install IDA Pro
When given the option, select to use bundled Python. Using the system Python may work, as long as the IDAPython plugin still runs and `python-protobuf` will install.
## Install helper libraries
### Install helper libraries
IDA does not install some of the libraries it needs to run. Lets help it out, and also install Google protocol buffer support for Python.
sudo dpkg --add-architecture x86
sudo apt-get update
sudo apt-get install libncurses5:i386 zlib1g:i386
sudo ln -s /lib/i386-linux-gnu/libncurses.so.5 /lib/i386-linux-gnu/libcurses.so
sudo apt-get install python-protobuf
sudo updatedb
```shell
sudo dpkg --add-architecture x86
sudo apt-get update
sudo apt-get install libncurses5:i386 zlib1g:i386
sudo ln -s /lib/i386-linux-gnu/libncurses.so.5 /lib/i386-linux-gnu/libcurses.so
sudo apt-get install python-protobuf
sudo updatedb
```
## Accept the License Agreement
### Accept the License Agreement
Run IDA as the user you will be using for McSema work and accept the license agreement
Run IDA as the user you will be using for McSema work and accept the license agreement.
docs/LifeOfAnInstruction.md
+5 -5
View File
@@ -2,7 +2,7 @@
This document describes how machine code instructions are lifted into LLVM bitcode. It should provide a detailed, inside-out overview of how McSema performs binary translation. This document omits some important steps, so consulting the [Navigating the code](NavigatingTheCode.md) document will be helpful.
### Running example
## Running example
This document will use the instructions in the following basic block as a running example.
@@ -52,7 +52,7 @@ Module {
} ];
} ];
...
}
}
```
## Step 2: Lifting to bitcode
@@ -74,7 +74,7 @@ block_804b7a3:
%7 = inttoptr i32 %6 to i32*; Cast ESP - 4 into a pointer
store i32 %4, i32* %7 ; Store EBX into [ESP - 4]
store i32 %6, i32* %3 ; ESP = ESP - 4 (complete the PUSH).
%8 = add i32 %5, 4 ; Compute ESP + 8, (pre-decrement ESP + 4)
%8 = add i32 %5, 4 ; Compute ESP + 8, (pre-decrement ESP + 4)
%9 = inttoptr i32 %8 to i32*; Cast ESP + 8 into a pointer
%10 = load i32, i32* %9 ; Load [ESP + 8]
store i32 %10, i32* %2 ; Store [ESP + 8] into EBX
@@ -84,7 +84,7 @@ block_804b7a3:
store i32 128, i32* %12 ; Set the hyper call vector to 0x80
%13 = getelementptr inbounds %struct.State, %struct.State* %state2, i32 0, i32 0, i32 0
store i32 4, i32* %13 ; Set the hyper call number to kX86IntN
; Call the Remill `__remill_async_hyper_call` intrinsic function to
; model the system effects of the `int 0x80` interrupt instruction.
%14 = tail call %struct.Memory* @__remill_async_hyper_call(%struct.State* nonnull %state2, i32 %11, %struct.Memory* %memory1)
@@ -95,6 +95,6 @@ block_804b7a3:
The produced LLVM functions have a predictable structure. They are named according to the `ea` (effective address, or entry address) field in the `Function` message. The `ea` of our example function is `0x804b7a3`, and so the lifted function is named `sub_804b7a3`.
All lifted functions accept three arguments: a pointer to a [`State`](https://github.com/trailofbits/remill/blob/master/remill/Arch/X86/Runtime/State.h#L520) structure, the current program counter, and an opaque pointer to a `Memory` structure. The lifted bitcode has a predictable structure. It emulates reads and writes of machine registers by using `load`s and `store`s to the `State` structure. It also emulates reads and writes to memory via `load`s and `store`s. To emulate memory differently, pass the `--keep_memops` option to `mcsema-lift-M.m`.
All lifted functions accept three arguments: a pointer to a [`State`](https://github.com/trailofbits/remill/blob/master/remill/Arch/X86/Runtime/State.h#L520) structure, the current program counter, and an opaque pointer to a `Memory` structure. The lifted bitcode has a predictable structure. It emulates reads and writes of machine registers by using `load`s and `store`s to the `State` structure. It also emulates reads and writes to memory via `load`s and `store`s. To emulate memory differently, pass the `--keep_memops` option to `mcsema-lift-M.m`.
In this case, we see the use of some Remill intrinsic functions, but in practice those don't show up, because `int 0x80`s and their ilk are typically found in libraries that wrap system calls, and not directly in the binaries themselves.
docs/Limitations.md
+7 -9
View File
@@ -1,6 +1,6 @@
# Limitations
# Limitations
The McSema toolset currently has some limitations which limit what programs it is able to translate to LLVM.
The McSema toolset currently has some limitations which limit what programs it is able to translate to LLVM.
## Building and Running
@@ -18,7 +18,7 @@ McSema's CFG recovery depends on IDA Pro being accurate. Sometimes its not. We a
### Self-modifying code
The translator is not designed to accommodate self-modifying code directly. An approach is possible to model its effects, where the modifications are described in a native control flow graph as branches to a new basic block, however this could be difficult to construct.
The translator is not designed to accommodate self-modifying code directly. An approach is possible to model its effects, where the modifications are described in a native control flow graph as branches to a new basic block, however this could be difficult to construct.
### Raw Binaries
@@ -42,7 +42,7 @@ The most conservative approach will greatly increase code size and reduce speed.
### External Function ABIs
We assume that external function ABIs are the default for the platform and processor combination of the target. While the ABI of internal functions inside a program does not matter, the ABI of external functions has to conform to the specification for the target platform.
We assume that external function ABIs are the default for the platform and processor combination of the target. While the ABI of internal functions inside a program does not matter, the ABI of external functions has to conform to the specification for the target platform.
External functions that take floating point arguments, or return floating point values are not yet supported, but there is no technical limitation peventing it; the glue code enabling this has just not been written. Pull requests are welcome.
@@ -50,16 +50,14 @@ External functions that take floating point arguments, or return floating point
Appliations that use the x87 FPU should run and report similar results, although sometimes at a lower precision. The x87 FPU is modelled using operations on `double` (64-bit floating point numbers), whereas the real hardware operates on `long double`s (80-bit floating point). Operating on `double`s was a deliberate design decision to improve the portability of the lifted bitcode.
#### Precision and Rounding Control
#### Precision and Rounding Control
Lifted bitcode does not explicitly represent the current FPU precision or rounding modes. Instead, it queries/controls the current FPU precision via standardized libc functions.
#### Last Instruction Pointer
#### Last Instruction Pointer
The last instruction pointer of the FPU is not modelled. Anything that reads this field and performs logic on it is self-referential code that we currently do not support.
### Memory Segmentation
### Memory Segmentation
McSema supports common segment prefixes that are used with TLS code, such as `fs` and `gs`. Setting the value of a segment register is not directly supported, but instead handled via `__remill_sync_hyper_call`.
docs/McsemaWalkthrough.md → docs/McSemaWalkthrough.md
+115 -88
View File
@@ -1,23 +1,25 @@
# Mcsema Walkthrough
# McSema Walkthrough
In this guide we'll show how to translate a Linux binary into LLVM bitcode, and then rebuild the resulting bitcode into a new, functional binary.
The binary we'll be translating is `xz`, a [compression and decompression suite that uses LZMA](https://en.wikipedia.org/wiki/XZ_Utils). The utility is common on Linux systems, and we'll be using the version of /bin/xz that ships with Ubuntu 16.04 amd64.
The binary we'll be translating is `xz`, a [compression and decompression suite that uses LZMA](https://en.wikipedia.org/wiki/XZ_Utils). The utility is common on Linux systems, and we'll be using the version of /bin/xz that ships with Ubuntu 16.04 amd64.
Here is the exact version and hash of our test binary:
$ ./xz -V
xz (XZ Utils) 5.1.0alpha
liblzma 5.1.0alpha
$ sha256sum ./xz
c3512fe134c78d7d734c6607a379b5f0d65276e8953a0a985e3e688356303223 ./xz
```shell
$ ./xz -V
xz (XZ Utils) 5.1.0alpha
liblzma 5.1.0alpha
$ sha256sum ./xz
c3512fe134c78d7d734c6607a379b5f0d65276e8953a0a985e3e688356303223 ./xz
```
This binary was chosen since it has complex features, it was easy to verify that the core functionality works, and at the time of writing, it demonstrated some common lifting failures and how to work around them.
## Assumptions
This guide assumes that you are working on a Linux system, have already built and installed a working version of mcsema, and that `mcsema-lift` and `mcsema-disass` are in your execution path. We assume that the mcsema repository was cloned into `$MCSEMA_DIR`.
This guide assumes that you are working on a Linux system, have already built and installed a working version of McSema, and that `mcsema-lift` and `mcsema-disass` are in your execution path. We assume that the McSema repository was cloned into `$MCSEMA_DIR`.
The guide also assumes that you have working version of IDA Pro, which is required for control flow recovery.
@@ -29,14 +31,17 @@ The first step in translation is to identify all the instructions, functions, an
First, let's place a copy of the binary in our working directory:
cd ~/mcsema
cp /usr/bin/xz ./
```shell
cd ~/mcsema
cp /usr/bin/xz ./
```
Next, we'll use `mcsema-disass`, the CFG recovery portion of mcsema, to recover `xz`'s control flow.
mcsema-disass --disassembler ~/ida-6.9/idal64 --os linux --arch amd64 --output xz.cfg --binary xz --entrypoint main --log_file xz.log
```shell
mcsema-disass --disassembler ~/ida-6.9/idal64 --os linux --arch amd64 --output xz.cfg --binary xz --entrypoint main --log_file xz.log
```
Let's walk through each option:
* `--disassembler ~/ida-6.9/idal64`: This is a path to the IDA Pro executable which will do the bulk of the disassembly work.
@@ -53,59 +58,69 @@ This section documents how to fix a common CFG recovery problem: undefined exter
The previous command may have failed (as in this snippet). If it did, read on. If not, skip this section and move on to Translation To Bitcode.
$ mcsema-disass --disassembler ~/ida-6.9/idal64 --os linux --arch amd64 --output xz.cfg --binary xz --entrypoint main --log_file xz.log
Generated an invalid (zero-sized) CFG. Please use the --log_file option to see an error log.
```shell
$ mcsema-disass --disassembler ~/ida-6.9/idal64 --os linux --arch amd64 --output xz.cfg --binary xz --entrypoint main --log_file xz.log
Generated an invalid (zero-sized) CFG. Please use the --log_file option to see an error log.
```
Let's take a look at the log, like the message suggests:
$ tail xz.log
File "/home/artem/.local/lib/python2.7/site-packages/mcsema_disass-0.0.1-py2.7.egg/mcsema_disass/ida/get_cfg.py", line 1693, in recoverFunction
recoverFunctionFromSet(M, F, blockset, new_eas)
File "/home/artem/.local/lib/python2.7/site-packages/mcsema_disass-0.0.1-py2.7.egg/mcsema_disass/ida/get_cfg.py", line 1681, in recoverFunctionFromSet
I, endBlock = instructionHandler(M, B, head, new_eas)
File "/home/artem/.local/lib/python2.7/site-packages/mcsema_disass-0.0.1-py2.7.egg/mcsema_disass/ida/get_cfg.py", line 817, in instructionHandler
if doesNotReturn(fn):
File "/home/artem/.local/lib/python2.7/site-packages/mcsema_disass-0.0.1-py2.7.egg/mcsema_disass/ida/get_cfg.py", line 220, in doesNotReturn
raise Exception("Unknown external: " + fname)
Exception: Unknown external: __open_2
```shell
$ tail xz.log
File "/home/artem/.local/lib/python2.7/site-packages/mcsema_disass-0.0.1-py2.7.egg/mcsema_disass/ida/get_cfg.py", line 1693, in recoverFunction
recoverFunctionFromSet(M, F, blockset, new_eas)
File "/home/artem/.local/lib/python2.7/site-packages/mcsema_disass-0.0.1-py2.7.egg/mcsema_disass/ida/get_cfg.py", line 1681, in recoverFunctionFromSet
I, endBlock = instructionHandler(M, B, head, new_eas)
File "/home/artem/.local/lib/python2.7/site-packages/mcsema_disass-0.0.1-py2.7.egg/mcsema_disass/ida/get_cfg.py", line 817, in instructionHandler
if doesNotReturn(fn):
File "/home/artem/.local/lib/python2.7/site-packages/mcsema_disass-0.0.1-py2.7.egg/mcsema_disass/ida/get_cfg.py", line 220, in doesNotReturn
raise Exception("Unknown external: " + fname)
Exception: Unknown external: __open_2
```
This means the binary is trying to call an external function, but mcsema does not know what calling convention it is, or how many arguments to give it. After searching for `__open_2` we can see that it's a C library function that takes two arguments. We can tell mcsema about it via a custom external definitions file.
Create a new file in your working directory named `xz_defs.txt` with the following content:
__open_2 2 C N
This tells mcsema that the function is named `__open_2`, it takes `2` arguments, its calling convention is `C`aller cleanup, and the function returns (or is `N`ot noreturn, as mcsema sees it).
Now let's tell `mcsema-disass` about our new definitions file:
$ mcsema-disass --disassembler ~/ida-6.9/idal64 --os linux --arch amd64 --output xz.cfg --binary xz --entrypoint main --log_file xz.log --std-defs xz_defs.txt
Generated an invalid (zero-sized) CFG. Please use the --log_file option to see an error log.
```shell
$ mcsema-disass --disassembler ~/ida-6.9/idal64 --os linux --arch amd64 --output xz.cfg --binary xz --entrypoint main --log_file xz.log --std-defs xz_defs.txt
Generated an invalid (zero-sized) CFG. Please use the --log_file option to see an error log.
```
Oh no! It's still failing? Let's check the log again:
$ tail xz.log
File "/home/artem/.local/lib/python2.7/site-packages/mcsema_disass-0.0.1-py2.7.egg/mcsema_disass/ida/get_cfg.py", line 1693, in recoverFunction
recoverFunctionFromSet(M, F, blockset, new_eas)
File "/home/artem/.local/lib/python2.7/site-packages/mcsema_disass-0.0.1-py2.7.egg/mcsema_disass/ida/get_cfg.py", line 1681, in recoverFunctionFromSet
I, endBlock = instructionHandler(M, B, head, new_eas)
File "/home/artem/.local/lib/python2.7/site-packages/mcsema_disass-0.0.1-py2.7.egg/mcsema_disass/ida/get_cfg.py", line 817, in instructionHandler
```shell
$ tail xz.log
File "/home/artem/.local/lib/python2.7/site-packages/mcsema_disass-0.0.1-py2.7.egg/mcsema_disass/ida/get_cfg.py", line 1693, in recoverFunction
recoverFunctionFromSet(M, F, blockset, new_eas)
File "/home/artem/.local/lib/python2.7/site-packages/mcsema_disass-0.0.1-py2.7.egg/mcsema_disass/ida/get_cfg.py", line 1681, in recoverFunctionFromSet
I, endBlock = instructionHandler(M, B, head, new_eas)
File "/home/artem/.local/lib/python2.7/site-packages/mcsema_disass-0.0.1-py2.7.egg/mcsema_disass/ida/get_cfg.py", line 817, in instructionHandler
if doesNotReturn(fn):
File "/home/artem/.local/lib/python2.7/site-packages/mcsema_disass-0.0.1-py2.7.egg/mcsema_disass/ida/get_cfg.py", line 220, in doesNotReturn
raise Exception("Unknown external: " + fname)
Exception: Unknown external: __vfprintf_chk
File "/home/artem/.local/lib/python2.7/site-packages/mcsema_disass-0.0.1-py2.7.egg/mcsema_disass/ida/get_cfg.py", line 220, in doesNotReturn
raise Exception("Unknown external: " + fname)
Exception: Unknown external: __vfprintf_chk
```
The culprit is another missing external function. Let's add this one is as well. Our new `xz_defs.txt` should now look like:
__open_2 2 C N
__vfprintf_chk 4 C N
Finally, our command should succeed. We can verify that the CFG recovery completed by looking at the size of the generated CFG:
$ mcsema-disass --disassembler ~/ida-6.9/idal64 --os linux --arch amd64 --output xz.cfg --binary xz --entrypoint main --log_file xz.log --std-defs xz_defs.txt
$ ls -lh xz.cfg
-rw-rw-r-- 1 artem artem 212K Mar 8 14:54 xz.cfg
```shell
$ mcsema-disass --disassembler ~/ida-6.9/idal64 --os linux --arch amd64 --output xz.cfg --binary xz --entrypoint main --log_file xz.log --std-defs xz_defs.txt
$ ls -lh xz.cfg
-rw-rw-r-- 1 artem artem 212K Mar 8 14:54 xz.cfg
```
If header files are available that declare these external functions, these files can be automatically generated using the [DEF file generation script](DEFFileGeneration.md).
## Translation to Bitcode
@@ -114,11 +129,13 @@ Once we have the program's control flow information, we can translate it to LLVM
Here is the command to translate the CFG into bitcode:
mcsema-lift-4.0 --os linux --arch amd64 --cfg xz.cfg --output xz.bc
```shell
mcsema-lift-4.0 --os linux --arch amd64 --cfg xz.cfg --output xz.bc
```
Let's explore the options one by one:
* `--os linux`: The CFG came from a binary for the Linux operating system. Currently the valid options are `linux` or `windows`. This option is required for certain aspects of translation, like ABI compatibility for external functions, etc.
* `--os linux`: The CFG came from a binary for the Linux operating system. Currently the valid options are `linux` or `windows`. This option is required for certain aspects of translation, like ABI compatibility for external functions, etc.
* `--arch amd64`: Use instruction semantics for the `amd64` architecture. Valid options are `x86` and `x86_avx` (32-bit x86 semantics), `amd64` and `amd64_avx` (64-bit x86), and `aarch64` (64-bit ARMv8).
* `--cfg xz.cfg`: The input control flow graph to convert into bitcode.
* `--output xz.bc`: Where to write the bitcode. If the `--output` option is not specified, the bitcode will be written to stdout.
@@ -127,59 +144,69 @@ The `mcsema-lift-4.0` program may print out errors or warnings to `stderr`. Ofte
And there will be a generated bitcode file in the output location we specified:
$ ls -lh xz.bc
-rw-rw-r-- 1 artem artem 1.9M Mar 8 14:57 xz.bc
```shell
$ ls -lh xz.bc
-rw-rw-r-- 1 artem artem 1.9M Mar 8 14:57 xz.bc
```
## Building a New Binary
The new bitcode can be used for a variety of purposes ranging from informational analyses to hardening and transformation. Eventually, though, you may want to re-create a new, working binary. Here is how to do that.
First, you'll need McSema's runtime libraries that are generated during installation. These are located at the installation prefix of McSema, typically in `/usr/local` on Unix machines. Alternatively, these can be found in the Remill build directory, under `tools/mcsema/mcsema/Arch/X86/Runtime` (for X86). You will also need to link against any libraries that the original program was linked against. For this example, make sure that the liblzma-dev package is installed on your machine:
$ sudo apt-get install liblzma-dev
```shell
sudo apt-get install liblzma-dev
```
When Remill/McSema is installed, so should a prefixed version of clang for the build toolchain. For example, building Remill/McSema with the LLVM 4.0 toolchain should also install `remill-clang-4.0`. If not, you can always find the toolchain-specific version of the LLVM binaries and libraries in the Remill build directory, under `libraries/llvm/bin`.
Now, let's re-create a new `xz` binary and see it in action!
$ remill-clang-4.0 -rdynamic -O3 -o xz.new xz.bc /usr/local/lib/libmcsema_rt64-4.0.a -llzma
```shell
remill-clang-4.0 -rdynamic -O3 -o xz.new xz.bc /usr/local/lib/libmcsema_rt64-4.0.a -llzma
```
This is a fairly ordinary clang command line; the only thing of note is `/usr/local/lib/libmcsema_rt64-4.0.a`, which is the path to the aforementioned runtime libraries. The `libmcsema_rt64-4.0.a` is the library to use for 64-bit bitcode using the LLVM 4.0 toolchain, and `libmcsema_rt32-4.0.a` is the library to use for 32-bit bitcode using the LLVM 4.0 toolchain. On Windows, these libraries are named `mcsema_rt64-4.0.lib` and `mcsema_rt32-4.0.lib`, respectively.
We can verify that our new binary, `xz.new`, works, and compresses output that can be read by `unxz`:
$ echo "testing compression" | ./xz.new | unxz
testing compression
Command line arguments to `xz.new` also work:
$ ./xz.new --version
xz (XZ Utils) 5.1.0alpha
liblzma 5.1.0alpha
$ ./xz.new --help
Usage: ./xz.new [OPTION]... [FILE]...
Compress or decompress FILEs in the .xz format.
```shell
$ echo "testing compression" | ./xz.new | unxz
testing compression
```
-z, --compress force compression
-d, --decompress force decompression
-t, --test test compressed file integrity
-l, --list list information about .xz files
-k, --keep keep (don't delete) input files
-f, --force force overwrite of output file and (de)compress links
-c, --stdout write to standard output and don't delete input files
-0 ... -9 compression preset; default is 6; take compressor *and*
decompressor memory usage into account before using 7-9!
-e, --extreme try to improve compression ratio by using more CPU time;
does not affect decompressor memory requirements
-q, --quiet suppress warnings; specify twice to suppress errors too
-v, --verbose be verbose; specify twice for even more verbose
-h, --help display this short help and exit
-H, --long-help display the long help (lists also the advanced options)
-V, --version display the version number and exit
Command line arguments to `xz.new` also work:
With no FILE, or when FILE is -, read standard input.
```shell
$ ./xz.new --version
xz (XZ Utils) 5.1.0alpha
liblzma 5.1.0alpha
$ ./xz.new --help
Usage: ./xz.new [OPTION]... [FILE]...
Compress or decompress FILEs in the .xz format.
-z, --compress force compression
-d, --decompress force decompression
-t, --test test compressed file integrity
-l, --list list information about .xz files
-k, --keep keep (don't delete) input files
-f, --force force overwrite of output file and (de)compress links
-c, --stdout write to standard output and don't delete input files
-0 ... -9 compression preset; default is 6; take compressor *and*
decompressor memory usage into account before using 7-9!
-e, --extreme try to improve compression ratio by using more CPU time;
does not affect decompressor memory requirements
-q, --quiet suppress warnings; specify twice to suppress errors too
-v, --verbose be verbose; specify twice for even more verbose
-h, --help display this short help and exit
-H, --long-help display the long help (lists also the advanced options)
-V, --version display the version number and exit
With no FILE, or when FILE is -, read standard input.
Report bugs to <lasse.collin@tukaani.org> (in English or Finnish).
XZ Utils home page: <http://tukaani.org/xz/>
```
Report bugs to <lasse.collin@tukaani.org> (in English or Finnish).
XZ Utils home page: <http://tukaani.org/xz/>
That's it for the walkthrough. Please let us know if any of the steps fail or change so that we can update this document. Happy translating!
docs/NavigatingTheCode.md
+16 -16
View File
@@ -4,15 +4,15 @@ This document describes the structure of the McSema codebase, where to find thin
There are three high-level steps to using McSema:
1. [Disassembling a program binary and producing a CFG file](#disass)
2. [Lifting the CFG file into LLVM bitcode](#lift)
3. Compiling the LLVM bitcode into a runnable binary
1. [Disassembling a program binary and producing a CFG file](#disass)
2. [Lifting the CFG file into LLVM bitcode](#lift)
3. Compiling the LLVM bitcode into a runnable binary
## File Layout
First, let's familiarize ourselve with essentials of the file layout of McSema.
First, let's familiarize ourselves with essentials of the file layout of McSema.
```
```shell
┌── mcsema
│   ├── Arch
│   │   ├── ... : Architecture-neutral files
@@ -33,7 +33,7 @@ First, let's familiarize ourselve with essentials of the file layout of McSema.
│      └── CFG.proto : CFG file format description
└── tools
├── mcsema_disass : mcsema-disass front-end, makes CFG files
│   ├── binja :
│   ├── binja :
│   │   └── ... : Binary Ninja backend for mcsema-disass
│   ├── defs
│   │   ├── linux.txt : List of known external library functions on Linux
@@ -44,7 +44,7 @@ First, let's familiarize ourselve with essentials of the file layout of McSema.
│   └── __main__.py
├── mcsema_lift
│   └── Lift.cpp : mcsema-lift front-end, makes bitcode from CFG files
└── regtrace : Intel PIN tool for identifying divergences in lifted code
└── regtrace : Intel PIN tool for identifying divergences in lifted code
```
## <a id="disass"></a> Producing a CFG file
@@ -61,10 +61,10 @@ The front-end is responsible for invoking the backend and disassembly engine. Th
The most important high-level structures recorded in the CFG file are:
- `Function`: functions in the binary with concrete implementations. The `Function` message contains all basic blocks and instructions of the function. A common example of this would be a program's `main` function.
- `Segment`: Data stored in the program binary. This includes things like global variables and `static` storage duration-defined variables in C/C++ code.
- `ExternalFunction`: functions called but not defined by the program. A common example of this would be libc functions like `malloc`, `strlen`, etc.
- `ExternalVariable`: data referenced but not defined by the program. An example of this would be the `getopt` C library's `optind` variable. You can things of these being like `extern`-declared global variables.
- `Function`: functions in the binary with concrete implementations. The `Function` message contains all basic blocks and instructions of the function. A common example of this would be a program's `main` function.
- `Segment`: Data stored in the program binary. This includes things like global variables and `static` storage duration-defined variables in C/C++ code.
- `ExternalFunction`: functions called but not defined by the program. A common example of this would be libc functions like `malloc`, `strlen`, etc.
- `ExternalVariable`: data referenced but not defined by the program. An example of this would be the `getopt` C library's `optind` variable. You can things of these being like `extern`-declared global variables.
`mcsema-lift-M.m` (where `M` is the major LLVM version, and `m` is the minor LLVM version, e.g. `mcsema-lift-4.0`) has different ways of turning each of the above structures into LLVM bitcode.
@@ -72,10 +72,10 @@ The most important high-level structures recorded in the CFG file are:
The `mcsema-lift-M.m` command is used to lift CFG files to LLVM bitcode. The four most important arguments to `mcsema-lift` are:
1. `--os`: The operating system of the code being lifted. In practice, each binary format is specific to an operating system. ELF files are for Linux, Mach-O files for macOS, and DLL files for Windows. This is one of `linux`, `macos`, or `windows`.
2. `--arch`: The architecture of the code being lifted. This is one of `x86`, `x86_avx`, `amd64`, `amd64_avx`, or `aarch64`.
3. `--cfg`: The path for the CFG file produced by `mcsema-disass`.
4. `--output`: The path to the bitcode file to save/produce.
1.`--os`: The operating system of the code being lifted. In practice, each binary format is specific to an operating system. ELF files are for Linux, Mach-O files for macOS, and DLL files for Windows. This is one of `linux`, `macos`, or `windows`.
2.`--arch`: The architecture of the code being lifted. This is one of `x86`, `x86_avx`, `amd64`, `amd64_avx`, or `aarch64`.
3.`--cfg`: The path for the CFG file produced by `mcsema-disass`.
4.`--output`: The path to the bitcode file to save/produce.
The above arguments instruct the [lifter](/tools/mcsema_lift/Lift.cpp) on how to configure the bitcode file.
@@ -97,7 +97,7 @@ bool LiftCodeIntoModule(const NativeModule *cfg_module) {
// Segments are inserted after the lifted function declarations are added
// so that cross-references to lifted code are handled.
AddDataSegments(cfg_module);
// Lift the blocks of instructions into the declared functions.
if (!DefineLiftedFunctions(cfg_module)) {
return false;
docs/UsingLibFuzzer.md
+160 -144
View File
@@ -1,143 +1,150 @@
# Using libFuzzer with Mcsema
# Using libFuzzer with McSema
[libFuzzer](http://blog.llvm.org/2015/04/fuzz-all-clangs.html) is an LLVM-based coverage-guided fuzzing framework similar to AFL. It is simple to integrate coverage-guided fuzzing with libFuzzer: just define a special function, update some build flags, and you have instant coverage-guided fuzzing.
Since libFuzzer works at the LLVM level, can we apply libFuzzer to mcsema translated bitcode and use libFuzzer on binaries?
Since libFuzzer works at the LLVM level, can we apply libFuzzer to McSema translated bitcode and use libFuzzer on binaries?
It turns out the answer is yes!
However, the 'yes' comes with caveats. First, mcsema assembly stubs do things that normal programs should never do (like calculate dynamic return addresses, allocate new stacks, etc). This behavior can conflict with address sanitizer, a feature that libFuzzer uses. Second, mcsema's control flow recovery is frequently wrong on large programs. Since libFuzzer explores new code paths, it has a very high likelihood of triggering a path where control flow recovery is incorrect. This means that some of the bugs found may be artifacts of translation that are not present in the original program.
However, the 'yes' comes with caveats. First, McSema assembly stubs do things that normal programs should never do (like calculate dynamic return addresses, allocate new stacks, etc). This behavior can conflict with address sanitizer, a feature that libFuzzer uses. Second, McSema's control flow recovery is frequently wrong on large programs. Since libFuzzer explores new code paths, it has a very high likelihood of triggering a path where control flow recovery is incorrect. This means that some of the bugs found may be artifacts of translation that are not present in the original program.
We hope to improve both of these issues in the future. For now, let's take a look at a proof of concept for using libFuzzer on binary code!
The code we will be fuzzing is a [simple program](../tests/libFuzzer/fuzzme.cc) that tries to dereference user input once it reads the word 'fuzz':
The code we will be fuzzing is a [simple program](../tests/libFuzzer/fuzzme.cc) that tries to dereference user input once it reads the word 'fuzz'. The following is `fuzzme.cc`:
$ cat fuzzme.cc
#include <stdio.h>
#include <stdint.h>
int vulnerable(const char *arg) {
if(arg[0] == 'f') {
if(arg[1] == 'u') {
if(arg[2] == 'z') {
if(arg[3] == 'z') {
if(arg[4] == '\0') {
return 0;
} else {
// lets deref some user specified memory
int** z = (int**)((void*)(arg+4));
return **z;
}
```c++
#include <stdio.h>
#include <stdint.h>
int vulnerable(const char *arg) {
if(arg[0] == 'f') {
if(arg[1] == 'u') {
if(arg[2] == 'z') {
if(arg[3] == 'z') {
if(arg[4] == '\0') {
return 0;
} else {
// lets deref some user specified memory
int** z = (int**)((void*)(arg+4));
return **z;
}
}
}
}
return -1;
}
#ifdef SOURCE_FUZZ
extern "C" int LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size) {
vulnerable((const char *)Data);
return 0; // Non-zero return values are reserved for future use.
return -1;
}
#ifdef SOURCE_FUZZ
extern "C" int LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size) {
vulnerable((const char *)Data);
return 0; // Non-zero return values are reserved for future use.
}
#else
int main(int argc, const char *argv[]) {
if(argc != 2) {
printf("Usage:\n");
printf("%s: <text>\n", argv[0]);
return 1;
}
#else
int main(int argc, const char *argv[]) {
if(argc != 2) {
printf("Usage:\n");
printf("%s: <text>\n", argv[0]);
return 1;
}
if(0 == vulnerable(argv[1])) {
printf("Processed correctly\n");
} else {
printf("Bad input\n");
}
return 0;
if(0 == vulnerable(argv[1])) {
printf("Processed correctly\n");
} else {
printf("Bad input\n");
}
#endif
return 0;
}
#endif
```
## Prepare
libFuzzer is constantly improving; this guide will use an older version of libFuzzer that comes with LLVM 3.8 because that is the LLVM version used by mcsema.
libFuzzer is constantly improving; this guide will use an older version of libFuzzer that comes with LLVM 3.8 because that is the LLVM version used by McSema.
For this guide, we will assume that mcsema was built in `$MCSEMA_DIR`, and all operations take place in [`$MCSEMA_DIR/mcsema/tests/libFuzzer`](../tests/libFuzzer).
For this guide, we will assume that McSema was built in `$MCSEMA_DIR`, and all operations take place in [`$MCSEMA_DIR/mcsema/tests/libFuzzer`](../tests/libFuzzer).
## Preparing and verifying libFuzzer
First, run the `buildFuzzer.sh` script to generate libFuzzer objects. This should generate several files matching the pattern `Fuzzer*.o`:
$ ./buildFuzzer.sh
$ ls -1 Fuzzer*.o
FuzzerCrossOver.o
FuzzerDriver.o
FuzzerInterface.o
FuzzerIO.o
FuzzerLoop.o
FuzzerMain.o
FuzzerMutate.o
FuzzerSanitizerOptions.o
FuzzerSHA1.o
FuzzerTraceState.o
FuzzerUtil.o
```shell
$ ./buildFuzzer.sh
$ ls -1 Fuzzer*.o
FuzzerCrossOver.o
FuzzerDriver.o
FuzzerInterface.o
FuzzerIO.o
FuzzerLoop.o
FuzzerMain.o
FuzzerMutate.o
FuzzerSanitizerOptions.o
FuzzerSHA1.o
FuzzerTraceState.o
FuzzerUtil.o
```
Next, let's build a normal, non-mcsema version of our test to make sure libFuzzer works.
Next, let's build a normal, non-McSema version of our test to make sure libFuzzer works.
clang++-3.8 -DSOURCE_FUZZ -o fuzzme fuzzme.cc Fuzzer*.o -fsanitize=address -fsanitize-coverage=edge
```shell
clang++-3.8 -DSOURCE_FUZZ -o fuzzme fuzzme.cc Fuzzer*.o -fsanitize=address -fsanitize-coverage=edge
```
The `-fsanitize` and `-fsanitize-coverage` arguments are standard arguments to instrument the built code for libFuzzer. The `-DSOURCE_FUZZ` argument sets a define to make sure the required `LLVMFuzzerTestOneInput` is emitted by our test code. For more details, see the [libFuzzer documentation](http://releases.llvm.org/3.8.0/docs/LibFuzzer.html).
Now, let's run the source-based libFuzzer executable to make sure it works. Very quickly, you should see output similar to the following:
$ ./fuzzme
Seed: 1013786530
PreferSmall: 1
#0 READ units: 1 exec/s: 0
#1 INITED cov: 5 units: 1 exec/s: 0
#437 NEW cov: 8 units: 2 exec/s: 0 L: 64 MS: 0
#620609 NEW cov: 11 units: 3 exec/s: 620609 L: 64 MS: 3 ShuffleBytes-ChangeASCIIInt-ChangeByte-
#978396 NEW cov: 13 units: 4 exec/s: 978396 L: 58 MS: 5 ShuffleBytes-ShuffleBytes-CrossOver-ChangeASCIIInt-ChangeByte-
ASAN:DEADLYSIGNAL
=================================================================
==21762==ERROR: AddressSanitizer: SEGV on unknown address 0x000000007566 (pc 0x0000004eebf6 bp 0x7fff690a6580 sp 0x7fff690a64f0 T0)
#0 0x4eebf5 (/store/artem/git/mcsema/tests/libFuzzer/fuzzme+0x4eebf5)
#1 0x4eed2d (/store/artem/git/mcsema/tests/libFuzzer/fuzzme+0x4eed2d)
#2 0x4f464a (/store/artem/git/mcsema/tests/libFuzzer/fuzzme+0x4f464a)
#3 0x4f57de (/store/artem/git/mcsema/tests/libFuzzer/fuzzme+0x4f57de)
#4 0x4f5db7 (/store/artem/git/mcsema/tests/libFuzzer/fuzzme+0x4f5db7)
#5 0x4f0546 (/store/artem/git/mcsema/tests/libFuzzer/fuzzme+0x4f0546)
#6 0x4ef232 (/store/artem/git/mcsema/tests/libFuzzer/fuzzme+0x4ef232)
#7 0x4ef183 (/store/artem/git/mcsema/tests/libFuzzer/fuzzme+0x4ef183)
#8 0x7f7cb7ce482f (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
#9 0x41a3b8 (/store/artem/git/mcsema/tests/libFuzzer/fuzzme+0x41a3b8)
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (/store/artem/git/mcsema/tests/libFuzzer/fuzzme+0x4eebf5)
==21762==ABORTING
DEATH:
0x66,0x75,0x7a,0x7a,0x66,0x75,
fuzzfu
artifact_prefix='./'; Test unit written to ./crash-643f1d111ee9b982169c2278898ce3228c4c6e2e
Base64: ZnV6emZ1
```shell
$ ./fuzzme
Seed: 1013786530
PreferSmall: 1
#0 READ units: 1 exec/s: 0
#1 INITED cov: 5 units: 1 exec/s: 0
#437 NEW cov: 8 units: 2 exec/s: 0 L: 64 MS: 0
#620609 NEW cov: 11 units: 3 exec/s: 620609 L: 64 MS: 3 ShuffleBytes-ChangeASCIIInt-ChangeByte-
#978396 NEW cov: 13 units: 4 exec/s: 978396 L: 58 MS: 5 ShuffleBytes-ShuffleBytes-CrossOver-ChangeASCIIInt-ChangeByte-
ASAN:DEADLYSIGNAL
=================================================================
==21762==ERROR: AddressSanitizer: SEGV on unknown address 0x000000007566 (pc 0x0000004eebf6 bp 0x7fff690a6580 sp 0x7fff690a64f0 T0)
#0 0x4eebf5 (/store/artem/git/mcsema/tests/libFuzzer/fuzzme+0x4eebf5)
#1 0x4eed2d (/store/artem/git/mcsema/tests/libFuzzer/fuzzme+0x4eed2d)
#2 0x4f464a (/store/artem/git/mcsema/tests/libFuzzer/fuzzme+0x4f464a)
#3 0x4f57de (/store/artem/git/mcsema/tests/libFuzzer/fuzzme+0x4f57de)
#4 0x4f5db7 (/store/artem/git/mcsema/tests/libFuzzer/fuzzme+0x4f5db7)
#5 0x4f0546 (/store/artem/git/mcsema/tests/libFuzzer/fuzzme+0x4f0546)
#6 0x4ef232 (/store/artem/git/mcsema/tests/libFuzzer/fuzzme+0x4ef232)
#7 0x4ef183 (/store/artem/git/mcsema/tests/libFuzzer/fuzzme+0x4ef183)
#8 0x7f7cb7ce482f (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
#9 0x41a3b8 (/store/artem/git/mcsema/tests/libFuzzer/fuzzme+0x41a3b8)
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (/store/artem/git/mcsema/tests/libFuzzer/fuzzme+0x4eebf5)
==21762==ABORTING
DEATH:
0x66,0x75,0x7a,0x7a,0x66,0x75,
fuzzfu
artifact_prefix='./'; Test unit written to ./crash-643f1d111ee9b982169c2278898ce3228c4c6e2e
Base64: ZnV6emZ1
```
Good news! libFuzzer found our magic prefix of 'fuzz' and triggered the error.
## Building The Binary
Mcsema operates on binaries, so let's give it a binary to translate and make sure it works:
McSema operates on binaries, so let's give it a binary to translate and make sure it works:
$ clang++-3.8 -o fuzzme.binary fuzzme.cc
$ ./fuzzme.binary
Usage:
./fuzzme.binary: <text>
$ ./fuzzme.binary aaaaa
Bad input
$ ./fuzzme.binary fuzza
Segmentation fault (core dumped)
```shell
$ clang++-3.8 -o fuzzme.binary fuzzme.cc
$ ./fuzzme.binary
Usage:
./fuzzme.binary: <text>
$ ./fuzzme.binary aaaaa
Bad input
$ ./fuzzme.binary fuzza
Segmentation fault (core dumped)
```
Great, the binary runs and exhibits the bad condition we want to fuzz for. Now let's lift it into bitcode!
@@ -145,71 +152,80 @@ Great, the binary runs and exhibits the bad condition we want to fuzz for. Now l
The function we really want to lift isn't `main`, but `vulnerable`. Since the binary is C++ based, `vulnerable`'s symbol name has been mangled. Let's find the [mangled name](https://en.wikipedia.org/wiki/Name_mangling) we need to use as the entry point.
$ nm fuzzme.binary | grep vulnerable
00000000004005f0 T _Z10vulnerablePKc
```shell
$ nm fuzzme.binary | grep vulnerable
00000000004005f0 T _Z10vulnerablePKc
```
Now, let's disassemble the binary, starting with the entry point of `_Z10vulnerablePKc`.
$ ../../bin/mcsema-disass --disassembler ~/ida-6.9/idal64 --arch amd64 --os linux --entrypoint _Z10vulnerablePKc --binary fuzzme.binary --output fuzzme.cfg --log_file fuzzme.log
```shell
../../bin/mcsema-disass --disassembler ~/ida-6.9/idal64 --arch amd64 --os linux --entrypoint _Z10vulnerablePKc --binary fuzzme.binary --output fuzzme.cfg --log_file fuzzme.log
```
And once we have the control flow graph, we can convert it to bitcode:
$ ../../bin/mcsema-lift --arch amd64 --os linux --entrypoint _Z10vulnerablePKc --cfg fuzzme.cfg --output fuzzme.bc
... lots of outputs ...
Adding entry point: _Z10vulnerablePKc
_Z10vulnerablePKc is implemented by sub_4005f0
```shell
$ ../../bin/mcsema-lift --arch amd64 --os linux --entrypoint _Z10vulnerablePKc --cfg fuzzme.cfg --output fuzzme.bc
... lots of outputs ...
Adding entry point: _Z10vulnerablePKc
_Z10vulnerablePKc is implemented by sub_4005f0
```
## Using libFuzzer on mcsema bitcode
## Using libFuzzer on McSema bitcode
To use libFuzzer, we need a function named `LLVMFuzzerTestOneInput`, so we have to create a small driver program to call into our bitcode. We have included a pre-made one, aptly named `driver.cc`:
$ cat driver.cc
#include <stdint.h>
#include <stdlib.h>
extern int vulnerable(const char *input);
extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
vulnerable((const char *)(data));
return 0;
}
```c++
#include <stdint.h>
#include <stdlib.h>
extern int vulnerable(const char *input);
extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
vulnerable((const char *)(data));
return 0;
}
```
The driver just says that there is an external function with the signature `int vulnerable(const char *)` (this is the function we lifted), and calls it.
Now let's combine the driver, our bitcode, mcsema assembly stubs, and libFuzzer instrumentation into one program:
Now let's combine the driver, our bitcode, McSema assembly stubs, and libFuzzer instrumentation into one program:
$ clang++-3.8 -O3 -o fuzzme.mcsema driver.cc fuzzme.bc ../../lib/libmcsema_rt64.a Fuzzer*.o -fsanitize=address -fsanitize-coverage=edge
```shell
clang++-3.8 -O3 -o fuzzme.mcsema driver.cc fuzzme.bc ../../lib/libmcsema_rt64.a Fuzzer*.o -fsanitize=address -fsanitize-coverage=edge
```
We can now try fuzzing the newly instrumented binary:
$ ./fuzzme.mcsema
Seed: 1024766608
PreferSmall: 1
#0 READ units: 1 exec/s: 0
#1 INITED cov: 5 units: 1 exec/s: 0
#742 NEW cov: 8 units: 2 exec/s: 0 L: 64 MS: 0
#238510 NEW cov: 11 units: 3 exec/s: 0 L: 5 MS: 4 CrossOver-InsertByte-EraseByte-ChangeByte-
#252530 NEW cov: 13 units: 4 exec/s: 0 L: 11 MS: 4 ShuffleBytes-InsertByte-CrossOver-ChangeBit-
ASAN:DEADLYSIGNAL
=================================================================
==21840==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x0000004ef9f0 bp 0x000000000001 sp 0x7f357e7bd99c T0)
#0 0x4ef9ef (/store/artem/git/mcsema/tests/libFuzzer/fuzzme.mcsema+0x4ef9ef)
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (/store/artem/git/mcsema/tests/libFuzzer/fuzzme.mcsema+0x4ef9ef)
==21840==ABORTING
DEATH:
0x66,0x75,0x7a,0x7a,0x75,0xf,0xf,0x3b,0x3b,0x66,0x66,
fuzzu\x0f\x0f;;ff
artifact_prefix='./'; Test unit written to ./crash-9991751ea97f13a1b7eeeb9e54c69be96cc782f7
Base64: ZnV6enUPDzs7ZmY=
```shell
$ ./fuzzme.mcsema
Seed: 1024766608
PreferSmall: 1
#0 READ units: 1 exec/s: 0
#1 INITED cov: 5 units: 1 exec/s: 0
#742 NEW cov: 8 units: 2 exec/s: 0 L: 64 MS: 0
#238510 NEW cov: 11 units: 3 exec/s: 0 L: 5 MS: 4 CrossOver-InsertByte-EraseByte-ChangeByte-
#252530 NEW cov: 13 units: 4 exec/s: 0 L: 11 MS: 4 ShuffleBytes-InsertByte-CrossOver-ChangeBit-
ASAN:DEADLYSIGNAL
=================================================================
==21840==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x0000004ef9f0 bp 0x000000000001 sp 0x7f357e7bd99c T0)
#0 0x4ef9ef (/store/artem/git/mcsema/tests/libFuzzer/fuzzme.mcsema+0x4ef9ef)
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (/store/artem/git/mcsema/tests/libFuzzer/fuzzme.mcsema+0x4ef9ef)
==21840==ABORTING
DEATH:
0x66,0x75,0x7a,0x7a,0x75,0xf,0xf,0x3b,0x3b,0x66,0x66,
fuzzu\x0f\x0f;;ff
artifact_prefix='./'; Test unit written to ./crash-9991751ea97f13a1b7eeeb9e54c69be96cc782f7
Base64: ZnV6enUPDzs7ZmY=
```
Success! It finds crashes with the same `fuzz` prefix as before.
## Remarks
As we mentioned in the introduction, mcsema generated bitcode and assembly stubs are not quite compatible with address sanitizer because they do things that normal programs should not do. The `-O3` in the final command line is necessary to produce code where the fuzzer-generated segfault can be reported. Try the same command line with `-O0`: libFuzzer will find the bug, but will not be able to properly report that it was found.
Using mcsema and libFuzzer on large programs is still a work in progress. We think that it can work, but currently CFG recovery is not accurate enough to use libFuzzer and mcsema on the normal libFuzzer samples. We hope to change that in the future.
As we mentioned in the introduction, McSema generated bitcode and assembly stubs are not quite compatible with address sanitizer because they do things that normal programs should not do. The `-O3` in the final command line is necessary to produce code where the fuzzer-generated segfault can be reported. Try the same command line with `-O0`: libFuzzer will find the bug, but will not be able to properly report that it was found.
Using McSema and libFuzzer on large programs is still a work in progress. We think that it can work, but currently CFG recovery is not accurate enough to use libFuzzer and McSema on the normal libFuzzer samples. We hope to change that in the future.