mirror of
https://github.com/lifting-bits/mcsema
synced 2026-06-08 15:31:09 +00:00
Update links and remotes to the new organization name (#598)
* Docs: Update links with the new organization name * CI/Travis: Update remotes with the new organization name * mcsema-disass: Update links with the new organization name * Dockerfile: Update remotes with the new organization name
This commit is contained in:
Alessandro Gario
committed by
Peter Goodman
Peter Goodman
parent
d134f3ad44
commit
1a96bd3712
+2
-2
@@ -20,7 +20,7 @@ To get started, use git to clone the `remill` repo, and start a new branch that
|
||||
|
||||
## Adding a New Instruction
|
||||
|
||||
To add support for lifting a new instruction, you will be extending the `remill` library. That is where the instruction semantics are implemented. We have created a whole guide to adding an instruction, which [you can find here](https://github.com/trailofbits/remill/blob/master/docs/ADD_AN_INSTRUCTION.md).
|
||||
To add support for lifting a new instruction, you will be extending the `remill` library. That is where the instruction semantics are implemented. We have created a whole guide to adding an instruction, which [you can find here](https://github.com/lifting-bits/remill/blob/master/docs/ADD_AN_INSTRUCTION.md).
|
||||
|
||||
## Creating Your GitHub Pull Request
|
||||
|
||||
@@ -48,7 +48,7 @@ Before you create your pull request, we ask that you conform your code to our pr
|
||||
## Advice and Guidance for New Contributors
|
||||
|
||||
Please take a look at our McSema documentation! We try to keep it updated with
|
||||
[debugging tips](https://github.com/trailofbits/mcsema/blob/master/docs/DebuggingTips.md), [common errors you might encounter](https://github.com/trailofbits/mcsema/blob/master/docs/CommonErrors.md), and [how to get acquainted with the codebase](https://github.com/trailofbits/mcsema/blob/master/docs/NavigatingTheCode.md).
|
||||
[debugging tips](https://github.com/lifting-bits/mcsema/blob/master/docs/DebuggingTips.md), [common errors you might encounter](https://github.com/lifting-bits/mcsema/blob/master/docs/CommonErrors.md), and [how to get acquainted with the codebase](https://github.com/lifting-bits/mcsema/blob/master/docs/NavigatingTheCode.md).
|
||||
|
||||
## Useful External Links
|
||||
|
||||
|
||||
@@ -7,11 +7,11 @@
|
||||
|
||||
McSema is an executable lifter. It translates ("lifts") executable binaries from native machine code to LLVM bitcode. LLVM bitcode is an [intermediate representation](https://en.wikipedia.org/wiki/Intermediate_representation) form of a program that was originally created for the [retargetable LLVM compiler](https://llvm.org), but which is also very useful for performing program analysis methods that would not be possible to perform on an executable binary directly.
|
||||
|
||||
McSema enables analysts to find and retroactively harden binary programs against security bugs, independently validate vendor source code, and generate application tests with high code coverage. McSema isn’t just for static analysis. The lifted LLVM bitcode can also be [fuzzed with libFuzzer](https://github.com/trailofbits/mcsema/blob/master/docs/UsingLibFuzzer.md), an LLVM-based instrumented fuzzer that would otherwise require the target source code. The lifted bitcode can even be [compiled](https://github.com/trailofbits/mcsema/blob/master/docs/UsingLibFuzzer.md) back into a [runnable program](https://github.com/trailofbits/mcsema/blob/master/docs/McSemaWalkthrough.md)! This is a procedure known as static binary rewriting, binary translation, or binary recompilation.
|
||||
McSema enables analysts to find and retroactively harden binary programs against security bugs, independently validate vendor source code, and generate application tests with high code coverage. McSema isn’t just for static analysis. The lifted LLVM bitcode can also be [fuzzed with libFuzzer](https://github.com/lifting-bits/mcsema/blob/master/docs/UsingLibFuzzer.md), an LLVM-based instrumented fuzzer that would otherwise require the target source code. The lifted bitcode can even be [compiled](https://github.com/lifting-bits/mcsema/blob/master/docs/UsingLibFuzzer.md) back into a [runnable program](https://github.com/lifting-bits/mcsema/blob/master/docs/McSemaWalkthrough.md)! This is a procedure known as static binary rewriting, binary translation, or binary recompilation.
|
||||
|
||||
McSema supports lifting both Linux (ELF) and Windows (PE) executables, and understands most x86 and amd64 instructions, including integer, X87, MMX, SSE and AVX operations. AARCH64 (ARMv8) instruction support is in active development.
|
||||
|
||||
Using McSema is a two-step process: control flow recovery, and instruction translation. Control flow recovery is performed using the `mcsema-disass` tool, which relies on IDA Pro, Binary Ninja, or DynInst to disassemble a binary file and produce a control flow graph. Instruction translation is then performed using the `mcsema-lift` tool, which converts the control flow graph into LLVM bitcode. Under the hood, the instruction translation capability of `mcsema-lift` is implemented in the [`remill` library](https://github.com/trailofbits/remill). The development of `remill` was a result of refactoring and improvements to McSema, and was first introduced with McSema version 2.0.0. Read more about `remill` [here](https://github.com/trailofbits/remill).
|
||||
Using McSema is a two-step process: control flow recovery, and instruction translation. Control flow recovery is performed using the `mcsema-disass` tool, which relies on IDA Pro, Binary Ninja, or DynInst to disassemble a binary file and produce a control flow graph. Instruction translation is then performed using the `mcsema-lift` tool, which converts the control flow graph into LLVM bitcode. Under the hood, the instruction translation capability of `mcsema-lift` is implemented in the [`remill` library](https://github.com/lifting-bits/remill). The development of `remill` was a result of refactoring and improvements to McSema, and was first introduced with McSema version 2.0.0. Read more about `remill` [here](https://github.com/lifting-bits/remill).
|
||||
|
||||
McSema and `remill` were developed and are maintained by Trail of Bits, funded by and used in research for DARPA and the US Department of Defense.
|
||||
|
||||
@@ -89,7 +89,7 @@ Why would anyone translate binaries *back* to bitcode?
|
||||
|
||||
#### Step 1: Download Dockerfile
|
||||
|
||||
`wget https://raw.githubusercontent.com/trailofbits/mcsema/master/tools/Dockerfile`
|
||||
`wget https://raw.githubusercontent.com/lifting-bits/mcsema/master/tools/Dockerfile`
|
||||
|
||||
#### Step 2: Add your disassembler
|
||||
|
||||
@@ -161,15 +161,15 @@ popd
|
||||
|
||||
#### Step 2: Clone the repository
|
||||
|
||||
The next step is to clone the [Remill](https://github.com/trailofbits/remill) repository. We then clone the McSema repository into the `tools` subdirectory of Remill. This is kind of like how Clang and LLVM are distributed separately, and the Clang source code needs to be put into LLVM's tools directory.
|
||||
The next step is to clone the [Remill](https://github.com/lifting-bits/remill) repository. We then clone the McSema repository into the `tools` subdirectory of Remill. This is kind of like how Clang and LLVM are distributed separately, and the Clang source code needs to be put into LLVM's tools directory.
|
||||
|
||||
**Notice that when building McSema, you should always use a specific Remill commit hash (the one we test). This hash can be found in the .remill_commit_id file**.
|
||||
|
||||
```shell
|
||||
git clone --depth 1 https://github.com/trailofbits/mcsema.git
|
||||
git clone --depth 1 https://github.com/lifting-bits/mcsema.git
|
||||
export REMILL_VERSION=`cat ./mcsema/.remill_commit_id`
|
||||
|
||||
git clone https://github.com/trailofbits/remill.git
|
||||
git clone https://github.com/lifting-bits/remill.git
|
||||
cd remill
|
||||
git checkout -b temp ${REMILL_VERSION}
|
||||
|
||||
@@ -243,8 +243,8 @@ In order to verify that McSema works correctly as built, head on over to [the do
|
||||
|
||||
#### Step 2: Obtaining the source code
|
||||
```
|
||||
git clone https://github.com/trailofbits/remill.git --depth=1
|
||||
git clone https://github.com/trailofbits/mcsema.git --depth=1 remill/tools/mcsema
|
||||
git clone https://github.com/lifting-bits/remill.git --depth=1
|
||||
git clone https://github.com/lifting-bits/mcsema.git --depth=1 remill/tools/mcsema
|
||||
```
|
||||
|
||||
Note that for production usage you should always use a specific remill commit (`remill/tools/mcsema/.remill_commit_id`) when building mcsema. At the time of writing, it is however best to use HEAD (or at least make sure that commit `e7795be` is present in the remill branch).
|
||||
@@ -310,7 +310,7 @@ Also update your PYTHONPATH: C:\mcsema\Lib\site-packages
|
||||
|
||||
* [McSema command line reference](docs/CommandLineReference.md)
|
||||
* [Common Errors](docs/CommonErrors.md) and [Debugging Tips](docs/DebuggingTips.md)
|
||||
* [How to add support for a new instruction](https://github.com/trailofbits/remill/blob/master/docs/ADD_AN_INSTRUCTION.md)
|
||||
* [How to add support for a new instruction](https://github.com/lifting-bits/remill/blob/master/docs/ADD_AN_INSTRUCTION.md)
|
||||
* [How to use McSema: A walkthrough](docs/McSemaWalkthrough.md)
|
||||
* [Life of an instruction](docs/LifeOfAnInstruction.md)
|
||||
* [Limitations](docs/Limitations.md)
|
||||
@@ -333,8 +333,8 @@ You don't! You can also use Binary Ninja or [Dyninst](tools/mcsema_disass/dynins
|
||||
|
||||
### What is Remill, and why does McSema need it
|
||||
|
||||
[Remill](https://github.com/trailofbits/remill) is a library that McSema uses to lift individual machine code instructions to LLVM IR. You can think of McSema being to Remill as Clang is to LLVM. Remill's scope is small: it focuses on instruction semantics only, and it provides semantics for x86, x86-64, and AArch64 instruction semantics. McSema's scope is much bigger: it focuses on lifting entire programs. To do so, McSema must lift the individual instructions, but there's a lot more to lifting programs than just the instructions; there are code and data cross-references, segments, etc.
|
||||
[Remill](https://github.com/lifting-bits/remill) is a library that McSema uses to lift individual machine code instructions to LLVM IR. You can think of McSema being to Remill as Clang is to LLVM. Remill's scope is small: it focuses on instruction semantics only, and it provides semantics for x86, x86-64, and AArch64 instruction semantics. McSema's scope is much bigger: it focuses on lifting entire programs. To do so, McSema must lift the individual instructions, but there's a lot more to lifting programs than just the instructions; there are code and data cross-references, segments, etc.
|
||||
|
||||
### I'm a student and I'd like to contribute to McSema: how can I help
|
||||
|
||||
We would love to take you on as an intern to help improve McSema. We have several project ideas labelled [`intern project`](https://github.com/trailofbits/mcsema/labels/intern%20project), as well as having smaller scale to-dos labelled under [`good first issue`](https://github.com/trailofbits/mcsema/labels/good%20first%20issue) and [`help wanted`](https://github.com/trailofbits/mcsema/labels/help%20wanted) on our issue tracker. You are not limited to those items: if you think of a great feature you want in McSema, let us know and we will sponsor it. Simply contact us on our [Slack channel](https://empireslacking.herokuapp.com/) or via mcsema@trailofbits.com and let us know what you'd want to work on and why.
|
||||
We would love to take you on as an intern to help improve McSema. We have several project ideas labelled [`intern project`](https://github.com/lifting-bits/mcsema/labels/intern%20project), as well as having smaller scale to-dos labelled under [`good first issue`](https://github.com/lifting-bits/mcsema/labels/good%20first%20issue) and [`help wanted`](https://github.com/lifting-bits/mcsema/labels/help%20wanted) on our issue tracker. You are not limited to those items: if you think of a great feature you want in McSema, let us know and we will sponsor it. Simply contact us on our [Slack channel](https://empireslacking.herokuapp.com/) or via mcsema@trailofbits.com and let us know what you'd want to work on and why.
|
||||
|
||||
@@ -49,7 +49,7 @@ Exception: Unknown external: cs_open
|
||||
|
||||
**Technical Background:** McSema needs to know how to call external functions, including the function calling convention and number of arguments. It knows how to call many common functions, but you hit one that it does not know about.
|
||||
|
||||
**Possible Fixes:** Add an entry to the [external function definitions file](https://github.com/trailofbits/mcsema/tree/master/tools/mcsema_disass/defs) describing the function's calling convention and number of arguments. Don't forget to re-build `mcsema-disass`. Submit a pull request so we can include it in future McSema releases. Alternatively, you can specify a custom definitions file location by using the `--std-defs` argument (e.g. `mcsema-disass --std-defs /path/to/my/defs/file.txt ...`).
|
||||
**Possible Fixes:** Add an entry to the [external function definitions file](https://github.com/lifting-bits/mcsema/tree/master/tools/mcsema_disass/defs) describing the function's calling convention and number of arguments. Don't forget to re-build `mcsema-disass`. Submit a pull request so we can include it in future McSema releases. Alternatively, you can specify a custom definitions file location by using the `--std-defs` argument (e.g. `mcsema-disass --std-defs /path/to/my/defs/file.txt ...`).
|
||||
|
||||
### Could not parse function type
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Things have gone wrong, now what
|
||||
|
||||
This document describes some approaches to debugging the lifted bitcode produced by McSema on Linux. This document does not describe [how to add missing instructions](https://github.com/trailofbits/remill/blob/master/docs/ADD_AN_INSTRUCTION.md), or how to [resolve common errors](CommonErrors.md).
|
||||
This document describes some approaches to debugging the lifted bitcode produced by McSema on Linux. This document does not describe [how to add missing instructions](https://github.com/lifting-bits/remill/blob/master/docs/ADD_AN_INSTRUCTION.md), or how to [resolve common errors](CommonErrors.md).
|
||||
|
||||
## Terminology
|
||||
|
||||
@@ -43,7 +43,7 @@ There are opportunities and drawbacks to debugging lifted code.
|
||||
|
||||
### Opportunities
|
||||
|
||||
* The [`State`](https://github.com/trailofbits/remill/blob/master/remill/Arch/X86/Runtime/State.h) is stored in memory. The advantage to this is that one can set data breakpoints (hardware watchpoints) on individual registers in the state structure. This is incredibly useful if you have a [time-travelling debugger](http://undo.io/products/undodb/).
|
||||
* The [`State`](https://github.com/lifting-bits/remill/blob/master/remill/Arch/X86/Runtime/State.h) is stored in memory. The advantage to this is that one can set data breakpoints (hardware watchpoints) on individual registers in the state structure. This is incredibly useful if you have a [time-travelling debugger](http://undo.io/products/undodb/).
|
||||
* Lifted bitcode can be instrumented using the LLVM toolchain. Some useful-for-debugging instrumentations come built-in to `mcsema-lift`.
|
||||
|
||||
#### Built-in instrumentation
|
||||
@@ -522,7 +522,7 @@ $4 = 30
|
||||
|
||||
Alright, we have confirmation, the lifted code for `cvtsi2ss` is definitely doing the wrong thing. Time to see the C++ code implementing this instruction.
|
||||
|
||||
**NOTE:** How McSema lifts instructions has changed since this walkthrough was originally written. The remainder of this document has been preserved for posterity. Instruction semantics are no longer implemented within McSema, and are instead implemented (and tested) in the [Remill](https://github.com/trailofbits/remill) project.
|
||||
**NOTE:** How McSema lifts instructions has changed since this walkthrough was originally written. The remainder of this document has been preserved for posterity. Instruction semantics are no longer implemented within McSema, and are instead implemented (and tested) in the [Remill](https://github.com/lifting-bits/remill) project.
|
||||
|
||||
First, we'll figure out what LLVM opcode implements this instruction.
|
||||
|
||||
@@ -573,6 +573,6 @@ Let's make this change, rebuild and install McSema, re-run `mcsema-lift`, compil
|
||||
|
||||
#### Closing comments
|
||||
|
||||
This document presented a few of the helpful commands provided for debugging, and then walked through the diagnosis and fixing of a specific bug that was present in McSema at commit [409abe3d31a7b0d09b1fee9c60e1d190de39cced](https://github.com/trailofbits/mcsema/commit/409abe3d31a7b0d09b1fee9c60e1d190de39cced).
|
||||
This document presented a few of the helpful commands provided for debugging, and then walked through the diagnosis and fixing of a specific bug that was present in McSema at commit [409abe3d31a7b0d09b1fee9c60e1d190de39cced](https://github.com/lifting-bits/mcsema/commit/409abe3d31a7b0d09b1fee9c60e1d190de39cced).
|
||||
|
||||
There is a lot of information in this document. Despite this, the same techniques used in this document can be productively applied to may similar types of bugs.
|
||||
|
||||
@@ -95,6 +95,6 @@ block_804b7a3:
|
||||
|
||||
The produced LLVM functions have a predictable structure. They are named according to the `ea` (effective address, or entry address) field in the `Function` message. The `ea` of our example function is `0x804b7a3`, and so the lifted function is named `sub_804b7a3`.
|
||||
|
||||
All lifted functions accept three arguments: a pointer to a [`State`](https://github.com/trailofbits/remill/blob/master/remill/Arch/X86/Runtime/State.h#L520) structure, the current program counter, and an opaque pointer to a `Memory` structure. The lifted bitcode has a predictable structure. It emulates reads and writes of machine registers by using `load`s and `store`s to the `State` structure. It also emulates reads and writes to memory via `load`s and `store`s. To emulate memory differently, pass the `--keep_memops` option to `mcsema-lift-M.m`.
|
||||
All lifted functions accept three arguments: a pointer to a [`State`](https://github.com/lifting-bits/remill/blob/master/remill/Arch/X86/Runtime/State.h#L520) structure, the current program counter, and an opaque pointer to a `Memory` structure. The lifted bitcode has a predictable structure. It emulates reads and writes of machine registers by using `load`s and `store`s to the `State` structure. It also emulates reads and writes to memory via `load`s and `store`s. To emulate memory differently, pass the `--keep_memops` option to `mcsema-lift-M.m`.
|
||||
|
||||
In this case, we see the use of some Remill intrinsic functions, but in practice those don't show up, because `int 0x80`s and their ilk are typically found in libraries that wrap system calls, and not directly in the binaries themselves.
|
||||
|
||||
@@ -54,7 +54,7 @@ Let's walk through each option:
|
||||
|
||||
### Fixing Errors
|
||||
|
||||
This section documents how to fix a common CFG recovery problem: undefined external functions. By the time you are reading this guide, the functions described here may have already been added to the [list of common external Linux functions that comes with mcsema](https://github.com/trailofbits/mcsema/blob/master/tools/mcsema_disass/defs/linux.txt) and you can skip this section.
|
||||
This section documents how to fix a common CFG recovery problem: undefined external functions. By the time you are reading this guide, the functions described here may have already been added to the [list of common external Linux functions that comes with mcsema](https://github.com/lifting-bits/mcsema/blob/master/tools/mcsema_disass/defs/linux.txt) and you can skip this section.
|
||||
|
||||
The previous command may have failed (as in this snippet). If it did, read on. If not, skip this section and move on to Translation To Bitcode.
|
||||
|
||||
|
||||
@@ -81,16 +81,16 @@ sudo pip install --upgrade pip
|
||||
sudo pip install 'protobuf==3.2.0'
|
||||
```
|
||||
|
||||
Now that we have the dependencies we need, we should clone [Remill](https://github.com/trailofbits/remill).
|
||||
Now that we have the dependencies we need, we should clone [Remill](https://github.com/lifting-bits/remill).
|
||||
|
||||
```bash
|
||||
cd ~/data
|
||||
git clone git@github.com:trailofbits/remill.git
|
||||
git clone https://github.com/lifting-bits/remill.git
|
||||
```
|
||||
|
||||
### Step 2: Build Remill, McSema, and KLEE
|
||||
|
||||
We have provided a convenient [script](https://github.com/trailofbits/remill/blob/master/scripts/build_klee.sh) for this walkthrough. The script will clone the latest version of McSema into the Remill checkout, as well as clone a version of KLEE that is compatible with Remill. You might already have McSema installed, and it might be a version that is not compatible with KLEE. That is not a problem. The script will ensure that the proper toolchain is built *within* the directory in which you invoke the script.
|
||||
We have provided a convenient [script](https://github.com/lifting-bits/remill/blob/master/scripts/build_klee.sh) for this walkthrough. The script will clone the latest version of McSema into the Remill checkout, as well as clone a version of KLEE that is compatible with Remill. You might already have McSema installed, and it might be a version that is not compatible with KLEE. That is not a problem. The script will ensure that the proper toolchain is built *within* the directory in which you invoke the script.
|
||||
|
||||
```bash
|
||||
mkdir /tmp/klee_ws
|
||||
|
||||
+1
-1
@@ -246,7 +246,7 @@ linux_build_helper() {
|
||||
return 1
|
||||
fi
|
||||
|
||||
git clone "https://github.com/trailofbits/remill.git" > "${log_file}" 2>&1
|
||||
git clone "https://github.com/lifting-bits/remill.git" > "${log_file}" 2>&1
|
||||
if [ $? -ne 0 ] ; then
|
||||
printf " x Failed to clone the remill repository. Error output follows:\n"
|
||||
printf "===\n"
|
||||
|
||||
+2
-2
@@ -15,9 +15,9 @@ RUN apt-get install git curl cmake python2.7 python-pip python-virtualenv wget b
|
||||
WORKDIR /home/ToB
|
||||
|
||||
# Download everything and set up folder structure
|
||||
RUN git clone --depth 1 https://github.com/trailofbits/mcsema.git && \
|
||||
RUN git clone --depth 1 https://github.com/lifting-bits/mcsema.git && \
|
||||
export REMILL_VERSION=`cat ./mcsema/.remill_commit_id` && \
|
||||
git clone https://github.com/trailofbits/remill.git && \
|
||||
git clone https://github.com/lifting-bits/remill.git && \
|
||||
cd remill && \
|
||||
git checkout -b temp ${REMILL_VERSION} && \
|
||||
mv ../mcsema tools
|
||||
|
||||
+1
-1
@@ -22,7 +22,7 @@ from setuptools import setup, find_packages
|
||||
setup(name="mcsema-disass",
|
||||
description="Binary program disassembler for McSema.",
|
||||
version="2.0",
|
||||
url="https://github.com/trailofbits/mcsema",
|
||||
url="https://github.com/lifting-bits/mcsema",
|
||||
author="Trail of Bits",
|
||||
author_email="mcsema@trailofbits.com",
|
||||
license='Apache 2.0',
|
||||
|
||||
Reference in New Issue
Block a user