diff --git a/README.rst b/README.rst new file mode 100644 index 000000000..9146ae5ea --- /dev/null +++ b/README.rst @@ -0,0 +1,52 @@ +******* +Purpose +******* + +`revamb` is a static binary translator. Given a input ELF binary for one of the +supported architectures (currently MIPS, ARM and x86-64) it will analyze it and +emit an equivalent LLVM IR. To do so, `revamb` employs the QEMU intermediate +representation (a series of TCG instructions) and then translates them to LLVM +IR. + +************ +How to build +************ + +`revamb` employs CMake as a build system. The build system will try to +automatically detect the QEMU installation and the GCC toolchains require to +build the test binaries. + +If everything is in standard locations, you can just run:: + + mkdir build/ + cd build/ + cmake .. + make -j$(nproc) + make install + +For further build options and more advanced configurations see +docs/BuildSystem.rst (TODO: reference). + +To run the test suite simply run:: + + make test + +*********** +Example run +*********** + +The simplest possible example consists in the following:: + + cd build + cat > hello.c < + + int main(int argc, char *argv[]) { + printf("Hello, world!\n"); + } + EOF + armv7a-hardfloat-linux-uclibceabi-gcc -static hello.c -o hello.arm + ./translate hello.arm + # ... + ./hello.arm.translated + Hello, world! diff --git a/docs/BuildSystem.rst b/docs/BuildSystem.rst new file mode 100644 index 000000000..a311159ac --- /dev/null +++ b/docs/BuildSystem.rst @@ -0,0 +1,83 @@ +*********************************** +Identifying the required components +*********************************** + +`revamb` requires three main components: LLVM, QEMU and one or more GCC +toolchains. + +LLVM +==== + +LLVM should be automatically detected if it's a standard location. If you want +to use a custom build in a non-standard location use the ``LLVM_DIR`` option, +making it point to the directory containing ``LLVMConfig.cmake`` in the install +directory (typically ``$INSTALL_PATH/share/llvm/cmake``): + +.. code-block:: sh + + cmake -DLLVM_DIR=/home/me/llvm-install/share/llvm/cmake ../ + +QEMU +==== + +By default the build system will look for the system QEMU installation (i.e., in +``/usr/``). If this is not what you want you can customize the QEMU install path +using the ``QEMU_INSTALL_PATH`` variable: + +.. code-block:: sh + + cmake -DQEMU_INSTALL_PATH="/home/me/qemu-install/" ../ + +GCC +=== + +The `revamb` build system will try to automatically detect toolchains to compile +code for the supported architectures. The toolchains are required to correctly +run the `revamb` test suite. + +The autodetction mechanism looks in all the directories in the ``PATH`` +environment variable for an executable matching the pattern ``$ARCH*-musl*-gcc`` +or ``$ARCH*-uclibc*-gcc`` (glibc is currently unsupported). + +It's possible to force the usage of a specific compiler for a certain +architecture using the CMake variable ``C_COMPILER_$ARCH``: + +.. code-block:: sh + + cmake -DC_COMPILER_mips="/home/me/my-mips-compiler" \ + -DC_COMPILER_miparm="/home/me/my-arm-compiler" \ + -DC_COMPILER_x86_64="/home/me/my-x86_64-compiler" \ + ../ + +The `revamb` build system also provides an option to specify additional flags to +employ when using the above mentioned commpilers for compilation and linking of +test binaries. This can be done using the variables ``TEST_CFLAGS_$ARCH`` (for +compile-time flags) and ``TEST_LINK_LIBRARIES_$ARCH`` (for linking). + +This is particularly useful to force x86-64 binaries to compile with software +floating point, which can be achieved with GCC as follows: + +.. code-block:: sh + + cmake -DTEST_CFLAGS_x86_64="-msoft-float -mfpmath=387 -mlong-double-64" \ + -DTEST_LINK_LIBRARIES_x86_64="-lc $LLVM_INSTALL_PATH/lib/linux/libclang_rt.builtins-x86_64.a" \ + ../ + +******************** +Common CMake options +******************** + +In the following we provide some useful-to-know CMake variables, that are not +specific to `revamb`. They can be specified using the ``-D`` flag, e.g.: + +.. code-block:: sh + + cmake -DCMAKE_INSTALL_PREFIX=/tmp ../ + +:CMAKE_INSTALL_PREFIX: Specify the install path. This is the destination + directory where all the necessary files will be copied + upon ``make install``. +:CMAKE_BUILD_TYPE: Specify the type of build. The two most useful values are + ``Debug``, for an unoptimized build with debugging + information, and ``RelWithdebinfo``, for an optimized build + with debugging information. diff --git a/docs/FromIRToExecutable.rst b/docs/FromIRToExecutable.rst new file mode 100644 index 000000000..0adfeb259 --- /dev/null +++ b/docs/FromIRToExecutable.rst @@ -0,0 +1,123 @@ +***************************** +From the IR to the executable +***************************** + +This document aims to describe how to obtain a working executable from the IR +generated by `revamb`. This for documentation purposes only, standard users can +simply use the `translate` script, which will take care of everything, + +From the IR to the object file +============================== + +As an optional step, before proceeding in compiling the IR, it's possible to run +the optimization pipeline on the generated IR using the LLVM `opt` tool: + +.. code-block:: sh + + opt -O2 -S translated.ll -o translated.opt.ll + +Depending on the size of the input program this might take some time. + +The first step to obtain the final executable consists in compiling the IR. To +do this, we can use `llc`, the LLVM static compiler, which takes an LLVM IR file +and produces an object file: + +.. code-block:: sh + + llc -O0 translated.ll -filetype=obj -o translated.o + +`llc` exposes the compiler backend. The backend can do another series of +optimizations: + +.. code-block:: sh + + llc -O2 translated.ll -filetype=obj -o translated.o -regalloc=fast + +Note: as for `opt`, optimizations can take a considerable amount of time, in +particular if slow register allocation methods are used (i.e., make sure to use +`-regalloc=fast`). Non-trivial register allocation techniques on the `root` +function can be prohibitively costly (see also `GeneratedIRReference.rst`_). + +Support functions +================= + +The IR produced by `revamb` is mostly self-contained, since all the QEMU helper +functions are statically linked in the output module (unless ``--no-link`` was +specified, in which case the module won't be functional). Howver, besides the +QEMU helper functions, some additional support functions are required to obtain +a working program, in particular for initialization purposes. + +`support.c` provides the default implementation for them. In the following we +provides a quick summary of the required functions as they are implemented in +`support.c`: + +:main: The entry point of the program, maps in memory the stack and the heap. It + also performs the stack initialization, which is a very delicate + operation. As it can be seen in the `prepare_stack` function the + environment variables, the arguments and the auxiliary vectors are + initialized mimicing the stack initialization performed by the Linux + kernel. `main` also performs a call to `syscall_init` a function defined + in the generated module, taken from QEMU, which carries on some + syscall-related initializations. Finally, the entry point of the module + generated by `revamb` (the `root` function) is called. + +:path: Function invoked by the QEMU Linux syscall emulation layer to *filter* + the path being opened by the translated program. For instance, it is + possible to catch attempts to open ``/proc/self/map`` and redirect it + somewhere else transparently. + +:unknownPC: Function handling the situation in which the code at an address that + should be executed has not been translated (see + `GeneratedIRReference.rst`_). It is basically an error handling + function that is supposed to never return. + +:newpc: If the ``--tracing`` option has been given, the calls to the `newpc` + functions emitted by `revamb` are turned into calls to an external + function, that the user can implement. This function can access the + status of the CPU and perform high performance tracing. + +The remaining functions in `support.c` are of little interest and might be +removed in the future. Note also that `support.c` works automatically with all +the supported architectures, but needs to know which was the input +architecture. For this reason, defined a macro ``TARGET_arch`` (e.g., +``TARGET_arm``) on the compilation command line. + +Linking +======= + +Once we have the object file generated by `revamb` we can compile `support.c` +and link them together. Be sure to link also `libm.so`, `libz.so` and +`librt.so`. + +In addition to this, we also have to tell the linker to force the *segment +variables* (see `GeneratedIRReference.rst`_) at the appropriate location in +memory. In fact, each segment of the original binary should be loaded exactly at +the same address where it was originally supposed to be loaded. `revamb` emits +the required mapping automatically in a file with the same name as the output +file plus a ``.li.csv`` suffix. This file is a CSV file composed by three +columns: + +:name: The name of the section in the generated module containing the data of + the considered segment. +:start: Start address where the segment originally was located and, therefore, + where it should be placed by the linker. +:end: Corresponding end address. + +The `li-csv-to-ld-options` script converts this CSV file into parameters for the +linker to enforce the location of this sections. + +As a result of this operation, the actual translated code might end up in an +unusual location, but the linker should be able to figure this out. + +In conclusion, to link the final program: + +.. code-block:: sh + + gcc $(li-csv-to-ld-options translated.ll.li.csv) \ + translated.o \ + support.c \ + -DTARGET_arm \ + -lz -lm -lrt -Wno-pointer-to-int-cast -Wno-int-to-pointer-cast -g \ + -o translated.elf + +.. _`GeneratedIRReference.rst`: GeneratedIRReference.rst diff --git a/docs/GeneratedIRReference.rst b/docs/GeneratedIRReference.rst new file mode 100644 index 000000000..d8abebd29 --- /dev/null +++ b/docs/GeneratedIRReference.rst @@ -0,0 +1,493 @@ +*********************** +revamb Output Reference +*********************** + +The main goal of revamb is to produce an LLVM module reproducing the behavior of +the input program. The module should be compilable and work out of the +box. However, such module contains also a rich set of additional information +recovered during the analysis process that the user can exploit to develop any +kind of analysis he wants. + +This document details how to interpret the information present in the generated +module. Please refer to the `LLVM Language Reference Manual`_ for details on the +LLVM language itself. + +The various sections of this document will present example to clarify the +presented concepts. All the examples originate from the translation of a simple +program compiled for x86-64: + +.. code-block:: c + + int myfunction(void) { + return 42; + } + + int _start(void) { + int a = 42; + return a + myfunction(); + } + +The program has been compiled as follows: + +.. code-block:: sh + + x86_64-gentoo-linux-musl-gcc -static -nostdlib -O0 -fomit-frame-pointer example.c -o example + +Producing the following assembly: + +.. code-block:: objdump + + 00000000004000e8 : + 4000e8: mov eax,0x2a + 4000ed: ret + + 00000000004000ee <_start>: + 4000ee: sub rsp,0x10 + 4000f2: mov DWORD PTR [rsp+0xc],0x2a + 4000fa: call 4000e8 + 4000ff: mov edx,eax + 400101: mov eax,DWORD PTR [rsp+0xc] + 400105: add eax,edx + 400107: add rsp,0x10 + 40010b: ret + +And it has been translated as follows: + +.. code-block:: sh + + ./revamb --no-link --functions-boundaries --use-sections --debug-info ll example example.ll + +Global variables +================ + +The CPU State Variables +----------------------- + +The CPU State Variables (or CSV) are global variables that represent a part of +the CPU. They vary from architecture to architecture and they are created +on-demand, which means that not all modules will have all of them. + +Some CSV variables have a name, in particular registers (e.g., `rsp`), some +others are instead identified by their position within the QEMU data structure +that contains them (e.g., ``state_0x123``). For example: + +.. code-block:: llvm + + @pc = global i64 0 + @rsp = global i64 0 + @rax = global i64 0 + @rdx = global i64 0 + @cc_src = global i64 0 + @cc_dst = global i64 0 + @cc_op = global i32 0 + +`@pc` represents the program counter (aka `rip`), `@rsp` the stack pointer +register, while `@rax` and `@rdx` are general purpose registers. The `@cc_*` are +helper variables used to compute the CPU flags. + +CSVs are used by the generated code and by the helper functions. This is also +the reason why they cannot be promoted to local variables in the `root` +function` + +Note that since they are global variables, the generated code interacts with +them using load and store operations, which might sound unusual for registers. + +Segment variables +----------------- + +The translated program expects the memory layout to be exactly as the one in the +original binary. This means that all the segments have to be loaded at the +original addresses. In the generated module, they are encoded as global +variables containing all the data of the segments. These variables have a name +similar to ``.o_permissions_address`` (e.g., ``.o_rx_0x10000``), where +*permissions* it's a string representing what type of accesses are allowed to +that segment (read, execute, write), and *address* is the starting address. + +These variables are associated to special sections which will be assigned to the +appropriate virtual address at link-time. + +In our example we have single segment, readable and writable: + +.. code-block:: llvm + + @.o_rx_0x400000 = constant [344 x i8] c"\7FELF\02\01\01\0...", section ".o_rx_0x400000", align 1 + +As you can see it is initalized with a copy of the original segment and its +assigned to the `.o_rx_0x400000` section. + +Other global variables +---------------------- + +Apart from CSVs and segment variables, the output module will contain a number +of other global variables, mainly for loading purposes (see ``support.c``). In +the following we report the most relevant ones. + +:.elfheaderhelper: a variable whose only purpose is to create the + `.elfheaderhelper` section, which is employed to force an + appropriate layout at link-time. It isn't of general + interest. +:e_phentsize: size of the ELF program header structure of the input binary. +:e_phnum: number of ELF program headers in the input binary. +:phdr_address: virtual address where the ELF program headers are loaded. + +For more information on the ELF program headers, see ``man elf``. In the +example program we have three program headers of 56 bytes, loaded at +`0x400040`: + +.. code-block:: llvm + + @.elfheaderhelper = constant i8 0, section ".elfheaderhelper", align 1 + @e_phentsize = constant i64 56 + @e_phnum = constant i64 3 + @phdr_address = constant i64 4194368 + + +Input architecture description +============================== + +The generated module also contains a *named metadata node*: +`revamb.input.architecture`. Currently it's composed by a metadata tuple with +two values: + +:u32 DelaySlotSize: the size, in number of instructions of the delay slot of the + input architecture. +:string PCRegName: the name of the CSV representing the program counter. + +Here's how this information appears in our example: + +.. code-block:: llvm + + !revamb.input.architecture = !{!0} + !0 = !{i32 0, !"pc"} + +There is no delay slot on x86-64 and the CSV representing the program counter is +`@pc`. + +The `root` function +=================== + +This section describes how the function collecting all the translated code is +organized. This fuction is knonw as the `root` function: + +.. code-block:: llvm + + define void @root() { + ; ... + } + + +The dispatcher +-------------- + +The first set of basic blocks are related to the dispatcher. Every time we have +an indirect branch that we cannot fully handle we jump to the *dispatcher*, +which basically maps (with a huge ``switch`` statement) the starting address of +each basic block A in the input program to the first basic block containing the +code generated due to A. + +:``dispatcher.entry``: the body of the dispatcher. Containes the ``switch`` + statement. If the requested address has not been + translated, execution is diverted to + ``dispatcher.default``. +:``dispatcher.default``: calls the `unknownPC` function, whose definition is + left to the user. +:``anypc``: handles the situation in which we were not able to fully enumerate + all the possible jump targets of an indirect jump. Typically will + just jump to ``dispatcher.entry``. +:``unexpectedpc``: handles the situation in which we though we were able to + enumerate all the possible jump targets, but an unexpected + program counter was requested. This indicates the presence of + a bug. It can either try to proceed with execution going to + ``dispatcher.entry`` or simply abort. + +The very first basic block is `entrypoint`. It's main purpose is to create all +the required local variables (``alloca`` instructions) and ensure that all the +basic blocks are reachable. In fact, it is terminated by a ``switch`` +instruction which make all the previously mentioned basic blocks reachable. This +ensures that we can compute a proper dominator tree and no basic blocks are +collected as dead code. + +Here's how it looks like in our example: + +.. code-block:: llvm + + define void @root() { + entrypoint: + %0 = alloca i64 + %1 = bitcast i64* %0 to i8* + switch i8 0, label %bb._start [ + i8 1, label %dispatcher.entry + i8 2, label %anypc + i8 3, label %unexpectedpc + ] + + dispatcher.entry: ; preds = %unexpectedpc, %anypc, %bb.myfunction, %bb._start.0x11, %entrypoint + %2 = load i64, i64* @pc + switch i64 %2, label %dispatcher.default [ + i64 4194536, label %bb.myfunction + i64 4194542, label %bb._start + i64 4194559, label %bb._start.0x11 + ], !revamb.block.type !1 + + dispatcher.default: ; preds = %dispatcher.entry + call void @unknownPC() + unreachable + + anypc: ; preds = %entrypoint + br label %dispatcher.entry, !revamb.block.type !2 + + unexpectedpc: ; preds = %entrypoint + br label %dispatcher.entry, !revamb.block.type !3 + + ; ... + + } + +As you can see, we have three jump targets: `myfunction`, `_start` and +`_start+0x11` (the return address after the function call. In this specific +example we decide to divert execution to the dispatcher both in `anypc` and +`unexpectedpc`. + +The translated basic blocks +--------------------------- + +The rest of the function is composed by basic blocks containing the translated +code. If symbols are available in the input binary, each basic block has name in +the form ``bb.closest_symbol.distance`` (e.g., ``bb.main.0x4`` means 4 bytes +after the symbol `main`). Otherwise the name is simply in the form +``bb.absolute_address`` (e.g., ``bb.0x400000``). + +In our example we have three basic blocks: + +.. code-block:: llvm + + define void @root() { + ; ... + + bb._start: ; preds = %dispatcher.entry, %entrypoint + ; ... + + bb._start.0x11: ; preds = %dispatcher.entry + ; ... + + bb.myfunction: ; preds = %dispatcher.entry, %bb._start + ; ... + + } + +Debug metadata +-------------- + +Each instruction we generate is associated with three types of metadata: + +:dbg: LLVM debug metadata, used to be able to step through the generated LLVM IR + (or input assembly or tiny code). +:oi: *original instruction* metadata, contains a string-integer pair. The string + represents the disassembled input instruction that generated the current + instruction. The integer is the program counter associated to that + instruction. +:pi: *portable tiny code instruction* metadata, contains a string representing + the textual representation of the TCG instruction that generated the + current instruction. + +Note: some optimizations passes might remove the metadata. + +For debugging purposes, the generated LLVM IR contains comments with information +derived from these metadata. + +As an example, let's see the first instruction of `myfunction`, ``mov +eax,0x2a``: + +.. code-block:: llvm + + define void @root() { + + ; ... + + bb.myfunction: ; preds = %dispatcher.entry, %bb._start + ; 0x00000000004000e8: mov eax,0x2a + + ; movi_i64 tmp0,$0x2a + ; ext32u_i64 rax,tmp0 + store i64 42, i64* @rax, !dbg !135, !oi !133, !pi !136 + + ; ... + + } + + ; ... + + !4 = distinct !DISubprogram(name: "root", ...) + !133 = distinct !{!"0x00000000004000e8: mov eax,0x2a\0A", i64 4194536} + !134 = distinct !{!"movi_i64 tmp0,$0x2a\0A"} + !135 = !DILocation(line: 244, scope: !4) + !136 = distinct !{!"ext32u_i64 rax,tmp0,\0A"} + +The `!dbg` metadata points to a `DILocation` object, which tells us that we're +at line 244 within the `root` function. This information will allow the debugger +(e.g., `gdb`) to perform step-by-step debugging. `!oi` points to a metadata node +containing the diassembled instruction that lead to generate this instruction +and its address (`4194536`). Finally, `!pi` points to the TCG instruction +leading to the creation of this instruction. + +Above the instruction, we also have, for easier reading, the corresponding +original and TCG instructions. + +Delimiting generated code +------------------------- + +The code generated due to a certain input instruction is delimited by calls to a +marker function `newpc`. This function takes three arguments plus a set of +variadic arguments: + +:u64 Address: the address of the instruction leading to the generation of the + code coming after the call of `newpc`. +:u64 InstructionSize: the size of the instruction at `Address`. +:u1 isJT: a boolean flag indicating whether the instruction at `Address` is a + jump target or not. +:u8 \*LocalVariables: a series of pointer to all the local variables used by + this instruction. + +The call to `newpc` prevents the optimizer to reorder instructions across its +boundaries and perform other optimizations. This is useful during analysis and +for debugging purposes, but to achieve optimal performances all these function +calls should be removed. + +Let's see how this works for the `bb.myfunction` basic block: + +.. code-block:: llvm + + bb.myfunction: ; preds = %dispatcher.entry, %bb._start + + ; 0x00000000004000e8: mov eax,0x2a + call void (i64, i64, i32, i8*, ...) @newpc(i64 4194536, i64 5, i32 1, i8* null), !oi !55, !pi !56 + + ; ... + + ; 0x00000000004000ed: ret + call void (i64, i64, i32, i8*, ...) @newpc(i64 4194541, i64 1, i32 0, i8* null), !oi !58, !pi !59 + + ; ... + +As you cane see there are two calls to `newpc`, the first for the ``mov`` +instruction at ``0x4000e8`` (5 bytes long) and the second one for the `ret` +instruction at ``0x4000ed`` (1 byte long). Note that the first instruction is a +jump target, in fact `newpc`'s third parameter is set to ``1``, unlike the +second call. + +Function calls +-------------- + +revamb can detect function calls. The terminator of a basic block can be +considered a function call if it's preceeded by a call to a function called +`function_call`. This function take three parameters: + +:BlockAddress Callee: reference to the callee basic block. The target of the + function call, most likely a function. +:BlockAddress Return: reference to the return basic block. It's the basic block + associated to the return address. +:u64 ReturnPC: the return address. + +In our example we had a function call in the `_start` basic block: + +.. code-block:: llvm + + bb._start: ; preds = %dispatcher.entry, %entrypoint + + ; ... + + ; 0x00000000004000fa: call 0x4000e8 + + ; ... + + store i64 4194536, i64* @pc, !dbg !58, !oi !46, !pi !59 + call void @function_call(i8* blockaddress(@root, %bb.myfunction), i8* blockaddress(@root, %bb._start.0x11), i32 4194559), !dbg !60 + br label %bb.myfunction, !dbg !61, !func.entry !62, !func.member.of !63 + +As expected, before the branch instruction representing the function call, we +have a call to `@function_call`. The first argument is the callee basic block +(`bb.myfunction`), the second argument is the return basic block (`_start+0x11`) +and the third one is the return address (``0x4000ff``). + +Function boundaries +------------------- + +revamb can identify function boundaries. This information is also encoded in the +generated module by associating two types of metadata (`func_entry` and `func`) +to the terminator instruction of each basic block. + +:func.entry: denotes that the current basic block is the entry block of a + certain function. The associated metadata tuple contains a single + string node representing the name assigned to the function. + +:func.member.of: denotes that the current basic block is part of a set of + functions. The associated metadata tuple contains a set of + string nodes representing the name assigned to the + corresponding functions. + +In our example we had three basic blocks: `_start`, `_start+0x11` and +`myfunction`. Let's see to what function they belong: + +.. code-block:: llvm + + define void @root() !dbg !4 { + + ; ... + + bb._start: ; preds = %dispatcher.entry, %entrypoint + ; ... + br label %bb.myfunction, !func.entry !62, !func.member.of !63 + + bb._start.0x11: ; preds = %dispatcher.entry + ; ... + br label %dispatcher.entry, !func.member.of !63 + + bb.myfunction: ; preds = %dispatcher.entry, %bb._start + ; ... + br label %dispatcher.entry, !func.entry !151, !func.member.of !152 + + ; ... + + } + + ; ... + + !62 = !{!"bb._start"} + !63 = !{!62} + ; ... + !151 = !{!"bb.myfunction"} + !152 = !{!151} + +As it can be seen, `bb._start` and `bb._start.0x11` belong to a single function, +identified by `bb._start`. `bb._start` is also marked as the entry point of the +function. On the other hand, `bb.myfunction` also belongs to (and it's the entry +point of) a single function, with the same name. + +Helper functions +================ + +Certain features of the input CPU would be to big to be expanded in TCG +instructions by QEMU (and therefore translate them in LLVM IR). For this reason, +call to *helper functions* are emitted. An example of an helper function is the +function handling a syscall or a floating point division. These functions can +take arguments and can read and modify freely all the CSV. + +Helper functions are obtained from QEMU in the form of LLVM IR (e.g., +``libtinycode-helpers-mips.ll``) and are statically linked by revamb before +emitting the module. + +The presence of helper functions also import a quite large number of data +structures, which are not directly related to revamb's output. + +Note that an helper function might be present multiple times with different +suffixes. This happens every time an helper function takes as an argument a +pointer to a CSV: for each different invocation we specialize that callee +function by fixing that argument. In this way, we can deterministically know +which parts of the CPU state is touched by an helper. + +Currently, there is no complete documentation of all the helper functions. The +best way to understand which helper function does what, is to create a simple +assembly snippet using a specific feature (e.g., a performing a syscall) and +translate it using revamb. + +.. _LLVM Language Reference Manual: http://llvm.org/docs/LangRef.html diff --git a/docs/PythonExample.rst b/docs/PythonExample.rst new file mode 100644 index 000000000..94dfd7236 --- /dev/null +++ b/docs/PythonExample.rst @@ -0,0 +1,191 @@ +*************************************************** +Using `revamb` with Python: a simple instrumenation +*************************************************** + +In this document we'll guide the user through using `revamb`'s output from +Python. Amomng the many possibilities that arouse from the LLVM IR provided by +`revamb`, in this document we'll show how it's possible to perform a simple +instrumentation of an existing binary, by injecting some code in the generated +LLVM IR and recompiling it. + +To manipulate LLVM IR from Python we'll use `llvmcpy`_, a lightweight wrapper +around the LLVM-C API for Python. + +An Hello World for ARM +====================== + +Let's first of all create a simple program in C called `hello.c`: + +.. code-block:: c + + #include + + int main(int argc, char *argv[]) { + printf("Hello world!\n"); + return 0; + } + +We can compile `hello.c` for ARM and link it statically: + +.. code-block:: sh + + armv7a-hardfloat-linux-uclibceabi-gcc hello.c -o hello -static + +Using the `translate`_ tool we can have `revamb` produce the +LLVM IR and recompile it for us. The output should be a working +``hello.translated`` program for x86-64 (our host architecture): + +.. code-block:: sh + + $ translate hello + $ ./hello.translated + Hello world! + +All good! + +Tracing all the syscall invocations +=================================== + +For this example, we'll write a simple Python script (``instrument.py``) which +takes in input the `revamb` generated LLVM IR, identifies all the syscalls and +instrument them injecting the code to print the number of syscall to be +performed. + +In the ARM architecture, syscalls are expressed as calls to a function whose +name begins with `helper_exception_with_syndrome_`. After this prefix there is +an index identifying the specialization of the function, as explained in +`GeneratedIRReference.rst`_, but this is not interesting for us in this case. + +Once we identified all the calls to the said functions, we will simply load the +value of the `r7` register, which holds the number of the syscall, and print it +to `stderr` using the `dprintf` function. + +First of all we need to import `llvmcpy`_,obtain the default `LLVMContext` +object and load the input LLVM IR: + +.. code-block:: python + + from llvmcpy import llvm + context = llvm.get_global_context() + buffer = llvm.create_memory_buffer_with_contents_of_file(sys.argv[1]) + module = context.parse_ir(buffer) + +Now that we a reference to the module produced by `revamb` we can collect the +objects required to perform the `dprintf` call, i.e., the function itself, the +CSV representing the register `r7`, a constant integer representing `stderr` and +the format string for `dprintf`: + +.. code-block:: python + + r7 = module.get_named_global("r7") + + dprintf = module.get_named_function("dprintf") + + two = context.int32_type().const_int(2, True) + + message_str = context.const_string("%d\n", 4, True) + message = module.add_global(message_str.type_of(), "message") + message.set_initializer(message_str) + message_ptr = message.const_bit_cast(context.int8_type().pointer(0)) + +Note that to build the format string we first have to create a new global +variable, then set its initializer with the constant string and finally cast it +to `int8 *` so that can be passed to `dprintf`, whose prototype is: + +.. code-block:: c + + int dprintf(int fd, const char *format, ...); + +At this point we have to iterate over all the instructions of the function +containing the generated code (`root`): + +.. code-block:: python + + root_function = module.get_named_function("root") + for basic_block in root_function.iter_basic_blocks(): + for instruction in basic_block.iter_instructions(): + # ... + +However, we are not interested in all instructions, but only in calls to +`helper_exception_with_syndrome_*` functions. Therefore, we check the opcode of +the instruction, and, if it's a call, we consider the last operand (which +represents the called function) and check it's name: + +.. code-block:: python + + if instruction.instruction_opcode == llvm.Call: + + last_operand_index = instruction.get_num_operands() - 1 + callee = instruction.get_operand(last_operand_index) + + if not callee.name: + assert(callee.get_num_operands() == 1) + callee = callee.get_operand(0) + + if callee.name.startswith("helper_exception_with_syndrome_"): + # ... + +Note that the called function is often casted to a slightly different function +type, but we are not interested in this cast. The ``if not callee.name:`` block +handles this situation by moving to the first operand of the cast instruction. + +Finally, we've found a location where we want to insert our instrumentation. To +do this, we create a *builder* object, position it right before the call +instruction, emit an instruction to load `r7`, prepare the other arguments and, +finally, emit the call to `dprintf`: + +.. code-block:: python + + builder = context.create_builder() + builder.position_builder_before(instruction) + load_r7 = builder.build_load(r7, "") + builder.build_call(dprintf, [two, message_ptr, load_r7], "") + +That's all. The last thing left to do is to serialize the new IR to file: + +.. code-block:: python + + module.print_module_to_file(sys.argv[2]) + +Let's now run our script and recompile the code: + +.. code-block:: sh + + $ mv hello.ll hello.ll.original + $ python instrument.py hello.ll.original hello.ll + $ translate -s hello + $ ./hello.translated + 45 + 45 + 983045 + 5 + 3 + 6 + 54 + 54 + 4 + Hello world! + 248 + +We can compare the result with a QEMU run of the original program: + +.. code-block:: sh + + $ qemu-arm -strace hello + 7346 brk(NULL) = 0x00039000 + 7346 brk(0x000394b0) = 0x000394b0 + 7346 open("/dev/urandom",O_RDONLY) = 3 + 7346 read(3,0xf6ffde84,4) = 4 + 7346 close(3) = 0 + 7346 ioctl(0,21505,-151003688,0,221184,0) = 0 + 7346 ioctl(1,21505,-151003688,1,221184,0) = 0 + 7346 write(1,0x372a8,13)Hello world! + = 13 + 7346 exit_group(0) + +The complete `instrument.py` script is available in `docs/instrument.py`_. + +.. _llvmcpy: https://rev.ng/llvmcpy +.. _translate: TranslateUsage.rst +.. _`GeneratedIRReference.rst`: GeneratedIRReference.rst +.. _`docs/instrument.py`: instrument.py diff --git a/docs/RevambDumpUsage.rst b/docs/RevambDumpUsage.rst new file mode 100644 index 000000000..03e1b717b --- /dev/null +++ b/docs/RevambDumpUsage.rst @@ -0,0 +1,49 @@ +*********** +revamb-dump +*********** + +---------------------------------------- +extract information from `revamb` output +---------------------------------------- + +:Author: Alessandro Di Federico +:Date: 2016-12-22 +:Copyright: MIT +:Version: 0.1 +:Manual section: 1 +:Manual group: rev.ng + +SYNOPSIS +======== + + revamb-dump [options] INFILE + +DESCRIPTION +=========== + +`revamb-dump` is a simple tool to extract some high level information from the +IR produced by `revamb`. + +OPTIONS +======= + +Note that all the options specifying a path support the special value ``-`` +which indicates ``stdout``. Note also that `revamb-dump` expresses the *name of +a basic block* as represented by `revamb` in the generated module (typically +``bb.0xaddress`` or ``bb.symbol.0xoffset``. + +:``-c``, ``--cfg``: Path where the control-flow graph should be stored. The + output will be a CSV file with two columns: `source` and + `destination`. Both of them contain the name of a basic + block. +:``-n``, ``--noreturn``: Path where the list of the ``noreturn`` basic block + should be stored. The output will be a CSV file with a + single column `noreturn`, containing the name of the + ``noreturn`` basic block. +:``-f``, ``--function-boundaries``: Path where the list of *function*<->*basic + block* pairs should be stored. The output + will be a CSV file with two column: + `function`, the name of the entry basic + block of the function, and `basicblock`, the + name of a basic block belonging to + `function`. diff --git a/docs/RevambUsage.rst b/docs/RevambUsage.rst new file mode 100644 index 000000000..b8da048cd --- /dev/null +++ b/docs/RevambUsage.rst @@ -0,0 +1,87 @@ +****** +revamb +****** + +--------------------------------------------------------- +lift a binary from any architecture to compilable LLVM IR +--------------------------------------------------------- + +:Author: Alessandro Di Federico +:Date: 2016-12-22 +:Copyright: MIT +:Version: 0.1 +:Manual section: 1 +:Manual group: rev.ng + +SYNOPSIS +======== + + revamb [options] [--] INFILE OUTFILE + +OPTIONS +======= + +`revamb` options allow to customize its output. In the following the main ones +are described: + +:``-e``, ``--entry``: Tell `revamb` to ignore the program entry point (typically + the `_start` function) and start to translated code at the + specified address (as *decimal* number). Note that this + option also disables the harvesting of the global data in + search for code pointers (see + `docs/GeneratedIRReference.rst`). This option is mainly + useful to test how `revamb` acts on a limited portion of + code without translating the whole program. Default: entry + point specified by the ELF header (``ElfN_Ehdr.e_entry``). +:``-i``, ``--linking-info``: Path where the CSV containing instructions for the + linker on how to position the segment variables + (see `docs/FromIRToExecutable.rst`). Default: + ``OUTFILE.li.csv``. +:``-g``, ``--debug-info``: Type of debug information to associate to the + generated module, i.e. the type of *source code* to + use in step-by-step debugging of the generated + program (e.g., using `gdb`). Possible values are: + `none` for no debugging information, `asm` to dump + the disassembly of the input code, `ptc` to dump TCG + instructions and `ll` to dump the LLVM IR. + ``--debug-path`` specifies the location of the + output. Default locations are ``OUTFILE.S`` for + `asm`, ``OUTFILE.ptc`` for `ptc` and ``OUTFILE`` + itself for `ll`. +:``-s``, ``--debug-path``: Path where the *debug source* should be saved. See + ``--debug-info`` for additional information and + default value. +:``-c``, ``--coverage-path``: Path where the list of the address ranges that + have been translated should be stored (in CSV + form). This option is deprecated in favor of using + `revamb-dump` and will be removed. Default: + ``OUTFILE.coverage.csv`` +:``-d``, ``--debug``: Enable verbose debugging info during execution. The + argument is a comma-separated list of keywords identifying + a type of debug information. This debug information are + for development usage and possible values can be + identified directly in the source code in the first + argument of the ``DBG(...)`` macro. +:``-O``, ``--no-osra``: Disable OSR Analysis. Useful for debugging purposes to + evaluate its effectiveness. +:``-L``, ``--no-link``: Do not link in QEMU helper functions. The call will + appear as calls to extern functions. This option makes + the resulting module non-functioncal but it's useful for + analysis-only purposes. +:``-t``, ``--tracing``: Function calls to `newpc` (which delimit the code + generated by an input instruction from another) target + an external function that the use can implement (see + `docs/FromIRToExecutable.rst`) instead of calling a + no-op function. This is useful to execute a certain + action, or inspect the CPU state at each instruction. +:``-S``, ``--use-sections``: If they are available, ELF sections are employed to + identify executable code. This options is useful to + evaluate `revamb` ignoring the code identification + issue. +:``-b``, ``--bb-summary``: Output path for the CSV containing statistics about + the translated basic blocks. This option is + deprecated in favor of using `revamb-dump` and will + be removed. Default: ``OUTFILE.bbsummary.csv``. +:``-f``, ``--function-boundaries``: Enable function boundaries detection. This + process currently can be quite expensive and + it's therefore disabled by default. diff --git a/docs/TranslateUsage.rst b/docs/TranslateUsage.rst new file mode 100644 index 000000000..c26cc12db --- /dev/null +++ b/docs/TranslateUsage.rst @@ -0,0 +1,42 @@ +********* +translate +********* + +---------------------------------------------------------------- +translate a program from an architecture to another using revamb +---------------------------------------------------------------- + +:Author: Alessandro Di Federico +:Date: 2016-12-22 +:Copyright: MIT +:Version: 0.1 +:Manual section: 1 +:Manual group: rev.ng + +SYNOPSIS +======== + + revamb-dump [options] [--] INFILE [revamb options] + +DESCRIPTION +=========== + +`translate` is a simple wrapper script which using `revamb` to lift an input +program to LLVM IR and then compiling it to x86-64. + +In practice, `translate` first invokes `revamb` then, depending on the options, +some optimizations are performed using `llc` and or `opt`, and finally the +generated object file is linked against the require libraries and `support.c`. + +Options after `INFILE` are forwarded as is to `revamb`. + +OPTIONS +======= + +:``-O0``: Disable optimizations both in the mid-end (`opt`) and the backend + (`llc`). This is default option. +:``-O1``: Enable backend optimizations (`llc`). +:``-O2``: Enable optimizations both in the mid-end (`opt`) and the backend + (`llc`). +:``-s``: Skip invoking `revamb`, assumes a file named `INFILE.ll` already + exists. This is useful for optimizing previously generated code. diff --git a/docs/instrument.py b/docs/instrument.py new file mode 100644 index 000000000..033e7476d --- /dev/null +++ b/docs/instrument.py @@ -0,0 +1,68 @@ +#!/usr/bin/env python + +import sys +from llvmcpy import llvm + +def main(): + context = llvm.get_global_context() + + # Load and parse the input file + buffer = llvm.create_memory_buffer_with_contents_of_file(sys.argv[1]) + module = context.parse_ir(buffer) + + # Identify the CSV (global variable) for the register containing the syscall + # number (r7 for ARM) + r7 = module.get_named_global("r7") + + # Prepare requirements to build our call to dprintf + + # 1) Identify the dprintf function (prototype) + dprintf = module.get_named_function("dprintf") + + # 2) Create an integer constant for stderr + two = context.int32_type().const_int(2, True) + + # 3) Create the message to print + message_str = context.const_string("%d\n", 4, True) + message = module.add_global(message_str.type_of(), "message") + message.set_initializer(message_str) + message_ptr = message.const_bit_cast(context.int8_type().pointer(0)) + + # Identify the function generated by revamb (root) + root_function = module.get_named_function("root") + + # Loop over all the generated instructions and add instrumentation code + builder = context.create_builder() + for basic_block in root_function.iter_basic_blocks(): + for instruction in basic_block.iter_instructions(): + # Check if the current instruction is a call + if instruction.instruction_opcode == llvm.Call: + # The callee is the last operand + last_operand_index = instruction.get_num_operands() - 1 + callee = instruction.get_operand(last_operand_index) + + # If there's a bitcast, skip it + if not callee.name: + assert(callee.get_num_operands() == 1) + callee = callee.get_operand(0) + + # Check if it's performing a syscall + if callee.name.startswith("helper_exception_with_syndrome_"): + # Set the builder's insert point before the call instruction + builder.position_builder_before(instruction) + + # Load the value of r7 (which contains the syscall number) + load_r7 = builder.build_load(r7, "") + + # Create the list of arguments for dprintf + arguments = [two, message_ptr, load_r7] + + # Create the function call to dprintf: + # dprintf(2, "%d\n", r7) + builder.build_call(dprintf, arguments, "") + + # Produce the instrumented LLVM IR to file + module.print_module_to_file(sys.argv[2]) + +if __name__ == "__main__": + main()