Commit Graph

49 Commits

Author SHA1 Message Date
Eric Kilmer aa49fcc4fb Support LLVM 11 (#729)
* Add guards based on LLVM version for 11 compatibility

* Use remill LLVM 11 compat headers

* CI support for LLVM 11
2021-02-13 11:22:32 -05:00
Peter Goodman 21970a5cfd Fixes one egregious bug in how we alias registers. What would happen is that AL/AH, DL/DH, etc. would all map to the same alias because their containing register and types would be the same. Modified register aliases to have the form <name>_<offset>_<type>. Also fixed an issue in mcsema-disass where the main function wouldn't be recognized as a function, and thus not lifted. (#710) 2020-11-11 18:07:48 -05:00
Peter Goodman b0f7790f43 Sparc support (#709)
* Adds sparc support; formats files

* Update the readme and walkthrough

* Minor tweaks to readme/walkthrough

* Update ci.yml
2020-11-05 12:59:20 -05:00
Peter Goodman c96402c313 Fixes issue identified by BoLei related to a function that has no associated basic blocks in the CFG (#707) 2020-10-25 22:32:37 -04:00
Peter Goodman 31f267bd5f Cmake refactor (#693)
* Update to latest remill APIs and way of using CMake.

* Minor fixes

* Bug fixes for AArch64

* Tweaks

* Adds some nifty functionality useful for debugging some aarch64 bugs

* Fixes a subtle bug

* Bug fixes

* Make the test suite use explicit args

* Bug fix in xrefs

* Disable tests for now
2020-09-22 23:22:56 -04:00
Peter Goodman c002b8505c Formats the code with clang-format and my additional stylistic mojojojo (#682) 2020-08-05 16:14:01 -04:00
Peter Goodman 2d55d02cc3 API improvements. Must be used with the api_improvements branches of Remill and Anvill (#666)
* API improvements. Must be used with the api_improvements branch of both Remill and McSema

fixes for x86 and running the lifted code with klee

* Update dockerfile to clone anvill

* update remill commit id

* Add python3 to dockerfile

* update python3

* disable abi script

* Updated cmake to find anvill

* Update main.cpp

* update find_package for anvill

* WIP:updated prebuild cfg

* update prebuild cfg files

* enable abi build for testsuite

* Fix memory leak

* install missing package for testcases

* frontend: Reflect cfg file changes in dyninst frontend.

* frontend: Update local files copyrights to reflect overall change to agplv3.

* fix failing testcases

* update test cfgs

* Fix test failure with local state pointer

* set the flag to use local state_ptr in default mode

Co-authored-by: kumarak <iit.akshay@gmail.com>
Co-authored-by: Lukas Korencik <xkorenc1@fi.muni.cz>
2020-07-23 02:14:56 -04:00
Aiethel 8ecfe165ef Various improvements (#628)
* bc: Inline calls to semantic functions and wrappers manually.

* bc: Inline calls to semantics and wrappers only if some flags are set.

* Refactor code that is responsible for abi libs loading.

* Wrap global pointers in RAII.

* Update .remill_commit_id

Co-authored-by: Peter Goodman <peter.goodman@gmail.com>
2020-02-10 13:20:05 -05:00
Auxy 8c8803d7c7 Additional check to prevent null target_segment (#627)
Co-authored-by: Peter Goodman <peter.goodman@gmail.com>
2020-01-21 13:18:03 -05:00
Peter Goodman 46362c1f69 Function annotations (#619)
* Annotate lifted functions

* Annotate and Tie entrypoints

* Annotate external functions

* Annote _attach_call

* Update comments

* Simplify code that generated VerifyRegState

Co-authored-by: Aiethel <korencikl@gmail.com>
2020-01-08 17:35:35 -05:00
Peter Goodman e13d8ffbc9 Llvm 900 (#605)
* Support for LLVM 8.0.0 and 9.0.0

* remill script fixes

* Update to master
2019-11-18 21:53:23 -05:00
Peter Goodman 782bd1dcfa Update for new remill APIs (#595)
* Update for new remill APIs

* Update remill commit id

* Cleanups

* resolved one issue
2019-10-27 17:48:43 -04:00
Peter Goodman e1a3a91aa8 Fix llvm 7 error (#534)
* Fix llvm 7 error

* Update Function.cpp

* Update Function.cpp
2019-03-22 16:21:54 -04:00
Peter Goodman 7d2dfe6e94 Attempt to resolve Issue #521 (#522)
* Attempt to resolve Issue #521

* minor fix

* minor fix

* minor fix

* Simplifications and comments

* If it's a copy of shared data, then don't treat it as an xref

* Error log if there is segment variable with the same name as external var

* Added flag and checks if the variable needs initialization if already defined;

* fix noreturn for the functions throwing exceptions;

* review changes

* Update get_cfg.py

* Update get_cfg.py

* Fix the recovery of exception action table chain

* Update the prebuild cfg for testsuite

* Fix entrypoint function not getting recovered in ida7

* Change the order of idc.GetFunctionName and idc.GetTrueName while looking for function name
2019-03-02 20:59:08 -05:00
Aiethel 5984250bc6 Native function improvements (#518)
* NativeFunction function is properly set

* Remove public keyword from structs
2019-02-26 11:07:54 -05:00
Aiethel 8eb4cbc16d Devirtualization can only happen in read_only segment (#475) 2018-09-26 11:05:18 -07:00
Peter Goodman 69cb35cc16 Call early init lazy xrefs (#466)
* More broadly applies lazy xref initialization to all initializations where the xref size is less than teh value size of the xref itself (pointer size, usually). Moves initialization of lazy xrefs into a new function, __mcsema_early_init, which guards itself against multiple executions. Makes sure that __mcsema_constructor calls __mcsema_early_init first. Finally, makes sure that all native-to-lifted entrypoints call __mcsema_early_init as well, as we have observed cases where the lifted binary contains weak implementations of c++ standard library functions, and these functions are called by native libraries initialized before the call to __mcsema_constructor, thereby resulting in re-entrancy issues.

* Update Function.cpp

* Make sure the __mcsema_early_init guard is in the module.

* Make sure the __mcsema_early_init guard is in the module.

* Fixes for __gmon_start__ and stuff.

* Minor fix

* Adds some symbols

* Minor fix for ida7

* Try to resolve things like calls through PLT thunks via xref entry lookups when a flow cross-reference is missing.
2018-09-23 15:25:14 -07:00
Peter Goodman 23800ebb8a Fstack protector (#434)
* Adds a new command-line option that adds a request for stack protection on lifted functions.

* Add the stack protector request to more locations

* Add gets to the libc abi.
2018-07-10 09:49:44 +10:00
Peter Goodman bda3af4c56 Dead store elim (#428)
* Update to use remill's dead store elimination, and be compatible with the indirection-removal done by Arch::PrepareModule.

* Fixes

* Update .remill_commit_id

* Update .remill_commit_id

* Update .remill_commit_id

* Update .remill_commit_id

* Update .remill_commit_id

* Update .remill_commit_id
2018-06-28 21:05:31 -04:00
kumarak a708bafc42 InlineAsm function type mismatch (#409)
* InlineAsm fails due to function type mismatch

* disable ada failing testcases;

* Generate ABI library for integration test

* Change ArrayRef to vector; getting corrupted for llvm >= 3.9
2018-05-18 05:58:34 -04:00
Peter Goodman a2fa36ab01 Llvm 600 compatibility (#399)
* Update code for LLVM 6.0 compatibility

* Move the OS dir.

* Minor edit

* Update .remill_commit_id

Update to LLVM 6.0-compatible Remill branch.
2018-04-13 21:25:45 -04:00
kumarak a3483e4bf1 C++ exception handling support (#391)
* Initial changes to parse the eh_frame and exception table

* linearly scan the function to find missing blocks

* Update protobuf format for lp; need to tag instructions or basic blocks to associate it with the lp

* tag instructions with lp; removed debug logs

* updated comments and protobuf; use utility functions for pointer size;

* travis build failure; make lp_ea optional field

* Collect the exception handling bb chunks; Create a passthrough for the lifted landingpad;

* Generate the passthrough IR and attach the landing pad with function invoke

* Generate the eh_frame/exception table and add personality function with one raising the exception.

* catch/cleanup wrapper for the exception handling; Disable CFG simplification pass temporarily for invoke inst;

* invoke wrapper to fix stack and base pointers

* recover type indices of the exception type

* fix exception table offset

* Add the function definition for setting the rsp/rpb correctly after eh return;

* Changes for abi_library support in exception handling

* Remove unwind attribute from ext_* functions to generate eh table

* Support the lifting of multiple exception types thrown;

* Fix the multiple declaration of personality function in lifted bitcode;

* Add --recover_exception flag to lift the exception table

* Update the list of external functions with c++ ABI's; Fix basic block split issue for exception chunks;

* Weak linkage for external function causing the garbage value for out_of_range destructor. Temporarily disabled;

* cleanup redundant changes and resolve conflicts

* Update --std-defs list with the c++ exception ABIs

* Fix merge fail

* exception test binaries for integration test; Move to clang++ for recompiling the bitcode.

* Fix the exception handling support for llvm 3.5

* Disable the optimization passes for the function having landingpad;

* Temporary changes to add ABI libraries for exception tests.

* Remove the noreturn attribute to fix the problem with -03 optimization

* review comment changes

* Change wrapper function name across invoke;

* review changes

* Updated comments text

* Remove duplicate clauses from the list; Add arguments for personalityfn and fix review comments.

* Update the flag name to --exception_personalityfn;

* Update the flag name to --exception_personality_func;

* Follow the variable naming convention;

* Update comments

* travis build fix for unused variable

* Create the map of type indices from original binary. Runtime routine to fix the RDX with correct type index;

* Add _Unwind_Resume to --abi_libraries functions list;

* Update comments & code restructure

* Exception handling changes and utility functions for 32 bit;

* DWORD for 32-bit system
2018-04-12 14:05:45 -04:00
Peter Goodman 8a9856ada3 Klee maze example (#369)
* In progress. Working on an example of using KLEE on a Maze, but with the maze program being compiled to x86, amd64, and aarch64.

* Making lots of progress on getting lifting and runnning an aarch64 maze program on amd64, but using --explicit_args. The key thing I'm working through right now is a jump offset table, but where the offset is a block pc, rather than a table base. Also adding various bits of code here and there to making runnning with klee more directly doable, and working on a debugging facility to track down when the emulated program counter gets out of sync with the original program.

* Fixed a subtle @PAGE and @PAGEOFF-related reference bug on AArch64. Partially disabled the special jump offset table handling I had in table.py, as it doesn't (yet) handle the shifted table values. However, I still have the code there, so that it can recognize that a basic block address is used as a possible offset, so that I can remove the block address as a reference, which permits a new heuristic on the C++ side to work. On the C++ side, when there's a jump instruction that isn't associated with a cross-reference flow, I try to auto-augment it with addition switch cases, targeting blocks with no predecessors (as present in the CFG). This seems to work reasonably well.

* Improved the scripts and updated the READMEs.

* Minor rephrase

* Minor rephrase
2018-01-13 23:47:55 -05:00
Peter Goodman e621164a7c Fixues issue #321. (#331)
* Fixues issue #321.

* This is an attempt to fix the issue Akshay observed, though not by restoring the original code. In a windows binary I am looking at, I have the jump table entries as 4-byte displacements, added to an 8-byte offset.

* This includes some pretty substantial changes. Some are related to jump tables, then things diverged into better handling of .idata sections in PE files, and specifically external calls and devirtualizing them in the lifted bitcode.
2017-11-19 23:51:57 -05:00
Peter Goodman a68f40142a Issue 323 unimplemented avx instructions (#328)
* Updates mcsema to understand that the InstructionLifter returns a LiftStatus now, instead of a bool. This allows us to partially resolve Issue #323. Next up is instruction support.

* Add missing function to runtime.
2017-11-16 21:40:02 -05:00
kumarak 11149d25e1 [WIP] Lift the stack variables using Mcsema2 (#303)
* collect the references of stack variables

* Initial changes to lift stack variables

* Set the stack variable name

* remove the entryBlock for allocating the stack variables

* addressed review comments

* Handling index and scale associated with stack variable; Build will fail because of the required changes in remill;

* Utility to load the register addresses; remove the dependency on remill changes

* Disable function parameters lifting;

* Fixed review comments

* IDA changes to lift the stack variables for apache ATD's. Issue with lifting some variables from status_handler;

* Change CFG proto names to match with other classes

* Remove debug logs

* Change stack allocation variables and general cleanup; Update the blacklist functions for which the stack shuffling does not work;

* Fixed code alignment

* fix tab indentation with spaces
2017-11-14 15:20:31 -05:00
Peter Goodman 497d4dbe04 Pretty substantial changes. I removed --partition_segments. Now all idautils.Entries that are in the data sections are treated as segment split points. This is all to handle Apache, which exports its symbols to its shared libs (apr and apr-util). 2017-08-23 00:10:49 -04:00
Peter Goodman 89c42a3e42 Support for jump offset tables in the bitcode. Different handling of ELF GOT pointers. It might still be a bit wonky for stdin/stdout.. not sure. 2017-08-22 13:05:20 -04:00
Peter Goodman 04e675ac48 Remove tail calls, fast calling convention, and fixup get_cfg.py to better support segmenting existing segments using global variables 2017-08-18 22:41:49 -04:00
Peter Goodman df1678412b Adds support for --legacy_mode, where drivers are added for the entrypoint and the callbacks 2017-08-18 15:09:25 -04:00
Peter Goodman 18abc11714 Added new --legacy_mode operation that tries to produce bitcode that is more similar to old bitcode. Also updates get_cfg.py to disassemble in such a way that mcsema2 will be able to support global variable recovery as though they were distinct segments 2017-08-16 23:06:53 -04:00
Peter Goodman a7609d1110 Made mcsema-disass recognize .idata section as containing externals. Provided preliminary implementation of __remill_sync_hyper_call in Runtime.cpp. Added a check to detect jump tables embedded in code so as to make mcsema-disass not think they are actual instructions. This came up with Notepad++. Added initial support that will eventually lead to be able to lift and run c++ code. McSema can now inject two functions, __mcsema_constructor and __mcsema_destructor into the bitcode. The constructor is added to the llvm.global_ctors array. This is there to handle relocations that can't actually be handled statically, e.g. having a global variable contain the value of stderr, which is an external. There's two new corresponding command-line options as well. --libc_constructor lets you tell mcsema that a specific function should be called before main executes. For example, for an ELF binary you would pass __libc_csu_init. In the case of C++, this should do some of the work of bringing up the C++ runtime support needed if the lifted program contains and globals that are constructed pre-main. Still to do is to handle jump tables that go backwards. This shows up in Notepad++, where we see: neg ecx; jmp ds:last_table_entry[ecx];. 2017-07-22 21:06:49 -04:00
Peter Goodman b6ab3f9c43 Swapped order of args to the lifted subroutines. Updated the callback assembly stubs to use an inline assembly feature I was previously unaware of. That inline asm change fixes one issue related to using llvm 3.6, which is that the callback functions would be compiled without their tail calls, and that would cause a mess of issues. 2017-07-19 17:35:07 -04:00
Peter Goodman 834adb7991 More progress on calling convention stuff. Moved some existing code around. Added implementations for the runtime argument getting stuff. Added functions for setting the return value. Not well tested. (#267) 2017-07-10 11:57:10 -04:00
car bauer 82d70afe5c Explicit args [WIP] (#265)
* explicit args to external APIs
* Make AArch64 disassembly detect refs to external functions
* Fix awful bug where code would become undefined since our callinst was not same convention as the called function
* Emit enough code so that AArch64 bitcode gets emitted
2017-06-29 13:34:51 -04:00
Peter Goodman 9dfedeef52 Updates to make code work by passing remill::Instruction by reference. 2017-06-14 21:32:23 -04:00
Peter Goodman 14c970c296 Changed the calling convention of breakpoint types. Have the old (fast) cc caused llvm to optimize away the contents of entire functions??? 2017-05-31 11:40:57 -04:00
Peter Goodman c1bd1668f4 Minor fixes here and there. May not yet build! 2017-05-25 11:00:02 -04:00
Peter Goodman 8452475933 Big improvements to get_cfg.py, especially for SQLite. Still, SQLite doesn't lift correctly. Also, ls using -la segfaults, although -lax does work. 2017-05-06 18:04:17 -04:00
Peter Goodman 939f408d69 Lifting and recompiling a simple hello world program is working. This uses a new runtime that does more direct register conversion between lifted and native. Updated the .gdbinit debugging script to work using the remill State structure. 2017-05-01 22:51:41 -04:00
Peter Goodman 5e508962f8 Fixes for lifting. Simple lifted binaries can now be recompiled when using the alessandro/refactor/llvmmultiversion-cmake branch of remill as a base. 2017-04-29 22:06:24 -04:00
Peter Goodman 0c8f2ec21f Minor updates. Trying to diagnose an issue where llvm optimizes a ton of stuff into unreachable insts. 2017-04-27 21:04:00 -04:00
Peter Goodman fd20ddf914 Changes to make things compile again. 2017-04-22 16:56:58 -04:00
Peter Goodman df97ad72e9 Fixes for LLVM compatibility 2017-04-21 13:41:06 -04:00
Peter Goodman 6649904888 Changes 2017-04-20 12:48:07 -04:00
Peter Goodman c6595839b3 Minor fixes 2017-04-18 17:37:50 -04:00
Peter Goodman 93b4aa56b4 WIP. Modified license to apache 2.0 to follow remill, also most code is new anyway. 2017-04-15 16:27:35 -04:00
Peter Goodman ce5bce402d More big improvements to get_cfg.py, with corresponding improvements on the lifting side of things. 2017-04-10 14:49:10 -04:00
Peter Goodman 23ef38c7c0 Things are moving steadily forward. Redid all cfg proto stuff, and am bringing the get_cfg.py script back up to producing all relevant info. 2017-04-05 23:20:41 -04:00