54 Commits

Author SHA1 Message Date
Peter Goodman b0f7790f43 Sparc support (#709)
* Adds sparc support; formats files

* Update the readme and walkthrough

* Minor tweaks to readme/walkthrough

* Update ci.yml
2020-11-05 12:59:20 -05:00
Peter Goodman 31f267bd5f Cmake refactor (#693)
* Update to latest remill APIs and way of using CMake.

* Minor fixes

* Bug fixes for AArch64

* Tweaks

* Adds some nifty functionality useful for debugging some aarch64 bugs

* Fixes a subtle bug

* Bug fixes

* Make the test suite use explicit args

* Bug fix in xrefs

* Disable tests for now
2020-09-22 23:22:56 -04:00
Peter Goodman c002b8505c Formats the code with clang-format and my additional stylistic mojojojo (#682) 2020-08-05 16:14:01 -04:00
Peter Goodman 2d55d02cc3 API improvements. Must be used with the api_improvements branches of Remill and Anvill (#666)
* API improvements. Must be used with the api_improvements branch of both Remill and McSema

fixes for x86 and running the lifted code with klee

* Update dockerfile to clone anvill

* update remill commit id

* Add python3 to dockerfile

* update python3

* disable abi script

* Updated cmake to find anvill

* Update main.cpp

* update find_package for anvill

* WIP:updated prebuild cfg

* update prebuild cfg files

* enable abi build for testsuite

* Fix memory leak

* install missing package for testcases

* frontend: Reflect cfg file changes in dyninst frontend.

* frontend: Update local files copyrights to reflect overall change to agplv3.

* fix failing testcases

* update test cfgs

* Fix test failure with local state pointer

* set the flag to use local state_ptr in default mode

Co-authored-by: kumarak <iit.akshay@gmail.com>
Co-authored-by: Lukas Korencik <xkorenc1@fi.muni.cz>
2020-07-23 02:14:56 -04:00
Peter Goodman 782bd1dcfa Update for new remill APIs (#595)
* Update for new remill APIs

* Update remill commit id

* Cleanups

* resolved one issue
2019-10-27 17:48:43 -04:00
Aiethel dfaeedcadc Remove required code section (#586)
* Build special code section from instructions

* Add some clarity comments around BuildCode

* Add code_segment into the module for the time of resolving instruction xrefs

* Add only instruction that do have some references to the list that is resolved later
2019-09-04 15:39:26 -04:00
kumarak 472675ba9c Check for the xref variable before calling lazy init (#560)
* Add check for the xref variable before lazy initialization

* Fix space
2019-05-06 14:10:22 -04:00
Peter Goodman a8176f141f Fix gcc build (#528)
* Should fix some errors when compiling with gcc

* Minor touch-up

* Whoops those were signed for a reason

* Whoops those were signed for a reason
2019-03-13 14:33:25 -04:00
Peter Goodman 7d2dfe6e94 Attempt to resolve Issue #521 (#522)
* Attempt to resolve Issue #521

* minor fix

* minor fix

* minor fix

* Simplifications and comments

* If it's a copy of shared data, then don't treat it as an xref

* Error log if there is segment variable with the same name as external var

* Added flag and checks if the variable needs initialization if already defined;

* fix noreturn for the functions throwing exceptions;

* review changes

* Update get_cfg.py

* Update get_cfg.py

* Fix the recovery of exception action table chain

* Update the prebuild cfg for testsuite

* Fix entrypoint function not getting recovered in ida7

* Change the order of idc.GetFunctionName and idc.GetTrueName while looking for function name
2019-03-02 20:59:08 -05:00
Aiethel 5984250bc6 Native function improvements (#518)
* NativeFunction function is properly set

* Remove public keyword from structs
2019-02-26 11:07:54 -05:00
MCMrARM 7a30febc0d Set the config file limit to 512M (#465)
* Set the cfg file limit to 512M

* Code style: Remove spaces before constructor arguments
2019-01-24 16:49:12 -05:00
kumarak a3483e4bf1 C++ exception handling support (#391)
* Initial changes to parse the eh_frame and exception table

* linearly scan the function to find missing blocks

* Update protobuf format for lp; need to tag instructions or basic blocks to associate it with the lp

* tag instructions with lp; removed debug logs

* updated comments and protobuf; use utility functions for pointer size;

* travis build failure; make lp_ea optional field

* Collect the exception handling bb chunks; Create a passthrough for the lifted landingpad;

* Generate the passthrough IR and attach the landing pad with function invoke

* Generate the eh_frame/exception table and add personality function with one raising the exception.

* catch/cleanup wrapper for the exception handling; Disable CFG simplification pass temporarily for invoke inst;

* invoke wrapper to fix stack and base pointers

* recover type indices of the exception type

* fix exception table offset

* Add the function definition for setting the rsp/rpb correctly after eh return;

* Changes for abi_library support in exception handling

* Remove unwind attribute from ext_* functions to generate eh table

* Support the lifting of multiple exception types thrown;

* Fix the multiple declaration of personality function in lifted bitcode;

* Add --recover_exception flag to lift the exception table

* Update the list of external functions with c++ ABI's; Fix basic block split issue for exception chunks;

* Weak linkage for external function causing the garbage value for out_of_range destructor. Temporarily disabled;

* cleanup redundant changes and resolve conflicts

* Update --std-defs list with the c++ exception ABIs

* Fix merge fail

* exception test binaries for integration test; Move to clang++ for recompiling the bitcode.

* Fix the exception handling support for llvm 3.5

* Disable the optimization passes for the function having landingpad;

* Temporary changes to add ABI libraries for exception tests.

* Remove the noreturn attribute to fix the problem with -03 optimization

* review comment changes

* Change wrapper function name across invoke;

* review changes

* Updated comments text

* Remove duplicate clauses from the list; Add arguments for personalityfn and fix review comments.

* Update the flag name to --exception_personalityfn;

* Update the flag name to --exception_personality_func;

* Follow the variable naming convention;

* Update comments

* travis build fix for unused variable

* Create the map of type indices from original binary. Runtime routine to fix the RDX with correct type index;

* Add _Unwind_Resume to --abi_libraries functions list;

* Update comments & code restructure

* Exception handling changes and utility functions for 32 bit;

* DWORD for 32-bit system
2018-04-12 14:05:45 -04:00
kumarak 1cc571b11f Support for --abi_library flag to call the external functions with explicit arguments. (#387)
* Add initial support for --abi_library flags

* support for abi_library

* Remove the weak linkage for __mcsema_debug_get_reg_state.

* Fix num of agruments for __cxa_allocate_exception

* Remove --library flag and cleanup

* Don't reload/overwrite mcsema::gModule after loading in the protobuf.

* Use PrepareModuleDataLayout to avoid checks for __remill_basic_block function.

* Bump the remill commit id to support --abi_library flag

* Fix travis build failure

* Initialize the test log filename;

* trim newline char from the eof
2018-03-22 11:09:13 -04:00
kumarak 737ffd38c1 Parsing eh_frame and language specific data area. (#358)
* Initial changes to parse the eh_frame and exception table

* linearly scan the function to find missing blocks

* Update protobuf format for lp; need to tag instructions or basic blocks to associate it with the lp

* tag instructions with lp; removed debug logs

* updated comments and protobuf; use utility functions for pointer size;

* travis build failure; make lp_ea optional field
2017-12-20 10:26:48 -05:00
Peter Goodman e621164a7c Fixues issue #321. (#331)
* Fixues issue #321.

* This is an attempt to fix the issue Akshay observed, though not by restoring the original code. In a windows binary I am looking at, I have the jump table entries as 4-byte displacements, added to an 8-byte offset.

* This includes some pretty substantial changes. Some are related to jump tables, then things diverged into better handling of .idata sections in PE files, and specifically external calls and devirtualizing them in the lifted bitcode.
2017-11-19 23:51:57 -05:00
kumarak 11149d25e1 [WIP] Lift the stack variables using Mcsema2 (#303)
* collect the references of stack variables

* Initial changes to lift stack variables

* Set the stack variable name

* remove the entryBlock for allocating the stack variables

* addressed review comments

* Handling index and scale associated with stack variable; Build will fail because of the required changes in remill;

* Utility to load the register addresses; remove the dependency on remill changes

* Disable function parameters lifting;

* Fixed review comments

* IDA changes to lift the stack variables for apache ATD's. Issue with lifting some variables from status_handler;

* Change CFG proto names to match with other classes

* Remove debug logs

* Change stack allocation variables and general cleanup; Update the blacklist functions for which the stack shuffling does not work;

* Fixed code alignment

* fix tab indentation with spaces
2017-11-14 15:20:31 -05:00
Peter Goodman 28265c4a8f Fixes to report not having a name for a variable segment, but not crashing on the condition. 2017-10-10 12:01:46 -04:00
Peter Goodman d84721d5e9 Require that the segment containing a cross-reference is part of the CFG 2017-09-26 13:05:16 -04:00
Peter Goodman 71c7e939df LLVM 5.0 compatibility. 2017-09-19 15:13:24 -04:00
Peter Goodman 7498b2d1ae Initial support for TLS. Some stuff is still not working, also it
is not thoroughly tested.
2017-08-24 22:52:36 -04:00
Peter Goodman 497d4dbe04 Pretty substantial changes. I removed --partition_segments. Now all idautils.Entries that are in the data sections are treated as segment split points. This is all to handle Apache, which exports its symbols to its shared libs (apr and apr-util). 2017-08-23 00:10:49 -04:00
Peter Goodman 89c42a3e42 Support for jump offset tables in the bitcode. Different handling of ELF GOT pointers. It might still be a bit wonky for stdin/stdout.. not sure. 2017-08-22 13:05:20 -04:00
Peter Goodman 04e675ac48 Remove tail calls, fast calling convention, and fixup get_cfg.py to better support segmenting existing segments using global variables 2017-08-18 22:41:49 -04:00
Peter Goodman df1678412b Adds support for --legacy_mode, where drivers are added for the entrypoint and the callbacks 2017-08-18 15:09:25 -04:00
Peter Goodman a3f5f65aed Minor improvements to handling of code with c++ exceptions. The exceptions should now be throwable, but not catchable/handle-able. Also fixed a related issue that crept up when there is an lazy xref in the gcc except table section to rtti stuff. 2017-08-08 15:07:24 -04:00
Peter Goodman 6ced3bd807 Transitioned to using a table-based approach for explicit arguments and return values. 2017-07-14 15:30:18 -04:00
Peter Goodman 1707c0c3cf Fixes some segfaults related to trying to print out the names of segments, but if the segment isn't recovered, e.g. it's an external segment, then we won't have the name for it. Also, produced an error message when it tries to fill in an external address itno a data section. 2017-07-10 16:31:22 -04:00
Peter Goodman 834adb7991 More progress on calling convention stuff. Moved some existing code around. Added implementations for the runtime argument getting stuff. Added functions for setting the return value. Not well tested. (#267) 2017-07-10 11:57:10 -04:00
car bauer 82d70afe5c Explicit args [WIP] (#265)
* explicit args to external APIs
* Make AArch64 disassembly detect refs to external functions
* Fix awful bug where code would become undefined since our callinst was not same convention as the called function
* Emit enough code so that AArch64 bitcode gets emitted
2017-06-29 13:34:51 -04:00
Peter Goodman 933abea11d Added support for @page and @pageoff memory relocations. 2017-05-27 23:20:31 -04:00
Peter Goodman 1acbb170fb Fixed up get_cfg.py to better understand data-to-code refs within the .text segment. This seems to be a common pattern for AArch64. Improved get_cfg to better understand AArch64 conditional branches. For example, IDA recognizes b.le as an idc.ARM_b and so to treat it as a conditional branch, we need to inspect IDA's insn_t.segpref field, and look for specific values in there. Finally, moved one of Alessandro's features for dumping the cfg out into its own tool, which is a more logical spot. This will simplify future additions, e.g. producing DOT graphs from the CFG file. 2017-05-27 18:56:56 -04:00
Peter Goodman c1bd1668f4 Minor fixes here and there. May not yet build! 2017-05-25 11:00:02 -04:00
Peter Goodman 8452475933 Big improvements to get_cfg.py, especially for SQLite. Still, SQLite doesn't lift correctly. Also, ls using -la segfaults, although -lax does work. 2017-05-06 18:04:17 -04:00
Peter Goodman 5e508962f8 Fixes for lifting. Simple lifted binaries can now be recompiled when using the alessandro/refactor/llvmmultiversion-cmake branch of remill as a base. 2017-04-29 22:06:24 -04:00
Peter Goodman 886ee0873f Use GlobalAlias instead of GlobalVariable for segment and variables. Seems to produce the right type of xref info in resulting object files. There's still a bunch of masking of ptrtoint'd vals that I need to find the source of and remove. 2017-04-28 20:11:21 -04:00
Peter Goodman 3ca5995e83 Working toward bringing up the mcsema runtime. Need to add --entrypoint support to get_cfg.py, so that something like main can be specified as the key entrypoint. Right now it's pulling in things like init, start, etc. 2017-04-25 11:45:32 -04:00
Peter Goodman 6d6d02ba7e Made it so that contiguous segments are placed within the same struct on the llvm side, just in case there's a reference to the beginning of one segment that is logically a reference just beyond the end of another one. Added some weak externs related to the end of some segments (used by brk I think) to the linux defs file. Improved some error reporting. 2017-04-23 12:52:48 -04:00
Peter Goodman df97ad72e9 Fixes for LLVM compatibility 2017-04-21 13:41:06 -04:00
Peter Goodman 6649904888 Changes 2017-04-20 12:48:07 -04:00
Peter Goodman 93b4aa56b4 WIP. Modified license to apache 2.0 to follow remill, also most code is new anyway. 2017-04-15 16:27:35 -04:00
Peter Goodman ce5bce402d More big improvements to get_cfg.py, with corresponding improvements on the lifting side of things. 2017-04-10 14:49:10 -04:00
Peter Goodman dfdc6b9648 More progress, especially on the get_cfg.py front. 2017-04-07 21:15:46 -04:00
Peter Goodman 23ef38c7c0 Things are moving steadily forward. Redid all cfg proto stuff, and am bringing the get_cfg.py script back up to producing all relevant info. 2017-04-05 23:20:41 -04:00
Peter Goodman 30e3c86e48 Improved jump table handling and started implementing jump table (via switch inst) support on the C++ side of things. Removed some optimization passes; it seems as though the instruction combining pass of llvm injected lots of undefs into the code, which resulted in all the things optimizing themselves away. 2017-03-31 23:37:16 -04:00
Peter Goodman dcaaa8683b Makign great progress on various forms of reference resolution. 2017-03-29 00:10:22 -04:00
Peter Goodman 0441f95426 Updated copyright messages. Added namespace mcsema 2017-03-24 19:33:54 -04:00
Peter Goodman a139d435ce Another good logical spot. Have basic lifting working (no mem refs, no jump tables, etc.), and also have optimization of the lifted bitcode working nicely, with removal of ISELs and SEMs. 2017-03-24 18:02:21 -04:00
Peter Goodman f17266f985 Making progress towards initial lifting 2017-03-23 23:23:49 -04:00
car 4445cc093d fixing typo in status print 2017-03-15 12:21:19 -04:00
artemdinaburg ed1d537db3 Initialize to nullptr (#165)
Closes #146
2017-03-08 17:45:31 -05:00