52 Commits

Author SHA1 Message Date
Stephen-lei 82c310697f a simple scripts to show the details in a cfg file (#780)
to use this ,you have to put the CFG_pb2.py file,which from the  mcsema/tools/mcsema_disass/ida7/ ,into the same dir with this script
 type  python show_cfg.py "xxx.cfg" "xxx.txt" 
 the first arg is the location of the cfg file you want to see, like "./maze.cfg"
 the second arg is the location of the generated cfg details if the cfg file, like "./cfgdetails.txt"
2022-02-22 22:41:48 -05:00
Eric Kilmer 339800eda3 Rework Docker Image build (#741)
* Try reworking Docker image

* Fix some old behavior for building abi libraries

I don't think the result is actually usable...

* Remove most Python2 references

Still left some references to python2 for helper scripts since they
aren't tested (afaik) in CI

* Fix CI

* Remove old CMakeLists.txt and remove support for old TRAILOFBITS_LIBRARIES method of building

* Update README
2021-03-05 16:43:20 -05:00
Eric Kilmer e42f45ffff Remove most Python2 references (#742)
* Remove most Python2 references

Still left some references to python2 for helper scripts since they
aren't tested (afaik) in CI

* Fix CI
2021-03-05 16:11:23 -05:00
Eric Kilmer aa49fcc4fb Support LLVM 11 (#729)
* Add guards based on LLVM version for 11 compatibility

* Use remill LLVM 11 compat headers

* CI support for LLVM 11
2021-02-13 11:22:32 -05:00
Artem Dinaburg 8af327d12a Rolling Releases on Master CI Pass (#685)
Create a new release every time master branch passes CI.
2020-08-11 15:25:33 -04:00
Eric Kilmer c170c28de2 Remill binary docker (#657)
* Install remill from published Docker images instead of from source
* Use anvill docker image for anvill parts
2020-08-10 15:18:09 -04:00
Peter Goodman c002b8505c Formats the code with clang-format and my additional stylistic mojojojo (#682) 2020-08-05 16:14:01 -04:00
Peter Goodman 2d55d02cc3 API improvements. Must be used with the api_improvements branches of Remill and Anvill (#666)
* API improvements. Must be used with the api_improvements branch of both Remill and McSema

fixes for x86 and running the lifted code with klee

* Update dockerfile to clone anvill

* update remill commit id

* Add python3 to dockerfile

* update python3

* disable abi script

* Updated cmake to find anvill

* Update main.cpp

* update find_package for anvill

* WIP:updated prebuild cfg

* update prebuild cfg files

* enable abi build for testsuite

* Fix memory leak

* install missing package for testcases

* frontend: Reflect cfg file changes in dyninst frontend.

* frontend: Update local files copyrights to reflect overall change to agplv3.

* fix failing testcases

* update test cfgs

* Fix test failure with local state pointer

* set the flag to use local state_ptr in default mode

Co-authored-by: kumarak <iit.akshay@gmail.com>
Co-authored-by: Lukas Korencik <xkorenc1@fi.muni.cz>
2020-07-23 02:14:56 -04:00
Eric Kilmer effd27daa9 Small final docker image (#655)
* Small final Docker image
* Ubuntu 20.04 support
* LLVM10 Fixes
2020-05-21 02:04:38 -04:00
Artem Dinaburg 28dd4a717d Set up Github Actions and use a new Dockerfile based on the remill image (#650)
* Set up Github Actions and use a new Dockerfile based on the remill image
2020-05-07 02:04:07 -04:00
Alessandro Gario b133125634 CMake: Add ccache support (#604) 2019-11-18 11:38:52 -05:00
Alessandro Gario 1a96bd3712 Update links and remotes to the new organization name (#598)
* Docs: Update links with the new organization name

* CI/Travis: Update remotes with the new organization name

* mcsema-disass: Update links with the new organization name

* Dockerfile: Update remotes with the new organization name
2019-10-27 12:56:23 -04:00
Aiethel a7d7793c6a Extended binary_info recognition of pie binaries (#585) 2019-08-28 11:12:02 -04:00
Aiethel 62a1319e17 Add brackets around print, for python3 compatiblity (#577) 2019-06-26 15:37:18 -04:00
Peter Goodman 678016598b Update lift_program.py 2019-04-25 16:03:28 -04:00
Peter Goodman 96d334cdaa Update lift_program.py 2019-04-25 16:02:27 -04:00
Peter Goodman 376479256d Enable builds with GCC (#527)
* Enable builds with GCC

* Update travis.sh
2019-03-13 09:49:27 -04:00
kumarak 91f6f26399 fix parallel make (#525) 2019-03-09 12:56:38 -05:00
Peter Goodman 18058dff3d Update travis.sh (#526)
Remove old version numbers, add new ones.
2019-03-08 12:37:30 -05:00
artemdinaburg 1ef3ed2cd3 Initial attempt at running Binary Ninja in CI instances (#489)
* Initial attempt at running Binary Ninja in CI instances

* Automatically update Binary Ninja to Dev Channel Latest

* is_update_installation_pending is a function
2018-11-01 14:16:06 -04:00
Kyle Martin f8fe086568 Resolve 473 (#478) 2018-09-27 14:24:03 -07:00
Peter Goodman 16cdff4f3b Update travis.sh (#469) 2018-09-23 15:25:41 -07:00
Peter Goodman 69cb35cc16 Call early init lazy xrefs (#466)
* More broadly applies lazy xref initialization to all initializations where the xref size is less than teh value size of the xref itself (pointer size, usually). Moves initialization of lazy xrefs into a new function, __mcsema_early_init, which guards itself against multiple executions. Makes sure that __mcsema_constructor calls __mcsema_early_init first. Finally, makes sure that all native-to-lifted entrypoints call __mcsema_early_init as well, as we have observed cases where the lifted binary contains weak implementations of c++ standard library functions, and these functions are called by native libraries initialized before the call to __mcsema_constructor, thereby resulting in re-entrancy issues.

* Update Function.cpp

* Make sure the __mcsema_early_init guard is in the module.

* Make sure the __mcsema_early_init guard is in the module.

* Fixes for __gmon_start__ and stuff.

* Minor fix

* Adds some symbols

* Minor fix for ida7

* Try to resolve things like calls through PLT thunks via xref entry lookups when a flow cross-reference is missing.
2018-09-23 15:25:14 -07:00
Kyle Martin 2498e6464d Add Binja Disassmbler Support (#472) 2018-09-23 09:09:26 -07:00
Alessandro Gario d176bf8f2e Add a new CMake variable to enable/disable ABI libraries (#456)
* Add a new CMake variable to enable/disable ABI libraries

By default, all ABI libraries are disabled.

To skip the 'Linux' ABI library:
  cmake -DMCSEMA_DISABLED_ABI_LIBRARIES:STRING="Linux"

* Enable all ABI libraries in Travis
2018-09-06 18:39:35 -04:00
Alessandro Gario 4143a74905 WIP: Add Windows support (#385)
* CMake: Small refactor, style changes

* CMake: Fix the install target for Windows

* Windows: Add build instructions

* CMake: Add support for find_package(remill)

* CMake: Fail when the Python package can't be installed

* CMake: Fix the 'install' target on Linux

* CMake: Copy the Updated settings.cmake from Remill

* Travis: Update the build script

* Update the remill commit id

* CMake: Only use C++14 when compiling on Windows

* CMake: Fix the Python package installer

* Update the Windows documentation

* Update .remill_commit_id

Update to track the tip of master

* Change default install location to /usr/local

* Indentation fixes

* Fix CMake error introduced in d2582c7

This regression has been in the branch for a while, and it is caused
by the ABI library in mcsema/OS/Linux. The commit that introduced the
problem enabled an 'add_subdirectory' directive that was initially
commented out.

The problem with CMake was primarly due to the 'project(name BC)'
statement, trying to enable back the BC language (which was removed
due to broken Visual Studio support).

I've kept the add_subdirectory enabled, but only when building on
linux. CMake works fine now, but compilation fails due to missing
include headers referenced by ABI_libc.h (like ultrasound.h, which
appears to come from the kernel headers).

=======

alessandro@tob-ubuntu1804-remill:~/Projects/remill/tools/mcsema$ git bisect run /tmp/bisect.sh
running /tmp/bisect.sh
CMake has failed to configure the project!
Bisecting: 5 revisions left to test after this (roughly 3 steps)
[67164c725e] Update README.md
running /tmp/bisect.sh
CMake has successfully configured the project!
Bisecting: 2 revisions left to test after this (roughly 2 steps)
[d2582c7abd] Build Tests in Travis (#406)
running /tmp/bisect.sh
CMake has failed to configure the project!
Bisecting: 0 revisions left to test after this (roughly 1 step)
[a708bafc42] InlineAsm function type mismatch (#409)
running /tmp/bisect.sh
CMake has successfully configured the project!
d2582c7abd is the first bad commit
commit d2582c7abd
Author: artemdinaburg <artem@dinaburg.org>
Date:   Tue May 22 21:13:43 2018 -0400
    Build Tests in Travis (#406)

    Always generate integration tests and then run the generated integration tests travis. We get around a lack of IDA by providing pre-built CFGs for examples.

    Closes #407
:100644 100644 4154fe1e11 8fbfcee113 M  CMakeLists.txt
:040000 040000 4bb5e0a484 491dba1a1d M  mcsema
:040000 040000 28398aa746 2461062d36 M  scripts
:040000 040000 97f8342b53 c1487df961 M  tests
bisect run success

* Fix for building 32-bit abi libraries

* Minor cmake tweaks

* Install 32-bit libraries on Travis-CI for Linux

* Fix installation of 32-bit libs to work around travis bug

* Handle C++ bitcode files

* Bump remill commit id

* Support bc file generation on ubuntu 14.04

* Bump the Remill commit ID.
2018-08-29 14:42:54 -04:00
artemdinaburg d2582c7abd Build Tests in Travis (#406)
Always generate integration tests and then run the generated integration tests travis. We get around a lack of IDA by providing pre-built CFGs for examples.

Closes #407
2018-05-22 21:13:43 -04:00
Peter Goodman 47751efc8c Add shell escaping to mcsema-disass-related stuff. Needed this because my IDA install dir on macOS has a space in it. 2018-05-09 13:43:37 -04:00
kumarak 1cc571b11f Support for --abi_library flag to call the external functions with explicit arguments. (#387)
* Add initial support for --abi_library flags

* support for abi_library

* Remove the weak linkage for __mcsema_debug_get_reg_state.

* Fix num of agruments for __cxa_allocate_exception

* Remove --library flag and cleanup

* Don't reload/overwrite mcsema::gModule after loading in the protobuf.

* Use PrepareModuleDataLayout to avoid checks for __remill_basic_block function.

* Bump the remill commit id to support --abi_library flag

* Fix travis build failure

* Initialize the test log filename;

* trim newline char from the eof
2018-03-22 11:09:13 -04:00
artemdinaburg 5dd79c35df Update README, Make --prefix work, check test results (#353)
* The test runner now knows about mcsema installations that aren't global

* We now check the output of the integration test suite in travis. This will cause travis to fail since we're currently failing tests

* Lots of README updates

* Use like 99% less sudo
* Better script to fix local IDA Python installations
* Support for building local installs in a virtualenv
* Remove the protobuf install that led people astray
2018-01-18 19:50:54 -05:00
Kareem El-Faramawi 329b6e2d8b [WIP] Binja cfg updates (#329)
* Fix incorrect stack var sizes

* Recover references to stack vars

* Fix requested changes

* Skip jump table entries in segment xrefs

* Fix sections being incorrectly considered code

* Ignore some symbols binja inserts

Binja inserts a few symbols it identifies that shouldn't be picked up as
globals, so skip these.

TODO: Look into a better way to identify globals than looking through
and filtering variable symbols

* Fix the links to our repos (#364)

* Bring back the mcsema-lift option for --list-supported (#365)

* Bring back the mcsema-lift option for --list-supported

* Discard changes to whitespace from last commit.

* Fix several xref warnings and issues

* Tail call targets now picked up as control flow xrefs
* Tail call targets added as successors
* Fix duplicate blocks being lifted as a result of tail calls being inlined
* Ignore duplicate xrefs as a result of how binja shows the instruction in IL
* Pick up missing xrefs when an instruction is expanded to multiple IL instructions
* Only classify the memory operand of a LOAD/STORE as a memory/displacement xref

* Fixes an issue where we assume that every symbol in the module passed to --library is external, whereas that's false. (#368)

* Bump up commit id to include support of atomic intrinsic (#367)

* Bump up commit id to include support of atomic intrinsic

* update remill commit id

* change cs_action to catch & cleanup type, not looking into catch types; (#371)

* Klee maze example (#369)

* In progress. Working on an example of using KLEE on a Maze, but with the maze program being compiled to x86, amd64, and aarch64.

* Making lots of progress on getting lifting and runnning an aarch64 maze program on amd64, but using --explicit_args. The key thing I'm working through right now is a jump offset table, but where the offset is a block pc, rather than a table base. Also adding various bits of code here and there to making runnning with klee more directly doable, and working on a debugging facility to track down when the emulated program counter gets out of sync with the original program.

* Fixed a subtle @PAGE and @PAGEOFF-related reference bug on AArch64. Partially disabled the special jump offset table handling I had in table.py, as it doesn't (yet) handle the shifted table values. However, I still have the code there, so that it can recognize that a basic block address is used as a possible offset, so that I can remove the block address as a reference, which permits a new heuristic on the C++ side to work. On the C++ side, when there's a jump instruction that isn't associated with a cross-reference flow, I try to auto-augment it with addition switch cases, targeting blocks with no predecessors (as present in the CFG). This seems to work reasonably well.

* Improved the scripts and updated the READMEs.

* Minor rephrase

* Minor rephrase

* Making the stack start a bit further back reduces things like KLEE messing up (#373)

* Changing indentation level, adding more logging statements to track what's going on.

* Manually merged in Kareem's changes before doing an auto merge.

* Got the Maze example working with binary ninja.
2018-01-17 23:41:33 -05:00
Peter Goodman aeda244f5f Adds support for the --legacy_mode flag to scripts/lift_directory.py, and ./scripts/lift_program.py. (#361) 2017-12-26 15:12:19 -05:00
Peter Goodman ad0be58242 Adds back two scripts that were lost in some commit (#357) 2017-12-19 10:29:05 -08:00
Alessandro Gario 98575e728f This PR adds support to LLVM 3.5/3.6/3.7/3.8/3.9/4/5 to Travis (#326)
* Travis: Enable LLVM 3.6.2, 3.7.1, 3.8.1, 3.9.1, 4.0.1, 5.0.0
* Travis: Cache the cxx-common packages in a dedicated folder
2017-11-11 23:45:13 +01:00
Alessandro Gario dc71c7b87d Travis - Automatically test both LLVM35 and LLVM40 (#325)
* Travis: Add LLVM35, do not use the Clang static analyzer when using LLVM < 4.0.0

* Travis: Use sudo when cleaning up between tests (see details).

Installing with 'sudo make install' leaves some files that are
owned by root in the build folders.
2017-11-10 17:45:32 -05:00
Alessandro Gario 58a09742ce Travis: Make sure the zlib1g-dev package is installed (#324) 2017-11-10 14:43:09 +01:00
Alessandro Gario 6015890d14 This PR enables the XZ test (#312)
* Travis: Enable the XZ test.

* Travis: Show the clang command line when recompilation fails.
2017-11-06 13:22:50 -05:00
Alessandro Gario 406e674636 Add Travis support to the use_remill_semantics branch (#310)
* Imported the new test framework

* Added the new Travis script

* Change Ubuntu version to Xenial (16.04)

* Travis: Various fixes; also enabled CMAKE_VERBOSE_MAKEFILE

* Travis: The test binaries were being excluded by gitignore. Add clang's static analysis

* Travis: the CC and CXX variables were not being set

* Travis: Explicitly set the compiler (CMAKE_CXX_COMPILER) along with the CXX env var

* Travis: Disable the XZ test (currently broken), show the clang static analyzer summary

* Travis: Move the CI shellscript to the script folder. Add src headers

* Travis: Test a known Remill version to avoid random breakages
2017-11-05 16:18:21 -05:00
Peter Goodman bcda6a120a Llvm38 (#112)
* - Upgrade to official llvm 3.8
- remove boost
- unify all cmake files into a single cmake file
- use official protobuf
- start factoring out x86-specific stuff to eventually make an arm port easier
- simplify the CLI; now use mcsema-lift, with -arch, -os, -cfg, -entrypoint, and -o. No more having to specify the target triple.
- moves source code slightly closer to our style guide

Note:
- lifted bitcode is not quite right in some cases, so this isn't a stable branch!
- TODO: re-add test cases to discover source of stability problems.

* Some minor fixes, one to make sure xmm regs in the state struct are properly aligned

* Added missing std defs for option parsing. This makes /bin/ls work properly :-)

* Remove old cmake files

* Minor changes to get_cfg.py and raiseX86.cpp in relation to those changes. Those changes don't fix anything, the purpose was to make symbol names for things match between python and cpp. E.g. get_cfg would name things like dta_0xf00, sub_0xf00, ext_... And it seems that it was dta_ instead of data_ for a reallly flaky and dumb reason but oh well. I also fixed a subtle bug related to saving and restoring of callee saved registers on elf 64. I have not made related changes to elf 32 or pe 32/64, though those may be necessary.

* Minor fix

* Adding mcsema-disass, which is a nice wrapper around get_cfg.py.

* Working on readme and cleaning out (currently) unused stuff from the repo

* Renaming mc-sema dir to mcsema

* new travis file

* Updates to bootstrap and build process

* Minor bootstrap fixes

* Well, don't have windows working yet but this is kind of progress I think

* Travis should work now

* Updating protobuf-cmake files so we can generate a VS2015 solution

* Removing and adding some choco packages from README

* Bootstrap now builds protobuf and generates protobuf files
LLVM should now be built on Windows

* Adding Win32 specific compiler options

* Renamed ConstantInt to CreateConstantInt to satisfy MSVC

* Build Release LLVM to not have linking conflicts of MD vs MDd

* Added some missing instructions

* Adding changes to generate runtimes

* Windows bootstrap works.
2017-02-23 13:23:31 -05:00
Peter Goodman d347c1232b Added defs for ssl, pcre, and asn1. Sorted and uniq'd the linux.txt defs, just to organize it. Updated the ida_get_cfg.sh script to use the linux.txt defs. Updated get_cfg.py to be able to take a file name for specifying where to store the debug log. The way to use it is --debug --debug_output /tmp/log. The default when only --debug is used is to log to stderr. 2016-10-04 12:08:49 -04:00
Artem Dinaburg ec8f9e083d Updating scripts 2016-06-08 06:21:43 -04:00
Artem Dinaburg 1109f050aa Properly display which compiler could not be found 2016-06-07 18:41:40 -04:00
Artem Dinaburg 0107822257 some sanity checks don't execute if we have -e for sanity_check 2016-06-07 18:36:46 -04:00
Artem Dinaburg 65cdbd75e5 Add option to output bitcode without a full link 2016-06-06 19:35:34 -04:00
Artem Dinaburg 496000936e Directory fix 2016-05-19 16:58:44 -04:00
Artem Dinaburg e11a9a73c2 More sanity checking, undo some 'get my dir' changes 2016-05-19 15:58:15 -04:00
Artem Dinaburg 308a53026a Fix script base dir retreival 2016-05-19 15:37:15 -04:00
Artem Dinaburg d446fe4a0f packaging updates 2016-05-19 00:59:45 -04:00
Artem Dinaburg 5565baa70d Update packaging and invocation scripts 2016-05-19 00:31:34 -04:00
Peter Goodman edac42eafb Fixed an issue with using the stack incorrectly. Still another blocking issue. Run script works nicely. 2015-11-06 19:21:17 -05:00