* This includes most of Alessandro's wonderful cmake changes, and starts into AArch64 support, though not using Capstone anymore.
* Trying to make travis work
* Minor additions, still not completely working on my ubuntu 14.04 vm or on travis
* Making progress on getting travis to build
* Some documentation improvements
* I push the travis button
* Change copyright notice in all the places. Trying to get cmake to download all the things
* CMake refactor
* Added the lib repository installer
* Fixed a couple of paths in library repository scripts
* Spelling and a missing .gitignore file.
* The google test library was not correctly linked
* Removed the pre-compiled XED library
* Removed unused files; fixed the INSTALL directives. Added automatic protobuf generation.
* Added a package generator script for ArchLinux
* Merged in Alessandro's cmake magic
* Cleanups, compatibility changes
* CMake cleanup.
o Removed the cmake folder as it is no longer used.
o Removed the library_repository_installer directory and
replaced it with a submodule to the newly introduced
cxx_common repository.
o Added the 'use_remill_semantics' branch of mcsema as a
submodule under tools/cmake.
o Updated the README instructions (added --recursive to the
git clone command).
Warning: the mcsema submodule is pointing to my fork of the
repository!
* CMake: avoid re-defining the C/CXX/ASM compiler.
This will prevent CMake from looping forever when using submodules.
* The tools/mcsema submodule now points to the official repository.
* mcsema submodule update
* Updated the mcsema git submodule to track the newest changes.
* Various CMake fixes (see details).
o External include headers were not correctly marked
as SYSTEM. This caused them to output warnings and
break the build.
o The PROJECT_SOURCE CMake variable has been replaced
with CMAKE_SOURCE_DIR.
o Updated the mcsema submodule to point to the latest
CMake fixes.
* Getting closer to having the test case runner work again without using the CFG protobufs.
* Trying to make things use ubuntu clang/llvm packages
* Update to the mcsema submodule.
* Minor changes for llvm version compatibility
* Test cases should run now. Had to compile the lifted testcases to a .S file, as with the .bc file, they were being optimized away.. I think. The compilation to assembly seems unusually slow, though.
* Minor change
* Some stuff for llvm 4.0 support
* Remove mcsema sub-module
* Remove cxx-common submodule
* First steps toward getting travis working again
* Attempt at getting travis working again
* Using https for cloning instead of ssh
* Minor build script update
* Added ISEL_ prefix to isels to make it easier for ForEachISel to find them. Commented out the defer_inlining intrinsics.
* Changes related to mcsema2
* Simplify runtime targets generation. (#110)
* CMake refactor: Added a new language 'BC' for the bitcode (see details).
The language is used to generate the runtimes used by the architecture
modules. The required executables (clang++ and llvm-link) are
automatically detected in the same way as other compilers are.
A new CMake function has been added and it can be used to easily
generate runtime targets in a way similar to add_executable:
add_runtime(<name> SOURCE <source list> ADDRESS_SIZE <n>
DEFINITIONS <definition list>)
Additionally, all files ending with the *.bcpp extensions will
automatically invoke the bitcode compiler when listed in an
active target.
This should open support for parallel compilation! You just have to
list all your .cpp files when calling add_runtime.
* CMake: Fixes to the BC language handler.
* CMake/BC: Use '.bo' for object files. X86 runtime: Add -g/-O flags.
* Initial commit of x86 program snapshotter. Haven't yet figured out how to save a core dump to a specific file.
Now finds and saves core dumps (in a sketchy way).
Kind of but didn't really fix remill-disass with core dumps. It almost seems like a bunch of stuff is missing from within a core dump. It may be the case that it will be simpler to use binary ninja directly on snapshot files.
Working toward the executor.
Have snapshot creation and loading working. Next up: decoding the first few instructions!
Starting to serverize remill and setting up lines of communicating between vmill and remill. Going to move on to finally implementing remill-opt.
Changed a bazillon things.
Fixed the dead register backward data-flow analysis.
Fixed soem bugs, added GEP re-association, working toward inserting stores that will kill values.
Got interprocedual dead store elimination working! :-D
Finally...dead store elimination
remill-opt is done.
Added Lifter interface
Spec'ing out the translation engine.
Got the dynamic decoder working!
Got bitcode caching working
Added caching layer for bytecode compilation.
Decided on how to access reg state and allocas in functions. The bytecode will treat the state struct and the alloca'd data as a contiguous, opaque, byte-addressible area.
Compiler seems to work
Refactor
Great progress...goodnight
Made it up to the first syscall
Fixed call to a hypercall intrinsics
Bug fixes, minor change to the CFG proto.
Refacotorings and changes
* Switching to trying nativeexec, and made memory32 map snapshot into low 32-bits of address space, preserving original addresses of program.
* About to make some interesting changes, so save save save
* Refactor done, now time to produce shared libraries
* Got initial execution of some stuff in PHP working.. it either seems like the code is in an infinite loop, or just horribly slow, not sure. Otherwise, amazing progress.
* Got initial compiling to a runtime dynamic library working
* Separated most DEF_ISEL_SEMs and tests still pass. Goodnight!
* Got incremental optimization and compilation working.
* Got caching of the bitcode file to disk working and periodically collapsing the shared libraries into a single library. OUT.
* Fixed some bugs
* Commit before the storm
* Made the JIT work again, still not that fast though.
* Minor logging fix
* Log an error that we're executing a missing instruction.
* JIT compile the whole module first, then incrementally JIT compile function partitions. Also, link in libm.
* Added new syscall
* Added breakpoint sync hypercall, useful for testing.
* Got a php5.4 unserialize bug to reproduce. Added remill-pinshot, which will use PIN to take a snapshot and print out a register trace. This is useful for debugging divergences. Fixed up the semantics of some instructions, and added semantics and tests for PSRLDQ.
* Simplifications to remill that removes all the various basic block arrays, and uses meta-data instead.
* Made cmake take over the test system
* Working on mega refactor to eventually permit klee support
* Made all semantics code return memory pointer; I think this makes LLVM's optimizer a bit happier. Made the __remill_sub_N things into global constants, referencing the actual functions, that have private linkage. Slowly getting back in the direction of execution. I've got some bitcode translation working (with a hard-coded address for my local php version). The address space stuff seems to work so far.
* Missing files.
* Other minor fixes
* rtti-related fixes for gtest
* minor travis config change
* minor travis config change
* sendfile no longer needed
* Disable building vmill on non-linux platforms
* Disable 32-bit test builds on macOS.
* Trying to get symbol names on mac right.
* Minor fix to save state code for tests, disable testing on macOS builds because of symbol mismatches in gtest.
* Fixup remill-disass to use the simplified proto structure. Something may be wrong with remill-opt, or installing remill opt. Remove caching from travis.
* Backporting changes from vmill to remill. One set of changes is to remove DEF_ISEL_SEM, requiring that one use DEF_ISEL and DEF_SEM separately. This makes some simple instruction semantics more tedious to implement. The main goal is for ALL semantics themselves to be treated as private, non-exported symbols. In the case of those semantics defined via DEF_ISEL_SEM, those weren't private, and this inhibited inlining optimizations. Another big change is that function cloning has been rewritten and no longer depends on LLVM's function cloning implementation. Another big change is that the meat of the optimizer has been put into the remill/BC subdirectory and wrapped in a class, enabling it to be used independently of remill-opt. There are some new instruction semantics, and fixes to those. Some new utility functions for iterating over lifted blocks in the module. Attach/detach intrinsics have been removed, as they don't (yet) explicitly fit within the current model. The AssemblyWriter, which would produce a .S file and debug info linking lifted bitcode to the .S file via source line debug info has been removed. Despite it being really cool, the actual benefit is not obvious, given that optimization of the bitcode doesn't actually propagate enough of that information after inlining.
* Eliminate sendfile dependency.
* Implements remill-opt
* Change back to google namespace
* ANother namespace change
* Should fix test runner.
* Try to fix deprecated use of fstat64 on macOS
* Start adding OS X compatibility
* wrong unzip command
* update README
* first shot at build.sh
* clean up CMakeLists
* unnecessary CMake flag
* build.sh finally works (on my machine)
* start work on travis integration with new build.sh
* fix directory issues
* fix script name typo
* fix unnecessary brew stuff for os x
* build.sh should work on travis now
* make tests not fail on apple
* forgot to add file to fix mac/macos confusion
* Starting work on Issue #50, implementing an IDA Python script for producing a CFG file. The approach taken is to fill in a generic data structure, and some other code will deal with the production of the CFG structure itself. This will enable moving beyond the protobuf format. The script can be used via remill--disass. Eventually Binary Ninja will support a batch mode, and I will work it into remill-disass when that happens.
* remill-disass can now be installed and executed directly. There were some issues with the PYTHONPATH but I think those are resolved.
* Added in command-line flag to change the log level.
* Added conditions to prevent non-termination, especially in the presence of noreturn calls.
* Try to fix travis
* remill-disass seems to work reasonably well now.
Issue 44 mmx support (#58)
* Should fix Issue #56. Used -isystem instead of -I, as it's an include of <cstdint> instead of "stdint.h". @erhlee-bird does this work on your system?
* Minor changes
* Remove old 'test' function.
* Start to MMX PUNPCK instructions.
* Initial example for a PUNPCK instruction.
* Cleanups to Operators.h. Specifically, to the signed versions of vector operations. They now more strictly enforce type safety. This caught a few bugs. Operators.h is arguable simpler as a result.
* MMX Instruction support(PADD/PSUB); Fix for handling signed operations;
* Should fix Issue #56. Used -isystem instead of -I, as it's an include of <cstdint> instead of "stdint.h". @erhlee-bird does this work on your system?
* Minor changes
* Cleanups to Operators.h. Specifically, to the signed versions of vector operations. They now more strictly enforce type safety. This caught a few bugs. Operators.h is arguable simpler as a result.
* merged conflict;
* MMX Instructions (PAVG/PHSUB/PMULHRS)
* Testcases for PADD/PSUB; Fix for signed/unsigned saturation;
* Test cases for PAVG/PABS/PACKSS; Fixes for MMX instructions handling;
* Fixed PACKSSWB instruction handling;
* PHADD/PHADDSW Instruction;
* PMAXSW/PMAXUB/PMINSW/PMINUB Instructions
* PCMP/PUNPCK/PMUL Instructions;
* Corrected space indentation;
* Fixed build issue;
* reimplemntation of instructions with saturation arithmetic
Minor syntax fixes. Added beginning of an implementation for EMMS.
Issue 44 mmx support (#65)
* Remove old 'test' function.
* Start to MMX PUNPCK instructions.
* Initial example for a PUNPCK instruction.
* MMX Instruction support(PADD/PSUB); Fix for handling signed operations;
* Should fix Issue #56. Used -isystem instead of -I, as it's an include of <cstdint> instead of "stdint.h". @erhlee-bird does this work on your system?
* Minor changes
* Cleanups to Operators.h. Specifically, to the signed versions of vector operations. They now more strictly enforce type safety. This caught a few bugs. Operators.h is arguable simpler as a result.
* merged conflict;
* MMX Instructions (PAVG/PHSUB/PMULHRS)
* Testcases for PADD/PSUB; Fix for signed/unsigned saturation;
* Test cases for PAVG/PABS/PACKSS; Fixes for MMX instructions handling;
* Fixed PACKSSWB instruction handling;
* PHADD/PHADDSW Instruction;
* PMAXSW/PMAXUB/PMINSW/PMINUB Instructions
* PCMP/PUNPCK/PMUL Instructions;
* Corrected space indentation;
* Fixed build issue;
* reimplemntation of instructions with saturation arithmetic
* PSLL/PSRL/PMUL instructions;
* Add in some missing arg parsing to Run.cpp. Set the minimum default log level in the test runner so that Travis logs don't explode so much, causing Travis to cancel an otherwise 'good' job.
* Update README.md (#62)
Fixed install path for install_gtest.sh
* Fixed failing testcases for PSLL/PSRA/PSRL/PACKSS
* Cleanup and resolved TODO comments;
* Added new testcases for MMX instructions;
* Fixes Issues #52 and #53. Addresses first comment of Issue #45.
* MIssing file
* Renaming test cases
* In progress changes. Moduler verifier is complaining about lack of \!dbg location entries. Adding a SourceWriter to produce something semi-useful.
* Linking lifted instruction code to printed out assembly code now works
* Added new instruction and testcase for CVTPI2PS. Made general improvements:
- made it so that disassembled instructions can be printed to a file, and linked back to the bitcode via debug info.
- eliminated the unnecessary _read and _write suffixes for registers in __remill_basic_block.
Mostly cosmetic changes. For example, limiting line length, putting spaces before the opening parens of loops, etc. Renamed all the semantics files to be cpp source files instead of header files. That stops cpplint from complaining about anonymous namespaces in headers ;-)
* Should fix Issue #56. Used -isystem instead of -I, as it's an include of <cstdint> instead of "stdint.h". @erhlee-bird does this work on your system?
* Minor changes
* Remove old 'test' function.
* Start to MMX PUNPCK instructions.
* Initial example for a PUNPCK instruction.
* Cleanups to Operators.h. Specifically, to the signed versions of vector operations. They now more strictly enforce type safety. This caught a few bugs. Operators.h is arguable simpler as a result.
* MMX Instruction support(PADD/PSUB); Fix for handling signed operations;
* Should fix Issue #56. Used -isystem instead of -I, as it's an include of <cstdint> instead of "stdint.h". @erhlee-bird does this work on your system?
* Minor changes
* Cleanups to Operators.h. Specifically, to the signed versions of vector operations. They now more strictly enforce type safety. This caught a few bugs. Operators.h is arguable simpler as a result.
* merged conflict;
* MMX Instructions (PAVG/PHSUB/PMULHRS)
* Testcases for PADD/PSUB; Fix for signed/unsigned saturation;
* Test cases for PAVG/PABS/PACKSS; Fixes for MMX instructions handling;
* Fixed PACKSSWB instruction handling;
* PHADD/PHADDSW Instruction;
* PMAXSW/PMAXUB/PMINSW/PMINUB Instructions
* PCMP/PUNPCK/PMUL Instructions;
* Corrected space indentation;
* Fixed build issue;
* reimplemntation of instructions with saturation arithmetic
* Working on factoring out external dependencies
* Making progress
* CMake for remill-lift seems to work :-)
* Improving build process and dependency list. Using a blank ubuntu 14.04 vm for testing.
* Still fighting with various dependencies.
* Making more progress with CMake stuff! Now have the semantics get compiled with cmake, and support installing stuff. I have also attempted to auto-find the semantics files when --bc_in is not provided.
* Adding protobuf compilation back to the steps.
* Improving readme
* Improving readme
* Switch to using the command.
* Updated travis. Updated README to include new build badge icons, to change where the ldconfig step goes, and to add a CMake notice for ubuntu trusty.
* Added script to generate tests. Added auto-yes in apt-add-repository commands.
* Minor fix
* Enable auto-finding of --bc_in based on --arch_in.
* Adding more test generation and compilation. Unfortunately compiling tests isn't working due to llvm::CloneFunctionInto not quite working anymore.
* llvm::CloneIntoFunction seems to work, which may resolve Issue #55. I'll need to double check with the official release version of LLVM; right now I'm using a custom debug build to try to catch issues. Added code to build and install gtest. Directly referenced which LLVM libraries must be linked. Added backup paths for deducing the semantics bitcode files based on the input arch. Made remill-lift report simple flag errors via stderr instead of using glog to abort the program. Fixed an issue where, when I added the various arch versions, xed was decoding avx-enabled instructions as 64-bit because the comparison condition deciding which decode state to use was too narrow.
* Tests generation and running scripts now work for me. Gotta test in fresh vms now.
* Addresses Issue #55. The problem was that the release version of clang 3.9 isn't guaranteed to preserve c++ variables name in the bitcode. I've added some stuff to re-introduce the variable names based on debug info.
* Build the test runner with rtti so that gtest doesn't complain on some versions of ubuntu.
* Fixing up Travis build stuff
* I am dumb; forgot to add the os-generic install file for travis to pick up.
* Change the trusty build type to generic, otherwise Travis uses a YUGE vm with all the things.
* Disable aptitude upgrading. dhclient is getting in the way.
* Re-order the linux-specific travis install script. Hope that protocol buffers gets installed correctly this time. Remove test generation and running from end of install script -- it doesn't belong there.
* Try to purge dhcp client, then add it back in, then do the normal update + upgrade
* Something is very dumb about Travi CI's generic linux build environment. It seems to be having trouble with the protobuf-compiler package -- that or somehting is going wrong just before that package is installed.
* Use the cpp build language -- I think this defaults to that mega vm though. Print out better OS information at the beginning of the travis install. Individually install needed packages, as opposed to doing it all at once.
* OK, using cpp language got me further, but builds take much longer. I was still missing cstdint, though. This time I'll try building with the generic trusty travis image, and with build-essential and the g++ 4.8 stdlib dev files installed.
* Lets try using the libstdc++ dev files
* Add in the x32 versions of the dev files. Hopefully that'll resolve the missing cstdint for the 32-bit builds of some of the instruction semantics bitcode files.
* Hopefully these are right!
* Install g++ multilib package.
* Simplifications to installer, and also to the README to now use g++-multilib.
Peter Goodman