* x86: Explicitly call __remill_undef for undefined arith flags.
* x86: Remove hack needed for pop [rsp] and introduce new semantic fn instead.
* test: Update test definitions with proper ignore flags.
* test: Propagate name of tests into gtest framework for better messages.
* arch: Make assert more verbose on failure.
* Refactors the code to improve directory layout, use more std::string_view in place of std::string, deletes some deprecated functions, deprecates some other functions for eventual deletion, and includes semantics for sparcv8 (sparc32) and sparcv9 (sparc64)
* Update new dir layout with llvm 11 support
* Whoops missing files
* Drop llvm 800 from workflow
* Minor fix
* Move where the install directives are in CMake
* Minor fixes
* Rename tools/ to bin/.
* Minor tweaks
* Should fix issues
* Running clang-format on files with some additional custom scripts for my style
* Fix missing unique_ptr in remill/BC/Optimizer.h
* Fixes and selective disabling of clang-format
* Add more diagnosis info to Run.cpp. Looks to be the x87 control, status, and tag word that are off after the signal handler catches the division by zero.
* Minor fixes, and more DIFF calls to drill down on differences.
* Should fix Issue #274, which is actually related to Arch::PrepareModule, and how it tries to get rid of double indirection with allocas and pointers.
* Fixes issue with finding SS_BASE on 32-bit x86 test suite.
* CMake: Make the bitcode generator Visual Studio friendly
* CMake: Always recompile the runtimes when the semantics change
* CMake: Style changes
* CMake: Fix missing variable when re-configuring
* CMake: Windows/Visual Studio fixes
* Windows: Add missing FPU macros
* Fix alignment issues with Visual Studio
* CMake: Fix the install target for Windows
* CMake: Add support for find_package(remill)
* Build script: Use verbose makefile
* CMake: Add missing ADDRESS_SIZE_BITS in BC compiler
* CMake: X86 Tests project refactor
* Windows: Port of BC/Utils and OS/FileSystem
* CMake: Update settings.cmake
* CMake: Generate the semantics path in remillConfig.cmake
* CMake: Only use C++14 when compiling on Windows
* Rename FPUFlags to Float and collect all FPU flags there
* Travis: Add macOS, test all LLVM versions, add clang static analyzer
* Travis: Disable LLVM50 tests, include verbose output for failed tests
* CMake: Update the find_package handler, add support for fcd headers
* Dead store elimination (draft) (#257)
* Add initial prototying of DeadStoreEliminator
* Improve upon DeadStoreEliminator prototype
Update `DeadStoreEliminator` prototyping with additional comments and
proper function prototype return values.
Fix a small warning in `Lifter.h` by adding `llvm::Value` and
`llvm::BasicBlock`.
* Add initial code for DeadStoreEliminator
Provide namespaced `remill::StateSlots` function which visits fields of
the module's state struct and returns `StateSlot` objects.
Remove `comment` field as the `State` object (an `llvm::StructType`)
does not track names in a useful way.
* Update state analyzer and fix type errors
Update `DeadStoreEliminator.cpp` to properly produce a vector of
`StateSlot`s without type errors (i.e. it compiles).
Code still needed for other container types besides structs.
* Add remill::VisitSequential for arrays and vectors
Add state analyzer code for LLVM's `ArrayType` and `VectorType`.
Add debugging use of `llvm::Type::dump()`.
Consider refactoring as a `StateVisitor` class.
* Begin refactor of Visit funcs to StateVisitor
* Fix segfault in StateVisitor::visit, code style
* Prototype ForwardAliasVisitor for alias analysis
Add code for ForwardAliasVisitor subclass of `llvm::InstVisitor`,
to be used for performing alias analysis of lifted functions.
* Add non-working visit functions for alias analysis
* Fix compilation of alias analysis visit funcs
* Change ForwardAliasVisitor to RetTy=bool
Update ForwardAliasVisitor to use booleans for return types to track
when we should add the instruction to the next_wl.
Add code to simplify StateSlots for vecs of ints.
* Add progress tracking to AliasAnalysis
* Correct progress tracker, use BasicBlockFunction()
* Change ForwardAliasAnalysis<RetTy = AliasResult>
Get some of those sweet sweet enums in there!
* Add FAV state pointer field, PHINode impl
* Allow non-const add and sub in FAV
Allow add and sub instructions with two pointers in the offset map.
Add to implementation of visitPHINode.
* Update PHINode impl in FAV
* Clean up use of StateSlots to create AAMDNodes
Add code to create AAMDNodes from StateSlot elements.
Change creation of StateSlot vector to have elements for every
byte offset of the state structure for fast indexing.
* Complete addition of AAMDNodes for load and store
Move function defs up for AAMDNode ops.
Add AliasMap typedef.
Finish generateAAMDNodesFromSlots.
* Add GenerateLiveSet func
* Initialize live set, add AAMDNodes to stores
* Change LiveSet creation to a block visitor class
* Add to_remove set to LSBV
* Update build script to use os-release for OS detection
* Update DSE code to conform to pag's review
* Refactor LiveSetBlockVisitor to one LiveSet per block
* Update VisitBlock to better check instruction type
* Fix VisitBlock CallInst and InvokeInst cases
* Stack allocate AAMDInfo in AnalyzeAliases
* Add remove pass option for VisitBlock
* Add DOT digraph generation
* Fix bugs in dot digraph
* Fix various bugs in AAMDNodes and LSBV
Move AAMDNodes functions later to match their use.
Clean up code conventions.
Fix small bugs in various spots in the code.
* Fix various function prototypes, overflow checks
* Merge AliasMap and OffsetMap
* Change add/sub insts to be safer
Add OpType enum class to replace use of `plus` bool.
Add more straightforward bounds checking on AddInst or SubInst values.
* Refactor GetUnsignedOffset style
* Fix illegal instruction error
* Add offset checking for GEP, provide log messages
Write a log message for the cases where `GetUnsignedOffset` returns
false (indicating an overflow or underflow).
* Fix APInt initialization in VisitGEP
* Re-add dead store elimination and alias map
* Move LSBV into alias analysis, expand callinst
Expand definition of cases where a CallInst should be considered to
touch the state struct or otherwise revive a slot.
* Improve DOT creation, fix errors for callinsts
Fix small errors in LiveSetBlockVisitor::CallAccessesState.
Clean up DOT digraph generation further.
* Clean up code per @pag's comments
* Clean up code, add selectinst, fix compile errors
Deal with a few small corner cases with FAV Load and Store instructions,
add SelectInst visitor.
Inline MarkLiveArgs to avoid the complications of C++ generics.
* Clean up logging
* More changes to please Peter
Clean up select and PHI node cases to improve circular dependency
handling.
* Correct errors in visitSelect
* Add load forwarding code
* Add code to run FBV
* Fix map usage error in FBV
* Add call, invoke cases for FBV
* Here are the changes whoops
* Minor API change
* Minor tweaks to DOT digraph printing, as well as attempts to handle the case where the forward analysis pass is incomplete. Still don't have it guarantee completion on everything, but seems 'good enough' for now.
* Change log level
* Fix compile errors due to messy merge
* Pag dead store (#262)
* Fix to script calling wrong function.
* Makes sure that value names are preserved (#249)
* Adds LLVM_VERSION() to accomodate llvm::LLVMContext::setDiscardValueNames() in <3.9 (#250)
* Makes sure that value names are preserved
* Adds LLVM_VERSION() macros to accomodate
* Update README.md
* Update README.md
* set state and memory as noalias (#254)
* Implements some ring 0 instructions in terms of hyper calls and new I/O port intrinsics. (#252)
* Random tests.
* More decode error info
* more playing around
* more system instructions. instrinsics for accessing I/O ports. Split our writes to individual control regs for better identification via hyper calls.
* CR8 read/write support (#255)
* Check argument index of function (#256)
* Fix NoAlias Attributes for older LLVM versions
* Fix for LLVM 4.0 and 3.9
* Fix typo
* Follow remill coding style
* Here are the changes whoops
* Minor API change
* Minor tweaks to DOT digraph printing, as well as attempts to handle the case where the forward analysis pass is incomplete. Still don't have it guarantee completion on everything, but seems 'good enough' for now.
* Change log level
* Fix compile errors due to messy merge
* Improve call/invoke case of FBV
* Improve call/invoke case of FBV
* Removes a level of indirection in the __remill_basic_block function
* Create dedicated stats tracker
Plus clean up the FBV visitor a teensy bit more.
* Move call/invoke LiveSet gen to static func
This commit is to pave the way for future improvements to where this
information is calculated (in FAV instead of LSBV).
* Add code to FAV for calls and invokes
Move call and invoke arg-based livesets to FAV to allow for module-level
LSBV code.
* Begin change of LSBV to module-level
* Minor bug fixes
* Remove some dead code
* Begin adding more code for call/invoke LSBV
* Add entry block checks for LSBV call/invoke
* Info about register names, as well as printing them in the digraphs.
* Make DOT printing only happen per function, as opposed to printing every function per DOT file. Minor tweaks to interprocedural analysis.
* Bug fixes and DOT printing improvements.
* Bug fixes and DOT printing improvements.
* Also print out DOT digraphs of functions after removing stuff
* Tried to make it treat everything before a call to __remill_error as dead but that didn't work out.
* Use datalayout and type sizes to handle the number of elements in a sequential type for LLVM 3.8 compatibility, also check for pointer type.
* More stats, hopefully fixes a bitcast issue.
* Minor bug fixes, and more DOT printing to help diagnose when the offset analysis terminates but there is still stuff in the work list.
* Some possible bug fixes
* Minor bug fix
* Travis build fixes for earlier compatibility
* Fix for LLVM less than 3.8 compatibility.
* LLVM 3.5 compatibility
* Missing condition in forwarding code that does casting.
* LLVM 5.0 compatibility
* Add LLVM 6.0 to build.sh. Change default install location to /usr/local
* Add back in vmill. Add some extra flags to the building runtimes.
* Maybe works
* Disable dse across indirect call (#267)
* Make DSE sensitive to indirect function calls
* FE_DENORM issue
* Added some compat code that implements the futimens syscall on mac os 10.12 for travis support.
* Disable dse across indirect call (#268)
* Make DSE sensitive to indirect function calls
* FE_DENORM issue
* Added some compat code that implements the futimens syscall on mac os 10.12 for travis support.
* Playing with CACHE and PARENT_SCOPE
* Minor stack address size check
* Fix to DSE I think.
* Minor DSE tweak
* Update Run.cpp
* Update travis.sh
Adds in 32-bit libraries for Linux builds.
* Random tests.
* More decode error info
* more playing around
* more system instructions. instrinsics for accessing I/O ports. Split our writes to individual control regs for better identification via hyper calls.
* Initial support for compilation on OS X
* Special-cased the size_t code for Apple, since it errors on Linux
* Support for OS X
* MACH-compatible ASM for OS X
* Working on macOS platform support for Remill.
* Support for OS X in the build script
* Use curl instead of wget.
* Remove illegal instruction from tests (#205)
The code segment (CS) register cannot be explicitly loaded, unlike the
other segment registers. The instruction `mov cs, rax` is thus illegal,
and triggered a SIGILL for each test case, causing all tests to
spuriously succeed.
* Remove unneeded DS register preservation around test cases
The commit that first added this included an illegal instruction, which
caused false positives and hid the fact that this is redundant.
* Revert "Restore previously failing FMA tests"
This reverts commit 9a9f370d08.
The bug in #205 is what made it seem like the underlying issue here had
been addressed.
* Include segment registers in `SaveState()`
* If in 32-bit mode, check if we should zero out x87 CS
* Added semantics and tests for SDIV
* Semantics and tests for SCVTF.
* Working on integer to/from float conversion bugs related to managing the fpu status register.
* Switch to using the check float function wrapper
* Remove unintentional call to fetestexcept
* Adding in a compiler reordering barrier.
* Added inline assembly that will force clang's optimizer to give up and filed a clang bug
* Adds some trailing spaces into some files. Re-enables all aarch64 tests.
* Added semantics and tests for SDIV
* Semantics and tests for SCVTF.
* Switch to using the check float function wrapper
* Remove unintentional call to fetestexcept
* Added inline assembly that will force clang's optimizer to give up and filed a clang bug
* Adds some trailing spaces into some files. Re-enables all aarch64 tests.
* Bringing the aarch64 branch back in line with master.
* Adds in a test and ISEL for LDR_BL_LDST_REGOFF, and fixes up the instructions in some SMOV and UMOV tests.
* Various fixes and some minor debugging aids for aarch64
* Resets the fenv before running lifted code.
* All existing aarch64 tests should now pass.
* Implements more accurate tracking of x87 exception flags, last instruction pointer, last data pointer, and floating point instruction opcode.
* Fix that masks out the MXCSR register from being compared at the end of tests. We don't yet have good enough info to do this well. Moves the FCMOV instruction set to be under the purview of X87.cpp and its tests.
* Tests should pass now.
* Addressing issues Mike mentioned.
* Post-decoder, semantics, and tests for BFM.
* Semantics and tests for ANDS
* Minor naming conflict fix
* Semantics and tests for SMADDL and SMULH
* Semantics and tests for MADD
* Minor symbol renaming fix
* Semantics and tests for EXTR, which also implements ROR
* Minor fix to number of arguments to test
* Semantics, but NO tests for LSLV, LSRV, ASRV, and RORV.
* Semantics and tests for ORN
* Minor fix
* Semantics for SBC, but no tests. Added an AddWithCarryNZCV primitive, untested
* Missing two's complement negation in subs using new addwithcarrynzcv
* Tests for SBC. Semantics and tests for SBCS
* Semantics and tests for UCVTF
* Update state save/restore code to record SIMD regs
* Attempt to fix save/restore code
* More attempts at getting the save/restore assembly right, this time with the fpsr bits.
* More fixes
* More tests extracting the fpsr state. Also trying without including RestoreState.S. I don't remember why I had that.
* Semantics for SVC and BRK
* Implemented tests for a few add and sub variants. Made it so that the post-decoders of some of the add/sub variants refer to eachother.
* Tests and another decoder for EOR
* Fixes to semantics generated for shift register operands
* Fixes and tests for cmp instruction. Really interesting because it does a - b, but as an adc(a, ~b, 1).
* Added post-decoders and tests for the add and sub variants that use extract, extend, and shift operators
* Minor fixes
* Minor fixes
* Minor fixes
* Minor improvements, though tests still failing
* Fix. Passed the wrong data size into the AddExtendRegOp function.
* Fix. Passed the wrong data size into the AddExtendRegOp function.
* Not sure if I did anything significant
* Changed the aarch64 test case lifter to support testing branches
* Tests for CBZ and CBNZ
* CBZ and CBNZ were testing the wrong operands
* Added semantics and tests for CMN
* Semantics and tests for SUBS
* Tests for ADDS
* Semantics and tests for AND, ORR, and BIC
* Reordered pc, state, and memory to help with mcsema's legacy mode
* Added in more aarch64 instructions. Fixed some x86 instructions. The x86 test cases now exercise each test through every possible combination of flags.
* Adding missing files
* Another missing file
* Rename file
* Fixup some macros
* IPR
* More improvements on the test runner
* Test runner fixes related to me not being familiar with aarch64 assembly
* Fixing default data layout
* Trying to use llc to compile bitcode to aarch64 assembly. wth.
* Revert back to using the CMAKE_BC_COMPILER for building the test assembly file instead of the whole CMAKE_LL_COMPILER stuff, now that I've adjusted cxx-common to use the right build target for aarch64.
* Documentation updates. Fixes for aarch64.
* Making progress. The native tests can run, but the first lifted test faults. Not yet sure why.
* Weirdest issue is happening on aarch64. A pointer argument is being compiled to an integer, and that's really screwing things up.
* Add caching of the libraries path to the main cmakelists to avoid having to re-run build.sh all the time when the TRAILOBITS_LIBRARIES env var is not globally defined. Experimenting with trying to force the semantics to be compiled using the x86_64 target, regardless of host arch, or modelled arch of the semantics. This is to try to get around the issue where a single-element struct containing a pointer is lowered into a uintptr_t when passed by value as an argument on aarch64.
* Alright, falling back on handling this problem in the lifter (for now, at least). Really not ideal.
* Test runner works afaict
* Change copyright notice in all the places. Trying to get cmake to download all the things
* CMake refactor
* Added the lib repository installer
* Fixed a couple of paths in library repository scripts
* Spelling and a missing .gitignore file.
* The google test library was not correctly linked
* Removed the pre-compiled XED library
* Removed unused files; fixed the INSTALL directives. Added automatic protobuf generation.
* Added a package generator script for ArchLinux
* Merged in Alessandro's cmake magic
* Cleanups, compatibility changes
* CMake cleanup.
o Removed the cmake folder as it is no longer used.
o Removed the library_repository_installer directory and
replaced it with a submodule to the newly introduced
cxx_common repository.
o Added the 'use_remill_semantics' branch of mcsema as a
submodule under tools/cmake.
o Updated the README instructions (added --recursive to the
git clone command).
Warning: the mcsema submodule is pointing to my fork of the
repository!
* CMake: avoid re-defining the C/CXX/ASM compiler.
This will prevent CMake from looping forever when using submodules.
* The tools/mcsema submodule now points to the official repository.
* mcsema submodule update
* Updated the mcsema git submodule to track the newest changes.
* Various CMake fixes (see details).
o External include headers were not correctly marked
as SYSTEM. This caused them to output warnings and
break the build.
o The PROJECT_SOURCE CMake variable has been replaced
with CMAKE_SOURCE_DIR.
o Updated the mcsema submodule to point to the latest
CMake fixes.
* Getting closer to having the test case runner work again without using the CFG protobufs.
* Trying to make things use ubuntu clang/llvm packages
* Update to the mcsema submodule.
* Minor changes for llvm version compatibility
* Test cases should run now. Had to compile the lifted testcases to a .S file, as with the .bc file, they were being optimized away.. I think. The compilation to assembly seems unusually slow, though.
* Minor change
* Some stuff for llvm 4.0 support
* Remove mcsema sub-module
* Remove cxx-common submodule
* First steps toward getting travis working again
* Attempt at getting travis working again
* Using https for cloning instead of ssh
* Minor build script update
* Added ISEL_ prefix to isels to make it easier for ForEachISel to find them. Commented out the defer_inlining intrinsics.
* Changes related to mcsema2
* Simplify runtime targets generation. (#110)
* CMake refactor: Added a new language 'BC' for the bitcode (see details).
The language is used to generate the runtimes used by the architecture
modules. The required executables (clang++ and llvm-link) are
automatically detected in the same way as other compilers are.
A new CMake function has been added and it can be used to easily
generate runtime targets in a way similar to add_executable:
add_runtime(<name> SOURCE <source list> ADDRESS_SIZE <n>
DEFINITIONS <definition list>)
Additionally, all files ending with the *.bcpp extensions will
automatically invoke the bitcode compiler when listed in an
active target.
This should open support for parallel compilation! You just have to
list all your .cpp files when calling add_runtime.
* CMake: Fixes to the BC language handler.
* CMake/BC: Use '.bo' for object files. X86 runtime: Add -g/-O flags.
* Initial commit of x86 program snapshotter. Haven't yet figured out how to save a core dump to a specific file.
Now finds and saves core dumps (in a sketchy way).
Kind of but didn't really fix remill-disass with core dumps. It almost seems like a bunch of stuff is missing from within a core dump. It may be the case that it will be simpler to use binary ninja directly on snapshot files.
Working toward the executor.
Have snapshot creation and loading working. Next up: decoding the first few instructions!
Starting to serverize remill and setting up lines of communicating between vmill and remill. Going to move on to finally implementing remill-opt.
Changed a bazillon things.
Fixed the dead register backward data-flow analysis.
Fixed soem bugs, added GEP re-association, working toward inserting stores that will kill values.
Got interprocedual dead store elimination working! :-D
Finally...dead store elimination
remill-opt is done.
Added Lifter interface
Spec'ing out the translation engine.
Got the dynamic decoder working!
Got bitcode caching working
Added caching layer for bytecode compilation.
Decided on how to access reg state and allocas in functions. The bytecode will treat the state struct and the alloca'd data as a contiguous, opaque, byte-addressible area.
Compiler seems to work
Refactor
Great progress...goodnight
Made it up to the first syscall
Fixed call to a hypercall intrinsics
Bug fixes, minor change to the CFG proto.
Refacotorings and changes
* Switching to trying nativeexec, and made memory32 map snapshot into low 32-bits of address space, preserving original addresses of program.
* About to make some interesting changes, so save save save
* Refactor done, now time to produce shared libraries
* Got initial execution of some stuff in PHP working.. it either seems like the code is in an infinite loop, or just horribly slow, not sure. Otherwise, amazing progress.
* Got initial compiling to a runtime dynamic library working
* Separated most DEF_ISEL_SEMs and tests still pass. Goodnight!
* Got incremental optimization and compilation working.
* Got caching of the bitcode file to disk working and periodically collapsing the shared libraries into a single library. OUT.
* Fixed some bugs
* Commit before the storm
* Made the JIT work again, still not that fast though.
* Minor logging fix
* Log an error that we're executing a missing instruction.
* JIT compile the whole module first, then incrementally JIT compile function partitions. Also, link in libm.
* Added new syscall
* Added breakpoint sync hypercall, useful for testing.
* Got a php5.4 unserialize bug to reproduce. Added remill-pinshot, which will use PIN to take a snapshot and print out a register trace. This is useful for debugging divergences. Fixed up the semantics of some instructions, and added semantics and tests for PSRLDQ.
* Simplifications to remill that removes all the various basic block arrays, and uses meta-data instead.
* Made cmake take over the test system
* Working on mega refactor to eventually permit klee support
* Made all semantics code return memory pointer; I think this makes LLVM's optimizer a bit happier. Made the __remill_sub_N things into global constants, referencing the actual functions, that have private linkage. Slowly getting back in the direction of execution. I've got some bitcode translation working (with a hard-coded address for my local php version). The address space stuff seems to work so far.
* Missing files.
* Other minor fixes
* rtti-related fixes for gtest
* minor travis config change
* minor travis config change
* sendfile no longer needed
* Disable building vmill on non-linux platforms
* Disable 32-bit test builds on macOS.
* Trying to get symbol names on mac right.
* Minor fix to save state code for tests, disable testing on macOS builds because of symbol mismatches in gtest.
* Fixup remill-disass to use the simplified proto structure. Something may be wrong with remill-opt, or installing remill opt. Remove caching from travis.
* Removing unnecessary file.
* Implementation and tests for pcmpistri
* Remove uncommented stuff
* Define issignalling when it's not available as a macro.
* tests: Add missing gflags include in tests/X86/Run.cpp.
* tools/remill_disasm: Fix invalid syntax in ida/disass.py.
Fixes#96.
Also removing trailing whitespace.
* Backporting changes from vmill to remill. One set of changes is to remove DEF_ISEL_SEM, requiring that one use DEF_ISEL and DEF_SEM separately. This makes some simple instruction semantics more tedious to implement. The main goal is for ALL semantics themselves to be treated as private, non-exported symbols. In the case of those semantics defined via DEF_ISEL_SEM, those weren't private, and this inhibited inlining optimizations. Another big change is that function cloning has been rewritten and no longer depends on LLVM's function cloning implementation. Another big change is that the meat of the optimizer has been put into the remill/BC subdirectory and wrapped in a class, enabling it to be used independently of remill-opt. There are some new instruction semantics, and fixes to those. Some new utility functions for iterating over lifted blocks in the module. Attach/detach intrinsics have been removed, as they don't (yet) explicitly fit within the current model. The AssemblyWriter, which would produce a .S file and debug info linking lifted bitcode to the .S file via source line debug info has been removed. Despite it being really cool, the actual benefit is not obvious, given that optimization of the bitcode doesn't actually propagate enough of that information after inlining.
* Eliminate sendfile dependency.
* Re-order state and memory pointers in all the things
* Fixes to remill-opt, added memory intrinsic lowering to remill-opt, removed the NEXT_PC pointer from within the basic block intrinsic template.
2over12