9 Commits

Author	SHA1	Message	Date
Tim Alberdingk Thijm	bf82eb39bb	Dead store elimination (draft) (#257) ... * Add initial prototying of DeadStoreEliminator * Improve upon DeadStoreEliminator prototype Update `DeadStoreEliminator` prototyping with additional comments and proper function prototype return values. Fix a small warning in `Lifter.h` by adding `llvm::Value` and `llvm::BasicBlock`. * Add initial code for DeadStoreEliminator Provide namespaced `remill::StateSlots` function which visits fields of the module's state struct and returns `StateSlot` objects. Remove `comment` field as the `State` object (an `llvm::StructType`) does not track names in a useful way. * Update state analyzer and fix type errors Update `DeadStoreEliminator.cpp` to properly produce a vector of `StateSlot`s without type errors (i.e. it compiles). Code still needed for other container types besides structs. * Add remill::VisitSequential for arrays and vectors Add state analyzer code for LLVM's `ArrayType` and `VectorType`. Add debugging use of `llvm::Type::dump()`. Consider refactoring as a `StateVisitor` class. * Begin refactor of Visit funcs to StateVisitor * Fix segfault in StateVisitor::visit, code style * Prototype ForwardAliasVisitor for alias analysis Add code for ForwardAliasVisitor subclass of `llvm::InstVisitor`, to be used for performing alias analysis of lifted functions. * Add non-working visit functions for alias analysis * Fix compilation of alias analysis visit funcs * Change ForwardAliasVisitor to RetTy=bool Update ForwardAliasVisitor to use booleans for return types to track when we should add the instruction to the next_wl. Add code to simplify StateSlots for vecs of ints. * Add progress tracking to AliasAnalysis * Correct progress tracker, use BasicBlockFunction() * Change ForwardAliasAnalysis<RetTy = AliasResult> Get some of those sweet sweet enums in there! * Add FAV state pointer field, PHINode impl * Allow non-const add and sub in FAV Allow add and sub instructions with two pointers in the offset map. Add to implementation of visitPHINode. * Update PHINode impl in FAV * Clean up use of StateSlots to create AAMDNodes Add code to create AAMDNodes from StateSlot elements. Change creation of StateSlot vector to have elements for every byte offset of the state structure for fast indexing. * Complete addition of AAMDNodes for load and store Move function defs up for AAMDNode ops. Add AliasMap typedef. Finish generateAAMDNodesFromSlots. * Add GenerateLiveSet func * Initialize live set, add AAMDNodes to stores * Change LiveSet creation to a block visitor class * Add to_remove set to LSBV * Update build script to use os-release for OS detection * Update DSE code to conform to pag's review * Refactor LiveSetBlockVisitor to one LiveSet per block * Update VisitBlock to better check instruction type * Fix VisitBlock CallInst and InvokeInst cases * Stack allocate AAMDInfo in AnalyzeAliases * Add remove pass option for VisitBlock * Add DOT digraph generation * Fix bugs in dot digraph * Fix various bugs in AAMDNodes and LSBV Move AAMDNodes functions later to match their use. Clean up code conventions. Fix small bugs in various spots in the code. * Fix various function prototypes, overflow checks * Merge AliasMap and OffsetMap * Change add/sub insts to be safer Add OpType enum class to replace use of `plus` bool. Add more straightforward bounds checking on AddInst or SubInst values. * Refactor GetUnsignedOffset style * Fix illegal instruction error * Add offset checking for GEP, provide log messages Write a log message for the cases where `GetUnsignedOffset` returns false (indicating an overflow or underflow). * Fix APInt initialization in VisitGEP * Re-add dead store elimination and alias map * Move LSBV into alias analysis, expand callinst Expand definition of cases where a CallInst should be considered to touch the state struct or otherwise revive a slot. * Improve DOT creation, fix errors for callinsts Fix small errors in LiveSetBlockVisitor::CallAccessesState. Clean up DOT digraph generation further. * Clean up code per @pag's comments * Clean up code, add selectinst, fix compile errors Deal with a few small corner cases with FAV Load and Store instructions, add SelectInst visitor. Inline MarkLiveArgs to avoid the complications of C++ generics. * Clean up logging * More changes to please Peter Clean up select and PHI node cases to improve circular dependency handling. * Correct errors in visitSelect * Add load forwarding code * Add code to run FBV * Fix map usage error in FBV * Add call, invoke cases for FBV * Here are the changes whoops * Minor API change * Minor tweaks to DOT digraph printing, as well as attempts to handle the case where the forward analysis pass is incomplete. Still don't have it guarantee completion on everything, but seems 'good enough' for now. * Change log level * Fix compile errors due to messy merge * Pag dead store (#262) * Fix to script calling wrong function. * Makes sure that value names are preserved (#249) * Adds LLVM_VERSION() to accomodate llvm::LLVMContext::setDiscardValueNames() in <3.9 (#250) * Makes sure that value names are preserved * Adds LLVM_VERSION() macros to accomodate * Update README.md * Update README.md * set state and memory as noalias (#254) * Implements some ring 0 instructions in terms of hyper calls and new I/O port intrinsics. (#252) * Random tests. * More decode error info * more playing around * more system instructions. instrinsics for accessing I/O ports. Split our writes to individual control regs for better identification via hyper calls. * CR8 read/write support (#255) * Check argument index of function (#256) * Fix NoAlias Attributes for older LLVM versions * Fix for LLVM 4.0 and 3.9 * Fix typo * Follow remill coding style * Here are the changes whoops * Minor API change * Minor tweaks to DOT digraph printing, as well as attempts to handle the case where the forward analysis pass is incomplete. Still don't have it guarantee completion on everything, but seems 'good enough' for now. * Change log level * Fix compile errors due to messy merge * Improve call/invoke case of FBV * Improve call/invoke case of FBV * Removes a level of indirection in the __remill_basic_block function * Create dedicated stats tracker Plus clean up the FBV visitor a teensy bit more. * Move call/invoke LiveSet gen to static func This commit is to pave the way for future improvements to where this information is calculated (in FAV instead of LSBV). * Add code to FAV for calls and invokes Move call and invoke arg-based livesets to FAV to allow for module-level LSBV code. * Begin change of LSBV to module-level * Minor bug fixes * Remove some dead code * Begin adding more code for call/invoke LSBV * Add entry block checks for LSBV call/invoke * Info about register names, as well as printing them in the digraphs. * Make DOT printing only happen per function, as opposed to printing every function per DOT file. Minor tweaks to interprocedural analysis. * Bug fixes and DOT printing improvements. * Bug fixes and DOT printing improvements. * Also print out DOT digraphs of functions after removing stuff * Tried to make it treat everything before a call to __remill_error as dead but that didn't work out. * Use datalayout and type sizes to handle the number of elements in a sequential type for LLVM 3.8 compatibility, also check for pointer type. * More stats, hopefully fixes a bitcast issue. * Minor bug fixes, and more DOT printing to help diagnose when the offset analysis terminates but there is still stuff in the work list. * Some possible bug fixes * Minor bug fix	2018-06-28 13:55:49 -04:00
Peter Goodman	8456f10787	Improvements and packuswb (#165) ... * This clarifies the difference between kCategoryInvalid and kCategoryError for instruction categories. * Removes the --use-system-compiler option, and requires that clang always be used. * Some general improvements, as well as semantics and tests for PACKUSWB.	2017-11-27 13:36:50 -05:00
Peter Goodman	6a3f112649	These changes help with some compatibility issues across LLVM versions, and help out with VMill as well. (#159)	2017-11-09 14:52:59 -05:00
Peter Goodman	35fb133a6d	Adds some code that got lost when doing some weird branching	2017-11-04 20:44:50 -04:00
Peter Goodman	b96aec8fd8	Issue 137 lift bin ls (#155) ... * [WIP] Float Instructions for /bin/ls (#146) * Added bunch of float instructions required by `ls` Not all variants implemented but should follow same structure * FADD * FMUL * FMOV * FCVTZU * FCMPE Fixed bugs with FMOV_n_to_n Implemented FMOV_D64 variant Fixed up FADD, need to doublecheck edgecases FMUL Added Weird interaction with checking for denormalized output fpclassify(prod) gives you FP_SUBNORMAL, but in the native code the idc bit on SR and FPSR is not set so commented out for now FCVTZU added - FPSR flags not being caught and set Type fix In<float64_t> Fixes for FCVTZU Added FCMPE Sn Cleanup testing * Check unallocated encoding for float insts * Minor modifications. * Trying to wrap up some fpu operations inside of a function that can test the inputs and outputs. * Semantics, but no tests, for some variants of FMUL, FADD, and FDIV. * Tests for FDIV and FSUB. Additional tests for FADD and FMUL. * Minor tweak. * Minor tweak to using fetestexcept only once * Added UNTESTED implementation of mrs with fpsr. * FMOV vector instructions, some fixes (#149) * FMOV vector instructions, some fixes * More tests for FMOV_VECTOR * Fix to STRH isel, uses the wrong semantics. Minor modification to add DEF_CONDs to the ForEachIsel iterator. * Various instructions for aarch64 lifting WIP * Tests working except where commented Specifically STRH test throws some lifter error FCVT has issues with flag determiniation * Bug fix with oxc fpsr bit in print state * FCVTS, FMOV_S, FMOV_D * FNEG, FMADD_S/D For FMADD still seeing weird issues with flag setting on multiplication step when using denormals as factor, look into later * More precise definition of segment selectors. Directory traversal utility function. * Minor tweaks to existing APIs. * FMADD has a bug with one type * Fixed reimport of test * Semantics and tests for BICS. * Minor fix to tests. * Semantics and tests for integer REV instructions. * Minor fix. * Minor fixes, forgot an ISEL. * Tests for MOVK. * Tests for MOVZ. * Forgot test file. * More STR-related tests. * Minor fix * Minor fix * Post-decoder fixes for STR. * Minor fix? * Semantics and tests for general simd DUP. * Minor fix for arrangement specifier. Found a possible bug in capstone/keystone. * Wrong isel in test * Semantics and tests for simd ADD and SUB. * Semantics and tests for LD1_ASISDLSEP_I2_I2 * Whoops, forgot TEST_INPUTS to test. * Semantics and tests for LD1_ASISDLSEP_I1_I1 * Used wrong isel names in tests. * Tests and semantics for two more variants of post-index LD1 * Minor fix * Minor fix to a check for large op sizes. Regenerated AArch64 Extract.cpp, ordering extractors by most constrained first. I observed an issue where LD1_ASISDLSEP_I3_I3 was incorrectly extracted as LD1_ASISDLSEP_R3_R3. * Minor fix of a decoder routine for loading multiple structures. * Semantics and tests for RBIT, SMOV, and UMOV. Also semantics, but NO tests for a large number of SIMD instructions. * Tests for simd smax, smin, umax, umin. * Tests for sminp, smaxp, uminp, umaxp. * Tests for ADDP. * Tests for the simd vector register of cmeq, cmgt, and cmge. * Minor fix. * Tests for cmeq, cmgt, and cmge that compare a vector against zero.	2017-11-04 15:52:23 -04:00
Peter Goodman	e1bdbdba9e	Alessandro/refactor/llvmmultiversion cmake (#111) ... * Change copyright notice in all the places. Trying to get cmake to download all the things * CMake refactor * Added the lib repository installer * Fixed a couple of paths in library repository scripts * Spelling and a missing .gitignore file. * The google test library was not correctly linked * Removed the pre-compiled XED library * Removed unused files; fixed the INSTALL directives. Added automatic protobuf generation. * Added a package generator script for ArchLinux * Merged in Alessandro's cmake magic * Cleanups, compatibility changes * CMake cleanup. o Removed the cmake folder as it is no longer used. o Removed the library_repository_installer directory and replaced it with a submodule to the newly introduced cxx_common repository. o Added the 'use_remill_semantics' branch of mcsema as a submodule under tools/cmake. o Updated the README instructions (added --recursive to the git clone command). Warning: the mcsema submodule is pointing to my fork of the repository! * CMake: avoid re-defining the C/CXX/ASM compiler. This will prevent CMake from looping forever when using submodules. * The tools/mcsema submodule now points to the official repository. * mcsema submodule update * Updated the mcsema git submodule to track the newest changes. * Various CMake fixes (see details). o External include headers were not correctly marked as SYSTEM. This caused them to output warnings and break the build. o The PROJECT_SOURCE CMake variable has been replaced with CMAKE_SOURCE_DIR. o Updated the mcsema submodule to point to the latest CMake fixes. * Getting closer to having the test case runner work again without using the CFG protobufs. * Trying to make things use ubuntu clang/llvm packages * Update to the mcsema submodule. * Minor changes for llvm version compatibility * Test cases should run now. Had to compile the lifted testcases to a .S file, as with the .bc file, they were being optimized away.. I think. The compilation to assembly seems unusually slow, though. * Minor change * Some stuff for llvm 4.0 support * Remove mcsema sub-module * Remove cxx-common submodule * First steps toward getting travis working again * Attempt at getting travis working again * Using https for cloning instead of ssh * Minor build script update * Added ISEL_ prefix to isels to make it easier for ForEachISel to find them. Commented out the defer_inlining intrinsics. * Changes related to mcsema2 * Simplify runtime targets generation. (#110) * CMake refactor: Added a new language 'BC' for the bitcode (see details). The language is used to generate the runtimes used by the architecture modules. The required executables (clang++ and llvm-link) are automatically detected in the same way as other compilers are. A new CMake function has been added and it can be used to easily generate runtime targets in a way similar to add_executable: add_runtime(<name> SOURCE <source list> ADDRESS_SIZE <n> DEFINITIONS <definition list>) Additionally, all files ending with the *.bcpp extensions will automatically invoke the bitcode compiler when listed in an active target. This should open support for parallel compilation! You just have to list all your .cpp files when calling add_runtime. * CMake: Fixes to the BC language handler. * CMake/BC: Use '.bo' for object files. X86 runtime: Add -g/-O flags.	2017-05-15 13:00:33 -04:00
Peter Goodman	7dd87902ea	Vmill (#106) ... * Initial commit of x86 program snapshotter. Haven't yet figured out how to save a core dump to a specific file. Now finds and saves core dumps (in a sketchy way). Kind of but didn't really fix remill-disass with core dumps. It almost seems like a bunch of stuff is missing from within a core dump. It may be the case that it will be simpler to use binary ninja directly on snapshot files. Working toward the executor. Have snapshot creation and loading working. Next up: decoding the first few instructions! Starting to serverize remill and setting up lines of communicating between vmill and remill. Going to move on to finally implementing remill-opt. Changed a bazillon things. Fixed the dead register backward data-flow analysis. Fixed soem bugs, added GEP re-association, working toward inserting stores that will kill values. Got interprocedual dead store elimination working! :-D Finally...dead store elimination remill-opt is done. Added Lifter interface Spec'ing out the translation engine. Got the dynamic decoder working! Got bitcode caching working Added caching layer for bytecode compilation. Decided on how to access reg state and allocas in functions. The bytecode will treat the state struct and the alloca'd data as a contiguous, opaque, byte-addressible area. Compiler seems to work Refactor Great progress...goodnight Made it up to the first syscall Fixed call to a hypercall intrinsics Bug fixes, minor change to the CFG proto. Refacotorings and changes * Switching to trying nativeexec, and made memory32 map snapshot into low 32-bits of address space, preserving original addresses of program. * About to make some interesting changes, so save save save * Refactor done, now time to produce shared libraries * Got initial execution of some stuff in PHP working.. it either seems like the code is in an infinite loop, or just horribly slow, not sure. Otherwise, amazing progress. * Got initial compiling to a runtime dynamic library working * Separated most DEF_ISEL_SEMs and tests still pass. Goodnight! * Got incremental optimization and compilation working. * Got caching of the bitcode file to disk working and periodically collapsing the shared libraries into a single library. OUT. * Fixed some bugs * Commit before the storm * Made the JIT work again, still not that fast though. * Minor logging fix * Log an error that we're executing a missing instruction. * JIT compile the whole module first, then incrementally JIT compile function partitions. Also, link in libm. * Added new syscall * Added breakpoint sync hypercall, useful for testing. * Got a php5.4 unserialize bug to reproduce. Added remill-pinshot, which will use PIN to take a snapshot and print out a register trace. This is useful for debugging divergences. Fixed up the semantics of some instructions, and added semantics and tests for PSRLDQ. * Simplifications to remill that removes all the various basic block arrays, and uses meta-data instead. * Made cmake take over the test system * Working on mega refactor to eventually permit klee support * Made all semantics code return memory pointer; I think this makes LLVM's optimizer a bit happier. Made the __remill_sub_N things into global constants, referencing the actual functions, that have private linkage. Slowly getting back in the direction of execution. I've got some bitcode translation working (with a hard-coded address for my local php version). The address space stuff seems to work so far. * Missing files. * Other minor fixes * rtti-related fixes for gtest * minor travis config change * minor travis config change * sendfile no longer needed * Disable building vmill on non-linux platforms * Disable 32-bit test builds on macOS. * Trying to get symbol names on mac right. * Minor fix to save state code for tests, disable testing on macOS builds because of symbol mismatches in gtest. * Fixup remill-disass to use the simplified proto structure. Something may be wrong with remill-opt, or installing remill opt. Remove caching from travis.	2017-03-23 15:49:33 -04:00
Peter Goodman	1a2e4d11f9	Vmill backport (#81) ... * Backporting changes from vmill to remill. One set of changes is to remove DEF_ISEL_SEM, requiring that one use DEF_ISEL and DEF_SEM separately. This makes some simple instruction semantics more tedious to implement. The main goal is for ALL semantics themselves to be treated as private, non-exported symbols. In the case of those semantics defined via DEF_ISEL_SEM, those weren't private, and this inhibited inlining optimizations. Another big change is that function cloning has been rewritten and no longer depends on LLVM's function cloning implementation. Another big change is that the meat of the optimizer has been put into the remill/BC subdirectory and wrapped in a class, enabling it to be used independently of remill-opt. There are some new instruction semantics, and fixes to those. Some new utility functions for iterating over lifted blocks in the module. Attach/detach intrinsics have been removed, as they don't (yet) explicitly fit within the current model. The AssemblyWriter, which would produce a .S file and debug info linking lifted bitcode to the .S file via source line debug info has been removed. Despite it being really cool, the actual benefit is not obvious, given that optimization of the bitcode doesn't actually propagate enough of that information after inlining. * Eliminate sendfile dependency.	2017-01-05 12:22:10 -05:00
Peter Goodman	65f3812cce	Opt (#76) ... * Implements remill-opt * Change back to google namespace * ANother namespace change * Should fix test runner. * Try to fix deprecated use of fstat64 on macOS	2016-11-28 20:44:02 -05:00