* Preliminary implementation of CFG edge spilling.
* Many tweaks.
* Use little endian; switch to native bytes and msgspec.
* More tweaking.
* Fix test cases.
* Get rid of _all_keys.
* Lint code.
* Type check.
* Preliminary implementation of a spilling CFG graph.
* Fix some test cases.
* Fix another test case.
* Fixes.
* Lint code.
* Implement InEdgeView and OutEdgeView.
* Implement InDegreeView and OutDegreeView.
* Fixes.
* Use (Node.addr, Node.size) as key.
* Fix test cases.
* More fixes.
* Fix CFGENode support.
* Fix the remaining issues.
* Get rid of SpillingCFGGraph.reverse().
* Some refactoring.
* Oops
* Fix perf issue.
* Fix set size change error during key iteration.
* Fix db_batch_size assignment. Fix some test cases.
* SpillingCFGNodeDict.__setstate__: Initialize self._all_keys.
* Getting rid of the nodes dictionary.
* Fix logic in cfg_emulated.py
* Multiple fixes.
* Type annotation fix.
* Fix missing edges after merging CFGNodes.
* Fix more logic.
* Add Soot CFGNodes to the CFG model.
* Lint and type check.
* Add USE_SPILLING_CFGNODE_DICT and CFGNode.dirty.
* Oops
* Lint code.
* No more pickling of CFGNodes.
* Fix no_ret being None.
* Lint code.
* Some refactor; Destroy the fallback mechanism.
* Fix node dirty bug after deserialization.
* FunctionManager: Spill to external storage.
* Remove atexit registration. Reduce map size.
* Introduce FuncNode in function graphs; Fix multiple issues with SpillingFunctionDict.
* Implement Function.dirty.
* A bit more optimization.
* FunctionManager loads only meta data for functions when graphs are not accessed; Save .info for functions.
* Only load meta data for Functions in more places.
* Remove FunctionManager.block_map because it's never really used.
* Retire blockaddr_to_function and replace it with blockaddr_to_funcaddr.
* More optimizations and fixes.
* More refactor and fixes.
* More optimizations.
* Fix the bug in lmdb spilling after raising MapFullError.
* Introduce RuntimeDb in KB. Migrate SpillingFunctionsDict to use RuntimeDb.
* Prioritize the basedir of the main executable for the runtime db path.
* Cache non-returning function addrs, unknown-returning function addrs, and function block count in FunctionManager.
* Bug fixes.
* Fix a bug in FunctionParser.
* CFGBase.make_functions: Copy over function metadata when creating functions in the first place.
* Update MockFunctionManager.
* Type check codenode.py and fix an RDA test case.
* Fix serialization tests; Introduce KnowledgeBasePlugin.set_kb(); SpillingFunctionDict now derives from UserDict; Fix KB.name stored in KB._plugins; Fix FunctionDict.__setstate__ swapping Function objects and function addresses.
* Update FactCollector to support FuncNode.
* FunctionInfo: Update Function.dirty and perform type checks on keys and values.
* Make function cache limit configurable.
* Adjust FunctionInfo type check.
* More fixes.
* More updates to account for FuncNode in function graphs.
* Update a test case.
* Fix another test case (do not use the size of FuncNodes).
* Fix Reassembler.
* CFGBase.make_functions: Add a missing insertion to _updated_nonreturning_functions.
* Update FunctionManager.rebuild_callgraph.
* FunctionParser: Call destinations must be FuncNodes.
* Serialize Function.is_default_name.
* Minor fixes.
* Fix CFunctionCall._is_target_ambiguous.
* Mark evicted Function instances as evicted.
* HashLookupAPIDeobfuscator: Take a list of function addresses instead of Function instances as arg.
* CC_NAMES: Fix the bug of missing SimCCCdecl.
* FunctionParser: Fix missing syscall function nodes.
* HookNode: Take a SimProcedure instance instead of the class as the sim_procedure argument.
* FunctionParser: Consider return-type edges when deserializing.
* SimTypeCppFunction: Fix to_json() serialization crash.
* FunctionParser: Fix missing return sites.
* Function.is_{syscall,simprocedure,alignment,plt} settings should mark the function dirty.
* RDA: Do not create blocks for FuncNodes or HookNodes.
* Update a test case.
* Fix issues with SimCppClass.to_json.
* HookNode: Fix HookNode.__eq__.
* Fix SootFunction.
* Lint function_manager.py.
* Lint and type check.
* More docs; Spill Function.ran_cca.
* RuntimeDb: Support specifying base dir using an environment variable.
* Lint and fix test cases.
* chore: Make SIM_LIBRARIES a dict of list of lib instead of a dict of lib
* lint
* Revert "Temporarily disabling a line in test_decompiling_4846..."
This reverts commit eafdbf857a.
* remove stray line
* support looking up sub_XXX names even when the function has been renamed.
* add ability to lookup functions by previous names after a renaming.
---------
Co-authored-by: Yan Shoshitaishvili <zardus@emotionlabs.io>
* fix typos
This mostly fixes comments, but in a handful of places fixes bugs due to typos.
One example:
```
- insn_op_idx=None if cmsg.operand_idx == -1 else cmsg.opearnd_idx,
+ insn_op_idx=None if cmsg.operand_idx == -1 else cmsg.operand_idx,
```
* more typo fixes
* address lint issues
* more import fixes?
* address feedback from PR
---------
Co-authored-by: Brian Caswell <bcaswell@microsoft.com>
* SimType & friends: Fully typecheck, fix observed bugs
* fix small bugs
* lint
* adjust the semantics of SimType.size - it can return None now
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* whack
* whack
* whack
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* rename functions_callable to functions_reachable
rename functions_callable to functions_reachable
* fix operator for Constant.__sub__
* print hex address of Function
* support multiple blocks with the same address in StackCanarySimplifier. This happens when doing inlined decompilation
* support specifying initial register values in the StackPointerTracker for inlining functions onto stacks of other functions
* add a StackPointerTracker method to retrieve all offsets of a register
* move stack pointer analysis later, in preparation of pre-inlining refactor
* split decompilation process in preparation for inlining logic
* inlined decompilation!
* testcase for inlined decompiler
* support spilled (e.g., callee-saved) register simplification in the inlined callees
* fix variable reference issue due to duplicated blocks (triggered via inlining)
* Fix more type annotations in clinic.py.
Co-authored-by: Fish <fishw@asu.edu>
* Add KB as parameter to KnowledgeBasePlugin for consistency
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Rattle the chains that bind us
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Florian Magin <fmagin@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Make CC and DEFAULT_CC platform-aware. Also enhance the input variable
discovery.
* Lint code.
* compatibility fixes
* SimProcedure.project can be None.
* Add a CC for CGC. Other fixes.
* Be more conservative when analyzing call sites.
* Initial pass, loosely following angr package layout
* Many updates to use the 'common' module for binaries repo
* Some unused imports and log handlers are cleaned up
* Attempt to provide an interface for writing RDA function handlers easily
* Aggressively touch up call and const atoms
* Totally rewrite function handlers
* Imports
* so many renames and fixes
* weh
* better errors
* Attempt to make decompiler work
* better behavior
* Fix a bunch of shit. thanks mypy
* Add DepGraph.find_path
* Use callsite insaddr for codeloc of function outputs
* Add LiveDefinitions: get_stack_values
* Don't require handle_impl_<func>
* wip: add logic for matching RDA definitions against SimVariables
* _narrow_exprs: Do not update call.ret_expr if ret_expr is None.
* Propagator: Fix an incorrect access to None in the AIL engine.
* RDA: Fix an incorrect CodeLocation definition in function handler.
* Clarify docs for CodeLocation
* fix handle_impl_ and clarify warning message
* Support pseudo function calls.
* Separate function effects applied at callsites and the ones applied inside callees.
* AILSimplifier._narrow_exprs: Filter away all definitions in callees.
* DepGraph no longer takes a reference to Project.
* Fix function argument inference for tail jumps.
* FunctionHandler: Use args. Support args_values.
* Tidy interfaces
* AILSimplifier._unify_local_variables: Ignore variables defined in callees.
* Fix CCA by enabling the function handler.
* Properly handle CallExprs.
* Fix call expression handling. Fix the test case.
* f
* DefinitionMatchPredicate: Fix default initializers
* Overhaul codelocs
* Fix context handling
* Fix values iter
* Sort some shit
* Add logic for handling incomplete prototypes in FunctionHandler
* FunctionHandler: apply effects from dependencies first.
* Properly handle Call.ret_expr in AIL. Fix a in VRA.
* Propagator: Be less strict about contatenation when the high bytes are concrete.
* oops
* Move uses to first pass of function handler effects
* Aggressive type hinting
* FunctionHandler: Try harder to resolve function address
* Type annotation
* Typing
* Adjust codegen to smooth over casts
* Tweak the function argument uses for the nth time
* ail simplifier: check that expr corresponding to use is not None
* Typing
* Objectively a correct bugfix
* Correctly generate args_atoms based on args_values
* in case of fire break glass
* Function handler: Add args uses even if there is no return value
* Add data.ret_values even if there is no ret atom (e.g. ccalls)
* Add the missing with_arch() call.
* function handler: hook prototypes don't have an arch
* Revert "Add the missing with_arch() call."
This reverts commit 319eb8f826f634c89106221e33359a2eb1009a48.
* Implement better is_testing detection.
* Uncomment an important fire extinguisher.
* RDA: AIL engine overwrites the whole register when the return value only occupies the partial register.
* FunctionHandler: Properly handle clobbered partial registers.
* Fix a type hint in atoms.Register.
* Do not add duplicated uses for values that are in args_values.
* copy guessed_prototype from hook to function data
* AILSimplifier: Ignore unused conflicting defs when unifying variables.
* c_args_as_atoms: Handle variadic functions.
* RDA: AILEngine: Do not redefine locals if call args are available.
* VRA: EngineAIL: Follow RDA's logic when handling call.ret_expr.
* Respect block_idx a few places; add caller-handled ret_defns to callsite info
* CallSiteMaker: Mark argument definitions in call stmts so RDA knows what to erase.
* RDA: AILEngine.handle_register: Use the optimal define location if possible.
* oops
* AILSimplifier._narrow_exprs: Fix the callee function check.
* Remove unused imports.
* docs
* Adapt StaticObjectFinder to the new function handler model.
* Make vex rda tests pass
* function handler: Apply bootstrap return value definition at callsite
* lint
* lint 2
---------
Co-authored-by: Audrey Dutcher <audrey@rhelmot.io>
Co-authored-by: Matt Borgerson <contact@mborgerson.com>
* SimMemoryObject: Use slots.
* Add LabeledMemory.
* CodeLocation: stmt_idx can be None.
* Propagator: Switch to LabeledMemory and claripy ASTs - Strike One.
* Implement SimLabeledMemoryObject.
* Propagator: Switch to LabeledMemory and claripy ASTs - Strike Two.
* Bug fixes. Migrate RDA live_definitions to LabeledMemory.
* Call the test case.
* Add MVListPage.
* some effort
* Update the test case.
* More progress
* More work done.
* Implement page merging.
* Implement add and sub for RDEngineVEX.
* More test cases passed
* More progress. Update test cases.
* RDA: `SimEngineRDVEX._handle_function_core` to use `MultiValues`
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
* Fix _process_block_end().
* Update _handle_CMPNE
* Update _handle_CMPLT
* Update _handle_CMPORD
* Update _handle_Sar
* Add none safety to some _handle_* functions
* Fix _handle_Not update
* Update _handle_Not1
* Update _handle_CCall
* Add exception checking during shifting
* Convert SimEngineRDAIL.
* bug fixes
* Fix the endianness bug.
* Several bug fixes.
* Do not resimplify a block if it is not changed.
* RDA: Do not reload register values.
* Fix DefinitionAnnotation.
* Fix MultiValues.__len__().
* Fix a test case.
* Propagator: Properly deal with stack addresses in Load.
* Migrate most of variable recovery analysis.
* Force AIL Store to use its own size.
* Fix incorrect propagation when bytes are extracted from an MO. Get rid of .processor_state in VRA.
* Get rid of ProcessorState.
* Minor bug fixes in _reference().
* bug fixes
* Propagator: Return 1-bit tops for CMPs.
* Cache TOPs.
* Fix variable indent for global variables.
* SimMemoryObject: Optimize __eq__().
* VRF: Fix four handlers in the AIL engine.
* Remove a breakpoint.
* MVListPage: Do not calculate changed bytes twice during merging.
* MVListPage: Reduce unnecessary scans of None in content.
* ListPage/MVListPage: Support custom SimMemoryObject comparators.
* MVListPage: Reduce calls to _contains().
* Propagator: Remove TOPs in replacements after analysis.
* Reduce max iterations of RDA and Propagator to 2.
* Bug fix in propagator.
* Fix an incorrect type annotation.
* Unify _post_analysis in Propagator.
* LiveDefinitions: Fix _mo_cmp.
* Revert "Reduce max iterations of RDA and Propagator to 2."
This reverts commit 536ea4d59f80095dc37ec9055ed25aab19ed7d83.
* _replace_and_build(): Returns if the replacement really happened.
* Propagator: Fix stack variable load/store.
* Compute less propagation and RDAs in AILSimplifier.
* RDA: Use cached and less-accurate dummy definitions for kill_definitions().
* Fix LiveDefinitions._mo_cmp.
* Propagator: Fix a minor issue in the Add handler in AIL engine.
* Propagator: Attempt to propagate everything coming out of tmps.
* Migrate variable recovery.
* RDA: Get rid of references to DataSet. Fix support for floating point values.
* LightEngineVEX: _handle_Const supports floating points.
* CallSiteMaker: Handle SimMemoryMissingError.
* extract_offset_to_sp: Handle __add__ with only one argument.
* Propagator: Do not store if the data is None.
* Implement get_all_definitions(). Fix bugs in SimEngineRDVEX.
* Fix more broken test cases.
* VariableRecovery: Fix a StopIteration bug.
* Fix more bugs.
* Lint the code.
* Fix test cases. Drop DataSet.
Co-authored-by: Pamplemousse <xav.maso@gmail.com>
Co-authored-by: Clasm <wfgibbs@asu.edu>
* RDA: Don't duplicate `get_sp` implementation
... accross `ReachingDefinitionsState` and `LiveDefinitions`
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
* RDA: `LiveDefinitions.get_sp()` not to fail
...when there are several SP definitions.
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
* RDA: Utilitary to get the size of several `DataSet`s
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
* RDA: More operations for `UnknownSize`
As the size of `DataSet`s represents a number of bits, the following
operation often happens in the code: `dataset._bits // 8` .
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
* RDA: Minor missing documention addition
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
* RDA: Favor `unittest` assertions to `nose`
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
* RDA: Basic heap memory modelisation
* have a `HeapAddress` dedicated type
* uber-basic simulation of an allocator behavior (addresses
generation)
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
* RDA: Add heap modelisation to the state
* content as part of the `LiveDefinitions`
* memory management using `HeapAllocator` and `HeapAddress`es
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
* RDA: Allow sum between `int` and `HeapAddress`
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
* RDA: Make `HeapAddress` hashable
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
* Fix functions_called
Infinite recursion bug in functions_called
* Add test for functions_called
Check cyclic dependencies of functions_called
* `Function.functions_called` split tests
* make the two cases explicitely separated
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
Co-authored-by: Pamplemousse <xav.maso@gmail.com>
Fish