ML: The current local model causes false positives with small functions, and functions with a huge difference in basic blocks. Diaphora will ignore such matches.
CORE: Increase the added similarity score in `deep_ratio` when constants (like strings or cryptographic constants) match.
HEUR: Remove the unreliable flag from heuristics "Pseudo-code fuzzy AST hash" and "Loop Count".
VULN: Do not use difflib.unified_diff as it's terribly slow; instead use difflib.ndiff.
CORE: Try to use `cdifflib` instead of Python's standard `difflib` when possible to get some performance gains.
BUG: High addresses in operands could cause the Python's sqlite3 module to crash when inserting into the database.
ML: Try to use the Ridge classifier as just another method to get a similarity ratio in `check_ratio`.
ML: Simplifications of the supervised learning based experimental engine.
CONFIG: Added parameter `COMMIT_AFTER_EACH_GUI_UPDATE` to force committing.
CONFIG: Added parameter `EXPORTING_COMPILATION_UNITS` to enable/disable exporting them (with some huge databases it might take even hours!).
CONFIG: Added parameters handling SQLite pragmas `SQLITE_JOURNAL_MODE` and `SQLITE_PRAGMA_SYNCHRONOUS`.
CONFIG: Added parameter `SHOW_IMPORT_WARNINGS` to enable/disable showing warnings when some important but optional Python packages aren't found.
BUG: Be sure to delete orphaned comments when importing pseudo-code comments.
BUG: The workaround for "max non-trivial tinfo_t count has been reached" was wrong. Now, the Hex-Rays functions cache is cleared every 10,000 rows.
GUI: Display the progress when exporting a large number of compilation units.
BUG: Inserting the link between functions and compilation units was terribly-utterly-horribly wrong.
VULN: Add pattern "UNC" to potentially detect vulnerabilities fixed in Windows components involving UNC paths.
EXTRAS: Added independent IDA plugin `extras/diaphora_local.py` to be able to diff functions inside the current binary.
BUG: Do a commit after all functions are exported so, in case IDA crashes for a reason/bug, Diaphora can properly recover from errors and have all the functions already exported there.
CORE: Train a Ridge classifier using known good and bad results after `find_partial_matches()` in order to try to better determine what looks like a good match and what does not.
MISC: Increase version number to 3.2.0.
The ML approach that is being tested in this version works as follows:
* Diaphora uses the usual approaches to find matches labelled as "Best" or "Partial".
* After `find_partial_matches()` is executed, it then generates a dataset to train the classifier.
* The dataset is generated by iterating through all the best and partial results previously found.
* Good results are labelled as "0", bad results are labelled as "0".
* The dataset generated contains a comparison of almost each field extracted by Diaphora.
* For integer columns, it calculates a ratio by doing `1 - ( abs(num1 - num2) / max(num1, num2) )`.
* For text columns, it uses `SequenceMatcher().quick_ratio()`.
* For lists of items, it tries to calculate a similarity ratio like this:
```
inter = len(s1.intersection(s2))
maxs = len(max(s1, s2))
val = 1. - (inter / maxs)
```
* After the dataset is generated and the model fitted, the classifier is ready to be used.
* Then, for any future function match `deep_ratio()` will add a value to the final similarity ratio if the classifier thinks it found a good match.
* This added value is specified in `diaphora_config.py`, the directive is `ML_DEEP_RATIO_ADDED_SCORE`.
Also, I have done various little refactorizations here and there mostly to try to use the same variable/field names everywhere, instead of different names on different parts of the code.
BUG: When checking if functions are 100% equals check more fields than just the id, address, mangled_function and nodes.
VULN: Mark as interesting differences with the functions `ProbeForRead` and `ProbeForWrite`.
EXPORT: Added a column to save how long it took to export a single function.
EXPORT: Use `cur.executemany()` instead of `cur.execute()` whenever it's possible.
DIFF: Added a ratios cache to speed up comparison operations.
HEUR: Added heuristic "Same constants related matches" to find functions using the same constants in different places.
HEUR: Added heuristic "Related compilation unit" to find functions by matching potential compilation units.
HEUR: Added absolutely experimental (and subject to be removed) heuristic "Similar export time".
BUG: The microcode instructions list was built a lot of times instead of being done only once.
GUI: Added menu item "Show assembly patch".
BUG: Diaphora was calling ida_lines.get_srcline() for every assembly line. Fixed by doing it once per basic block.
BUG: The code for calculating the primes assigned to a compilation unit was terribly slow.
MISC: Replace multiple "SELECT *" appearances with just the required fields, where appropriate.
VULN: Added a few new patterns to try to find potentially fixed vulnerabilities.
VULN: Added heuristic to try to find fixed signedness issues for x86 and ARM.
MISC: Refactored the code for finding potentially fixed vulnerabilities.
GUI: BUG: The main UI dialog might not be 100% visible with some screen resolutions.
GUI: BUG: Adding manual matches was partially wrong.
GUI: BUG: After closing the "Interesting matches" tab there was no way to reshow it again (like F3 does).
CORE: By default, run a helper script that tries to find potentially fixed vulnerabilities when a patch diffing session with function names is detected.
Joxean Koret