mirror of
https://github.com/basicmachines-co/basic-memory
synced 2026-06-21 13:47:35 +00:00
Drew Cain
d175879940
Benchmarked the #951 entity-aware ranking boost against the LoCoMo retrieval suite (hybrid mode) and a hand-built adversarial corpus. LoCoMo is insensitive to the boost: sweeping the weight across 0.15/0.3/0.5/1.0/2.0 produced identical recall@5, recall@10, MRR, and content-hit at every point (no query reordered, no score changed). LoCoMo docs are keyed by session id and expose speaker names only in body text, never as entity titles or relation names, so the title/relation-matching boost never fires there. An adversarial check found a real regression mode: Title-Case queries inject spurious entity terms. 'What Is The Plan For Q3' extracts 'Q3' and, even at weight 0.15, promotes a literal-'Q3' document over the more relevant 'third quarter' document. Clean proper nouns (Katze) work; lowercase-leading identifiers (getUserById) are correctly ignored. Decision: keep search_entity_boost_enabled default off and the weight at 0.15. LoCoMo provides no signal to raise the weight, and the adversarial check is not clean. Document the findings and guidance; no code/default changes. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Drew Cain <groksrc@gmail.com>