basicmachines-co-basic-memory/docs/semantic-search-test-log.md

Semantic Search Manual Test Log

Overview

Manual test session for semantic (vector) search on the main project.

• Date: 2026-02-15
• Database: ~/.basic-memory/memory.db (SQLite)
• Entities: 456 embedded, 2714 vector chunks
• Search index: 2390 FTS entries
• Embedding model: default (384-dim, sqlite-vec)

Test Plan

1. Search Type Routing — verify vector/hybrid/text dispatch, invalid search_type handling
2. Conceptual Queries — natural language where vector should beat FTS
3. Keyword Queries — exact terms where FTS should be strong
4. Hybrid Ranking — queries where both FTS and vector contribute
5. Result Types — entities, observations, relations in vector results
6. Filters + Vector — combine vector with types/entity_types/after_date
7. Edge Cases — short queries, long queries, empty, special chars, no-match
8. Pagination — page > 1, page_size respected

---

Test Results

Test 1: Search Type Routing

1a: search_type="semantic" (invalid value)

• Input: query="how does the knowledge graph work", search_type="semantic"
• Expected: error or explicit fallback
• Actual: Silently falls through to text search (else branch in search.py:430)
• Verdict: BUG — should either be a recognized alias for "vector" or return an error

1b: search_type="vector"

• Input: query="keeping AI context between sessions", search_type="vector"
• Actual: 5 results, scores ~0.58-0.59, found "Maintaining context across conversation boundaries" observation
• Verdict: PASS

1c: search_type="text" with conceptual query

• Input: query="keeping AI context between sessions", search_type="text"
• Actual: 0 results (no exact keyword match)
• Verdict: PASS (expected — FTS requires token overlap)

1d: search_type="hybrid" with conceptual query

• Input: query="keeping AI context between sessions", search_type="hybrid"
• Actual: 5 results, same ranking as vector (FTS contributed nothing here)
• Verdict: PASS

1e: search_type="text" with keyword query

• Input: query="OAuth authentication", search_type="text"
• Actual: 3 results — AUTH.md Supabase OAuth, OAuth Rip-and-Replace, OAuth Integration Analysis
• Verdict: PASS

1f: search_type="vector" with keyword query

• Input: query="OAuth authentication", search_type="vector"
• Actual: Same top results as text (keyword-rich content also scores well in vector space)
• Verdict: PASS

---

Test 2: Conceptual Queries (vector advantage)

2a: Natural language question

• Input: query="why do AI assistants forget things", search_type="vector"
• Actual: 5 results — Manual Testing Session, "Balance security and usability" observation, "Tools should match thought patterns" observation. Scores ~0.56-0.57
• Vector advantage: Found conceptually related content despite no exact keyword overlap
• Verdict: PASS

2b: Same query, text search

• Input: query="why do AI assistants forget things", search_type="text"
• Actual: 1 result — "What is Basic Memory?" (likely matched on "AI" token)
• Verdict: PASS (demonstrates vector advantage — text barely matched)

2c: Domain concept with no jargon

• Input: query="pricing strategy for cloud product", search_type="vector"
• Actual: 3 results — SPEC-16 MCP Cloud Service Consolidation, knowledge architecture observation, Visual Knowledge Spaces relation. Scores ~0.56-0.57
• Verdict: PASS (found cloud-related content conceptually)

2d: Technical concept, long query

• Input: query="SQLite performance optimization WAL mode concurrent writes", search_type="vector"
• Actual: 3 results — SPEC-11 API Performance Optimization, Real-Time Updates with WebSockets, marketing status update. Scores ~0.55-0.58
• Verdict: PASS (found performance-related content)

---

Test 3: Keyword Queries (FTS strength)

3a: Exact term match — "OAuth authentication"

• Text: 3 results with high relevance (exact matches in titles)
• Vector: Same top results (keyword overlap helps vector too)
• Verdict: PASS — FTS and vector converge on keyword-rich queries

3b: "OAuth" single keyword, hybrid mode

• Input: query="OAuth", search_type="hybrid"
• Actual: 5 results — Basic Memory Coding Guide, AI Collaboration Examples, SPEC-18, daily note, Manual Testing Session. FTS + vector blended. Scores ~0.016-0.032
• Note: Top hybrid result is "Basic Memory Coding Guide" not an OAuth-specific doc — suggests hybrid scoring may dilute strong FTS matches
• Verdict: PASS but hybrid ranking questionable for single-keyword queries

---

Test 4: Hybrid Ranking

4a: Hybrid vs vector on "OAuth authentication"

• Hybrid with entity_types=["entity"]: 5 results — RLS Implementation Lessons, Cloud Readiness Assessment, AUTH.md OAuth, Core Service Implementation, OAuth Rip-and-Replace. Scores ~0.016-0.023
• Vector with entity_types=["entity"]: 5 results — Core Service Implementation, SPEC-13 CLI Auth, Coding Guide, Authentication Service, ADR Production Auth. Scores ~0.55-0.60
• Observation: Hybrid surfaces different top results than vector-only. Hybrid found RLS and Cloud Readiness docs that vector didn't prioritize. Different ranking is expected from RRF fusion.
• Verdict: PASS — hybrid produces meaningfully different ranking

---

Test 5: Result Types

5a: Vector returns all result types

• Input: query="keeping AI context between sessions", search_type="vector"
• Entities: SPEC-18 AI Memory Management Tool (type=entity)
• Relations: Prompt Builder integrates_with (type=relation)
• Observations: "Translation layer is key" (type=observation), "Maintaining context across conversation boundaries" (type=observation)
• Verdict: PASS — all three types appear in vector results

5b: Observations carry metadata

• Observation result: category="challenge", content="Maintaining context across conversation boundaries", from_entity="research/ai-knowledge-management-research"
• Verdict: PASS — category, content, from_entity, tags all present

5c: Relations carry link info

• Relation result: relation_type="integrates_with", from_entity="development/features/prompt-builder...", to_entity (present but truncated in some)
• Verdict: PASS — relation metadata present

---

Test 6: Filters + Vector Search

6a: entity_types=["entity"] with vector

• Input: query="OAuth authentication", search_type="vector", entity_types=["entity"]
• Actual: 5 results, all type="entity" (Core Service Implementation, SPEC-13, Coding Guide, Authentication Service, ADR Auth)
• Verdict: PASS — filter correctly restricts to entities only

6b: types=["note"] with vector

• Input: query="OAuth authentication", search_type="vector", types=["note"]
• Actual: Same 5 results (all have entity_type="note" in metadata)
• Verdict: PASS — types filter works with vector search

6c: after_date with vector

• Input: query="OAuth authentication", search_type="vector", after_date="2025-06-01"
• Actual: 3 results — Core Service Implementation, Cloud Web App analysis observation, SPEC-13. Filtered out older OAuth docs.
• Verdict: PASS — date filter applied correctly

6d: entity_types=["entity"] with hybrid

• Input: query="OAuth authentication", search_type="hybrid", entity_types=["entity"]
• Actual: 5 results, all type="entity" — RLS lessons, Cloud Readiness, AUTH.md OAuth, Core Service, OAuth Rip-and-Replace
• Verdict: PASS — filter works with hybrid mode too

6e: types=["entity"] with vector (WRONG filter name)

• Input: query="OAuth authentication", search_type="vector", types=["entity"]
• Actual: 0 results
• Note: types filters by entity_type metadata (e.g., "note", "person"), NOT by SearchItemType. Using types=["entity"] looks for entity_type="entity" which few/no notes have. This is a UX confusion point — the param names are ambiguous.
• Verdict: PASS (correct behavior) but USABILITY ISSUE — easy to confuse types vs entity_types

---

Test 7: Edge Cases

7a: Single character query

• Input: query="x", search_type="vector"
• Actual: 3 results — "Self-contained application bundle" observation, Non-Markdown File Support relation, quick-win-tools entity. Scores ~0.57-0.59
• Note: Single character still produces an embedding and returns results. Quality is low/random as expected.
• Verdict: PASS (no crash, returns results)

7b: Whitespace-only query

• Input: query=" ", search_type="vector"
• Actual: 0 results
• Verdict: PASS (handled gracefully — _check_vector_eligible strips and rejects empty)

7c: Query with no relevant content

• Input: query="quantum computing blockchain", search_type="vector"
• Actual: 3 results — Inter-Agent Communication relation, Self-contained bundle observation, JSON-LD interop observation. Scores ~0.54
• Note: Still returns results because vector search always finds nearest neighbors. Scores are lower (~0.54) than relevant queries (~0.58-0.60). No relevance threshold applied.
• Verdict: PASS (expected behavior) but NOTE — no relevance cutoff means irrelevant queries always return something

---

Test 8: Pagination

8a: Vector search page 2

• Input: query="keeping AI context between sessions", search_type="vector", page=2, page_size=3
• Actual: 3 results on page 2, current_page=2. Different results from page 1. Top: "Maintaining context across conversation boundaries" observation (score 0.587)
• Note: Interestingly, page 2 had a higher-scoring result than some page 1 results. This may indicate pagination doesn't sort globally — it might be paginating within a pre-scored set.
• Verdict: PASS (pagination works) but POSSIBLE ISSUE — result ordering across pages needs investigation

---

Summary

Passing Tests: 20/21

Bugs Found

1. search_type="semantic" silently falls through (Test 1a) — Invalid search_type values fall to the else branch and default to text search without any warning. Should either alias "semantic" to "vector" or raise an error.

Usability Issues

2. types vs entity_types confusion (Test 6e) — types filters by entity_type metadata (note, person, etc.) while entity_types filters by SearchItemType (entity, observation, relation). The naming is ambiguous and easy to mix up.
3. No relevance threshold (Test 7c) — Vector search always returns nearest neighbors even for completely irrelevant queries. Consider adding a minimum score threshold or at least documenting expected score ranges.
4. Hybrid ranking for single keywords (Test 3b) — Hybrid mode on simple keyword queries produced less intuitive rankings than pure FTS or pure vector. The RRF fusion may dilute strong FTS signals.

Observations

• Vector search successfully finds conceptually related content that FTS misses entirely
• Score ranges: relevant queries ~0.56-0.60, irrelevant queries ~0.54 (narrow spread)
• All three result types (entity, observation, relation) appear correctly in vector results
• Filters (entity_types, types, after_date) all work correctly with vector and hybrid modes
• Pagination works but cross-page ordering may need investigation