admin/basicmachines-co-basic-memory
1
0
Fork 0
mirror of https://github.com/basicmachines-co/basic-memory synced 2026-06-21 13:47:35 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: merge search_by_metadata into search_notes with optional query

Browse Source 
Make `query` optional in `search_notes` so it becomes the single search tool.
Remove `search_by_metadata` entirely — it was unreleased and redundant since
`search_notes` already supports `metadata_filters`, `tags`, and `status`.

- 🔧 `query` param is now `Optional[str] = None`
- 🛡️ Added None guards for project detection and URL resolution
- ✅ Added `no_criteria()` validation with helpful error message
- 🗑️ Deleted `search_by_metadata` tool, imports, tests, and contract entry
- 📝 Updated docs, README, and v0.19.0 release notes

Closes #605

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
This commit is contained in:
phernandez
2026-02-25 13:30:42 -06:00
parent b09eca1698
commit 73413486bc
9 changed files with 128 additions and 276 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+1 -2
View File
@@ -438,8 +438,7 @@ list_directory(dir_name, depth) - Browse directory contents with filtering
**Search & Discovery:**
```
search(query, page, page_size) - Search across your knowledge base
search_notes(query, page, page_size, search_type, types, entity_types, after_date, metadata_filters, tags, status, project) - Search with filters
search_by_metadata(filters, limit, offset, project) - Structured frontmatter search
search_notes(query, page, page_size, search_type, types, entity_types, after_date, metadata_filters, tags, status, project) - Search with filters (query is optional for filter-only searches)
```
**Project Management:**
docs/ai-assistant-guide-extended.md
+7 -14
View File
@@ -1060,9 +1060,9 @@ results = await search_notes(
project="main"
)
# Metadata-only search
results = await search_by_metadata(
filters={"type": "spec", "status": "in-progress"},
# Metadata-only search (no query needed)
results = await search_notes(
metadata_filters={"type": "spec", "status": "in-progress"},
project="main"
)
```
@@ -2915,18 +2915,11 @@ results = await search_notes(
)
```
**search_by_metadata(filters, limit, offset, project)**
- Metadata-only search using structured frontmatter
- Parameters:
- `filters` (required): Dict of field -> value (supports $in, $gt/$gte/$lt/$lte, $between)
- `limit` (optional): Max results (default: 20)
- `offset` (optional): Pagination offset (default: 0)
- `project` (required unless default_project_mode): Target project
- Returns: Matching entities
- Example:
**Metadata-only search (via search_notes)**
- Use `search_notes` with `metadata_filters` and no `query` for metadata-only searches:
```python
results = await search_by_metadata(
filters={"type": "spec", "status": "in-progress"},
results = await search_notes(
metadata_filters={"type": "spec", "status": "in-progress"},
project="main"
)
```
docs/metadata-search.md
+13 -47
View File
@@ -2,14 +2,9 @@
Basic Memory automatically indexes custom frontmatter fields so you can query them with structured filters. Any YAML key in a note's frontmatter beyond the standard set (`title`, `type`, `tags`, `permalink`, `schema`) is stored as `entity_metadata` and becomes searchable.
## Two Ways to Query
## Querying with `search_notes`
| Tool | Use When |
|------|----------|
| `search_by_metadata` | You only need metadata filters (no text query) |
| `search_notes` | You want to combine a text query with metadata filters |
Both tools accept the same filter syntax.
`search_notes` is the single search tool for all queries — text, metadata filters, or both. The `query` parameter is optional, so you can use metadata filters alone without passing an empty string.
## Filter Syntax
@@ -88,45 +83,16 @@ This queries the `version` key inside a `schema` object in frontmatter.
- `$in` and array-contains require non-empty lists.
- `$between` requires exactly two values `[min, max]`.
## MCP Tools
## MCP Tool — `search_notes`
### `search_by_metadata` — metadata-only search
Searches entities by structured frontmatter metadata without a text query. Results are scoped to entity-level items.
**Parameters:**
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `filters` | dict | Yes | Metadata filter dictionary (see syntax above) |
| `project` | string | No | Project to search in (uses default if omitted) |
| `limit` | int | No | Max results (default 20) |
| `offset` | int | No | Skip N results for pagination (default 0) |
**Example:**
```python
# Find all notes with status "in-progress"
await search_by_metadata({"status": "in-progress"})
# Find high-priority specs in the research project
await search_by_metadata(
{"type": "spec", "priority": {"$in": ["high", "critical"]}},
project="research",
limit=10,
)
```
### `search_notes` with metadata — combined text + metadata
The `search_notes` tool accepts `metadata_filters`, `tags`, and `status` parameters alongside the text `query`. This lets you combine full-text search with structured filtering.
`search_notes` is the single search tool for text queries, metadata filters, or both. The `query` parameter is optional.
**Relevant parameters:**
| Parameter | Type | Description |
|-----------|------|-------------|
| `query` | string | Text search query (can be empty when using only filters) |
| `metadata_filters` | dict | Structured filter dict (same syntax as `search_by_metadata`) |
| `query` | string (optional) | Text search query. Omit for filter-only searches. |
| `metadata_filters` | dict | Structured filter dict (see syntax above) |
| `tags` | list[str] | Convenience shorthand — merged into `metadata_filters["tags"]` |
| `status` | string | Convenience shorthand — merged into `metadata_filters["status"]` |
@@ -138,8 +104,8 @@ The `search_notes` tool accepts `metadata_filters`, `tags`, and `status` paramet
# Text search filtered by metadata
await search_notes("authentication", metadata_filters={"status": "draft"})
# Filter-only search (empty query)
await search_notes("", metadata_filters={"type": "spec"})
# Filter-only search (no query needed)
await search_notes(metadata_filters={"type": "spec"})
# Combine text, tags shortcut, and metadata
await search_notes(
@@ -150,7 +116,7 @@ await search_notes(
# Convenience shortcuts
await search_notes("planning", status="active")
await search_notes("", tags=["tier1", "alpha"])
await search_notes(tags=["tier1", "alpha"])
```
## Tag Search Shortcuts
@@ -260,19 +226,19 @@ confidence: 0.6
```python
# Find all in-progress specs
await search_by_metadata({"status": "in-progress", "type": "spec"})
await search_notes(metadata_filters={"status": "in-progress", "type": "spec"})
# → Auth Design
# Find high-confidence specs
await search_by_metadata({"confidence": {"$gt": 0.7}})
await search_notes(metadata_filters={"confidence": {"$gt": 0.7}})
# → Auth Design (confidence: 0.85)
# Find specs with priority high or medium
await search_by_metadata({"priority": {"$in": ["high", "medium"]}})
await search_notes(metadata_filters={"priority": {"$in": ["high", "medium"]}})
# → Auth Design, Search Redesign
# Find specs in a confidence range
await search_by_metadata({"confidence": {"$between": [0.5, 0.9]}})
await search_notes(metadata_filters={"confidence": {"$between": [0.5, 0.9]}})
# → Auth Design (0.85), Search Redesign (0.6)
# Find notes tagged with security
docs/releases/v0.19.0.md
+8 -5
View File
@@ -94,13 +94,16 @@ All MCP tools now support `output_format="json"` for machine-readable responses.
### Structured Metadata Search
New `search_by_metadata` tool for searching by frontmatter fields.
`search_notes` now supports filter-only searches — `query` is optional, so you can search
purely by frontmatter metadata without passing an empty string.
```
search_by_metadata({"status": "in-progress"})
search_by_metadata({"tags": ["security", "oauth"]})
search_by_metadata({"priority": {"$in": ["high", "critical"]}})
search_by_metadata({"schema.confidence": {"$gt": 0.7}})
search_notes(metadata_filters={"status": "in-progress"})
search_notes(metadata_filters={"tags": ["security", "oauth"]})
search_notes(metadata_filters={"priority": {"$in": ["high", "critical"]}})
search_notes(metadata_filters={"schema.confidence": {"$gt": 0.7}})
search_notes(tags=["security"]) # convenience shorthand
search_notes(status="draft") # convenience shorthand
```
### `tag:` Search Shorthand
src/basic_memory/cli/commands/tool.py
+1 -1
View File
@@ -485,7 +485,7 @@ def search_notes(
with force_routing(local=local, cloud=cloud):
result = run_with_cleanup(
mcp_search(
query=query or "",
query=query or None,
project=project,
workspace=workspace,
search_type=search_type,
src/basic_memory/mcp/tools/__init__.py
+1 -2
View File
@@ -18,7 +18,7 @@ from basic_memory.mcp.tools.view_note import view_note
from basic_memory.mcp.tools.write_note import write_note
from basic_memory.mcp.tools.cloud_info import cloud_info
from basic_memory.mcp.tools.release_notes import release_notes
from basic_memory.mcp.tools.search import search_notes, search_by_metadata
from basic_memory.mcp.tools.search import search_notes
from basic_memory.mcp.tools.canvas import canvas
from basic_memory.mcp.tools.list_directory import list_directory
from basic_memory.mcp.tools.edit_note import edit_note
@@ -58,7 +58,6 @@ __all__ = [
"schema_infer",
"schema_validate",
"search",
"search_by_metadata",
"search_notes",
# "search_notes_ui",
"view_note",
src/basic_memory/mcp/tools/search.py
+54 -113
View File
@@ -257,7 +257,7 @@ Error searching for '{query}': {error_message}
annotations={"readOnlyHint": True, "openWorldHint": False},
)
async def search_notes(
query: str,
query: Optional[str] = None,
project: Optional[str] = None,
workspace: Optional[str] = None,
page: int = 1,
@@ -333,8 +333,10 @@ async def search_notes(
- Nested keys use dot notation (e.g., `"schema.confidence"`).
### Filter-only Searches
You can pass an empty query string when only using structured filters:
- `search_notes("my-project", "", metadata_filters={"type": "spec"})`
Omit `query` (or pass None) when only using structured filters:
- `search_notes(metadata_filters={"type": "spec"}, project="my-project")`
- `search_notes(tags=["security"], project="my-project")`
- `search_notes(status="draft", project="my-project")`
### Convenience Filters
`tags` and `status` are shorthand for metadata_filters. If the same key exists in
@@ -347,7 +349,8 @@ async def search_notes(
- `search_notes("archive", "docs/2024-*", search_type="permalink")` - Year-based permalink search
Args:
query: The search query string (supports boolean operators, phrases, patterns)
query: Optional search query string (supports boolean operators, phrases, patterns).
Omit or pass None for filter-only searches using metadata_filters, tags, or status.
project: Project name to search in. Optional - server will resolve using hierarchy.
If unknown, use list_memory_projects() to discover available projects.
page: The page number of results to return (default 1)
@@ -436,47 +439,55 @@ async def search_notes(
entity_types = entity_types or []
# Detect project from memory URL prefix before routing
if project is None:
if project is None and query is not None:
detected = detect_project_from_url_prefix(query, ConfigManager().config)
if detected:
project = detected
async with get_project_client(project, workspace, context) as (client, active_project):
# Handle memory:// URLs by resolving to permalink search
_, resolved_query, is_memory_url = await resolve_project_and_path(
client, query, project, context
)
is_memory_url = False
if query is not None:
_, resolved_query, is_memory_url = await resolve_project_and_path(
client, query, project, context
)
if is_memory_url:
query = resolved_query
effective_search_type = search_type or _default_search_type()
if is_memory_url:
query = resolved_query
effective_search_type = "permalink"
try:
# Create a SearchQuery object based on the parameters
search_query = SearchQuery()
# Map search_type to the appropriate query field and retrieval mode
valid_search_types = {"text", "title", "permalink", "vector", "semantic", "hybrid"}
if effective_search_type == "text":
search_query.text = query
search_query.retrieval_mode = SearchRetrievalMode.FTS
elif effective_search_type in ("vector", "semantic"):
search_query.text = query
search_query.retrieval_mode = SearchRetrievalMode.VECTOR
elif effective_search_type == "hybrid":
search_query.text = query
search_query.retrieval_mode = SearchRetrievalMode.HYBRID
elif effective_search_type == "title":
search_query.title = query
elif effective_search_type == "permalink" and "*" in query:
search_query.permalink_match = query
elif effective_search_type == "permalink":
search_query.permalink = query
else:
raise ValueError(
f"Invalid search_type '{effective_search_type}'. "
f"Valid options: {', '.join(sorted(valid_search_types))}"
)
# Only map search_type to query fields when there is an actual query string.
# When query is None/empty, skip the search mode block — filters-only path.
effective_query = (query or "").strip()
if effective_query:
valid_search_types = {
"text", "title", "permalink", "vector", "semantic", "hybrid",
}
if effective_search_type == "text":
search_query.text = effective_query
search_query.retrieval_mode = SearchRetrievalMode.FTS
elif effective_search_type in ("vector", "semantic"):
search_query.text = effective_query
search_query.retrieval_mode = SearchRetrievalMode.VECTOR
elif effective_search_type == "hybrid":
search_query.text = effective_query
search_query.retrieval_mode = SearchRetrievalMode.HYBRID
elif effective_search_type == "title":
search_query.title = effective_query
elif effective_search_type == "permalink" and "*" in effective_query:
search_query.permalink_match = effective_query
elif effective_search_type == "permalink":
search_query.permalink = effective_query
else:
raise ValueError(
f"Invalid search_type '{effective_search_type}'. "
f"Valid options: {', '.join(sorted(valid_search_types))}"
)
# Add optional filters if provided (empty lists are treated as no filter)
if entity_types:
@@ -494,6 +505,14 @@ async def search_notes(
if min_similarity is not None:
search_query.min_similarity = min_similarity
# Reject searches with no criteria at all
if search_query.no_criteria():
return (
"# No Search Criteria\n\n"
"Please provide at least one of: `query`, `metadata_filters`, "
"`tags`, `status`, `note_types`, `entity_types`, or `after_date`."
)
logger.info(f"Searching for {search_query} in project {active_project.name}")
# Import here to avoid circular import (tools → clients → utils → tools)
from basic_memory.mcp.clients import SearchClient
@@ -520,88 +539,10 @@ async def search_notes(
return result
except Exception as e:
logger.error(f"Search failed for query '{query}': {e}, project: {active_project.name}")
logger.error(
f"Search failed for query '{query or ''}': {e}, project: {active_project.name}"
)
# Return formatted error message as string for better user experience
return _format_search_error_response(
active_project.name, str(e), query, effective_search_type
)
@mcp.tool(
description="Search entities by structured frontmatter metadata.",
annotations={"readOnlyHint": True, "openWorldHint": False},
)
async def search_by_metadata(
filters: Dict[str, Any],
project: Optional[str] = None,
workspace: Optional[str] = None,
limit: int = 20,
offset: int = 0,
context: Context | None = None,
) -> SearchResponse | str:
"""Search entities by structured frontmatter metadata.
Args:
filters: Dictionary of metadata filters (e.g., {"status": "in-progress"})
project: Project name to search in. Optional - server will resolve using hierarchy.
limit: Maximum number of results to return
offset: Number of results to skip (for pagination)
context: Optional FastMCP context for performance caching.
Returns:
SearchResponse with results, or helpful error guidance if search fails
"""
if limit <= 0:
return "# Error\n\n`limit` must be greater than 0."
# Build a structured-only search query
search_query = SearchQuery()
search_query.metadata_filters = filters
search_query.entity_types = [SearchItemType.ENTITY]
# Convert offset/limit to page/page_size (API uses paging)
page_size = limit
page = (offset // limit) + 1
offset_within_page = offset % limit
async with get_project_client(project, workspace, context) as (client, active_project):
logger.info(
f"Structured search in project {active_project.name} filters={filters} limit={limit} offset={offset}"
)
try:
from basic_memory.mcp.clients import SearchClient
search_client = SearchClient(client, active_project.external_id)
result = await search_client.search(
search_query.model_dump(),
page=page,
page_size=page_size,
)
# Apply offset within page, fetch next page if needed
if offset_within_page:
remaining = result.results[offset_within_page:]
if len(remaining) < limit:
next_page = page + 1
extra = await search_client.search(
search_query.model_dump(),
page=next_page,
page_size=page_size,
)
remaining.extend(extra.results[: max(0, limit - len(remaining))])
result = SearchResponse(
results=remaining[:limit],
current_page=page,
page_size=page_size,
)
return result
except Exception as e:
logger.error(
f"Metadata search failed for filters '{filters}': {e}, project: {active_project.name}"
)
return _format_search_error_response(
active_project.name, str(e), str(filters), "metadata"
active_project.name, str(e), query or "", effective_search_type
)
tests/mcp/test_tool_contracts.py
-2
View File
@@ -73,7 +73,6 @@ EXPECTED_TOOL_SIGNATURES: dict[str, list[str]] = {
"schema_infer": ["note_type", "threshold", "project", "workspace", "output_format"],
"schema_validate": ["note_type", "identifier", "project", "workspace", "output_format"],
"search": ["query"],
"search_by_metadata": ["filters", "project", "workspace", "limit", "offset"],
"search_notes": [
"query",
"project",
@@ -126,7 +125,6 @@ TOOL_FUNCTIONS: dict[str, object] = {
"schema_infer": tools.schema_infer,
"schema_validate": tools.schema_validate,
"search": tools.search,
"search_by_metadata": tools.search_by_metadata,
"search_notes": tools.search_notes,
"view_note": tools.view_note,
"write_note": tools.write_note,
tests/mcp/test_tool_search.py
+43 -90
View File
@@ -6,7 +6,7 @@ from datetime import datetime, timedelta
from basic_memory.mcp.tools import write_note
from basic_memory.mcp.tools.search import search_notes, _format_search_error_response
from basic_memory.schemas.search import SearchItemType, SearchResponse
from basic_memory.schemas.search import SearchResponse
@pytest.mark.asyncio
@@ -526,14 +526,12 @@ async def test_search_notes_passes_metadata_filters(monkeypatch):
assert captured_payload["status"] == "published"
# --- Tests for search_by_metadata tool (lines 505-556) ---------------------
# --- Tests for filter-only search (query=None) --------------------------------
@pytest.mark.asyncio
async def test_search_by_metadata_basic(monkeypatch):
"""search_by_metadata calls SearchClient with correct structured query."""
from basic_memory.mcp.tools.search import search_by_metadata
async def test_search_notes_filter_only_metadata(monkeypatch):
"""search_notes with metadata_filters only (no query) sends correct payload."""
import importlib
search_mod = importlib.import_module("basic_memory.mcp.tools.search")
@@ -560,37 +558,22 @@ async def test_search_by_metadata_basic(monkeypatch):
monkeypatch.setattr(search_mod, "get_project_client", fake_get_project_client)
monkeypatch.setattr(clients_mod, "SearchClient", MockSearchClient)
result = await search_by_metadata(
filters={"status": "in-progress"},
result = await search_mod.search_notes(
project="test-project",
limit=10,
offset=0,
metadata_filters={"status": "in-progress"},
)
assert isinstance(result, SearchResponse)
assert captured_payload["metadata_filters"] == {"status": "in-progress"}
assert captured_payload["entity_types"] == ["entity"]
# No text/title/permalink should be set
assert captured_payload.get("text") is None
assert captured_payload.get("title") is None
assert captured_payload.get("permalink") is None
@pytest.mark.asyncio
async def test_search_by_metadata_limit_zero():
"""search_by_metadata rejects limit <= 0 with error string."""
from basic_memory.mcp.tools.search import search_by_metadata
result = await search_by_metadata(
filters={"status": "active"},
limit=0,
)
assert isinstance(result, str)
assert "limit" in result.lower()
@pytest.mark.asyncio
async def test_search_by_metadata_offset_within_page(monkeypatch):
"""When offset doesn't align to page boundary, results are trimmed."""
from basic_memory.mcp.tools.search import search_by_metadata
async def test_search_notes_filter_only_tags(monkeypatch):
"""search_notes with tags only (no query) sends correct payload."""
import importlib
search_mod = importlib.import_module("basic_memory.mcp.tools.search")
@@ -604,80 +587,50 @@ async def test_search_by_metadata_offset_within_page(monkeypatch):
async def fake_get_project_client(*args, **kwargs):
yield (object(), StubProject())
from basic_memory.schemas.search import SearchResult
fake_items = [
SearchResult(
title=f"Item {i}",
permalink=f"item-{i}",
file_path=f"item-{i}.md",
type=SearchItemType.ENTITY,
score=1.0 - i * 0.1,
)
for i in range(5)
]
class MockSearchClient:
def __init__(self, *args, **kwargs):
self.call_count = 0
async def search(self, payload, page, page_size):
self.call_count += 1
if page == 1:
return SearchResponse(results=fake_items, current_page=1, page_size=page_size)
return SearchResponse(results=[], current_page=page, page_size=page_size)
monkeypatch.setattr(search_mod, "get_project_client", fake_get_project_client)
monkeypatch.setattr(clients_mod, "SearchClient", MockSearchClient)
# offset=2, limit=5 → page=1, offset_within_page=2
result = await search_by_metadata(
filters={"status": "active"},
project="test-project",
limit=5,
offset=2,
)
assert isinstance(result, SearchResponse)
# Should have sliced off the first 2 items
assert result.results[0].title == "Item 2"
@pytest.mark.asyncio
async def test_search_by_metadata_error_handling(monkeypatch):
"""search_by_metadata returns error string on exception."""
from basic_memory.mcp.tools.search import search_by_metadata
import importlib
search_mod = importlib.import_module("basic_memory.mcp.tools.search")
clients_mod = importlib.import_module("basic_memory.mcp.clients")
class StubProject:
name = "test-project"
external_id = "test-external-id"
@asynccontextmanager
async def fake_get_project_client(*args, **kwargs):
yield (object(), StubProject())
captured_payload: dict = {}
class MockSearchClient:
def __init__(self, *args, **kwargs):
pass
async def search(self, *args, **kwargs):
raise RuntimeError("database connection lost")
async def search(self, payload, page, page_size):
captured_payload.update(payload)
return SearchResponse(results=[], current_page=page, page_size=page_size)
monkeypatch.setattr(search_mod, "get_project_client", fake_get_project_client)
monkeypatch.setattr(clients_mod, "SearchClient", MockSearchClient)
result = await search_by_metadata(
filters={"status": "active"},
result = await search_mod.search_notes(
project="test-project",
tags=["security", "oauth"],
)
assert isinstance(result, SearchResponse)
assert captured_payload["tags"] == ["security", "oauth"]
assert captured_payload.get("text") is None
@pytest.mark.asyncio
async def test_search_notes_no_criteria_returns_error(monkeypatch):
"""search_notes with no args at all returns a helpful error string."""
import importlib
search_mod = importlib.import_module("basic_memory.mcp.tools.search")
class StubProject:
name = "test-project"
external_id = "test-external-id"
@asynccontextmanager
async def fake_get_project_client(*args, **kwargs):
yield (object(), StubProject())
monkeypatch.setattr(search_mod, "get_project_client", fake_get_project_client)
result = await search_mod.search_notes(project="test-project")
assert isinstance(result, str)
assert "Search Failed" in result
assert "No Search Criteria" in result
@pytest.mark.asyncio