fix: enhance character conflict detection and error handling for sync operations (#201)

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
This commit is contained in:
Paul Hernandez
2025-08-01 21:35:56 -05:00
committed by GitHub
parent 7585a29c96
commit fb1350b294
5 changed files with 734 additions and 4 deletions
+241
View File
@@ -0,0 +1,241 @@
# Character Handling and Conflict Resolution
Basic Memory handles various character encoding scenarios and file naming conventions to provide consistent permalink generation and conflict resolution. This document explains how the system works and how to resolve common character-related issues.
## Overview
Basic Memory uses a sophisticated system to generate permalinks from file paths while maintaining consistency across different operating systems and character encodings. The system normalizes file paths and generates unique permalinks to prevent conflicts.
## Character Normalization Rules
### 1. Permalink Generation
When Basic Memory processes a file path, it applies these normalization rules:
```
Original: "Finance/My Investment Strategy.md"
Permalink: "finance/my-investment-strategy"
```
**Transformation process:**
1. Remove file extension (`.md`)
2. Convert to lowercase (case-insensitive)
3. Replace spaces with hyphens
4. Replace underscores with hyphens
5. Handle international characters (transliteration for Latin, preservation for non-Latin)
6. Convert camelCase to kebab-case
### 2. International Character Support
**Latin characters with diacritics** are transliterated:
- `ø``o` (Søren → soren)
- `ü``u` (Müller → muller)
- `é``e` (Café → cafe)
- `ñ``n` (Niño → nino)
**Non-Latin characters** are preserved:
- Chinese: `中文/测试文档.md``中文/测试文档`
- Japanese: `日本語/文書.md``日本語/文書`
## Common Conflict Scenarios
### 1. Hyphen vs Space Conflicts
**Problem:** Files with existing hyphens conflict with generated permalinks from spaces.
**Example:**
```
File 1: "basic memory bug.md" → permalink: "basic-memory-bug"
File 2: "basic-memory-bug.md" → permalink: "basic-memory-bug" (CONFLICT!)
```
**Resolution:** The system automatically resolves this by adding suffixes:
```
File 1: "basic memory bug.md" → permalink: "basic-memory-bug"
File 2: "basic-memory-bug.md" → permalink: "basic-memory-bug-1"
```
**Best Practice:** Choose consistent naming conventions within your project.
### 2. Case Sensitivity Conflicts
**Problem:** Different case variations that normalize to the same permalink.
**Example on macOS:**
```
Directory: Finance/investment.md
Directory: finance/investment.md (different on filesystem, same permalink)
```
**Resolution:** Basic Memory detects case conflicts and prevents them during sync operations with helpful error messages.
**Best Practice:** Use consistent casing for directory and file names.
### 3. Character Encoding Conflicts
**Problem:** Different Unicode normalizations of the same logical character.
**Example:**
```
File 1: "café.md" (é as single character)
File 2: "café.md" (e + combining accent)
```
**Resolution:** Basic Memory normalizes Unicode characters using NFD normalization to detect these conflicts.
### 4. Forward Slash Conflicts
**Problem:** Forward slashes in frontmatter or file names interpreted as path separators.
**Example:**
```yaml
---
permalink: finance/investment/strategy
---
```
**Resolution:** Basic Memory validates frontmatter permalinks and warns about path separator conflicts.
## Error Messages and Troubleshooting
### "UNIQUE constraint failed: entity.file_path, entity.project_id"
**Cause:** Two entities trying to use the same file path within a project.
**Common scenarios:**
1. File move operation where destination is already occupied
2. Case sensitivity differences on macOS
3. Character encoding conflicts
4. Concurrent file operations
**Resolution steps:**
1. Check for duplicate file names with different cases
2. Look for files with similar names but different character encodings
3. Rename conflicting files to have unique names
4. Run sync again after resolving conflicts
### "File path conflict detected during move"
**Cause:** Enhanced conflict detection preventing potential database integrity violations.
**What this means:** The system detected that moving a file would create a conflict before attempting the database operation.
**Resolution:** Follow the specific guidance in the error message, which will indicate the type of conflict detected.
## Best Practices
### 1. File Naming Conventions
**Recommended patterns:**
- Use consistent casing (prefer lowercase)
- Use hyphens instead of spaces for multi-word files
- Avoid special characters that could conflict with path separators
- Be consistent with directory structure casing
**Examples:**
```
✅ Good:
- finance/investment-strategy.md
- projects/basic-memory-features.md
- docs/api-reference.md
❌ Problematic:
- Finance/Investment Strategy.md (mixed case, spaces)
- finance/Investment Strategy.md (inconsistent case)
- docs/API/Reference.md (mixed case directories)
```
### 2. Permalink Management
**Custom permalinks in frontmatter:**
```yaml
---
type: knowledge
permalink: custom-permalink-name
---
```
**Guidelines:**
- Use lowercase permalinks
- Use hyphens for word separation
- Avoid path separators unless creating sub-paths
- Ensure uniqueness within your project
### 3. Directory Structure
**Consistent casing:**
```
✅ Good:
finance/
investment-strategies.md
portfolio-management.md
❌ Problematic:
Finance/ (capital F)
investment-strategies.md
finance/ (lowercase f)
portfolio-management.md
```
## Migration and Cleanup
### Identifying Conflicts
Use Basic Memory's built-in conflict detection:
```bash
# Sync will report conflicts
basic-memory sync
# Check sync status for warnings
basic-memory status
```
### Resolving Existing Conflicts
1. **Identify conflicting files** from sync error messages
2. **Choose consistent naming convention** for your project
3. **Rename files** to follow the convention
4. **Re-run sync** to verify resolution
### Bulk Renaming Strategy
For projects with many conflicts:
1. **Backup your project** before making changes
2. **Standardize on lowercase** file and directory names
3. **Replace spaces with hyphens** in file names
4. **Use consistent character encoding** (UTF-8)
5. **Test sync after each batch** of changes
## System Enhancements
### Recent Improvements (v0.13+)
1. **Enhanced conflict detection** before database operations
2. **Improved error messages** with specific resolution guidance
3. **Character normalization utilities** for consistent handling
4. **File swap detection** for complex move scenarios
5. **Proactive conflict warnings** during permalink resolution
### Monitoring and Logging
The system now provides detailed logging for conflict resolution:
```
DEBUG: Detected potential file path conflicts for 'Finance/Investment.md': ['finance/investment.md']
WARNING: File path conflict detected during move: entity_id=123 trying to move from 'old.md' to 'new.md'
```
These logs help identify and resolve conflicts before they cause sync failures.
## Support and Resources
If you encounter character-related conflicts not covered in this guide:
1. **Check the logs** for specific conflict details
2. **Review error messages** for resolution guidance
3. **Report issues** with examples of the conflicting files
4. **Consider the file naming best practices** outlined above
The Basic Memory system is designed to handle most character conflicts automatically while providing clear guidance for manual resolution when needed.
+49 -3
View File
@@ -15,6 +15,7 @@ from basic_memory.markdown.entity_parser import EntityParser
from basic_memory.markdown.utils import entity_model_from_markdown, schema_to_markdown
from basic_memory.models import Entity as EntityModel
from basic_memory.models import Observation, Relation
from basic_memory.models.knowledge import Entity
from basic_memory.repository import ObservationRepository, RelationRepository
from basic_memory.repository.entity_repository import EntityRepository
from basic_memory.schemas import Entity as EntitySchema
@@ -44,6 +45,39 @@ class EntityService(BaseService[EntityModel]):
self.file_service = file_service
self.link_resolver = link_resolver
async def detect_file_path_conflicts(self, file_path: str) -> List[Entity]:
"""Detect potential file path conflicts for a given file path.
This checks for entities with similar file paths that might cause conflicts:
- Case sensitivity differences (Finance/file.md vs finance/file.md)
- Character encoding differences
- Hyphen vs space differences
- Unicode normalization differences
Args:
file_path: The file path to check for conflicts
Returns:
List of entities that might conflict with the given file path
"""
from basic_memory.utils import detect_potential_file_conflicts
conflicts = []
# Get all existing file paths
all_entities = await self.repository.find_all()
existing_paths = [entity.file_path for entity in all_entities]
# Use the enhanced conflict detection utility
conflicting_paths = detect_potential_file_conflicts(file_path, existing_paths)
# Find the entities corresponding to conflicting paths
for entity in all_entities:
if entity.file_path in conflicting_paths:
conflicts.append(entity)
return conflicts
async def resolve_permalink(
self, file_path: Permalink | Path, markdown: Optional[EntityMarkdown] = None
) -> str:
@@ -54,18 +88,30 @@ class EntityService(BaseService[EntityModel]):
2. If markdown has permalink but it's used by another file -> make unique
3. For existing files, keep current permalink from db
4. Generate new unique permalink from file path
Enhanced to detect and handle character-related conflicts.
"""
file_path_str = str(file_path)
# Check for potential file path conflicts before resolving permalink
conflicts = await self.detect_file_path_conflicts(file_path_str)
if conflicts:
logger.warning(
f"Detected potential file path conflicts for '{file_path_str}': "
f"{[entity.file_path for entity in conflicts]}"
)
# If markdown has explicit permalink, try to validate it
if markdown and markdown.frontmatter.permalink:
desired_permalink = markdown.frontmatter.permalink
existing = await self.repository.get_by_permalink(desired_permalink)
# If no conflict or it's our own file, use as is
if not existing or existing.file_path == str(file_path):
if not existing or existing.file_path == file_path_str:
return desired_permalink
# For existing files, try to find current permalink
existing = await self.repository.get_by_file_path(str(file_path))
existing = await self.repository.get_by_file_path(file_path_str)
if existing:
return existing.permalink
@@ -75,7 +121,7 @@ class EntityService(BaseService[EntityModel]):
else:
desired_permalink = generate_permalink(file_path)
# Make unique if needed
# Make unique if needed - enhanced to handle character conflicts
permalink = desired_permalink
suffix = 1
while await self.repository.get_by_permalink(permalink):
+50 -1
View File
@@ -453,6 +453,36 @@ class SyncService:
entity = await self.entity_repository.get_by_file_path(old_path)
if entity:
# Check if destination path is already occupied by another entity
existing_at_destination = await self.entity_repository.get_by_file_path(new_path)
if existing_at_destination and existing_at_destination.id != entity.id:
# Handle the conflict - this could be a file swap or replacement scenario
logger.warning(
f"File path conflict detected during move: "
f"entity_id={entity.id} trying to move from '{old_path}' to '{new_path}', "
f"but entity_id={existing_at_destination.id} already occupies '{new_path}'"
)
# Check if this is a file swap (the destination entity is being moved to our old path)
# This would indicate a simultaneous move operation
old_path_after_swap = await self.entity_repository.get_by_file_path(old_path)
if old_path_after_swap and old_path_after_swap.id == existing_at_destination.id:
logger.info(f"Detected file swap between '{old_path}' and '{new_path}'")
# This is a swap scenario - both moves should succeed
# We'll allow this to proceed since the other file has moved out
else:
# This is a conflict where the destination is occupied
raise ValueError(
f"Cannot move entity from '{old_path}' to '{new_path}': "
f"destination path is already occupied by another file. "
f"This may be caused by: "
f"1. Conflicting file names with different character encodings, "
f"2. Case sensitivity differences (e.g., 'Finance/' vs 'finance/'), "
f"3. Character conflicts between hyphens in filenames and generated permalinks, "
f"4. Files with similar names containing special characters. "
f"Try renaming one of the conflicting files to resolve this issue."
)
# Update file_path in all cases
updates = {"file_path": new_path}
@@ -477,7 +507,26 @@ class SyncService:
f"new_checksum={new_checksum}"
)
updated = await self.entity_repository.update(entity.id, updates)
try:
updated = await self.entity_repository.update(entity.id, updates)
except Exception as e:
# Catch any database integrity errors and provide helpful context
if "UNIQUE constraint failed" in str(e):
logger.error(
f"Database constraint violation during move: "
f"entity_id={entity.id}, old_path='{old_path}', new_path='{new_path}'"
)
raise ValueError(
f"Cannot complete move from '{old_path}' to '{new_path}': "
f"a database constraint was violated. This usually indicates "
f"a file path or permalink conflict. Please check for: "
f"1. Duplicate file names, "
f"2. Case sensitivity issues (e.g., 'File.md' vs 'file.md'), "
f"3. Character encoding conflicts in file names."
) from e
else:
# Re-raise other exceptions as-is
raise
if updated is None: # pragma: no cover
logger.error(
+73
View File
@@ -181,6 +181,79 @@ def setup_logging(
logging.getLogger(logger_name).setLevel(level)
def normalize_file_path_for_comparison(file_path: str) -> str:
"""Normalize a file path for conflict detection.
This function normalizes file paths to help detect potential conflicts:
- Converts to lowercase for case-insensitive comparison
- Normalizes Unicode characters
- Handles path separators consistently
Args:
file_path: The file path to normalize
Returns:
Normalized file path for comparison purposes
"""
import unicodedata
# Convert to lowercase for case-insensitive comparison
normalized = file_path.lower()
# Normalize Unicode characters (NFD normalization)
normalized = unicodedata.normalize('NFD', normalized)
# Replace path separators with forward slashes
normalized = normalized.replace('\\', '/')
# Remove multiple slashes
normalized = re.sub(r'/+', '/', normalized)
return normalized
def detect_potential_file_conflicts(file_path: str, existing_paths: List[str]) -> List[str]:
"""Detect potential conflicts between a file path and existing paths.
This function checks for various types of conflicts:
- Case sensitivity differences
- Unicode normalization differences
- Path separator differences
- Permalink generation conflicts
Args:
file_path: The file path to check
existing_paths: List of existing file paths to check against
Returns:
List of existing paths that might conflict with the given file path
"""
conflicts = []
# Normalize the input file path
normalized_input = normalize_file_path_for_comparison(file_path)
input_permalink = generate_permalink(file_path)
for existing_path in existing_paths:
# Skip identical paths
if existing_path == file_path:
continue
# Check for case-insensitive path conflicts
normalized_existing = normalize_file_path_for_comparison(existing_path)
if normalized_input == normalized_existing:
conflicts.append(existing_path)
continue
# Check for permalink conflicts
existing_permalink = generate_permalink(existing_path)
if input_permalink == existing_permalink:
conflicts.append(existing_path)
continue
return conflicts
def parse_tags(tags: Union[List[str], str, None]) -> List[str]:
"""Parse tags from various input formats into a consistent list.
+321
View File
@@ -0,0 +1,321 @@
"""Test character-related sync conflicts and permalink generation."""
import asyncio
from pathlib import Path
from textwrap import dedent
import pytest
from sqlalchemy.exc import IntegrityError
from basic_memory.config import ProjectConfig
from basic_memory.models import Entity
from basic_memory.repository import EntityRepository
from basic_memory.services import EntityService
from basic_memory.sync.sync_service import SyncService
from basic_memory.utils import (
generate_permalink,
normalize_file_path_for_comparison,
detect_potential_file_conflicts
)
async def create_test_file(path: Path, content: str = "test content") -> None:
"""Create a test file with given content."""
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(content)
class TestUtilityFunctions:
"""Test utility functions for file path normalization and conflict detection."""
def test_normalize_file_path_for_comparison(self):
"""Test file path normalization for conflict detection."""
# Case sensitivity normalization
assert normalize_file_path_for_comparison("Finance/Investment.md") == "finance/investment.md"
assert normalize_file_path_for_comparison("FINANCE/INVESTMENT.MD") == "finance/investment.md"
# Path separator normalization
assert normalize_file_path_for_comparison("Finance\\Investment.md") == "finance/investment.md"
# Multiple slash handling
assert normalize_file_path_for_comparison("Finance//Investment.md") == "finance/investment.md"
def test_detect_potential_file_conflicts(self):
"""Test the enhanced conflict detection function."""
existing_paths = [
"Finance/Investment.md",
"finance/Investment.md",
"docs/my-feature.md",
"docs/my feature.md"
]
# Case sensitivity conflict
conflicts = detect_potential_file_conflicts("FINANCE/INVESTMENT.md", existing_paths)
assert "Finance/Investment.md" in conflicts
assert "finance/Investment.md" in conflicts
# Permalink conflict (space vs hyphen)
conflicts = detect_potential_file_conflicts("docs/my_feature.md", existing_paths)
assert "docs/my-feature.md" in conflicts
assert "docs/my feature.md" in conflicts
class TestPermalinkGeneration:
"""Test permalink generation with various character scenarios."""
def test_hyphen_handling(self):
"""Test that hyphens in filenames are handled consistently."""
# File with existing hyphens
assert generate_permalink("docs/my-feature.md") == "docs/my-feature"
assert generate_permalink("docs/basic-memory bug.md") == "docs/basic-memory-bug"
# File with spaces that become hyphens
assert generate_permalink("docs/my feature.md") == "docs/my-feature"
# Mixed scenarios
assert generate_permalink("docs/my-old feature.md") == "docs/my-old-feature"
def test_forward_slash_handling(self):
"""Test that forward slashes are handled properly."""
# Normal directory structure
assert generate_permalink("Finance/Investment.md") == "finance/investment"
# Path with spaces in directory names
assert generate_permalink("My Finance/Investment.md") == "my-finance/investment"
def test_case_sensitivity_normalization(self):
"""Test that case differences are normalized consistently."""
# Same logical path with different cases
assert generate_permalink("Finance/Investment.md") == "finance/investment"
assert generate_permalink("finance/Investment.md") == "finance/investment"
assert generate_permalink("FINANCE/INVESTMENT.md") == "finance/investment"
def test_unicode_character_handling(self):
"""Test that international characters are handled properly."""
# Italian characters as mentioned in user feedback
assert generate_permalink("Finance/Punti Chiave di Peter Lynch.md") == "finance/punti-chiave-di-peter-lynch"
# Chinese characters (should be preserved)
assert generate_permalink("中文/测试文档.md") == "中文/测试文档"
# Mixed international characters
assert generate_permalink("docs/Café München.md") == "docs/cafe-munchen"
def test_special_punctuation(self):
"""Test handling of special punctuation characters."""
# Apostrophes should be removed
assert generate_permalink("Peter's Guide.md") == "peters-guide"
# Other punctuation should become hyphens
assert generate_permalink("Q&A Session.md") == "q-a-session"
@pytest.mark.asyncio
class TestSyncConflictHandling:
"""Test sync service handling of file path and permalink conflicts."""
async def test_file_path_conflict_detection(
self,
sync_service: SyncService,
project_config: ProjectConfig,
entity_repository: EntityRepository,
):
"""Test that file path conflicts are detected during move operations."""
project_dir = project_config.home
# Create two files
content1 = dedent("""
---
type: knowledge
---
# Document One
This is the first document.
""")
content2 = dedent("""
---
type: knowledge
---
# Document Two
This is the second document.
""")
await create_test_file(project_dir / "doc1.md", content1)
await create_test_file(project_dir / "doc2.md", content2)
# Initial sync
await sync_service.sync(project_config.home)
# Verify both entities exist
entities = await entity_repository.find_all()
assert len(entities) == 2
# Now simulate a move where doc1.md tries to move to doc2.md's location
# This should be handled gracefully, not throw an IntegrityError
# First, get the entities
entity1 = await entity_repository.get_by_file_path("doc1.md")
entity2 = await entity_repository.get_by_file_path("doc2.md")
assert entity1 is not None
assert entity2 is not None
# Simulate the conflict scenario
with pytest.raises(Exception) as exc_info:
# This should detect the conflict and handle it gracefully
await sync_service.handle_move("doc1.md", "doc2.md")
# The exception should be a meaningful error, not an IntegrityError
assert not isinstance(exc_info.value, IntegrityError)
async def test_hyphen_filename_conflict(
self,
sync_service: SyncService,
project_config: ProjectConfig,
entity_repository: EntityRepository,
):
"""Test conflict when filename with hyphens conflicts with generated permalink."""
project_dir = project_config.home
# Create file with spaces (will generate permalink with hyphens)
content1 = dedent("""
---
type: knowledge
---
# Basic Memory Bug
This file has spaces in the name.
""")
# Create file with hyphens (already has hyphens in filename)
content2 = dedent("""
---
type: knowledge
---
# Basic Memory Bug Report
This file has hyphens in the name.
""")
await create_test_file(project_dir / "basic memory bug.md", content1)
await create_test_file(project_dir / "basic-memory-bug.md", content2)
# Sync should handle this without conflict
await sync_service.sync(project_config.home)
# Verify both entities were created with unique permalinks
entities = await entity_repository.find_all()
assert len(entities) == 2
# Check that permalinks are unique
permalinks = [entity.permalink for entity in entities if entity.permalink]
assert len(set(permalinks)) == len(permalinks), "Permalinks should be unique"
async def test_case_sensitivity_conflict(
self,
sync_service: SyncService,
project_config: ProjectConfig,
entity_repository: EntityRepository,
):
"""Test conflict handling when case differences cause issues."""
project_dir = project_config.home
# Create directory structure that might cause case conflicts
(project_dir / "Finance").mkdir(parents=True, exist_ok=True)
(project_dir / "finance").mkdir(parents=True, exist_ok=True)
content1 = dedent("""
---
type: knowledge
---
# Investment Guide
Upper case directory.
""")
content2 = dedent("""
---
type: knowledge
---
# Investment Tips
Lower case directory.
""")
await create_test_file(project_dir / "Finance" / "investment.md", content1)
await create_test_file(project_dir / "finance" / "investment.md", content2)
# Sync should handle case differences properly
await sync_service.sync(project_config.home)
# Verify entities were created
entities = await entity_repository.find_all()
assert len(entities) >= 2 # Allow for potential other test files
# Check that file paths are preserved correctly
file_paths = [entity.file_path for entity in entities]
assert "Finance/investment.md" in file_paths
assert "finance/investment.md" in file_paths
async def test_move_conflict_resolution(
self,
sync_service: SyncService,
project_config: ProjectConfig,
entity_repository: EntityRepository,
):
"""Test that move conflicts are resolved with proper error handling."""
project_dir = project_config.home
# Create three files in a scenario that could cause move conflicts
await create_test_file(project_dir / "file-a.md", "# File A")
await create_test_file(project_dir / "file-b.md", "# File B")
await create_test_file(project_dir / "temp.md", "# Temp File")
# Initial sync
await sync_service.sync(project_config.home)
# Simulate a complex move scenario where files swap locations
# This is the kind of scenario that caused the original bug
# Get the entities
entity_a = await entity_repository.get_by_file_path("file-a.md")
entity_b = await entity_repository.get_by_file_path("file-b.md")
entity_temp = await entity_repository.get_by_file_path("temp.md")
assert all([entity_a, entity_b, entity_temp])
# Try to move file-a to file-b's location (should detect conflict)
try:
await sync_service.handle_move("file-a.md", "file-b.md")
# If this doesn't raise an exception, the conflict was resolved
# Verify the state is consistent
updated_entities = await entity_repository.find_all()
file_paths = [entity.file_path for entity in updated_entities]
# Should not have duplicate file paths
assert len(file_paths) == len(set(file_paths)), "File paths should be unique"
except Exception as e:
# If an exception is raised, it should be a meaningful error
assert "conflict" in str(e).lower() or "already exists" in str(e).lower()
assert not isinstance(e, IntegrityError), "Should not be a raw IntegrityError"
@pytest.mark.asyncio
class TestEnhancedErrorMessages:
"""Test that error messages provide helpful guidance for character conflicts."""
async def test_helpful_error_for_hyphen_conflict(
self,
sync_service: SyncService,
project_config: ProjectConfig,
):
"""Test that hyphen conflicts generate helpful error messages."""
# This test will be implemented after we enhance the error handling
pass
async def test_helpful_error_for_case_conflict(
self,
sync_service: SyncService,
project_config: ProjectConfig,
):
"""Test that case sensitivity conflicts generate helpful error messages."""
# This test will be implemented after we enhance the error handling
pass