basicmachines-co-basic-memory/specs/SPEC-8 TigrisFS Integration.md

---
title: 'SPEC-8: TigrisFS Integration for Tenant API'
Date: September 22, 2025
Status: Phase 3.6 Complete - Tenant Mount API Endpoints Ready for CLI Implementation
Priority: High
Goal: Replace Fly volumes with Tigris bucket provisioning in production tenant API
permalink: spec-8-tigris-fs-integration
---

## Executive Summary

Based on SPEC-7 Phase 4 POC testing, this spec outlines productizing the TigrisFS/rclone implementation in the Basic Memory Cloud tenant API. 
We're moving from proof-of-concept to production integration, replacing Fly volume storage with Tigris bucket-per-tenant architecture.

## Current Architecture (Fly Volumes)

### Tenant Provisioning Flow
```python
# apps/cloud/src/basic_memory_cloud/workflows/tenant_provisioning.py
async def provision_tenant_infrastructure(tenant_id: str):
    # 1. Create Fly app
    # 2. Create Fly volume  ← REPLACE THIS
    # 3. Deploy API container with volume mount
    # 4. Configure health checks
```

### Storage Implementation
- Each tenant gets dedicated Fly volume (1GB-10GB)
- Volume mounted at `/app/data` in API container
- Local filesystem storage with Basic Memory indexing
- No global caching or edge distribution

## Proposed Architecture (Tigris Buckets)

### New Tenant Provisioning Flow
```python
async def provision_tenant_infrastructure(tenant_id: str):
    # 1. Create Fly app
    # 2. Create Tigris bucket with admin credentials  ← NEW
    # 3. Store bucket name in tenant record  ← NEW
    # 4. Deploy API container with TigrisFS mount using admin credentials
    # 5. Configure health checks
```

### Storage Implementation
- Each tenant gets dedicated Tigris bucket
- TigrisFS mounts bucket at `/app/data` in API container
- Global edge caching and distribution
- Configurable cache TTL for sync performance

## Implementation Plan

### Phase 1: Bucket Provisioning Service

**✅ IMPLEMENTED: StorageClient with Admin Credentials**
```python
# apps/cloud/src/basic_memory_cloud/clients/storage_client.py
class StorageClient:
    async def create_tenant_bucket(self, tenant_id: UUID) -> TigrisBucketCredentials
    async def delete_tenant_bucket(self, tenant_id: UUID, bucket_name: str) -> bool
    async def list_buckets(self) -> list[TigrisBucketResponse]
    async def test_tenant_credentials(self, credentials: TigrisBucketCredentials) -> bool
```

**Simplified Architecture Using Admin Credentials:**
- Single admin access key with full Tigris permissions (configured in console)
- No tenant-specific IAM user creation needed
- Bucket-per-tenant isolation for logical separation
- Admin credentials shared across all tenant operations

**Integrate with Provisioning workflow:**
```python
# Update tenant_provisioning.py
async def provision_tenant_infrastructure(tenant_id: str):
    storage_client = StorageClient(settings.aws_access_key_id, settings.aws_secret_access_key)
    bucket_creds = await storage_client.create_tenant_bucket(tenant_id)
    await store_bucket_name(tenant_id, bucket_creds.bucket_name)
    await deploy_api_with_tigris(tenant_id, bucket_creds)
```

### Phase 2: Simplified Bucket Management

**✅ SIMPLIFIED: Admin Credentials + Bucket Names Only**

Since we use admin credentials for all operations, we only need to track bucket names per tenant:

1. **Primary Storage (Fly Secrets)**
   ```bash
   flyctl secrets set -a basic-memory-{tenant_id} \
     AWS_ACCESS_KEY_ID="{admin_access_key}" \
     AWS_SECRET_ACCESS_KEY="{admin_secret_key}" \
     AWS_ENDPOINT_URL_S3="https://fly.storage.tigris.dev" \
     AWS_REGION="auto" \
     BUCKET_NAME="basic-memory-{tenant_id}"
   ```

2. **Database Storage (Bucket Name Only)**
   ```python
   # apps/cloud/src/basic_memory_cloud/models/tenant.py
   class Tenant(BaseModel):
       # ... existing fields
       tigris_bucket_name: Optional[str] = None  # Just store bucket name
       tigris_region: str = "auto"
       created_at: datetime
   ```

**Benefits of Simplified Approach:**
- No credential encryption/decryption needed
- Admin credentials managed centrally in environment
- Only bucket names stored in database (not sensitive)
- Simplified backup/restore scenarios
- Reduced security attack surface

### Phase 3: API Container Updates

**Update API container configuration:**
```dockerfile
# apps/api/Dockerfile
# Add TigrisFS installation
RUN curl -L https://github.com/tigrisdata/tigrisfs/releases/latest/download/tigrisfs-linux-amd64 \
    -o /usr/local/bin/tigrisfs && chmod +x /usr/local/bin/tigrisfs
```

**Startup script integration:**
```bash
# apps/api/tigrisfs-startup.sh (already exists)
# Mount TigrisFS → Start Basic Memory API
exec python -m basic_memory_cloud_api.main
```

**Fly.toml environment (optimized for < 5s startup):**
```toml
# apps/api/fly.tigris-production.toml
[env]
  TIGRISFS_MEMORY_LIMIT = '1024'     # Reduced for faster init
  TIGRISFS_MAX_FLUSHERS = '16'       # Fewer threads for faster startup
  TIGRISFS_STAT_CACHE_TTL = '30s'    # Balance sync speed vs startup
  TIGRISFS_LAZY_INIT = 'true'        # Enable lazy loading
  BASIC_MEMORY_HOME = '/app/data'

# Suspend optimization for wake-on-network
[machine]
  auto_stop_machines = "suspend"     # Faster than full stop
  auto_start_machines = true
  min_machines_running = 0
```

### Phase 4: Local Access Features

**CLI automation for local mounting:**
```python
# New CLI command: basic-memory cloud mount
async def setup_local_mount(tenant_id: str):
    # 1. Fetch bucket credentials from cloud API
    # 2. Configure rclone with scoped IAM policy
    # 3. Mount via rclone nfsmount (macOS) or FUSE (Linux)
    # 4. Start Basic Memory sync watcher
```

**Local mount configuration:**
```bash
# rclone config for tenant
rclone mount basic-memory-{tenant_id}: ~/basic-memory-{tenant_id} \
  --nfs-mount \
  --vfs-cache-mode writes \
  --cache-dir ~/.cache/rclone/basic-memory-{tenant_id}
```

### Phase 5: TigrisFS Cache Sync Solutions

**Problem**: When files are uploaded via CLI/bisync, the tenant API container doesn't see them immediately due to TigrisFS cache (30s TTL) and lack of inotify events on mounted filesystems.

**Multi-Layer Solution:**

**Layer 1: API Sync Endpoint** (Immediate)
```python
# POST /sync - Force TigrisFS cache refresh
# Callable by CLI after uploads
subprocess.run(["sync", "fsync /app/data"], check=True)
```

**Layer 2: Tigris Webhook Integration** (Real-time)
https://www.tigrisdata.com/docs/buckets/object-notifications/#webhook
```python
# Webhook endpoint for bucket changes
@app.post("/webhooks/tigris/{tenant_id}")
async def handle_bucket_notification(tenant_id: str, event: TigrisEvent):
    if event.eventName in ["OBJECT_CREATED_PUT", "OBJECT_DELETED"]:
        await notify_container_sync(tenant_id, event.object.key)
```

**Layer 3: CLI Sync Notification** (User-triggered)
```bash
# CLI calls container sync endpoint after successful bisync
basic-memory cloud bisync  # Automatically notifies container
curl -X POST https://basic-memory-{tenant-id}.fly.dev/sync
```

**Layer 4: Periodic Sync Fallback** (Safety net)
```python
# Background task: fsync /app/data every 30s as fallback
# Ensures eventual consistency even if other layers fail
```

**Implementation Priority:**
1. Layer 1 (API endpoint) - Quick testing capability
2. Layer 3 (CLI integration) - Improved UX
3. Layer 4 (Periodic fallback) - Safety net
4. Layer 2 (Webhooks) - Production real-time sync


## Performance Targets

### Sync Latency
- **Target**: < 5 seconds local→cloud→container
- **Configuration**: `TIGRISFS_STAT_CACHE_TTL = '5s'`
- **Monitoring**: Track sync metrics in production

### Container Startup
- **Target**: < 5 seconds including TigrisFS mount
- **Fast retry**: 0.5s intervals for mount verification
- **Fallback**: Container fails fast if mount fails

### Memory Usage
- **TigrisFS cache**: 2GB memory limit per container
- **Concurrent uploads**: 32 flushers max
- **VM sizing**: shared-cpu-2x (2048mb) minimum

## Security Considerations

### Bucket Isolation
- Each tenant has dedicated bucket
- IAM policies prevent cross-tenant access
- No shared bucket with subdirectories

### Credential Security
- Fly secrets for runtime access
- Encrypted database backup for disaster recovery
- Credential rotation capability

### Data Residency
- Tigris global edge caching
- SOC2 Type II compliance
- Encryption at rest and in transit

## Operational Benefits

### Scalability
- Horizontal scaling with stateless API containers
- Global edge distribution
- Better resource utilization

### Reliability
- No cold starts between tenants
- Built-in redundancy and caching
- Simplified backup strategy

### Cost Efficiency
- Pay-per-use storage pricing
- Shared infrastructure benefits
- Reduced operational overhead

## Risk Mitigation

### Data Loss Prevention
- Dual credential storage (Fly + database)
- Automated backup workflows to R2/S3
- Tigris built-in redundancy

### Performance Degradation
- Configurable cache settings per tenant
- Monitoring and alerting on sync latency
- Fallback to volume storage if needed

### Security Vulnerabilities
- Bucket-per-tenant isolation
- Regular credential rotation
- Security scanning and monitoring

## Success Metrics

### Technical Metrics
- Sync latency P50 < 5 seconds
- Container startup time < 5 seconds
- Zero data loss incidents
- 99.9% uptime per tenant

### Business Metrics
- Reduced infrastructure costs vs volumes
- Improved user experience with faster sync
- Enhanced enterprise security posture
- Simplified operational overhead

## Open Questions

1. **Tigris rate limits**: What are the API limits for bucket creation?
2. **Cost analysis**: What's the break-even point vs Fly volumes?
3. **Regional preferences**: Should enterprise customers choose regions?
4. **Backup retention**: How long to keep automated backups?

## Implementation Checklist

### Phase 1: Bucket Provisioning Service ✅ COMPLETED
- [x] **Research Tigris bucket API** - Document bucket creation and S3 API compatibility
- [x] **Create StorageClient class** - Implemented with admin credentials and comprehensive integration tests
- [x] **Test bucket creation** - Full test suite validates API integration with real Tigris environment
- [x] **Add bucket provisioning to DBOS workflow** - Integrated StorageClient with tenant_provisioning.py

### Phase 2: Simplified Bucket Management ✅ COMPLETED
- [x] **Update Tenant model** with tigris_bucket_name field (replaced fly_volume_id)
- [x] **Implement bucket name storage** - Database migration and model updates completed
- [x] **Test bucket provisioning integration** - Full test suite validates workflow from tenant creation to bucket assignment
- [x] **Remove volume logic from all tests** - Complete migration from volume-based to bucket-based architecture

### Phase 3: API Container Integration ✅ COMPLETED
- [x] **Update Dockerfile** to install TigrisFS binary in API container with configurable version
- [x] **Optimize tigrisfs-startup.sh** with production-ready security and reliability improvements
- [x] **Create production-ready container** with proper signal handling and mount validation
- [x] **Implement security fixes** based on Claude code review (conditional debug, credential protection)
- [x] **Add proper process supervision** with cleanup traps and error handling
- [x] **Remove debug artifacts** - Cleaned up all debug Dockerfiles and test scripts

### Phase 3.5: IAM Access Key Management ✅ COMPLETED
- [x] **Research Tigris IAM API** - Documented create_policy, attach_user_policy, delete_access_key operations
- [x] **Implement bucket-scoped credential generation** - StorageClient.create_tenant_access_keys() with IAM policies
- [x] **Add comprehensive security test suite** - 5 security-focused integration tests covering all attack vectors
- [x] **Verify cross-bucket access prevention** - Scoped credentials can ONLY access their designated bucket
- [x] **Test credential lifecycle management** - Create, validate, delete, and revoke access keys
- [x] **Validate admin vs scoped credential isolation** - Different access patterns and security boundaries
- [x] **Test multi-tenant isolation** - Multiple tenants cannot access each other's buckets

### Phase 3.6: Tenant Mount API Endpoints ✅ COMPLETED
- [x] **Implement GET /tenant/mount/info** - Returns mount info without exposing credentials
- [x] **Implement POST /tenant/mount/credentials** - Creates new bucket-scoped credentials for CLI mounting
- [x] **Implement DELETE /tenant/mount/credentials/{cred_id}** - Revoke specific credentials with proper cleanup
- [x] **Implement GET /tenant/mount/credentials** - List active credentials without exposing secrets
- [x] **Add TenantMountCredentials database model** - Tracks credential metadata (no secret storage)
- [x] **Create comprehensive test suite** - 28 tests covering all scenarios including multi-session support
- [x] **Implement multi-session credential flow** - Multiple active credentials per tenant supported
- [x] **Secure credential handling** - Secret keys never stored, returned once only for immediate use
- [x] **Add dependency injection for StorageClient** - Clean integration with existing API architecture
- [x] **Fix Tigris configuration for cloud service** - Added AWS environment variables to fly.template.toml
- [x] **Update tenant machine configurations** - Include AWS credentials for TigrisFS mounting with clear credential strategy

**Security Test Results:**
```
✅ Cross-bucket access prevention - PASS
✅ Deleted credentials access revoked - PASS
✅ Invalid credentials rejected - PASS
✅ Admin vs scoped credential isolation - PASS
✅ Multiple scoped credentials isolation - PASS
```

**Implementation Details:**
- Uses Tigris IAM managed policies (create_policy + attach_user_policy)
- Bucket-scoped S3 policies with Actions: GetObject, PutObject, DeleteObject, ListBucket
- Resource ARNs limited to specific bucket: `arn:aws:s3:::bucket-name` and `arn:aws:s3:::bucket-name/*`
- Access keys follow Tigris format: `tid_` prefix with secure random suffix
- Complete cleanup on deletion removes both access keys and associated policies

### Phase 4: Local Access CLI
- [x] **Design local mount CLI command** for automated rclone configuration
- [x] **Implement credential fetching** from cloud API for local setup
- [x] **Create rclone config automation** for tenant-specific bucket mounting
- [x] **Test local→cloud→container sync** with optimized cache settings
- [x] **Document local access setup** for beta users

### Phase 5: Webhook Integration (Future)
- [ ] **Research Tigris webhook API** for object notifications and payload format
- [ ] **Design webhook endpoint** for real-time sync notifications
- [ ] **Implement notification handling** to trigger Basic Memory sync events
- [ ] **Test webhook delivery** and sync latency improvements

## Success Metrics
- [ ] **Container startup < 5 seconds** including TigrisFS mount and Basic Memory init
- [ ] **Sync latency < 5 seconds** for local→cloud→container file changes
- [ ] **Zero data loss** during bucket provisioning and credential management
- [ ] **100% test coverage** for new TigrisBucketService and credential functions
- [ ] **Beta deployment** with internal users validating local-cloud workflow



## Implementation Notes

## Phase 4.1: Bidirectional Sync with rclone bisync (NEW)

### Problem Statement
During testing, we discovered that some applications (particularly Obsidian) don't detect file changes over NFS mounts. Rather than building a custom sync daemon, we can leverage `rclone bisync` - rclone's built-in bidirectional synchronization feature.

### Solution: rclone bisync
Use rclone's proven bidirectional sync instead of custom implementation:

**Core Architecture:**
```bash
# rclone bisync handles all the complexity
rclone bisync ~/basic-memory-{tenant_id} basic-memory-{tenant_id}:{bucket_name} \
  --create-empty-src-dirs \
  --conflict-resolve newer \
  --resilient \
  --check-access
```

**Key Benefits:**
- ✅ **Battle-tested**: Production-proven rclone functionality
- ✅ **MIT licensed**: Open source with permissive licensing
- ✅ **No custom code**: Zero maintenance burden for sync logic
- ✅ **Built-in safety**: max-delete protection, conflict resolution
- ✅ **Simple installation**: Works with Homebrew rclone (no FUSE needed)
- ✅ **File watcher compatible**: Works with Obsidian and all applications
- ✅ **Offline support**: Can work offline and sync when connected

### bisync Conflict Resolution Options

**Built-in conflict strategies:**
```bash
--conflict-resolve none     # Keep both files with .conflict suffixes (safest)
--conflict-resolve newer    # Always pick the most recently modified file
--conflict-resolve larger   # Choose based on file size
--conflict-resolve path1    # Always prefer local changes
--conflict-resolve path2    # Always prefer cloud changes
```

### Sync Profiles Using bisync

**Profile configurations:**
```python
BISYNC_PROFILES = {
    "safe": {
        "conflict_resolve": "none",      # Keep both versions
        "max_delete": 10,                # Prevent mass deletion
        "check_access": True,            # Verify sync integrity
        "description": "Safe mode with conflict preservation"
    },
    "balanced": {
        "conflict_resolve": "newer",     # Auto-resolve to newer file
        "max_delete": 25,
        "check_access": True,
        "description": "Balanced mode (recommended default)"
    },
    "fast": {
        "conflict_resolve": "newer",
        "max_delete": 50,
        "check_access": False,           # Skip verification for speed
        "description": "Fast mode for rapid iteration"
    }
}
```

### CLI Commands

**Manual sync commands:**
```bash
basic-memory cloud bisync                    # Manual bidirectional sync
basic-memory cloud bisync --dry-run          # Preview changes
basic-memory cloud bisync --profile safe     # Use specific profile
basic-memory cloud bisync --resync           # Force full baseline resync
```

**Watch mode (Step 1):**
```bash
basic-memory cloud bisync --watch            # Long-running process, sync every 60s
basic-memory cloud bisync --watch --interval 30s  # Custom interval
```

**System integration (Step 2 - Future):**
```bash
basic-memory cloud bisync-service install    # Install as system service
basic-memory cloud bisync-service start      # Start background service
basic-memory cloud bisync-service status     # Check service status
```

### Implementation Strategy

**Phase 4.1.1: Core bisync Implementation**
- [ ] Implement `run_bisync()` function wrapping rclone bisync
- [ ] Add profile-based configuration (safe/balanced/fast)
- [ ] Create conflict resolution and safety options
- [ ] Test with sample files and conflict scenarios

**Phase 4.1.2: Watch Mode**
- [ ] Add `--watch` flag for continuous sync
- [ ] Implement configurable sync intervals
- [ ] Add graceful shutdown and signal handling
- [ ] Create status monitoring and progress indicators

**Phase 4.1.3: User Experience**
- [ ] Add conflict reporting and resolution guidance
- [ ] Implement dry-run preview functionality
- [ ] Create troubleshooting and diagnostic commands
- [ ] Add filtering configuration (.gitignore-style)

**Phase 4.1.4: System Integration (Future)**
- [ ] Generate platform-specific service files (launchd/systemd)
- [ ] Add service management commands
- [ ] Implement automatic startup and recovery
- [ ] Create monitoring and logging integration

### Technical Implementation

**Core bisync wrapper:**
```python
def run_bisync(
    tenant_id: str,
    bucket_name: str,
    profile: str = "balanced",
    dry_run: bool = False
) -> bool:
    """Run rclone bisync with specified profile."""

    local_path = Path.home() / f"basic-memory-{tenant_id}"
    remote_path = f"basic-memory-{tenant_id}:{bucket_name}"
    profile_config = BISYNC_PROFILES[profile]

    cmd = [
        "rclone", "bisync",
        str(local_path), remote_path,
        "--create-empty-src-dirs",
        "--resilient",
        f"--conflict-resolve={profile_config['conflict_resolve']}",
        f"--max-delete={profile_config['max_delete']}",
        "--filters-file", "~/.basic-memory/bisync-filters.txt"
    ]

    if profile_config.get("check_access"):
        cmd.append("--check-access")

    if dry_run:
        cmd.append("--dry-run")

    return subprocess.run(cmd, check=True).returncode == 0
```

**Default filter file (~/.basic-memory/bisync-filters.txt):**
```
- .DS_Store
- .git/**
- __pycache__/**
- *.pyc
- .pytest_cache/**
- node_modules/**
- .conflict-*
- Thumbs.db
- desktop.ini
```

**Advantages Over Custom Daemon:**
- ✅ **Zero maintenance**: No custom sync logic to debug/maintain
- ✅ **Production proven**: Used by thousands in production
- ✅ **Safety features**: Built-in max-delete, conflict handling, recovery
- ✅ **Filtering**: Advanced exclude patterns and rules
- ✅ **Performance**: Optimized for various storage backends
- ✅ **Community support**: Extensive documentation and community

## Phase 4.2: NFS Mount Support (Direct Access)

### Solution: rclone nfsmount
Keep the existing NFS mount functionality for users who prefer direct file access:

**Core Architecture:**
```bash
# rclone nfsmount provides transparent file access
rclone nfsmount basic-memory-{tenant_id}:{bucket_name} ~/basic-memory-{tenant_id} \
  --vfs-cache-mode writes \
  --dir-cache-time 10s \
  --daemon
```

**Key Benefits:**
- ✅ **Real-time access**: Files appear immediately as they're created/modified
- ✅ **Transparent**: Works with any application that reads/writes files
- ✅ **Low latency**: Direct access without sync delays
- ✅ **Simple**: No periodic sync commands needed
- ✅ **Homebrew compatible**: Works with Homebrew rclone (no FUSE required)

**Limitations:**
- ❌ **File watcher compatibility**: Some apps (Obsidian) don't detect changes over NFS
- ❌ **Network dependency**: Requires active connection to cloud storage
- ❌ **Potential conflicts**: Simultaneous edits from multiple locations can cause issues

### Mount Profiles (Existing)

**Already implemented profiles from SPEC-7 testing:**
```python
MOUNT_PROFILES = {
    "fast": {
        "cache_time": "5s",
        "poll_interval": "3s",
        "description": "Ultra-fast development (5s sync)"
    },
    "balanced": {
        "cache_time": "10s",
        "poll_interval": "5s",
        "description": "Fast development (10-15s sync, recommended)"
    },
    "safe": {
        "cache_time": "15s",
        "poll_interval": "10s",
        "description": "Conflict-aware mount with backup",
        "extra_args": ["--conflict-suffix", ".conflict-{DateTimeExt}"]
    }
}
```

### CLI Commands (Existing)

**Mount commands already implemented:**
```bash
basic-memory cloud mount                     # Mount with balanced profile
basic-memory cloud mount --profile fast     # Ultra-fast caching
basic-memory cloud mount --profile safe     # Conflict detection
basic-memory cloud unmount                  # Clean unmount
basic-memory cloud mount-status             # Show mount status
```

## User Choice: Mount vs Bisync

### When to Use Each Approach

| Use Case | Recommended Solution | Why |
|----------|---------------------|-----|
| **Obsidian users** | `bisync` | File watcher support for live preview |
| **CLI/vim/emacs users** | `mount` | Direct file access, lower latency |
| **Offline work** | `bisync` | Can work offline, sync when connected |
| **Real-time collaboration** | `mount` | Immediate visibility of changes |
| **Multiple machines** | `bisync` | Better conflict handling |
| **Single machine** | `mount` | Simpler, more transparent |
| **Development work** | Either | Both work well, user preference |
| **Large files** | `mount` | Streaming access vs full download |

### Installation Simplicity

**Both approaches now use simple Homebrew installation:**
```bash
# Single installation command for both approaches
brew install rclone

# No macFUSE, no system modifications needed
# Works immediately with both mount and bisync
```

### Implementation Status

**Phase 4.1: bisync** (NEW)
- [ ] Implement bisync command wrapper
- [ ] Add watch mode with configurable intervals
- [ ] Create conflict resolution workflows
- [ ] Add filtering and safety options

**Phase 4.2: mount** (EXISTING - ✅ IMPLEMENTED)
- [x] NFS mount commands with profile support
- [x] Mount management and cleanup
- [x] Process monitoring and health checks
- [x] Credential integration with cloud API

**Both approaches share:**
- [x] Credential management via cloud API
- [x] Secure rclone configuration
- [x] Tenant isolation and bucket scoping
- [x] Simple Homebrew rclone installation


Key Features:

1. Cross-Platform rclone Installation (rclone_installer.py):
- macOS: Homebrew → official script fallback
- Linux: snap → apt → official script fallback
- Windows: winget → chocolatey → scoop fallback
- Automatic version detection and verification

2. Smart rclone Configuration (rclone_config.py):
- Automatic tenant-specific config generation
- Three optimized mount profiles from your SPEC-7 testing:
- fast: 5s sync (ultra-performance)
- balanced: 10-15s sync (recommended default)
- safe: 15s sync + conflict detection
- Backup existing configs before modification

3. Robust Mount Management (mount_commands.py):
- Automatic tenant credential generation
- Mount path management (~/basic-memory-{tenant-id})
- Process lifecycle management (prevent duplicate mounts)
- Orphaned process cleanup
- Mount verification and health checking

4. Clean Architecture (api_client.py):
- Separated API client to avoid circular imports
- Reuses existing authentication infrastructure
- Consistent error handling and logging

User Experience:

One-Command Setup:
basic-memory cloud setup
```bash 
# 1. Installs rclone automatically
# 2. Authenticates with existing login
# 3. Generates secure credentials  
# 4. Configures rclone
# 5. Performs initial mount
```

Profile-Based Mounting:
basic-memory cloud mount --profile fast      # 5s sync
basic-memory cloud mount --profile balanced  # 15s sync (default)
basic-memory cloud mount --profile safe      # conflict detection

Status Monitoring:
basic-memory cloud mount-status
```bash 
# Shows: tenant info, mount path, sync profile, rclone processes
```
### local mount api 

Endpoint 1: Get Tenant Info for user
Purpose: Get tenant details for mounting
- pass in jwt
- service returns mount info

**✅ IMPLEMENTED API Specification:**

**Endpoint 1: GET /tenant/mount/info**
- Purpose: Get tenant mount information without exposing credentials
- Authentication: JWT token (tenant_id extracted from claims)

Request:
```
GET /tenant/mount/info
Authorization: Bearer {jwt_token}
```

Response:
```json
{
    "tenant_id": "434252dd-d83b-4b20-bf70-8a950ff875c4",
    "bucket_name": "basic-memory-434252dd",
    "has_credentials": true,
    "credentials_created_at": "2025-09-22T16:48:50.414694"
}
```

**Endpoint 2: POST /tenant/mount/credentials**
- Purpose: Generate NEW bucket-scoped S3 credentials for rclone mounting
- Authentication: JWT token (tenant_id extracted from claims)
- Multi-session: Creates new credentials without revoking existing ones

Request:
```
POST /tenant/mount/credentials
Authorization: Bearer {jwt_token}
Content-Type: application/json
```
*Note: No request body needed - tenant_id extracted from JWT*

Response:
```json
{
    "tenant_id": "434252dd-d83b-4b20-bf70-8a950ff875c4",
    "bucket_name": "basic-memory-434252dd",
    "access_key": "test_access_key_12345",
    "secret_key": "test_secret_key_abcdef",
    "endpoint_url": "https://fly.storage.tigris.dev",
    "region": "auto"
}
```

**🔒 Security Notes:**
- Secret key returned ONCE only - never stored in database
- Credentials are bucket-scoped (cannot access other tenants' buckets)
- Multiple active credentials supported per tenant (work laptop + personal machine)

Implementation Notes

Security:
- Both endpoints require JWT authentication
- Extract tenant_id from JWT claims (not request body)
- Generate scoped credentials (not admin credentials)
- Credentials should have bucket-specific access only

Integration Points:
- Use your existing StorageClient from SPEC-8 implementation
- Leverage existing JWT middleware for tenant extraction
- Return same credential format as your Tigris bucket provisioning

Error Handling:
- 401 if not authenticated
- 403 if tenant doesn't exist
- 500 if credential generation fails

**🔄 Design Decisions:**

1. **Secure Credential Flow (No Secret Storage)**

Based on CLI flow analysis, we follow security best practices:
- ✅ API generates both access_key + secret_key via Tigris IAM
- ✅ Returns both in API response for immediate use
- ✅ CLI uses credentials immediately to configure rclone
- ✅ Database stores only metadata (access_key + policy_arn for cleanup)
- ✅ rclone handles secure local credential storage
- ❌ **Never store secret_key in database (even encrypted)**

2. **CLI Credential Flow**
```bash
# CLI calls API
POST /tenant/mount/credentials → {access_key, secret_key, ...}

# CLI immediately configures rclone
rclone config create basic-memory-{tenant_id} s3 \
  access_key_id={access_key} \
  secret_access_key={secret_key} \
  endpoint=https://fly.storage.tigris.dev

# Database tracks metadata only
INSERT INTO tenant_mount_credentials (tenant_id, access_key, policy_arn, ...)
```

3. **Multiple Sessions Supported**

- Users can have multiple active credential sets (work laptop, personal machine, etc.)
- Each credential generation creates a new Tigris access key
- List active credentials via API (shows access_key but never secret)

4. **Failure Handling & Cleanup**

- **Happy Path**: Credentials created → Used immediately → rclone configured
- **Orphaned Credentials**: Background job revokes unused credentials
- **API Failure Recovery**: Retry Tigris deletion with stored policy_arn
- **Status Tracking**: Track tigris_deletion_status (pending/completed/failed)

5. **Event Sourcing & Audit**

- MountCredentialCreatedEvent
- MountCredentialRevokedEvent
- MountCredentialOrphanedEvent (for cleanup)
- Full audit trail for security compliance

6. **Tenant/Bucket Validation**

- Verify tenant exists and has valid bucket before credential generation
- Use existing StorageClient to validate bucket access
- Prevent credential generation for inactive/invalid tenants

📋 **Implemented API Endpoints:**

```
✅ IMPLEMENTED:
GET    /tenant/mount/info                    # Get tenant/bucket info (no credentials exposed)
POST   /tenant/mount/credentials             # Generate new credentials (returns secret once)
GET    /tenant/mount/credentials             # List active credentials (no secrets)
DELETE /tenant/mount/credentials/{cred_id}   # Revoke specific credentials
```

**API Implementation Status:**
- ✅ **GET /tenant/mount/info**: Returns tenant_id, bucket_name, has_credentials, credentials_created_at
- ✅ **POST /tenant/mount/credentials**: Creates new bucket-scoped access keys, returns access_key + secret_key once
- ✅ **GET /tenant/mount/credentials**: Lists active credentials without exposing secret keys
- ✅ **DELETE /tenant/mount/credentials/{cred_id}**: Revokes specific credentials with proper Tigris IAM cleanup
- ✅ **Multi-session support**: Multiple active credentials per tenant (work laptop + personal machine)
- ✅ **Security**: Secret keys never stored in database, returned once only for immediate use
- ✅ **Comprehensive test suite**: 28 tests covering all scenarios including error handling and multi-session flows
- ✅ **Dependency injection**: Clean integration with existing FastAPI architecture
- ✅ **Production-ready configuration**: Tigris credentials properly configured for tenant machines

🗄️ **Secure Database Schema:**

```sql
CREATE TABLE tenant_mount_credentials (
  id UUID PRIMARY KEY,
  tenant_id UUID REFERENCES tenant(id),
  access_key VARCHAR(255) NOT NULL,
  -- secret_key REMOVED - never store secrets (security best practice)
  policy_arn VARCHAR(255) NOT NULL,           -- For Tigris IAM cleanup
  tigris_deletion_status VARCHAR(20) DEFAULT 'pending',  -- Track cleanup
  created_at TIMESTAMP DEFAULT NOW(),
  updated_at TIMESTAMP DEFAULT NOW(),
  revoked_at TIMESTAMP NULL,
  last_used_at TIMESTAMP NULL,               -- Track usage for orphan cleanup
  description VARCHAR(255) DEFAULT 'CLI mount credentials'
);
```

**Security Benefits:**
- ✅ Database breach cannot expose secrets
- ✅ Follows "secrets don't persist" security principle
- ✅ Meets compliance requirements (SOC2, etc.)
- ✅ Reduced attack surface
- ✅ CLI gets credentials once and stores securely via rclone