basicmachines-co-basic-memory/specs/SPEC-8 TigrisFS Integration.md

title, Date, Status, Priority, Goal, permalink

title	Date	Status	Priority	Goal	permalink
SPEC-8: TigrisFS Integration for Tenant API	September 22, 2025	Phase 3.6 Complete - Tenant Mount API Endpoints Ready for CLI Implementation	High	Replace Fly volumes with Tigris bucket provisioning in production tenant API	spec-8-tigris-fs-integration

Executive Summary

Based on SPEC-7 Phase 4 POC testing, this spec outlines productizing the TigrisFS/rclone implementation in the Basic Memory Cloud tenant API. We're moving from proof-of-concept to production integration, replacing Fly volume storage with Tigris bucket-per-tenant architecture.

Current Architecture (Fly Volumes)

Tenant Provisioning Flow

# apps/cloud/src/basic_memory_cloud/workflows/tenant_provisioning.py
async def provision_tenant_infrastructure(tenant_id: str):
    # 1. Create Fly app
    # 2. Create Fly volume  ← REPLACE THIS
    # 3. Deploy API container with volume mount
    # 4. Configure health checks

Storage Implementation

• Each tenant gets dedicated Fly volume (1GB-10GB)
• Volume mounted at /app/data in API container
• Local filesystem storage with Basic Memory indexing
• No global caching or edge distribution

Proposed Architecture (Tigris Buckets)

New Tenant Provisioning Flow

async def provision_tenant_infrastructure(tenant_id: str):
    # 1. Create Fly app
    # 2. Create Tigris bucket with admin credentials  ← NEW
    # 3. Store bucket name in tenant record  ← NEW
    # 4. Deploy API container with TigrisFS mount using admin credentials
    # 5. Configure health checks

Storage Implementation

• Each tenant gets dedicated Tigris bucket
• TigrisFS mounts bucket at /app/data in API container
• Global edge caching and distribution
• Configurable cache TTL for sync performance

Implementation Plan

Phase 1: Bucket Provisioning Service

✅ IMPLEMENTED: StorageClient with Admin Credentials

# apps/cloud/src/basic_memory_cloud/clients/storage_client.py
class StorageClient:
    async def create_tenant_bucket(self, tenant_id: UUID) -> TigrisBucketCredentials
    async def delete_tenant_bucket(self, tenant_id: UUID, bucket_name: str) -> bool
    async def list_buckets(self) -> list[TigrisBucketResponse]
    async def test_tenant_credentials(self, credentials: TigrisBucketCredentials) -> bool

Simplified Architecture Using Admin Credentials:

• Single admin access key with full Tigris permissions (configured in console)
• No tenant-specific IAM user creation needed
• Bucket-per-tenant isolation for logical separation
• Admin credentials shared across all tenant operations

Integrate with Provisioning workflow:

# Update tenant_provisioning.py
async def provision_tenant_infrastructure(tenant_id: str):
    storage_client = StorageClient(settings.aws_access_key_id, settings.aws_secret_access_key)
    bucket_creds = await storage_client.create_tenant_bucket(tenant_id)
    await store_bucket_name(tenant_id, bucket_creds.bucket_name)
    await deploy_api_with_tigris(tenant_id, bucket_creds)

Phase 2: Simplified Bucket Management

✅ SIMPLIFIED: Admin Credentials + Bucket Names Only

Since we use admin credentials for all operations, we only need to track bucket names per tenant:

1. Primary Storage (Fly Secrets)

   flyctl secrets set -a basic-memory-{tenant_id} \
     AWS_ACCESS_KEY_ID="{admin_access_key}" \
     AWS_SECRET_ACCESS_KEY="{admin_secret_key}" \
     AWS_ENDPOINT_URL_S3="https://fly.storage.tigris.dev" \
     AWS_REGION="auto" \
     BUCKET_NAME="basic-memory-{tenant_id}"
   

2. Database Storage (Bucket Name Only)

   # apps/cloud/src/basic_memory_cloud/models/tenant.py
   class Tenant(BaseModel):
       # ... existing fields
       tigris_bucket_name: Optional[str] = None  # Just store bucket name
       tigris_region: str = "auto"
       created_at: datetime
   

Benefits of Simplified Approach:

• No credential encryption/decryption needed
• Admin credentials managed centrally in environment
• Only bucket names stored in database (not sensitive)
• Simplified backup/restore scenarios
• Reduced security attack surface

Phase 3: API Container Updates

Update API container configuration:

# apps/api/Dockerfile
# Add TigrisFS installation
RUN curl -L https://github.com/tigrisdata/tigrisfs/releases/latest/download/tigrisfs-linux-amd64 \
    -o /usr/local/bin/tigrisfs && chmod +x /usr/local/bin/tigrisfs

Startup script integration:

# apps/api/tigrisfs-startup.sh (already exists)
# Mount TigrisFS → Start Basic Memory API
exec python -m basic_memory_cloud_api.main

Fly.toml environment (optimized for < 5s startup):

# apps/api/fly.tigris-production.toml
[env]
  TIGRISFS_MEMORY_LIMIT = '1024'     # Reduced for faster init
  TIGRISFS_MAX_FLUSHERS = '16'       # Fewer threads for faster startup
  TIGRISFS_STAT_CACHE_TTL = '30s'    # Balance sync speed vs startup
  TIGRISFS_LAZY_INIT = 'true'        # Enable lazy loading
  BASIC_MEMORY_HOME = '/app/data'

# Suspend optimization for wake-on-network
[machine]
  auto_stop_machines = "suspend"     # Faster than full stop
  auto_start_machines = true
  min_machines_running = 0

Phase 4: Local Access Features

CLI automation for local mounting:

# New CLI command: basic-memory cloud mount
async def setup_local_mount(tenant_id: str):
    # 1. Fetch bucket credentials from cloud API
    # 2. Configure rclone with scoped IAM policy
    # 3. Mount via rclone nfsmount (macOS) or FUSE (Linux)
    # 4. Start Basic Memory sync watcher

Local mount configuration:

# rclone config for tenant
rclone mount basic-memory-{tenant_id}: ~/basic-memory-{tenant_id} \
  --nfs-mount \
  --vfs-cache-mode writes \
  --cache-dir ~/.cache/rclone/basic-memory-{tenant_id}

Phase 5: TigrisFS Cache Sync Solutions

Problem: When files are uploaded via CLI/bisync, the tenant API container doesn't see them immediately due to TigrisFS cache (30s TTL) and lack of inotify events on mounted filesystems.

Multi-Layer Solution:

Layer 1: API Sync Endpoint (Immediate)

# POST /sync - Force TigrisFS cache refresh
# Callable by CLI after uploads
subprocess.run(["sync", "fsync /app/data"], check=True)

Layer 2: Tigris Webhook Integration (Real-time) https://www.tigrisdata.com/docs/buckets/object-notifications/#webhook

# Webhook endpoint for bucket changes
@app.post("/webhooks/tigris/{tenant_id}")
async def handle_bucket_notification(tenant_id: str, event: TigrisEvent):
    if event.eventName in ["OBJECT_CREATED_PUT", "OBJECT_DELETED"]:
        await notify_container_sync(tenant_id, event.object.key)

Layer 3: CLI Sync Notification (User-triggered)

# CLI calls container sync endpoint after successful bisync
basic-memory cloud bisync  # Automatically notifies container
curl -X POST https://basic-memory-{tenant-id}.fly.dev/sync

Layer 4: Periodic Sync Fallback (Safety net)

# Background task: fsync /app/data every 30s as fallback
# Ensures eventual consistency even if other layers fail

Implementation Priority:

1. Layer 1 (API endpoint) - Quick testing capability
2. Layer 3 (CLI integration) - Improved UX
3. Layer 4 (Periodic fallback) - Safety net
4. Layer 2 (Webhooks) - Production real-time sync

Performance Targets

Sync Latency

• Target: < 5 seconds local→cloud→container
• Configuration: TIGRISFS_STAT_CACHE_TTL = '5s'
• Monitoring: Track sync metrics in production

Container Startup

• Target: < 5 seconds including TigrisFS mount
• Fast retry: 0.5s intervals for mount verification
• Fallback: Container fails fast if mount fails

Memory Usage

• TigrisFS cache: 2GB memory limit per container
• Concurrent uploads: 32 flushers max
• VM sizing: shared-cpu-2x (2048mb) minimum

Security Considerations

Bucket Isolation

• Each tenant has dedicated bucket
• IAM policies prevent cross-tenant access
• No shared bucket with subdirectories

Credential Security

• Fly secrets for runtime access
• Encrypted database backup for disaster recovery
• Credential rotation capability

Data Residency

• Tigris global edge caching
• SOC2 Type II compliance
• Encryption at rest and in transit

Operational Benefits

Scalability

• Horizontal scaling with stateless API containers
• Global edge distribution
• Better resource utilization

Reliability

• No cold starts between tenants
• Built-in redundancy and caching
• Simplified backup strategy

Cost Efficiency

• Pay-per-use storage pricing
• Shared infrastructure benefits
• Reduced operational overhead

Risk Mitigation

Data Loss Prevention

• Dual credential storage (Fly + database)
• Automated backup workflows to R2/S3
• Tigris built-in redundancy

Performance Degradation

• Configurable cache settings per tenant
• Monitoring and alerting on sync latency
• Fallback to volume storage if needed

Security Vulnerabilities

• Bucket-per-tenant isolation
• Regular credential rotation
• Security scanning and monitoring

Success Metrics

Technical Metrics

• Sync latency P50 < 5 seconds
• Container startup time < 5 seconds
• Zero data loss incidents
• 99.9% uptime per tenant

Business Metrics

• Reduced infrastructure costs vs volumes
• Improved user experience with faster sync
• Enhanced enterprise security posture
• Simplified operational overhead

Open Questions

1. Tigris rate limits: What are the API limits for bucket creation?
2. Cost analysis: What's the break-even point vs Fly volumes?
3. Regional preferences: Should enterprise customers choose regions?
4. Backup retention: How long to keep automated backups?

Implementation Checklist

Phase 1: Bucket Provisioning Service ✅ COMPLETED

• Research Tigris bucket API - Document bucket creation and S3 API compatibility
• Create StorageClient class - Implemented with admin credentials and comprehensive integration tests
• Test bucket creation - Full test suite validates API integration with real Tigris environment
• Add bucket provisioning to DBOS workflow - Integrated StorageClient with tenant_provisioning.py

Phase 2: Simplified Bucket Management ✅ COMPLETED

• Update Tenant model with tigris_bucket_name field (replaced fly_volume_id)
• Implement bucket name storage - Database migration and model updates completed
• Test bucket provisioning integration - Full test suite validates workflow from tenant creation to bucket assignment
• Remove volume logic from all tests - Complete migration from volume-based to bucket-based architecture

Phase 3: API Container Integration ✅ COMPLETED

• Update Dockerfile to install TigrisFS binary in API container with configurable version
• Optimize tigrisfs-startup.sh with production-ready security and reliability improvements
• Create production-ready container with proper signal handling and mount validation
• Implement security fixes based on Claude code review (conditional debug, credential protection)
• Add proper process supervision with cleanup traps and error handling
• Remove debug artifacts - Cleaned up all debug Dockerfiles and test scripts

Phase 3.5: IAM Access Key Management ✅ COMPLETED

• Research Tigris IAM API - Documented create_policy, attach_user_policy, delete_access_key operations
• Implement bucket-scoped credential generation - StorageClient.create_tenant_access_keys() with IAM policies
• Add comprehensive security test suite - 5 security-focused integration tests covering all attack vectors
• Verify cross-bucket access prevention - Scoped credentials can ONLY access their designated bucket
• Test credential lifecycle management - Create, validate, delete, and revoke access keys
• Validate admin vs scoped credential isolation - Different access patterns and security boundaries
• Test multi-tenant isolation - Multiple tenants cannot access each other's buckets

Phase 3.6: Tenant Mount API Endpoints ✅ COMPLETED

• Implement GET /tenant/mount/info - Returns mount info without exposing credentials
• Implement POST /tenant/mount/credentials - Creates new bucket-scoped credentials for CLI mounting
• Implement DELETE /tenant/mount/credentials/{cred_id} - Revoke specific credentials with proper cleanup
• Implement GET /tenant/mount/credentials - List active credentials without exposing secrets
• Add TenantMountCredentials database model - Tracks credential metadata (no secret storage)
• Create comprehensive test suite - 28 tests covering all scenarios including multi-session support
• Implement multi-session credential flow - Multiple active credentials per tenant supported
• Secure credential handling - Secret keys never stored, returned once only for immediate use
• Add dependency injection for StorageClient - Clean integration with existing API architecture
• Fix Tigris configuration for cloud service - Added AWS environment variables to fly.template.toml
• Update tenant machine configurations - Include AWS credentials for TigrisFS mounting with clear credential strategy

Security Test Results:

✅ Cross-bucket access prevention - PASS
✅ Deleted credentials access revoked - PASS
✅ Invalid credentials rejected - PASS
✅ Admin vs scoped credential isolation - PASS
✅ Multiple scoped credentials isolation - PASS

Implementation Details:

• Uses Tigris IAM managed policies (create_policy + attach_user_policy)
• Bucket-scoped S3 policies with Actions: GetObject, PutObject, DeleteObject, ListBucket
• Resource ARNs limited to specific bucket: arn:aws:s3:::bucket-name and arn:aws:s3:::bucket-name/*
• Access keys follow Tigris format: tid_ prefix with secure random suffix
• Complete cleanup on deletion removes both access keys and associated policies

Phase 4: Local Access CLI

• Design local mount CLI command for automated rclone configuration
• Implement credential fetching from cloud API for local setup
• Create rclone config automation for tenant-specific bucket mounting
• Test local→cloud→container sync with optimized cache settings
• Document local access setup for beta users

Phase 5: Webhook Integration (Future)

• Research Tigris webhook API for object notifications and payload format
• Design webhook endpoint for real-time sync notifications
• Implement notification handling to trigger Basic Memory sync events
• Test webhook delivery and sync latency improvements

Success Metrics

• Container startup < 5 seconds including TigrisFS mount and Basic Memory init
• Sync latency < 5 seconds for local→cloud→container file changes
• Zero data loss during bucket provisioning and credential management
• 100% test coverage for new TigrisBucketService and credential functions
• Beta deployment with internal users validating local-cloud workflow

Implementation Notes

Phase 4.1: Bidirectional Sync with rclone bisync (NEW)

Problem Statement

During testing, we discovered that some applications (particularly Obsidian) don't detect file changes over NFS mounts. Rather than building a custom sync daemon, we can leverage rclone bisync - rclone's built-in bidirectional synchronization feature.

Solution: rclone bisync

Use rclone's proven bidirectional sync instead of custom implementation:

Core Architecture:

# rclone bisync handles all the complexity
rclone bisync ~/basic-memory-{tenant_id} basic-memory-{tenant_id}:{bucket_name} \
  --create-empty-src-dirs \
  --conflict-resolve newer \
  --resilient \
  --check-access

Key Benefits:

• ✅ Battle-tested: Production-proven rclone functionality
• ✅ MIT licensed: Open source with permissive licensing
• ✅ No custom code: Zero maintenance burden for sync logic
• ✅ Built-in safety: max-delete protection, conflict resolution
• ✅ Simple installation: Works with Homebrew rclone (no FUSE needed)
• ✅ File watcher compatible: Works with Obsidian and all applications
• ✅ Offline support: Can work offline and sync when connected

bisync Conflict Resolution Options

Built-in conflict strategies:

--conflict-resolve none     # Keep both files with .conflict suffixes (safest)
--conflict-resolve newer    # Always pick the most recently modified file
--conflict-resolve larger   # Choose based on file size
--conflict-resolve path1    # Always prefer local changes
--conflict-resolve path2    # Always prefer cloud changes

Sync Profiles Using bisync

Profile configurations:

BISYNC_PROFILES = {
    "safe": {
        "conflict_resolve": "none",      # Keep both versions
        "max_delete": 10,                # Prevent mass deletion
        "check_access": True,            # Verify sync integrity
        "description": "Safe mode with conflict preservation"
    },
    "balanced": {
        "conflict_resolve": "newer",     # Auto-resolve to newer file
        "max_delete": 25,
        "check_access": True,
        "description": "Balanced mode (recommended default)"
    },
    "fast": {
        "conflict_resolve": "newer",
        "max_delete": 50,
        "check_access": False,           # Skip verification for speed
        "description": "Fast mode for rapid iteration"
    }
}

CLI Commands

Manual sync commands:

basic-memory cloud bisync                    # Manual bidirectional sync
basic-memory cloud bisync --dry-run          # Preview changes
basic-memory cloud bisync --profile safe     # Use specific profile
basic-memory cloud bisync --resync           # Force full baseline resync

Watch mode (Step 1):

basic-memory cloud bisync --watch            # Long-running process, sync every 60s
basic-memory cloud bisync --watch --interval 30s  # Custom interval

System integration (Step 2 - Future):

basic-memory cloud bisync-service install    # Install as system service
basic-memory cloud bisync-service start      # Start background service
basic-memory cloud bisync-service status     # Check service status

Implementation Strategy

Phase 4.1.1: Core bisync Implementation

• Implement run_bisync() function wrapping rclone bisync
• Add profile-based configuration (safe/balanced/fast)
• Create conflict resolution and safety options
• Test with sample files and conflict scenarios

Phase 4.1.2: Watch Mode

• Add --watch flag for continuous sync
• Implement configurable sync intervals
• Add graceful shutdown and signal handling
• Create status monitoring and progress indicators

Phase 4.1.3: User Experience

• Add conflict reporting and resolution guidance
• Implement dry-run preview functionality
• Create troubleshooting and diagnostic commands
• Add filtering configuration (.gitignore-style)

Phase 4.1.4: System Integration (Future)

• Generate platform-specific service files (launchd/systemd)
• Add service management commands
• Implement automatic startup and recovery
• Create monitoring and logging integration

Technical Implementation

Core bisync wrapper:

def run_bisync(
    tenant_id: str,
    bucket_name: str,
    profile: str = "balanced",
    dry_run: bool = False
) -> bool:
    """Run rclone bisync with specified profile."""

    local_path = Path.home() / f"basic-memory-{tenant_id}"
    remote_path = f"basic-memory-{tenant_id}:{bucket_name}"
    profile_config = BISYNC_PROFILES[profile]

    cmd = [
        "rclone", "bisync",
        str(local_path), remote_path,
        "--create-empty-src-dirs",
        "--resilient",
        f"--conflict-resolve={profile_config['conflict_resolve']}",
        f"--max-delete={profile_config['max_delete']}",
        "--filters-file", "~/.basic-memory/bisync-filters.txt"
    ]

    if profile_config.get("check_access"):
        cmd.append("--check-access")

    if dry_run:
        cmd.append("--dry-run")

    return subprocess.run(cmd, check=True).returncode == 0

Default filter file (~/.basic-memory/bisync-filters.txt):

- .DS_Store
- .git/**
- __pycache__/**
- *.pyc
- .pytest_cache/**
- node_modules/**
- .conflict-*
- Thumbs.db
- desktop.ini

Advantages Over Custom Daemon:

• ✅ Zero maintenance: No custom sync logic to debug/maintain
• ✅ Production proven: Used by thousands in production
• ✅ Safety features: Built-in max-delete, conflict handling, recovery
• ✅ Filtering: Advanced exclude patterns and rules
• ✅ Performance: Optimized for various storage backends
• ✅ Community support: Extensive documentation and community

Phase 4.2: NFS Mount Support (Direct Access)

Solution: rclone nfsmount

Keep the existing NFS mount functionality for users who prefer direct file access:

Core Architecture:

# rclone nfsmount provides transparent file access
rclone nfsmount basic-memory-{tenant_id}:{bucket_name} ~/basic-memory-{tenant_id} \
  --vfs-cache-mode writes \
  --dir-cache-time 10s \
  --daemon

Key Benefits:

• ✅ Real-time access: Files appear immediately as they're created/modified
• ✅ Transparent: Works with any application that reads/writes files
• ✅ Low latency: Direct access without sync delays
• ✅ Simple: No periodic sync commands needed
• ✅ Homebrew compatible: Works with Homebrew rclone (no FUSE required)

Limitations:

• ❌ File watcher compatibility: Some apps (Obsidian) don't detect changes over NFS
• ❌ Network dependency: Requires active connection to cloud storage
• ❌ Potential conflicts: Simultaneous edits from multiple locations can cause issues

Mount Profiles (Existing)

Already implemented profiles from SPEC-7 testing:

MOUNT_PROFILES = {
    "fast": {
        "cache_time": "5s",
        "poll_interval": "3s",
        "description": "Ultra-fast development (5s sync)"
    },
    "balanced": {
        "cache_time": "10s",
        "poll_interval": "5s",
        "description": "Fast development (10-15s sync, recommended)"
    },
    "safe": {
        "cache_time": "15s",
        "poll_interval": "10s",
        "description": "Conflict-aware mount with backup",
        "extra_args": ["--conflict-suffix", ".conflict-{DateTimeExt}"]
    }
}

CLI Commands (Existing)

Mount commands already implemented:

basic-memory cloud mount                     # Mount with balanced profile
basic-memory cloud mount --profile fast     # Ultra-fast caching
basic-memory cloud mount --profile safe     # Conflict detection
basic-memory cloud unmount                  # Clean unmount
basic-memory cloud mount-status             # Show mount status

User Choice: Mount vs Bisync

When to Use Each Approach

Use Case	Recommended Solution	Why
Obsidian users	bisync	File watcher support for live preview
CLI/vim/emacs users	mount	Direct file access, lower latency
Offline work	bisync	Can work offline, sync when connected
Real-time collaboration	mount	Immediate visibility of changes
Multiple machines	bisync	Better conflict handling
Single machine	mount	Simpler, more transparent
Development work	Either	Both work well, user preference
Large files	mount	Streaming access vs full download

Installation Simplicity

Both approaches now use simple Homebrew installation:

# Single installation command for both approaches
brew install rclone

# No macFUSE, no system modifications needed
# Works immediately with both mount and bisync

Implementation Status

Phase 4.1: bisync (NEW)

• Implement bisync command wrapper
• Add watch mode with configurable intervals
• Create conflict resolution workflows
• Add filtering and safety options

Phase 4.2: mount (EXISTING - ✅ IMPLEMENTED)

• NFS mount commands with profile support
• Mount management and cleanup
• Process monitoring and health checks
• Credential integration with cloud API

Both approaches share:

• Credential management via cloud API
• Secure rclone configuration
• Tenant isolation and bucket scoping
• Simple Homebrew rclone installation

Key Features:

1. Cross-Platform rclone Installation (rclone_installer.py):

• macOS: Homebrew → official script fallback
• Linux: snap → apt → official script fallback
• Windows: winget → chocolatey → scoop fallback
• Automatic version detection and verification

2. Smart rclone Configuration (rclone_config.py):

• Automatic tenant-specific config generation
• Three optimized mount profiles from your SPEC-7 testing:
• fast: 5s sync (ultra-performance)
• balanced: 10-15s sync (recommended default)
• safe: 15s sync + conflict detection
• Backup existing configs before modification

3. Robust Mount Management (mount_commands.py):

• Automatic tenant credential generation
• Mount path management (~/basic-memory-{tenant-id})
• Process lifecycle management (prevent duplicate mounts)
• Orphaned process cleanup
• Mount verification and health checking

4. Clean Architecture (api_client.py):

• Separated API client to avoid circular imports
• Reuses existing authentication infrastructure
• Consistent error handling and logging

User Experience:

One-Command Setup: basic-memory cloud setup

# 1. Installs rclone automatically
# 2. Authenticates with existing login
# 3. Generates secure credentials  
# 4. Configures rclone
# 5. Performs initial mount

Profile-Based Mounting: basic-memory cloud mount --profile fast # 5s sync basic-memory cloud mount --profile balanced # 15s sync (default) basic-memory cloud mount --profile safe # conflict detection

Status Monitoring: basic-memory cloud mount-status

# Shows: tenant info, mount path, sync profile, rclone processes

local mount api

Endpoint 1: Get Tenant Info for user Purpose: Get tenant details for mounting

• pass in jwt
• service returns mount info

✅ IMPLEMENTED API Specification:

Endpoint 1: GET /tenant/mount/info

• Purpose: Get tenant mount information without exposing credentials
• Authentication: JWT token (tenant_id extracted from claims)

Request:

GET /tenant/mount/info
Authorization: Bearer {jwt_token}

Response:

{
    "tenant_id": "434252dd-d83b-4b20-bf70-8a950ff875c4",
    "bucket_name": "basic-memory-434252dd",
    "has_credentials": true,
    "credentials_created_at": "2025-09-22T16:48:50.414694"
}

Endpoint 2: POST /tenant/mount/credentials

• Purpose: Generate NEW bucket-scoped S3 credentials for rclone mounting
• Authentication: JWT token (tenant_id extracted from claims)
• Multi-session: Creates new credentials without revoking existing ones

Request:

POST /tenant/mount/credentials
Authorization: Bearer {jwt_token}
Content-Type: application/json

Note: No request body needed - tenant_id extracted from JWT

Response:

{
    "tenant_id": "434252dd-d83b-4b20-bf70-8a950ff875c4",
    "bucket_name": "basic-memory-434252dd",
    "access_key": "test_access_key_12345",
    "secret_key": "test_secret_key_abcdef",
    "endpoint_url": "https://fly.storage.tigris.dev",
    "region": "auto"
}

🔒 Security Notes:

• Secret key returned ONCE only - never stored in database
• Credentials are bucket-scoped (cannot access other tenants' buckets)
• Multiple active credentials supported per tenant (work laptop + personal machine)

Implementation Notes

Security:

• Both endpoints require JWT authentication
• Extract tenant_id from JWT claims (not request body)
• Generate scoped credentials (not admin credentials)
• Credentials should have bucket-specific access only

Integration Points:

• Use your existing StorageClient from SPEC-8 implementation
• Leverage existing JWT middleware for tenant extraction
• Return same credential format as your Tigris bucket provisioning

Error Handling:

• 401 if not authenticated
• 403 if tenant doesn't exist
• 500 if credential generation fails

🔄 Design Decisions:

1. Secure Credential Flow (No Secret Storage)

Based on CLI flow analysis, we follow security best practices:

• ✅ API generates both access_key + secret_key via Tigris IAM
• ✅ Returns both in API response for immediate use
• ✅ CLI uses credentials immediately to configure rclone
• ✅ Database stores only metadata (access_key + policy_arn for cleanup)
• ✅ rclone handles secure local credential storage
• ❌ Never store secret_key in database (even encrypted)

2. CLI Credential Flow

# CLI calls API
POST /tenant/mount/credentials → {access_key, secret_key, ...}

# CLI immediately configures rclone
rclone config create basic-memory-{tenant_id} s3 \
  access_key_id={access_key} \
  secret_access_key={secret_key} \
  endpoint=https://fly.storage.tigris.dev

# Database tracks metadata only
INSERT INTO tenant_mount_credentials (tenant_id, access_key, policy_arn, ...)

3. Multiple Sessions Supported

• Users can have multiple active credential sets (work laptop, personal machine, etc.)
• Each credential generation creates a new Tigris access key
• List active credentials via API (shows access_key but never secret)

4. Failure Handling & Cleanup

• Happy Path: Credentials created → Used immediately → rclone configured
• Orphaned Credentials: Background job revokes unused credentials
• API Failure Recovery: Retry Tigris deletion with stored policy_arn
• Status Tracking: Track tigris_deletion_status (pending/completed/failed)

5. Event Sourcing & Audit

• MountCredentialCreatedEvent
• MountCredentialRevokedEvent
• MountCredentialOrphanedEvent (for cleanup)
• Full audit trail for security compliance

6. Tenant/Bucket Validation

• Verify tenant exists and has valid bucket before credential generation
• Use existing StorageClient to validate bucket access
• Prevent credential generation for inactive/invalid tenants

📋 Implemented API Endpoints:

✅ IMPLEMENTED:
GET    /tenant/mount/info                    # Get tenant/bucket info (no credentials exposed)
POST   /tenant/mount/credentials             # Generate new credentials (returns secret once)
GET    /tenant/mount/credentials             # List active credentials (no secrets)
DELETE /tenant/mount/credentials/{cred_id}   # Revoke specific credentials

API Implementation Status:

• ✅ GET /tenant/mount/info: Returns tenant_id, bucket_name, has_credentials, credentials_created_at
• ✅ POST /tenant/mount/credentials: Creates new bucket-scoped access keys, returns access_key + secret_key once
• ✅ GET /tenant/mount/credentials: Lists active credentials without exposing secret keys
• ✅ DELETE /tenant/mount/credentials/{cred_id}: Revokes specific credentials with proper Tigris IAM cleanup
• ✅ Multi-session support: Multiple active credentials per tenant (work laptop + personal machine)
• ✅ Security: Secret keys never stored in database, returned once only for immediate use
• ✅ Comprehensive test suite: 28 tests covering all scenarios including error handling and multi-session flows
• ✅ Dependency injection: Clean integration with existing FastAPI architecture
• ✅ Production-ready configuration: Tigris credentials properly configured for tenant machines

🗄️ Secure Database Schema:

CREATE TABLE tenant_mount_credentials (
  id UUID PRIMARY KEY,
  tenant_id UUID REFERENCES tenant(id),
  access_key VARCHAR(255) NOT NULL,
  -- secret_key REMOVED - never store secrets (security best practice)
  policy_arn VARCHAR(255) NOT NULL,           -- For Tigris IAM cleanup
  tigris_deletion_status VARCHAR(20) DEFAULT 'pending',  -- Track cleanup
  created_at TIMESTAMP DEFAULT NOW(),
  updated_at TIMESTAMP DEFAULT NOW(),
  revoked_at TIMESTAMP NULL,
  last_used_at TIMESTAMP NULL,               -- Track usage for orphan cleanup
  description VARCHAR(255) DEFAULT 'CLI mount credentials'
);

Security Benefits:

• ✅ Database breach cannot expose secrets
• ✅ Follows "secrets don't persist" security principle
• ✅ Meets compliance requirements (SOC2, etc.)
• ✅ Reduced attack surface
• ✅ CLI gets credentials once and stores securely via rclone