M4A

Local-First Agent Memory

Privacy-preserving architecture with on-device memory and optional sync

intermediateprivacylocal-firston-devicesync

Overview

Local-first memory keeps user data on their own devices by default, with optional encrypted sync. This architecture maximizes privacy, works offline, and gives users complete control over their data. Cloud sync is opt-in and always end-to-end encrypted.

Core Principles

Data Sovereignty

Users own their memory:

  • Data stored on user's device
  • No cloud requirement
  • Export anytime in open formats
  • Delete means actually delete

    • Offline-First

    Works without internet:

  • Full functionality offline
  • Sync when connected
  • Graceful degradation
  • No cloud dependency for core features

    • Privacy by Default

    Minimal data exposure:

  • No server-side storage required
  • End-to-end encryption for sync
  • Local processing preferred
  • Explicit consent for any cloud use

    • Architecture Layers

    ┌─────────────────────────────────────────────────────────────┐

    │ User's Device │

    │ │

    │ ┌──────────────────────────────────────────────────────┐ │

    │ │ Application │ │

    │ └────────────────────────┬─────────────────────────────┘ │

    │ │ │

    │ ┌────────────────────────┴─────────────────────────────┐ │

    │ │ Local Memory Store │ │

    │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │

    │ │ │ SQLite │ │ Vector │ │ Index │ │ │

    │ │ │ (Facts) │ │ Store │ │ Cache │ │ │

    │ │ └─────────────┘ └─────────────┘ └─────────────┘ │ │

    │ └────────────────────────┬─────────────────────────────┘ │

    │ │ │

    │ ┌────────────────────────┴─────────────────────────────┐ │

    │ │ Sync Engine (Optional) │ │

    │ │ ├── End-to-end encryption │ │

    │ │ ├── Conflict resolution │ │

    │ │ └── Multi-device coordination │ │

    │ └────────────────────────┬─────────────────────────────┘ │

    └───────────────────────────┼─────────────────────────────────┘

    │ encrypted sync (opt-in)

    ▼

    ┌─────────────────────────────────────────────────────────────┐

    │ Sync Server (Dumb Storage) │

    │ ├── Stores encrypted blobs only │

    │ ├── Cannot read user data │

    │ ├── Facilitates device sync │

    │ └── Optional, user can self-host │

    └─────────────────────────────────────────────────────────────┘

    Local Storage

    On-Device Vector Store

    Lightweight vector search:

  • SQLite with vector extensions
  • Or embedded vector DB (Chroma, LanceDB)
  • Efficient for personal-scale data
  • No network latency

    • Memory Schema

    memories:

    ├── id: local unique id

    ├── content: memory text

    ├── embedding: vector (computed locally)

    ├── created_at: timestamp

    ├── type: fact | preference | conversation

    ├── importance: score

    ├── sync_status: local | synced | conflict

    └── version: for conflict resolution

    Local Processing

    Compute on device when possible:

  • Embedding generation (small models)
  • Memory extraction
  • Search and retrieval
  • Only use cloud AI for generation

    • Sync Design

    Encryption

    All synced data is encrypted:

  • User-held encryption keys
  • Server stores only ciphertext
  • Key derivation from user secret
  • Zero-knowledge server design

    • Conflict Resolution

    When devices diverge:

  • Last-write-wins for simple conflicts
  • Merge for additive changes
  • User resolution for complex conflicts
  • Preserve all versions until resolved

    • Selective Sync

    User controls what syncs:

  • All memories
  • Only conversations
  • Only specific topics
  • Nothing (local only)

    • Privacy Features

    Data Minimization

    Only store what's needed:

  • Automatic expiration options
  • User-controlled retention
  • No analytics on memory content
  • Minimal metadata

    • Transparency

    Users can see everything:

  • Memory browser UI
  • Export in readable formats
  • Audit log of access
  • Clear deletion confirmation

    • Consent

    Explicit permission for everything:

  • Opt-in to sync
  • Opt-in to any cloud features
  • Per-feature consent
  • Easy to revoke

    • Offline Capabilities

    Full Functionality

    Without internet:

  • Memory storage works
  • Retrieval works
  • Agent responds (with local or cached model)
  • Queues sync for later

    • Graceful Degradation

    When connectivity is limited:

  • Prioritize essential syncs
  • Defer large transfers
  • Continue local operation
  • Clear status indication

    • Implementation Approaches

    Web Application

  • IndexedDB for storage
  • Web Crypto API for encryption
  • Service Worker for offline
  • WebAssembly for vector ops

    • Desktop Application

  • SQLite with extensions
  • Native crypto libraries
  • Full offline support
  • System integration

    • Mobile Application

  • Core Data / Room for storage
  • Platform crypto
  • Background sync
  • Battery-aware operations

    • Trade-offs

    Advantages

  • Maximum privacy
  • Works offline
  • User control
  • No server costs for storage

    • Challenges

  • Limited compute on device
  • Sync complexity
  • Storage limits on some devices
  • Backup responsibility on user
    • Back to Architectures