Entity Memory System

Overview

Entity memory organizes agent knowledge around entities: people, places, organizations, projects, and concepts. Rather than storing isolated facts, this architecture builds a knowledge graph of entities and their relationships, enabling rich contextual understanding.

Entity Types

People

Information about individuals:

Name and aliases

Relationship to user (friend, colleague, family)

Contact information

Preferences and interests

Interaction history

Key facts learned

Organizations

Companies, groups, institutions:

Name and type

User's relationship (employer, client, etc.)

Key people associated

Relevant context

Past interactions

Projects

Work or personal endeavors:

Name and description

Status (active, completed, planned)

Associated people

Key milestones

Related documents/resources

Concepts

Abstract topics and themes:

Definition in user's context

Related entities

User's interest level

Knowledge depth

Entity Schema

Entity:

├── id: unique identifier

├── type: person | organization | project | concept

├── name: primary name

├── aliases: alternative names

├── attributes: key-value facts

│ ├── [attribute_name]: value

│ └── ...

├── relationships: connections to other entities

│ ├── [relationship_type]: entity_id[]

│ └── ...

├── mentions: interaction history

│ ├── first_mentioned: timestamp

│ ├── last_mentioned: timestamp

│ ├── mention_count: number

│ └── contexts: session_id[]

├── embedding: vector for semantic search

└── updated_at: timestamp

Architecture

┌─────────────────────────────────────────────────────────────┐

│ Conversation │

│ "I had lunch with Sarah from Acme Corp about the Q4 deal" │

└──────────────────────────┬──────────────────────────────────┘

│

▼

┌─────────────────────────────────────────────────────────────┐

│ Entity Extraction │

│ ├── Sarah (person) │

│ ├── Acme Corp (organization) │

│ └── Q4 deal (project) │

└──────────────────────────┬──────────────────────────────────┘

│

▼

┌─────────────────────────────────────────────────────────────┐

│ Entity Resolution │

│ ├── Sarah → existing entity #123 (Sarah Chen) │

│ ├── Acme Corp → existing entity #456 │

│ └── Q4 deal → new entity created │

└──────────────────────────┬──────────────────────────────────┘

│

▼

┌─────────────────────────────────────────────────────────────┐

│ Entity Update │

│ ├── Sarah: add relationship to Acme Corp │

│ ├── Sarah: update last_mentioned │

│ ├── Acme Corp: add relationship to Q4 deal │

│ └── Q4 deal: create with Sarah, Acme relationships │

└──────────────────────────┬──────────────────────────────────┘

│

▼

┌─────────────────────────────────────────────────────────────┐

│ Entity Store │

│ ┌─────────┐ ┌─────────┐ ┌─────────┐ │

│ │ Sarah │────▶│ Acme │────▶│ Q4 Deal │ │

│ │ (person)│ │ Corp │ │(project)│ │

│ └─────────┘ └─────────┘ └─────────┘ │

└─────────────────────────────────────────────────────────────┘

Entity Extraction

From Conversations

Use NLP to identify entities:

Input: "Meeting with John about the website redesign tomorrow"

Extraction:

John (person)

website redesign (project)

tomorrow (temporal, not stored as entity)

Extraction Prompt

Extract entities from this text:

[TEXT]

For each entity, provide:

name: the entity name

type: person | organization | project | concept | place

attributes: any facts about it

relationships: connections to other entities in the text

Return as JSON array.

Entity Resolution

The Challenge

Same entity, different references:

"Sarah", "Sarah Chen", "my colleague Sarah"

"the company", "Acme", "Acme Corp"

"that project", "website redesign", "the Q4 initiative"

Resolution Strategy

**Exact match**: Same name string

**Alias match**: Check known aliases

**Semantic match**: Embedding similarity

**Context match**: Same relationships/attributes

5. **Recency match**: Recently mentioned similar entity

Disambiguation

When uncertain:

Store as potential match

Ask user for confirmation

Use context to decide

Learn from corrections

Relationship Types

Person-to-Person

family_of

friend_of

colleague_of

reports_to

manages

Person-to-Organization

works_at

client_of

founder_of

member_of

Entity-to-Project

works_on

owns

stakeholder_in

created

General

related_to

mentioned_with

part_of

Retrieval Patterns

Entity Lookup

When entity is mentioned:

User: "What do you know about Sarah?"

Lookup:

Find entity by name/alias

Load all attributes

Load relationships

Get related entities (1 hop)

5. Get recent mentions

Contextual Retrieval

For relevant context:

User: "Schedule a meeting about the website"

Retrieval:

Find "website" project entity

Get associated people

Get related projects

Include in context for scheduling

Graph Traversal

For complex queries:

User: "Who at Acme Corp have I worked with?"

Query:

Find Acme Corp entity

Find people with works_at relationship

Filter by has interacted_with user

Return matching people

Storage Options

Document Store + Index

Simple approach:

Store entities as documents

Vector index for semantic search

Filter by type and relationships

Good for smaller graphs

Graph Database

For complex relationships:

Native relationship queries

Efficient traversal

Schema flexibility

Better for large, connected graphs

Hybrid

Best of both:

Graph DB for structure

Vector DB for semantic search

Join results as needed

Maintenance

Merging Duplicates

When duplicates are detected:

Merge attributes (keep all)

Combine relationships

Preserve history

Update references

Pruning Stale Entities

Over time:

Reduce importance of unmentioned entities

Archive or remove abandoned entities

Keep relationship history

Configurable retention