Norns Memory System

Long-term episodic and semantic memory for Norns Agent using pgvector embeddings.

---

Overview

The Norns memory system (Muninn) implements two types of memory:

1. Episodic Memory: Stores specific conversation turns with semantic embeddings for similarity search
2. Semantic Memory: Extracts and stores patterns, preferences, and learned insights

┌─────────────────────────────────────────────────────────────────┐
│                     CONVERSATION FLOW                            │
└─────────────────────────────────────────────────────────────────┘
                              ↓
┌─────────────────────────────────────────────────────────────────┐
│  Memory Layer (muninn_context.py)                               │
│                                                                  │
│  1. Generate embedding (via Ollama nomic-embed-text)            │
│  2. Search similar memories (pgvector cosine similarity)        │
│  3. Store new memory with embedding                             │
└─────────────────────────────────────────────────────────────────┘
                              ↓
┌─────────────────────────────────────────────────────────────────┐
│  PostgreSQL with pgvector                                       │
│                                                                  │
│  • episodic_memories (vector embeddings)                        │
│  • semantic_patterns (extracted insights)                       │
└─────────────────────────────────────────────────────────────────┘

---

Database Schema

Episodic Memories Table

CREATE TABLE episodic_memories (
    id UUID PRIMARY KEY,
    user_id UUID NOT NULL,
    content TEXT NOT NULL,
    embedding vector(768),  -- pgvector type
    created_at TIMESTAMP DEFAULT NOW(),
    metadata JSONB
);

CREATE INDEX ON episodic_memories 
USING ivfflat (embedding vector_cosine_ops) 
WITH (lists = 100);

Semantic Patterns Table

CREATE TABLE semantic_patterns (
    id UUID PRIMARY KEY,
    user_id UUID NOT NULL,
    pattern_type VARCHAR(50),
    pattern_data JSONB,
    embedding vector(768),
    created_at TIMESTAMP DEFAULT NOW()
);

---

Embedding Generation

Embeddings are generated using Ollama's nomic-embed-text model (768 dimensions):

async def generate_embedding(text: str) -> list[float]:
    """Generate embedding vector for text"""
    async with httpx.AsyncClient() as client:
        response = await client.post(
            f"{OLLAMA_URL}/api/embeddings",
            json={
                "model": "nomic-embed-text",
                "prompt": text
            }
        )
        result = response.json()
        return result["embedding"]  # Returns Python list of floats

---

pgvector Integration

CRITICAL: pgvector embeddings must be passed as Python lists, not strings.

Correct Implementation

The pgvector.asyncpg.register_vector() call in main.py enables automatic conversion:

# In main.py startup event
import pgvector.asyncpg

@app.on_event("startup")
async def startup():
    await database.connect()
    
    # Register pgvector type adapter for asyncpg
    conn = await asyncpg.connect(DATABASE_URL)
    await pgvector.asyncpg.register_vector(conn)
    await conn.close()

With this registration, you can pass Python lists directly:

# CORRECT: Pass list directly
embedding = [0.123, 0.456, 0.789, ...]  # Python list
await db.execute(
    "INSERT INTO episodic_memories (id, user_id, content, embedding) VALUES (:id, :user_id, :content, :embedding)",
    {
        "id": memory_id,
        "user_id": user_id,
        "content": text,
        "embedding": embedding  # Python list
    }
)

Previous Incorrect Implementation

Before the fix (2026-01-03), embeddings were incorrectly converted to strings:

# INCORRECT: Don't do this!
embedding_str = '[' + ','.join(map(str, embedding)) + ']'
await db.execute(
    "INSERT INTO episodic_memories (..., embedding) VALUES (..., :embedding)",
    {"embedding": embedding_str}  # String, not list
)

This caused type errors because pgvector expected a vector type, not a string.

---

Memory Search

Similarity search uses pgvector's cosine similarity operator (<=>):

async def search_similar_memories(
    user_id: str, 
    query_embedding: list[float], 
    limit: int = 5
) -> list[dict]:
    """Find similar memories using vector search"""
    
    query = """
    SELECT id, content, created_at,
           1 - (embedding <=> :query_embedding) as similarity
    FROM episodic_memories
    WHERE user_id = :user_id
    ORDER BY embedding <=> :query_embedding
    LIMIT :limit
    """
    
    results = await db.fetch_all(
        query,
        {
            "user_id": user_id,
            "query_embedding": query_embedding,  # Python list
            "limit": limit
        }
    )
    
    return [dict(row) for row in results]

Note: The <=> operator returns distance (lower is better), so we use 1 - distance for similarity score.

---

Memory Storage Flow

User Message
    ↓
Generate embedding (Ollama)
    ↓
Search similar past memories (pgvector)
    ↓
Add context to agent prompt
    ↓
Agent responds
    ↓
Store conversation turn + embedding
    ↓
Extract patterns (if applicable)
    ↓
Store semantic pattern + embedding

---

Files Affected by Embedding Fix

The fix (2026-01-03) removed string conversion in these files:

1. episodic.py: Episodic memory storage

   • Changed: Pass embedding as list, not string
   • Changed: Pass query_embedding as list in search

2. muninn_context.py: Memory context builder

   • Changed: Pass embedding as list in all queries
   • Changed: Removed '[' + ','.join(...) + ']' conversions

3. main.py: Already had pgvector.asyncpg.register_vector()

   • No changes needed - registration was already correct

---

Testing Memory System

# Check memory count
ssh ravenhelm@100.115.101.81 "docker exec -i postgres psql -U ravenhelm -d ravenmaskos -c '
SELECT 
  (SELECT COUNT(*) FROM episodic_memories) as episodic,
  (SELECT COUNT(*) FROM semantic_patterns) as patterns;
'"

# View recent memories
ssh ravenhelm@100.115.101.81 "docker exec -i postgres psql -U ravenhelm -d ravenmaskos -c '
SELECT id, LEFT(content, 50) as content, created_at 
FROM episodic_memories 
ORDER BY created_at DESC 
LIMIT 10;
'"

# Test similarity search (requires embedding)
ssh ravenhelm@100.115.101.81 "docker exec -i postgres psql -U ravenhelm -d ravenmaskos -c '
SELECT LEFT(content, 50) as content, 
       1 - (embedding <=> (SELECT embedding FROM episodic_memories LIMIT 1)) as similarity
FROM episodic_memories
ORDER BY similarity DESC
LIMIT 5;
'"

---

Troubleshooting

TypeError: Cannot pass list to asyncpg

Symptoms:

TypeError: Cannot convert Python list to PostgreSQL type

Diagnosis: Check if pgvector.asyncpg.register_vector() was called during startup.

Solution: Ensure main.py contains:

import pgvector.asyncpg

@app.on_event("startup")
async def startup():
    conn = await asyncpg.connect(DATABASE_URL)
    await pgvector.asyncpg.register_vector(conn)
    await conn.close()

Embedding dimension mismatch

Symptoms:

ERROR: expected 768 dimensions, got 384

Diagnosis: Wrong embedding model used.

Solution: Use nomic-embed-text (768 dims), not all-MiniLM-L6-v2 (384 dims):

ssh ravenhelm@100.115.101.81 "docker exec ollama ollama pull nomic-embed-text"

Slow vector search

Symptoms: Queries take >1 second

Diagnosis: Missing or incorrect index.

Solution: Create IVFFlat index:

CREATE INDEX ON episodic_memories 
USING ivfflat (embedding vector_cosine_ops) 
WITH (lists = 100);

Adjust lists parameter based on data size:

• lists = 100 for <10k rows
• lists = 1000 for 100k-1M rows

---

Memory Retention Policy

Current: No automatic deletion (memories stored indefinitely)

Future considerations:

• Prune old episodic memories (keep last 1000 per user)
• Consolidate into semantic patterns
• Archive cold data to object storage

---

References

• pgvector Documentation
• nomic-embed-text Model
• Vector Similarity Search