Life Archive is a full RAG (Retrieval-Augmented Generation) pipeline that indexes ~278K personal records spanning decades — Evernote notes, emails, magazine archives, Tana nodes, and Paperless-NGX documents — into a searchable knowledge base. It runs entirely on the Mac Studio using local embeddings (gte-Qwen2-7B on Apple MPS) and LanceDB for vector storage.

What it answers: “What did I write about X?”, “When did I meet Y?”, “What happened during Z trip?” — any question against a lifetime of personal documents.

Key paths:

Data flow:

1. Source extraction — Raw documents parsed from Evernote exports, email archives, magazine PDFs, Tana JSON, Paperless-NGX API
2. Enrichment — Text cleaning, section splitting, paragraph chunking, QA pair generation
3. Embedding — gte-Qwen2-7B encodes text into dense vectors (local, MPS-accelerated)
4. Storage — LanceDB tables for docs, sections, paragraphs, QA pairs; SQLite for knowledge graph
5. Query — Multi-strategy retrieval with fusion and reranking

Retrieval strategies (all run in parallel per query):

Results from all strategies are fused via Reciprocal Rank Fusion (RRF), then reranked with a cross-encoder model for final ordering.

Four persistent services on Mac Studio, all managed via launchd:

All launchd plists are in ~/Library/LaunchAgents/.

API endpoints (port 8900):

MCP endpoint (port 8901): http://192.168.8.180:8901/mcp — Streamable HTTP transport for Claude Desktop, Claude Code, or any MCP client.

Remote access:

LanceDB (as of 2026-03-12):

Knowledge Graph:

Entity types: person (92,377) · org (85,519) · thing (52,346) · location (46,106)

Source breakdown:

The Life Archive is also available as MCP tools inside Claude Code and Cowork, enabling natural-language queries without the HTTP API.

Two transport modes:

Remote MCP client config (Claude Desktop / Claude Code):

"mcpServers": {
    "life-archive": {
        "url": "http://100.96.128.20:8901/mcp"
    }
}

All scripts live in ~/Sync/ED/life_archive/:

Check service status:

launchctl list | grep beedifferent

Restart embed server:

launchctl kickstart -k gui/$(id -u)/com.beedifferent.embed-server

Restart Life Archive API:

launchctl kickstart -k gui/$(id -u)/com.beedifferent.life-archive-api

Test API health:

curl http://localhost:8900/health

Run a search via API:

curl -X POST http://localhost:8900/search \
  -H "Content-Type: application/json" \
  -d '{"query": "beekeeping notes from 2023"}'

View logs:

tail -f ~/Sync/ED/life_archive/http_api.stdout.log
tail -f ~/Sync/ED/life_archive/http_api.stderr.log

Load new data into LanceDB:

cd ~/Sync/ED/life_archive
.venv/bin/python load_lancedb.py --source <source_name>

Rebuild knowledge graph:

cd ~/Sync/ED/life_archive
.venv/bin/python load_knowledge_graph.py

The knowledge graph is exposed as a live API that any client can query — Claude, Obsidian, Tana, local LLMs, browsers, scripts. Three endpoints provide entity exploration, multi-hop traversal, and search, all with source document links back to the original archive content.

Live endpoints (port 8900):

Web explorer: http://192.168.8.180:1313/kg/ — interactive D3.js force-directed graph backed by the live API.

Example: Explore an entity

curl -X POST http://192.168.8.180:8900/graph/explore \
  -H "Content-Type: application/json" \
  -d '{"entity": "thomas brown", "max_connections": 20, "max_sources": 5}'

Returns: entity info, all connections with relationship labels, source documents with titles and summaries, total document count.

Example: Traverse the graph (2 hops from Colorado)

curl -X POST http://192.168.8.180:8900/graph/traverse \
  -H "Content-Type: application/json" \
  -d '{"entity": "colorado", "depth": 2, "max_per_hop": 15}'

Returns: full subgraph of nodes and edges reachable within N hops. Each node tagged with hop distance from root.

Example: Search entities

curl -X POST http://192.168.8.180:8900/graph/search \
  -H "Content-Type: application/json" \
  -d '{"query": "brown", "entity_type": "person", "limit": 10}'

MCP tools (same functionality): life_archive_graph_explore, life_archive_graph_traverse, life_archive_graph_search — available via stdio and HTTP MCP servers. Any Claude session or MCP-compatible LLM can call these.

Client compatibility:

Key files:

The knowledge graph can be exported to GEXF format for interactive exploration in Gephi or Cosmograph.

Export script: ~/Sync/ED/life_archive/export_kg_gexf.py

Pre-built exports (in ~/Sync/ED/life_archive/exports/):

Color scheme:

Node sizes scale logarithmically by mention count.

Viewing in Gephi:

1. Install: brew install --cask gephi
2. File → Open → choose a .gexf export
3. Layout → ForceAtlas 2 → Run (let settle 30–60 sec) → Stop
4. Appearance → Nodes → Color → Partition → entity_type
5. Statistics → Modularity → Run → then color by modularity class to see communities
6. Use Data Laboratory tab to search/filter entities by name

Viewing in Cosmograph:

1. Go to cosmograph.app
2. Drag and drop the .gexf file
3. WebGL renders instantly — supports the full 276K-node graph

Custom exports:

cd ~/Sync/ED/life_archive

# Only people and orgs
python3 export_kg_gexf.py --types person org

# Entities mentioned 5+ times
python3 export_kg_gexf.py --min-mentions 5

# Top 10,000 by mention count
python3 export_kg_gexf.py --top 10000

Last updated: 2026-03-24

Contextual re-embedding is the most important pending item. All existing embeddings were generated without document-level context prefixed to chunks. New runpod_embed.py adds this (35-50% retrieval improvement). Previous RunPod run (2026-03-17 to 2026-03-21) failed at source 3/7 with OOM. Scripts fixed 2026-03-21 — ready for new pod.

See ~/Sync/ED/TASKS.md for step-by-step next actions.