odysseus/docs/memory-graph-design.md

# Memory Graph View — Design

Status: **approved and partially implemented.** This proposal builds on the findings in `docs/memory-graph-analysis.md`. The user approved this design and requested implementation; Milestone 1 (backend) is complete and committed, Milestone 2 (frontend) is written but uncommitted and not yet visually verified. See `docs/progress.md` for the session log, `docs/todos.md` for the live checklist, and `docs/handoff.md` for the exact next task and a "Resolved open questions" note below. The rest of this document is kept as originally written (the plan), with corrections/decisions layered in inline where implementation diverged from the original proposal.

## Goals / non-goals

**Goals**: an interactive, pannable/zoomable node-link visualization of a user's own memories (Obsidian-graph-like), where nodes are memories and edges represent derived relationships (semantic similarity, shared session, shared category), reachable from the existing "Brain" memory modal, scoped strictly to the owning user.

**Non-goals for v1**: cross-user graphs, graphs spanning memories + documents + sessions in one view (a plausible v2 extension, not required now), real-time multi-tab collaborative editing of the graph, migrating the dormant SQL `memories` table into active use.

## 1. Architecture

The feature is additive and follows the existing "component bag" / manual-wiring conventions already used throughout the codebase — no new architectural pattern is introduced.

```
Browser (static/js/memoryGraph.js, new tab in #memory-modal)
        │  fetch('/api/memory/graph?...')
        ▼
FastAPI route: routes/memory/memory_graph_routes.py
  setup_memory_graph_routes(memory_manager, memory_vector, session_manager)
        │
        ▼
Service: src/memory_graph.py  (new, pure-ish module)
  build_graph(owner, filters) -> {nodes, edges, meta}
        │                              │
        ▼                              ▼
MemoryManager.load(owner=...)   MemoryVectorStore.search(...) per-node kNN via Chroma
(data/memory.json, existing)   (odysseus_memories collection, existing)
```

- **New backend module**: `routes/memory/memory_graph_routes.py`, mounted in `app.py` next to the existing `setup_memory_routes(...)` call, receiving the same already-constructed `memory_manager` and `memory_vector` singletons from `app_initializer.initialize_managers()`. No new singleton, no new startup step.
- **New service module**: `src/memory_graph.py` holds the graph-building logic (node assembly + edge derivation), kept separate from the route file so the edge-derivation logic is unit-testable without FastAPI in the loop (see §10).
- **New frontend module**: `static/js/memoryGraph.js`, imported from `static/app.js` alongside the existing `memory.js` import, following the same default-export-object + optional `window.*` exposure convention.
- **New tab**, not a new modal: added to the existing `.memory-tabs` strip in `index.html` (`Browse | Skills | Add | Settings | Graph`) and registered as a fifth tab panel inside `#memory-modal`. This reuses the modal's existing `Modals.register('memory-modal', ...)` entry, its drag/dock/minimize/z-order behavior, and its existing auth/ownership context — no new modal-manager registration needed. A standalone modal was considered and rejected for v1 (more registration/dock plumbing for no functional benefit; a tab is also the closer analogue to Obsidian's own "Graph view" living alongside the file browser).
- **Graph-rendering library**: recommend vendoring **Cytoscape.js** (single UMD file) into `static/lib/cytoscape.min.js`, following the existing vendoring convention used for `docx.umd.min.js`, `xlsx.full.min.js`, etc. Cytoscape is purpose-built for node-link graphs, ships a canvas renderer (needed for performance past a few hundred nodes — see §5), and bundles pan/zoom/drag interaction out of the box, so no additional libraries (separate zoom/drag/force packages) need to be vendored. D3 (force + zoom + drag composed manually) was considered as an alternative; Cytoscape was chosen because it is a single dependency rather than several composed low-level pieces, and its declarative style/selector API maps naturally onto category-based node styling. **This library choice is a recommendation, not a decision already made — flagging it explicitly for approval before implementation** (see Open Questions).

## 2. Database changes

**None required for v1.** Edges are computed on request from existing data (`data/memory.json` + the `odysseus_memories` Chroma collection); nothing new is persisted. This deliberately avoids two riskier paths identified in the analysis: touching the dormant SQL `memories` table (unused by the runtime today, so "fixing" it is an unrelated, larger migration) and adding a hand-rolled SQLite migration for a brand-new table.

If a later phase adds **explicit user-drawn links** (an Obsidian-style manual connection between two memories, as opposed to a derived one), the proposed change is still additive and still avoids SQL:
- Add an optional `links: []` array field to entries in `data/memory.json`, defaulting to `[]` when absent (`entry.get("links", [])`) so every existing entry remains valid without a migration pass.
- `MemoryManager` already owns read/write of this file; the change is a new optional key, not a schema/format break. Old app versions reading a file with this key simply ignore it (JSON is forward-tolerant), and rollback requires no data cleanup (see §9).

No changes to `core/database.py`, no new Alembic-equivalent `_migrate_add_*` function, no new SQLAlchemy model.

## 3. API design

New endpoints, added to the existing `/api/memory` prefix, following the file's existing conventions (owner-scoped via `get_current_user`/`effective_user`, 404-not-403 on cross-owner id access, `HTTPException` for errors):

### `GET /api/memory/graph`

Returns the full owner-scoped graph (nodes + derived edges), subject to server-side limits (see §5).

Query parameters (all optional):
| Param | Type | Default | Meaning |
|---|---|---|---|
| `category` | string, repeatable | none | Filter nodes to one or more of the existing `MEMORY_CATEGORIES` |
| `min_similarity` | float 0–1 | 0.75 | Minimum cosine similarity to draw a semantic edge |
| `max_edges_per_node` | int | 5 | Cap on nearest-neighbor edges per node (bounds edge count) |
| `include_session_edges` | bool | true | Whether to draw "extracted from the same session" edges |
| `include_category_edges` | bool | false | Whether to draw "same category" edges (off by default — high fan-out) |
| `limit` | int | 1000 | Max nodes returned (see §5 for behavior beyond this) |
| `since` | unix ts | none | Only include memories created/updated after this time (for incremental/lazy loading) |

Response shape:
```json
{
  "nodes": [
    {"id": "mem_123", "text": "...", "category": "preference", "pinned": false,
     "uses": 3, "timestamp": 1769..., "session_id": "sess_abc"}
  ],
  "edges": [
    {"source": "mem_123", "target": "mem_456", "type": "similarity", "weight": 0.86},
    {"source": "mem_123", "target": "mem_789", "type": "session", "weight": 1.0}
  ],
  "meta": {"node_count": 214, "truncated": false, "generated_at": 1769...}
}
```

Auth/ownership: identical pattern to `GET /api/memory` — `require_user(request)` (no `can_manage_memory` needed, matching that read-only memory listing today requires no special privilege), owner resolved via `effective_user(request)` so Bearer-token callers are scoped correctly, and the response is built exclusively from `memory_manager.load(owner=user)` — no id-based lookup path exists on this endpoint, so there's no cross-owner leakage vector to guard beyond the load-time filter.

### `GET /api/memory/graph/{id}/neighbors`

Lazy-expansion endpoint for a single node's immediate neighbors, for progressive loading on large graphs (see §5). Reuses the existing `_verify_memory_owner(id, owner)` 404-on-mismatch helper before returning anything.

### `POST /api/memory/{id}/links` / `DELETE /api/memory/{id}/links/{target_id}` (phase 2, optional)

Manual link create/remove, gated by `require_privilege(request, "can_manage_memory")` exactly like other memory mutations (`PUT`/`DELETE /api/memory/{id}`). Not required for v1; documented here so the API surface is planned coherently rather than bolted on later.

### API token scope

The graph read endpoint is exposed under the **existing** `memory:read` scope already defined in `routes/api_token_routes.py` — no new scope is introduced, consistent with the analysis's note about avoiding further scope proliferation.

## 4. UI design

**Entry point**: new "Graph" tab in the existing `.memory-tabs` strip (`index.html`), alongside Browse/Skills/Add/Settings. Selecting it lazily creates the Cytoscape instance on first open (not on modal load) to avoid any cost for users who never click it.

**Canvas**: fills the tab panel body, resizing with the modal (a `ResizeObserver` on the panel calls `cy.resize()` + `cy.fit()`, since `windowDrag.js`/`tileManager.js` already support the user resizing/tiling the modal itself).

**Node styling** (mirrors the existing Browse-tab visual language rather than inventing a new one):
- Fill color by category, reusing the same category color mapping already defined for the Browse tab's category chips/badges in `style.css`.
- Size scaled by `uses` (more-referenced memories render larger, echoing their real weight in chat context injection).
- Pinned memories get a distinct border/ring, matching the pinned badge already used in the list view.
- Node label: truncated memory text (first ~40 chars), full text on hover tooltip.

**Edge styling**:
- Semantic-similarity edges: solid line, opacity/thickness scaled by `weight` (cosine similarity).
- Same-session edges: dashed line, fixed low opacity, to visually separate "structurally related" from "semantically related" without cluttering the primary read.
- A small legend (reusing the existing modal's compact panel styling) explains the two edge types and lets the user toggle each on/off — mapped directly to the `include_session_edges`/`include_category_edges` query params.

**Interactions**:
- Pan/zoom: native Cytoscape gestures.
- Click node: opens a side panel (or reuses the existing inline-edit affordance from the Browse tab) showing full text, category, pin state, use count, and Pin/Edit/Delete actions — deliberately reusing `memory.js`'s existing edit/pin/delete logic rather than re-implementing it, so behavior (and any future changes to it) stays in one place.
- Double-click node: same inline-edit flow already used in the Browse tab.
- Drag node: repositions it client-side only (not persisted), matching how Obsidian's own graph behaves — dragged position is not sent back to the server.
- Search box (reusing the existing `#memory-search` pattern): highlights/centers matching nodes instead of filtering a list.
- Category filter chips (reusing the existing Browse-tab chip component): toggle node visibility by category.
- Similarity threshold slider: re-queries `min_similarity` and redraws edges (debounced, see §5).
- Click-to-isolate: clicking a node dims everything outside its connected component, a common Obsidian-graph affordance, implemented purely client-side against the already-loaded graph (no extra request).

**Empty/loading/error states**: follow existing modal conventions — spinner (`spinner.js`) while the initial `GET /api/memory/graph` is in flight, an empty-state message reusing the Browse tab's "no memories yet" copy/style when the user has zero memories, and a plain inline error message (no full-modal takeover) on request failure, consistent with how other tabs degrade.

**Accessibility note**: a pure canvas graph is not screen-reader navigable; the existing Browse tab remains the accessible list view of the same data, so the Graph tab is explicitly an additional visualization, not a replacement — nothing about memory access requires the graph to work.

## 5. Performance considerations

- **Server-side edge computation must avoid O(n²) pairwise similarity.** Use Chroma's own ANN query per node (`MemoryVectorStore.search(text, k=max_edges_per_node)`) rather than brute-force pairwise cosine over all memories. This is a set of N approximate-nearest-neighbor queries (already fast, since Chroma is built for this), not N² raw comparisons.
- **Cap response size.** Default `limit=1000` nodes; if an owner has more memories than that, the endpoint returns the most-recent/most-used `limit` nodes plus `meta.truncated=true`, and the frontend shows a "showing N of M — refine filters to see more" notice rather than silently dropping data. The `/graph/{id}/neighbors` endpoint exists precisely so a user can drill into the truncated remainder on demand instead of the server ever needing to return everything at once.
- **Canvas over SVG.** Cytoscape's canvas renderer is required, not optional, once node count exceeds roughly 200–300 — an SVG-per-node approach (as some hand-rolled D3 examples use) degrades badly at that scale by DOM node count alone. This is why Cytoscape (canvas-first) was preferred over a naive D3+SVG approach in §1.
- **Debounce filter changes.** The similarity-threshold slider and category toggles should debounce their re-fetch (e.g. 250ms) rather than firing a request per slider tick.
- **Server-side short-lived cache.** Cache the computed `{nodes, edges}` graph per owner (in-process, e.g. a small dict keyed by `owner` with a TTL of a minute or two, or invalidated eagerly on `memory_added`/`memory_updated`/`memory_deleted` if the optional SSE work in §1 of the analysis is picked up later). This avoids recomputing kNN edges on every tab-open within a short window, at negligible memory cost given the existing single-process assumption already baked into the app.
- **Target scale.** This is a personal/local-assistant memory store, not a general knowledge base — the design explicitly targets smooth interaction for roughly 200–2,000 memories per user, with graceful truncation (not failure) beyond that, rather than engineering for unbounded scale from day one.

## 6. Scalability

- **Multi-user isolation carries no new risk**: the graph endpoint reads through the same `memory_manager.load(owner=user)` owner filter every other memory route already uses, so per-user data volume and per-user isolation both scale exactly as well (or as poorly) as the existing memory list/search endpoints do today.
- **No new infrastructure dependency.** No new database engine, no new container (confirmed in the analysis: ChromaDB is already a separate container the app already depends on). Scaling the graph feature scales exactly with scaling the app's existing single-process/single-SQLite/single-Chroma-container deployment model — this proposal does not change that model or its known limits.
- **Phase-2 scaling lever (only if needed)**: if a user's memory count grows large enough that on-request kNN computation becomes noticeably slow even with caching, the next lever is an incrementally-maintained nearest-neighbor cache (refreshed only for the new/changed memory on each `memory_added`/`memory_updated`, not recomputed from scratch), rather than anything to do with the graph *rendering* — the render side is already the cheap part once node count is capped per §5.

## 7. Security

- **No new privilege boundary.** `GET /api/memory/graph` requires only `require_user`, matching the existing `GET /api/memory` and `/timeline` endpoints — consistent with `THREAT_MODEL.md` explicitly listing memory management as available to non-admins. Mutating endpoints (optional manual links, phase 2) require `can_manage_memory`, matching existing memory-mutation routes.
- **No new cross-owner leakage vector.** The list endpoint has no id-based lookup path (built entirely from an owner-filtered load), and the neighbors/link endpoints reuse the existing `_verify_memory_owner` 404-on-mismatch helper rather than introducing a new ownership check pattern.
- **XSS hygiene in rendering.** Memory text is user-authored/LLM-extracted content rendered into node labels/tooltips — it must go through the same escaping helper already used elsewhere in the frontend (`uiModule.esc()` per the existing markdown-rendering convention) before being placed in any DOM tooltip; Cytoscape's own canvas node labels are not DOM-injected and are safe by construction, but any HTML side-panel showing full memory text on click must still escape it.
- **API token scope reuse, not expansion.** Exposing the read endpoint under the existing `memory:read` scope avoids adding to the token-scope surface, per the analysis's note on the "coarse scopes" gap already tracked in `THREAT_MODEL.md`.
- **Light rate limiting.** Even in a trusted-local-network threat model, add a light per-owner limit (reusing `src/rate_limiter.py`, the same mechanism already used for login) on `GET /api/memory/graph` so a misbehaving client/tab can't hammer Chroma with repeated kNN queries — cheap insurance, not a response to a specific known threat.
- **No prompt-injection surface introduced.** This is a display endpoint; it does not feed memory text back into an LLM prompt, so the existing `untrusted_context_message()` wrapping requirement (which already governs memory text reaching the agent loop) is unaffected and does not need to be duplicated here.

## 8. Migration strategy

No data migration is required for v1 (§2). The rollout is staged purely at the feature-flag / code level:

- **Phase 1 — backend only.** Ship `routes/memory/memory_graph_routes.py` + `src/memory_graph.py`, mounted in `app.py`, with no frontend entry point yet. Fully testable via the existing route-factory test convention (§10) and manual `curl`, with zero user-visible surface — the lowest-risk possible increment.
- **Phase 2 — frontend behind an opt-in setting.** Ship the "Graph" tab gated by a Settings-tab toggle (e.g. "Enable Memory Graph (beta)"), defaulting **off**, so existing users see no change until they opt in. This mirrors how a genuinely new, unproven interaction pattern should be introduced into an app whose UI conventions (flat list, modal-based navigation) this feature is a first departure from.
- **Phase 3 — default-on.** After a soak period with no material bug reports, flip the default to on and drop the "(beta)" label. The toggle itself can remain as a permanent "hide Graph tab" preference for users who simply don't want it, or be removed — a call to make at that time, not now.
- **Phase 4 — optional extensions**, proposed separately and not part of this design's approval scope: manual linking (§2/§3 phase-2 bits), live SSE-based updates (per the analysis's §14 finding that no such channel exists today), and cross-linking memories to the documents/sessions they were extracted from.

Each phase is independently shippable, independently revertable, and does not block on the next phase being designed yet.

## 9. Rollback strategy

- **Backend rollback**: the new route module is additive and isolated — removing its `include_router(...)` call (or the whole file) from `app.py` has zero blast radius on any other route, since it shares only already-existing, already-stable singletons (`memory_manager`, `memory_vector`) and defines no new schema. Rollback is a plain code revert.
- **Frontend rollback**: with the Phase-2 opt-in flag, disabling the flag hides the tab immediately with no deploy needed. A full revert (removing the module, the tab markup, and the flag) is likewise a plain code revert — no other module should come to depend on `memoryGraph.js` internals, which should be verified (grep for imports of it) before merging any phase past Phase 1.
- **No data to roll back.** Because v1 stores nothing new (§2), there is no "undo a migration" step, no backfill to reverse, and no risk of orphaned rows or half-migrated JSON — rollback is strictly a code-level operation at every phase of v1.
- **If the optional `links` JSON field (§2, phase-2 extension) ships later**: rolling back the code that reads/writes it leaves harmless, inert data behind (`links: []` or a populated array), which older `MemoryManager` code already tolerates by construction (unknown/extra JSON keys are never rejected) — no cleanup pass is required unless the user explicitly wants the field purged, which would be a separate, deliberate data-cleanup action, not an automatic part of rollback.

## 10. Testing strategy

- **Backend route tests** (`tests/test_memory_graph_routes.py`, new): follow the repo's established convention exactly — build the router via `setup_memory_graph_routes(...)` directly (no `TestClient`/ASGI app), look up the endpoint function off `router.routes` by path, call it directly with a hand-built `Request` stand-in, and bypass auth via `monkeypatch.setattr(memory_graph_routes, "get_current_user", ...)` / `require_user`. Required cases:
  - Owner isolation: a graph built for owner A contains none of owner B's memories (mirrors `tests/test_memory_owner_isolation.py`).
  - `min_similarity`/`max_edges_per_node`/`category` filters change the returned edge/node sets as expected.
  - `limit`/`meta.truncated` behavior when an owner has more memories than the limit.
  - `/graph/{id}/neighbors` 404s on a foreign-owned id (reusing the existing `_verify_memory_owner` test pattern).
- **Edge-derivation unit tests** (`tests/test_memory_graph_edges.py`, new): the similarity/session/category edge-building logic in `src/memory_graph.py` should be a pure function taking pre-fetched memory + neighbor data and returning edge lists, so it's table-driven-testable without Chroma or FastAPI in the loop — pass in synthetic embeddings/neighbor lists and assert threshold and top-k truncation behavior deterministically.
- **Frontend pure-logic tests**: any pure-JS piece of `memoryGraph.js` that isn't DOM-dependent (mapping an API response into Cytoscape `elements`, category→color lookup, client-side connected-component isolation on click) gets a `.test.mjs` file run via `node --test`, wrapped by a thin pytest shim — following the exact precedent of `tests/test_streaming_segmenter_js.py` / `tests/streaming/invariant.test.mjs`.
- **Manual/DOM verification**: since the repo has no DOM/browser test harness (§17 of the analysis), pan/zoom/drag/click interactions and visual correctness (category colors, edge styling, legend, resize behavior) are verified manually against the running dev server before merge, and this should be stated explicitly in the PR description rather than implied — consistent with how the repo already documents this limitation for other rendering-heavy modules.
- **Performance smoke test**: seed a temporary `MemoryManager` with a few hundred and a few thousand synthetic entries and assert `GET /api/memory/graph` completes within a defined latency budget with a warm cache — a regression guard against an accidental reintroduction of O(n²) pairwise comparison.
- **Marker tagging**: new tests get the existing `area_routes`/`area_services` markers per `tests/_taxonomy.py` so they're picked up by the same `-m area_routes` selective-run convention the rest of the suite already uses.

## Open questions — resolved

The four questions originally posed here have since been answered by explicit user instruction (see `docs/progress.md`, Session 1 → Session 2 transition). Recorded here for anyone re-reading this doc without that context:

1. **Graph library**: **Cytoscape.js**, as recommended. Vendored at `static/lib/cytoscape.min.js` (3.34.0, MIT), fetched via `npm pack cytoscape@3` rather than a hand-typed CDN URL. D3 and a from-scratch renderer were not pursued.
2. **Tab-in-modal vs. standalone modal**: **standalone.** The user explicitly asked for "a new Memory Graph navigation item" and "a dedicated Graph page" — this superseded the original tab-in-Brain-modal recommendation. What's built: a new `#tool-memory-graph-btn` in the sidebar Tools section, opening its own `memory-graph-modal` (built and registered exactly like `calendar.js`'s modal — see `docs/handoff.md` for why that template was chosen over the Brain modal's own pattern).
3. **Manual linking**: **in scope**, from Milestone 1 onward, not deferred to a later phase. `POST /api/memory/{id}/links` / `DELETE /api/memory/{id}/links/{target_id}` shipped in the same backend milestone as the read endpoint, and the frontend's "Link mode" toolbar toggle + per-node "Start link" action + per-relationship remove (×) in the detail panel shipped in Milestone 2.
4. **Opt-in beta flag vs. default-on**: **neither, precisely.** The user's instruction was "Feature flag enabled (beta)," which was implemented as: the nav item is visible by default (gated only by the existing `can_manage_memory` privilege, same as every other memory entry point) and carries a static "beta" label in the modal header — there is no separate togglable setting. This is flagged as an explicit open question needing confirmation in `docs/handoff.md`, since "feature flag" could reasonably have meant an actual on/off switch (the original Phase 2 proposal below, under "Migration strategy") rather than just a visible label.

## Deviations from the original plan worth knowing

Beyond the four resolved questions above, implementation diverged from the letter of this document in two ways, both deliberate and both recorded in more detail in `docs/handoff.md`:

- **§3 API design** describes per-request query params (`category`, `min_similarity`, `max_edges_per_node`, etc.) driving what the server returns. What's actually built: the frontend fetches once per open (or after a mutation) at a generous floor (`min_similarity=0.5`, `max_edges_per_node=8`) and does all category/similarity/search filtering **client-side** against that cached graph via Cytoscape's own display/class toggling. The query params themselves still exist and work server-side (see `routes/memory/memory_graph_routes.py`); the frontend just doesn't re-hit them on every filter interaction, trading a slightly-higher initial payload for instant filter feedback.
- **§4 UI design**'s edge-type distinction (similarity solid/opacity-by-weight, session dashed, manual distinct-color-solid) was carried through as specified. The legend, search-highlight, and category-filter-chip pieces were all built in Milestone 2 rather than deferred to "Milestone 3 polish" as this document's phasing loosely implied — the user's Milestone 2 instructions listed them as hard requirements for that milestone, not nice-to-haves.

No further action will be taken beyond what's recorded in `docs/todos.md` until the user resumes work.