Compare

Last updated: dev@df2fad2 | 2026-07-12

Scope

This spec covers model A/B comparison behavior in:

routes/compare_routes.py;
routes/session_routes.py;
routes/chat_routes.py and routes/chat_helpers.py;
routes/model_routes.py;
routes/search_routes.py;
core/database.py model Comparison;
src/llm_core.py and src/endpoint_resolver.py;
frontend modules under static/js/compare/;
static/js/chat.js, static/js/sessions.js, static/js/models.js, and static/js/slashCommands.js;
tests/test_compare_* and focused blind-compare redaction tests.

Runtime Behavior

The active text compare UI creates ordinary [CMP] sessions through /api/session, then streams each pane through /api/chat_stream with compare_mode=true. Search compare is a separate branch: it can query /api/search/query directly and its synthesis sessions use ordinary chat streaming without compare_mode=true. static/js/compare/index.js owns compare orchestration, session creation, execution order, search-mode branching, and export actions. static/js/compare/panes.js owns pane add/remove/swap/reroll lifecycle. static/js/compare/stream.js owns pane streaming and event rendering.

routes/compare_routes.py owns the /api/compare HTTP surface for alternate/legacy start/vote/history/delete behavior and the active /api/compare/record vote-summary endpoint. Legacy /api/compare/start now uses neutral helper-session names and withholds model identities/mapping from the start response while blind mode is active. It does not own provider-specific payload behavior.

Current call sites include:

/api/session compare session creation and cleanup in compare frontend modules;
/api/chat_stream pane execution through chat routes and detached stream infrastructure, streamed directly into panes so upstream generation stops promptly when panes are stopped;
/api/models and probe routes for model/endpoint selection;
search-provider compare mode through routes/search_routes.py;
/api/compare/record as a fire-and-forget backend vote summary, while active scoreboard state is localStorage-backed.

Comparison rows currently persist vote/history metadata: prompt, first model identifiers, winner, blind flag, optional N-model JSON in blind_mapping, vote timestamp, and owner. Response and metric columns exist in the schema but are not populated by the active compare UI flow. Compare history must be owner-scoped.

Frontend compare behavior is split by responsibility:

state.js owns local compare state;
selector.js, models.js, and probe.js own endpoint/model selection and probe UI;
panes.js and stream.js own paired response rendering;
vote.js and scoreboard.js own voting and history display.

Mobile compare layout collapses multi-pane grids to a single column so panes remain readable on narrow screens while the desktop grid still uses the selected column count.

Ownership Boundaries

Compare owns paired evaluation flow and pane state. Chat routes own the actual stream execution path for compare panes. LLM provider code owns model-call mechanics. Session/model routes own endpoint-id resolution, owner-filtered endpoint/model visibility, header copying, and deleted-endpoint failures.

compare_mode in chat strips compare-breaking tools, disables document tools for [CMP] sessions, skips some research clarification, and suppresses memory, skill, and webhook side effects after pane responses.

Compare frontend code is part of the app DOM security surface. Current stream/search rendering sanitizes probe labels and tool labels, constrains search-result links to HTTP(S), uses safe generated-image display sources, and opens compare export/image popups with opener isolation.

Policy Notes

Current blind compare is UI/API masking until vote/reveal, not a full confidentiality boundary. [CMP] session names and session-list model fields are redacted for helper sessions, and legacy /api/compare/start withholds model identity/mapping while blind. Client-side selected model state and privileged/local inspection can still expose identity.
Compare endpoint lists and secondary endpoint lookups use owner filtering so users see and resolve only shared or owned endpoints.
Non-admin compare session creation must use registered owner-visible endpoints; compare must not allow arbitrary raw endpoint URLs to bypass session-route endpoint policy.
Prefetched search, URL, RAG, and research context entering compare panes must use the untrusted-context wrapper.
If endpoint fallback behavior changes, verify each pane still goes to the intended model and that fallback notices/errors are visible in compare panes.

Degraded And Compatibility Behavior

Missing/offline endpoints are surfaced by model/session routes; chat can clear orphaned endpoint references and recover empty models when possible.
Compare streams inherit chat's pre-output-only model fallback and provider-normalized SSE events, but compare frontend handling for event: error and type: "fallback" is thinner than chat's stream path.
Shared legacy ModelEndpoint.owner == NULL rows remain visible through owner filters. Legacy Comparison.owner == NULL rows are not treated as shared for authenticated vote/delete/history flows.
/api/compare/start and /{comp_id}/vote remain implemented but are not the active frontend path.

Current Gaps

Blind mode is not a confidentiality boundary; client/local state can still expose model identity before vote.
/api/compare/start accepts raw endpoint URLs and can diverge from /api/session endpoint-owner/raw-endpoint policy.
src/agent_loop.py advertises stale compare app API endpoints.
Compare streaming and chat streaming are separate frontend paths but share model/provider infrastructure; regressions can happen when provider event shape changes.
Compare frontend needs explicit fallback/error event handling parity with chat streaming.
Compare tests cover endpoint owner helper behavior, blind compare redaction, and portable JS helpers, but not full active /api/session pane creation, frontend pane lifecycle, or SSE fallback/error handling.

6.1 KiB Raw Blame History