From edba0d92fa0e062cf1b69bf7c8aefea6d7eb55fd Mon Sep 17 00:00:00 2001 From: RaresKeY <158580472+RaresKeY@users.noreply.github.com> Date: Sun, 26 Jul 2026 13:45:29 +0000 Subject: [PATCH] docs(specs): restore bootstrap after dev rewrite --- specs/_readme.md | 89 +++ specs/agent-tools.md | 157 +++++ specs/architecture-runtime-inventory.md | 572 ++++++------------ specs/auth-security.md | 159 +++++ specs/calendar-tasks-notes.md | 185 ++++++ specs/chat.md | 145 +++++ specs/compare.md | 77 +++ specs/context-building.md | 111 ++++ specs/cookbook-hwfit.md | 194 ++++++ specs/documents-rag-uploads.md | 202 +++++++ specs/email-contacts.md | 198 ++++++ specs/frontend.md | 151 +++++ specs/gallery-editor-media.md | 162 +++++ specs/integrations.md | 195 ++++++ specs/llm-models.md | 151 +++++ specs/memory-skills.md | 121 ++++ specs/model-capability-canonical.md | 184 ++++++ specs/model-providers/_readme.md | 103 ++++ specs/model-providers/anthropic.md | 46 ++ specs/model-providers/atlas-cloud.md | 21 + specs/model-providers/azure-openai.md | 26 + specs/model-providers/bedrock.md | 23 + specs/model-providers/cerebras.md | 23 + specs/model-providers/chatgpt-subscription.md | 47 ++ .../model-providers/cloudflare-workers-ai.md | 21 + specs/model-providers/cohere.md | 56 ++ specs/model-providers/deepseek.md | 28 + specs/model-providers/fireworks.md | 22 + specs/model-providers/github-copilot.md | 46 ++ specs/model-providers/github-models.md | 21 + specs/model-providers/google.md | 54 ++ specs/model-providers/groq.md | 24 + specs/model-providers/hugging-face.md | 41 ++ specs/model-providers/llama-cpp.md | 47 ++ specs/model-providers/lm-studio.md | 45 ++ .../local-compatible-engines.md | 37 ++ specs/model-providers/minimax.md | 48 ++ specs/model-providers/mistral.md | 49 ++ specs/model-providers/moonshot-kimi.md | 30 + specs/model-providers/nvidia-nim.md | 28 + specs/model-providers/ollama.md | 54 ++ specs/model-providers/openai-compatible.md | 55 ++ specs/model-providers/openai.md | 40 ++ specs/model-providers/opencode.md | 23 + specs/model-providers/openrouter.md | 44 ++ specs/model-providers/perplexity.md | 20 + specs/model-providers/sglang.md | 47 ++ specs/model-providers/siliconflow.md | 21 + specs/model-providers/together.md | 27 + specs/model-providers/venice.md | 19 + specs/model-providers/vllm.md | 44 ++ specs/model-providers/xai.md | 21 + specs/model-providers/zai.md | 23 + specs/model-quirks.md | 88 +++ specs/persistence.md | 133 ++++ specs/research.md | 153 +++++ specs/runtime.md | 110 ++++ specs/search.md | 137 +++++ specs/settings-admin.md | 187 ++++++ specs/shell-mcp.md | 172 ++++++ specs/speech.md | 130 ++++ specs/testing-devops.md | 210 +++++++ 62 files changed, 5309 insertions(+), 388 deletions(-) create mode 100644 specs/_readme.md create mode 100644 specs/agent-tools.md create mode 100644 specs/auth-security.md create mode 100644 specs/calendar-tasks-notes.md create mode 100644 specs/chat.md create mode 100644 specs/compare.md create mode 100644 specs/context-building.md create mode 100644 specs/cookbook-hwfit.md create mode 100644 specs/documents-rag-uploads.md create mode 100644 specs/email-contacts.md create mode 100644 specs/frontend.md create mode 100644 specs/gallery-editor-media.md create mode 100644 specs/integrations.md create mode 100644 specs/llm-models.md create mode 100644 specs/memory-skills.md create mode 100644 specs/model-capability-canonical.md create mode 100644 specs/model-providers/_readme.md create mode 100644 specs/model-providers/anthropic.md create mode 100644 specs/model-providers/atlas-cloud.md create mode 100644 specs/model-providers/azure-openai.md create mode 100644 specs/model-providers/bedrock.md create mode 100644 specs/model-providers/cerebras.md create mode 100644 specs/model-providers/chatgpt-subscription.md create mode 100644 specs/model-providers/cloudflare-workers-ai.md create mode 100644 specs/model-providers/cohere.md create mode 100644 specs/model-providers/deepseek.md create mode 100644 specs/model-providers/fireworks.md create mode 100644 specs/model-providers/github-copilot.md create mode 100644 specs/model-providers/github-models.md create mode 100644 specs/model-providers/google.md create mode 100644 specs/model-providers/groq.md create mode 100644 specs/model-providers/hugging-face.md create mode 100644 specs/model-providers/llama-cpp.md create mode 100644 specs/model-providers/lm-studio.md create mode 100644 specs/model-providers/local-compatible-engines.md create mode 100644 specs/model-providers/minimax.md create mode 100644 specs/model-providers/mistral.md create mode 100644 specs/model-providers/moonshot-kimi.md create mode 100644 specs/model-providers/nvidia-nim.md create mode 100644 specs/model-providers/ollama.md create mode 100644 specs/model-providers/openai-compatible.md create mode 100644 specs/model-providers/openai.md create mode 100644 specs/model-providers/opencode.md create mode 100644 specs/model-providers/openrouter.md create mode 100644 specs/model-providers/perplexity.md create mode 100644 specs/model-providers/sglang.md create mode 100644 specs/model-providers/siliconflow.md create mode 100644 specs/model-providers/together.md create mode 100644 specs/model-providers/venice.md create mode 100644 specs/model-providers/vllm.md create mode 100644 specs/model-providers/xai.md create mode 100644 specs/model-providers/zai.md create mode 100644 specs/model-quirks.md create mode 100644 specs/persistence.md create mode 100644 specs/research.md create mode 100644 specs/runtime.md create mode 100644 specs/search.md create mode 100644 specs/settings-admin.md create mode 100644 specs/shell-mcp.md create mode 100644 specs/speech.md create mode 100644 specs/testing-devops.md diff --git a/specs/_readme.md b/specs/_readme.md new file mode 100644 index 000000000..6e82de44e --- /dev/null +++ b/specs/_readme.md @@ -0,0 +1,89 @@ +# Specs DocumentMap + +Last updated: dev@e57f60b | 2026-07-20 + +This folder is the compact implementation-truth map for humans and coding agents working on Odysseus. Read this file first, then open only the subsystem specs that match the work. + +Specs are living notes about current code shape and intended contracts. They are not product marketing, not PR planning, not templates, and not a replacement for source inspection or tests. + +This `_readme.md` is the DocumentMap and control document. It is intentionally exempt from subsystem `Scope` and `Current Gaps` sections; keep it limited to the quality contract, working rules, subsystem map, and cross-cutting update triggers. + +## Quality Contract + +Each subsystem spec should stay compact and useful under context pressure: + +- Start with `Last updated: dev@ | YYYY-MM-DD`, using the + upstream `dev` commit the spec text was inspected against. +- Use a concrete `Scope` section that names real files, route surfaces, frontend modules, data stores, and integration points. +- Use domain-specific sections. Do not force every spec into the same headings when the subsystem needs `Streaming`, `Tool Results`, `Optional Dependencies`, `Current Gaps`, or another focused section. +- State ownership clearly: which file owns a mapping, which layer only forwards state, and which caller requests behavior without owning implementation. +- Include runtime behavior bullets for flows that matter. +- Include "Current call sites include" when behavior is spread across many files. +- Record transitional compatibility notes, especially `src/` versus `services/` duplication. +- Record degraded, optional, or platform behavior where it changes runtime expectations. +- Record policy/provenance where relevant: untrusted context, encrypted secrets, API token scopes, optional dependency/license implications, generated media, or user data. +- End with `Current Gaps` only when there is a real known gap, not as filler. + +If code and specs disagree, treat code as ground truth. Update specs only when +the current task explicitly includes spec maintenance or the PR intentionally +includes specs; otherwise report the drift in the relevant issue, PR review, or +project documentation. + +## Working Rules + +- Start here before substantial work. +- Read the related subsystem spec before changing code in that area. For cross-cutting work, include the owning domain spec plus route/runtime, auth/security, persistence, frontend, tool/context, integration, and testing/devops specs as applicable. +- Treat specs as read-only context during ordinary project work, PR review, and code review. Do not edit specs unless the user explicitly asks for spec work or the current PR intentionally includes spec changes. +- During explicit spec-maintenance work, update the related spec when source inspection shows behavior, ownership, security boundaries, data shape, import paths, or implementation contracts have changed. +- During ordinary work, record source/spec drift in the relevant issue, PR review, or project documentation instead of mutating specs. +- Keep specs dense but readable. Prefer current facts and invariants over broad explanation. +- Every non-index `specs/*.md` file should appear exactly once in the Subsystem Map with a one-line description and no dead link. +- Specs contain implementation truth. Planning, research, branch notes, and decisions belong in tracked project docs. Drafts, audit reports, raw exports, and exploratory gap lists are not authoritative until promoted into tracked docs or specs. +- Use repo source and these specs as the authority for Odysseus architecture. Do not treat global skill registries or external agent metadata as repo ground truth. + +## Subsystem Map + +- [runtime.md](runtime.md): FastAPI startup, router registration, static serving, lifespan, app-wide middleware. +- [architecture-runtime-inventory.md](architecture-runtime-inventory.md): current runtime/module inventory, import-risk map, and behavior-preserving refactor candidates. +- [auth-security.md](auth-security.md): auth, privileges, API tokens, security headers, untrusted data, SSRF and admin boundaries. +- [persistence.md](persistence.md): SQLite models, startup migrations, encrypted columns, ownership columns, data directory rules. +- [chat.md](chat.md): chat routes, sessions, streaming, uploads-in-chat, compare handoff, research/chat mode dispatch. +- [compare.md](compare.md): model A/B comparison runs, voting/history, compare frontend panes, compare ownership. +- [llm-models.md](llm-models.md): LLM provider calls, endpoint discovery, model context length, fallbacks, model endpoints. +- [model-capability-canonical.md](model-capability-canonical.md): canonical provider/model capability shapes, evidence, payload resolution, and safe fallback. +- [model-quirks.md](model-quirks.md): model-specific behavior observations, evidence, and promotion gates. +- [model-providers/_readme.md](model-providers/_readme.md): provider-by-provider API/catalog shape index and compatibility status. +- [agent-tools.md](agent-tools.md): agent loop, tool schemas, tool execution, tool retrieval, tool security, MCP tool exposure. +- [context-building.md](context-building.md): URL/search/RAG/memory/skills/YouTube/email/tool-output context, untrusted wrapping, unavailable context, intent boundaries. +- [search.md](search.md): web search providers, ranking, cache/analytics, URL fetch/content extraction, `src.search`/`services.search` split. +- [documents-rag-uploads.md](documents-rag-uploads.md): uploads, documents, PDF/form handling, personal docs, RAG/vector stores. +- [memory-skills.md](memory-skills.md): memory storage, semantic memory, skill extraction/formatting, owner isolation. +- [research.md](research.md): deep research jobs, synthesis, sources, research library, research UI panel. +- [calendar-tasks-notes.md](calendar-tasks-notes.md): CalDAV calendars, scheduled tasks, reminders, assistant runs, notes/todos. +- [email-contacts.md](email-contacts.md): IMAP/SMTP email, email library, scheduled mail, contacts/CardDAV. +- [gallery-editor-media.md](gallery-editor-media.md): gallery, generated media, image editor drafts, signatures, emoji/font helpers. +- [cookbook-hwfit.md](cookbook-hwfit.md): model downloads, local/remote model serving, hardware detection, fit ranking. +- [speech.md](speech.md): STT and TTS services, routes, settings, optional dependencies. +- [frontend.md](frontend.md): static SPA, module loading, UI conventions, major JS areas, no-build frontend shape. +- [integrations.md](integrations.md): Codex/Claude scoped APIs, companion pairing, webhooks, external agent access. +- [shell-mcp.md](shell-mcp.md): shell execution, background jobs, MCP manager, built-in MCP servers. +- [settings-admin.md](settings-admin.md): settings, preferences, presets, backup/import/export, diagnostics, admin wipe. +- [testing-devops.md](testing-devops.md): pytest, JS tests, Docker, scripts, requirements, local dev expectations. + +## Cross-Cutting Spec Update Triggers + +Use these triggers only during explicit spec-maintenance work or a PR that +intentionally includes specs. For ordinary work and code review, use the same +list to choose which specs to read and where to report drift. + +- New route file or route prefix: update [runtime.md](runtime.md) and the owning subsystem spec. +- New SQLAlchemy model, column migration, durable JSON/local store, data directory, backup/import domain, or non-SQL persistence behavior: update [persistence.md](persistence.md) and the owning subsystem spec. +- New tool, tool schema, agent prompt rule, or tool security behavior: update [agent-tools.md](agent-tools.md) and [context-building.md](context-building.md) if it adds model context. +- New MCP runtime/config/built-in behavior: update [shell-mcp.md](shell-mcp.md), [agent-tools.md](agent-tools.md), and [context-building.md](context-building.md) when MCP tool results enter model context. +- New external content source, tool result, MCP/app API result, or integration result shown to an LLM: update [context-building.md](context-building.md) and [auth-security.md](auth-security.md). +- New API-token scope, scoped external API, webhook, companion/pairing route, generic integration provider, or external-agent helper bundle: update [integrations.md](integrations.md), [auth-security.md](auth-security.md), and the owning subsystem spec. +- New secret store, decrypted-secret return path, settings backup/import/export behavior, diagnostics/log output, vault/tool secret flow, `.env*` policy change, or credential-bearing CLI output: update [auth-security.md](auth-security.md), [settings-admin.md](settings-admin.md), [testing-devops.md](testing-devops.md), and the owning subsystem spec. +- New optional dependency, degraded fallback, platform/Docker/native/launcher difference, GPU overlay behavior, or retired compatibility shim: update [testing-devops.md](testing-devops.md) and the owning subsystem spec; also update [runtime.md](runtime.md), [llm-models.md](llm-models.md), [shell-mcp.md](shell-mcp.md), [cookbook-hwfit.md](cookbook-hwfit.md), or [persistence.md](persistence.md) when that layer owns the behavior. +- New frontend module or modal/tool surface: update [frontend.md](frontend.md) and the owning subsystem spec. +- New static/PWA/service-worker/cache/CSP behavior: update [frontend.md](frontend.md), [runtime.md](runtime.md), and [auth-security.md](auth-security.md) when headers or trust boundaries change. +- New CLI script: update [testing-devops.md](testing-devops.md) and the owning subsystem spec. diff --git a/specs/agent-tools.md b/specs/agent-tools.md new file mode 100644 index 000000000..81ca52f7b --- /dev/null +++ b/specs/agent-tools.md @@ -0,0 +1,157 @@ +# Agent Tools + +Last updated: dev@df2fad2 | 2026-07-12 + +## Scope + +This spec covers agent/tool behavior in: + +- `src/agent_loop.py`; +- `src/llm_core.py`; +- `src/tool_schemas.py`; +- `src/tool_execution.py`; +- `src/tool_policy.py`; +- `src/tool_index.py`; +- `src/tool_parsing.py`; +- `src/tool_security.py`; +- `src/attachment_refs.py` and shared upload lifecycle helpers in + `src/upload_handler.py` / `src/tool_utils.py`; +- `src/tool_implementations.py`; +- `src/tools/*.py`; +- `src/builtin_actions.py`; +- `src/ai_interaction.py`; +- `src/action_intents.py`; +- `src/goal_based_extractor.py`; +- `src/teacher_escalation.py`; +- `src/agent_tools/` modules and compatibility facade; +- `src/mcp_manager.py`; +- `src/builtin_mcp.py`; +- `src/bg_jobs.py` and `src/bg_monitor.py`; +- `routes/chat_routes.py`, `routes/chat_helpers.py`, `routes/model_routes.py`, `routes/skills_routes.py`, `routes/mcp_routes.py`, and `routes/workspace_routes.py`; +- `mcp_servers/*.py`; +- frontend stream/admin/settings files that display tool events, workspaces, and disabled tools; +- `tests/test_agent_loop.py`, `tests/test_tool_*`, and focused MCP/public-policy/schema tests. + +## Agent Loop + +`src.agent_loop` owns agent prompt assembly, request-local current date/time insertion, tool retrieval, prompted tool-block handling, native tool-call consumption after `llm_core` normalizes provider events, multi-round execution, tool result insertion, final metrics, and fallback responses. It requests context from documents, skills, tool retrieval, and messages; it should not own domain-specific business logic for every tool. Its prompt rules now bias structured/long-form writing toward living documents, route active compose/email drafts back into existing email documents, and prefer first-class `web_search`/`web_fetch` tools over shell/Python/curl for current web lookups when web tools are enabled. + +`src.llm_core` owns provider payloads, native tool-schema emission, and provider stream parsing. `agent_loop` consumes normalized tool-call events and decides whether and how to execute them. + +Agent mode enters through chat routes, including auto-escalation from intent helpers, detached `agent_runs` streaming, resume/stop behavior, and frontend tool-event rendering. + +Guide-only/no-tools turns are runtime policy, not prompt advice. `src.tool_policy` detects strong latest-turn directives such as guide-only mode, no-tools mode, and explicit requests not to use tools; it builds a `ToolPolicy` that hides schemas, disables known native tools, disables MCP for that turn, skips tool retrieval, suppresses local/workspace context injection, blocks document streaming/teacher escalation, and gives `tool_execution` a final execution backstop. + +Plan mode is a read-only investigation path inside the same loop. It adds a denylist for known mutating tools, filters write/unknown MCP tools, prepends plan-mode instructions, and uses the `update_plan` tool only after a plan is approved for execution. The backend path still exists for compatibility, but current browser chat forces incoming `plan_mode` off and the old plan-window UI module is gone. + +Workspace mode is request-scoped. Admin chat can send a workspace directory selected through `static/js/workspace.js`; `agent_loop` injects that fact early in the prompt and `tool_execution` confines bash, python, read/write/edit-file, and code-navigation tools to that root. `routes.workspace_routes` owns admin-only browse/vet APIs, skips hidden/symlink directory traversal, caps listings, and rejects sensitive/root paths before a workspace reaches chat. + +## Tool Registry + +Tool registration is split: + +- `src.agent_tools` is now a package/facade. `TOOL_HANDLERS` maps native tool names to handler functions across filesystem, subprocess, web, document, interaction, model-interaction, background-job, session, and admin modules, while `TOOL_TAGS` keeps compatibility metadata and the global MCP manager handle; +- `src.tools` owns domain do_* implementations for calendar, contacts, Cookbook, image, notes, research, search, system, and vault tools. `src.tool_implementations` is now a compatibility facade that re-exports those symbols and lazy-loads admin manage_* symbols to avoid circular imports; +- `src.agent_tools.admin_tools` owns admin manage_* tools for endpoints, MCP, webhooks, tokens, and settings, including command validation for `manage_mcp`; +- `src.tool_parsing._TOOL_NAME_MAP` owns aliases and prompted-block parsing; +- `src.tool_schemas.FUNCTION_TOOL_SCHEMAS` and `function_call_to_tool_block()` own native schema and native-call conversion; +- `src.tool_index.BUILTIN_TOOL_DESCRIPTIONS` owns retrieval text; +- `src.tool_execution.execute_tool_block()` owns dispatch and hard execution gates; +- `routes.model_routes.py` and frontend settings/admin surfaces expose global disabled-tool controls. + +When adding, removing, or renaming a tool, update the registry chain, execution dispatch, retrieval text, prompt wording, disabled-tool UI, and tests together. + +`src.tool_index.ALWAYS_AVAILABLE` is the retrieval catalog for high-frequency tools such as shell/python, web search/fetch, read/write/edit-file, code-nav, `manage_memory`, `ask_user`, `update_plan`, selected Cookbook serve controls, and `app_api`. Current prompt/schema assembly preserves only selected base tools unconditionally, then adds intent-, skill-, and retrieval-relevant tools so unrelated schemas do not flood small contexts. + +## Tool Retrieval And Execution + +`src.tool_index.ToolIndex` owns candidate retrieval using embeddings/keywords and cached index data. Security filtering is not its hard boundary: `agent_loop` hides unavailable schemas, and `tool_execution` blocks disabled, admin-only, and public-restricted calls before dispatch. + +`src.tool_execution` owns built-in tool execution, MCP dispatch, path confinement, background markers, output truncation, internal HTTP loopback, owner/admin checks, policy-blocked execution results, and formatting tool results for the model/UI. File tools support exact edit diffs, full-file writes, read line ranges, and workspace confinement. Code-navigation tools (`grep`, `glob`, `ls`) prefer `rg`/structured filesystem traversal over ad hoc shell commands. Uploaded-file context uses stable `attachment_ref` manifests and owner-checked URIs; a compatibility local path is exposed only after upload-root and tool-root confinement. Shared truncation, upload-handler registration, and MCP manager compatibility helpers live in `src.tool_utils`. + +Tool retrieval has domain-specific hooks beyond generic similarity: contact queries can surface `resolve_contact`/`manage_contact`; matched skills can add `manage_skills` and their required toolsets to the relevant tool set; explicit admin intents can include admin schemas so prompt text and native schema emission match. + +Interaction/session/model helper tools are native first-class tools, not prompt-only conventions. `ask_user` and `update_plan` live in `src.agent_tools.interaction_tools`, model delegation/listing helpers live in `model_interaction_tools`, session creation/list/send/manage helpers live in `session_tools`, and `manage_bg_jobs` lives in `bg_job_tools`. + +Prompted-tool parsing includes recovery paths for local/provider text leaks: +bare JSON after a web-tool mention, OpenAI-style raw +`{"function": ...}` payloads, StepFun/Gemma/DSML markup, and +`......` +wrappers from local MLX/Exo models. Non-dict JSON arguments are rejected back to +empty args instead of crashing the turn, common `tex` typos normalize to +`text`, and delimiter scans are forward-only so unterminated tool markup cannot +drive quadratic regex rescans. Executed raw tool JSON is stripped from assistant +text afterward; this is still not a general-purpose JSON-command parser. + +Current call sites include: + +- agent mode tool calls from `src.agent_loop`; +- MCP route configuration and built-in MCP registration; +- background job monitoring and auto-continue; +- skill tests, teacher escalation, scheduled tasks, and background follow-up loops; +- UI-control and AI interaction helpers. + +## Streaming And Continuations + +Agent streaming emits normal content plus tool progress/output, document stream/update, ask-user choices, plan updates, budget, round exhaustion, loop-breaker, intent-nudge exhaustion, metrics, teacher escalation, research anchor, and finish/error events. Frontend chat stream code and detached replay depend on stable event names. If the stream generator closes while awaiting an in-flight tool, the loop cancels and awaits that tool task so subprocess-backed work is not left orphaned. + +Long-running bash jobs can be detached with background markers. `src.bg_jobs` owns persistent job state/result files; `src.bg_monitor` owns auto-continuation when jobs finish. Detached chat runs are in-memory and do not survive server restart, while background job state is disk-backed. + +Loop-breaker final-answer rounds, explicit repeated-tool/intent-nudge guard events, round-cap continuation signals, optional verifier retries, and teacher escalation are recovery behavior owned by `agent_loop` and `src.teacher_escalation`. + +## Security And Policy + +- `src.tool_security` owns non-admin blocked-tool decisions. +- Non-admin users must not reach admin tools through agent mode, MCP, retrieval, or loopback calls. +- Agent owner is passed from chat route `get_current_user(request)`. In `AUTH_ENABLED=false` mode this is `None`, not the `""` value returned by route dependencies. `blocked_tools_for_owner()`, schema hiding, and `execute_tool_block()` all use that owner. +- Current dev tool security treats explicit `AUTH_ENABLED=false` as single-user even when an auth store exists, while auth-enabled pre-setup callers remain non-admin. +- Path-based tools must remain confined to allowed roots and reject sensitive paths. Sensitive-path checks are case-insensitive and apply to direct file tools and code-navigation tools; `grep`/`glob`/`ls` must not become existence or content oracles for `.env`, SSH/GPG material, `id_rsa`, and similar denylisted paths. +- Tool output is bounded/truncated where native execution owns the path, including displayed agent-tool output through the shared truncation helper. MCP output must be treated as untrusted; central MCP-output truncation before model re-entry remains a gap. +- Provider-emitted native tool calls are requests, not authorization. `tool_execution` and route-level policy remain the authority. +- Attachment-bearing document, note, and calendar tools owner-reserve internal + upload references before durable writes and fail without mutation when the + referenced upload is unavailable. +- Guide-only/no-tools mode blocks tools before prompt assembly, before execution, and in chat preprocessing paths that would otherwise fetch context or start tool-backed research. +- Plan mode is policy, not prompt advice: mutating native tools are disabled through schema-derived detection plus a static backstop, and write/unknown MCP tools are hidden and runtime-blocked for that turn. + +## Internal Loopback + +`do_app_api()` is implemented in `src.tools.system` and re-exported by `src.tool_implementations`. It owns generic app API loopback, OpenAPI discovery, method/path blocklists, and fixed local target behavior. `_internal_headers()` adds the process-secret internal-tool token and optional `X-Odysseus-Owner`; `core.middleware.require_admin()` and auth middleware own the corresponding bypass and owner-stamping rules. Route-specific owner handling must still be audited. + +## MCP + +`src.mcp_manager` owns configured MCP server lifecycle, discovered tool state, qualified MCP names, OpenAI schema conversion, call routing, generation invalidation, and connect/disconnect status. It supports stdio, SSE, and Streamable HTTP transports; Streamable HTTP can publish a `needs_auth` state and uses `src.mcp_oauth` for OAuth/OIDC-style authorization, token refresh, and encrypted token storage. `src.builtin_mcp` owns built-in server registration and the native-vs-MCP split. `mcp_servers/` owns server-specific tools for email, image generation, memory, RAG, and optional browser tooling. + +Native bash, python, file, web search, and web fetch tools continue through native fallback even when MCP is unavailable. Browser MCP is optional and can be skipped when cached Playwright/NPX packages are missing. Public users get no MCP schemas, and any `mcp__*` execution attempt must be blocked. + +MCP prompt/schema rendering includes server-provided input schemas, but names, types, and parameter hint text are sanitized and length-capped before entering the prompt. Per-server disabled tools filter listings, prompt descriptions, and function schemas; execution-time disabled-tool enforcement remains a separate hardening item. + +## Intent And Recovery Helpers + +`src.action_intents` owns deterministic chat-to-agent promotion hints and returns a category/reason so route logs can explain auto-escalation decisions. Explicit web-search language is category `web`; it can promote the turn into agent mode and narrow tools toward web search/fetch, but route policy requires explicit web-search enablement and honors explicit denial. It must avoid promoting explanatory questions into agent mode. `src.builtin_actions` owns scheduler/background actions outside the normal live agent loop. `src.teacher_escalation` owns recovery/escalation and skill-creation flows. `src.goal_based_extractor` is research-adjacent and should stay cross-referenced from research behavior rather than treated as ordinary tool execution. + +When an email reader is active, browser chat passes active email metadata and the agent loop injects it as protected, untrusted context so default reply/draft behavior targets the selected message. Active email compose documents are handled as existing email drafts rather than generic new-document requests. + +## Degraded Behavior + +- ToolIndex can degrade to keyword selection when embeddings, Chroma, index + warmup, or vector retrieval timeouts fail. +- Agent mode can degrade from native function schemas to prompted fenced-block parsing based on provider/tool-support heuristics. Local Ollama `/v1` and native `/api` endpoints default to text tools unless the endpoint explicitly advertises `supports_tools`; `gpt-oss` remains text-tool by default unless the endpoint opts in. +- MCP startup failure is non-critical; route/status surfaces expose per-server errors. +- `ODYSSEUS_DISABLE_MCP`, missing `mcp`, uncached browser MCP packages, and per-server disabled tools can remove tools without blocking the app. +- Global `builtin_browser` disable behavior may not currently match qualified `mcp__builtin_browser__*` tool names. + +## Current Gaps + +- Tool descriptions are duplicated across `FUNCTION_TOOL_SCHEMAS`, agent prompt sections, and `BUILTIN_TOOL_DESCRIPTIONS`. +- Agent prompts remain heavy for small local context windows. +- Some AI-control helpers are still globally wired from app startup rather than a narrower service layer. +- Tool registry consistency is manual across handler maps, tags, aliases, schemas, retrieval descriptions, execution dispatch, settings/model routes, and frontend toggles. +- MCP disabled-tool changes can stale-cache tool retrieval because disabled maps are not always an index generation input. +- External MCP output truncation and tool-result prompt-injection wrapping need stronger guarantees. +- Auth-disabled/no-login owner propagation is inconsistent between route dependencies and chat/agent execution, so tool-security and native tool storage behavior need dedicated regression coverage. +- Agent tests mostly cover helpers and targeted regressions, including round-cap + and disconnect cancellation paths, but not an end-to-end fake-LLM + `stream_agent_loop` path with retrieval, native schemas, prompted blocks, + disabled/admin hiding, MCP tools, plan/workspace state, user-time context, and + tool-result SSE. diff --git a/specs/architecture-runtime-inventory.md b/specs/architecture-runtime-inventory.md index 5c8e4bc21..17ad918e6 100644 --- a/specs/architecture-runtime-inventory.md +++ b/specs/architecture-runtime-inventory.md @@ -1,412 +1,208 @@ # Architecture Runtime Inventory -> **Purpose**: Phase 0 planning baseline for codebase readability improvements (#4071). -> **Parent issue**: [#4082](https://github.com/odysseus-dev/odysseus/issues/4082) -> **Last updated**: dev@b58af42 | 2026-06-16 -> **Status**: Draft — to be reviewed before follow-up slices open. -> **Snapshot basis**: Importer / file / import-line counts are refreshed to `dev@b58af42` (2026-06-16) and are recomputable via the commands in §3.4. **Line counts** in §2.1 / §2.2 are a snapshot from an earlier baseline and drift as `dev` moves — recompute any of them with `wc -l `. This inventory tracks structure and risk, not live metrics. +Last updated: dev@e57f60b | 2026-07-20 -This document maps the current runtime module structure, identifies high-risk boundaries, and recommends safe first refactor slices. It does **not** move files, change imports, or alter runtime behavior. +> Purpose: current runtime/module inventory for codebase readability work +> originally discussed around #4071/#4082. This is a source snapshot, not a +> refactor plan. Recompute metrics against current `dev` before treating any +> count as authoritative. ---- +This document maps the current runtime module structure, high-risk boundaries, +and behavior-preserving refactor candidates. It does not move files, change +imports, or alter runtime behavior. -## 1. Current Structure Overview - -### 1.1 Top-Level Layout +## Current Structure ``` odysseus/ -├── app.py # FastAPI app entrypoint (1,145 lines) -├── conf/ # Configuration (config.py, settings.py, settings_scrub.py) -├── src/ # 95 flat .py files + 2 subdirectories -│ ├── agent_tools/ # Tool helpers: document, filesystem, subprocess, web -│ └── search/ # Search subsystem -├── routes/ # 54 flat .py files — HTTP route handlers -├── core/ # 10 files — database models, auth, middleware, session -├── mcp_servers/ # 5 files — MCP server implementations -├── scripts/ # CLI tools and one-shot scripts -├── static/ # Frontend HTML/CSS/JS -├── tests/ # 583 test files (~54,800 lines) -└── services/ # (exists as needed) +├── app.py # FastAPI app entrypoint and route registry +├── conf/ # Configuration helpers +├── core/ # database, auth, middleware, session helpers +├── routes/ # HTTP routes plus selected domain subpackages +│ ├── contacts/ # canonical contacts/CardDAV route package +│ ├── gallery/ # canonical gallery route/helper package +│ ├── history/ # canonical chat history route package +│ ├── memory/ # canonical memory route package +│ ├── note/ # canonical notes/reminders route package +│ └── research/ # canonical research route package +├── src/ # agent/model/runtime services and facades +│ ├── agent_tools/ # native tool handler classes +│ ├── model_capability_readers/ # provider model-metadata normalization +│ ├── search/ # compatibility aliases for services.search +│ └── tools/ # split do_* tool implementation domains +├── services/ # service facades and canonical search/youtube paths +├── mcp_servers/ # built-in MCP server implementations +├── scripts/ # local CLI tools and one-shot scripts +├── static/ # no-build browser SPA +├── tests/ # pytest, Node, and static source-shape tests +└── specs/ # implementation-truth notes ``` -### 1.2 Directory Flatness Metric +### Directory Flatness -| Directory | Flat `.py` Files | Subdirectories | Concern | -|-----------|-----------------|----------------|---------| -| `src/` | **95** | 2 (`agent_tools/`, `search/`) | No domain grouping; 95 files in one directory | -| `routes/` | **54** | 0 | All route handlers in one flat directory | -| `core/` | 10 | 0 | Manageable, but `database.py` is oversized | +| Directory | Flat `.py` Files | Subdirectories | Current Concern | +|-----------|------------------|----------------|-----------------| +| `src/` | 100 | `agent_tools/`, `model_capability_readers/`, `search/`, `tools/` | Still broad, but tool handlers, capability readers, and do_* implementations now have packages. | +| `routes/` | 54 | `contacts/`, `gallery/`, `history/`, `memory/`, `note/`, `research/` | Route grouping has started; most domains remain flat top-level route files. | +| `core/` | 11 | none | Manageable count, but `database.py` remains oversized and highly imported. | ---- +## Largest Runtime Modules -## 2. Largest Runtime Modules +### Python Backend -### 2.1 Python Backend +| File | Lines | Notes | +|------|-------|-------| +| `routes/email_routes.py` | 5,226 | Largest HTTP domain; route, cache, compose, OAuth, and mutation behavior. | +| `src/agent_loop.py` | 4,529 | Agent orchestration, tool rounds, prompt/context assembly, recovery. | +| `routes/cookbook_routes.py` | 4,386 | Cookbook setup/download/serve/state flows. | +| `src/llm_core.py` | 2,869 | Provider payloads, streaming, fallbacks, provider quirks. | +| `src/builtin_actions.py` | 2,776 | Scheduler/background built-in action helpers. | +| `routes/model_routes.py` | 2,657 | Endpoint CRUD, probing, catalog cache, provider auth links. | +| `src/task_scheduler.py` | 2,627 | Task runner, runs, chained/event/webhook execution. | +| `core/database.py` | 2,562 | SQLAlchemy models plus manual SQLite migrations. | +| `routes/gallery/gallery_routes.py` | 1,966 | Canonical gallery/media route package. | +| `routes/note/note_routes.py` | 937 | Canonical notes/reminders route package. | +| `routes/contacts/contacts_routes.py` | 916 | Canonical contacts/CardDAV route package. | +| `routes/research/research_routes.py` | 783 | Canonical research route package. | +| `routes/history/history_routes.py` | 794 | Canonical chat history route package. | +| `routes/memory/memory_routes.py` | 552 | Canonical memory route package. | +| `src/tool_implementations.py` | 115 | Compatibility facade over `src/tools/*` and admin tool handlers. | -| Rank | File | Lines | Classes | Functions | Risk | -|------|------|-------|---------|-----------|------| -| 1 | `src/tool_implementations.py` | **4,032** | 0 | ~48 | **HIGH** | -| 2 | `routes/email_routes.py` | **3,245** | — | — | **MEDIUM** | -| 3 | `routes/cookbook_routes.py` | **2,969** | — | — | **MEDIUM** | -| 4 | `src/agent_loop.py` | **2,961** | 0 | ~24 | **HIGH** | -| 5 | `src/task_scheduler.py` | **2,330** | — | 5 | MEDIUM | -| 6 | `routes/model_routes.py` | **2,266** | — | — | MEDIUM | -| 7 | `core/database.py` | **2,265** | 28 | ~59 helpers | **HIGH** | -| 8 | `src/builtin_actions.py` | **2,262** | 2 | ~24 | MEDIUM | -| 9 | `src/llm_core.py` | **2,164** | — | — | MEDIUM | -| 10 | `mcp_servers/email_server.py` | 2,197 | — | — | LOW (separate process) | -| 11 | `src/visual_report.py` | 1,918 | — | — | LOW | -| 12 | `routes/gallery_routes.py` | 1,896 | — | — | LOW | -| 13 | `src/ai_interaction.py` | 1,846 | — | — | MEDIUM | -| 14 | `routes/document_routes.py` | 1,717 | — | — | LOW | -| 15 | `routes/skills_routes.py` | 1,648 | — | — | LOW | +### Frontend -**Heuristic**: Files > 2,000 lines with 20+ public symbols and many importers are the highest-risk splits. Files 1,000–2,000 lines are medium-risk if tightly coupled. +| File | Lines | Notes | +|------|-------|-------| +| `static/style.css` | 40,453 | App-wide CSS remains the largest frontend risk. | +| `static/js/document.js` | 11,038 | Large document editor/library coordinator. | +| `static/js/emailLibrary.js` | 7,784 | Email library UI and cache behavior. | +| `static/js/settings.js` | 5,795 | Settings modal and provider/admin-adjacent wiring. | +| `static/js/chat.js` | 5,457 | Main chat streaming/UI coordinator. | +| `static/app.js` | 4,389 | SPA orchestration and compatibility bridges. | -### 2.2 Frontend +## Import Dependency Snapshot -| File | Lines | Concern | -|------|-------|---------| -| `static/style.css` | **36,653** | Entire app CSS in one file (tracked separately in #2617) | -| `static/js/document.js` | **9,776** | Single JS file for document functionality | -| `static/js/slashCommands.js` | 6,498 | | -| `static/js/settings.js` | 5,266 | | -| `static/js/emailLibrary.js` | 5,217 | | -| `static/js/notes.js` | 5,124 | | -| `static/js/chat.js` | 4,985 | | -| `static/app.js` | 4,090 | | +| Relationship | Count | Notes | +|--------------|-------|-------| +| `core.database` importers | 118 | Highest-risk split target; routes, services, tests, and helpers depend on it. | +| `src.tool_implementations` importers | 22 | Still a live facade even after the tool split. | +| `src.agent_loop` importers | 32 | Agent loop is an orchestration hub. | +| `src/` import lines referencing `routes` | 36 | Mostly function-local compatibility/runtime coupling. | +| `routes/` import lines referencing `src` | 391 | Expected route-to-service direction. | +| `routes/` import lines referencing `core` | 139 | Expected DB/auth/session dependencies. | -**Note**: Frontend modularization is tracked separately in #2617 (CSS) and is not the focus of this Phase 0 inventory. Frontend is listed here for completeness but follow-up slices should target Python backend boundaries first. - ---- - -## 3. Import Dependency Graph - -### 3.1 Who Depends on `core/database.py` - -**102 files** import from `core.database` — this is the most depended-upon module: - -- All route handlers (`routes/*.py`) -- Most `src/*.py` files -- `core/session_manager.py`, `core/auth.py` -- Multiple test files - -**Implication**: Any split of `core/database.py` is the highest-risk refactor. It should be tackled **last**, never first. - -### 3.2 Who Depends on `src/tool_implementations.py` - -**17 files** import from `src.tool_implementations`: -- `src/agent_loop.py`, `src/builtin_actions.py`, `src/tool_index.py` -- `src/task_scheduler.py`, `src/tool_policy.py` -- Various tests - -### 3.3 Who Depends on `src/agent_loop.py` - -**22 files** import from `src.agent_loop`: - -- `src/tool_policy.py`, `src/teacher_escalation.py`, `src/bg_monitor.py` -- `src/task_scheduler.py` -- Multiple test files - -### 3.4 Cross-Layer Import Violations - -**`src/` importing from `routes/`** (backwards dependency — domain logic depending on HTTP layer): - -``` -src/tool_implementations.py ──→ routes/calendar_routes.py -src/tool_implementations.py ──→ routes/cookbook_helpers.py -src/tool_implementations.py ──→ routes/email_helpers.py -src/tool_implementations.py ──→ routes/email_pollers.py -src/tool_implementations.py ──→ routes/email_routes.py -src/tool_implementations.py ──→ routes/model_routes.py -src/tool_implementations.py ──→ routes/note_routes.py -src/tool_implementations.py ──→ routes/prefs_routes.py -``` - -> These are **runtime imports** (inside function bodies, not at module top), which mitigates circular import risk but indicates fuzzy layer boundaries. Function-level inline imports from the HTTP layer into business logic are a code smell. - -**Import counts (top-level)**: -| Direction | Count | Notes | -|-----------|-------|-------| -| `routes/` → `src/` | **374** | Expected: HTTP handlers call domain logic | -| `routes/` → `core/` | **126** | Expected: handlers access DB models | -| `src/` → `routes/` | **31** | **Unexpected**: domain logic reaching into HTTP layer (direct grep of import lines referencing `routes/`) | -| `src/` → `core/` | **106** | Acceptable but could be reduced with a data-access layer | - -> **How the metrics in this document are computed** — recompute against current `dev` before treating any count as authoritative (the tree drifts; these numbers are a snapshot, not a live value): -> - `src/` flat `.py` files: `find src -maxdepth 1 -name '*.py' | wc -l` -> - `tests/` test files: `find tests -name 'test_*.py' | wc -l` -> - `core.database` importers: `grep -rlE '(from|import) +core\.database' --include='*.py' . | grep -v core/database.py | wc -l` -> - `src.agent_loop` importers: `grep -rlE '(from|import) +src\.agent_loop' --include='*.py' . | grep -v src/agent_loop.py | wc -l` -> - Cross-layer import lines: `grep -rhE '(from|import) +' --include='*.py' / | wc -l` (e.g. `(from|import) +routes` over `src/`) - ---- - -## 4. Route Ownership Map - -Routes can be grouped into logical feature domains. Current flat structure obscures these boundaries: - -| Domain | Route Files | Total Lines | Review Complexity | -|--------|-------------|-------------|-------------------| -| **Email** | `email_routes.py`, `email_helpers.py`, `email_pollers.py` | 5,936 | HIGH — most complex domain | -| **Chat / Agent** | `chat_routes.py`, `chat_helpers.py`, `shell_routes.py`, `codex_routes.py`, `skills_routes.py` | 6,365 | HIGH — core interaction surface | -| **Cookbook** | `cookbook_routes.py`, `cookbook_helpers.py`, `cookbook_output.py` | 4,110 | MEDIUM | -| **Model / LLM** | `model_routes.py`, `assistant_routes.py`, `copilot_routes.py` | 2,764 | MEDIUM | -| **Calendar / Contacts** | `calendar_routes.py`, `contacts_routes.py` | 2,336 | MEDIUM | -| **Documents** | `document_routes.py`, `document_helpers.py` | 1,954 | LOW | -| **Auth** | `auth_routes.py`, `api_token_routes.py`, `device_flow.py` | 1,171 | LOW | -| **Tasks** | `task_routes.py` (standalone) | 1,157 | LOW | -| **Session** | `session_routes.py` (standalone) | 1,287 | LOW | -| **Gallery** | `gallery_routes.py`, `gallery_helpers.py` | 1,896 | LOW | -| **Memory** | `memory_routes.py` | — | LOW | -| **Research** | `research_routes.py` | — | LOW | -| **MCP** | `mcp_routes.py` | — | LOW | -| **Notes** | `note_routes.py` | — | LOW | -| **Other** | `prefs_routes.py`, `upload_routes.py`, `vault_routes.py`, `webhook_routes.py`, `workspace_routes.py`, `search_routes.py`, `history_routes.py`, `hwfit_routes.py`, `preset_routes.py`, `signature_routes.py`, `backup_routes.py`, `cleanup_routes.py`, `diagnostics_routes.py`, `embedding_routes.py`, `emoji_routes.py`, `font_routes.py`, `stt_routes.py`, `tts_routes.py`, `compare_routes.py`, `personal_routes.py`, `editor_draft_routes.py`, `admin_wipe_routes.py`, `chatgpt_subscription_routes.py` | 2,000+ | LOW individual, HIGH cumulative | - ---- - -## 5. Tool Registry & Implementation Boundaries - -### 5.1 Current Tool Architecture - -| Component | File | Lines | Role | -|-----------|------|-------|------| -| Tool schemas | `src/tool_schemas.py` | 1,392 | JSON Schema tool definitions (Duck-TypedDict) | -| Tool index | `src/tool_index.py` | 542 | RAG-based tool retrieval from ChromaDB | -| Tool implementations | `src/tool_implementations.py` | 4,032 | 33 `do_*` functions — all tool execution logic | -| Tool security | `src/tool_security.py` | — | Owner-scoped tool blocking | -| Tool policy | `src/tool_policy.py` | — | Guide-only directive, plan-mode disabled tools | -| Tool utils | `src/tool_utils.py` | — | Shared tool helpers | - -### 5.2 Tool Implementation Categories - -The 33 `do_*` functions in `tool_implementations.py` fall into natural domain groups — the basis for slice 1's split in §6.2: - -| Category | `do_*` functions | Count | -|----------|------------------|-------| -| **System / config** | `do_manage_skills`, `do_manage_tasks`, `do_manage_endpoints`, `do_manage_mcp`, `do_manage_webhooks`, `do_manage_tokens`, `do_manage_settings`, `do_api_call`, `do_app_api` | 9 | -| **Cookbook / model serving** | `do_download_model`, `do_serve_model`, `do_list_served_models`, `do_stop_served_model`, `do_tail_serve_output`, `do_list_downloads`, `do_cancel_download`, `do_search_hf_models`, `do_adopt_served_model`, `do_list_cookbook_servers`, `do_list_serve_presets`, `do_serve_preset`, `do_list_cached_models` | 13 | -| **Notes** | `do_manage_notes` | 1 | -| **Calendar** | `do_manage_calendar` | 1 | -| **Search** | `do_search_chats` | 1 | -| **Research** | `do_manage_research`, `do_trigger_research` | 2 | -| **Contacts** | `do_resolve_contact`, `do_manage_contact` | 2 | -| **Vault** | `do_vault_search`, `do_vault_get`, `do_vault_unlock` | 3 | -| **Image** | `do_edit_image` | 1 | -| | **Total** | **33** | - -> Low-level tools (filesystem, subprocess, web fetch, document parsing) live in `src/agent_tools/`, **not** in `tool_implementations.py` — out of scope for this split. - ---- - -## 6. Risk Assessment & Candidate Slice Ranking - -> **Candidate proposals, not a committed plan.** The rankings, package shapes (e.g. `src/pkg/`, `src/domain/`, `src/infra/`, `src/api/`), split ordering, and route-grouping strategy below are **options for maintainer discussion**. Per #4082/#4071, slice ownership and order are settled by maintainers before any follow-up PR. §1–§3 above are the factual current-state inventory. - -### 6.1 Risk Scale - -| Level | Criteria | -|-------|----------| -| **LOW** | File has ≤3 importers AND ≤500 lines, OR is a pure refactor with clear boundaries | -| **MEDIUM** | File has 4–15 importers OR 500–1,500 lines | -| **HIGH** | File has 16+ importers OR >2,000 lines, OR has cross-layer import violations | - -### 6.2 Ranked Split Candidates - -| Priority | Target | Risk | Rationale | -|----------|--------|------|-----------| -| **1** | `src/tool_implementations.py` → `src/tools/*.py` | **MEDIUM** | 4,032 lines → ~10 files by tool category. Already has natural boundaries. 17 importers, tracked in #3629. Use `__init__.py` shim to keep existing imports working. | -| **2** | `routes/` → domain subdirectories (one domain per PR) | **MEDIUM** | 54 flat files. Done **one domain at a time** (e.g. a standalone PR for the email domain, then chat, …), not a broad reorganization — route modules carry helper imports, registration assumptions, and test import paths. | -| **3** | `src/agent_loop.py` → `src/agent/loop.py` + submodules | **MEDIUM-HIGH** | 2,961 lines, 24 functions. Can extract prompt building, classification, verification, and runaway detection. Tracked in #3266. | -| **4** | `src/` → `src/pkg/`, `src/domain/`, `src/infra/`, `src/api/` | **MEDIUM** | Structural reorganization. Split flat `src/` into layered packages. Must come after routes and tools are stable. | -| **5** | `routes/email_*.py` consolidation | **LOW** | Already grouped by filename prefix. Low-risk cleanup within the email domain. | -| **6** | `core/database.py` → `src/infra/database/models/*.py` | **HIGH** | 28 classes, 102 importers. Highest-risk split. Must be **last** in any sequence. Requires careful import shim strategy. | -| **7** | Frontend CSS modularization | **MEDIUM** | 36,653 lines. Tracked in #2617. Separate timeline from backend work. | -| **8** | Frontend JS modularization | **MEDIUM** | 9,776 lines in `document.js`. Introduce ES modules at minimum. | - -### 6.3 Candidate First 3 Behavior-Preserving Slices - -**Slice 1: Split `tool_implementations.py`** (Lowest-risk high-impact) - -- Create `src/tools/` package with one file per tool category -- Add `src/tools/__init__.py` re-exporting all symbols with current names -- Update 17 importers to use new paths (can be deferred via shim) -- Validation: `python -m pytest tests/ -x -q` + manual smoke test of tool execution -- Reference: #3629 - -**Slice 2: Group `routes/` by domain** (one domain per PR, not a broad sweep) - -Route modules carry helper imports, router registration assumptions, and test import paths, so this must be done **one domain at a time** rather than as a single reorganization PR. Example sequence (each its own PR): - -- PR 2a: move the **email** domain (`email_routes.py`, `email_helpers.py`, `email_pollers.py`) → `routes/email/` + shim -- PR 2b: move the **chat/agent** domain → `routes/chat/` + shim -- PR 2c: move the **cookbook** domain → `routes/cookbook/` + shim -- …and so on per domain from §4 - -Each PR: add `__init__.py` re-exporting old names, update `app.py` router imports, validation `python app.py` starts clean. **No behavior change** — pure file reorganization. - -**Slice 3: Extract `agent_loop.py` submodules** (Improve reviewability) - -- Move prompt assembly → `src/agent/prompt.py` -- Move request classification → `src/agent/classifier.py` -- Move sub-agent verification → `src/agent/verifier.py` -- Move runaway detection → `src/agent/runaway.py` -- Move context management → `src/agent/context.py` -- Keep `src/agent/loop.py` as the main orchestration module -- Validation: `python -m pytest tests/test_agent_loop.py tests/test_loop_breaker_runaway.py -v` - ---- - -## 7. Safety Guardrails for Follow-Up Work - -Per maintainer guidance in #4082 and #4071: - -- [ ] **One domain/slice per PR** — never mix multiple reorganizations -- [ ] **No behavior changes** mixed with file moves — pure reorganization only -- [ ] **Keep compatibility shims** — `__init__.py` re-exports for all existing import paths -- [ ] **Add or identify focused tests** before risky splits -- [ ] **Do not start with `core/database.py`** or broad route movement unless this inventory shows a safe boundary -- [ ] **Prefer small, reviewable slices** over large restructures -- [ ] **No packaging/runtime/tooling migration** mixed into file moves -- [ ] **No frontend framework migration** inside this stabilization lane -- [ ] **Validate with `python -m compileall`** — every PR must pass CI checks -- [ ] **Validate with `pytest`** — run the full test suite before opening each PR - ---- - -## 8. Validation Commands - -Each follow-up PR should be verifiable with these commands before submission: +Recompute examples: ```bash -# Syntax check — must pass with zero errors -python -m compileall src/ routes/ core/ conf/ +find src -maxdepth 1 -name '*.py' | wc -l +find routes -maxdepth 1 -name '*.py' | wc -l +find tests -name 'test_*.py' | wc -l +wc -l app.py core/database.py src/agent_loop.py src/tool_implementations.py +rg -l '(^| )from core.database|(^| )import core.database' --glob '*.py' | wc -l +rg -n '(^| )from routes|(^| )import routes' src --glob '*.py' | wc -l +``` -# Full test suite — must match baseline pass rate +## Route Ownership Map + +Route modules are still mostly flat, with six landed domain packages: + +- `routes/gallery/gallery_routes.py` and `routes/gallery/gallery_helpers.py` + are canonical. `routes/gallery_routes.py` and `routes/gallery_helpers.py` + are compatibility shims that replace their `sys.modules` entries with the + canonical module object. +- `routes/memory/memory_routes.py` is canonical. `routes/memory_routes.py` is + a compatibility shim. +- `routes/research/research_routes.py` is canonical. `routes/research_routes.py` + is a compatibility shim. +- `routes/history/history_routes.py` is canonical. `routes/history_routes.py` + is a compatibility shim. +- `routes/contacts/contacts_routes.py` is canonical. + `routes/contacts_routes.py` is a compatibility shim. +- `routes/note/note_routes.py` is canonical. `routes/note_routes.py` is a + compatibility shim. + +Other major domains remain top-level route modules: + +| Domain | Primary Route Files | Current Risk | +|--------|---------------------|--------------| +| Email | `email_routes.py`, `email_helpers.py`, `email_pollers.py` | High: largest route surface and many side tables/caches. | +| Chat / Agent | `chat_routes.py`, `chat_helpers.py`, `shell_routes.py`, `codex_routes.py`, `skills_routes.py` | High: cross-cuts sessions, tools, research, compare, uploads. | +| Cookbook | `cookbook_routes.py`, `cookbook_helpers.py`, `cookbook_output.py` | Medium-high: code execution, SSH, model serving, state. | +| Model / LLM | `model_routes.py`, `assistant_routes.py`, `copilot_routes.py`, `chatgpt_subscription_routes.py` | Medium-high: secrets, endpoint ownership, provider auth. | +| Calendar / Contacts | `calendar_routes.py`, `contacts/contacts_routes.py` plus shim | Medium: remote sync/writeback and credential handling. | +| Documents | `document_routes.py`, `document_helpers.py`, `personal_routes.py`, `upload_routes.py` | Medium: files, ownership, optional renderers, RAG. | +| Auth / Admin | `auth_routes.py`, `api_token_routes.py`, `backup_routes.py`, `diagnostics_routes.py`, `admin_wipe_routes.py` | Medium: security-critical but more modular. | + +## Tool Registry And Implementation Boundaries + +Tool execution is no longer concentrated in one 4k-line module. + +| Component | Owner | Role | +|-----------|-------|------| +| Native handler registry | `src/agent_tools/__init__.py` | Maps native tool names to handler classes for bash/python/web/files/documents/interaction/model/session/background/admin tools. | +| Low-level native handlers | `src/agent_tools/*.py` | Filesystem, subprocess, web, document, interaction, model interaction, background job, session, and admin handler classes. | +| Domain do_* implementations | `src/tools/*.py` | Calendar, contacts, Cookbook, image, notes, research, search, system, and vault do_* functions. | +| Compatibility facade | `src/tool_implementations.py` | Re-exports old do_* names and lazy-loads admin do_* symbols so legacy imports and tests keep working. | +| Tool schemas | `src/tool_schemas.py` | Native OpenAI-style schemas and native-call conversion. | +| Tool retrieval | `src/tool_index.py` | Built-in and MCP tool retrieval text/indexing. | +| Tool parsing | `src/tool_parsing.py` | Prompted/fenced tool-call parsing and aliases. | +| Tool execution gates | `src/tool_execution.py` | Dispatch, path confinement, admin/non-admin gates, MCP dispatch, truncation, and UI formatting. | + +Current `src/tools/*` do_* domain counts: + +| Domain File | Count | +|-------------|-------| +| `calendar.py` | 1 | +| `contacts.py` | 2 | +| `cookbook.py` | 13 | +| `image.py` | 1 | +| `notes.py` | 1 | +| `research.py` | 2 | +| `search.py` | 1 | +| `system.py` | 4 | +| `vault.py` | 3 | + +Admin manage tools for endpoints, MCP, webhooks, tokens, and settings live in +`src/agent_tools/admin_tools.py` and are registered through `ADMIN_TOOL_HANDLERS`. + +## Risk Ranking For Future Refactors + +| Priority | Target | Risk | Notes | +|----------|--------|------|-------| +| 1 | Remaining route domains into packages | Medium | Do one domain per PR with `sys.modules` compatibility shims and route-import tests. | +| 2 | `src/agent_loop.py` submodules | Medium-high | Extract prompt/context assembly, classification, verification/recovery, and stream-round helpers without changing behavior. | +| 3 | Email route/service split | High | Valuable but risky because account ownership, IMAP cache, side DBs, OAuth, and compose/send are intertwined. | +| 4 | Cookbook route/service split | Medium-high | Preserve command validation, shell/SSH boundaries, and state semantics. | +| 5 | `core/database.py` model/migration split | High | Most imported module; should be late and shim-heavy. | +| 6 | Frontend CSS/large coordinator splits | Medium | Requires browser/module-order verification, not just source movement. | + +Already-landed structure that should not be treated as future work: + +- `src/tool_implementations.py` has already been split behind a facade. +- Gallery, research, memory, history, contacts, and note route packages already have canonical subpackage + locations plus top-level compatibility shims. + +## Safety Guardrails For Follow-Up Work + +- One domain/slice per PR. +- No behavior changes mixed with file moves. +- Keep compatibility shims for existing import paths until all call sites and + tests are intentionally migrated. +- Add import-parity tests for every moved module. +- Validate with focused tests for the moved domain plus compile checks. +- Do not start with `core/database.py` unless the change is a small migration or + helper extraction with dedicated tests. +- Avoid packaging/runtime/tooling migration inside route or module moves. + +## Validation Commands + +```bash +python -m compileall app.py core routes src conf services python -m pytest tests/ -x -q - -# Import shim verification — existing import paths must still work -python -c "from src.tool_implementations import do_search_chats; print('OK')" - -# App startup smoke test (if backend touched) -timeout 5 python app.py 2>&1 | head -5 || true -``` - ---- - -## 9. Open Questions - -1. Is `#2538` (specs ground truth) the canonical behavior map baseline, and should this inventory be kept in sync with those specs once merged? -2. Should route grouping follow the domain map proposed here, or is there a different taxonomy preferred by maintainers? -3. For the `tool_implementations.py` split (#3629), is the tool categorization in §5.2 acceptable, or should it follow a different grouping? -4. Should compatibility shims (`__init__.py`) be temporary (removed in a follow-up wave) or permanent? -5. Should an ADR (Architecture Decision Record) document be started to track decisions made during this process? - ---- - -## 10. Future Direction (NOT current state) - -The following are **future refactor targets** (candidate directions **pending maintainer agreement**, not committed), recorded here so this inventory does not imply they exist today. None of them are present in the current `dev` tree: - -- `main.py` — proposed rename of the `app.py` entrypoint. Today the app boots via `app.py`. -- `src/agent/` — proposed package to hold `agent_loop.py` submodules (prompt/classifier/verifier/runaway/context). Today `agent_loop.py` is a single flat file in `src/`. -- `src/infra/`, `src/domain/`, `src/pkg/`, `src/api/` — proposed layered reorganization of the flat `src/` directory (slice 4 in §6). - -These become real only when the corresponding slices land. - ---- - -## Appendix A: File Listing - -### `src/` (95 files — 61 shown; run `ls src/*.py` for the full list) - -``` -agent_loop.py tool_implementations.py tool_schemas.py -tool_index.py tool_security.py tool_policy.py -tool_utils.py builtin_actions.py task_scheduler.py -llm_core.py model_context.py model_discovery.py -session_search.py context_budget.py context_compactor.py -ai_interaction.py action_intents.py agent_runs.py -app_helpers.py app_initializer.py config.py -database.py memory.py memory_provider.py -secret_storage.py prompt_security.py url_security.py -url_safety.py rate_limiter.py cleanup_service.py -readiness.py service_health.py exceptions.py -request_models.py assistant_log.py bg_monitor.py -builtin_mcp.py chat_helpers.py chroma_client.py -document_processor.py embedding_lanes.py deep_research.py -research_handler.py research_utils.py personal_docs.py -rag_manager.py rag_singleton.py topic_analyzer.py -visual_report.py youtube_handler.py pdf_forms.py -pdf_form_doc.py pdf_runtime.py caldav_writeback.py -email_thread_parser.py text_helpers.py user_time.py -teacher_escalation.py cookbook_serve_lifecycle.py -chatgpt_subscription.py mcp_manager.py -``` - -### `routes/` (54 files) - -``` -__init__.py _validators.py -auth_routes.py api_token_routes.py device_flow.py -chat_routes.py chat_helpers.py shell_routes.py -codex_routes.py skills_routes.py -email_routes.py email_helpers.py email_pollers.py -cookbook_routes.py cookbook_helpers.py cookbook_output.py -model_routes.py assistant_routes.py copilot_routes.py -calendar_routes.py contacts_routes.py -document_routes.py document_helpers.py -gallery_routes.py gallery_helpers.py -task_routes.py session_routes.py -note_routes.py memory_routes.py research_routes.py -mcp_routes.py search_routes.py history_routes.py -webhook_routes.py workspace_routes.py upload_routes.py -vault_routes.py prefs_routes.py preset_routes.py -signature_routes.py personal_routes.py hwfit_routes.py -backup_routes.py cleanup_routes.py diagnostics_routes.py -embedding_routes.py emoji_routes.py font_routes.py -stt_routes.py tts_routes.py compare_routes.py -editor_draft_routes.py chatgpt_subscription_routes.py admin_wipe_routes.py -``` - -### `core/` (10 files) - -``` -__init__.py constants.py database.py models.py -auth.py middleware.py session_manager.py exceptions.py -atomic_io.py platform_compat.py -``` - ---- - -## Appendix B: Key Import Relationships - -``` -core/database.py ←── 102 importers (routes/*, src/*, core/*, tests/*) - ↑ - ├── routes/auth_routes.py - ├── routes/email_routes.py - ├── src/builtin_actions.py - ├── src/task_scheduler.py - ├── src/tool_implementations.py (inline) - └── ...97 more - -src/tool_implementations.py ←── 17 importers - ↑ - ├── src/agent_loop.py - ├── src/builtin_actions.py - ├── src/tool_index.py - ├── src/task_scheduler.py - ├── src/tool_policy.py - └── ...12 more (mostly tests) - -src/agent_loop.py ←── 22 importers - ↑ - ├── src/tool_policy.py - ├── src/teacher_escalation.py - ├── src/bg_monitor.py - ├── src/task_scheduler.py - └── 18 more (incl. tests) +python -c "from src.tool_implementations import do_search_chats; print('tool facade OK')" +python -c "import routes.gallery_routes as g; import routes.gallery.gallery_routes as c; print(g is c)" +python -c "import routes.memory_routes as m; import routes.memory.memory_routes as c; print(m is c)" +python -c "import routes.research_routes as r; import routes.research.research_routes as c; print(r is c)" +python -c "import routes.history_routes as h; import routes.history.history_routes as c; print(h is c)" +python -c "import routes.contacts_routes as c0; import routes.contacts.contacts_routes as c1; print(c0 is c1)" +python -c "import routes.note_routes as n; import routes.note.note_routes as c; print(n is c)" ``` diff --git a/specs/auth-security.md b/specs/auth-security.md new file mode 100644 index 000000000..47ad6aee1 --- /dev/null +++ b/specs/auth-security.md @@ -0,0 +1,159 @@ +# Auth And Security + +Last updated: dev@e57f60b | 2026-07-20 + +## Scope + +This spec covers current security and trust-boundary behavior in: + +- `core/auth.py`; +- `core/middleware.py`; +- `core/log_safety.py`; +- `core/database.py`; +- `app.py` auth middleware and token cache; +- `src/auth_helpers.py`; +- `src/tool_security.py`; +- `src/tool_execution.py`; +- `src/task_action_policy.py`; +- `src/prompt_security.py`; +- `src/url_safety.py` and `src/url_security.py`; +- `src/host_docker_access.py`; +- `src/attachment_refs.py` and upload lifecycle enforcement in + `src/upload_handler.py` / `routes/upload_routes.py`; +- `src/secret_storage.py`; +- `src/api_key_manager.py`; +- `src/integrations.py`; +- `src/webhook_manager.py`; +- `src/generated_images.py`; +- `scripts/diffusion_server.py`; +- `companion/routes.py` and `companion/pairing.py`; +- `routes/auth_routes.py`, `routes/api_token_routes.py`, `routes/vault_routes.py`; +- admin-gated call sites in route files; +- `THREAT_MODEL.md` and `SECURITY.md`. + +## Trust Boundary + +Odysseus is a trusted-user private-network app. Admins intentionally have powerful local capabilities: shell, files, email, calendar, MCP, model serving, vault, settings, and API token management. The security model prevents unauthenticated access, non-admin escalation, prompt-injection through untrusted content, and accidental exposure of internal services. + +`THREAT_MODEL.md` owns high-level security framing, but implementation claims here should be verified against current code when the threat model is stale. This spec records the implementation map that contributors should check before changing auth or untrusted-context flows. Security-header runtime details live in `runtime.md`. + +## Auth Ownership + +- `core.auth.AuthManager` owns users, password hashing, TOTP/backup codes, reserved usernames, privilege defaults, admin promote/demote state, and auth settings stored in `data/auth.json`. Auth config/setup mutations are lock-guarded, and session tokens are persisted separately in `data/sessions.json` behind their own lock. +- `app.py` owns request-time auth middleware, token-cache rebuild/invalidation, auth exemptions, API-token verification, and internal-tool identity stamping. +- `routes/auth_routes.py` owns HTTP endpoints for setup, signup/login/logout, 2FA, users, privileges, auth features, and integration settings. +- `core.middleware.require_admin()` owns the normal admin gate. Local wrappers must document and test any intentional divergence from that boundary. +- `src.auth_helpers.effective_user()` owns cookie/API-token owner attribution for selected route code. `require_user()` owns route-level degraded user resolution, `require_privilege()` owns privilege checks, and `owner_filter()` owns shared/null-owner query compatibility. + +Reserved usernames include `internal-tool`, `api`, `demo`, and `system`. Loaded auth data drops reserved user records, and create/rename flows must reject real users with those names. + +## Auth Runtime Flow + +`AuthMiddleware` is the outer request gate because FastAPI middleware executes in reverse add order. It can return API `401` JSON or browser `/login` redirects before timeout/security-header middleware reaches the route. + +Public/auth-exempt surfaces are limited to setup, signup/login/logout/status, feature/settings/integration preset reads, health/version/login, `/static/*`, and task webhook trigger paths. `routes/task_routes.py` owns validation of `POST /api/tasks/{task_id}/webhook/{token}` path credentials. + +Login issues an `HttpOnly`, `SameSite=Lax` cookie, with `SECURE_COOKIES` opt-in and a seven-day max age when "remember" is enabled. TOTP is checked before session issuance. Logout, password changes, user deletion, rename flows, expired sessions, and deleted-user sessions must keep revocation/migration behavior intact. + +Deleting a user revokes that user's browser sessions and API-token rows, then the admin delete route invalidates the in-memory bearer-token cache so already-cached tokens stop authenticating. + +Rename first changes the auth username, then migrates owner-bearing DB rows and disk-backed stores. Current rename coverage includes user preferences, active/disk research state, `memory.json`, upload metadata and owner-qualified upload index keys, skills frontmatter/usage state, cached browser sessions, and API-token cache invalidation. If owner migration fails after the auth rename, the route attempts to roll auth back to the old username instead of leaving a split identity. + +Admin promotion/demotion is a live auth flag change through `AuthManager.set_admin()` and `PUT /api/auth/users/{username}/admin`. Demotion refuses to remove the last admin, permits self-demotion when another admin remains, restores the pre-admin privilege map when available, and does not revoke sessions or API tokens because later admin checks read the current `is_admin` flag. + +## Owner Attribution + +Cookie requests use the real username. Bearer-token requests are stamped as `request.state.current_user = "api"` plus `api_token_owner`, `api_token_scopes`, and token id. Routes that support API-token access must explicitly use `effective_user()` or route-local scope helpers instead of treating `"api"` as an owner. + +Internal loopback calls may stamp `current_user = "internal-tool"` or a validated `X-Odysseus-Owner` username. Network/proxy validation for that bypass lives in `app.py`; `require_admin()` trusts the stamped sentinel or raw internal header and should be used behind equivalent middleware control. + +Missing-owner values are state-dependent and are not one canonical identity: + +- Auth-enabled, configured auth with no `current_user` is unauthenticated and should fail closed at route dependencies. +- `AUTH_ENABLED=false` is an explicit local single-user/no-login mode. Route helpers return `""`, and admin gates allow the local operator. +- Chat/agent code that reads `get_current_user(request)` directly gets `None` when auth middleware is disabled, because no middleware stamps request state. +- SQL `NULL`/JSON missing owners remain legacy/shared compatibility data, not the same thing as a logged-out authenticated caller. +- `"api"` and `"internal-tool"` are request sentinels. They must not be persisted as normal storage owners unless a route explicitly defines that behavior. + +Authenticated `manage_tasks` mutations require an exact stored task-owner +match and reject both cross-owner and legacy null-owner rows. The `owner=None` +agent path keeps deliberate auth-disabled single-user compatibility, including +unscoped list/create/mutation behavior. + +Owner-scoped route code should use `require_user()` or equivalent policy before querying per-owner data. Current note CRUD/reorder/reminder routes do this so an auth-enabled request that reaches the route without identity returns `401` instead of falling into single-user/null-owner compatibility behavior. + +## API Tokens And Scoped Integrations + +`routes/api_token_routes.py` owns token CRUD and scope normalization. Partial updates preserve existing scopes unless new scopes are supplied, write scopes imply the matching read scopes where applicable, and Cookbook scopes are part of the normalized scope set. `app.py` caches active token prefix rows and verifies bearer tokens with bcrypt. API-token requests set `request.state.current_user = "api"` plus token owner/scopes. + +Current call sites include Codex/Claude scoped APIs, `/api/v1/chat`, webhooks, selected session routes, companion pairing, and external integrations. `/api/codex/*` and `/api/v1/chat` enforce route-local scopes; companion and selected session routes use owner attribution. `companion/pairing.py` can mint chat-scoped tokens outside normal token CRUD. + +Admin token CRUD is cookie/admin gated. Update/delete operations check token ownership, and cache rebuild ignores active tokens whose owner no longer maps to a known auth user. Scoped route code must use the token owner and declared scopes instead of falling back to cookie-user assumptions. + +## Internal Tool Loopback + +Agent tools call admin-gated HTTP routes through an in-process loopback. `core.middleware.INTERNAL_TOOL_TOKEN` owns the random per-process secret. `app.py` only accepts this bypass from direct loopback clients without proxy-forwarding headers. + +`src.tool_security` owns non-admin tool blocking. Non-admin users must not reach admin tools through agent mode, MCP tools, or loopback calls. + +`src.tool_security.owner_is_admin_or_single_user()` treats explicit `AUTH_ENABLED=false` as intentional single-user mode even when an auth store already exists, while keeping pre-setup auth-enabled callers non-admin. + +Current admin gates include `require_admin()` call sites across admin wipe, backup, contacts, Cookbook, diagnostics, embeddings, MCP, model, personal docs, presets, skills, uploads, vault, webhook, and companion routes. Local wrappers also exist in auth routes, shell routes, and task action policy; changes to those wrappers need the same trust-boundary review as `require_admin()`. Scheduled task action policy treats `run_local`, `run_script`, `ssh_command`, and `cookbook_serve` as admin-only action tasks across create/update/manual-run/webhook/scheduler execution. + +## Untrusted Context Policy + +`src.prompt_security` owns the model-facing untrusted data contract: + +- `UNTRUSTED_CONTEXT_POLICY` states the policy in system prompt text. +- `untrusted_context_message(label, content)` wraps external content as user-role data with `metadata.trusted = False`. + +Current untrusted surfaces include fetched URLs, web results, emails, memories, skills, notes, documents, active editor content, and tool output sourced from outside the server. Injecting those as trusted system instructions is a security bug. + +## URL, Path, And Secret Policy + +- `src/url_security.py` owns public HTTP(S) validation for integration/API-token supplied URLs. It should fail closed for private IP, loopback, invalid scheme, and unsafe redirect targets. +- `src/url_safety.py` owns local-first outbound URL safety for model endpoints and similar local services. Loopback/LAN can be allowed by default, and private-IP blocking is an explicit caller policy. Strict `block_private=True` also rejects RFC 6598 shared/CGNAT space (`100.64.0.0/10`) explicitly because Python does not classify that range as private. +- `core.log_safety.redact_url()` strips URL userinfo, query strings, and fragments before endpoint URLs enter logs. Model, chat/research endpoint, contact/CardDAV, and similar diagnostics should use this helper instead of logging raw admin-configured URLs. +- `src.webhook_manager` validates webhook URLs at create and delivery time, + rejects private/internal targets, disables redirects, and pins delivery to + the public IP set that passed validation immediately before the request. +- `src.integrations` owns admin-configured integration base URLs and secret + masking. `api_call` accepts only relative paths, rejects link-local/metadata + destinations through `src.url_safety`, and can additionally block + RFC1918/loopback/private targets with `INTEGRATION_API_BLOCK_PRIVATE_IPS=true`. +- `services.search.content` validates every redirect hop, rejects private/local + resolved addresses, and pins the HTTP connection to the validated public IP + while preserving original URL/SNI/Host semantics. +- Path-based tools, upload/document/gallery/signature/generated-image routes, embedding cache paths, and research JSON helpers must stay confined to allowed roots and owner-scoped files. Native file/code-navigation tools also apply a case-insensitive sensitive-path denylist so `grep`, `glob`, `ls`, direct reads, and writes cannot reveal `.env`, SSH/GPG material, private-key filenames, or similar secret paths. +- Durable upload references are owner-reserved before chat/session, document, + note, or calendar writes. Cleanup scans every current durable reference + surface and fails closed on incomplete discovery or inconsistent upload-index + state rather than deleting a possibly live upload. +- File-backed SQLite startup restricts `app.db` and existing rollback/WAL/SHM + sidecars to `0600` on POSIX after resolving the real path from the parsed + engine URL. Windows, in-memory, and non-SQLite databases are excluded, and + failed POSIX restriction is logged as a secret-file warning. +- Secret-like DB columns use `EncryptedText` or `src.secret_storage`. Email passwords and Google OAuth mail tokens are encrypted manually in `EmailAccount` string columns; Google OAuth state is HMAC-signed and callback writes are owner-checked before token storage. `src.api_key_manager` keeps provider API keys encrypted in `data/api_keys.json`, writes by loading the raw encrypted dict so saving one provider does not rewrite other providers' keys as plaintext, and restricts local key-file permissions where the platform supports chmod. Vault state in `data/vault.json` is a chmod-restricted JSON secret store, not Fernet-encrypted DB storage. Do not log or return decrypted secrets except for intentional admin vault retrieval flows with audit/reason checks. +- `.env` files are secrets-only inputs and should not be read or printed during agent work. + +`scripts/diffusion_server.py` is a local model-serving helper with its own web surface. It defaults CORS to deny, installs a trusted-host allowlist for loopback/bind addresses, and only extends Host/CORS through explicit CLI flags. + +Host Docker socket access is a high-trust admin/deployment choice, not a normal container capability. Default Docker Compose does not mount `/var/run/docker.sock`; `src.host_docker_access` only reports local Docker available inside a container when `ODYSSEUS_ENABLE_HOST_DOCKER=true` and the socket exists. Remote SSH Docker/Cookbook workflows remain the safer default. + +## Degraded And Compatibility Behavior + +- `AUTH_ENABLED=false` skips `AuthMiddleware` and `src.auth_helpers.require_user()` returns `""` from any host. This preserves local single-user/no-login operation; it is not permission for auth-enabled logged-out callers. Route code should still avoid assuming a non-empty owner, and chat/agent code must handle that direct `get_current_user()` reads return `None` in this mode. Owner-scoped routes that tolerate no-login mode should still call `require_user()` so auth-enabled anonymous requests fail closed. +- First-run setup mode redirects browser requests to `/login`, returns API `401 Setup required`, and keeps setup/status/login surfaces auth-exempt. Setup/signup/login are rate-limited; status is exempt but not rate-limited. Route helper fallbacks only tolerate unconfigured anonymous access from loopback. +- User privilege checks distinguish legacy empty `allowed_models=[]` from explicit no-model access through `allowed_models_restricted=True`. +- `LOCALHOST_BYPASS` in `app.py` only applies to direct loopback clients and excludes proxy/tunnel headers. Helper fallback code is weaker and should not be treated as the primary bypass boundary. +- Legacy migrations claim null-owner SQL/JSON data for the primary admin when possible, and startup repeats a null-owner sweep hourly. Remaining null-owner rows are surface-specific compatibility data that must be deliberately included, no-oped for single-user mode, or rejected for strict ownership gates. +- `.env` is loaded with `utf-8-sig`, so Windows BOM auth flags still parse. + +## Current Gaps + +- There is no shell/filesystem sandbox for admin tools. +- Token scopes remain coarse for some surfaces. +- `app.py` AuthMiddleware lacks direct regression coverage for bearer-token state/cache behavior, trusted-loopback proxy-header rejection, and internal-tool owner stamping. +- Codex/Claude scoped route enforcement and untrusted tool-result reinjection need stronger regression coverage. +- `THREAT_MODEL.md` still has stale token-scope and `/api/v1/chat` SSRF gap text that should be reconciled with current route validation. +- The no-login owner model is split across route helper `""`, chat/agent `None`, SQL/JSON null-owner compatibility, and calendar fallback owner behavior. It has targeted tool-access coverage, but still needs a canonical cross-domain policy. diff --git a/specs/calendar-tasks-notes.md b/specs/calendar-tasks-notes.md new file mode 100644 index 000000000..124da8310 --- /dev/null +++ b/specs/calendar-tasks-notes.md @@ -0,0 +1,185 @@ +# Calendar, Tasks, And Notes + +Last updated: dev@e57f60b | 2026-07-20 + +## Scope + +This spec covers calendar, reminders, tasks, assistant runs, and notes in: + +- app route wiring, auth exemptions, and scheduler startup in `app.py`; +- canonical database models in `core/database.py`, with `src/database.py` as a compatibility re-export; +- `routes/calendar_routes.py`, `src/caldav_sync.py`, and `src/caldav_writeback.py`; +- `routes/task_routes.py`, `src/task_scheduler.py`, `src/task_endpoint.py`, `src/event_bus.py`, and `src/interactive_gate.py`; +- shared privileged task-action policy in `src/task_action_policy.py`; +- `routes/assistant_routes.py`; +- canonical `routes/note/note_routes.py`, compatibility shim + `routes/note_routes.py`, `src/builtin_actions.py`, and `src/action_intents.py`; +- agent/tool call sites in `src/tool_index.py` and `src/tool_implementations.py`; +- scoped Codex wrappers in `routes/codex_routes.py`; +- database models `CalendarCal`, `CalendarEvent`, `ScheduledTask`, `TaskRun`, `Note`, and `CrewMember`; +- direct DB CLIs `scripts/odysseus-calendar`, `scripts/odysseus-notes`, and `scripts/odysseus-tasks`; +- frontend modules `static/js/calendar.js`, `static/js/calendar/*`, `static/js/tasks.js`, `static/js/notes.js`, and `static/js/assistant.js`; +- tests covering calendar routes/utilities, CalDAV, recurrence, timezone handling, scheduler behavior, task webhooks, notes CLI/tool behavior, and task CLI behavior. + +## Calendar + +`routes/calendar_routes.py` owns `/api/calendar` behavior: config, multi-account CalDAV CRUD, connection test, sync, local calendar CRUD, event CRUD, recurrence expansion, ICS import/export, quick parse, and user timezone offset handling. + +`src.caldav_sync` owns CalDAV fetch/sync. `src.caldav_writeback` owns pushing local changes back to remote calendars. Calendar routes request those behaviors; they do not own CalDAV protocol details. + +Runtime behavior: + +- local default calendars are created per owner as needed; +- route-level no-login calendar access normalizes empty owner values to `ODYSSEUS_FALLBACK_OWNER` or `owner@localhost`, so route-created calendar rows do not use the empty string as their storage owner; +- CalDAV account config lives in per-user prefs as `caldav_accounts`, with the legacy `/api/calendar/config` route reading/upserting the first account; +- recurring rules are expanded server-side, including compound recurrence IDs; +- RRULE expansion is capped and marks truncated responses; +- event datetimes preserve UTC/local metadata through `CalendarEvent.is_utc` where supported; +- CalDAV pull uses a bounded sync window, scopes existing UID lookups to the synced calendar, stamps account ids and remote metadata on local calendars, maps Google principal URLs to event collections, preserves locally-created or writeback-pending events that are not yet remote-owned, and deletes stale in-window remote events only when remote object parsing did not fail; +- CalDAV writeback stores `remote_href`/`remote_etag`, clears `caldav_sync_pending` only after successful remote writes, and leaves create/update/delete pending markers for retry on failure; +- pull and writeback paths always close their `DAVClient`, including discovery, + database, and remote-write failure paths; +- sync direction can be pull, push, or both, and pending local writeback rows are included even before remote href metadata exists; +- ICS import is per-owner, capped, creates fresh local IDs in the target import calendar, and preserves zero-duration events as visible imported rows rather than dropping them as empty ranges; +- writeback is best-effort and local SQLite remains source of truth when remote writes fail. + +Calendar credentials are encrypted at rest and are not returned to clients. CalDAV URL validation rejects unsafe schemes, credentials, fragments, localhost names, bad ports, unsafe IP literals, and hostnames resolving to disallowed addresses, with `ODYSSEUS_ALLOW_PRIVATE_CALDAV=1` as the explicit private-IP escape hatch. CalDAV sync/writeback clients disable redirects so credentials are not followed to another origin. The connection-test client keeps proxy/environment trust disabled but explicitly loads an operator `SSL_CERT_FILE` or `REQUESTS_CA_BUNDLE` when the file exists so private/self-signed deployments use the same CA trust intent as real sync. + +## Tasks And Assistant Runs + +`src.task_scheduler.TaskScheduler` owns scheduled task execution, next-run computation, strict single-slot execution, queued/running cleanup at startup, overdue next-run advancement, webhook-triggered tasks, notifications, run records, chained tasks, and event-triggered actions. + +Cookbook serve scheduling crosses this domain. The Cookbook UI creates `cookbook_serve` scheduled tasks, can mirror them as Cookbook calendar events with `cookbook_event_uid`, and task deletion cleans up the linked event when present, falling back to exact-summary matching for legacy events without a stored UID. Cookbook command execution/lifecycle details stay in `cookbook-hwfit.md`. + +`routes/task_routes.py` owns task CRUD, status, manual run/stop/cancel, pause/resume, owner-scoped run/activity history, metadata, onboarding defaults, cache clearing, parse endpoints, and webhook-token regeneration. Chained-task `then_task_id` values are validated as same-owner relationships on create/update, and scheduler execution also rejects cross-owner or cyclic chains. + +Task webhook paths are auth-exempt at the app middleware layer only for `/api/tasks/{task_id}/webhook/{token}`. The route still validates active task state plus task-specific webhook token before dispatch. + +Task runtime behavior: + +- task runs move through queued/running/success/error/skipped/aborted states; +- scheduler/background execution can wait for `src.interactive_gate` to report a quiet foreground window, and running background work can use browser heartbeat/chat-stream activity as a cancellation/defer signal where implemented; +- output targets include chat sessions, notifications, email, and MCP delivery paths; +- LLM and research tasks can carry a built-in `character_id` persona prompt that the scheduler prepends at execution time; +- task-created chat sessions can be foldered under `Tasks`, and startup migration backfills task/research folders for legacy sessions; +- event-bus triggers persist counters and `next_run` before scheduler handoff; +- the in-process scheduler is gated by `ODYSSEUS_INPROCESS_TASKS`, and multiple enabled app processes can double-run work. +- action tasks with `run_local`, `run_script`, `ssh_command`, or + `cookbook_serve` are admin-only. `routes.task_routes` enforces this on + create/update/manual run and hides those actions from `/meta/actions` for + non-admin owners; webhook and scheduler execution pause the task and clear + `next_run` if an admin-only action belongs to a non-admin owner. +- background LLM task execution uses the background workload path, and the + scheduler can abort/cancel active in-process task runs when foreground browser + activity appears. + +`routes.assistant_routes.py` owns crew/assistant settings and run-status surfaces that use the scheduler. `TaskScheduler.ensure_assistant_defaults()` currently seeds the personal assistant crew member and pinned assistant session, but no longer auto-creates Morning/Midday/Evening check-in tasks. Existing crew-linked check-in tasks are still rendered and managed when present. + +## Notes And Reminders + +`routes.note.note_routes` owns notes/todos/reminders, and `app.py` imports that +canonical path. `routes.note_routes` replaces its module entry with the +canonical module for legacy import and monkeypatch compatibility. Notes are +SQLAlchemy `Note` rows and can include due dates, ordering, images, repeat +state, AI classification, source/session provenance, and agent session +linkage. + +Notes CRUD/reorder/reminder routes resolve the acting owner through `require_user()`: auth-enabled anonymous requests fail closed before hitting owner-scoped queries, while documented no-login/single-user modes still resolve to the compatibility owner path. + +Reminder policy: + +- "remind me at 5pm" should become a todo/note with a due date; +- calendar event alarm/reminder UI writes reminder Notes; +- calendar events are for scheduled time blocks, meetings, appointments, or explicit calendar requests; +- creating a calendar event named "Reminder" does not create notification behavior. + +Built-in reminder/persona prompt text is mirrored server-side for reminder synthesis and scheduled task execution; frontend persona selectors are UI over that server-owned id map, not the authority. + +Reminder dispatch is Note-owned: + +- `dispatch_reminder()` owns browser, email, ntfy, generic webhook, in-app notification, optional LLM reminder text, and dedupe behavior; +- the scheduler note scanner calls note-ping actions for backend due-note delivery with per-owner notification state, and calendar-event reminders are treated as Note-owned reminders rather than separate scheduler event pings; +- the notes frontend has a browser-tab fallback for visible sessions; +- calendar frontend reminder UI stores reminder records as Notes, not calendar-event notification jobs. + +Email/ntfy failures degrade into channel result fields rather than blocking every reminder path. ntfy and generic webhook reminder URLs run through outbound URL safety checks, with `REMINDER_WEBHOOK_BLOCK_PRIVATE_IPS` controlling whether private/LAN targets are allowed. ntfy notification titles are converted to ASCII with replacement and capped at 200 characters before entering HTTP headers. Reminder dedupe uses owner-scoped cache files under `data/`. + +## Agent, Codex, And CLI Surfaces + +`do_manage_tasks`, `do_manage_notes`, and `do_manage_calendar` own agent-side writes. `do_manage_calendar` supports batch event creation plus list range aliases (`start`, `start_time`, `start_date`, `range_start`, `from`, `dtstart`, `since`, and matching end aliases), calendar name/short-id lookup, importance/tag aliases, and reminder offsets expressed as numbers, minute/hour words, or common abbreviations such as `min`/`mins`/`hr`/`hrs`. If a model supplies a loose `query`, `date_range`, or `range` without explicit start/end datetimes, `list_events` returns an error asking the caller to resolve the range and call again instead of guessing. Event classification reads `Memory.text` for personal context before LLM classification. `src.tool_index` encodes the reminder policy that notes/todos own reminders while calendar events own time blocks. + +Agent native tool owner handling is not uniform today. `do_manage_tasks()` filters lists only when `owner` is truthy and creates tasks with the passed owner, so `owner=None` can create legacy/null-owner tasks. For authenticated/non-empty owners, edit/delete/pause/resume/run require an exact stored owner match and reject both cross-owner and null-owner rows; `owner=None` retains single-user compatibility. `do_manage_notes()` list/query behavior distinguishes `None` from `""`, with `None` acting as broader single-user compatibility while `""` filters to empty-owner rows in some paths. `do_manage_calendar()` query helpers filter only when owner is not `None`, while calendar creation routes through the calendar fallback owner for default calendars. These are compatibility behaviors, not a cross-user sharing model. + +Note and calendar route/tool writers owner-reserve any canonical internal upload +references in content, checklist/color/image fields, descriptions, and +locations before their database writes. Missing or wrong-owner uploads fail the +write instead of creating a dangling durable reference; reservations serialize +with upload cleanup. + +Chat forwards browser timezone offset so natural-language note/calendar tools can anchor dates to the user clock. Chat can auto-promote note/calendar/reminder intents to agent mode. + +Codex todo/calendar wrappers enforce bearer-token owner and `todos:*` or `calendar:*` scopes, then delegate to note/calendar behavior as the token owner. Normal calendar/task/note routes are current-user/cookie routes and should not be treated as scoped bearer-token APIs unless they explicitly use token owner/scope policy. + +Direct DB CLIs are local compatibility tools. They bypass HTTP route behavior, CalDAV writeback, and some owner/timezone parsing policy. + +## Event Bus + +`src.event_bus` owns event-triggered task counters and scheduler handoff. Current emitters include chat/session/document/memory/research/email/skill paths. Ownerless events resolve to a primary configured user instead of broadcasting to every owner. + +The current event bus is not a calendar-event emitter despite the adjacent calendar/task/reminder domain. + +## Timezone And Date Semantics + +- calendar events store offset-aware input as UTC/naive fields plus `is_utc`; +- note `due_date` uses ISO-like strings interpreted through note/tool parsers; +- chat forwards browser UTC offset into `routes.calendar_routes` request-local state for natural-language date anchoring in calendar/note tool parsing; +- generic scheduled task clock times are stored as UTC values after local conversion; +- assistant check-ins can use an IANA timezone on `CrewMember`, with UTC fallback. + +Dateutil fallbacks strip timezone-aware parser results back to the naive-UTC contract before recurrence/window comparisons. Calendar agent list tools accept current range aliases implemented by `src.tool_implementations`, and equal/same-day start/end ranges are normalized to a one-day window instead of silently returning no rows. + +Natural-language parsers prefer time-first interpretations for short reminder/event phrases where the user supplies a clock time before a date phrase. + +Calendar frontend week-start preference is browser-local (`cal-week-start`) with Monday/Sunday controls; it is not persisted as a server preference. + +Natural-language date parsing and timezone behavior are compatibility-sensitive and need route/tool/frontend regression coverage when changed. Request-local timezone context is ephemeral and must not be persisted as user state. + +## Degraded And Optional Behavior + +- CalDAV sync no-ops with shaped errors when unconfigured, invalid, offline, or missing the optional `caldav` dependency. +- CalDAV writeback failures are non-fatal to local calendar writes and are mostly visible through logs. +- Missing or invalid `croniter` rejects cron schedules or yields no next run. +- Missing timezone support falls back to UTC or legacy behavior. +- ICS import depends on `icalendar`; missing dependency can fail before route-shaped error handling today. +- Notes reminders can still use local browser fallback when backend email/ntfy channels fail. +- App backup import/export does not currently include calendar events, scheduled tasks, task runs, or notes; calendar ICS import/export is separate and calendar-only. + +## Security And Provenance + +Calendar, task, note, and assistant routes are owner-scoped for normal users. Legacy null-owner behavior is compatibility-sensitive and should not silently grant authenticated owners broad mutation rights. + +Because auth-disabled chat owners can arrive as `None`, tool-created rows may not use the same owner value as route-created rows. Multi-user or owner-model changes must audit both route and agent paths. + +Task creation/update/manual run/webhook/scheduler execution blocks shell-like and Cookbook serve action types for non-admin users through `src.task_action_policy`, and tool security blocks privileged task/calendar tools for non-admin use. Assistant defaults reject synthetic owners such as `api` and `internal-tool`. + +Note routes store caller-provided `source`, `session_id`, `image_url`, and agent-session provenance. Canonical internal upload references in persisted note/calendar fields are owner-reserved before writes, and upload-backed bytes remain protected when fetched through upload routes. Arbitrary non-upload image/provenance URLs are not otherwise normalized or validated by note storage. + +## Testing Coverage + +Existing coverage is strongest around CalDAV URL hardening/writeback, client cleanup and operator CA handling, bidirectional/pending CalDAV sync markers, CalDAV UID calendar scoping, calendar recurrence/timezone helpers, owner-scoped calendar basics, exact-owner task-tool mutations, scheduler restart/cancel/next-run behavior, webhook auth-exemption source shape, canonical/legacy note-module identity, note-route unauthenticated fail-closed behavior, note/calendar attachment reservations, notes CLI/tool due-date behavior, calendar reminder abbreviation parsing, task CLI preview, task persona fields, and same-owner chained task validation. + +Route-level coverage is thinner for full calendar route behavior, task CRUD/security/run controls, live webhook token dispatch, notes owner CRUD/reminder delivery, assistant defaults/run status, event-bus triggers, Codex todo/calendar scopes, and frontend panel wiring. + +## Current Gaps + +- CardDAV still needs URL hardening parity with CalDAV; CalDAV now resolves hostnames during validation and revalidates writeback URLs. +- `do_manage_notes()` should match HTTP note-route owner behavior for legacy null-owner notes. +- Auth-disabled agent tools can produce or read broader owner scopes than route handlers because they receive `owner=None`; tasks, notes, and calendar need aligned policy/tests. +- Task webhook tests should keep exercising live route token behavior and + admin-only action blocking, not only middleware/source strings. +- Reminder delivery needs tests across frontend `/fire-reminder`, backend `dispatch_reminder()`, scheduler note pings, channel degradation, and dedupe. +- Codex todo/calendar scope and owner mapping needs dedicated regression coverage. +- Direct DB CLIs need either documented route-bypassing support status or shared helpers to avoid owner/timezone/writeback drift. +- `scripts/odysseus-webhook` appears to reference an older webhook URL shape. +- Assistant default documentation/code comments still mention check-ins that are no longer auto-seeded. +- App backup import/export does not cover the calendar/task/note rows described by this spec. diff --git a/specs/chat.md b/specs/chat.md new file mode 100644 index 000000000..1f71f095c --- /dev/null +++ b/specs/chat.md @@ -0,0 +1,145 @@ +# Chat + +Last updated: dev@e57f60b | 2026-07-20 + +## Scope + +This spec covers current chat behavior in: + +- `routes/chat_routes.py` and `routes/chat_helpers.py`; +- `routes/session_routes.py` and canonical `routes/history/history_routes.py`, + with `routes/history_routes.py` as a compatibility shim; +- `src/chat_helpers.py`; +- `src/agent_runs.py`; +- `src/chat_handler.py` and `src/chat_processor.py`; +- `core/session_manager.py` and `core/models.py`; +- `src/attachment_refs.py` and `src/upload_handler.py` for durable attachment + references and write reservations; +- `src/context_budget.py`, `src/context_compactor.py`, and `src/topic_analyzer.py`; +- `routes/workspace_routes.py` for workspace selection support; +- frontend modules `static/js/chat.js`, `static/js/chatStream.js`, `static/js/chatRenderer.js`, `static/js/sessions.js`, `static/js/search-chat.js`, `static/js/compare/stream.js`, `static/js/workspace.js`, `static/js/composerArrowUpRecall.js`, `static/js/streamingSegmenter.js`, `static/js/group.js`, and `static/js/notes.js`; +- integration points with uploads, documents, compare, research, agent tools, memory, RAG, search, and model endpoints. + +## Session Ownership + +`core.session_manager.SessionManager` owns session persistence and message writes. `routes/session_routes.py` owns session list/create/update/archive/delete/folder/importance behavior for the sidebar. `routes.history.history_routes` owns history/topic surfaces, with `routes/history_routes.py` kept as a compatibility shim. + +`core.models.Session` and `ChatMessage` are pure data containers. They do not own persistence; `Session.add_message()` delegates to the configured session manager when present. + +## Streaming + +`routes/chat_routes.py` owns `/api/chat`, `/api/chat_stream`, detached stream resume/stop/status, injected context, chat-message search, and rewrite routes. Streaming is the main UI path. + +`static/js/chat.js` owns send/abort/continue UI state, the main fetch/read loop, SSE parsing, rendering dispatch, workspace form wiring, and background/resumable stream tracking. `static/js/chatStream.js` owns UI-control event handling and stream/research notification helpers. `static/js/sessions.js` polls server stream status after refresh or session switch. `static/js/composerArrowUpRecall.js` owns prompt recall from the composer when the caret is at the top of an empty input. + +Runtime behavior: + +- the `/api/chat*` prefix is exempt from the global request hard timeout; +- browser chat sends `X-Tz-Offset`; route code forwards it into `routes.calendar_routes` request-local state so note/calendar tool parsing can anchor natural-language dates to the user clock; +- browser chat can send a selected workspace path; route code only resolves it for admin/single-user flows, validates it as an existing directory, and forwards it so agent file/shell tools are confined by `src.tool_execution`; +- stream callbacks can outlive a deleted session, so persistence must fail closed instead of recreating orphan messages; +- message metadata carries timestamps, metrics, tool events, sources, hidden + thinking/reasoning text when providers expose it separately, context-trim + metrics, structured attachment references, and related UI state; +- metadata preserves both requested and actual reply models when provider streams or fallbacks report them, and stable session ids are kept available so prompt/sequence-memory and KV-cache paths can address the same conversation consistently; +- multimodal content can be a list of content blocks for the live provider call, + while persistence collapses raw media into readable text and stable + attachment-reference lines; +- agent streams forward explicit round-cap, tool-budget, repeated-tool-loop, + and intent-without-action guard events so the frontend can distinguish a + controlled stop from a stalled response. + +`src.agent_runs` owns detached in-memory stream runs, replay buffers, replacement cancellation, resume subscribers, explicit stop, and terminal-buffer eviction. Closing the SSE connection does not necessarily stop generation. `static/js/chat.js` can live-resume a still-running detached stream through `/api/chat/resume/{session_id}`; rich responses reload from DB for canonical rendering. Detached runs are process-local and do not survive server restart. + +Provider adapters live below chat in `src.llm_core`. Chat consumes normalized SSE output, fallback/error events, reasoning/tool deltas, and metrics. Model fallback only commits a candidate after substantive text/reasoning or tool-call output; metadata-only and empty/DONE-only streams can advance to the next candidate without exposing stale metadata. After substantive output, errors are surfaced to the stream instead of silently retrying a new model. + +## Context Preface + +`routes.chat_helpers.build_chat_context()` owns the shared route pipeline: preset extraction, preprocessing, user-message persistence, incognito/no-memory/RAG/skills flags, prefetched compare search, YouTube transcript context, research-spinoff grounding, model normalization, and compaction. + +`src.chat_processor.ChatProcessor.build_context_preface()` owns source preface construction. It can add memory, RAG, web search, URL page content, and skills index context before the model call. + +Chat preface enhances the model's context. It must not rewrite the user message or force literal-vs-fetch interpretation before the model sees the request. See [context-building.md](context-building.md). + +Chat-owned external context must enter the model through `untrusted_context_message()` unless a different treatment is explicitly documented. This includes memory, RAG, web search, URL fetches, prefetched search context, YouTube transcripts, research injection, and manual context injection. + +## Modes And Handoffs + +Chat can dispatch to normal LLM calls, agent mode, research mode, or compare-related flows. Session mode is stored on `sessions.mode`. + +Legacy plan-mode backend plumbing still exists below chat, but `routes/chat_routes.py` currently forces browser/form `plan_mode` input off and the old visible plan window frontend module is not part of the current SPA. Treat plan-mode changes as compatibility work unless the UI contract is intentionally reintroduced. + +Current call sites include: + +- chat/research dispatch in `routes/chat_routes.py`; +- agent execution in `src/agent_loop.py`; +- deep research orchestration in `src/research_handler.py`; +- compare entry points in `routes/compare_routes.py` and frontend compare modules. + +Agent-mode tool access is gated in layers. Chat route toggles and privileges +build a disabled-tool set; incognito and compare mode remove persistence-heavy +or UI-breaking tools; `src.action_intents.message_needs_tools()` provides +conservative regex auto-escalation hints; `src.agent_loop`, +`src.tool_security`, `src.tool_execution`, and internal loopback validation +remain server-side enforcement owners. + +`allow_bash` and `allow_web_search` can be read from the JSON request body for browser chat posts that do not submit traditional form fields. + +Web search tools are per-turn explicit opt-in. Either `allow_web_search=true` +or `use_web=true` can enable `web_search`/`web_fetch`, but an explicit +`allow_web_search=false` wins over `use_web=true` and keeps those tools +disabled. Explicit latest-turn web-search intent can still auto-escalate into +agent mode and narrows the available tool set toward `web_search`/`web_fetch`, +but it no longer re-enables web tools after an explicit denial or global +disable. + +Guide-only/no-tools requests build an effective tool policy before preprocessing and agent dispatch. That policy suppresses tool-backed preprocessing/background extraction/research, disables schemas and MCP for the turn, and is still enforced by `src.tool_execution` if a model emits a tool call anyway. + +When route context is trimmed without full compaction, chat emits a +`context_trimmed` SSE event and carries before/after message/token counts into +metrics. Provider reasoning/thinking deltas are streamed for live UI handling +but kept out of the visible saved assistant content and stored in metadata when +available. + +## Attachments + +`src.chat_handler.ChatHandler.preprocess_message()` owns owner-scoped upload-id resolution, attachment metadata, YouTube transcript/comment preprocessing, image/VL behavior, and enhanced text used by chat. `src.document_processor.build_user_content()` owns conversion of uploaded/chat-attached files into model-ready text or multimodal blocks. `src.attachment_refs` owns persisted text/reference normalization, and `SessionManager` owner-reserves attachment ids before appending or replacing durable message rows. `static/js/fileHandler.js` owns frontend pending-file state. + +Attachment-only sends are valid. Missing or unauthorized ids are skipped during preprocessing, while a missing/wrong-owner durable reference aborts a message/history replacement before existing transcript rows are removed. Upload failures keep pending files for retry, unsupported media can degrade to text markers, optional Office/PDF/VL dependencies can emit extraction banners, Office attachments can create markdown documents when extracted server-side, and fillable-PDF auto-document failures fall back to normal PDF extraction. `chat_messages.content` and FTS do not retain provider data URLs; structured references stay in metadata for reloads. Chat does not own upload bytes or durable document storage; it requests document/upload behavior from those subsystems. + +Frontend chat distinguishes normal resend from regenerate-from-here: normal resend appends a fresh user copy and carries upload IDs where available, while regeneration truncates from the selected point. AI-message delete prompts before removing the AI response plus preceding user turn. Desktop Enter submits; mobile Enter inserts a newline unless another platform-specific send control is used. + +Native document tool outputs can open or refresh the document editor from +tool-result metadata, so the UI can recover if a later `doc_update` stream event +is missed. The chat renderer also hides raw/incomplete leaked tool JSON and +document fences from normal transcript text. + +## Security And Provenance + +`/api/chat` and `/api/chat_stream` verify session ownership before loading the session. Chat privilege gates enforce allowed models and daily message caps before LLM work. Active document injection, session auth/header recovery, endpoint repair, upload-id resolution and reservation, memory/RAG retrieval, and post-response work must stay owner-scoped. + +The scoped API-token chat surface is `/api/v1/chat`. Browser chat routes can receive bearer-auth state from middleware, but route code must not assume `"api"` is a durable owner; API-token support requires explicit scope checks and token-owner attribution. + +Incognito disables memory, skill, and chat-history tools and skips assistant DB persistence, but current user-message persistence and later cleanup are not a strict no-write guarantee. Treat incognito changes as security-sensitive until that contract is clarified. + +## Search Boundary + +`GET /api/search` in `routes/chat_routes.py` is chat-message search for the UI and slash commands. Web search routes are owned by `routes/search_routes.py`; chat and agent web context call through `src.search`, compatibility shims, and search content fetchers. Do not confuse chat-history search with external web retrieval. + +## Degraded And Compatibility Behavior + +- Missing ChromaDB, embeddings, memory vectors, RAG managers, or skills indexes should remove injected context or fall back to keyword/text behavior without failing chat. +- Sessions hydrate legacy string headers and multimodal JSON-array content, export text/HTML/Markdown after flattening non-string blocks, can lazy-load from DB when cached state is empty, and preserve old history/index delete behavior where needed. +- Chat repairs empty selected models and orphaned endpoint references before provider calls when possible. +- Deleted-session stream writes fail closed. +- Docker/native endpoint differences are owned by runtime/model setup, but chat sessions depend on the saved endpoint URLs and headers. +- Copying a response from the UI copies the displayed answer text and omits hidden reasoning/thinking segments. + +## Current Gaps + +- Chat, agent, research, and compare orchestration still meet in a large route file. +- Context preface behavior is spread across `routes/chat_helpers.py`, `src/chat_processor.py`, route injections, and agent/tool paths. +- Detached stream lifecycle spans `routes/chat_routes.py`, `src/agent_runs.py`, `static/js/chat.js`, `static/js/sessions.js`, and non-chat callers. +- Some frontend stream state is still global/module-level in `static/js/chat.js` and needs careful session isolation when adding background or resumable flows. +- Chat lacks route-level SSE regression tests for `/api/chat_stream`, live resume/stop/status, mode handoff, persistence metadata, partial-save behavior, attachment/doc-update events, browser timezone offset/workspace handling, and literal URL context intent. +- Bearer-token behavior on browser chat routes and incognito persistence need explicit contract decisions and regression coverage. diff --git a/specs/compare.md b/specs/compare.md new file mode 100644 index 000000000..f43500c04 --- /dev/null +++ b/specs/compare.md @@ -0,0 +1,77 @@ +# Compare + +Last updated: dev@df2fad2 | 2026-07-12 + +## Scope + +This spec covers model A/B comparison behavior in: + +- `routes/compare_routes.py`; +- `routes/session_routes.py`; +- `routes/chat_routes.py` and `routes/chat_helpers.py`; +- `routes/model_routes.py`; +- `routes/search_routes.py`; +- `core/database.py` model `Comparison`; +- `src/llm_core.py` and `src/endpoint_resolver.py`; +- frontend modules under `static/js/compare/`; +- `static/js/chat.js`, `static/js/sessions.js`, `static/js/models.js`, and `static/js/slashCommands.js`; +- `tests/test_compare_*` and focused blind-compare redaction tests. + +## Runtime Behavior + +The active text compare UI creates ordinary `[CMP]` sessions through `/api/session`, then streams each pane through `/api/chat_stream` with `compare_mode=true`. Search compare is a separate branch: it can query `/api/search/query` directly and its synthesis sessions use ordinary chat streaming without `compare_mode=true`. `static/js/compare/index.js` owns compare orchestration, session creation, execution order, search-mode branching, and export actions. `static/js/compare/panes.js` owns pane add/remove/swap/reroll lifecycle. `static/js/compare/stream.js` owns pane streaming and event rendering. + +`routes/compare_routes.py` owns the `/api/compare` HTTP surface for alternate/legacy start/vote/history/delete behavior and the active `/api/compare/record` vote-summary endpoint. Legacy `/api/compare/start` now uses neutral helper-session names and withholds model identities/mapping from the start response while blind mode is active. It does not own provider-specific payload behavior. + +Current call sites include: + +- `/api/session` compare session creation and cleanup in compare frontend modules; +- `/api/chat_stream` pane execution through chat routes and detached stream infrastructure, streamed directly into panes so upstream generation stops promptly when panes are stopped; +- `/api/models` and probe routes for model/endpoint selection; +- search-provider compare mode through `routes/search_routes.py`; +- `/api/compare/record` as a fire-and-forget backend vote summary, while active scoreboard state is localStorage-backed. + +`Comparison` rows currently persist vote/history metadata: prompt, first model identifiers, winner, blind flag, optional N-model JSON in `blind_mapping`, vote timestamp, and owner. Response and metric columns exist in the schema but are not populated by the active compare UI flow. Compare history must be owner-scoped. + +Frontend compare behavior is split by responsibility: + +- `state.js` owns local compare state; +- `selector.js`, `models.js`, and `probe.js` own endpoint/model selection and probe UI; +- `panes.js` and `stream.js` own paired response rendering; +- `vote.js` and `scoreboard.js` own voting and history display. + +Mobile compare layout collapses multi-pane grids to a single column so panes +remain readable on narrow screens while the desktop grid still uses the +selected column count. + +## Ownership Boundaries + +Compare owns paired evaluation flow and pane state. Chat routes own the actual stream execution path for compare panes. LLM provider code owns model-call mechanics. Session/model routes own endpoint-id resolution, owner-filtered endpoint/model visibility, header copying, and deleted-endpoint failures. + +`compare_mode` in chat strips compare-breaking tools, disables document tools for `[CMP]` sessions, skips some research clarification, and suppresses memory, skill, and webhook side effects after pane responses. + +Compare frontend code is part of the app DOM security surface. Current stream/search rendering sanitizes probe labels and tool labels, constrains search-result links to HTTP(S), uses safe generated-image display sources, and opens compare export/image popups with opener isolation. + +## Policy Notes + +- Current blind compare is UI/API masking until vote/reveal, not a full confidentiality boundary. `[CMP]` session names and session-list model fields are redacted for helper sessions, and legacy `/api/compare/start` withholds model identity/mapping while blind. Client-side selected model state and privileged/local inspection can still expose identity. +- Compare endpoint lists and secondary endpoint lookups use owner filtering so users see and resolve only shared or owned endpoints. +- Non-admin compare session creation must use registered owner-visible endpoints; compare must not allow arbitrary raw endpoint URLs to bypass session-route endpoint policy. +- Prefetched search, URL, RAG, and research context entering compare panes must use the untrusted-context wrapper. +- If endpoint fallback behavior changes, verify each pane still goes to the intended model and that fallback notices/errors are visible in compare panes. + +## Degraded And Compatibility Behavior + +- Missing/offline endpoints are surfaced by model/session routes; chat can clear orphaned endpoint references and recover empty models when possible. +- Compare streams inherit chat's pre-output-only model fallback and provider-normalized SSE events, but compare frontend handling for `event: error` and `type: "fallback"` is thinner than chat's stream path. +- Shared legacy `ModelEndpoint.owner == NULL` rows remain visible through owner filters. Legacy `Comparison.owner == NULL` rows are not treated as shared for authenticated vote/delete/history flows. +- `/api/compare/start` and `/{comp_id}/vote` remain implemented but are not the active frontend path. + +## Current Gaps + +- Blind mode is not a confidentiality boundary; client/local state can still expose model identity before vote. +- `/api/compare/start` accepts raw endpoint URLs and can diverge from `/api/session` endpoint-owner/raw-endpoint policy. +- `src/agent_loop.py` advertises stale compare app API endpoints. +- Compare streaming and chat streaming are separate frontend paths but share model/provider infrastructure; regressions can happen when provider event shape changes. +- Compare frontend needs explicit fallback/error event handling parity with chat streaming. +- Compare tests cover endpoint owner helper behavior, blind compare redaction, and portable JS helpers, but not full active `/api/session` pane creation, frontend pane lifecycle, or SSE fallback/error handling. diff --git a/specs/context-building.md b/specs/context-building.md new file mode 100644 index 000000000..5a0730f9f --- /dev/null +++ b/specs/context-building.md @@ -0,0 +1,111 @@ +# Context Building + +Last updated: dev@df2fad2 | 2026-07-12 + +## Scope + +This spec covers model-context construction in: + +- `src/chat_processor.py`; +- `src/chat_handler.py` and `src/youtube_handler.py`; +- `routes/chat_helpers.py` and context injection in `routes/chat_routes.py`; +- `src/agent_loop.py`; +- `src/tool_execution.py`; +- `src/attachment_refs.py` and uploaded-file manifest construction in + `routes/chat_helpers.py`; +- `src/tool_policy.py`; +- `src/prompt_security.py`; +- URL fetchers in `src/search/content.py` and `services/search/content.py`; +- search orchestration in `services/search/core.py` and the compatibility wrapper in `src/search/core.py`; +- RAG and personal docs in `src/rag_singleton.py`, `src/rag_vector.py`, `src/rag_manager.py`, and `src/personal_docs.py`; +- research flows in `src/deep_research.py`, `src/research_handler.py`, and `services/research/research_handler.py`; +- memory and skills in `src/memory.py` and `services/memory/*`; +- related policy in `THREAT_MODEL.md`. + +## Contract + +Context-building tools gather evidence. They do not own user-intent routing. + +Runtime rules: + +- if external context is available, add it as compact untrusted source data; +- if an attempted source is unavailable and relevant, represent the unavailable state explicitly with source and reason when known; +- preserve the user's original message for the model; +- do not use regex preprocessing to force literal-vs-fetch intent; +- do not disable tools or force a reply style solely because preprocessing found a URL. + +## Untrusted Data + +`src.prompt_security` owns the untrusted wrapper: + +- `UNTRUSTED_CONTEXT_POLICY` states global model policy; +- `untrusted_context_message(label, content)` wraps source content as user-role data with `metadata.trusted = False`. + +Current untrusted context sources include: + +- fetched URLs and web search results; +- webpage content passed into deep-research extraction; +- YouTube transcripts/comments; +- RAG/personal document chunks; +- memories and skills; +- notes and active editor documents; +- emails and attachments; +- tool output from external/user-controlled data. + +Live multimodal provider blocks can contain data URLs, but persisted and +tool-facing context uses stable attachment references. Tool manifests carry an +`odysseus://attachment/` URI and owner-checked read policy; local paths are +compatibility data added only after owner and root-confinement checks. Persisted +chat context keeps readable text/reference lines rather than reinserting raw +media bytes into later turns or search state. + +## URL, Search, And Tool-Derived Context + +Chat URL prefetch and agent `web_fetch` are different paths. Chat prefetch happens before the model call; `web_fetch` is a tool the model may choose later. Both should converge on the same intent: enrich context when content is available, represent unavailable content when it is not, and let the model interpret the user request. + +Search results and fetched pages are evidence. `web_search` should not force a page fetch unless its explicit contract says it does. Failed fetches should not crash chat or silently imply content was read. Canonical search content fetchers can extract readable text from HTML, `text/*`, Markdown, `.txt`, `.json`, and `.jsonl` responses and should return shaped error results for HTTP status failures. URL fetches validate every redirect hop and pin the outbound connection to a public IP resolved during validation, so context-building callers do not need a second DNS-rebinding guard. + +Current behavior is not yet unified: + +- successful chat URL prefetch is wrapped as untrusted context, but failed chat URL prefetch can be dropped; +- agent `web_fetch` returns explicit URL-specific tool errors for timeout, unsupported scheme, fetch failure, or no readable text; +- comprehensive search reports provider-chain failures, but individual page-fetch failures can be logged and omitted; +- YouTube fetching is owned by `ChatHandler`/`youtube_handler`, while `routes.chat_helpers` only wraps the resulting transcript/comment strings. + +`services/search/core.py` owns `comprehensive_web_search()` orchestration. `src/search/core.py` is a compatibility wrapper. `src/search/content.py` now aliases the canonical `services.search.content` module so old imports do not create a second fetch/extract implementation. + +## Tool Result Envelope + +`src.tool_execution` executes and formats tools. Tool output caps live in `src.constants` and are re-exported through older facades; shared native-tool truncation lives in `src.tool_utils`. `src.agent_loop._append_tool_results()` owns model re-entry: native tool calls return as provider-style `role: "tool"` messages, while fenced-tool results can become a bracketed user message. These results are untrusted, but they do not all currently use `untrusted_context_message()` or `metadata.trusted = False`. + +Context budgeting uses known model context windows when available. `src.context_budget` treats the default 6000-token value as an automatic sentinel, scales to a capped fraction of known context length for non-explicit budgets, and leaves unknown windows on conservative defaults. + +Side-effect enforcement lives outside context building. Chat route disabled-tool policy, `src.tool_security`, `src.tool_execution`, and `do_app_api()` block unsafe tool execution; prompt wording alone is not the authority. + +Guide-only/no-tools policy can suppress context acquisition before the model call. `src.tool_policy` feeds chat route preprocessing and agent-loop assembly so tool-backed search/research/memory/RAG/skills/local-context paths are skipped when the latest user turn explicitly forbids tools. + +## Degraded And Optional Dependencies + +- ChromaDB, HTTP embeddings, and FastEmbed are installed/expected in normal setups but must degrade cleanly when a service, package, or embedding backend is unavailable. +- `src.rag_singleton.get_rag_manager()` owns RAG startup retry throttling; `src.rag_vector.VectorRAG` is the live owner-filtered path; `src.rag_manager.RAGManager` is compatibility/backward-compat behavior. +- Memory-vector and tool-index retrieval can fall back to keyword/text behavior when vector stores or embeddings fail. +- Docker compose and native installs use different Chroma host defaults; model endpoint loopback rewriting is owned by model/runtime specs. + +## Current Call Sites Include + +- `ChatProcessor.build_context_preface()` for memory, RAG, web search, URL content, and skills index; +- `ChatHandler.preprocess_message()` and the canonical `services.youtube.youtube_handler` import path for YouTube fetch/format, then `routes/chat_helpers.py` for wrapping prefetched search/Youtube context; +- `routes/chat_routes.py` research context injection; +- `src.agent_loop` for active editor document, skill context, and tool-result reinsertion; +- uploaded-file manifest/reference context for agent tools and later chat turns; +- `src.tool_execution` for `web_search`, `web_fetch`, file, shell, MCP, and other tool outputs; +- `src.deep_research` and research handlers for search/fetch/extract flows used by research jobs, with fetched webpage text wrapped before extraction and analyzed URLs tracked separately from source snippets. + +## Current Gaps + +- URL/search context result shape is not unified across chat prefetch, agent tools, and research. +- Some failed fetch states are still easier for code to drop than to represent explicitly. +- Tool/context wording is spread across schema, prompt, and retrieval surfaces. +- Agent tool-result reinjection lacks a unified untrusted wrapper/metadata envelope across native, fenced, MCP, and app API outputs. +- Source-specific wrapping and unavailable-state behavior need focused tests for chat URL prefetch, literal URL intent, search context, deep-research extraction, RAG/memory/skills, YouTube, and tool results. +- Compare pre-search context is computed but may not be submitted through the current compare stream form. diff --git a/specs/cookbook-hwfit.md b/specs/cookbook-hwfit.md new file mode 100644 index 000000000..2dbfc6824 --- /dev/null +++ b/specs/cookbook-hwfit.md @@ -0,0 +1,194 @@ +# Cookbook And Hardware Fit + +Last updated: dev@df2fad2 | 2026-07-12 + +## Scope + +This spec covers model setup/serving and hardware fit in: + +- app route registration in `app.py`; +- `routes/cookbook_routes.py`; +- `src/cookbook_serve_lifecycle.py`; +- `src/host_docker_access.py`; +- Cookbook package/rebuild/shell integration in `routes/shell_routes.py`; +- `routes/cookbook_helpers.py`; +- `routes/hwfit_routes.py`; +- `services/hwfit/*` and `services/hwfit/data/hf_models.json`; +- durable Cookbook state through `routes.cookbook_helpers.COOKBOOK_STATE_FILE`; +- helper/CLI scripts `scripts/odysseus-cookbook`, `scripts/add_hwfit_models.py`, `scripts/hf_download.py`, and `scripts/diffusion_server.py`; +- Docker overlays `docker-compose.gpu-*.yml`, `docker/gpu.*.yml`, `docker/host-docker.yml`, `scripts/check-docker-gpu.sh`, and `scripts/check-docker-amd-gpu.sh`; +- frontend modules `static/js/cookbook*.js`, including Cookbook running, serve, download, diagnosis, progress, and HW Fit modules; +- tests covering Cookbook helpers, routes, CLI state, package detection, frontend progress, HW Fit services, serve profiles, Docker GPU overlays, and GPU diagnostic scripts. + +## Current Call Sites Include + +- Cookbook modal and state modules in `static/js/cookbook*.js`; +- package readiness/install and rebuild flows through `routes/shell_routes.py`; +- direct shell exec/stream integration used by Cookbook task controls; +- model endpoint setup and serve flows; +- hardware-fit recommendations for model choices; +- image-model recommendations for diffusion serving; +- APFEL/local platform dependency paths where supported; +- Docker GPU helper scripts and compose overlays; +- the `odysseus-cookbook` CLI using the same Cookbook state file. + +## Cookbook Runtime + +`routes.cookbook_routes` owns model download, setup, SSH key, cached model scan, serve, GPU state, kill-pid, state sync, Hugging Face latest lookup, vLLM recipe lookup, serve diagnosis, and task-status endpoints. `src.cookbook_serve_lifecycle` bridges scheduled `cookbook_serve` tasks into serve/stop behavior; task/calendar scheduling ownership stays in `calendar-tasks-notes.md`. + +Access policy is split by surface: + +- download/setup/SSH key/cache scan/serve/GPU/kill/state/task-status are admin/internal-tool surfaces; +- `/api/cookbook/hf-latest` is authenticated-user gated; +- HW Fit routes are authenticated read/probe routes through normal middleware, not admin-only operations; +- bearer API tokens do not satisfy Cookbook admin gates. + +Runtime behavior: + +- POSIX and most remote flows run detached through tmux; +- local Windows uses detached process/log/pid behavior under `%TEMP%\\odysseus-tmux`; +- remote Windows uses PowerShell runner scripts; +- missing `tmux`, `docker`, or serve-engine binaries return shaped errors where possible; +- local Docker inside the Odysseus container is available only when the Docker CLI exists, `ODYSSEUS_ENABLE_HOST_DOCKER=true`, and `/var/run/docker.sock` is actually mounted as a socket; otherwise Cookbook should show the host-Docker access hint and prefer remote SSH Docker workflows; +- model serve auto-registers LLM or image `ModelEndpoint` rows immediately, then frontend readiness probing can repair/create fallback endpoints; +- diffusion-server serves are registered as image endpoints; +- vLLM recipe routes fetch and cache model recipe manifests/YAML from `vllm-project/recipes`, normalize base args/env/dependencies/tool-calling/reasoning variants, and expose compatible strategy metadata for serve setup; +- Hugging Face download/setup paths can detect and persist encrypted HF tokens for later Cookbook/agent use; +- local and remote model paths can contain spaces or non-ASCII characters when helper validation/quoting accepts them; +- task status handles tmux, remote Windows logs, local Windows PID/log files, HF cache completion checks, stale browser-state download guards, pip dependency-install success sentinels, exit-code wrappers, serve diagnosis snapshots, and scheduled serve lifecycle hooks; +- scheduled serve lifecycle stop attempts only persist `status=stopped`, clear `_scheduledStopAtMs`, and delete auto-registered endpoints for sessions whose tmux/remote stop command succeeded or were already gone; failed stop attempts are logged without marking unrelated expired serves as stopped. + +`routes.cookbook_helpers` owns validation and command construction: + +- repository and model IDs; +- local directories, SSH hosts/ports, GPU selectors, and tokens; +- shell quoting for Bash and PowerShell; +- pip/install fallback chains; +- safe environment prefixes; +- serve command validation; +- user-shell PATH bootstrap, Git-Bash drive-path conversion, preflight, and exit-code helpers. + +Cookbook routes request shell/SSH behavior; they do not relax shell security. + +## Shell Dependencies + +`routes.shell_routes.py` owns Cookbook-adjacent package readiness/install, shell execution/streaming, and llama.cpp rebuild endpoints. The Cookbook UI calls these routes for dependency diagnosis, install/update actions, engine rebuilds, and tmux/reconnect/stop/kill flows. Windows uses detached log/PID wrappers where POSIX tmux is unavailable. + +These are admin-only code-execution surfaces and should be reviewed with Cookbook changes even though they are implemented outside `routes.cookbook_routes.py`. + +## State, Secrets, And Provenance + +Cookbook state lives under the shared data dir through the `COOKBOOK_STATE_FILE` constant, normally `data/cookbook_state.json`. Routes and the `odysseus-cookbook` CLI use the same state path. + +State behavior: + +- browser-facing state masks secrets; +- server-side `env.hfToken` is encrypted before storage; +- task payloads strip raw HF tokens; +- browser local storage strips HF token values; +- state POST has anti-wipe guards for server lists; +- state POST rejects stale `done` download state when the latest shard/cache markers still show an incomplete download; +- recent server-side tasks are preserved against stale browser overwrites; +- task-status validates saved shell-bound fields before SSH/tmux commands. + +Cookbook auto-registered endpoints are currently shared/null-owner rows with no API key when created by backend serve registration. Browser fallback registration goes through the normal model-endpoint route. The desired ownership policy for Cookbook-created endpoints should remain explicit. + +HW Fit is an MIT-licensed llmfit adaptation; attribution lives in project acknowledgments/licenses. + +## Hardware Fit + +`services/hwfit/hardware.py` owns hardware detection across NVIDIA, AMD, Apple Silicon, Windows, CPU, RAM, available RAM, remote SSH, container/native probe context, and cached host detections. + +`services/hwfit/models.py`, `fit.py`, `profiles.py`, `image_models.py`, and +`hf_discovery.py` own model catalog loading, normalization, API-backed dynamic +catalog refresh, memory estimates, quantization labels, fit scoring, serve +profile computation, image model ranking, and backend/format servability +filtering. + +`routes/hwfit_routes.py` owns the HTTP surface and manual hardware override application. + +Runtime behavior: + +- hardware detection uses a cache with `fresh=true` bypass; +- probe results include scope/container visibility metadata, and containerized no-GPU/low-RAM states can return user-facing visibility warnings with rescan/manual/copy-diagnostics actions; +- manual hardware replacement is a what-if simulator, not additive hardware; +- manual hardware accepts `cuda`, `rocm`, `metal`, `cpu_x86`, and `cpu_arm` + backends and must stay in lock-step with backend support in `fit.py`. Metal + simulation marks unified memory and filters toward locally servable GGUF/MLX + choices instead of CUDA/vLLM-only formats. +- ignore switches can drop detected GPU/RAM before ranking; +- homogeneous GPU grouping targets realistic multi-GPU pools; +- image model ranking normalizes to a single-GPU fit view; +- Metal/RDNA/backend restrictions can filter otherwise fit models. +- Apple Silicon bandwidth estimates use chip/core-specific tables for M-series Max/Pro/Ultra variants and avoid matching non-Apple GPU names. +- Windows and Apple/consumer-AMD paths filter toward GGUF/llama.cpp-compatible + choices. On multi-GPU systems, fixed GGUF target quantization that cannot be + served by the selected backend returns `no_fit` rather than `None`. + +## Platform And Degraded Behavior + +- Linux, Windows/PowerShell, macOS, Docker, NVIDIA, AMD, Apple Silicon, and CPU-only systems have different command paths. +- Remote hosts are accessed through SSH helpers; Cookbook host/port/path inputs must be validated before command construction. +- HW Fit remote host/port query values currently do not share all Cookbook route-level validation before SSH probing. +- Missing local tools or failed installs should surface command/output/error detail where possible. +- GPU overlays remain optional and do not break CPU-only deployments. +- Docker GPU overlays pass host devices/env; they do not install CUDA/ROCm engines by themselves. +- Default Docker Compose intentionally does not mount the host Docker socket. `docker/host-docker.yml` is an explicit high-trust overlay for operators who accept broad host-Docker control from inside the container. +- NVIDIA Docker diagnostics are read-only by default, and `.env` edits/install actions require explicit flags. +- AMD Docker diagnostics are read-only and do not mutate `.env`. +- vLLM is rejected on unsupported Windows/macOS paths. +- llama.cpp CPU-only and GPU fallback scripts should preserve usable CPU paths. +- SSH probe failures, GPU driver errors, and no-GPU states should be distinguishable. +- Remote SSH host/port validation is shared through route validators for Cookbook/HWFit paths. +- Windows launcher/runtime Git Bash discovery includes per-user installs under `%LocalAppData%\\Programs\\Git`, and WSL/Git Bash detection shapes PATH handling for NVIDIA/remote flows. +- macOS startup helpers start ChromaDB alongside the app path. +- Ollama serve can auto-pick an available port, and scheduled task stop paths + verify stop success before persisting a stopped state. + +## Model Catalog And Latest Lookup + +HW Fit model scoring depends on bundled `services/hwfit/data/hf_models.json`, +bundled `services/hwfit/data/mlx_community_models.json`, runtime dynamic caches +under `DATA_DIR/hwfit/`, catalog normalization, and assumptions about model +formats and quantization. `scripts/add_hwfit_models.py` updates the static HF +catalog. + +Hugging Face latest lookup and HW Fit dynamic refresh use external Hub metadata +and can degrade to empty, unknown-size, partial, or malformed-result behavior. +`refresh_catalog=1` refreshes API-backed collection caches for MLX community +and selected HF organization collections, with a 24-hour freshness guard and +bundled JSON fallbacks when the network/cache is unavailable. HW Fit tolerates +non-numeric `gpu_count` values from callers. Model normalization also treats +non-string `parameter_count` and quantization fields as unknown rather than +calling string methods and aborting the ranking pass. Catalog drift and dynamic +latest-model metadata are separate sources of recommendation drift. + +## Security Policy + +Admin gates must stay in place for install, serve, kill, setup, state mutation, and shell-like actions. `/api/shell/exec` is an admin primitive used by Cookbook task control and must stay in this review boundary. Scheduled `cookbook_serve` tasks are admin-only action tasks; task create/update/manual run/webhook/scheduler execution must all reject or pause them for non-admin owners. + +Kill-pid guardrails: + +- admin-only; +- PID floor; +- signal allowlist; +- validated remote host/port; +- frontend confirmation for TERM/KILL cleanup. + +Shell-bound Cookbook inputs must pass helper validation before command construction. HF tokens, Cookbook state secrets, and endpoint API keys must remain encrypted or masked and must not be written back to clients in raw form. Host Docker socket access must stay opt-in and clearly distinguished from merely having a Docker CLI in the container. + +## Testing Coverage + +Existing coverage is strongest for helper validation/quoting, SSH host validation, pip fallback and dependency-completion regressions, cached scan scripts, serve profile computation, scheduled serve lifecycle state persistence, hardware detection/ranking across AMD/NVIDIA/macOS/manual/container modes, MLX/Metal ranking, manual backend simulation, Docker GPU compose overlays, Cookbook CLI state, package detection, Windows path/task helpers, non-numeric GPU counts, non-string model catalog fields, and selected frontend progress regressions. + +Route-level auth/security and degraded-return coverage is thinner for Cookbook admin routes, shell dependency routes, `/api/cookbook/hf-latest`, state/status edge cases, HW Fit routes, frontend JS behavior, and helper scripts such as `hf_download.py`, `add_hwfit_models.py`, and `diffusion_server.py`. + +## Current Gaps + +- Cookbook-created model endpoint ownership/shared/null-owner policy needs a deliberate decision. +- `/api/shell/exec` and Cookbook package/rebuild routes need to remain cross-referenced with shell/admin specs because they are Cookbook-critical code-execution surfaces. +- Cookbook route auth/security and degraded-return behavior need route-level tests. +- `/api/cookbook/hf-latest` needs tests locking its user-authenticated access policy and failure behavior. +- HW Fit routes need route-level tests around missing catalogs, manual overrides, `fit_only`, profiles, and image-model cases. +- Dependency install/serve diagnosis remains split across Cookbook routes, shell routes, frontend diagnosis, optional binaries, and platform-specific scripts, even though longer serve-output tails are centralized through `routes/cookbook_output.py`. +- Model catalog, quantization, backend, and Hugging Face metadata drift need ongoing maintenance. diff --git a/specs/documents-rag-uploads.md b/specs/documents-rag-uploads.md new file mode 100644 index 000000000..ee1adcda7 --- /dev/null +++ b/specs/documents-rag-uploads.md @@ -0,0 +1,202 @@ +# Documents, RAG, And Uploads + +Last updated: dev@df2fad2 | 2026-07-12 + +## Scope + +This spec covers file/document context, document storage, and vector retrieval in: + +- `app.py` and `src/app_initializer.py` route/manager wiring; +- `routes/upload_routes.py`, `routes/personal_routes.py`, `routes/embedding_routes.py`, `routes/document_routes.py`, and `routes/document_helpers.py`; +- chat attachment paths in `routes/chat_routes.py`, `routes/chat_helpers.py`, `src/chat_handler.py`, and `src/chat_processor.py`; +- `core/session_manager.py`, `src/attachment_refs.py`, `src/upload_handler.py`, + `src/upload_limits.py`, and the public reference contract in + `docs/attachments.md`; +- `src/document_processor.py`, `src/document_actions.py`, `src/personal_docs.py`, and `src/markitdown_runtime.py`; +- `src/rag_singleton.py`, `src/rag_vector.py`, `src/rag_manager.py`, `src/chroma_client.py`, `src/embeddings.py`, and `src/embedding_lanes.py`; +- PDF/form helpers in `src/pdf_runtime.py`, `src/pdf_forms.py`, and `src/pdf_form_doc.py`; +- `services/docs/service.py`; +- document, upload, RAG, chat, email, and admin frontend callers in `static/app.js`, `static/js/chat.js`, `static/js/chatRenderer.js`, `static/js/fileHandler.js`, `static/js/document.js`, `static/js/documentLibrary.js`, `static/js/rag.js`, `static/js/admin.js`, `static/js/emailInbox.js`, and `static/js/slashCommands.js`; +- tests covering upload, document, attachment, PDF, RAG, Chroma, MarkItDown, and embedding behavior. + +## Runtime Integration + +`app.py` registers upload, personal-doc/RAG, embedding, document, diagnostics, and Codex document routes. `src.app_initializer.initialize_managers()` creates `UploadHandler` and `PersonalDocsManager`, installs the upload handler on `SessionManager` and the shared tool helper, and startup attempts to initialize the RAG singleton. App route wiring passes that same handler to session/history, document, note, and calendar writers that can persist upload references. + +`src.rag_singleton.get_rag_manager()` returns the live `VectorRAG` instance when Chroma/embedding dependencies are reachable. Personal routes can retry the singleton and return explicit 503s when unavailable. Chat RAG uses the `PersonalDocsManager.rag_manager` captured during app initialization and can silently skip RAG if that manager is absent. + +## Uploads And Attachments + +`src.upload_handler.UploadHandler` owns upload IDs, safe filenames, upload metadata, owner rename rewrites, atomic `uploads.json` writes, content-type detection, and file storage under `data/uploads`. Upload IDs accept extensionless values or one sanitized alphanumeric extension. + +`src.upload_limits` owns central upload-size caps and environment overrides for chat attachments, gallery, transforms, memory import, personal uploads, email compose, STT audio, and ICS imports. Invalid configured limits fail fast at import so routes do not silently accept unsafe sizes. Docker installs `libmagic1` plus `python-magic` so `UploadHandler.detect_content_type()` can sniff bytes in the official image; native installs can fall back to extension/MIME guesses when `python-magic` is unavailable. + +`routes/upload_routes.py` owns: + +- `POST /api/upload`, returning uploaded file metadata; +- reference-aware admin upload cleanup and stats; +- `GET /api/upload/{file_id}`; +- `GET/PUT /api/upload/{file_id}/vision` for editable OCR/vision cache; +- thumbnail and masked owner/admin access behavior. + +It does not currently expose a general upload list/delete route. Download/preview responses that serve uploaded content should include `X-Content-Type-Options: nosniff` where route code owns the response so browser MIME sniffing does not widen accepted upload types. + +Readable/code-like upload handling includes common text/code extensions plus `.nix`; document processing renders recognized code-like text into fenced blocks with language metadata. + +Chat does not own attachment extraction. Runtime flow: + +- the frontend uploads files and submits attachment IDs; +- `ChatHandler.preprocess_message()` resolves IDs with the session owner through `UploadHandler.resolve_upload()`, which enforces owner/admin access and no longer treats missing owner context as permission to read owned uploads; +- vision/OCR cache and attachment metadata are prepared before model calls; +- text-only models receive stripped multimodal blocks; +- `src.document_processor.build_user_content()` produces model-ready text, PDF text, Office/EPUB text when MarkItDown or the DOCX fallback is available, image/multimodal blocks, truncation, and PDF/Office auto-document updates; +- chat streams attachment, PDF-created `doc_update`, and `rag_sources` events where applicable. + +Extensionless image and audio attachments derive their data-URI subtype from +the detected MIME type, so `image/png` and `audio/mpeg` uploads do not become +invalid `data:image/;base64` or `data:audio/;base64` blocks when the filename +has no extension. + +## Durable References And Cleanup + +`src.attachment_refs` owns the stable `attachment_ref` shape used outside raw +upload storage: attachment id, name, MIME type, size, and optional checksum, +creation time, dimensions, vision text/model, and gallery id. Live provider +calls may still receive multimodal data URLs for the current turn, but durable +chat content is normalized to readable text plus compact reference lines. +Structured references remain in message attachment metadata, and chat FTS +triggers omit inline media while startup migration scrubs legacy indexed data +URLs. + +Agent/tool manifests expose `odysseus://attachment/` with +`read_policy: "owner_checked_upload"`. A compatibility filesystem path is +included only after owner-aware upload resolution, upload-root confinement, and +tool-readable-root checks; the stable contract for external tools is the URI +and attachment id, not host layout. + +Writers reserve referenced uploads before committing durable state. This +includes session message append/replace and history rewrites, document +create/update and native document edits, note route/tool create/update, +calendar/event route/tool create/update, and attachment-bearing session +updates. A missing or wrong-owner reference aborts before destructive +replacement and surfaces a route conflict or tool error. Reservations serialize +with cleanup through the upload-index lock and refresh access time. + +Admin cleanup first scans chat content and attachment metadata, current and +versioned documents including PDF markers, gallery filenames/hashes, note +image/color/content/checklist fields, and calendar color/description/location +fields. Reference discovery or index-integrity failure aborts cleanup; the +lower-level API removes nothing without both completed id and hash snapshots. +Only expired, unreferenced files with coherent id/path/owner/checksum/timestamp +metadata are candidates. Matching index rows are persisted away before byte +deletion and restored if deletion fails. This lock is process-local, so the +documented race protection assumes the current single-worker deployment. + +## Living Documents And PDF + +`routes/document_routes.py` owns the HTTP document API: create/read/update/archive/delete, library listing, import/export, version history, tidy/AI tidy, PDF rendering/export, PDF form helpers, and email-attachment reply preparation. + +`static/js/documentLibrary.js` owns local library state after archive/delete actions, including total counts and language chips. Server route truth still owns durable document state. + +`static/js/document.js` owns the browser document editor and markdown preview. Preview rendering applies code highlighting when highlight.js is present, renders Mermaid diagrams when the Mermaid runtime is available, refreshes after AI edits, and discards pending AI diffs before switching the active document. + +Document mutations also happen through agent tools, Codex document routes, email attachment import, and scripts. HTTP and native-agent document writers owner-reserve any internal upload/PDF references before persisting new current content or versions. Native document tool outputs include metadata that the browser can use to open/update the editor if a later stream update is missed. Those callers must preserve document owner, attachment, and version semantics. + +Email draft documents are a first-class document language. Create/update paths +detect the `To`/`Subject`/header shape, coerce language to `email`, and preserve +protected reply/forward headers such as `In-Reply-To`, `References`, +`X-Source-UID`, `X-Source-Folder`, attachment headers, and quoted/original +history when model or UI edits replace the draft body. Creating a draft for the +same source UID/folder in the same session updates the active draft instead of +creating a duplicate. + +`Document` rows own current content and owner. `DocumentVersion` rows own immutable snapshots. Document access should be owner-filtered, not session-id-only; the session document listing path still needs regression coverage for per-document owner filtering after the session owner check. + +PDF runtime behavior: + +- direct PDF import stores the upload through `UploadHandler`; +- PDF library entries preserve metadata/preview behavior for source PDFs; +- pypdf text extraction remains core; +- PyMuPDF enables form detection, page rendering, page PNGs, annotation fill, render/export PDF, and form filling; +- PDF render routes should return a shaped 503 when PyMuPDF is absent and use same-origin framing/download behavior for rendered pages; +- imported PDFs become either plain `pdf_source` markdown or `pdf_form_source` markdown with sidecar field data; +- PDF markers must resolve back through an upload owned by the caller; +- signed-reply preparation uses document `source_email_*` provenance and verifies the document owner and signature owner. Source email account resolution still needs explicit owner-scoped coverage. + +Office/EPUB attachment extraction is optional and MarkItDown-backed for `.docx`, `.pptx`, `.xlsx`, `.xls`, and `.epub`; a pure-Python DOCX fallback can extract `word/document.xml`. When a session id is present, full extraction can be saved as a markdown `Document` while the chat-inline copy remains capped. + +## Personal Docs And RAG + +`src.personal_docs.PersonalDocsManager` owns personal-directory indexing and keyword retrieval. + +`src.rag_vector.VectorRAG` owns Chroma/embedding-backed indexing and owner-filtered retrieval. Chunk ids are owner-scoped so byte-identical chunks from different owners do not suppress each other. `src.rag_singleton` owns lazy initialization, retry throttling, and reset behavior. + +`routes/personal_routes.py` owns personal-doc and direct RAG-upload routes. Directory list/index/delete routes are admin-gated. Direct RAG upload is user-authenticated, requires document privilege, forwards owner into the manager wrapper, writes unique files under per-owner subdirectories of `data/personal_uploads`, and has looser file-type validation than normal uploads. + +Current call sites include: + +- admin RAG pages and slash commands; +- chat RAG preface building; +- AI interaction and MCP RAG management tools; +- CLI scripts for document/personal indexing. + +Some non-route tool/script paths can index ownerless or arbitrary directories and should be treated as compatibility-sensitive management surfaces. + +## Embedding Models + +`routes/embedding_routes.py` owns admin-gated embedding model and custom endpoint management. It validates custom endpoints with outbound URL checks, can persist and process-expose `EMBEDDING_API_KEY`, resets embedding/RAG/tool-index/Chroma state, and does not own document extraction. + +`src.embeddings` owns HTTP embedding fallback to FastEmbed and process-level endpoint state. `src.embedding_lanes` keeps custom HTTP embedding vectors separate from FastEmbed fallback vectors with lane-specific Chroma collections, migrates legacy unsuffixed collections into empty lanes, and dedupes query results across lanes. `src.chroma_client` owns native Chroma defaults and fast reachability checks. + +## Compatibility State + +`src.rag_manager.RAGManager` is a backward-compat wrapper. The live owner-aware vector path is `VectorRAG`. + +`services/docs/service.py` is a separate facade and currently has result-shape drift from `VectorRAG`: it maps legacy `text`/`content` and `indexed`/`failed` keys while the live vector path returns `document`/`similarity` and `indexed_count`/`failed_count`. + +`src.database` re-exports `core.database`; document models and migrations live in `core.database`. + +## Optional And Degraded Behavior + +- ChromaDB/FastEmbed are default installed dependencies, but Chroma can be offline or unreachable. +- Native Chroma defaults to `localhost:8100`; Docker uses the `chromadb:8000` compose service and persistent Chroma storage. +- HTTP embeddings can fall back to FastEmbed; when both lanes exist, lane separation avoids Chroma dimension conflicts. +- MarkItDown is optional for Office/EPUB extraction; chat attachments and personal directory indexing have clear degraded behavior, while direct RAG upload does not share the same extraction path. +- PyMuPDF is optional, unlocks PDF form/render/fill paths, and carries AGPL implications when installed. +- PyMuPDF-dependent document routes should use the shared runtime helper/error text so missing-dependency and license policy stay visible. +- pypdf text extraction is core and should remain available without PyMuPDF. + +## Security And Provenance + +Uploaded files, documents, RAG chunks, extracted attachment text, OCR/vision text, PDF marker content, and source-email metadata are untrusted external or user-provided context when sent to an LLM. + +Concrete enforcement points include: + +- `UploadHandler.resolve_upload()` for upload ID validation, owner/admin access, and upload-dir confinement; +- owner-checked write reservations before durable attachment references are + stored, sharing the upload-index lock with reference-aware cleanup; +- PDF marker ownership checks before resolving source uploads; +- personal-directory and personal-upload confinement helpers, including symlink/realpath checks before deleting uploaded files or removing indexed directories; +- owner-filtered `VectorRAG.search(owner=...)`; +- shared untrusted-context wrappers for RAG preface insertion. + +Extracted attachment text is currently appended into the user message rather than wrapped as a separate untrusted-context message. That is current behavior and a prompt-injection hardening gap. + +Bearer-token callers are not a scoped document/upload API surface today. Routes that treat token-authenticated users as owners need explicit scope/effective-user policy before they are considered safe token APIs. + +## Testing Coverage + +Existing useful coverage includes upload owner scope, upload IDs, upload atomicity, durable attachment reference normalization, message/document/note/calendar write reservations, fail-closed reference-aware cleanup, attachment budgets, `.nix` text upload handling, upload/PDF security regressions, Docker `libmagic`/`python-magic` upload detection, RAG owner fallback, Chroma fast-fail, MarkItDown runtime, PDF runtime, document-library counter updates, and selected document helper behavior. + +Route-level coverage is thinner for document CRUD, PDF import/render/export/fill, direct RAG upload, embedding admin/security behavior, and RAG unavailable states. + +## Current Gaps + +- Direct RAG upload still needs clearer file-type validation and MarkItDown/PDF extraction parity decisions. +- Document `session_id` relinking and session document listing need owner-scope regressions. +- `services/docs/service.py` return-shape mapping is stale relative to `VectorRAG`. +- Chat RAG can remain degraded after startup even if personal routes later initialize the RAG singleton. +- PyMuPDF-dependent routes do not all share the same optional-runtime helper/error behavior. +- Signed-reply preparation needs owner-scoped source email account/signature regression coverage. +- Document/upload routes need explicit bearer-token scope/effective-user policy. +- User-facing document/PDF/RAG route matrices need more regression coverage for owner denial, admin gates, unavailable services, and degraded optional dependencies. diff --git a/specs/email-contacts.md b/specs/email-contacts.md new file mode 100644 index 000000000..54ad77d21 --- /dev/null +++ b/specs/email-contacts.md @@ -0,0 +1,198 @@ +# Email And Contacts + +Last updated: dev@df2fad2 | 2026-07-12 + +## Scope + +This spec covers mail and contacts in: + +- app wiring in `app.py`; +- `core.database.EmailAccount`; +- `routes/email_routes.py`, `routes/email_helpers.py`, and `routes/email_pollers.py`; +- email threading in `src/email_thread_parser.py`; +- email MCP tools in `mcp_servers/email_server.py`; +- canonical contact/CardDAV routes in `routes/contacts/contacts_routes.py`, + with `routes/contacts_routes.py` as a compatibility shim; +- Codex email bridge in `routes/codex_routes.py`; +- document signed-reply flows in `routes/document_routes.py` and document `source_email_*` fields; +- reminder/task email senders in `routes/note_routes.py` and `src/task_scheduler.py`; +- email/contact agent surfaces in `src/tool_implementations.py`, `src/tool_schemas.py`, `src/tool_index.py`, and `src/agent_loop.py`; +- CLI wrappers `scripts/odysseus-mail` and `scripts/odysseus-contacts`; +- frontend modules `static/js/emailInbox.js`, `static/js/emailLibrary.js`, `static/js/emailLibrary/*`, `static/js/emailShared.js`, `static/js/chatStream.js`, `static/js/document.js`, and `static/js/settings.js`; +- tests under `tests/test_email_*`, `tests/test_contacts_*`, `tests/test_mail_cli_*`, `tests/test_mcp_email_*`, `tests/test_schedule_email_*`, email/contact JS tests, and email security regressions. + +## Current Call Sites Include + +- browser email inbox/library, compose, schedule, account, and attachment actions; +- document-editor compose, recipient autocomplete, compose uploads, and signed-reply handoff; +- Codex email read/draft/send routes using API-token scopes; +- note reminder and task-output email delivery; +- built-in email summary/reply/calendar/urgency actions; +- scheduled email pollers and CLI one-shot pollers; +- MCP email tools; +- contact manager settings, compose contact autocomplete, agent contact tools, and contacts CLI. + +## Email Accounts And Transport + +`EmailAccount` rows own IMAP/SMTP configuration. Password fields are string columns containing encrypted ciphertext written with `src.secret_storage`; startup migrations handle legacy plaintext rows. Google OAuth account rows also carry `oauth_provider`, encrypted access/refresh tokens, token expiry, and an optional outbound `display_name`. Do not return decrypted credentials or OAuth tokens, or write them to logs. + +`routes.email_helpers` owns: + +- account owner assertions and config fallback order; +- IMAP/SMTP connection helpers and related transport utilities; +- Google OAuth2 state signing/verification, token refresh, and XOAUTH2 framing; +- SMTP security modes (`ssl`, `starttls`, `none`); +- envelope recipients and Odysseus headers; +- attachment extraction helpers; +- email pre-retrieval context for AI reply drafting; +- scheduled email, summary, reply, tag, calendar extraction, urgency, and signature-boundary side databases. + +Email config can fall back to legacy `data/settings.json` or environment variables when no scoped account is configured. Account discovery now owner-scopes the default/first-enabled fallback and can still match legacy account rows by IMAP username or from-address. That fallback remains compatibility-sensitive in multi-user contexts. + +Email owner semantics are route-local and compatibility-sensitive: + +- `routes.email_helpers._require_auth()` returns `""` in `AUTH_ENABLED=false` mode, rejects configured auth with no user, and only tolerates first-run anonymous loopback fallback. +- Empty owner is treated as single-user compatibility: account-ownership assertions no-op, default/first-enabled account fallback can be global, and email cache clauses include `owner = '' OR owner IS NULL`. +- Non-empty owners scope account/config/cache queries. Legacy ownerless account + rows are visible to an authenticated owner only when the row's IMAP username + or from-address matches that owner, so old unowned rows do not become global + cross-user accounts in configured multi-user deployments. + +`routes.email_routes` owns the HTTP mail surface: + +- account CRUD, test, default, and masked config reads; +- Google OAuth authorize/callback for Workspace and .edu Gmail-style accounts; +- list, search, read, folders, and contacts; +- folder role resolution and UID fetch/search helpers used by the route surface; +- owner-scoped route caches and IMAP pool behavior; +- attachments, bulk attachment ZIP downloads, and attachment-to-document flows; +- compose upload, draft/send, `wait_for_delivery`, Sent append, and source `\Answered` marking; +- schedule/list/delete scheduled emails; +- pending agent-draft approval/cancel flows; +- mark read/unread/answered, spam flags, move, archive, and delete. IMAP move/delete/archive operations use UID commands for message identity and fail safe when the requested UID no longer exists; they never reinterpret a missing UID as a sequence number, which could mutate or expunge an unrelated message. + +Google OAuth behavior is account-owned: + +- `/api/email/oauth/google/authorize` requires an authenticated owner, checks account ownership, HMAC-signs state with account id, owner, and nonce, and redirects to Google with mail/userinfo scopes; +- `/api/email/oauth/google/callback` verifies signed state before token exchange, re-checks the target account owner before writing tokens, stores access/refresh tokens encrypted, stores token expiry as a timestamp, and redirects with generic success/error codes rather than raw provider errors; +- token refresh uses `GOOGLE_OAUTH_CLIENT_ID` and `GOOGLE_OAUTH_CLIENT_SECRET`, stores refreshed access tokens encrypted, and logs only generic/account-id context on failures; +- SMTP and IMAP use XOAUTH2 when `oauth_provider == "google"`; OAuth accounts are send-capable without an SMTP password when host and user are configured; +- outbound mail formats the `From` header with `display_name` when present. + +MCP full-message read/reply/attachment fetches use IMAP `BODY.PEEK[]` rather than bare `RFC822`, so iCloud-style servers return the full body without marking messages seen. Poller UID handling must tolerate both bytes and string UIDs. Built-in signature-learning and daily-brief actions also use UID SEARCH/FETCH rather than sequence-number commands. + +IMAP helpers quote mailbox names, raise the Python IMAP line cap for large messages, close sockets after connect/login failures, and preserve Gmail FETCH attributes that follow header literals so unread flag state is not lost. Browser list routes offload blocking IMAP work from async handlers; browser search runs in FastAPI's threadpool, rejects CRLF query input, tokenizes quoted phrases/terms, searches FROM/TO/CC/SUBJECT/TEXT, can search Gmail All Mail when an INBOX query should include archived or labelled messages, and supports `scope=folder` when callers intentionally want the selected folder only. The local index fallback can return indexed results when IMAP returns empty or fails. + +## Runtime And Pollers + +Scheduled email rows live in `data/scheduled_emails.db` and are owner-scoped. Scheduled send times are normalized before storage. + +`routes.email_pollers` owns the scheduled-send poller and single-shot/task/CLI automation passes. Before SMTP work, each poller atomically claims a due row with a conditional `pending` to `sending` update; concurrent in-process/CLI pollers that lose the claim skip the row instead of sending a duplicate. Only the scheduled-send poller starts in-process by default when `ODYSSEUS_INPROCESS_POLLERS` allows it; Docker forwards that gate. Background email automation can also consult the foreground activity gate so auto actions do not compete with active browser/model work. Native cron/systemd can drive one-shot pollers through `scripts/odysseus-mail`. + +Transport degraded behavior: + +- IMAP timeouts are clamped by configuration; +- providers can use implicit SSL, STARTTLS, or plain connections; +- poisoned IMAP sockets are reconnected around known provider failures; +- SMTP-capable account fallback is used where supported; +- route helpers, MCP, and CLI do not all share identical SMTP/IMAP parsing and security behavior today. + +## Caching And Staleness + +Email list/read behavior uses short route caches, longer read caches, capped warm prefetch, and owner/account-aware pool/cache keys. The frontend email library has its own session SWR cache, cache-buster refreshes, scheduled/search cache exclusions, and stale-row behavior when refresh fails. + +List/read route caches are owner/account-aware. Helper-side summary, AI-reply, tag, calendar-extraction, urgency-alert, and learned sender-signature tables carry owner columns and owner clauses. Thread-boundary rows are still keyed by message shape rather than a full owner/account/mailbox key, so they remain cross-owner audit points when identical messages appear in multiple mailboxes. + +## Attachments And Signed Replies + +Compose uploads live under `ODYSSEUS_MAIL_ATTACHMENTS_DIR`; missing staged files are skipped with warnings. Attachment-to-document supports PDF, DOCX, TXT, and MD. DOCX depends on `python-docx`; PDF form/open-in-doc flows can depend on optional PyMuPDF. + +Email attachment-as-document flows stamp `Document.source_email_*` provenance. `GET /api/email/attachments-download/{uid}` builds an owner-scoped ZIP of visible non-signature attachments using safe names. `compose-from-odysseus` and `compose-from-odysseus-zip` can stage owner-visible documents and gallery images as compose uploads, preserving legacy session fallback only where the source object remains visible to the owner. `prepare-signed-reply` verifies document ownership, reconstructs reply headers, flattens/stages signed PDFs as compose uploads, and leaves final send/draft review to the compose flow. + +Email bodies and attachments are untrusted model context. + +## Threading And Rendering + +`src.email_thread_parser` owns splitting plaintext/HTML email threads into quoted conversation parts. Frontend email library modules own reply-recipient logic, signature folding, local state, and rendering behavior. Bulk selections are cleared when folder/account loads, search text, search pills, or result scope changes so actions cannot carry stale UIDs into a different visible context. `static/js/emailShared.js` owns shared email UI helpers used across inbox/library surfaces. + +Remote inbound email HTML is sanitized by frontend email-library utilities before `innerHTML` insertion. Server-side email routes sanitize composed/generated outbound HTML with an allowlist before draft/send, dropping scripts/styles and unsafe attributes. Both sides are part of the rendering invariant. + +When the email reader is active, browser chat sends selected-message metadata. `src.tool_implementations` stores that request-local active email reference, `src.agent_loop` injects it as protected untrusted context, and `static/js/chatStream.js` handles `ui_control open_email_reply` so default reply/draft behavior opens the selected message's compose flow instead of a generic new document. + +## MCP Email + +`mcp_servers/email_server.py` exposes email tools for MCP/agent use. It has its own account discovery, IMAP/SMTP, attachment, cache, and send paths, but account visibility now mirrors the HTTP owner policy. The active owner comes from a hidden `_odysseus_owner` argument when the caller provides one, or from `ODYSSEUS_MCP_EMAIL_OWNER` / `ODYSSEUS_EMAIL_OWNER`. If any enabled account is owner-scoped and no current/configured owner exists, email MCP returns an owner-scope error instead of listing global accounts. + +MCP email account filtering includes owner-owned rows and legacy ownerless rows +whose mailbox/from-address matches the owner. Confirmation-first `send_email` +resolves the selected account before stashing an `agent_draft`, so drafts cannot +be staged against another owner's account. MCP-created draft documents use the +resolved hidden/configured owner when available, with `ODYSSEUS_DOCUMENT_OWNER` +and single-admin fallback only as document-visibility compatibility. + +MCP email send behavior is confirmation-first by default: `send_email` and reply send paths stash a `scheduled_emails` row with `status='agent_draft'` when `agent_email_confirm` is true, and browser routes expose pending drafts for approval or cancellation. Separate MCP draft tools create Odysseus compose documents for user review without sending. + +MCP email remains a separate local/admin trust boundary. Public and non-admin users must not see or execute email MCP tools. It still needs route-helper parity audits for attachment path containment, sanitization, transport behavior, and pending-draft result text, but global all-account behavior is no longer the current owner model. + +## Contacts + +`routes.contacts.contacts_routes` owns global/admin contacts and CardDAV behavior. The top-level `routes.contacts_routes` module is a compatibility shim. The canonical package supports local contacts, CardDAV config, list/search/add/update/delete, VCF/CSV import/export, and clear. + +Contact runtime behavior: + +- contacts routes are admin-gated; +- local `data/contacts.json` is used when CardDAV is unconfigured; +- import paths tolerate malformed or non-string contact bodies by skipping invalid rows instead of crashing the import; +- configured CardDAV uses REPORT with GET fallback and a short in-memory cache; +- configured-but-offline CardDAV can return cached reads but writes fail instead of falling back to local JSON; +- CardDAV config reads mask the password, settings-stored passwords are encrypted with `src.secret_storage`, omitted password updates preserve the existing secret, and an explicit empty password clears it; +- the native contacts CLI is CardDAV-oriented and does not fully match web JSON fallback behavior; +- agent contact tools reuse helper functions in-process because the HTTP routes require browser/admin auth. + +Contacts are global admin-only data today. There is no per-user contact sharing model unless a future spec defines one. + +## Security Policy + +Email HTTP access is owner-scoped, including account selection, scheduled email rows, and attachment routes. Null-owner/single-user compatibility paths are security-sensitive and must not allow cross-user mailbox access. + +Codex email routes are the scoped bearer-token email API. They enforce `email:read`, `email:draft`, and `email:send` scopes and use token-owner attribution before borrowing email route handlers. + +Known security policy details: + +- decrypted email credentials stay process-local; +- account/config reads mask passwords and expose only OAuth status fields, not access or refresh token values; +- SMTP/IMAP security mode behavior is part of the credential contract; +- Google OAuth state and callback owner checks are part of the account-boundary contract; +- scheduled emails must remain owner-scoped; +- email pre-retrieval contacts context is allowed only for admin/single-user situations; +- MCP attachment downloads need route-level path-containment parity; current MCP paths are separate from the HTTP compose/attachment helper path. + +CardDAV credentials and URLs are security-sensitive. CardDAV URL setup and derived href writes/deletes pass through outbound URL validation; absolute hrefs from a CardDAV server are constrained back to the configured origin before credentials are reused. CardDAV passwords in settings are encrypted and masked on read; environment-sourced legacy password values are used as supplied. + +## Degraded Behavior + +- IMAP/SMTP providers can be slow or inconsistent; folder resolution, pooled connections, and reconnect behavior should fail with clear errors. +- Google OAuth requires external Google endpoints plus configured `GOOGLE_OAUTH_CLIENT_ID`/`GOOGLE_OAUTH_CLIENT_SECRET`; missing client credentials or refresh failures degrade to reconnect-required or generic OAuth error paths. +- Scheduled email delivery depends on `scheduled_emails.db`, poller runtime, and configured SMTP. +- Attachment handling must tolerate missing staged files, unsupported formats, and inaccessible remote messages. +- CardDAV local fallback applies only when CardDAV is unconfigured; configured CardDAV outages are not treated as local-write mode. +- Multi-account list/search behavior can be sequential and cache-sensitive. + +## Testing Coverage + +Existing coverage includes header decoding, envelope recipients, IMAP timeout, SMTP security, IMAP reconnect, Google OAuth state/callback/token-refresh/XOAUTH2 behavior, OAuth account token non-disclosure, UID-only IMAP move/flag/fetch behavior, concurrent scheduled-email claim behavior, iCloud-compatible MCP full-message fetch shape, owner scope, ownerless account mailbox-match guards, owner-keyed sender signatures, Gmail flag parsing, scheduled offset normalization, active-email reply guard behavior, thread parsing, HTML sanitizer source checks, MCP header decoding, MCP owner-account scope, MCP multi-account/search shapes, CardDAV password encryption, mail CLI behavior, contacts parsing/add basics, bulk-selection reset, reply-recipient JS, signature folding, Gmail quote attribution, and selected security regressions. + +Route-level and duplicate-path coverage is still thin for email list/read/search/mutations, account CRUD/security outside the OAuth path, send/draft security, attachments, scheduled-poller failures, contacts admin/CardDAV routes, MCP account/scope behavior, CardDAV degraded mode, and executable frontend behavior. + +## Current Gaps + +- Owner-keyed cache policy still needs an explicit decision for thread boundaries, plus continued migration/query audits for every email side table. +- CardDAV still needs redirect/proxy policy and broader route-level tests for URL validation, private-address blocking configuration, and same-origin href enforcement. +- MCP email needs continued route-helper parity for attachment path containment, + sanitization, transport behavior, and pending-draft result text. +- Empty-owner route compatibility and ownerless email cache rows need + end-to-end owner-boundary tests. +- CLI send/contact paths need parity decisions for SMTP security, recipient parsing, local fallback, and normalized contact shapes. +- Email HTTP route coverage is concentrated in scheduling/account-test helpers rather than full list/read/search/mutation/send/draft/account/attachment flows. +- Contacts coverage lacks admin-gate, config masking, import/export, CardDAV fallback, and CardDAV write-failure tests. +- Multi-account performance and cache staleness remain known audit areas. diff --git a/specs/frontend.md b/specs/frontend.md new file mode 100644 index 000000000..a0db034a1 --- /dev/null +++ b/specs/frontend.md @@ -0,0 +1,151 @@ +# Frontend + +Last updated: dev@e57f60b | 2026-07-20 + +## Scope + +This spec covers the current browser app in: + +- static serving and SPA routes in `app.py`; +- CSP/security headers in `core/middleware.py`; +- `static/index.html`; +- `static/login.html`; +- `static/app.js`; +- `static/style.css`; +- `static/js/*.js` and `static/js/*/*.js`; +- vendor libraries under `static/lib/*`; +- custom fonts and static assets under `static/fonts/*`; +- `static/sw.js` and `static/manifest.json`; +- frontend-oriented tests in `tests/*_js.py`, `tests/*.mjs`, `tests/bombadil-spec.ts`, static DOM/CSS/source-shape tests, and app/static tests such as `tests/test_app_static_mime.py`. + +`/backgrounds` currently targets `static/backgrounds.html`; if that route remains, the file must exist or the route should be removed. + +`static/manifest.json` and `static/index.html` reference PWA icon files under `static/icons/`; the current 192px, 512px, and maskable icon files exist and should stay aligned with those references. + +## Current Call Sites Include + +- `static/index.html` script tags and modulepreloads; +- `static/sw.js` `PRECACHE`; +- app-owned SPA deep links for notes, calendar, cookbook, email, memory, gallery, tasks, and library; +- `/login` and app-owned static/HTML routes; +- `/api/activity/heartbeat` browser visibility pings used by the foreground activity gate; +- `static/app.js` route opener/sidebar/tool-window wiring; +- frontend JS helper tests and static HTML/CSS/source-shape regressions; +- CDN dependencies, local vendor libraries, service worker, and PWA manifest. + +## Runtime Shape + +The frontend is a raw static SPA served by FastAPI. There is no Vite, React, TypeScript, bundler, or generated build output. + +`app.py` owns: + +- stable `.js`/`.mjs` MIME registration; +- the `/static` mount; +- no-cache headers for `.js`, `.css`, and `.html` static source files; +- nonce-injected SPA/login HTML serving; +- SPA deep-link routes. + +`static/index.html` owns the DOM shell and script loading order. It loads browser ES modules directly. Current boot order includes nonce-bearing inline boot scripts, self-hosted highlight.js, async CDN KaTeX/Mermaid, modulepreloads, ordered module script tags, `static/app.js`, `static/js/init.js`, `static/js/a11y.js`, workspace/chat helpers, provider device-flow helpers, and service-worker registration. + +Exact script URL identity matters. Versioned script tags, unversioned imports, and service-worker precache entries must stay aligned. Current service-worker precache coverage is not a full mirror of the `index.html` module graph, so changes there need direct verification. + +## Security Policy + +`core/middleware.py` owns CSP and security headers. `app.py` injects the per-request nonce into served HTML. New inline scripts or external scripts/styles/images/media must fit the CSP contract or explicitly update it. + +`/static/*` is public/auth-exempt. Frontend privilege gates are display-only; backend routes enforce authorization. + +XSS/DOM policy: + +- prefer DOM construction, `textContent`, and shared escaping helpers; +- Markdown raw HTML preservation must remain constrained through sanitizer helpers; +- remote email `body_html` must pass through the email-library sanitizer before insertion; +- Mermaid, code-runner iframe `srcdoc`, visual reports, remote media, and scattered `innerHTML` templates require explicit review. +- Visual report Markdown HTML is server-rendered and should be treated as security-sensitive alongside frontend entry points and remote media. + +Storage/secrets policy: + +- localStorage/sessionStorage are for preferences, UI state, offline caches, and user-switch sentinels; +- `static/js/init.js` owns user-switch storage cleanup; +- raw API tokens, provider keys, HF tokens, and other credentials must not be persisted in browser storage unless a feature documents masking/stripping and backend storage ownership. + +## Service Worker And PWA + +`static/sw.js` owns PWA cache behavior: + +- API and non-GET requests are bypassed; +- root navigation uses stale-while-revalidate; +- JS/CSS use network-first behavior; +- other static assets use cache-first with background refresh; +- `CACHE_NAME` bumps and `PRECACHE` updates must accompany cache policy or shell asset changes. + +`static/manifest.json` owns default PWA metadata. Route-specific manifests can be generated as Blob URLs when supported. Current default icon references must match real files under `static/icons/`. + +Offline/PWA behavior is not fully self-contained: KaTeX, Mermaid, and Pyodide use jsDelivr paths, while other vendor libraries are self-hosted under `static/lib`. + +## Module Ownership + +Current major frontend areas include: + +- chat, stream handling, rendering, sessions, markdown, uploads, voice recorder, TTS, and keyboard shortcuts; +- models, provider setup, pure model-key matching helpers, model picker, presets, search, RAG, settings, and admin; +- compare modules under `static/js/compare/`, including sanitized popup/search/image handling; +- document editor/library in `static/js/document.js` and `static/js/documentLibrary.js`; +- image editor integration in `static/js/galleryEditor.js` plus leaves under `static/js/editor/`; +- gallery, email inbox/library, calendar, research panel/jobs/synapse, notes/tasks, assistant, memory/skills, Cookbook/HW Fit, workspace picker, provider device flow, composer ArrowUp recall, theme, modal/window utilities, storage, and accessibility helpers. + +Coordinator ownership: + +- `static/app.js` owns late orchestration, global fetch 401 redirects, sidebar/tool route wiring, and many `window.*` compatibility bridges; +- `static/js/init.js` owns post-load cleanup, user-switch storage wipe, and cosmetic privilege gates; +- `static/js/storage.js` owns shared key constants and safe JSON helpers; +- feature modules own feature state where possible. + +`static/js/MODULE_SUMMARY.md` is a refreshed ownership/navigation map for the no-build frontend. The current `static/js/` tree, `static/app.js`, `static/index.html`, and executable behavior remain the authority when the summary drifts. + +Current small frontend helper contracts include `static/js/model/matchKey.js` for longest-substring model info/pricing matches, `static/js/models.js` for in-flight `/api/models` request sharing, `static/js/providerDeviceFlow.js` for Copilot/ChatGPT Subscription device-flow polling UI, `static/js/composerArrowUpRecall.js` for prompt recall from an empty composer, `static/js/fileHandler.js` for capped pending-file state and collapsed attachment-chip display, `static/js/streamingSegmenter.js` for incremental markdown/code-fence segmentation, `static/js/emojiShortcodes.js` for shortcode replacement, `static/js/documentLibrary.js` for keeping document counters/language chips in sync after archive/delete, `static/js/keyboard-shortcuts.js` for rejecting empty or non-string persisted keybinds before combo parsing, `static/js/modalSnap.js` for reusable desktop modal edge docking, `static/js/toolWindowZOrder.js` for shared portal/window z-index allocation, and `static/js/emailShared.js` for common email UI helpers. + +Recent browser behavior contracts include mobile chat Enter inserting newlines while desktop Enter submits, queued prompts preserving mobile composer behavior during active streams, regenerate-from-here truncating history while normal resend appends a fresh turn, AI-message delete confirmation, user-message edit textareas expanding the bubble to the normal 85% editing width, native document tool results opening/updating the editor even if a later stream event is missed, chat rendering that hides raw leaked tool JSON/document fences and displays explicit agent loop/intent guard stops, Markdown inline-math delimiter rules that leave currency ranges such as `$5 to $10` as text, notes search reset on reopen, email bulk-selection reset across folder/account/search context changes, calendar Monday/Sunday week-start localStorage preference, CardDAV unchanged-password placeholders, Google Workspace/.edu email OAuth controls in both email account forms, Google Gemini model-endpoint creation omitting an automatic refresh value so the backend manual default applies, gallery editor AI edit command routing, OpenDyslexic/text-size accessibility preferences, admin promote/demote buttons, admin diagnostics log polling, and dismissible toasts that do not block pointer interaction longer than intended. + +## UI Policy + +- New code must run as browser ES modules without a build step. +- Reuse existing CSS variables, modal/window patterns, icon style, storage helpers, and route conventions. +- Custom font handling includes bundled OpenDyslexic assets plus user-supplied fonts exposed through `/api/fonts/custom`; font and text-size settings must stay coordinated between settings UI, theme helpers, and CSS variables. +- Avoid relying on stale module summaries. +- API shape changes must update the owning JS module and tests. +- Add behavior to large coordinators such as `static/app.js`, `static/js/chat.js`, `static/js/document.js`, or `static/js/settings.js` only when it matches their existing wiring ownership. + +## Degraded And Platform Behavior + +- Server no-cache applies to `.js`, `.css`, and `.html` source files, not every static asset. +- Service-worker cache changes can affect frontend behavior even when source files revalidate. +- Mobile behavior uses separate CSS/media/hover/safe-area/`100dvh` handling and JS layout code; check it directly. +- Browser APIs such as service workers, Blob route manifests, Web Speech, `getUserMedia`, visual viewport, and storage can be absent or restricted. +- Local libraries and CDN globals degrade differently; document, markdown, math, diagrams, and code runner flows should handle missing globals where possible. +- localStorage migrations and cross-user cleanup are part of compatibility. + +## Testing Coverage + +Existing frontend coverage is a mix of Node-executed helper tests, `.mjs` tests, static DOM/CSS/source-shape tests, browser exploration specs, and app/static tests. Many tests are useful source-shape regressions but do not replace browser/module-graph execution. + +Recent focused coverage includes model-key matching under Node, document-library counter source-shape checks, chat resend/delete/mobile Enter, agent guard events, editable message width, and queued prompt behavior, malformed keybind handling, currency-safe inline math, email bulk-selection reset, notes search reset, calendar week-start preference, modal snap zones, manifest icon existence, admin log fetching, Markdown DOM XSS helpers, and CardDAV unchanged-password handling. + +Missing coverage includes: + +- SPA route/static auth and no-cache headers; +- CSP header contents and nonce injection for `/` and `/login`; +- service-worker API/non-GET bypass and cache strategy; +- service-worker precache versus `index.html` script/module tags, including query strings; +- ongoing manifest/icon reference drift; +- module graph/load-order validation; +- degraded vendor/CDN/browser API behavior. + +## Current Gaps + +- `static/style.css` and large coordinators remain high-risk owners: `static/js/document.js`, `static/js/settings.js`, `static/js/chat.js`, and `static/app.js`. +- There is no build-time type checking, module graph validation, script-order validation, or service-worker precache validation. +- Frontend state is mostly module/global/localStorage driven, so cross-session and cross-user behavior needs explicit care. +- `window.*` compatibility bridges remain widespread. +- PWA/static-serving behavior may deserve a separate spec if service worker, manifests, route-specific icons, and cache policy keep growing. +- A static asset/route manifest regression should verify files referenced by `index.html`, `manifest.json`, `sw.js`, and app-owned HTML routes actually exist. diff --git a/specs/gallery-editor-media.md b/specs/gallery-editor-media.md new file mode 100644 index 000000000..c90a8a47a --- /dev/null +++ b/specs/gallery-editor-media.md @@ -0,0 +1,162 @@ +# Gallery, Editor, And Media + +Last updated: dev@df2fad2 | 2026-07-12 + +## Scope + +This spec covers media surfaces in: + +- app route registration and generated-file serving in `app.py`; +- canonical models in `core/database.py`, with `src.database` as a compatibility import path; +- canonical route package `routes/gallery/gallery_routes.py` and `routes/gallery/gallery_helpers.py`, with top-level `routes/gallery_routes.py` and `routes/gallery_helpers.py` compatibility shims; +- generated-image writers in `src/ai_interaction.py` and `mcp_servers/image_gen_server.py`; +- image tool schemas/dispatch/implementations in `src/tool_schemas.py`, `src/tool_execution.py`, and `src/tool_implementations.py`; +- `routes/editor_draft_routes.py`; +- `routes/signature_routes.py` and document signature consumers in `routes/document_routes.py`; +- `routes/emoji_routes.py`; +- `routes/font_routes.py`; +- `src/generated_images.py`; +- `src/visual_report.py` plus research image hide/unhide routes; +- database models `GalleryImage`, `GalleryAlbum`, `EditorDraft`, and `Signature`; +- generated files under `data/generated_images`; +- frontend modules `static/js/gallery.js`, `static/js/galleryEditor.js`, `static/js/editor/*`, `static/js/signature.js`, `static/js/emojiPicker.js`, `static/js/chatRenderer.js`, `static/js/document.js`, `static/js/markdown.js`, and `static/js/theme.js`; +- CLI surfaces `scripts/odysseus-gallery` and `scripts/odysseus-signature`; +- tests covering gallery helpers/routes, generated-image serving, editor drafts, signatures, visual reports, fonts, upload limits, and image endpoint security. + +## Current Call Sites Include + +- gallery upload, library, album, tag, favorite, ZIP, delete, and saved-project views; +- chat-generated image rendering/edit/delete bubbles; +- agent `generate_image` and stale `edit_image` tool paths; +- MCP image-generation rows/files; +- image editor AI tools and model endpoint pickers; +- document PDF signing with stored signatures; +- visual-report hero/section image insertion and research hide/unhide controls; +- emoji picker/markdown emoji SVG proxy calls; +- theme custom-font loading; +- local gallery/signature CLI inspection. + +## Gallery + +`routes.gallery.gallery_routes` owns gallery upload/import/library/editor transform behavior: upload dedupe, image/video extension handling, EXIF extraction for images, albums, favorites, tags, generated media metadata, search/filter/sort, owner filtering, ZIP downloads, soft delete, disk cleanup, and chat-history cleanup after image delete. Top-level `routes.gallery_routes` is a `sys.modules` compatibility shim to the canonical module. + +Frontend gallery behavior includes upload progress, folder-drop album import, stale-while-revalidate cards, saved editor projects, detail actions, bulk delete/download, and cache-busted image refreshes. + +Album assignment and gallery image detail/update endpoints enforce owner scope and fail closed when no authenticated owner is available instead of falling back to broad access. + +Generated media provenance: + +- generated filenames are opaque hex-like media names, not trusted content hashes; +- upload `file_hash` is a separate metadata field; +- generated files live under `data/generated_images`; +- chat image generation writes files and inserts `GalleryImage` rows through `src.ai_interaction`; +- MCP image generation can create ownerless rows/files; +- generated-but-not-yet-imported images can have no gallery row; +- once a gallery row exists, owner checks decide visibility where the route enforces them. + +`app.py` owns direct `/api/generated-image/{filename}` serving through `src.generated_images.resolve_generated_image_path()`. It validates hex-like image/video filenames, rejects path escape and missing files, serves rowless generated files, checks row owner when a row exists, allows null-owner compatibility rows, and uses immutable/nosniff cache headers. Gallery replace/rotate/save/delete/ZIP paths also resolve filenames through a shared generated-image path helper so database filenames cannot escape `data/generated_images`. Replace/rotate/save-over-original flows can mutate bytes under the same filename, so frontend cache busting matters. + +## Image Tools And Providers + +Gallery/editor image transforms are split across: + +- `/api/gallery/ai-upscale` and `/api/gallery/style-transfer`; +- `/api/image/inpaint`; +- `/api/image/harmonize`; +- `/api/image/sharpen`; +- `/api/image/denoise`; +- `/api/image/upscale-local`; +- `/api/image/remove-bg`; +- `/api/image/enhance-face`. + +AI image endpoints mostly require image-generation privilege in the gallery route layer. The sharpen route is explicitly auth-gated; utility routes that live outside gallery still need their own route-level gate checks rather than assuming a shared decorator. The chat image-generation session path calls `do_generate_image()` separately and has its own privilege/tool-listing behavior. + +Provider behavior: + +- OpenAI image edits use multipart `/images/edits`, mask conversion, size coercion, model restrictions, and source compositing where needed; +- diffusion/self-hosted paths use JSON APIs such as inpaint, img2img, variations, harmonize, or A1111-compatible fallbacks; +- client-supplied endpoint URLs on selected routes must pass outbound endpoint validation; DB-selected image endpoints should be resolved through owner-visible endpoint queries before decrypted headers/keys are used; +- provider-returned image result URLs are validated with `src.url_safety.check_outbound_url()` before server-side download, with private-IP blocking controlled by image-route settings; +- AI endpoint path suffixes are allowlisted before proxy/download use so arbitrary endpoint paths cannot be selected through gallery/editor requests; +- editor model pickers load `/api/model-endpoints` and classify image-capable endpoints. + +Optional dependency behavior: + +- Pillow-backed paths are effectively core for EXIF, rotate, sharpen, and image preparation; +- Real-ESRGAN powers denoise/upscale when installed and otherwise returns install guidance; import-time torchvision compatibility patches run before Real-ESRGAN imports; +- remove-bg tries `rembg`, then transformers-style fallback, then an error; +- face enhancement falls back from GFPGAN/OpenCV toward PIL behavior; +- video uploads intentionally skip EXIF/ffprobe metadata today. + +## Editor Drafts + +`routes.editor_draft_routes` owns server-backed image editor project payloads. `EditorDraft` rows store title, payload JSON, thumbnail, source image, timestamps, and owner. + +Frontend editor behavior is split across `static/js/editor/*` and `static/js/galleryEditor.js`: canvas state, layer panel, masks, history, snapping, stroke pipeline, inpaint/rembg/harmonize tools, AI tool runner, model pickers, an AI edit command box that routes natural-language edit requests into existing inpaint/remove/upscale/background/style actions where possible, import wiring, topbar controls, auto-save, resume by draft ID or source image, draft-only open, and cleanup after close. + +Draft compatibility behavior: + +- v2 server drafts store payloads and thumbnails server-side; +- legacy/local raw payloads can still be restored by the frontend; +- PUT 404 can recreate a missing draft row; +- broken image drafts can fall back to the source image; +- final close persist is best-effort. + +## Signatures, Emoji, Fonts + +`routes.signature_routes` owns reusable signature/stamp rows. Signature image payloads are normalized to bounded PNG base64, encrypted at rest, and owner-filtered; SVG signature input is not preserved. Document PDF render/export paths owner-filter signature IDs before stamping. + +`routes.emoji_routes` owns same-origin OpenMoji black SVG proxy/caching. It validates codepoint filenames, caches SVGs under `data/emoji_cache`, and returns transparent no-store SVGs for invalid, unknown, or unreachable codepoints. `static/js/emojiPicker.js` is a curated inline monochrome picker. + +`routes.font_routes` owns deriving available custom font family names from static font files under `static/fonts/custom`. + +## Visual Reports + +`src.visual_report` owns generated research/report HTML image behavior: HTTPS Open Graph image filtering, hero images, section images, icon/logo filtering, hide/reroll client controls, and inline JSON escaping for scripts. + +Research routes and handler code own hidden-image persistence. Visual reports render model/source-influenced Markdown to HTML, so raw HTML/link/image sanitization remains security-sensitive. + +## Security Policy + +Media routes are cookie/current-user surfaces unless they explicitly implement token owner/scope handling. Bearer-token callers that arrive as synthetic `api` users should not be treated as owner-scoped media API clients without explicit policy. + +Known boundaries: + +- image-generation routes require `can_generate_images`; +- image proxy/editor endpoints currently resolve client-selected, DB-selected, or fallback image model endpoints without full owner-scoped endpoint-key policy or uniform outbound revalidation; +- generated-file serving allows rowless files and null-owner compatibility rows; +- uploads are byte-limited and extension-gated, with content sniffing available through `UploadHandler.detect_content_type()` when `python-magic`/`libmagic` is installed; +- several base64 JSON editor routes accept large decoded image payloads and need route-level size discipline; +- gallery DB filenames should be joined through shared generated-media path helpers before filesystem operations; +- editor draft source image IDs, payloads, and thumbnails are owner-scoped by draft owner but do not fully validate source-gallery ownership or payload size; +- emoji proxy constrains codepoint filenames and degrades invalid, unknown, or unreachable SVGs to transparent no-store placeholders, but remote SVG content still deserves security review; +- visual report Markdown HTML/link/image output needs continued sanitization coverage. + +## Degraded And Compatibility Behavior + +- Uploaded images record display dimensions with EXIF orientation when possible; EXIF failures warn/degrade. +- Video uploads skip EXIF and have no metadata extraction yet. +- Missing generated files are skipped in ZIP downloads; if all are missing, the route returns no files found. +- Soft delete commits the gallery row state before removing the disk file, so a failed DB write does not orphan a missing image row. +- AI tagging can fail when disk files are missing. +- Static JS/CSS/HTML assets revalidate because there is no frontend build/versioning. +- Gallery/editor frontend state includes stale-while-revalidate and listener cleanup to avoid stale handlers. +- `edit_image` tool schema/implementation currently appears stale against implemented `/api/image/*` and `/api/gallery/*` routes. + +## Testing Coverage + +Existing tests cover EXIF dimensions, owner-filter helper behavior, direct upload limits, image-generation privilege source shape, sharpen auth, gallery null-user denial, endpoint SSRF/source checks, editor draft payload validation, font family derivation, visual-report helper behavior, gallery CLI previews, and selected security regressions. + +Route-level coverage is thin for full gallery CRUD/album/tag/download/delete flows, generated-image serving, editor draft owner CRUD, signature owner CRUD, emoji proxy/cache behavior, image-tool degraded responses, optional dependency fallbacks, and frontend editor behavior. + +## Current Gaps + +- Owner-scoped endpoint-key resolution is needed for image proxy/editor routes. +- Media routes need a clear API-token policy: reject token callers, or implement owner/scope handling. +- Generated-image serving needs live route tests for invalid filenames, rowless files, owned rows, null-owner rows, MIME/cache headers, and cross-owner behavior. +- Mutable generated filenames plus immutable cache headers need cache-busting tests for replace/save-over-original flows. +- Base64 JSON editor payload size limits need hardening; upload content sniffing should keep native/Docker parity coverage as dependencies change. +- MCP image generation needs an owner attribution decision or explicit admin-only documentation. +- `edit_image` tool route mapping appears stale. +- Emoji SVG proxy/cache and visual-report raw HTML/link sanitization need stronger tests. +- Optional image dependency fallbacks are mostly untested. diff --git a/specs/integrations.md b/specs/integrations.md new file mode 100644 index 000000000..70e7992c6 --- /dev/null +++ b/specs/integrations.md @@ -0,0 +1,195 @@ +# Integrations + +Last updated: dev@df2fad2 | 2026-07-12 + +## Scope + +This spec covers external integration surfaces in: + +- `routes/codex_routes.py`; +- `integrations/codex/*` and `integrations/claude/*`; +- `routes/api_token_routes.py` and bearer-token handling in `app.py`; +- `routes/auth_routes.py` integration CRUD/test routes; +- `src/integrations.py` and `data/integrations.json`; +- `routes/webhook_routes.py` and `src/webhook_manager.py`; +- task webhook generation/triggering in `routes/task_routes.py`, `app.py`, and `static/js/tasks.js`; +- companion/mobile pairing in `companion/routes.py` and `companion/pairing.py`; +- provider OAuth/device-flow endpoint links in `routes/copilot_routes.py`, `routes/chatgpt_subscription_routes.py`, `routes/device_flow.py`, and `ProviderAuthSession` rows; +- integration UI surfaces in `static/js/settings.js` and `static/js/admin.js`; +- database models `ApiToken` and `Webhook`. + +The SQLAlchemy `Integration` model exists in `core/database.py`, but current Settings generic integration CRUD uses `src/integrations.py` and `data/integrations.json`. + +## Scoped Agent Runtime + +`/api/codex/*` is the canonical scoped HTTP surface for external coding agents. Claude Code uses the same runtime endpoints; `/api/claude/plugin.zip` only delivers the Claude skill bundle. + +`routes.codex_routes` owns: + +- `/api/codex/capabilities`; +- todos list/manage through `do_manage_notes()`; +- email list/read/draft/send; +- memory list/add/delete; +- calendar list/create/delete; +- document list/read/create/delete; +- Cookbook task/server/output/cached-model/preset/serve/adopt/stop controls. + +`_scope_owner()` owns scope checks and token-owner resolution. `_as_owner()` temporarily runs borrowed route handlers as the scoped owner and restores request state afterward. Borrowed email, memory, calendar, and document route handlers own their domain behavior; Codex routes only adapt them behind scoped access. + +Runtime behavior: + +- missing scopes return 403; +- invalid payloads return 400; +- unavailable borrowed route surfaces return 503; +- capabilities expose scope-derived booleans and partial availability flags; +- email send and destructive actions remain described as confirmation-required behavior in bundled agent instructions. +- Cookbook adopt/stop paths validate stored remote SSH host and port before interpolating them into SSH commands. + +The local integration skill/helper files require `ODYSSEUS_URL` and `ODYSSEUS_API_TOKEN`. They must use `/api/codex/*` and must not bypass Settings/token scopes through SSH, Docker, direct DB access, local files, MCP internals, or app imports. Helper scripts refuse non-`/api/codex/*` paths. + +## Bundle Distribution + +`/api/codex/plugin.zip` ships the Codex plugin tree from `integrations/codex/`. `/api/claude/plugin.zip` ships only the Claude `skills/` subtree from `integrations/claude/skills/`. These routes require an authenticated browser/user request and do not embed an API token. + +Setup instructions are duplicated in integration READMEs and `static/js/settings.js`; they need to stay aligned with live route surfaces and `/api/codex/capabilities`. + +## API Tokens + +`routes.api_token_routes` owns token profiles, allowed scopes, scope normalization, token creation/update/revocation, and profile metadata shown in Settings. Partial updates preserve existing scopes unless new scopes are supplied, owner checks apply to update/delete, and write scopes auto-include their read scope where applicable. + +`app.py` owns bearer-token validation. It accepts `Bearer ody_...`, checks a bcrypt hash through a prefix cache, updates `last_used_at` asynchronously, and stamps: + +- `request.state.current_user = "api"`; +- `request.state.api_token = True`; +- `request.state.api_token_owner`; +- `request.state.api_token_scopes`. + +The raw token is returned only on creation. Stored state is hash, prefix, owner, scopes, active flag, and timestamps. Token create/update/delete invalidates the auth middleware cache. Companion pairing also mints chat-scoped `ApiToken` rows and invalidates that cache. + +Current API-token consumers include: + +- `/api/codex/*` scoped agent routes; +- `/api/v1/chat` synchronous external chat; +- `/api/models` catalog reads for `chat`-scoped token owners; +- companion read endpoints; +- selected session and owner-attribution helpers described in `auth-security.md`. + +The Cookbook scoped-agent surface currently exposes `cookbook:read` and `cookbook:launch` in Settings and checks them in Codex routes; those scope names must stay reconciled with `routes.api_token_routes.ALLOWED_SCOPES`. + +## Generic API Integrations + +`src.integrations` owns generic API integration presets, `data/integrations.json`, API-key encryption/decryption, secret masking, plaintext-key migration, enabled integration prompt text, and `execute_api_call()`. + +`routes.auth_routes` owns admin-only HTTP CRUD/test routes for these integrations. Presets are public metadata. The ntfy test route is special: it publishes a real test notification to the configured reminder topic instead of only probing server health. + +`api_call` is the agent/tool execution path for configured integrations. It is blocked for non-admin/public users by tool security, accepts only relative paths, uses the admin-configured base URL/auth settings, and returns truncated external responses to the model, including a sentinel when long JSON lists are shortened. Admin-authored integration descriptions are prompt context; external responses remain untrusted data. + +`execute_api_call()` normalizes base URLs to HTTP(S) scheme, hostname, and +path-only values, rejects request paths that are not relative absolute paths +(`/...`) or that carry schemes/fragments, treats `/` as the base URL without +appending an extra slash, and checks the final URL through `src.url_safety`. +Link-local/metadata targets are always rejected; setting +`INTEGRATION_API_BLOCK_PRIVATE_IPS=true` also rejects loopback/RFC1918/private +addresses for operators who do not need LAN integrations. + +Current call sites include: + +- `src.agent_loop` injecting enabled integration descriptions; +- `src.tool_implementations.do_api_call()`; +- task scheduler discovery/check-ins; +- note reminder delivery through ntfy integrations and the generic webhook reminder channel. + +## Webhooks And External Chat + +Outgoing webhooks are admin-managed `Webhook` rows. `routes.webhook_routes` owns CRUD/test/toggle/delete and `/api/v1/chat`. `src.webhook_manager` owns allowed event validation, public URL validation, delivery-time URL revalidation, DNS-rebinding-safe pinned-IP delivery, HMAC signing, fire-and-forget delivery, in-flight task references, and delivery status/error persistence. Sanitized delivery errors redact IPv6-style address details. + +Allowed outgoing events are: + +- `session.created`; +- `chat.message`; +- `chat.completed`; +- `webhook.test`. + +Current webhook event emitters include session creation, chat message/completion paths, and `/api/v1/chat` completion. + +`/api/v1/chat` is an inbound external chat endpoint. It requires a `chat` API token, checks session ownership before resume, can create a session from a direct API key, and otherwise falls back to the first owner-visible enabled model endpoint. Token-supplied direct `base_url` values use public-URL validation; configured endpoints remain admin-trusted. Logs and delivery/error text that include endpoint URLs should pass through URL redaction helpers before persistence or diagnostics. + +## Task Webhooks And Event Triggers + +Task webhook triggers are separate inbound webhooks. `app.py` exempts only `/api/tasks/{task_id}/webhook/{token}` from normal auth so external callers can trigger tasks without cookies. `routes.task_routes` owns token generation/regeneration and validates task id, token, and active status before queueing a run. + +`static/js/tasks.js` displays the live task webhook URL. `scripts/odysseus-webhook` is stale against this route shape and is not authoritative. + +Event-triggered tasks use `src.event_bus`; task execution and scheduling ownership lives in `calendar-tasks-notes.md`. + +## Companion Pairing + +`companion.routes` owns companion/mobile HTTP routes: + +- `/api/companion/ping`; +- `/api/companion/info`; +- `/api/companion/models`; +- `/api/companion/pair`. + +Read endpoints accept session or bearer-token callers and resolve the effective owner for visible rows. Model responses omit API keys. Pairing `GET` renders the admin form; pairing `POST` is admin-cookie only, mints a normal chat-scoped API token, invalidates the auth token cache, and returns a host/port/token payload as HTML or JSON. + +`companion.pairing` owns LAN host detection, pairing payload shape, token minting, and optional QR generation. QR rendering depends on optional `qrcode`; if unavailable or failing, pairing still returns the text payload. + +## Unified Settings Surface + +The Settings Integrations view aggregates several subsystem surfaces: + +- generic API integrations; +- Codex/Claude agent token setup; +- CalDAV, CardDAV, email accounts including Google Workspace/.edu OAuth connect flows, MCP/OAuth links, provider device-flow links, and agent tokens. +- provider-auth backed model endpoints such as ChatGPT Subscription and Copilot, where device-flow credentials live in provider auth rows rather than endpoint API-key fields. + +Vault and companion/mobile setup are separate settings/route surfaces today, not entries in the unified add-integration list. + +This spec owns the cross-integration framing and agent/token/webhook surfaces. Domain internals stay with their subsystem specs: calendar, email/contacts, shell-MCP, vault/auth, and settings-admin. + +## Degraded And Compatibility Behavior + +- 403 from scoped APIs means a settings/scope restriction. +- 503 from Codex borrowed routes means the domain route surface is unavailable. +- Missing or corrupt `data/integrations.json` loads as an empty list; non-object rows are ignored. +- Plaintext generic integration API keys migrate to encrypted storage on load. +- Webhook delivery has no retry/backoff queue; the persisted state is last status or sanitized last error. +- Webhook URLs are validated at create and delivery time, redirects are disabled, + and delivery connects to the IP set validated immediately before the request. +- Companion LAN detection is best-effort and falls back to local host/port defaults. +- `ODYSSEUS_URL` must be reachable from the external coding agent; no Docker/native URL rewrite is performed. + +## Security And Provenance + +- API-token routes must either enforce a relevant scope or document an explicit exception. +- Codex/Claude plugin zips must not expose secrets beyond source instructions and helper files. +- Webhook list responses expose `has_secret`, not the secret value. +- Webhook secrets are encrypted when an API key manager is available; plaintext fallback is legacy/degraded behavior. +- Outgoing webhook signatures use `X-Odysseus-Signature`. +- Generic integration API keys are encrypted at rest and masked in API responses. +- Generic integration base URLs are admin-configured and not the same public-only policy as webhook URLs. +- `api_call` output and remote integration responses are untrusted model context. +- Pairing payloads expose the raw chat token once through HTML/JSON/QR; persisted token storage is hash/prefix only. + +## Testing Notes + +Current targeted coverage includes API-token CRUD basics, chat-scoped `/api/models` token access, companion pairing/read-only owner scoping, webhook SSRF validation, webhook auth-exempt source checks, webhook CLI token masking, integration-store shape/encryption migration, Google email OAuth route/helper behavior, Cookbook API-token scopes, Cookbook adopt SSH host validation, and `/api/v1/chat` base-url/fallback owner scoping. + +The integration audit also ran the targeted venv subset covering those areas with 52 passing tests and one warning. + +## Current Gaps + +- Codex/Claude scoped routes, owner restoration, degraded 503 behavior, plugin zip contents, and helper-script path refusal need focused regression tests. +- Token profile/update behavior and Settings agent-token scope toggles need direct coverage. +- Codex Cookbook scopes need continued Settings, route-check, and `ALLOWED_SCOPES` regression coverage. +- Generic integration HTTP CRUD/test routes, `execute_api_call()` auth modes, response shaping, and frontend Settings/Admin flows need direct coverage. +- `do_manage_tokens()` does not match `/api/tokens` semantics for `ody_` prefix, owner, scopes, and cache invalidation. +- `do_manage_webhooks()` bypasses route behavior and does not cover signing-secret parity. +- Companion read endpoints should either require `chat` scope or be documented as an explicit scope-policy exception. +- Decide whether webhook secret plaintext fallback should remain accepted when the API key manager is unavailable. +- Decide whether generic integration base URLs should stay LAN-capable by + default or make `INTEGRATION_API_BLOCK_PRIVATE_IPS=true` the default. +- Decide whether admin-authored integration descriptions and `api_call` results need a shared untrusted-context wrapper. +- The dormant SQLAlchemy `Integration` model should be removed, migrated into use, or documented as legacy. +- `scripts/odysseus-webhook` still emits the removed `/api/webhook/{token}` path. diff --git a/specs/llm-models.md b/specs/llm-models.md new file mode 100644 index 000000000..66880abba --- /dev/null +++ b/specs/llm-models.md @@ -0,0 +1,151 @@ +# LLM Models And Endpoints + +Last updated: dev@e57f60b | 2026-07-20 + +## Scope + +This spec covers model/provider behavior in: + +- `src/llm_core.py`; +- `src/endpoint_resolver.py`; +- `src/model_discovery.py`; +- `src/model_context.py`; +- `src/model_capabilities.py`; +- `src/model_capability_readers/`; +- `src/task_endpoint.py`; +- `src/tls_overrides.py`; +- `src/copilot.py`; +- `routes/copilot_routes.py`; +- `routes/chatgpt_subscription_routes.py` and `routes/device_flow.py`; +- `routes/model_routes.py`; +- `routes/session_routes.py`; +- `routes/cookbook_routes.py`, `routes/hwfit_routes.py`, and `services/hwfit/`; +- `src/settings.py`; +- `core/database.py` model `ModelEndpoint`; +- frontend modules `static/js/models.js`, `static/js/modelPicker.js`, `static/js/model/matchKey.js`, `static/js/providers.js`, `static/js/settings.js`, `static/js/admin.js`, `static/js/compare/`, and Cookbook model-serving modules; +- chat, compare, research, STT/TTS, and utility-model call sites. + +## Provider Calls + +`src.llm_core` owns provider-call mechanics. It handles OpenAI-compatible calls, Ollama normalization, Anthropic payload conversion, GitHub Copilot and ChatGPT Subscription provider detection/header injection, NVIDIA provider routing, streaming, fallback calls, upstream error formatting, async/streaming host liveness caching, configured model-list cache reads, tool-call sanitization, reasoning/thinking stream routing, and provider-specific parameter rules. GitHub Copilot OAuth/device-flow orchestration lives in `routes/copilot_routes.py` and `src/copilot.py`; ChatGPT Subscription device flow uses `routes/chatgpt_subscription_routes.py`, shared device-flow helpers, and `ProviderAuthSession` rows. + +`llm_core` owns payload shape. Route files and chat/agent code should request a call; they should not duplicate provider-specific payload quirks. + +Kimi Code User-Agent discovery has both sync and async implementations. Async +post and stream paths probe `/models` through their existing async client and +await each candidate, so header negotiation does not block the event loop; both +paths share the accepted-value cache and 403 fallback policy. + +Provider-specific behavior is part of this layer: `LLM_CONNECT_TIMEOUT` controls the connect budget for sync and streaming calls, Kimi Code endpoints retry a small whitelisted User-Agent set on 403 and cache the accepted value, official Moonshot/Kimi Code and Anthropic Opus 4.7+ payloads omit `temperature` where required, reasoning models omit or clamp unsupported temperature values, and self-hosted compatible endpoints keep normal OpenAI-compatible parameters unless detected otherwise. Mistral/Moonshot/Kimi reasoning content, `gpt-oss` harmony channel output, and native/OpenAI-compatible Ollama thinking formats are normalized so hidden reasoning and visible text stay separated where the provider exposes that structure. Copilot request metadata is treated defensively; malformed/non-dict `request_flags` in the last message must not crash payload construction. + +## Canonical Provider And Model Shape + +`src.model_capabilities` owns canonical model family, task, modality, +capability, limit, evidence, assertion, deterministic-control, probe-result, +reasoning-control token, and display-query values. +`src.model_capability_readers` owns endpoint-scoped stable identity, lightweight +provider detection, record serialization, and normalization of already-fetched +provider payloads. Readers do no network I/O. Model-specific observations are +kept in `model-quirks.md`, not a runtime registry without a consumer. + +Provider support and model support are different facts. A provider may expose +tools, reasoning, vision, or multiple APIs while individual models differ. +Provider-native readers describe where model evidence can appear. Current +concrete readers cover generic OpenAI-compatible identity, OpenAI, OpenRouter, +Google, Ollama, LM Studio, and llama.cpp. Identity-only model lists remain +unknown. + +Reader dispatch uses an explicit vendor first, then endpoint kind, hostname +suffix, and common local-port hints. Generic payload handling accepts `data[]` +or `models[]` items with `id`, `name`, or `model`; it does not accept a bare +list and never promotes capability-looking fields. Unknown fields remain in +the in-memory raw record. See [model-capability-canonical.md](model-capability-canonical.md), +[model-quirks.md](model-quirks.md), and the +[provider map](model-providers/_readme.md). + +This canonical layer is currently exercised by focused unit tests but is not +wired into runtime discovery, endpoint resolution, model context, request +shaping, or frontend pickers. `routes/model_routes.py` model probes continue to +return model IDs through their existing runtime path. + +Route-level probe helpers in `routes/model_routes.py` are the current exception: they build minimal provider-specific probe payloads using `llm_core` detection helpers. Keep probe behavior aligned with `llm_core` provider adapters. LLM provider HTTP clients and endpoint probes share `src.tls_overrides.llm_verify()`, which can add an operator-provided `LLM_CA_BUNDLE` on top of normal certificate verification without turning verification off or widening that trust to arbitrary URL fetches. + +## Endpoint Resolution + +`src.endpoint_resolver` owns endpoint normalization and URL construction: + +- base URL normalization; +- chat and model-list URL construction; +- endpoint ID resolution; +- chat, utility, and vision fallback candidate selection; +- Tailscale hostname resolution where available. + +OpenAI-compatible model-list URL construction preserves `/v1` bases and inserts `/v1/models` for bare local bases such as LM Studio `http://localhost:1234`. + +`routes/model_routes.py` owns model endpoint CRUD, admin provider discovery/probing, visible/hidden/pinned model lists, endpoint kind and refresh policy, curated/extra model partitioning, `/api/models` catalog caching, Docker loopback rewriting, tool-support probing, provider-auth linkage, endpoint-dependent settings cleanup, and owner filtering. Endpoint dedupe allows the same base URL under different API keys and surfaces API-key fingerprints/key presence without returning secrets. + +`routes/session_routes.py` owns binding sessions to endpoint IDs, owner-scoped header construction, raw-endpoint rejection for non-admin users, model validation, and persisted session headers. Compare panes and normal chat session creation use this path. + +`ModelEndpoint` rows own API keys, base URLs, cached/hidden/pinned models, model type, endpoint kind, refresh mode/interval/timeout, supports-tools state, nullable owner, optional provider-auth linkage, and provider metadata. `owner = NULL` means legacy/shared; non-null rows are private to that owner, while admins can see all. Secret fields must remain encrypted and scrubbed in responses. + +Decrypted endpoint headers can be copied into session metadata for chat use. Endpoint deletion must clear dependent settings and copied session headers. + +## Model Discovery And Lists + +`src.model_discovery` owns host/env/Tailscale/local-port scanning for model servers. Admin `/api/providers` and `/api/discover` use that scanner; endpoint CRUD, test, refresh, and hidden-model controls are frontend-owned by `static/js/admin.js`. + +`/api/models` is the normal picker/catalog surface. It is auth/owner scoped, per-user/admin-flag cached briefly, can trigger background refresh, preserves offline endpoint rows, filters hidden models, and preserves pinned model IDs for UI selection. API-token callers must carry `chat` scope and a token owner before they can list models. Proxy/API endpoints can be marked cached-first/manual so large upstream catalogs are not repeatedly probed, while explicit refresh paths use longer manual timeouts. Local endpoints get cheap reachability probes before expensive refreshes where possible, and endpoint responses can include explicit `supports_tools` state for schema-emission heuristics. Google Gemini API endpoints use the native paginated `generativelanguage.googleapis.com/v1beta/models` catalog, send API keys in `x-goog-api-key`, retain only content-generation model IDs, and default to manual refresh unless the caller explicitly chooses another mode. Probe failure returns no curated Google fallback. `static/js/models.js` and `static/js/modelPicker.js` own the sidebar/picker catalog; `static/js/model/matchKey.js` owns longest-substring model-info/pricing key matching; `static/js/settings.js` owns default, utility, vision, image, TTS, STT, and fallback selectors. + +`src.task_endpoint` owns background-task endpoint/model resolution for task routes and scheduler callers. It resolves `task_endpoint_id`/`task_model` through the normal endpoint resolver with owner context. + +Cookbook and HWFit own local model download, serve, ranking, and auto-registration flows. They can create LLM or image `ModelEndpoint` rows, but provider dispatch remains owned by `llm_core`/endpoint resolution. + +## Context Length + +`src.model_context` owns model context-length lookup/query and token estimation. Cache keys include endpoint plus model so identical model names on different endpoints do not bleed context-window data. Unknown proxy/API models can pick up real context windows from endpoint catalog metadata such as `context_length`; otherwise unknown lengths stay explicit unknowns rather than default values. Known lengths feed chat/agent token-budget scaling through `src.context_budget`. Token estimation counts assistant `tool_calls` arguments so compaction sees tool-only turns instead of underestimating them. Chat/agent context budgeting should call this layer instead of hardcoding model windows. + +## Runtime Fallback And Routing + +Streaming chat and agent mode use configured fallback candidates through `stream_llm_with_fallback()`. A candidate commits only after a non-empty text/reasoning delta, a tool-call delta, or a non-empty completed tool-call event. Metadata is buffered until that point, so metadata followed by `[DONE]` or a clean empty completion can fall through to the next candidate without leaking stale model/usage state. Tool-call deltas are forwarded immediately and prevent fallback. If every candidate completes without substantive output, the stream emits one terminal 502-shaped error. Non-streaming chat and rewrite routes do not automatically get the same fallback path. Utility callers may use `llm_call_async_with_fallback()`, and vision uses its own fallback loop. + +Model selection has three layers: endpoint resolver hidden-model and first-chat-model selection, `/api/default-chat` per-user default/fallback resolution, and frontend picker auto-selection for empty sessions. + +Image routing uses model-name prefixes and `ModelEndpoint.model_type == "image"` to bypass text chat and generate media. Vision analysis uses configured vision models and `vision_model_fallbacks`; image and vision endpoint lifecycle changes should update chat, document processing, Cookbook, and settings UI together. + +Provider tool calls are untrusted requests, not authorization. `supports_tools` controls schema emission only; `llm_core` normalizes provider tool-call payloads, while execution authority remains in `src.tool_execution`, `src.tool_security`, and agent-tool policy. + +## Degraded And Platform Behavior + +- Provider offline or probe failures should surface actionable errors without crashing the app. Async calls retry transient 429/502/503/504 responses before failing. +- Docker deployments may need loopback URL rewriting from `127.0.0.1` to host-accessible addresses. +- Fallback selection must preserve endpoint identity and owner scope. User/API-token LLM dispatch that can carry configured endpoint keys must pass the effective owner into resolver calls. +- Async and streaming calls use dead-host cooldown; sync utility/vision calls do not have identical cooldown coverage. +- llama.cpp slot-affinity routing is local-endpoint behavior only and must not be applied to cloud/provider endpoints. +- Hidden, pinned, cached, endpoint-kind, refresh-policy, and offline model state are UI/runtime compatibility data. Pinned models may not participate in every resolver auto-pick path unless code explicitly includes them. +- SSE/stream parsers tolerate null choice/usage/tool-call entries and null streaming tool-call arguments; provider events should degrade to empty text or shaped stream errors instead of crashing the chat loop. +- Provider adapters carry small model-specific quirks: Opus 4.7+ and official Kimi/Moonshot code payloads omit `temperature`, Kimi/Moonshot/Mistral reasoning content is preserved separately, ChatGPT Subscription refreshes bearer credentials, native Ollama can handle multimodal content, and Ollama `/v1` responses for Qwen3/Gemma4-style thinking can suppress thinking text when requested. + +## Security Policy + +- Endpoint API keys are encrypted in `ModelEndpoint.api_key` and never returned by endpoint APIs; admin surfaces return key presence only. +- Endpoint CRUD, probes, provider discovery, and most endpoint configuration are admin-cookie or internal-tool gated. +- `/api/models` is auth/owner scoped for configured deployments; API-token access requires `chat` scope and token-owner attribution. +- Admin-created model endpoints may target local/LAN servers. Non-admin chat session creation must use registered endpoint IDs. API-token `/api/v1/chat` requires `chat` scope and validates direct `base_url` with public-only URL checks. + +## Current Call Sites Include + +- chat streaming and non-streaming calls; +- agent loop calls with optional tool schemas; +- compare pane calls; +- research synthesis/probe calls; +- utility model fallbacks for summarization/extraction; +- frontend Settings and model picker endpoint management. + +## Current Gaps + +- Runtime provider detection, model curation, and frontend logos are still split across `llm_core`, `model_routes`, and `providers.js`; the canonical reader package has no production consumer yet. +- Provider-specific behavior is concentrated in `llm_core.py`, which is large and easy to regress. +- Several runtime request builders still use model-name heuristics. They should migrate only after endpoint/provider code supplies structured identity and a real consumer contract; the canonical catalog does not add a parallel quirk matcher. +- Endpoint identity and fallback behavior need careful review when new OAuth/subscription providers are added. +- Owner must continue to be threaded through new utility/research/default endpoint-resolution call sites so provider keys stay isolated. +- `/api/models` owner-scoped listing/cache behavior, shared/private endpoint dedupe, endpoint-kind refresh policy, fallback-chain owner scope, and image endpoint create/list/update lifecycle need stronger route-level regression coverage. diff --git a/specs/memory-skills.md b/specs/memory-skills.md new file mode 100644 index 000000000..62256620d --- /dev/null +++ b/specs/memory-skills.md @@ -0,0 +1,121 @@ +# Memory And Skills + +Last updated: dev@28d27ee | 2026-07-17 + +## Scope + +This spec covers persistent memory and user skills in: + +- app wiring in `app.py` and `src/app_initializer.py`; +- active legacy memory managers `src/memory.py` and `src/memory_vector.py`; +- canonical memory routes in `routes/memory/memory_routes.py`, with `routes/memory_routes.py` as a compatibility shim; +- chat memory/skill gating in `routes/chat_helpers.py`; +- memory compatibility modules in `services/memory/memory.py`, `services/memory/memory_vector.py`, and `services/memory/service.py`; +- provider abstractions in `src/memory_provider.py`; +- LLM extraction/audit in `services/memory/memory_extractor.py`; +- skill storage, format, import, and extraction in `services/memory/skills.py`, `services/memory/skill_format.py`, `services/memory/skill_importer.py`, and `services/memory/skill_extractor.py`; +- skill routes in `routes/skills_routes.py`; +- prompt/tool call sites in `src/chat_processor.py`, `src/agent_loop.py`, `src/ai_interaction.py`, `src/tool_implementations.py`, `src/tool_execution.py`, `src/tool_schemas.py`, and `src/tool_security.py`; +- MCP and Codex surfaces in `mcp_servers/memory_server.py` and `routes/codex_routes.py`; +- backup/admin/CLI surfaces in `routes/backup_routes.py`, `routes/admin_wipe_routes.py`, `scripts/odysseus-memory`, `scripts/odysseus-skills`, and `scripts/odysseus-backup`; +- frontend modules `static/js/memory.js` and `static/js/skills.js`; +- tests under `tests/test_memory_*`, `tests/test_builtin_memory_consolidation.py`, `tests/test_skill_*`, and `tests/test_skills_*`. + +## Memory Runtime + +`src.app_initializer.initialize_managers()` creates the active `src.memory.MemoryManager` and `src.memory_vector.MemoryVectorStore` used by app startup. `routes.memory.memory_routes` imports through `services.memory` but is passed the startup manager instances; top-level `routes.memory_routes` is a `sys.modules` compatibility shim. + +`MemoryManager` owns JSON-backed memory storage in `data/memory.json`, validation, owner fields, pinned state, use counts, and text/keyword similarity. `MemoryVectorStore` owns semantic lookup when Chroma and embeddings are reachable. `services/memory/memory.py` and `services/memory/memory_vector.py` are compatibility imports for these canonical `src` classes. `src.memory_provider.NativeMemoryProvider` is the provider interface around the native manager/vector pair. + +Chat memory behavior: + +- chat preferences and incognito state gate memory preface use; +- pinned memories are loaded for the owner; +- retrieved memories use keyword matching plus optional vector scoring; +- inserted memory is wrapped as untrusted context; +- memory use counts are incremented after insertion. + +`services/memory/memory_extractor.py` owns LLM-assisted extraction, audit, and validation flows. It requests model behavior and writes through the memory manager; it does not own chat session persistence. + +Extraction handles reasoning-model response shapes and records explicit dislike/drop preferences as `dislikes` rather than losing them to generic fact handling. + +## Skills Runtime + +`services/memory/skills.py` owns disk-backed skill storage under `data/skills///SKILL.md`, plus `_usage.json` usage/audit sidecars. Legacy `data/skills.json` is a read-only fallback/import source, not the current write shape. + +`services/memory/skill_format.py` owns frontmatter/body parsing and emission. `services/memory/skill_importer.py` resolves public GitHub/skills URLs, fetches bundle files with strict public-network URL safety, and chooses/imports `SKILL.md`. Import fetches disable automatic HTTP redirects, follow at most five redirects manually, and re-run the private/loopback/link-local guard before every hop; GitHub final-host checks still apply after the network guard. `routes/skills_routes.py` owns HTTP skill CRUD/search/index/markdown/import behavior, owner filtering, tag/search handling, skill test jobs, audit-all jobs, scheduled audit entry points, and admin-gated built-in tool instruction overrides. + +Skill extraction is owned by `services/memory/skill_extractor.py`. It can suggest or save skills from conversations, tries valid brace-delimited JSON candidates with `JSONDecoder.raw_decode()`, rejects ambiguous multiple top-level JSON objects instead of guessing, and saved skills remain user-editable data. + +Agent skill behavior: + +- matched skills are owner-scoped, confidence-gated, usage-counted, and wrapped as untrusted context; +- `index_for()` exposes published skills plus teacher-escalation drafts gated by platform and toolsets; `active_toolsets=None` means the caller has no explicit toolset knowledge and does not hide `requires_toolsets` skills, while an explicit list applies the gate; +- user prefs such as skills enabled, auto-approve, and max injected skills shape runtime insertion; +- the level-0 base skill index currently calls `index_for(owner=None)`, so it is not fully owner-scoped. + +## Tools, MCP, And Backup + +Native `manage_memory` and `manage_skills` tool paths pass owner context and use in-process policy gates. Manual memory add can choose a memory category instead of always defaulting to `fact`, and route-side manual add validates the source session owner before attaching session-derived memories. `mcp_servers/memory_server.py` lazy-initializes `src` managers and exposes list/add/edit/delete/search. It can scope to `ODYSSEUS_MCP_MEMORY_OWNER` or `ODYSSEUS_MEMORY_OWNER`; if the JSON store contains owner-bearing entries and no owner env is configured, it returns an owner-scope error instead of listing or mutating across owners. Ownerless stores remain ownerless compatibility mode. + +The direct `odysseus-memory add` CLI tolerates non-object legacy/corrupt rows +when checking whether its newly added entry is already present; it ignores +those rows instead of calling mapping methods on them and crashing the add. + +`/api/export` owner-filters memories and skills. `/api/import` imports skills through current disk-backed `SkillsManager` APIs, stamping missing owners to the importer and preserving supported skill metadata. Full data snapshots through `scripts/odysseus-backup` preserve on-disk skill trees, memory JSON, and caches differently from JSON import/export. + +## Compatibility State + +Memory and skills are partially migrated: + +- app startup, MCP, and some tools still use `src.memory*`; +- services memory modules remain relevant for imports/tests, with memory and vector modules re-exporting canonical `src` implementations; +- `services/memory/service.py` is a compatibility facade around the canonical managers, but it remains ownerless and should not be assumed equivalent to route/tool owner policy; +- skills are service-owned and disk-backed, while backup import and some compatibility paths still expect older JSON/list shapes. + +## Degraded Vector Memory + +Chroma is an external HTTP service. Native defaults use `localhost:8100`; Docker uses `chromadb:8000`. Embeddings prefer configured HTTP endpoints and can fall back to local FastEmbed. + +Startup can degrade to keyword-only memory when vector initialization fails. Extraction/audit paths catch vector failures and continue with text/JSON behavior. Vector dedup is checked against the current owner before suppressing a candidate, and audit rebuilds preserve other owners' vector rows. Chat retrieval assumes a healthy startup vector store remains usable, so post-start vector failures can still break memory retrieval unless handled by the caller. + +Admin wipe currently has a vector cleanup compatibility gap because it imports a nonexistent helper before attempting vector clearing. + +## Policy + +Saved memories and skills are untrusted source data when shown to the model. A stored skill may contain useful instructions, but it is still user-editable content and must be framed consistently with prompt-injection policy. + +Owner isolation is surface-specific: + +- HTTP memory and skills routes are expected to owner-filter normal user data; +- native memory/skill tools are expected to pass owner context; +- Codex exposes scoped token memory behavior separately; +- normal memory/skills routes are cookie/current-user surfaces, not scoped token APIs; +- MCP memory uses an environment-configured owner for owner-scoped stores, while the agent level-0 skill index currently has ownerless/global behavior; +- vector dedup during memory extraction suppresses only same-owner or legacy-ownerless vector matches. + +Skill test/audit flows intentionally run user-editable `SKILL.md` content as instructions inside controlled jobs. Those jobs rely on route owner checks, admin gates where applicable, and tool execution policy. + +Skill import is admin-gated defense-in-depth, but imported URLs are still +untrusted network input. Initial and redirected targets must remain public; +`httpx` redirect following must stay disabled so validation happens before +each connection rather than after a redirect has already reached a private +service. + +User rename flows update skill frontmatter owner fields and `_usage.json` owner keys alongside memory/upload/research ownership migrations. + +## Testing Coverage + +Existing tests cover memory bullet extraction, extraction degraded-vector behavior, manager owner isolation, auto-memory owner isolation, MCP memory list shape and owner-scope behavior, skill owner update/delete, skill prompt-injection wrapping, toolset gating, skill save no-rename behavior, skill-import private-target and redirect-hop SSRF rejection, CLI non-object row handling, and selected route owner checks. + +Route-level memory CRUD/security, skills route security, MCP memory behavior, vector degraded writes, compatibility facade owner behavior, backup skill import, admin vector cleanup, and frontend endpoint wiring need broader coverage. + +## Current Gaps + +- `services/memory/service.py` needs an explicit owner-scope/support decision before it is treated as a public memory API. +- The agent level-0 skill index should thread owner or be documented as an intentional local/global index. +- MCP memory still needs a deliberate multi-user UX/config decision, but current behavior avoids cross-owner access when owner-bearing rows exist without an explicit MCP owner env. +- Memory JSON import does not rebuild vector indexes. +- Admin wipe vector clearing is currently ineffective. +- Chat memory retrieval needs a graceful path for vector failures after startup. +- Route-level memory and skills security coverage is incomplete. diff --git a/specs/model-capability-canonical.md b/specs/model-capability-canonical.md new file mode 100644 index 000000000..68350ae05 --- /dev/null +++ b/specs/model-capability-canonical.md @@ -0,0 +1,184 @@ +# Canonical Provider And Model Capability Layer + +Last updated: dev@e57f60b | 2026-07-20 + +## Scope + +This spec covers the implementation introduced on current `dev` in: + +- canonical model values and query helpers in `src/model_capabilities.py`; +- record, identity, and provider-detection helpers in + `src/model_capability_readers/base.py`; +- reader dispatch in `src/model_capability_readers/__init__.py`; +- concrete readers for generic OpenAI-compatible, OpenAI, OpenRouter, Google, + Ollama, LM Studio, and llama.cpp payloads; +- regression coverage in `tests/test_model_capabilities.py` and + `tests/test_model_capability_readers.py`. + +The layer normalizes already-fetched JSON-compatible values. It performs no +network I/O, does not shape provider requests, does not persist its output, and +does not authorize model or tool use. No production caller currently consumes +the canonical records outside this package; runtime integration remains later +work. + +There is no `src/provider_capability_schemas.py`, capability-specific +diagnostics module, or runtime model-quirk registry on current `dev`. + +## Layer Boundaries + +- `src.model_capabilities` defines normalized families, tasks, modalities, + capabilities, evidence sources/confidence, assertion states, deterministic + controls, probe results, reasoning-control tokens, and display-surface + queries. +- `ModelCapability` owns family, primary task, input/output modalities, + capability tokens, limits, source, and confidence. +- `CapabilityAssertion` records claimed, verified, unsupported, or unknown + status for one capability. Missing evidence is not an unsupported claim. +- `DeterministicControl` records support evidence for controls such as + temperature, top-p, seed, tool choice, or prompt caching. A supported + request control is not itself a model capability. +- `CapabilityProbeResult` is an in-memory evidence shape that converts pass, + fail, or partial probe state into an assertion. No current runtime probe + stores or merges these objects. +- `CapabilityQuery` and `display_surfaces_for()` map a normalized capability + into candidate surfaces such as chat, vision chat, image generation, + embeddings, or reranking. They are not wired into current pickers. +- Reader `ModelCapabilityRecord` binds a vendor/model identity to the nested + capability object, assertions, deterministic controls, and optional raw + provider evidence. + +Provider transport support and per-model support are separate facts. Request +and response adapters remain in `src.llm_core` and related provider modules. +Model-specific observations remain in [model-quirks.md](model-quirks.md). + +## Current Serialized Shapes + +`ModelCapability.to_dict()` emits the nested capability shape: + +```json +{ + "family": "chat", + "primary_task": "chat.completions", + "modalities": { + "input": ["text", "image"], + "output": ["text"] + }, + "capabilities": ["tool_call", "vision"], + "limits": {"context_tokens": 131072}, + "source": "provider_reader", + "confidence": "provider_reported" +} +``` + +`ModelCapabilityRecord.to_dict()` wraps that value with `vendor`, `model_id`, +`stable_model_id`, `display_name`, `capability_assertions`, and +`deterministic_controls`. It does not currently emit a schema version or the +flat `provider`/`model`/`features`/`controls` shape. Raw provider fields are +included only when the caller passes `include_raw=True`. + +Endpoint configuration can explicitly map `model_type=llm` to chat and +`model_type=image` to image generation. Missing or unrecognized endpoint types +stay unknown rather than silently becoming chat-capable in this schema layer. + +## Identity And Reader Dispatch + +`records_from_payload()` selects a reader from an explicit `vendor`, or from +`detect_vendor(base_url, endpoint_kind)` when no vendor is supplied. + +Current detection order and behavior are: + +1. a recognized explicit endpoint kind; +2. hostname suffix checks for OpenRouter, OpenAI, Anthropic, Google APIs, and + Ollama Cloud; +3. common local ports: `11434` for Ollama, `1234` for LM Studio, `8000` for + vLLM, and `30000` for SGLang; +4. generic OpenAI-compatible for any other parsed host, otherwise unknown. + +These are normalization hints, not authorization. Current hostname checks use +plain string suffixes, and the local-port mappings are intentionally covered by +tests; callers must not treat the result as proof of endpoint trust. + +Implemented reader modules are `generic_openai`, `openai`, `openrouter`, +`google`, `llamacpp`, `ollama`, and `lmstudio`. Anthropic, Hugging Face, +SGLang, and vLLM have placeholder vendor IDs but currently dispatch through the +generic identity-only reader. Other explicitly supplied vendor strings are +also preserved while using that generic reader. + +Stable model identity is scoped in this order: + +- explicit endpoint ID; +- a short hash of normalized base URL when an endpoint ID is absent; +- `global` when neither endpoint identity is supplied. + +## Generic Identity-Only Contract + +The generic reader accepts mapping payloads containing `data[]` or `models[]`. +Each item must itself be a mapping and provide `id`, `name`, or `model`. +Bare-list payloads and `key`/`slug`-only items are not accepted by the current +implementation. + +The reader deliberately returns unknown family, modalities, capabilities, and +controls. It preserves the raw item on the in-memory record but does not parse +type/task fields, descriptions, ownership, supported-parameter lists, +capability-looking booleans, or token limits. + +## Provider-Native Readers + +- OpenAI keeps the official Models API identity-only. +- OpenRouter maps explicit architecture modalities, supported parameters, + limits, voices, and default parameters into family/capability/control state. +- Google maps the native Models resource. Embedding-only methods map to the + embedding family; content-generation methods do not prove modality or chat + family. Explicit thinking, limits, sampling fields, caching, and batch + methods are retained without parsing product names. +- Ollama treats `/api/tags` as identity-only and maps selected-model + `/api/show` capability tokens. Context can come from structured fields or a + parsed `num_ctx` line in the serialized `parameters` value. +- LM Studio maps native v1 `models[]` and v0-style `data[]` fields. A plain + OpenAI-compatible list without native type/capability fields stays unknown. +- llama.cpp can merge `/v1/models`, `/props`, and `/slots` evidence for one + served model. It records tool/streaming claims, explicit unsupported + vision/audio assertions, controls, and runtime/training/size limits. + +Readers tolerate non-object entries and unknown fields where their helpers +permit it. They do not infer authoritative capability from model IDs or display +names. + +## Evidence Semantics + +The canonical vocabulary includes admin override, endpoint configuration, +provider reader, Cookbook/Hugging Face, maintained registries, heuristic, +probe, and unknown sources. It also defines explicit, provider-reported, +registry, heuristic, and unknown confidence values. + +Those tokens make evidence representable; current `dev` does not implement a +global precedence, merge, expiry, or conflict-resolution engine. Assertions +generated by readers are usually `claimed`; a `CapabilityProbeResult` maps pass +to verified, fail to unsupported, and partial to claimed at the scope carried +by that object. + +## Tests + +Focused tests pin: + +- endpoint-kind, host, and common-port vendor detection; +- endpoint/base-URL-scoped stable IDs; +- unknown behavior for generic and official OpenAI lists; +- canonical normalization and display-surface matching; +- assertion, deterministic-control, and probe-result shapes; +- OpenRouter, Google, Ollama, LM Studio, and llama.cpp mappings; +- negative cases that avoid name-based media/capability inference. + +## Current Gaps + +- Canonical records are not yet used by runtime discovery, endpoint + resolution, model context, request shaping, or frontend pickers. +- Reader output is not persisted, refreshed, merged, or expired. +- Provider detection uses common-port hints and non-label-bounded hostname + suffix checks; consumers must not promote those hints into trust decisions. +- Only seven concrete readers exist; placeholder and other providers use the + identity-only generic reader. +- Generic fallback does not accept bare-list or `key`/`slug`-only payloads. +- There is no capability-specific diagnostic/logging path. +- Runtime request builders still contain model-name heuristics outside this + canonical layer. diff --git a/specs/model-providers/_readme.md b/specs/model-providers/_readme.md new file mode 100644 index 000000000..61847cb3a --- /dev/null +++ b/specs/model-providers/_readme.md @@ -0,0 +1,103 @@ +# Provider Capability Specs + +Last updated: dev@e57f60b | 2026-07-20 + +## Scope + +This directory maps serving-provider observations and current model-catalog +normalization into the canonical layer defined by +[model-capability-canonical.md](../model-capability-canonical.md). It records +current Odysseus implementation evidence, merged fixes, reproducible user +observations, and provider documentation without treating any single source as +global model truth. + +## General To Specific Resolution + +Read specs in this order: + +1. [openai-compatible.md](openai-compatible.md) for the conservative general + identity-only reader; +2. the serving-provider file for native endpoints, headers, request/response + observations, and catalog fields; +3. [model-quirks.md](../model-quirks.md) for model-specific observations. + +Provider files document transport; runtime adapters still own it. Model quirks +record only deviations and are not a second runtime matcher. Shared model facts +must not be copied into every provider file. An OpenAI-compatible provider is +not OpenAI: an explicitly supplied vendor string is preserved even when it uses +the generic reader. + +Current reader dispatch does not infer a provider from payload shape. It uses +an explicit vendor, then endpoint kind, hostname suffix, and common local-port +hints. The port hints map 11434 to Ollama, 1234 to LM Studio, 8000 to vLLM, and +30000 to SGLang. Those hints are normalization behavior, not endpoint trust. + +## Provider Map + +### Implemented canonical readers + +- [openai.md](openai.md): identity-only Models API plus Chat/Responses dialects. +- [openai-compatible.md](openai-compatible.md): generic compatible catalog and runtime dialect boundaries. +- [openrouter.md](openrouter.md): rich architecture, modalities, parameters, and limits. +- [google.md](google.md): native paginated Gemini Models API and GenerateContent. +- [ollama.md](ollama.md): `/api/tags`, `/api/show`, native chat, and OpenAI compatibility. +- [lm-studio.md](lm-studio.md): native v1 catalog/chat, explicit v0 compatibility, and OpenAI compatibility. +- [llama-cpp.md](llama-cpp.md): `/props`, `/slots`, OpenAI/Responses/Anthropic surfaces. + +### Placeholder identities using the generic reader + +- [anthropic.md](anthropic.md): identity-only Models API and native Messages runtime adapter. +- [vllm.md](vllm.md): common-port identity hint; deployment capability remains unknown. +- [sglang.md](sglang.md): common-port identity hint; parser/config-dependent capability remains unknown. +- [hugging-face.md](hugging-face.md): Hub observations and download/fit metadata without a canonical reader. + +### Provider observations without a dedicated canonical reader + +- [mistral.md](mistral.md): rich model cards, reasoning controls, and structured runtime content. +- [github-copilot.md](github-copilot.md): account model-list observations and required runtime headers. +- [chatgpt-subscription.md](chatgpt-subscription.md): Codex model identity and Responses event shape. +- [cohere.md](cohere.md): native endpoint/catalog observations; not currently normalized. + +### Other provider identity and general/identity-only observations + +- [moonshot-kimi.md](moonshot-kimi.md) +- [deepseek.md](deepseek.md) +- [groq.md](groq.md) +- [nvidia-nim.md](nvidia-nim.md) +- [cerebras.md](cerebras.md) +- [together.md](together.md) +- [fireworks.md](fireworks.md) +- [xai.md](xai.md) +- [zai.md](zai.md) +- [opencode.md](opencode.md) +- [perplexity.md](perplexity.md) +- [github-models.md](github-models.md) +- [venice.md](venice.md) +- [azure-openai.md](azure-openai.md) +- [bedrock.md](bedrock.md) +- [cloudflare-workers-ai.md](cloudflare-workers-ai.md) +- [atlas-cloud.md](atlas-cloud.md) +- [siliconflow.md](siliconflow.md) +- [minimax.md](minimax.md) + +### Other local/proxy serving identities + +- [local-compatible-engines.md](local-compatible-engines.md): MLX LM, TGI, + LMDeploy, LiteLLM, and unknown compatible deployments. + +## Provider Spec Template + +Each provider file records: + +- provider identity and API dialects; +- latest observed native catalog endpoint/envelope and capability-bearing fields; +- whether current source has a dedicated reader or only generic fallback; +- observed request, tool, text, reasoning, and control paths owned by runtime + adapters rather than the catalog reader; +- what remains per-model/unknown; +- Odysseus evidence and regressions; +- fallback/safety behavior and current gaps. + +Marketing capability lists and curated picker lists may guide research but do +not automatically become model claims. Provider-returned false values can be +negative evidence only at the same provider/endpoint/model scope. diff --git a/specs/model-providers/anthropic.md b/specs/model-providers/anthropic.md new file mode 100644 index 000000000..3b2b6a400 --- /dev/null +++ b/specs/model-providers/anthropic.md @@ -0,0 +1,46 @@ +# Anthropic Provider Shape + +Last updated: dev@e57f60b | 2026-07-20 + +## Scope + +Canonical placeholder vendor ID `anthropic`; Anthropic Messages runtime +adapter in `src/llm_core.py`. There is no dedicated Anthropic capability-reader +module; explicit/auto-detected Anthropic payloads use the generic identity-only +reader. + +## Catalog Shape + +`GET /v1/models` returns `data[]` model resources with `id`, `type: model`, +`display_name`, and `created_at`, plus pagination metadata. These fields prove +identity/availability only. Do not assume all listed Claude models share +vision, tools, reasoning, sampling, or context limits. + +## Request And Response Shape + +Native Messages uses a top-level `system`, alternating `messages`, content +blocks, `tools[].input_schema`, `tool_use` assistant blocks, and `tool_result` +user blocks. Text, thinking, signatures, server-tool blocks, and tool calls are +typed content rather than OpenAI roles/fields. Preserve block IDs/signatures +needed for continuation. + +Sampling and thinking support can be version/model specific. The Opus 4.7+ +sampling omission is a model-scoped runtime observation, not an Anthropic-wide +rule. Anthropic-compatible proxies are Anthropic dialect only when configured +or their exact payload/endpoint shape proves it (#3110). + +## Fallback And Safety + +Runtime provider detection uses label-bounded Anthropic host matching. The +canonical reader helper separately uses a plain `endswith("anthropic.com")` +hostname hint or explicit endpoint kind. A provider using Anthropic Messages +through another host must be explicit. Identity-only model cards remain +unknown. + +## Current Gaps + +- The public model list does not provide per-model canonical capability data. +- There is no dedicated Anthropic canonical reader; only `id`, `name`, or + `model` identity survives generic normalization. +- Runtime model-version parsing needs structured identity before a later + consumer can centralize sampling exceptions without another name matcher. diff --git a/specs/model-providers/atlas-cloud.md b/specs/model-providers/atlas-cloud.md new file mode 100644 index 000000000..71a8aaa87 --- /dev/null +++ b/specs/model-providers/atlas-cloud.md @@ -0,0 +1,21 @@ +# Atlas Cloud Provider Shape + +Last updated: dev@28d27ee | 2026-07-17 + +## Scope + +Canonical provider ID `atlas_cloud`; OpenAI-compatible provider proposed in +#5566 with live `/v1/models` observations for current Qwen/DeepSeek offerings. + +## Shape + +Treat the observed list as identity-only. Even capability-looking item fields +remain raw until an Atlas-specific discriminating shape intentionally maps +them. The model IDs observed by a PR demonstrate availability at that time, +not permanent capability or a reason to hardcode family-name behavior. + +## Fallback And Current Gaps + +Exact Atlas Cloud host or explicit kind preserves identity; otherwise use the +inventory fallback. The provider work is open/unmerged and has no independently +versioned rich catalog schema, so evidence remains provisional. diff --git a/specs/model-providers/azure-openai.md b/specs/model-providers/azure-openai.md new file mode 100644 index 000000000..f4f2ca77a --- /dev/null +++ b/specs/model-providers/azure-openai.md @@ -0,0 +1,26 @@ +# Azure OpenAI Provider Shape + +Last updated: dev@28d27ee | 2026-07-17 + +## Scope + +Canonical provider ID `azure_openai`; Azure deployment-scoped OpenAI dialects; +custom endpoints use explicit configuration. + +## Shape + +Azure commonly identifies deployments rather than globally stable model IDs. +Preserve endpoint, deployment ID, API version, and underlying model/version as +separate structured identity when returned. A standard OpenAI-compatible model +list is identity-only until an Azure-specific reader intentionally maps its +deployment fields. + +Request paths and authentication can be deployment/API-version specific; do +not blindly append public OpenAI paths or copy provider quirks. Capability and +limits are deployment scoped. + +## Fallback And Current Gaps + +Known `*.openai.azure.com` hosts select Azure OpenAI; other Azure gateways need +explicit kind. Odysseus lacks a native Azure deployment catalog reader and +structured API-version persistence in the canonical record. diff --git a/specs/model-providers/bedrock.md b/specs/model-providers/bedrock.md new file mode 100644 index 000000000..d970b7eed --- /dev/null +++ b/specs/model-providers/bedrock.md @@ -0,0 +1,23 @@ +# AWS Bedrock Provider Shape + +Last updated: dev@28d27ee | 2026-07-17 + +## Scope + +Canonical provider ID `bedrock`; UI/provider mapping currently recognizes AWS +Bedrock, but the canonical layer has no native Bedrock runtime reader. + +## Shape + +Bedrock is not generally an OpenAI-compatible host: model IDs, inference +profiles, request/response unions, signing, and per-family payloads differ. +Only an explicitly configured OpenAI/Anthropic-compatible gateway may use those +dialects. Native Bedrock capability must come from a versioned Bedrock model +catalog plus exact foundation-model/inference-profile identity. + +## Fallback And Current Gaps + +Do not classify all `amazonaws.com` hosts as Bedrock; use explicit kind or a +future region-aware exact host/path shape. General fallback is safe only behind +an explicitly compatible gateway. Native signing, catalogs, and family payload +mappings remain unimplemented. diff --git a/specs/model-providers/cerebras.md b/specs/model-providers/cerebras.md new file mode 100644 index 000000000..eba288ca6 --- /dev/null +++ b/specs/model-providers/cerebras.md @@ -0,0 +1,23 @@ +# Cerebras Provider Shape + +Last updated: dev@28d27ee | 2026-07-17 + +## Scope + +Canonical provider ID `cerebras`; OpenAI-compatible cloud transport; runtime +provider detection and cache-affinity safeguards in `src/llm_core.py`. + +## Shape And Observations + +Model lists use the general identity-only inventory reader. Cerebras rejects +llama.cpp-only `session_id` and `cache_prompt` fields (#4640), so cloud identity +must suppress local slot-affinity extensions. Current regressions pin this +provider boundary. + +Tool, reasoning, structured output, and limits remain per model. Do not promote +them from the fact that the API accepts OpenAI Chat. + +## Fallback And Current Gaps + +Exact `*.cerebras.ai` selects provider identity. Compatible proxies require +explicit configuration. No rich per-model Cerebras catalog reader is present. diff --git a/specs/model-providers/chatgpt-subscription.md b/specs/model-providers/chatgpt-subscription.md new file mode 100644 index 000000000..a961c1714 --- /dev/null +++ b/specs/model-providers/chatgpt-subscription.md @@ -0,0 +1,47 @@ +# ChatGPT Subscription Provider Shape + +Last updated: dev@e57f60b | 2026-07-20 + +## Scope + +Canonical provider ID `chatgpt_subscription`; Codex Responses transport; +auth and runtime code in `src/chatgpt_subscription.py`, +`routes/chatgpt_subscription_routes.py`, and `src/llm_core.py`. +There is no dedicated ChatGPT Subscription canonical reader on current `dev`. + +## Catalog Shape + +The account-scoped Codex models endpoint returns root `models[]`; `slug` is the +request identity and `visibility`/`priority` control availability/order. These +fields do not prove tools, reasoning, vision, or context. Null/malformed model +lists fail soft rather than crashing discovery (#5280/#5281). + +The canonical generic reader does not accept `slug`-only items, so this runtime +catalog is not currently normalized into `ModelCapabilityRecord` values. + +## Request And Response Shape + +Transport uses a ChatGPT backend Responses endpoint, `input` items, flattened +function tools, streamed function-call argument events, exact `call_id`, and +`function_call_output` continuation. Parallel calls and encrypted reasoning +continuity require preserving typed output/history rather than coercing all +roles to text. This shape is supported by the existing adapter and the focused +tool-calling follow-up evidence in #5490; unmerged observations remain claimed +until integrated/reproduced. + +OAuth/device credentials and refresh are provider-session behavior. Expired +credentials should return an actionable reconnect error, not generic model +failure. + +## Fallback And Safety + +Only the explicit internal base/ChatGPT host selects this provider. Never send +subscription credentials to a custom OpenAI-compatible URL. Catalog slugs stay +identity-only unless account-scoped fields or probes supply capability. + +## Current Gaps + +- Comprehensive Responses tool/reasoning parity is still evolving. +- Account model slugs are not consumed by the canonical reader package. +- The account catalog does not currently provide a complete canonical + capability card for every slug. diff --git a/specs/model-providers/cloudflare-workers-ai.md b/specs/model-providers/cloudflare-workers-ai.md new file mode 100644 index 000000000..480501299 --- /dev/null +++ b/specs/model-providers/cloudflare-workers-ai.md @@ -0,0 +1,21 @@ +# Cloudflare Workers AI Provider Shape + +Last updated: dev@28d27ee | 2026-07-17 + +## Scope + +Canonical provider ID `cloudflare_workers_ai`; OpenAI-compatible Workers AI +endpoint observations in #5175; explicit provider configuration required. + +## Shape + +Cloudflare account/path identity is part of the endpoint. Use the general +OpenAI-compatible inventory reader for returned model cards, preserving full +model IDs but no capability fields. +Do not identify the provider from broad `api.cloudflare.com` alone or infer +capability from Workers AI catalog prose. + +## Fallback And Current Gaps + +Provider identity must be explicit until a narrow account/AI path matcher is +implemented. There is no rich normalized capability catalog reader. diff --git a/specs/model-providers/cohere.md b/specs/model-providers/cohere.md new file mode 100644 index 000000000..f1ad23296 --- /dev/null +++ b/specs/model-providers/cohere.md @@ -0,0 +1,56 @@ +# Cohere Provider Shape + +Last updated: dev@e57f60b | 2026-07-20 + +## Scope + +Documented provider identity `cohere`; native Chat v2 plus the OpenAI +Compatibility API. Current `dev` has no dedicated Cohere capability reader or +direct Cohere request adapter; compatible endpoints use the general runtime +path when explicitly configured. + +## Catalog Shape + +`GET /v1/models` returns a paginated `models[]` envelope. Each model can carry +`name`, `endpoints`, `default_endpoints`, `context_length`, `features`, and +`sampling_defaults`; the root can carry `next_page_token`. + +These are candidate fields for a future dedicated reader: + +- a single canonical family from `endpoints`: `chat`/`generate`, `embed`, + `rerank`, or `classify`; +- `context_length` to the endpoint/model context limit; +- known sampling-default keys to deterministic controls. + +Current canonical normalization does not map them. When the generic reader is +explicitly selected with vendor `cohere`, it preserves only item identity plus +the raw item; family, context, features, and sampling controls stay unknown. + +## Request And Response Shape + +Native `POST /v2/chat` uses `messages`, structured content blocks, tools, +`response_format`, sampling fields, and an optional structured `thinking` +object. Text lives in `message.content[type=text].text`; reasoning-capable +models use `message.content[type=thinking].thinking`. Streaming uses typed +events rather than one generic text delta. + +The OpenAI compatibility base is `/compatibility/v1`. Its current chat subset +includes tools, structured output, sampling, and `reasoning_effort`, but model +support remains per-model. In the compatibility dialect only `none` and `high` +currently map to native thinking off/on; do not assume low/medium support. + +## Fallback And Safety + +No Cohere host or payload-shape detection exists in the canonical reader +registry. The caller must supply provider/endpoint configuration. Marketing +pages and provider-wide endpoint features do not grant every listed model +tools, vision, or reasoning. + +## Evidence And Gaps + +- Official List/Get Models resources define the catalog fields. +- Official Chat v2, Reasoning, and Compatibility API resources define the + transport and thinking controls. +- Odysseus has no direct Cohere request adapter, canonical reader, or sanitized + canonical fixtures yet; both normalization and runtime integration remain + follow-up work. diff --git a/specs/model-providers/deepseek.md b/specs/model-providers/deepseek.md new file mode 100644 index 000000000..dd0d131d9 --- /dev/null +++ b/specs/model-providers/deepseek.md @@ -0,0 +1,28 @@ +# DeepSeek Provider Shape + +Last updated: dev@28d27ee | 2026-07-17 + +## Scope + +Canonical provider ID `deepseek`; official cloud OpenAI-compatible API; +curation/detection in `routes/model_routes.py` and runtime reasoning handling in +`src/llm_core.py`. + +## Shape And Observations + +Use the general model-list inventory shape; capability-looking fields remain +unknown until a DeepSeek-native reader maps them. Cloud response history can use +`reasoning_content`; preserve it structurally for reasoning turns and tool +continuation (#968, #3152). `deepseek-chat`, reasoning models, distilled local +variants, and future V4 models do not share one capability record. + +Cloud endpoint evidence can support tools while a local DeepSeek-R1 deployment +may not have a working tool parser. Existing tool-support tests intentionally +separate official host from local engine/model-name heuristics. + +## Fallback And Current Gaps + +Exact `*.deepseek.com` selects provider identity; self-hosted checkpoints use +Ollama/vLLM/SGLang/llama.cpp identity. Curated model IDs and pricing/context +tables are compatibility data, not authoritative capability. A rich official +model-card reader is still absent. diff --git a/specs/model-providers/fireworks.md b/specs/model-providers/fireworks.md new file mode 100644 index 000000000..ccb0a5bc1 --- /dev/null +++ b/specs/model-providers/fireworks.md @@ -0,0 +1,22 @@ +# Fireworks AI Provider Shape + +Last updated: dev@28d27ee | 2026-07-17 + +## Scope + +Canonical provider ID `fireworks`; OpenAI-compatible cloud transport with path +prefixes such as `/inference/v1`; curation and URL handling in +`routes/model_routes.py` and `src/endpoint_resolver.py`. + +## Shape + +Use the general identity-only inventory reader. Fireworks IDs can contain +account/model paths; preserve the full request ID and endpoint scope. Item +modalities, supported parameters, task/type, and limits require a +Fireworks-native mapped shape before promotion. + +## Fallback And Current Gaps + +Exact `*.fireworks.ai` preserves provider identity and its configured path +prefix. Do not normalize account-qualified IDs by taking the last path segment. +No verified rich Fireworks capability catalog is currently mapped. diff --git a/specs/model-providers/github-copilot.md b/specs/model-providers/github-copilot.md new file mode 100644 index 000000000..4ab4e37f9 --- /dev/null +++ b/specs/model-providers/github-copilot.md @@ -0,0 +1,46 @@ +# GitHub Copilot Provider Shape + +Last updated: dev@e57f60b | 2026-07-20 + +## Scope + +Canonical provider ID `copilot`; OpenAI-compatible chat with Copilot headers +and OAuth; runtime adapter `src/copilot.py` and routes in +`routes/copilot_routes.py`. There is no dedicated Copilot canonical reader on +current `dev`. + +## Catalog Shape + +The observed Copilot `/models` response uses `data[]` entries with: + +- `id`; +- `model_picker_enabled`; +- `capabilities.supports.tool_calls` and `.vision`; +- optional limit/family metadata. + +Runtime model discovery uses picker state for availability. The canonical +reader package does not map the nested support fields; an explicitly supplied +`copilot` vendor currently uses generic identity-only normalization, and +`model_picker_enabled` does not become canonical capability. + +## Request And Response Shape + +Chat is OpenAI-compatible but requires Copilot/GitHub API version, editor/plugin +identity, intent, integration, and initiator headers; image requests add the +vision request flag. Header derivation must tolerate malformed message entries. +OAuth token exchange and access policies are provider authentication, not model +capability. + +## Fallback And Safety + +Use exact GitHub Copilot host or explicit kind, including the constrained +enterprise `copilot-api.*.ghe.com` form. Do not treat arbitrary `ghe.com` hosts +as Copilot. Official model availability tables are useful registry context but +do not replace the account-scoped catalog response. + +## Current Gaps + +- The catalog shape is implementation-observed and needs ongoing fixture + comparison with current Copilot clients. +- Copilot catalog capability fields are not normalized by current `dev`. +- Account/plan/policy availability must remain endpoint-user scoped. diff --git a/specs/model-providers/github-models.md b/specs/model-providers/github-models.md new file mode 100644 index 000000000..d6a26e66d --- /dev/null +++ b/specs/model-providers/github-models.md @@ -0,0 +1,21 @@ +# GitHub Models Provider Shape + +Last updated: dev@28d27ee | 2026-07-17 + +## Scope + +Canonical provider ID `github_models`; OpenAI-compatible GitHub Models/Azure +inference endpoint observed in #2995; distinct from GitHub Copilot. + +## Shape + +Use general identity-only inventory. Deployment IDs and account access +can differ from upstream model IDs. Do not copy Copilot picker metadata, +headers, plan rules, or capabilities into GitHub Models; they are separate +providers despite shared GitHub branding. + +## Fallback And Current Gaps + +The known `models.inference.ai.azure.com` host selects GitHub Models. Other +Azure deployment hosts require explicit provider configuration. No rich +account-scoped capability catalog is currently mapped. diff --git a/specs/model-providers/google.md b/specs/model-providers/google.md new file mode 100644 index 000000000..a465c540e --- /dev/null +++ b/specs/model-providers/google.md @@ -0,0 +1,54 @@ +# Google Gemini Provider Shape + +Last updated: dev@e57f60b | 2026-07-20 + +## Scope + +Canonical provider ID `google`; native GenerateContent plus optional Google +OpenAI-compatible chat; readers `google.py` and +`google_ai_studio_mapping.py`; catalog/probe ownership in +`routes/model_routes.py`. + +## Catalog Shape + +Use the native paginated `GET /v1beta/models` endpoint, including +`nextPageToken`, with `x-goog-api-key` when configured. `models[]` can contain: + +- `name`, `baseModelId`, `version`, and `displayName`; +- `inputTokenLimit` and `outputTokenLimit`; +- `supportedGenerationMethods`; +- `thinking`, `temperature`, `maxTemperature`, `topP`, and `topK`. + +Embedding-only methods map to embedding. Generation methods prove a native +method, not chat/image/video/audio modality, so those records remain unknown +unless stronger structured evidence exists. `thinking: true` and explicit +sampling fields map to a reasoning claim and controls. Model IDs such as +Imagen, Veo, or TTS names are not parsed. + +## Request And Response Shape + +Native generation uses `contents`, `systemInstruction`, +`generationConfig`, `tools[].functionDeclarations`, and +`models/{model}:generateContent|streamGenerateContent`. Responses use +`candidates[].content.parts[]` for `text`, `functionCall`, `functionResponse`, +`thought`, and `thoughtSignature`; token accounting is in `usageMetadata`. +Native Google tool/thought continuity must not be flattened through an +OpenAI-only history shape. + +## Fallback And Safety + +Prefer native model metadata even when chat is configured through Google's +OpenAI compatibility URL. Pagination parameters must remain stable between +pages. The route probe activates only for the exact +`generativelanguage.googleapis.com` hostname, filters the picker list to +content-generation methods, returns no curated fallback after probe failure, +and defaults those endpoints to manual catalog refresh unless explicitly +overridden. The canonical Google reader is not yet called by that probe. +Unknown methods and fields stay raw; unrecognized prediction models remain +unknown. + +## Current Gaps + +- The Models resource does not expose full modalities for every Google media + family. +- Native Gemini request/response support is not yet the only runtime path. diff --git a/specs/model-providers/groq.md b/specs/model-providers/groq.md new file mode 100644 index 000000000..2dfddcab3 --- /dev/null +++ b/specs/model-providers/groq.md @@ -0,0 +1,24 @@ +# Groq Provider Shape + +Last updated: dev@28d27ee | 2026-07-17 + +## Scope + +Canonical provider ID `groq`; OpenAI-compatible cloud transport; detection and +request behavior in `src/llm_core.py`. + +## Shape + +Model discovery falls back to the general `data[].id` identity shape. Richer +fields require a Groq-native mapped shape even when the payload happens to +supply modalities, supported parameters, or limits. Groq transport may accept OpenAI-style tools and streaming extensions, +but support remains per model and account. + +Runtime currently exempts Groq/OpenRouter from some parameter stripping paths; +that is transport compatibility, not a provider-wide model capability claim. + +## Fallback And Current Gaps + +Exact `*.groq.com` preserves Groq identity. Do not infer Llama/Gemma model +capabilities from IDs. There is no canonical rich Groq model-card reader or +freshness policy yet. diff --git a/specs/model-providers/hugging-face.md b/specs/model-providers/hugging-face.md new file mode 100644 index 000000000..7fa74c8f3 --- /dev/null +++ b/specs/model-providers/hugging-face.md @@ -0,0 +1,41 @@ +# Hugging Face Provider And Registry Shape + +Last updated: dev@e57f60b | 2026-07-20 + +## Scope + +Canonical placeholder vendor ID `huggingface`; download/fit metadata in +`services/hwfit/`; OpenAI-compatible inference providers/TGI handled as their +serving dialect. There is no dedicated Hugging Face canonical reader on +current `dev`. + +## Hub Model Shape + +Hub model info can provide `modelId`/`id`, `pipeline_tag`, `tags`, `config`, and +card metadata. Current canonical normalization does not map `pipeline_tag`, +`config.model_type`, or Hub task/modality fields. An explicitly selected +Hugging Face vendor uses generic identity-only normalization. + +This source is `cookbook_hf`/registry confidence, not live endpoint truth. +Free-form tags, README/card prose, repository names, and architecture names do +not automatically claim capability. A serving engine can load a model with +missing projection, different template, or disabled parser. + +## Serving Shape + +Hugging Face routed inference and TGI can expose OpenAI-compatible endpoints; +their model list may be identity-only. Keep Hub identity separate from the +serving endpoint and merge only when exact revision/model identity is known. + +## Fallback And Safety + +Hub metadata can fill a scoped registry record after provider payload fields +and probes, but must not overwrite fresh endpoint-negative evidence. Treat +remote code, model cards, and repository files as untrusted content. + +## Current Gaps + +- Revision/digest linkage between downloads, Hub records, and serving + endpoints is incomplete. +- Hub task/family metadata is not consumed by the canonical reader package. +- Pipeline tags can be missing or overly broad; unknown stays unknown. diff --git a/specs/model-providers/llama-cpp.md b/specs/model-providers/llama-cpp.md new file mode 100644 index 000000000..6073c5bbb --- /dev/null +++ b/specs/model-providers/llama-cpp.md @@ -0,0 +1,47 @@ +# llama.cpp Provider Shape + +Last updated: dev@e57f60b | 2026-07-20 + +## Scope + +Canonical provider ID `llamacpp`; OpenAI Chat/Responses and Anthropic Messages +compatibility plus native server metadata; reader +`src/model_capability_readers/llamacpp.py`. + +## Metadata Shapes + +`/v1/models` provides served identity and can include server model entries; +native `/props` is authoritative for the running model/server combination: + +- `model_alias`/`model_path`; +- `default_generation_settings.n_ctx` and sampling `params`; +- `total_slots` and optional `/slots[].n_ctx` fallback; +- `chat_template_caps` for tools/system role; +- `modalities.vision|audio`; +- current server/build state. + +Capability depends on weights, projection/model assets, chat template, parser, +and launch flags. It is endpoint evidence, not a checkpoint-name claim. +`/props` and `/v1/models` can be merged only for the same served identity. + +## Request And Response Shape + +llama-server supports several OpenAI-compatible tasks and native extensions. +Do not infer embeddings/rerank/chat solely from the OpenAI model card; use an +explicit server model capability field or endpoint configuration. Tool and +reasoning correctness can depend on selected chat template and parser. + +## Fallback And Safety + +The registry selects llama.cpp through an explicit vendor or endpoint kind; it +does not auto-detect `/props` from payload shape. Port 8000 currently maps to +the vLLM placeholder, while 8080 falls through to generic OpenAI-compatible. +llama.cpp-only `session_id` and `cache_prompt` affinity fields must remain local +endpoint behavior and never leak to strict cloud providers (#4640 and current +affinity tests). + +## Current Gaps + +- Multi-model routing requires per-served-model `/props` association. +- Parser/template configuration is not yet fully represented in canonical + endpoint metadata. diff --git a/specs/model-providers/lm-studio.md b/specs/model-providers/lm-studio.md new file mode 100644 index 000000000..0046ad6a3 --- /dev/null +++ b/specs/model-providers/lm-studio.md @@ -0,0 +1,45 @@ +# LM Studio Provider Shape + +Last updated: dev@e57f60b | 2026-07-20 + +## Scope + +Canonical provider ID `lmstudio`; native LM Studio v1 plus OpenAI Chat and +Responses compatibility; reader `src/model_capability_readers/lmstudio.py`. + +## Catalog Shapes + +Preferred shape is `GET /api/v1/models` with root `models[]`. Current fields +include `key`, `type` (`llm` or `embedding`), display/publisher data, +`architecture`, quantization/format/size, `max_context_length`, +`loaded_instances[].config.context_length`, and a capability object containing +`vision`, `trained_for_tool_use`, and reasoning options/defaults. + +Compatibility shape `GET /api/v0/models` uses `data[]` with `id`, `type` +(`llm`, `vlm`, or embeddings), `arch`, `compatibility_type`, state, and +context metadata. It is an explicit older shape, not a loose fallback. +OpenAI `/v1/models` is identity-only when native endpoints are unavailable. + +Loaded-instance context is the effective runtime context; maximum context is a +separate limit. Model type maps family, explicit capability booleans map +vision/tools/reasoning, and architecture is provider-reported model family. + +## Request And Response Shape + +Native v1 chat is `/api/v1/chat` and can expose stateful/MCP-oriented output; +LM Studio also supports OpenAI Chat and Responses compatibility. Keep dialect +selection explicit because tool/MCP features differ between native and +compatible paths. + +## Fallback And Safety + +Current reader detection identifies port 1234 as LM Studio. Prefer pathless +native `/api/v1/models` discovery where configured (#1122, #3615), then v0, +then general identity. The port mapping is a normalization hint, not endpoint +trust. An error object from an unsupported native route is not a model list. + +## Current Gaps + +- Runtime discovery does not yet persist native capability records. +- LM Studio API capabilities continue to evolve; each new native version needs + an explicit shape fixture before promotion. diff --git a/specs/model-providers/local-compatible-engines.md b/specs/model-providers/local-compatible-engines.md new file mode 100644 index 000000000..a0f1034e9 --- /dev/null +++ b/specs/model-providers/local-compatible-engines.md @@ -0,0 +1,37 @@ +# Other Local And Proxy Compatible Engines + +Last updated: dev@28d27ee | 2026-07-17 + +## Scope + +Canonical explicit identities `mlx_lm`, `text_generation_inference`, +`lmdeploy`, and `litellm`, plus unknown OpenAI-compatible deployments not +covered by the native Ollama, LM Studio, llama.cpp, vLLM, or SGLang specs. + +## Shape + +Use explicit endpoint kind when known; otherwise use only the general model +list envelopes for inventory identity. Capability-looking structural fields +remain raw. Local host and port do not distinguish these engines. +MLX/Cookbook launch recipes, TGI task configuration, LMDeploy +adapters, and LiteLLM upstream routing can all change capability independently +of the model ID. + +Proxy model aliases are endpoint scoped. A proxy may return richer fields, but +unknown keys remain raw until a versioned shape is added. Provider-specific +headers/extensions must not be applied based on a port or upstream model name. + +## Fallback And Safety + +Discovery can probe cheap native identity endpoints when available, but +capability probes execute only explicit bounded test contracts. Never read +broad server/environment dumps as ordinary model metadata. Unknown compatible +servers should still list identities and make conservative text calls where +explicitly configured, without appearing on capability-gated surfaces. + +## Current Gaps + +- These engines need individual safe metadata fixtures before they can graduate + from general fallback. +- Gateway upstream identity and effective downstream model capability are not + yet represented as a chain. diff --git a/specs/model-providers/minimax.md b/specs/model-providers/minimax.md new file mode 100644 index 000000000..d54a67457 --- /dev/null +++ b/specs/model-providers/minimax.md @@ -0,0 +1,48 @@ +# MiniMax Provider Shape + +Last updated: dev@28d27ee | 2026-07-17 + +## Scope + +Canonical provider ID `minimax`; international host `api.minimax.io`, China +host `api.minimaxi.com`; current OpenAI-compatible and recommended +Anthropic-compatible text transports. Odysseus contains MiniMax-oriented tool +output handling and local-serving guidance but no dedicated catalog reader. + +## Catalog Shape + +Current `GET /v1/models` is an OpenAI-compatible identity list: +`object: list`, `data[]`, and model cards containing `id`, `object: model`, +`created`, and `owned_by: minimax`. The `owned_by` discriminator identifies the +provider shape, but the card exposes no per-model capability or modality +fields. Keep these records unknown and preserve raw identity metadata. + +Do not backfill current model capabilities, token limits, or modalities from +the platform overview into this list response. Those tables are useful scoped +registry evidence only after model/version identity and freshness are carried +explicitly. + +## Request And Response Shape + +- OpenAI compatibility uses `/v1/chat/completions` and structured + `reasoning_content` alongside normal message content. +- Anthropic compatibility uses `/anthropic/v1/messages`; the current M2.7 + family supports typed thinking blocks and interleaved thinking, making this + the preferred reasoning/tool-continuation transport in provider guidance. +- Native audio, image, video, music, and file endpoints are separate product + shapes. They must not be inferred from presence in the text model list. + +## Local Deployments + +The current provider guide documents vLLM, SGLang, and MLX deployment. Those +instances retain serving-engine identity and configuration-derived capability; +the checkpoint name alone does not turn a vLLM/SGLang card into the hosted +MiniMax provider shape. + +## Fallback And Current Gaps + +Exact MiniMax hosts or the discriminating `owned_by: minimax` model-list shape +select provider identity. Unknown compatible proxies retain the general shape. +The identity list does not safely distinguish M2 reasoning behavior from +speech/image/video/music products, so exact model quirks remain documentation +until structured model-version evidence reaches runtime request builders. diff --git a/specs/model-providers/mistral.md b/specs/model-providers/mistral.md new file mode 100644 index 000000000..b48568008 --- /dev/null +++ b/specs/model-providers/mistral.md @@ -0,0 +1,49 @@ +# Mistral Provider Shape + +Last updated: dev@e57f60b | 2026-07-20 + +## Scope + +Canonical provider ID `mistral`; OpenAI-compatible chat with Mistral response +extensions and runtime handling in `src/llm_core.py`. There is no dedicated +Mistral canonical reader on current `dev`. + +## Catalog Shape + +`GET /v1/models` returns `data[]` cards with `id`, `root`, aliases, +`max_context_length`, and `capabilities` booleans including +`completion_chat`, `completion_fim`, `function_calling`, `vision`, +`classification`, and lifecycle/fine-tuning fields. These are candidate fields +for a future dedicated reader: + +- chat/FIM or classification family; +- vision input; +- function calling; +- explicitly reported reasoning/structured output when present; +- context limit and root family. + +Fine-tuning availability and archived status are not inference capabilities. +The current generic reader retains identity/raw data only and does not map any +of these fields. Different Mistral models retain independent identities. + +## Request And Response Shape + +Reasoning-capable models accept graded `reasoning_effort`. Mistral can return +`content` as typed blocks: a `thinking` block containing text fragments plus a +normal `text` block. Normalize these structured blocks into separate reasoning +and visible channels; do not stringify the list or scan text tags (#4698). + +## Fallback And Safety + +Runtime `llm_core` detects label-bounded Mistral hosts for request/response +handling. The canonical registry has no Mistral host or rich-payload detector; +an explicitly supplied `mistral` vendor falls back to generic identity. A +Mistral model served through another engine uses that serving engine's dialect. + +## Current Gaps + +- Catalog reasoning fields vary across model-card generations; absent remains + unknown. +- Mistral catalog capability fields are not normalized by current `dev`. +- Runtime thinking-family selection still uses names and should migrate to + structured root/capability identity. diff --git a/specs/model-providers/moonshot-kimi.md b/specs/model-providers/moonshot-kimi.md new file mode 100644 index 000000000..35c5e7e42 --- /dev/null +++ b/specs/model-providers/moonshot-kimi.md @@ -0,0 +1,30 @@ +# Moonshot And Kimi Provider Shape + +Last updated: dev@28d27ee | 2026-07-17 + +## Scope + +Provider IDs `moonshot` for official Moonshot API and `kimi_code` for the Kimi +Code surface; OpenAI-compatible transport with provider-specific headers and +model-specific behavior in `src/llm_core.py`. + +## Shape And Observations + +Model lists use the general OpenAI-compatible identity shape unless a richer +account response is returned. Official Kimi K2.5/K2.6 fixes temperature by +thinking mode, so Odysseus omits `temperature` rather than sending an invalid +value (#3960). Thinking tool-call continuation requires preservation of +assistant `reasoning_content` (#3118). Kimi Code negotiates a small exact +User-Agent set on 403 and caches the accepted value; this is provider transport, +not model capability. + +Reports distinguish K2.5/K2.6 multimodality from older K2 variants (#2522). +Promote those claims only through exact structured model IDs/families, not a +`kimi` name match. + +## Fallback And Current Gaps + +Keep Moonshot and Kimi Code identities distinct even when both use OpenAI Chat. +Self-hosted Kimi checkpoints inherit their serving engine shape, not official +Moonshot sampling rules. The provider catalog does not yet yield a complete +canonical capability card. diff --git a/specs/model-providers/nvidia-nim.md b/specs/model-providers/nvidia-nim.md new file mode 100644 index 000000000..8f68d4f00 --- /dev/null +++ b/specs/model-providers/nvidia-nim.md @@ -0,0 +1,28 @@ +# NVIDIA NIM Provider Shape + +Last updated: dev@28d27ee | 2026-07-17 + +## Scope + +Canonical provider ID `nvidia`; OpenAI-compatible NVIDIA/NIM endpoints; current +provider detection, catalog routing, and reasoning stream handling in +`src/llm_core.py`, `routes/model_routes.py`, and tests. + +## Shape And Observations + +Model lists use the general identity-only shape; capability-looking fields +require a provider-native mapped shape. +NIM/vLLM-style responses have emitted structured `reasoning` while older paths +used `reasoning_content`; Odysseus routes either to the reasoning channel +(#602). This response compatibility does not claim that every NIM model +reasons. + +NVIDIA endpoints can host many unrelated model families with different tools, +vision, context, and parser support. Keep endpoint/model stable identity and +prefer provider fields or probes. + +## Fallback And Current Gaps + +Exact NVIDIA host preserves provider identity; private NIM installations need +explicit endpoint kind because a local port/hostname is not distinctive. No +safe normalized native NIM capability endpoint is currently consumed. diff --git a/specs/model-providers/ollama.md b/specs/model-providers/ollama.md new file mode 100644 index 000000000..607ffed55 --- /dev/null +++ b/specs/model-providers/ollama.md @@ -0,0 +1,54 @@ +# Ollama Provider Shape + +Last updated: dev@e57f60b | 2026-07-20 + +## Scope + +Canonical provider ID `ollama`; native Ollama chat/generate plus OpenAI +compatibility; reader `src/model_capability_readers/ollama.py`; discovery and +runtime code in `routes/model_routes.py` and `src/llm_core.py`. + +## Catalog And Detail Shapes + +Use two native steps: + +1. `GET /api/tags` returns `models[]` identity (`name`/`model`, digest, + `details.family|families`, format, parameter size, quantization). Tags do not + claim capabilities. +2. `POST /api/show` for a selected model returns explicit `capabilities[]`, + `details`, and `model_info`. Map completion/chat, embedding, vision, tools, + and thinking/reasoning tokens. Map context from exact `context_length` or + native `.context_length` fields. + +The reader does not parse model names or architecture names. It does parse a +two-column serialized `parameters` value and can take `num_ctx` from it before +falling back to exact or suffix `*.context_length` keys in structured mappings. +The parameters text is used only for that keyed limit lookup, not capability +inference. + +## Request And Response Shape + +Native chat uses `/api/chat`, `messages`, optional OpenAI-shaped tool +definitions, `format`, `options`, and model-dependent `think`. Responses use +`message.content`, `message.thinking`, and `message.tool_calls`. Generate uses +top-level `response` and `thinking`. OpenAI compatibility is a separate dialect +and can change control names independently. + +Thinking control is model-specific: most documented reasoning families accept +a native bool, while GPT-OSS accepts low/medium/high and cannot be fully +disabled. A reported Ollama 0.20.6 Qwen3.5 OpenAI-compat path requires +`reasoning_effort: none` rather than `think: false` (#5503); keep it versioned +and low-confidence until corroborated. + +## Fallback And Safety + +Current reader detection identifies port 11434 as Ollama, in addition to an +explicit endpoint kind or an `ollama.com` hostname suffix. This is a +normalization hint, not endpoint trust or capability evidence. Names that +contain `vision`, `embed`, or `qwen` are not capability evidence (#3743, +#4487). + +## Current Gaps + +- List discovery needs an orchestrated `/api/show` detail step per model. +- Runtime OpenAI-compat thinking suppression still contains name heuristics. diff --git a/specs/model-providers/openai-compatible.md b/specs/model-providers/openai-compatible.md new file mode 100644 index 000000000..b1763fb09 --- /dev/null +++ b/specs/model-providers/openai-compatible.md @@ -0,0 +1,55 @@ +# General OpenAI-Compatible Inventory Fallback + +Last updated: dev@e57f60b | 2026-07-20 + +## Scope + +Canonical compatibility identity `generic_openai`; identity-only reader +`src/model_capability_readers/generic_openai.py`; shared envelope and identity +helpers in `src/model_capability_readers/base.py`. + +This is not a universal OpenAI-compatible capability schema. Transport request +and response behavior remains in `src.llm_core` and provider adapters. + +## Accepted Inventory Shape + +- `{"data": [...]}`; +- `{"models": [...]}`. + +Within an item, the reader recovers identity from `id`, `name`, or `model`. +Bare-list payloads and `key`/`slug`-only items are not supported. It preserves +the raw item on the in-memory record, while `to_dict()` includes it only when +the caller explicitly requests `include_raw=True`. Capability remains unknown. + +## Disabled Capability Paths + +The generic reader does not inspect capability-looking fields, including: + +- `type`, `model_type`, `task`, and `pipeline_tag`; +- top-level or nested modality fields; +- capability booleans/maps/lists; +- `supported_parameters`; +- context, input, output, and model-length fields. + +Names, descriptions, ownership, pricing, and serialized text also never +promote capability through this reader. + +## Forward Compatibility + +An explicitly configured but unknown provider ID is preserved when the generic +reader is selected. That allows endpoint-scoped stable IDs to keep working +while every family, modality, capability, limit, and control remains unknown. +Non-object entries are skipped; null or malformed roots return no records. + +Provider-specific headers, request extensions, and reasoning channels must be +selected by explicit provider/endpoint adapters. They never leak through this +fallback. + +## Current Gaps + +- Compatible providers differ on path prefixes, null handling, tools, + streaming usage, and strict extra-field rejection. +- Bare-list and `key`/`slug`-only inventories need explicit normalization if a + runtime consumer later requires them. +- Safe request shaping still requires explicit endpoint/provider + configuration even when identity normalization succeeds. diff --git a/specs/model-providers/openai.md b/specs/model-providers/openai.md new file mode 100644 index 000000000..2cff3e35f --- /dev/null +++ b/specs/model-providers/openai.md @@ -0,0 +1,40 @@ +# OpenAI Provider Shape + +Last updated: dev@e57f60b | 2026-07-20 + +## Scope + +Canonical provider ID `openai`; API dialects OpenAI Chat Completions and +Responses; catalog reader `src/model_capability_readers/openai.py`. + +## Catalog Shape + +`GET /v1/models` returns `object: list` with `data[]` model cards containing +`id`, `object`, `created`, and `owned_by`. This is identity and availability +metadata only. It does not claim vision, tools, reasoning, modality, task, or +context length. The record remains unknown and keeps the raw fields. + +## Request And Response Shape + +Chat uses `messages`, `tools[].function`, `tool_choice`, and +`choices[].message|delta`; Responses uses `input`, flattened tools, output +items, and typed stream events. OpenAI may support a parameter at the platform +level while individual models differ. A later model registry or probe must +scope that fact before it becomes canonical model capability. + +## Fallback And Safety + +An explicit endpoint kind selects this provider. Automatic reader detection +currently uses a plain hostname `endswith("openai.com")` check; it is a +normalization hint rather than a trust boundary. Do not parse model IDs or +ownership labels. If a proxy returns richer fields while explicitly configured +as OpenAI, the reader preserves them as raw evidence but keeps capability +unknown. + +## Current Gaps + +- OpenAI's Models API does not publish the per-model capability shape needed + for automatic canonical classification. +- Runtime model-specific sampling/reasoning behavior still needs a maintained + structured registry or endpoint probes. +- Automatic reader host matching is not label-boundary checked. diff --git a/specs/model-providers/opencode.md b/specs/model-providers/opencode.md new file mode 100644 index 000000000..82ba100f3 --- /dev/null +++ b/specs/model-providers/opencode.md @@ -0,0 +1,23 @@ +# OpenCode Provider Shape + +Last updated: dev@28d27ee | 2026-07-17 + +## Scope + +Canonical provider identity `opencode` with Zen/Go endpoint variants; +OpenAI-compatible transport and webhook presets in `src/llm_core.py` and +`routes/webhook_routes.py`. + +## Shape + +Keep Zen and Go path identity in endpoint metadata even though the canonical +provider family is OpenCode. Model discovery uses general identity-only +fallback. Path/version, account policy, and model selection can differ between +variants; do not flatten them into OpenAI. + +## Fallback And Current Gaps + +Exact `*.opencode.ai` plus configured `/zen` or `/zen/go` selects this family. +No provider-specific rich capability catalog is mapped, and runtime still has +separate variant labels that should eventually become structured endpoint +metadata. diff --git a/specs/model-providers/openrouter.md b/specs/model-providers/openrouter.md new file mode 100644 index 000000000..87beabbf2 --- /dev/null +++ b/specs/model-providers/openrouter.md @@ -0,0 +1,44 @@ +# OpenRouter Provider Shape + +Last updated: dev@e57f60b | 2026-07-20 + +## Scope + +Canonical provider ID `openrouter`; OpenAI-compatible chat dialect; rich reader +`src/model_capability_readers/openrouter.py`. + +## Catalog Shape + +`GET /api/v1/models` returns `data[]`. Canonical fields are: + +- `id` (falling back to `name`) and display `name`; +- `architecture.input_modalities`, `architecture.output_modalities`, and + compatibility `architecture.modality`; +- `context_length` and `top_provider.max_completion_tokens`; +- `supported_parameters`, `default_parameters`, `supported_voices`, and + `per_request_limits`. + +Modalities determine family and vision/file/audio/image/video behavior. +Recognized supported parameters claim tools, JSON/structured output, +reasoning, and web search. Sampling/default parameters become controls, not +capabilities. Descriptions, pricing, author slugs, and tokenizer names do not. + +## Provider Versus Routed Endpoint + +OpenRouter normalizes requests while routing a model to one of several +underlying providers. The catalog model record is OpenRouter-scoped. Do not +copy a direct-provider quirk to OpenRouter unless its normalized API and exact +model/endpoint evidence require it. `top_provider` limits describe the current +route class, not a permanent global model limit. + +## Fallback And Safety + +The reader receives OpenRouter through explicit selection or the current +hostname suffix hint. Future fields remain raw. If modalities are absent, it +falls back to an identity-only OpenRouter record and does not parse the model +slug; supported-parameter controls are not retained on that fallback path. + +## Current Gaps + +- Per-upstream endpoint differences can still invalidate an aggregate claim. +- Catalog values change frequently and need freshness/expiry when persisted. diff --git a/specs/model-providers/perplexity.md b/specs/model-providers/perplexity.md new file mode 100644 index 000000000..0f0a616de --- /dev/null +++ b/specs/model-providers/perplexity.md @@ -0,0 +1,20 @@ +# Perplexity Provider Shape + +Last updated: dev@28d27ee | 2026-07-17 + +## Scope + +Canonical provider ID `perplexity`; OpenAI-compatible cloud endpoint recognized +by current UI/provider host maps and agent cloud-host safeguards (#3015). + +## Shape + +Use general identity-only inventory mapping. Perplexity products may perform +search, but `web_search` becomes a canonical model capability only when an +exact model card, maintained registry, or probe reports it. Provider identity +alone and product descriptions are insufficient. + +## Fallback And Current Gaps + +Exact `*.perplexity.ai` preserves provider identity. No rich per-model catalog +or search-control mapping is currently consumed. diff --git a/specs/model-providers/sglang.md b/specs/model-providers/sglang.md new file mode 100644 index 000000000..cc402a505 --- /dev/null +++ b/specs/model-providers/sglang.md @@ -0,0 +1,47 @@ +# SGLang Provider Shape + +Last updated: dev@e57f60b | 2026-07-20 + +## Scope + +Canonical provider ID `sglang`; OpenAI Chat/Responses plus native generation; +Cookbook launch behavior in `routes/cookbook_routes.py` and serving UI modules. +There is no dedicated SGLang canonical reader on current `dev`. + +## Metadata Shapes + +Preferred native `GET /model_info` (legacy `/get_model_info`) returns: + +- `model_path` and `tokenizer_path`; +- `is_generation`; +- `has_image_understanding` and `has_audio_understanding`; +- `model_type`, `architectures`, `weight_version`; +- `preferred_sampling_params`. + +These are provider observations for a future dedicated reader. Current generic +normalization does not map `is_generation`, modality booleans, sampling keys, +or `max_model_len`. + +`GET /v1/models` returns served IDs with `owned_by: sglang`, `root`, and +`max_model_len`; it supplies identity/context but not parser capability. + +## Runtime Capability + +Tools and reasoning depend on explicit `--tool-call-parser` and +`--reasoning-parser`; multimodality and context can also be launch-configured. +Cookbook recipes for Qwen, DeepSeek, GLM, Kimi, MiniMax, StepFun, and other +families are deployment observations, not universal model-name rules. Persist +the selected parser/config as endpoint evidence before canonical promotion. + +## Fallback And Safety + +Current reader detection identifies port 30000 as SGLang, or accepts an +explicit endpoint kind, then dispatches to the generic identity-only reader. +It does not infer SGLang from `/model_info` payload shape. Avoid normal +discovery through the broad admin `/server_info` dump. + +## Current Gaps + +- Endpoint records do not yet store parser/task configuration canonically. +- Non-generation task classification needs explicit serving metadata. +- No dedicated reader maps SGLang metadata today. diff --git a/specs/model-providers/siliconflow.md b/specs/model-providers/siliconflow.md new file mode 100644 index 000000000..d77983a28 --- /dev/null +++ b/specs/model-providers/siliconflow.md @@ -0,0 +1,21 @@ +# SiliconFlow Provider Shape + +Last updated: dev@28d27ee | 2026-07-17 + +## Scope + +Canonical provider ID `siliconflow`; global/CN OpenAI-compatible provider +proposed in #5562. + +## Shape + +Use the general `/v1/models` identity-only inventory reader for both regional +surfaces. Region/base URL and API key remain endpoint identity. A regional +provider-native schema is required before any item fields are promoted; model +tokens in returned IDs or PR examples are never capability evidence. + +## Fallback And Current Gaps + +Exact SiliconFlow hosts or explicit kind preserve provider identity. The open +provider work has no confirmed rich capability card; regional path/host details +and current payload fixtures need revalidation before runtime integration. diff --git a/specs/model-providers/together.md b/specs/model-providers/together.md new file mode 100644 index 000000000..53b82f14c --- /dev/null +++ b/specs/model-providers/together.md @@ -0,0 +1,27 @@ +# Together AI Provider Shape + +Last updated: dev@e57f60b | 2026-07-20 + +## Scope + +Canonical provider ID `together`; OpenAI-compatible cloud transport; curated +models and discovery compatibility in `routes/model_routes.py`. + +## Shape And Observations + +Together has returned both standard `data[]` and bare model-card lists. The +current generic reader accepts the standard envelope when the caller supplies +the Together vendor, but it does not accept a bare root list. It keeps +identity/provider scope and promotes no capability fields. Task, modality, +parameter, and limit data needs a dedicated Together reader before it becomes +canonical; model names and the curated picker list are not capability evidence. + +Together can serve many upstream families. Direct-provider quirks do not +automatically apply because Together may normalize requests and responses. + +## Fallback And Current Gaps + +Both `*.together.xyz` and `*.together.ai` identify the provider. Malformed/null +lists fail soft. A provider-specific rich capability schema has not been +confirmed, so general fallback remains intentional. Bare-list catalogs require +route-specific preprocessing or a future reader update. diff --git a/specs/model-providers/venice.md b/specs/model-providers/venice.md new file mode 100644 index 000000000..8972f6db0 --- /dev/null +++ b/specs/model-providers/venice.md @@ -0,0 +1,19 @@ +# Venice Provider Shape + +Last updated: dev@28d27ee | 2026-07-17 + +## Scope + +Canonical provider ID `venice`; paid OpenAI-compatible cloud API represented in +webhook presets and cloud/self-hosted classification tests. + +## Shape + +Use general identity-only inventory mapping. Treat `api.venice.ai` as a remote API +for routing/security, while keeping model capability per returned model. Do not +infer privacy, tools, reasoning, or context from provider marketing or names. + +## Fallback And Current Gaps + +Exact `*.venice.ai` preserves provider identity. No verified rich model-card +schema is currently mapped. diff --git a/specs/model-providers/vllm.md b/specs/model-providers/vllm.md new file mode 100644 index 000000000..209582fe0 --- /dev/null +++ b/specs/model-providers/vllm.md @@ -0,0 +1,44 @@ +# vLLM Provider Shape + +Last updated: dev@e57f60b | 2026-07-20 + +## Scope + +Canonical placeholder provider ID `vllm`; OpenAI Chat and Responses serving; +generic identity-only inventory normalization. There is no dedicated vLLM +reader or model-card detector on current `dev`. + +## Catalog Shape + +Current `GET /v1/models` returns `object: list`, `data[]` model cards with +`id`, `object`, `owned_by: vllm`, `root`, `parent`, `max_model_len`, and +`permission[]`. The generic reader retains only identity/raw data and does not +inspect `owned_by`, `root`, `parent`, `max_model_len`, or `permission`. The card +does not prove chat template, tools, +reasoning parser, vision assets, embeddings, transcription, or rerank. + +LoRA cards can use a different `id`, root path, and parent. Keep each served ID +endpoint scoped and do not merge it globally with the base checkpoint. + +## Runtime Capability + +vLLM's supported API surface is broad, but actual behavior depends on the +loaded model task, chat template, multimodal assets, tool-call parser, +reasoning parser, structured-output configuration, and launch flags. Current +Odysseus reasoning regressions cover structured `reasoning`, legacy +`reasoning_content`, and compatible fields (#602). These response channels are +transport evidence, not a claim that every vLLM model reasons. + +## Fallback And Safety + +Current reader detection identifies port 8000 as vLLM, or accepts an explicit +endpoint kind, then dispatches to the generic identity-only reader. It does not +infer vLLM from the model-card payload. Do not consume `/server_info` +environment/config dumps for normal discovery because they can be large and +operationally sensitive. + +## Current Gaps + +- A small safe native capability endpoint is not part of the canonical probe. +- Deployment parser/template flags are not persisted with endpoint capability. +- No dedicated reader maps vLLM model-card fields today. diff --git a/specs/model-providers/xai.md b/specs/model-providers/xai.md new file mode 100644 index 000000000..c46e49a4d --- /dev/null +++ b/specs/model-providers/xai.md @@ -0,0 +1,21 @@ +# xAI Provider Shape + +Last updated: dev@28d27ee | 2026-07-17 + +## Scope + +Canonical provider ID `xai`; OpenAI-compatible xAI cloud transport; provider +labels/curation in `src/llm_core.py` and `routes/model_routes.py`. + +## Shape + +Model discovery uses general identity-only inventory. Reasoning effort, tools, +image input, or other Grok behavior must be +scoped per returned model/registry/probe. The provider's broad API feature set +does not grant every listed model every capability. + +## Fallback And Current Gaps + +Exact `*.x.ai` selects xAI. Preserve provider identity through OpenAI-compatible +fallback and reject lookalikes. A current rich model catalog schema and +structured version registry are not yet mapped. diff --git a/specs/model-providers/zai.md b/specs/model-providers/zai.md new file mode 100644 index 000000000..8f07f52c1 --- /dev/null +++ b/specs/model-providers/zai.md @@ -0,0 +1,23 @@ +# Z.AI Provider Shape + +Last updated: dev@28d27ee | 2026-07-17 + +## Scope + +Canonical provider ID `zai`; Z.AI/GLM OpenAI-compatible endpoints including +coding-plan variants; curated discovery in `routes/model_routes.py` and prior +vision/reasoning fixes such as #664. + +## Shape And Observations + +Use general identity-only inventory mapping. Some working coding-plan models may be +absent from `/models`, so pinned/curated IDs are availability compatibility, +not capability truth. GLM reasoning controls have appeared as structured +objects or serving-template kwargs depending on direct cloud versus local +engine (#3031). Keep those scopes separate. + +## Fallback And Current Gaps + +Exact `*.z.ai` or explicit endpoint kind preserves Z.AI identity. Never infer +vision/reasoning/tool support from `glm` in a name. A rich official model-card +reader and direct-versus-coding-plan schema split are still missing. diff --git a/specs/model-quirks.md b/specs/model-quirks.md new file mode 100644 index 000000000..89b456b86 --- /dev/null +++ b/specs/model-quirks.md @@ -0,0 +1,88 @@ +# Model Behavior Observations + +Last updated: dev@e57f60b | 2026-07-20 + +## Scope + +This file records model- or provider+model-specific behavior observed in +Odysseus code, tests, Issues, PRs, commits, and provider documentation. It is a +compact evidence map, not a runtime matcher. General canonical rules belong in +[model-capability-canonical.md](model-capability-canonical.md); provider-wide +transport belongs in [the provider map](model-providers/_readme.md). + +The canonical capability layer intentionally has no +`src/model_behavior_quirks.py`. +Adding a registry before runtime call sites carry structured provider, model, +version, and dialect identity would create another model-name matching layer. + +## General Observation Template + +Record only the fields supported by the evidence: + +- provider and endpoint/dialect scope; +- exact provider-returned model ID or family; +- structured model/provider version when available; +- capability or request/response behavior observed; +- exact native request field/value and response field when relevant; +- source, confidence, status, and reproduction date; +- whether the behavior is already implemented in runtime code. + +If exact structured identity is unavailable, keep the observation here and in +its current tested runtime location. Do not promote it through substring, +regex, prose, or serialized-prompt parsing in the canonical layer. + +## Model-Specific Observation Map + +| Observation | Scope | Behavior | Evidence/status | +| --- | --- | --- | --- | +| Moonshot Kimi K2.5/K2.6 fixed temperature | official Moonshot, K2.5/K2.6, OpenAI Chat | omit `temperature`; thinking mode owns its fixed value | #3960, `f5d3e509`; implemented in current runtime | +| Moonshot reasoning tool history | same provider/models/dialect | preserve assistant `reasoning_content` across tool continuation | #3118, `2e6fff22`; implemented | +| Claude Opus 4.7+ sampling omission | Anthropic Messages, Opus 4.7+ | omit `temperature`, `top_p`, and `top_k` | #3117, `4f48cfa9`; implemented through current runtime identity logic | +| Mistral structured reasoning | reasoning-capable Mistral model through native/compatible response shape | use graded effort where accepted; keep typed thinking separate from text | #4698, `bd9149f7`, provider docs; partly implemented | +| Ollama native reasoning control | selected reasoning model/deployment | native `think`; reasoning in `message.thinking`/`thinking` | #3031 and provider docs; deployment scoped | +| Ollama native `gpt-oss` reasoning level | `gpt-oss` served through Ollama native | `think` accepts low/medium/high and does not represent off | provider docs; deployment scoped | +| Ollama compatibility disable observation | Ollama 0.20.6+, observed Qwen3.5 compatibility path | `reasoning_effort: none` was reported to disable reasoning | #5503; unmerged/low confidence until reproduced | + +Issue and commit references are evidence identifiers, not runtime dependencies. +Open or unmerged observations remain provisional until reproduced or supported +by current provider documentation. + +## Other Model-Level Observations + +- Kimi K2.5/K2.6 multimodality differs from older K2 variants (#2522). Promote + only from an exact provider card or scoped registry, never the `kimi` token. +- Google product names suggest media tasks to humans, but its Models resource + does not publish complete modalities. Keep those modalities unknown without + stronger model-scoped evidence. +- Ollama `/api/tags` names can omit vision markers (#3743, #4487). Use selected + model `/api/show.capabilities`, not its name. +- Local reasoning controls vary by serving template/config: message/system + directives, `chat_template_kwargs.enable_thinking`, native booleans, + structured objects, budgets, and effort levels were all observed (#3031). + These are endpoint/deployment facts, not universal checkpoint properties. +- DeepSeek, vLLM/NIM, Mistral, Moonshot, Ollama, and harmony-style servers use + different structured reasoning channels. Provider/dialect evidence chooses + the channel; generic response-text scanning is not capability discovery. +- Cohere native and compatibility transports expose different thinking + controls/channels. The Cohere model list does not itself prove reasoning. +- MiniMax M2.7 exposes different thinking channels through Anthropic and + OpenAI-compatible transports. Its current model list is identity-only. +- Gemma/Phi/Qwen vision behavior has changed across serving engines (#1430, + #1704, #1478). Native engine metadata or a verified endpoint probe outranks + a model-family name list. + +## Promotion Gate + +Before an observation becomes canonical runtime behavior, a consumer must +already have the necessary structured identity and tests must cover both its +positive scope and a neighboring negative scope. Request control and response +visibility remain separate: hiding reasoning text is not the same as disabling +reasoning at the provider (#2905). + +## Current Gaps + +- Runtime still contains model-name helpers for several implemented behaviors; + this spec records them but the canonical catalog does not duplicate them. +- Hosted aliases and provider behavior can change; there is no durable + observation expiry/revalidation layer yet. +- Detail/probe-only model facts cannot safely be populated from list discovery. diff --git a/specs/persistence.md b/specs/persistence.md new file mode 100644 index 000000000..f5e3ad298 --- /dev/null +++ b/specs/persistence.md @@ -0,0 +1,133 @@ +# Persistence + +Last updated: dev@df2fad2 | 2026-07-12 + +## Scope + +This spec covers durable state in: + +- `core/database.py`; +- `src/database.py`; +- `src/runtime_paths.py`; +- `src/constants.py`; +- `core/models.py`; +- `core/session_manager.py`; +- `core/atomic_io.py`; +- `src/attachment_refs.py`, `src/upload_handler.py`, and + `routes/upload_routes.py` for durable upload references and retention; +- JSON stores managed by `core/auth.py`, `src/settings.py`, `src/api_key_manager.py`, `src/preset_manager.py`, `src/integrations.py`, `src/upload_handler.py`, `src/personal_docs.py`, `src/research_handler.py`, `src/bg_jobs.py`, `routes/prefs_routes.py`, canonical `routes/contacts/contacts_routes.py` plus its shim, `routes/vault_routes.py`, `routes/cookbook_routes.py`, and memory/skills managers; +- `routes/email_helpers.py` scheduled-email storage; +- `routes/backup_routes.py` and `scripts/odysseus-backup`; +- runtime data under `data/`. + +## Database Shape + +`core/database.py` owns SQLAlchemy models and startup migrations. `src/database.py` is a compatibility re-export for legacy imports. Route and service code commonly owns its own `SessionLocal()` lifecycle instead of using one central unit-of-work wrapper. + +The default database is SQLite at `DATA_DIR/app.db`. `src.runtime_paths` and `src.constants` own the data-dir default: source runs use the repository `data/` directory, frozen builds default to `~/.odysseus/data`, and `ODYSSEUS_DATA_DIR` overrides both. SQLAlchemy can point at a non-SQLite `DATABASE_URL`, but current startup migrations/backfills are SQLite-first and often use `sqlite3`, `PRAGMA`, or SQLite catalog queries. External DBs are not fully migration-compatible unless those helpers are made backend-neutral. + +After `Base.metadata.create_all()`, `init_db()` resolves file-backed SQLite +paths from SQLAlchemy's parsed engine URL and attempts to restrict the main +database plus existing `-journal`, `-wal`, and `-shm` sidecars to `0600` on +POSIX. Driver-qualified, query-tagged, and local `file:` URI forms are covered; +non-SQLite, in-memory SQLite, and Windows paths are skipped. A failed POSIX +chmod is logged because the database and sidecars can contain password/token +hashes and encrypted provider material. + +Timestamp defaults use `utcnow_naive()` so existing naive `DateTime` columns stay UTC without the deprecated `datetime.utcnow()` default. + +Current model families include: + +- chat sessions, messages, and `chat_messages_fts` transcript-search state/triggers; +- documents and document versions; +- gallery albums/images, editor drafts, signatures, generated-media metadata; +- email accounts, model endpoints, MCP servers, comparisons; +- provider auth sessions for OAuth/device-flow-backed provider credentials; +- API tokens, admin-global webhooks, user tools/tool data, integrations; +- crew members, scheduled tasks, task runs, notes; +- memory rows, calendar calendars, and calendar events. + +Chat persistence stores model-readable text plus compact attachment-reference +lines in `chat_messages.content`, while structured references remain in message +metadata. Provider data URLs used by the live turn are not duplicated into the +durable transcript. The FTS migration recreates insert/update triggers to omit +inline media and scrubs legacy indexed rows that still contain data URLs. + +Current calendar/task persistence includes CalDAV remote identity columns (`CalendarCal.remote_href`, `CalendarCal.remote_etag`, `CalendarEvent.remote_href`, `CalendarEvent.remote_etag`), `CalendarEvent.caldav_sync_pending` for retryable writeback state, and `ScheduledTask.character_id` for built-in task persona selection. + +`EmailAccount` includes encrypted password fields plus Google OAuth fields (`oauth_provider`, encrypted access/refresh tokens, token expiry) and optional `display_name`. Startup migrations add those OAuth/display columns idempotently for older databases. + +`core/models.py` owns pure dataclasses used by `SessionManager`. It does not own database persistence. + +`routes/email_helpers.py` owns a second SQLite database at `data/scheduled_emails.db` for scheduled email, summary, reply, tag, sender-signature, urgency-alert, calendar-extraction, and cache state. Its migrations and owner backfills are local to that module, not `core/database.py`, and those auxiliary tables are owner-scoped. + +## Migration Policy + +Odysseus does not use Alembic. `core.database.init_db()` runs at module import, before FastAPI lifespan startup. `Base.metadata.create_all()` creates missing tables; hand-written `_migrate_*` functions add or reshape legacy columns. + +Runtime behavior: + +- migrations must be idempotent; +- SQLite foreign keys are enabled for every engine connection; +- new SQLAlchemy columns need matching startup migration code; +- legacy ownerless/shared rows may exist and must be handled by owner-aware route helpers. + +Startup backfills include document-owner backfill from linked sessions, blanket legacy owner assignment for SQL and selected JSON stores, `user_prefs.json` per-user nesting, email account seeding from legacy settings, and encryption rewrites for legacy plaintext endpoint, signature, and email secrets. Failed encryption rewrites are logged and retried on later startup. + +Owner-claiming is partly automatic and partly manual. `core.database._migrate_assign_legacy_owner()` assigns many ownerless SQL rows and selected JSON records to the primary admin when auth data exists, while `scripts/claim_ownerless.py` is an explicit local utility for claiming older ownerless memories, skills, sessions, documents, gallery rows, and comparisons. + +## Ownership And Access + +Owner columns are security-relevant. Current owner-bearing domains include sessions, documents, gallery images/albums, editor drafts, model endpoints, signatures, API tokens, user tools/tool data, comparisons, crew members, scheduled tasks/task runs, memories, notes, calendars/events, email accounts, and integrations. Webhooks are admin-global today and do not have an owner column. + +Route code owns filtering for its domain. `src.auth_helpers.owner_filter()` is the common helper where available; gallery, documents, calendar, email, skills, and other surfaces also use local filters. Null-owner compatibility is domain-specific: shared endpoints may include null owners, while strict gates and disk stores may reject them. Do not rely on frontend filtering for access control. + +There is no single anonymous/local owner value today. SQL `NULL` and missing JSON owners usually mean legacy/shared/unscoped compatibility; route-level no-login helpers use the empty string `""` when `AUTH_ENABLED=false`; chat/agent paths can pass `owner=None`; and calendar routes normalize empty owners to `ODYSSEUS_FALLBACK_OWNER` or `owner@localhost`. Email account helpers treat ownerless rows as single-user/global only for empty-owner mode; for non-empty owners, old ownerless account rows are visible only when the mailbox/from-address matches the owner. Lower-level helpers such as `get_upcoming_events(owner=None)` treat `None` as no owner scoping, so multi-user callers must pass a non-empty owner deliberately. + +## Secrets And Local Stores + +`ModelEndpoint` includes cached/hidden/pinned model lists, endpoint kind, refresh mode/interval/timeout, model type, supports-tools, owner, optional `provider_auth_id`, provider metadata, and encrypted API key columns. New endpoint columns need matching startup migration helpers. + +`ProviderAuthSession` rows hold OAuth/device-flow credential state for providers such as ChatGPT Subscription. Endpoints can reference those rows through `provider_auth_id`; deletion/cleanup must preserve auth rows still referenced by another endpoint and remove orphaned provider-auth rows only after the last endpoint reference is gone. + +`McpServer` includes stdio/SSE/HTTP transport config, plaintext env JSON, OAuth config, disabled tool names, and encrypted generic OAuth token/client state in `oauth_tokens`. Generic MCP token storage treats valid non-object JSON as empty state on reads and replaces it with an object on the next write instead of crashing callers. + +`CalendarCal.account_id` links synced local calendars back to one saved CalDAV account so multi-account sync/writeback can round-trip remote calendar identity. Remote href/etag columns on calendars and events preserve CalDAV server identity across pull/push cycles, while `caldav_sync_pending` marks local create/update/delete work that still needs remote writeback. + +`EncryptedText` owns transparent encrypted-at-rest DB columns via `src.secret_storage` for model endpoint keys and signatures. Email passwords and Google OAuth access/refresh tokens are `String` columns encrypted/decrypted manually. Integrations, CalDAV/CardDAV prefs, and other JSON stores can use `src.secret_storage` directly. API tokens are bcrypt-hashed, API-key manager state uses `data/.key` plus `data/api_keys.json` with restrictive chmod where supported, and vault state in `data/vault.json` is chmod-restricted JSON. Legacy plaintext rows are tolerated until migration or rewrite. + +Current JSON/local stores include: + +- `data/auth.json` for users, password hashes, TOTP, privileges, and auth settings; +- `data/sessions.json` for persisted browser session tokens; +- `data/settings.json`, user preferences, feature flags, integration settings, and `data/embedding_endpoint.json`; +- presets, API key manager state, memory/skills state, upload metadata, personal docs indexes, research JSON, background jobs, contacts/vault JSON, and task/cookbook auxiliary state. + +Cookbook state lives under the shared `DATA_DIR` path through the `COOKBOOK_STATE_FILE` constant. Search cache/analytics, FastEmbed cache fallback, uploads, generated media, logs, and auxiliary SQLite stores also resolve from shared data-dir constants and must work with source, Docker, and frozen data-dir defaults. + +`core.atomic_io` owns atomic file-write behavior for auth/settings/integration-style stores. Upload metadata uses its own locked atomic writer with `.bak` recovery and can rewrite owner fields plus owner-qualified index keys during user rename. Attachment-bearing chat/session, document, note, and calendar writers take owner-checked upload reservations before their durable writes; reservations share the upload-index lock with cleanup and refresh access time. Cleanup receives a complete reference snapshot from chat content/metadata, document current/version content, gallery rows, notes, and calendar rows, and removes only expired uploads proven unreferenced with coherent index state. Missing/incomplete scans fail closed, and index rows are restored when byte deletion fails. Memory and user prefs use temp-and-rename. API keys preserve encrypted values when saving one provider, presets persist atomically, and settings/feature loads degrade to defaults when the store is unreadable. + +Persisted memories, skills, documents, email, RAG chunks, notes, and other user-editable data are untrusted when reintroduced to model context. Route and processor code must pass them through the untrusted-context contract described in `context-building.md` and `auth-security.md`. + +## Backup And Restore + +`routes/backup_routes.py` owns narrow admin HTTP JSON export/import for memories, presets, skills, settings, features, and prefs. Skill import writes through the disk-backed skills manager API. This is not a full system restore path. + +`scripts/odysseus-backup` owns local `data/` snapshot/restore, with some large/runtime subtrees such as deep research and mail attachments behind flags. It uses SQLite backup APIs, includes secret-bearing key files and stores, validates restore archives against path escapes and link entries, and skips list entries that disappear or become unstatable during directory iteration. Backup artifacts should be treated as sensitive. + +## Transitional Notes + +The repo still mixes database-backed and JSON-backed persistence. Some domains have both legacy manager state and newer SQLAlchemy rows. `src.database` remains a live compatibility import path. `services/memory/memory.py` and `services/memory/memory_vector.py` now re-export canonical `src` memory classes; preserve compatibility unless the change explicitly migrates a store and includes backfill/tests. + +Docker bind-mounts `data/`, `logs/`, cache/local state, and optional Chroma state. The entrypoint repairs ownership for `PUID`/`PGID` before dropping privileges. POSIX secret files attempt restrictive chmod; Windows permission hardening is best-effort/no-op through platform compatibility helpers. + +ChromaDB/vector stores are optional durable storage outside `data/app.db`; missing Chroma degrades RAG, memory-vector, and tool-index features without blocking core SQLite/JSON persistence. Vector collections can be lane-suffixed for custom HTTP embeddings versus FastEmbed fallback. See `documents-rag-uploads.md`. + +## Current Gaps + +- Migration behavior is centralized but long and manual. +- Ownerless legacy rows make access-control reasoning harder. +- Some JSON store shapes are only documented by manager code and tests. +- Startup migrations lack a legacy-schema/idempotence test harness for owner backfills, encrypted-secret rewrites, and repeated runs. +- JSON-store atomicity is inconsistent across stores, though prefs and upload metadata now have focused atomic-write paths. +- Agent filesystem tools currently allow broad `data/` access; secret-bearing files under `data/` need explicit deny coverage. diff --git a/specs/research.md b/specs/research.md new file mode 100644 index 000000000..7e0d44d97 --- /dev/null +++ b/specs/research.md @@ -0,0 +1,153 @@ +# Research + +Last updated: dev@e57f60b | 2026-07-20 + +## Scope + +This spec covers deep research behavior in: + +- app wiring and timeout policy in `app.py` and `src/app_initializer.py`; +- canonical browser/API routes in `routes/research/research_routes.py`, with `routes/research_routes.py` as a compatibility shim; +- chat-triggered research in `routes/chat_routes.py`; +- diagnostics in `routes/diagnostics_routes.py`; +- scheduled research in `routes/task_routes.py` and `src/task_scheduler.py`; +- active runtime code in `src/research_handler.py`, `src/deep_research.py`, `src/research_utils.py`, and `src/visual_report.py`; +- search/fetch dependencies in `src.search`, `services.search`, and the `src.search.content` compatibility alias; +- compatibility/public service code in `services/research/research_handler.py` and `services/research/service.py`; +- agent tools in `src/tool_implementations.py`, `src/tool_execution.py`, and `src/tool_index.py`; +- research CLI access in `scripts/odysseus-research`; +- frontend modules `static/js/research/panel.js`, `static/js/research/jobs.js`, `static/js/researchSynapse.js`, `static/js/chat.js`, `static/js/chatRenderer.js`, `static/js/chatStream.js`, `static/js/documentLibrary.js`, `static/js/sessions.js`, and compare stream research UI; +- persisted reports under `data/deep_research/*.json`; +- tests under `tests/test_research_*`, `tests/test_deep_research_*`, `tests/test_visual_report*.py`, `tests/test_services_research_low_quality_sources.py`, `tests/test_svc_research_sources_nondict.py`, research auth regressions, endpoint fallback tests, and research CLI tests. + +## Current Call Sites Include + +- panel-launched research through `/api/research/start`; +- chat-stream research mode, including clarification, continuation from prior research JSON, progress events, and consumed results; +- non-streaming chat inline research context; +- compare/chat frontend research indicators; +- agent `trigger_research` and `manage_research`; +- scheduled research tasks that write compatible report JSON directly; +- diagnostics `/api/test-research`; +- report library, visual report, hide/unhide image, archive/delete, spinoff, and CLI list/show/report/search/delete flows. + +## Job Ownership + +`src.research_handler.ResearchHandler` owns panel and chat-stream active research jobs: validation, query synthesis, model probing, endpoint/model selection inputs, task registry state, cancellation, progress, raw findings, result persistence, average-duration caching, owner stamping, and owner rename for active/disk-backed task state. + +`routes.research.research_routes` owns the browser/API surface: auth and privileges, active/status/cancel/result/result-peek/stream routes, report HTML, hide/unhide images, library/detail/archive/delete, endpoint resolution for panel launch, and spinoff chat creation. Top-level `routes.research_routes` is a `sys.modules` compatibility shim. + +`TaskScheduler` owns scheduled research execution. It uses `DeepResearcher` directly, creates `[Research]` chat sessions, and writes `data/deep_research/*.json` in a compatible library/report shape without going through `ResearchHandler.start_research()`. + +Agent tools and the CLI read and mutate persisted research JSON directly. They are separate policy surfaces and must not be assumed to inherit browser route owner gates. + +## Research Runtime + +`src.deep_research.DeepResearcher` owns multi-round research work: + +- date/context setup; +- search provider selection and fallback through `src.search.providers` and `src.search.core`; +- URL/content fetching through `src.search.fetch_webpage_content`; +- separate tracking of analyzed URLs, last search errors, and empty-round limits; +- source summarization/extraction; +- synthesis into final answers/reports; +- partial/fallback reports when extraction or synthesis fails. + +Panel runtime behavior: + +- reconnects to active jobs through `/api/research/active`; +- starts jobs through `/api/research/start`; +- streams progress over `/api/research/stream/{id}`; +- falls back to status polling when SSE is unavailable; +- reads non-destructive results through `/api/research/result-peek/{id}`; +- opens visual reports from persisted JSON. + +Chat-stream runtime behavior: + +- first vague research messages can ask clarifying questions and set `research_pending`; +- later messages synthesize a focused research query; +- prior persisted research can seed continuation; +- progress, sources, raw findings, and `research_done` are emitted as SSE events; +- `/api/research/result/{id}` is destructive for chat consumption and marks/clears consumed in-memory results. + +Spinoff/Discuss creates a new chat session from a saved report. It seeds the report text as a system primer with `research_spinoff_from` metadata, uses the source session owner/endpoint context where available, disables RAG by default for the new session, and keeps source details out of the chat context to avoid fabricated citations. + +## Reports And Persistence + +Research persistence uses `data/deep_research/.json`. Current JSON can include result/report text, raw report, sources, raw findings, stats, category, archived state, hidden images, owner, timestamps, and consumed state. + +Route access to persisted report files is path-confined. Browser routes validate +session ids against `^[a-zA-Z0-9-]{1,128}$`, enumerate trusted `*.json` files +under the resolved research storage root, match by exact filename, reject +symlink/path escapes after `resolve().relative_to(root)`, and then perform owner +checks before detail/archive/delete/result-peek/spinoff reads or mutations. +Invalid ids return 400; missing or cross-owner reports return 404. + +`src.visual_report` owns HTML report generation from markdown-like research output, heading/TOC processing, category styling, image injection, allowlist sanitization of untrusted rendered HTML, and client-side controls for hiding images and discussing reports. + +Research library thumbnails prefer visible source/report images and Open Graph images, while avoiding obvious logos/icons and blocked/hidden images. + +`clear_result()` marks/clears in-memory state; it does not delete the on-disk report. Library/detail/report/archive/delete routes operate on persisted JSON. + +## Frontend Panel + +`static/js/research/panel.js` owns the research modal/panel UI, settings, provider controls, job cards, result rendering, destructive actions, progress display, and library counts. + +`static/js/research/jobs.js` owns active-job adoption, SSE connection, polling fallback, cancel, and result-peek flow. `researchSynapse.js` owns the compact running-state indicator. Chat and library frontend modules own report buttons, discuss/spinoff entry points, and older library views. + +## Degraded Runtime + +- `/api/research*` is exempt from the app-level hard request timeout. +- `ResearchHandler.start_research()` applies `research_run_timeout_seconds`; `0` means unlimited and bounded settings protect accidental extremes. User-selected round count is threaded into `DeepResearcher`; `max_rounds=0` means automatic mode capped by the route/handler rather than unbounded research. +- Deep extraction has separate timeout and concurrency controls. +- Scheduled research currently uses its own fixed max-time behavior. +- Probe failures are formatted before long jobs start. +- Search provider failure records `_last_search_error` and degrades through provider chains or empty results. +- Fetch/extraction failures skip individual sources when possible. +- Synthesis/final-report failures should preserve gathered material where possible. +- Provider, search, fetch, or model offline states should become failed/degraded job state, not app crashes. + +Native/Docker endpoint behavior is delegated to model endpoint registration and `src.endpoint_resolver`. Research does not guarantee useful output without a working model plus some usable search/fetch source path. + +## Compatibility State + +The active FastAPI app path uses `src.research_handler.ResearchHandler`. + +`services/research/service.py` is a public wrapper around a duplicate `services.research.research_handler.ResearchHandler`. That services handler remains compatibility/cleanup surface rather than canonical runtime truth; check parity before assuming it has every active-route field or policy behavior. + +Its source extraction skips non-dict finding rows so one malformed cached or +generated entry does not discard later valid URL/title/summary sources. + +Search compatibility also matters: `src.search.core`, `src.search.providers`, and `src.search.content` alias the service search path so old imports stay live without a second fetch implementation. + +## Security Policy + +Research routes require an authenticated user, and start routes require research privilege. Persisted report access and mutations should return 404 for cross-owner or null-owner JSON. Archive/delete/hide-image/unhide-image must preserve owner gates. + +Endpoint secret policy: + +- `/api/research/start` must use owner-scoped enabled endpoints before decrypted API keys/base URLs are passed to the handler; +- endpoint/model selectors should resolve `ProviderAuthSession`-backed endpoints for the acting owner and filter non-chat/image-only models out of research model lists; +- spinoff/follow-up endpoint selection should keep using owner-scoped endpoint context when present; +- token-authenticated behavior must preserve token owner/scope expectations before being treated as an API surface. + +Research sources, fetched pages, summaries, generated reports, and saved research context are untrusted data when reused in chat or another model call. Fetched webpage content in `DeepResearcher` is wrapped with `untrusted_context_message("webpage", content)` before extraction; other reuse paths should keep the same user-role/metadata policy. + +Visual reports render model/source-influenced Markdown into HTML with inline JavaScript and remote images. Markdown HTML is allowlist-sanitized; category-derived CSS/classes, links, and image URLs need continued policy coverage. Report HTML remains a security-sensitive rendering surface. + +## Testing Coverage + +Existing useful coverage includes deep-research runtime/degraded tests, handler/service tests, persisted route owner-scope tests, endpoint selection tests, auth regressions, visual report tests, query fallback tests, and CLI preview/store tests. + +Coverage is still thin around live job route ownership, `/api/research/start` route behavior, SSE/result-peek/cancel edges, spinoff endpoint ownership, tool/CLI direct JSON access, remote-image policy, and frontend panel/jobs behavior. + +## Current Gaps + +- Consolidate, retire, or clearly deprecate `services/research/research_handler.py`. +- Decide whether direct JSON access by `manage_research` and `scripts/odysseus-research` must be owner-filtered like browser routes or is local/tool-only. +- Spinoff endpoint fallback needs continued owner-scoped endpoint regression coverage. +- Spinoff research context is preserved during trimming through metadata, but the system-message primer still needs an explicit policy decision versus the shared untrusted-context role/metadata wrapper. +- Research search/fetch logic does not yet share a single result shape with chat prefetch and agent tools. +- Visual report remote image policy needs stronger regressions. +- Scheduled research persistence needs dedicated route/library/report visibility coverage. +- Frontend research jobs/panel/SSE fallback behavior lacks direct tests. diff --git a/specs/runtime.md b/specs/runtime.md new file mode 100644 index 000000000..08c0c4f94 --- /dev/null +++ b/specs/runtime.md @@ -0,0 +1,110 @@ +# Runtime + +Last updated: dev@e57f60b | 2026-07-20 + +## Scope + +This spec covers current app runtime wiring in: + +- `app.py`; +- `src/app_initializer.py`; +- `src/runtime_paths.py`; +- `src/config.py`; +- `core/constants.py`; +- `src/constants.py`; +- `src/interactive_gate.py`; +- `src/host_docker_access.py`; +- `core/middleware.py`; +- all route setup functions registered from `app.py`, including canonical + `routes/gallery/`, `routes/memory/`, `routes/research/`, + `routes/history/`, `routes/contacts/`, and `routes/note/` packages plus top-level + compatibility shims; +- `routes/prefs_routes.py`, `routes/workspace_routes.py`, and `companion/routes.py`; +- `src/generated_images.py` for generated-media file resolution; +- `launcher.py`, `Odysseus.spec`, and platform launcher scripts where frozen/native startup changes runtime paths; +- static entrypoints in `static/index.html`, `static/login.html`, and `static/app.js`. + +## App Orchestrator + +`app.py` owns process-level startup and HTTP composition. It configures MIME types, `.env` loading, logging under `DATA_DIR/logs`, CORS, gzip compression, auth middleware, request timeout middleware, static files, generated-image serving, router registration, SPA HTML routes, health/readiness/runtime endpoints, and lifespan hooks. Its console, rotating-file, and direct-uvicorn logging levels use the existing `LOG_LEVEL` environment toggle and default to `INFO`; invalid levels also fall back to `INFO`. `core/middleware.py` owns security headers, admin helpers, and internal-tool token constants. + +`src/app_initializer.initialize_managers()` owns shared manager construction. It creates memory, skills, sessions, uploads, personal docs, API keys, presets, chat processor/handler, research handler, model discovery, and optional memory vector store. Route modules receive these dependencies from `app.py`; they should not recreate manager singletons. + +`app.py` separately owns runtime singletons and integration hooks for auth, vector RAG, TTS/STT, webhooks, scheduled tasks, MCP, assistant log globals, event bus wiring, AI interaction globals, API-token cache invalidation, and foreground activity tracking. `src.runtime_paths` owns source-versus-frozen app/data path resolution; `src.constants` derives `DATA_DIR` from `ODYSSEUS_DATA_DIR` or that runtime default. `core/constants.py` and `src/constants.py` are both live import paths and are not fully identical today, so new constants need explicit placement/compatibility decisions. + +The shared upload handler is also installed on the session manager and tool +helper, and `app.py` injects it into attachment-bearing route factories so +durable writers and cleanup use one lifecycle owner. + +## Routes And Static Serving + +Current router call sites include: + +- auth, uploads, emoji, sessions, admin wipe, memory, skills, chat, workspace, research, history, search, presets, diagnostics, cleanup, personal docs, embeddings, model endpoints; +- TTS/STT, documents, signatures, gallery, editor drafts, scheduled tasks, assistant, calendar, shell, Cookbook, HW Fit, compare, preferences, backup, fonts, Copilot and ChatGPT Subscription auth; +- MCP, webhooks, API tokens, notes, email, Codex/Claude scoped APIs, vault, contacts, and companion routes. + +Gallery, memory, research, history, contacts, and note routes have canonical +subpackage modules (`routes.gallery.gallery_routes`, +`routes.memory.memory_routes`, `routes.research.research_routes`, +`routes.history.history_routes`, `routes.contacts.contacts_routes`, and +`routes.note.note_routes`). The old top-level modules replace their +`sys.modules` entries with the canonical module object so legacy imports, +`importlib`, and monkeypatch tests target the same route module that `app.py` +uses. + +The SPA routes `/`, `/notes`, `/calendar`, `/cookbook`, `/email`, `/memory`, `/gallery`, `/tasks`, and `/library` all serve `static/index.html`. `static/` is served with revalidation for `.js`, `.css`, and `.html` because the frontend ships raw browser modules with no hashed build output. + +Direct app-owned endpoints include `/api/generated-image/{filename}`, `/backgrounds`, `/login`, `/api/version`, `/api/health`, `/api/ready`, `/api/runtime`, and `/api/activity/heartbeat`. `/backgrounds` points at `static/backgrounds.html`; if that file is absent or the route remains auth-gated, that is route/static drift rather than an intentional public contract. + +`/static/*` is auth-exempt and public. SPA HTML routes are auth-gated except `/login`, and they are nonce-injected dynamic `HTMLResponse` values outside the static mount. Generated images and videos are served from `data/generated_images` through the generated-image resolver with immutable/nosniff caching. + +## Runtime Security Boundaries + +Effective middleware order matters. CORS, `SecurityHeadersMiddleware`, `_RequestTimeoutMiddleware`, and GZip middleware are added before `AuthMiddleware`; auth short-circuit responses can therefore bypass downstream app handlers and should be tested when changing response headers or auth behavior. Text responses can be compressed when they pass through the app stack. + +Security headers include HSTS and a restrictive `Permissions-Policy` that disables camera/geolocation and only allows microphone from self. + +`_TIMEOUT_EXEMPT_PREFIXES` owns hard-timeout bypass policy. It is prefix-based and currently exempts all subroutes under `/api/chat`, `/api/shell/stream`, `/api/research`, `/api/model/download`, `/api/model/probe`, `/api/model-endpoints`, `/api/cookbook/setup`, `/api/upload`, `/api/image`, and `/api/memory/audit`. Memory audit has its own longer inactivity timeout. + +Generated-image path resolution fails closed for invalid names, path escape, and missing files. Ownership checks are best-effort when a current user exists: gallery rows owned by a different user return 404, rowless generated files are allowed, and DB/helper failures fail open. See `auth-security.md` for `LOCALHOST_BYPASS`, internal-tool loopback, proxy-header exclusion, and owner-impersonation policy. + +## Runtime Behavior + +- Request hard timeout applies to non-exempt paths that reach `_RequestTimeoutMiddleware`. +- `src.interactive_gate` tracks foreground requests, browser heartbeats, and active chat streams. Background task/email work can wait for a quiet window so scheduled jobs do not compete with visible browser or model activity. +- YouTube support is initialized through `services.youtube.init_youtube()`. +- Vector document RAG is initialized lazily through `src.rag_singleton.get_rag_manager()` and may be unavailable at startup. +- `routes.workspace_routes` lets the browser choose a server directory for agent turns; execution confinement is enforced below the route layer by tool execution. + +## Lifespan Startup + +Upload cleanup first snapshots durable chat, document, gallery, note, and +calendar references and aborts on scan or upload-index integrity failure. + +Startup purges leftover incognito sessions, reconciles default scheduled tasks before the task runner starts, and backfills legacy skill owners when possible. + +Startup fire-and-forget work includes upload cleanup, background-job monitoring, MCP built-in registration and user-server connection, tool-index warmup, model-endpoint warmup, endpoint keepalive, Cookbook serve lifecycle monitoring, hourly null-owner sweeps, and nightly skill audit. The in-process task scheduler is gated by `ODYSSEUS_INPROCESS_TASKS`; email polling is started from email route setup and gated separately by `ODYSSEUS_INPROCESS_POLLERS`. Foreground-gate knobs are `BACKGROUND_TASK_FOREGROUND_GATE`, `BACKGROUND_TASK_QUIET_MS`, `BACKGROUND_TASK_MAX_WAIT_SECONDS`, and `BACKGROUND_TASK_BROWSER_ACTIVE_SECONDS`. + +Shutdown cancels upload cleanup, stops the task scheduler, closes the webhook manager, and disconnects MCP servers. + +## Degraded And Platform Behavior + +- On Windows, HuggingFace symlink warnings are disabled so model files copy instead of symlink on network/UNC paths. +- `.env` is loaded with `utf-8-sig` to tolerate Notepad BOM files. +- Process-wide MIME registration forces stable `.js` and `.mjs` types across native platforms. +- Frozen/PyInstaller builds use `src.runtime_paths` so bundled app assets resolve from the executable payload while persistent data defaults to `~/.odysseus/data`; normal source runs still default to the repository `data/` directory unless `ODYSSEUS_DATA_DIR` overrides it. +- Docker detection in `/api/runtime` selects `host.docker.internal` as the Ollama default inside containers and `127.0.0.1` natively. Compose sets Chroma to `chromadb:8000`; native Chroma defaults live in `src/chroma_client.py`. +- `src.host_docker_access` treats host Docker access from inside the container as opt-in. Default Compose does not mount `/var/run/docker.sock`; `docker/host-docker.yml` plus `ODYSSEUS_ENABLE_HOST_DOCKER=true` are required before local container code considers the host Docker daemon available. +- Chroma-backed consumers degrade independently: personal-doc RAG can return route-level 503s, semantic memory vectors can be dropped from chat/memory wiring, and the tool index can fall back when vector retrieval is unavailable. +- RAG startup failure is throttled so failed clients do not poison later retries. +- MCP startup is asynchronous and non-critical. User-server connection is bounded, failures surface through MCP status routes, and builtin MCP calls can reconnect after crashes. +- `/api/health` is liveness only. `/api/ready` checks database reachability, writable data dir, and local-first storage metadata; it does not prove optional subsystem health for RAG, Chroma, MCP, memory vectors, tool index, or endpoint warmups. +- `/api/diagnostics/services` is an admin diagnostics endpoint for optional service health. It reports bounded, non-intrusive checks for ChromaDB, SearXNG, email accounts, ntfy, and model provider endpoints with `ok`/`degraded`/`down`/`disabled` style status values and strips secret-bearing URLs/errors. `/api/diagnostics/logs` returns a bounded tail of the app log for admin troubleshooting. + +## Current Gaps + +- `app.py` is still a large route registry and runtime orchestrator. There is no generated route manifest or smaller runtime composition layer yet. +- Long-running route timeout exemptions are manual and prefix-based; new SSE/proxy/task paths can be missed, while broad prefixes can exempt more routes than intended. +- Runtime tests cover small helper slices, but not full app import/TestClient behavior for mounted static cache headers, generated-image serving, timeout middleware, middleware order, lifespan startup wiring, or route/static drift. +- The diagnostics service-health endpoint is not a readiness gate and does not cover every optional subsystem. diff --git a/specs/search.md b/specs/search.md new file mode 100644 index 000000000..22949d526 --- /dev/null +++ b/specs/search.md @@ -0,0 +1,137 @@ +# Search + +Last updated: dev@df2fad2 | 2026-07-12 + +## Scope + +This spec covers web search, URL fetching, and search-derived context in: + +- `routes/search_routes.py`; +- `services/search/*` and exported `services.search.SearchService`; +- `src/search/*` compatibility aliases around canonical service modules; +- search call sites in `src/chat_processor.py`, `src/tool_execution.py`, `src/session_search.py`, `src/research_handler.py`, `src/deep_research.py`, and `services/research/research_handler.py`; +- search settings in `src/settings.py`, `static/js/settings.js`, and compare/research frontend search callers; +- YouTube context paths in `src/youtube_handler.py` and `services/youtube/youtube_handler.py`; +- research visual/report consumers in `src/visual_report.py` and `routes/research/research_routes.py`; +- tests under `tests/test_search_*`, `tests/test_service_search_*`, `tests/test_services_search_*`, `tests/test_security_regressions.py`, `tests/test_agent_loop.py`, `tests/test_deep_research_*`, `tests/test_research_handler_*`, `tests/test_youtube_*`, and `tests/test_og_image_extraction.py`. + +`routes/chat_routes.py` also exposes `GET /api/search`, but that route searches chat messages and belongs to chat history behavior, not web search. + +## Route Flows + +`routes/search_routes.py` owns the browser/API web-search routes: + +- `GET /api/search/config` returns search configuration with provider key presence, not secret values; +- `POST /api/search` calls `comprehensive_web_search(..., return_sources=True)` and returns `{context, sources, error?}`; +- `GET /api/search/providers` returns provider metadata and availability; +- `POST /api/search/query` calls one provider directly and returns `{results, provider, time, error?}` without ranking, fallback chains, cache formatting, or content fetch. + +Compare mode uses both route shapes: shared presearch uses `/api/search`, while provider/search comparison panes use `/api/search/query`. Research panels can pass provider override settings through research routes into the deep-research search path. + +Research provider naming is not fully normalized in the UI: some frontend selectors still use `google`, while provider dispatch expects `google_pse`. + +## Search Pipeline + +`services/search/core.py` owns `comprehensive_web_search()`. It coordinates provider selection, fallback chains, ranking, optional fetch/content extraction, formatted prompt context, cache invalidation, and analytics. + +`services/search/service.py` owns `SearchService`, the async facade exported by `services.search` and `services`. It wraps the synchronous comprehensive search path off the event loop and maps route-style output into service result rows. + +`services/search/providers.py` owns provider-specific calls for SearXNG, Brave, DuckDuckGo, Google PSE, Tavily, and Serper. `PROVIDER_INFO`, provider availability, missing-key behavior, and provider dispatch live there. + +`services/search/query.py` owns query enhancement and sanitization, including stripping markdown/code-fence noise from model- or user-supplied queries before provider calls and extracting Unicode/non-ASCII capitalized entity names. `services/search/ranking.py` owns result ranking, including word-boundary title/snippet/subject matching so short query terms do not match unrelated substrings. + +## Provider Settings And Fallback + +`src/settings.py` owns default provider settings. The default provider is SearXNG, with DuckDuckGo as the default fallback chain. `static/js/settings.js` owns the admin search settings UI, provider key presence display, provider selection, and fallback ordering. SafeSearch is a backend/provider setting today, not a visible Settings control. + +Provider API keys come from settings or environment at call time. Web config routes expose availability/presence only, non-admin settings reads are scrubbed, and chat settings tools cannot set provider credentials. + +Runtime behavior: + +- disabled search returns disabled/unavailable text in the comprehensive path; +- missing keyed-provider secrets return empty provider results instead of exposing secrets; +- SearXNG retries through JSON variants before HTML fallback, pins English/general-engine defaults where needed, and maps news/recency settings into provider time filters; +- comprehensive search retries providers and then walks the fallback chain; +- `/api/search/query` is a direct provider test/query path and does not use the comprehensive fallback chain. Direct provider result limits can be controlled dynamically by the caller. + +## Content Fetching + +`services/search/content.py` owns webpage fetch/extract behavior for the services path: + +- public HTTP/HTTPS URL checks; +- DNS fail-closed behavior; +- rejection of localhost, metadata, private, reserved, multicast, and link-local targets; +- redirect revalidation on each hop; +- one-time public DNS resolution per hop plus an `httpcore`/`httpx` pinned + transport that connects to the validated public IP while preserving the + original URL, Host header, and TLS SNI, closing DNS-rebinding time-of-check + drift; +- metadata, Open Graph image, list, table, code block, PDF, and text extraction; +- readable text extraction for `text/*`, Markdown, `.txt`, `.json`, `.jsonl`, and JSON content types; +- central User-Agent behavior through `WEB_FETCH_USER_AGENT`; +- soft and hard download byte caps through `WEB_FETCH_SOFT_MAX_BYTES` and `WEB_FETCH_HARD_MAX_BYTES`, with declared-length and streaming-budget checks; requests prefer identity transfer encoding so compressed bodies cannot bypass the effective body cap; +- JS-heavy empty result hints; +- cache writes; +- empty/error result shape, including explicit HTTP-status failures instead of raising through callers. + +`src/search/content.py` is now a compatibility alias to `services.search.content`; chat URL auto-fetch, agent `web_fetch`, and deep research keep the `src.search` import path but share the services implementation. + +Agent `web_fetch` raises the per-call budget only within the global hard cap, leads tool output with a partial-content notice when the download budget truncated the page, and then applies normal tool-output truncation so the notice survives. + +Content failures are caller-shaped: + +- comprehensive search drops failed page fetches and keeps usable search context; +- `web_fetch` returns tool errors, including bot-protection and HTTP-status failures; +- direct URL chat prefetch expects unavailable context rather than fabricated content; +- deep research records search/provider failures separately from extraction failures. + +## Result Shapes + +Search does not have one canonical result shape yet. Current shapes include: + +- `/api/search`: `{context, sources, error?}`; +- `/api/search/query`: `{results, provider, time, error?}`; +- `comprehensive_web_search(return_sources=True)`: formatted context plus `{url, title}` sources; +- `SearchService.search()`: service result rows; +- agent `web_search`: tool output text plus a hidden sources marker stripped by the agent loop; +- agent `web_fetch`: fetched page text or tool error; +- deep research: findings, cited sources, optional source images, and `_last_search_error` state. + +Chat/session transcript search is separate from web search but now uses `chat_messages_fts` when available, sanitizes FTS queries, and batches message lookup after FTS hits to avoid per-hit database reads. + +Search owns Open Graph image extraction for fetched pages. Research owns promotion of those images into research sources and visual reports. This is not a standalone web image-search provider or gallery image proxy. + +## YouTube + +`services/youtube/youtube_handler.py` owns YouTube URL detection, id extraction, transcript, comment, and formatting behavior. `src/youtube_handler.py` is a compatibility alias to the canonical services module so startup `init_youtube()` state and chat imports share one implementation. + +YouTube transcript and comment content is search-like external context. URL parsing covers common watch, mobile/music, embed, `/v/`, shorts, live, and `youtu.be` forms and must tolerate non-string input. + +## Compatibility State + +`src/search/core.py`, `src/search/providers.py`, `src/search/ranking.py`, `src/search/cache.py`, `src/search/content.py`, `src/search/query.py`, and `src/search/analytics.py` are compatibility shims or module aliases around `services.search`. Ranking helpers exposed through `src.search.ranking` include recency scoring, result ranking, naive-UTC handling, `_SPORTS_HINT_RE`, and age formats. + +`src.youtube_handler` remains a compatibility import path, but it should resolve to the same module object as `services.youtube.youtube_handler`. + +## Context Policy + +Search results, fetched pages, Open Graph metadata, and YouTube transcript/comment content are untrusted context. + +Chat search, chat URL prefetch, compare presearch, and YouTube context wrap inserted content through the shared untrusted-context message helpers. Agent `web_search`/`web_fetch` results are read-only tool outputs and must not be treated as instructions. + +Deep research wraps fetched webpage content through `untrusted_context_message("webpage", content)` before extractor calls, though search result/failure shapes still differ from chat and agent tools. + +## Optional And Platform Behavior + +`ddgs` is optional; provider code has an HTML fallback. Search cache and analytics state live under the shared data dir and mkdir failures in read-only image layers are tolerated where possible. PDF extraction uses `pdfminer.six` only when installed. Native SearXNG defaults to `http://localhost:8080`; Docker uses the compose `searxng` service URL and pins the SearXNG image with a healthcheck. + +`httpx` and BeautifulSoup are required runtime dependencies for the active search/fetch path. + +## Current Gaps + +- Search route handlers need direct tests for request body formats, provider validation, provider availability, and route error/empty-result shapes. +- Agent search, chat search prefetch, and research search do not yet share a single result/failure shape. +- `src/search` and `services/search` are mostly consolidated through shims, but import-path parity tests remain important. +- Deep-research webpage-content extraction uses the shared untrusted wrapper, but synthesis/reuse boundaries still need route/tool tests. +- Search-sourced `og_image` URLs need an explicit privacy/security decision: documented direct browser loads, public-URL validation, or a same-origin proxy. +- Route and integration tests do not fully pin chat/compare/YouTube untrusted-context insertion. diff --git a/specs/settings-admin.md b/specs/settings-admin.md new file mode 100644 index 000000000..b91566899 --- /dev/null +++ b/specs/settings-admin.md @@ -0,0 +1,187 @@ +# Settings And Admin Surfaces + +Last updated: dev@e57f60b | 2026-07-20 + +## Scope + +This spec covers settings and admin surfaces in: + +- `app.py` auth-exempt and route-registration wiring; +- `routes/auth_routes.py` for setup, login/status, users, features, settings, and integration settings routes; +- `core/auth.py` and `core/middleware.py` admin/privilege behavior; +- `src/settings.py` and `src/settings_scrub.py`; +- `routes/prefs_routes.py`; +- `src/preset_manager.py` and `routes/preset_routes.py`; +- `routes/backup_routes.py` and `scripts/odysseus-backup`; +- `routes/diagnostics_routes.py`; +- `routes/admin_wipe_routes.py`; +- `routes/cleanup_routes.py` and `src/cleanup_service.py`; +- `routes/vault_routes.py` and vault-related tool implementations; +- `routes/font_routes.py`; +- `routes/model_routes.py` for `/api/tools` and settings-bound model endpoint references; +- `src/agent_tools/admin_tools.py`, `src/tool_implementations.py`, `src/tool_execution.py`, `src/tool_schemas.py`, and `src/tool_index.py` for `manage_settings`; +- `src/agent_loop.py` for stale agent prompt references to settings APIs; +- frontend modules `static/js/settings.js`, `static/js/admin.js`, `static/js/presets.js`, `static/js/theme.js`, and `static/js/storage.js`; +- CLI helpers `scripts/odysseus-preset` and `scripts/odysseus-theme`. + +Generic API integrations are cross-referenced in `integrations.md`. Model endpoint CRUD and endpoint cleanup are covered in `llm-models.md`. Email/contact/calendar legacy setting fallbacks stay with their domain specs. + +## Data Stores + +`src.settings` owns `data/settings.json` and `data/features.json`. Settings and features are merged over defaults and cached briefly. Missing, corrupt, unreadable, or non-object stores fall back to defaults. + +`routes.prefs_routes` owns `data/user_prefs.json`. It supports: + +- `_users` multi-user storage; +- legacy flat prefs; +- auth-disabled first-user compatibility without clobbering the rest of `_users`. + +`src.settings.get_user_setting()` overlays only a whitelist of per-user prefs over global settings. That whitelist is mostly model/media endpoint choices. + +Other active stores include: + +- `data/presets.json`; +- `data/vault.json`; +- `static/fonts/custom`; +- DB-backed domain tables used by admin wipe and cleanup; +- browser localStorage/sessionStorage for theme, preset, privacy, and transient UI state. + +## Bootstrap, Auth, And Settings Routes + +`routes.auth_routes` owns first-run setup, login/logout/status, password/TOTP flows, signup controls, user CRUD, admin promote/demote, privilege edits, feature flags, and app settings. `app.py` exposes setup/status/features/settings routes before cookie auth so first-run and frontend bootstrap can work. + +Settings runtime: + +- `GET /api/auth/features` is public feature visibility metadata; +- `POST /api/auth/features` is admin-only; +- `GET /api/auth/settings` returns full settings to admins; +- non-admin or unauthenticated `GET /api/auth/settings` returns `scrub_settings()` output; +- `POST /api/auth/settings` is admin-only and only writes keys present in `DEFAULT_SETTINGS`. + +`src.settings_scrub` owns deep secret-key scrubbing for non-admin settings reads, including snake_case and camelCase secret-like key names. It preserves structure while blanking secret-shaped string values. + +Admin gates inherit the auth contracts in `auth-security.md`: normal deployments require an admin user, while `AUTH_ENABLED=false`, first-run/setup mode, validated internal-tool loopback, and direct localhost bypass have explicit behavior in auth middleware/helpers. + +## Preferences And Frontend State + +`routes.prefs_routes` owns per-user key/value preferences. Theme and custom-theme code uses localStorage first, syncs selected prefs through `/api/prefs/*`, and falls back from server prefs when local theme state is absent. + +`static/js/theme.js` owns: + +- theme and custom-theme persistence; +- old theme-name migrations; +- custom font selection and `/api/fonts/custom` discovery; +- bundled accessibility font selection such as OpenDyslexic and text-size variable application; +- CSS variable application. + +`static/js/settings.js` owns the Settings modal shell, non-admin settings panels, admin visibility sync, provider/model/search/research/reminder/email/CalDAV/CardDAV/vault panels, accessibility/font/text-size controls, scoped-token helpers, and unified integrations forms. Its email account forms include provider presets, Google Workspace/.edu OAuth connect/reconnect controls, display-name fields, password-field hiding for OAuth flows, and redirect result banners. `static/js/admin.js` owns user/admin panels, admin promote/demote controls, model endpoints, builtin tool toggles, MCP admin forms, feature toggles, token/webhook panels, diagnostics logs, backup/import, and danger-zone wipes. Google Gemini API endpoint creation omits `model_refresh_mode` so the backend can apply its manual default; proxies remain manual and other API endpoint forms submit auto refresh. + +Logout/user-switch flows clear local/session storage to avoid stale cross-account UI state. + +## Presets + +`src.preset_manager.PresetManager` owns preset persistence, atomic writes, default preset healing, corrupt-store fallback, and legacy custom-preset migration. `routes.preset_routes` owns HTTP behavior. + +Runtime behavior: + +- preset list/templates/groups/expand routes are read or utility surfaces; +- custom preset/template/group mutations are admin-gated; +- preset expansion can call the configured model; +- frontend activation combines persisted `custom.enabled` with local selected-preset UI state; +- presets, user templates, and group presets are currently shared stores, not owner-scoped stores. + +`scripts/odysseus-preset` is a local CLI for preset store maintenance and backup of `presets.json`. + +## Tools Settings + +`routes.model_routes` owns `/api/tools`, which writes `settings.json:disabled_tools` for global builtin tool toggles. + +`src.agent_tools.admin_tools.do_manage_settings()` owns the model-facing settings tool and is re-exported through `src.tool_implementations`. It is admin-only through tool execution/security policy, writes real global settings, refuses secret-shaped setting writes, refuses structured clobbers, resolves model aliases to endpoints, and can enable/disable tools. + +The stale `app_api` prompt text that mentions `/api/settings` is not the canonical settings surface; the live HTTP route is `/api/auth/settings`, and `manage_settings` is the intended agent settings tool. The `manage_settings` schema also still describes free-form preferences even though implementation only accepts keys in `DEFAULT_SETTINGS`. + +## Backup And Import + +`routes.backup_routes` owns admin JSON export/import for selected app state: + +- owner-filtered memories; +- shared presets; +- owner-filtered skills; +- raw global settings; +- feature flags; +- per-user preferences. + +HTTP export is secret-bearing because it includes raw settings. Treat exported files as sensitive admin artifacts. + +HTTP import is best-effort and section-based. It rejects invalid top-level JSON, ignores unrecognized or wrongly typed sections, merges recognized sections, and may partially write earlier sections before a later failure. Memory dedup is scoped to the importing user; imported memories/skills without owners are stamped to the caller, while explicit owner fields are preserved. Skill import writes through the disk-backed `SkillsManager.add_skill()` API, not the removed JSON-era `save()` shape. + +`scripts/odysseus-backup` is a separate local `data/` snapshot/restore tool, with some large/runtime subtrees behind flags. It uses SQLite backup where applicable, rejects archives written inside `data/`, validates restore members, refuses links/special files, and skips entries that disappear or become unstatable while a backup directory listing is assembled. + +## Diagnostics, Cleanup, And Wipe + +`routes.diagnostics_routes` owns admin diagnostics for DB, RAG, YouTube, research status, aggregate optional service health, and application log tails. The service-health endpoint checks ChromaDB, SearXNG, email accounts, ntfy, and model provider endpoints with bounded probes and redacted output. URL-bearing diagnostics should use log-safety redaction helpers so credentials/query strings do not leak. `/api/diagnostics/logs` reads a bounded tail from `DATA_DIR/logs/app.log`, with missing logs returning an empty result. Diagnostics are operational and must avoid growing into broad secret/environment dumps. + +`routes.cleanup_routes` is owner-scoped, not admin-only. It previews and applies session cleanup for the current user through `src.cleanup_service`; when auth is disabled, cleanup can operate as a single-user unscoped flow. + +`routes.admin_wipe_routes` owns global per-domain destructive wipe actions. Current kinds include chats, memory, skills, notes, tasks, documents, gallery, and calendar. Server enforcement is admin gate plus kind allowlist. Frontend double confirmation in `static/js/admin.js` is user-interface protection, not server authorization. + +## Vault + +`routes.vault_routes` owns Vaultwarden/Bitwarden CLI config, login, unlock, lock, logout, and `bw_installed` checks. + +Runtime behavior: + +- `GET /api/vault/config` returns no `session` value; +- `data/vault.json` stores config and `BW_SESSION`; +- POSIX saves attempt `0600` permissions; +- master passwords are passed to `bw` on stdin, not argv; +- missing `bw` degrades to route error/status responses; +- corrupt or non-object vault config loads as empty config; +- lock/logout clear the saved session. + +Vault tool paths duplicate some route behavior and can return vault item secrets to an admin tool result after a reason check and audit log. They are admin/local trust-boundary surfaces. + +## Fonts + +`routes.font_routes` lists user-supplied font files under `static/fonts/custom`. It is a support/discovery route, not an admin operation. `static/js/theme.js` owns consuming this list for theme font selection. + +## Security And Provenance + +- Non-admin and unauthenticated settings reads are scrubbed. +- Admin settings reads, admin edit forms, vault flows, backup files, and local CLI artifacts can contain secrets and must remain admin-only or locally protected. +- Backup artifacts are sensitive because settings may include API keys, passwords, tokens, and endpoint credentials. +- Diagnostics and logs should avoid adding secret-bearing values. +- Admin wipe is global per kind and crosses owners. +- Cleanup is owner-scoped in normal auth mode. +- `manage_settings` blocks secret-shaped setting writes and structured setting clobbers. +- Vault master passwords must not appear in process argv. +- Client-side confirmations are not server authorization controls. + +## Degraded And Compatibility Behavior + +- Settings/features fall back to defaults on missing/corrupt/unreadable/non-object stores. +- `is_setting_overridden()` has a narrower error contract than `load_settings()`. +- Prefs support legacy flat files and auth-disabled first-user writes. +- Presets heal missing built-ins and legacy custom state without clobbering user edits. +- `/api/import` is non-atomic section merge. +- Vault route and vault tool degraded behavior are not identical. +- Theme/preset frontend helpers tolerate malformed localStorage values. +- CLI helpers are local maintenance surfaces and may bypass HTTP route policy. + +## Testing Notes + +Current targeted coverage includes settings store fallback/error paths, settings scrub, prefs no-clobber behavior, atomic preset store/migration/CLI/localStorage helpers, backup import cross-user dedup, backup CLI restore/list-race safety, cleanup owner scope, diagnostics admin-gate/source/service-health/log-tail checks, admin promote/demote, admin wipe gallery, font family derivation, theme helper behavior, vault password-not-in-argv checks, setup/auth regressions, reserved usernames, Google email OAuth route/helper behavior, and a token-budget `manage_settings` path. + +## Current Gaps + +- Add route tests for `/api/auth/settings`: anonymous/non-admin scrubbed reads, admin full reads, non-admin POST rejection, and unknown-key ignore behavior. +- Add route tests for `/api/auth/features` admin writes. +- Add `/api/tools` and `manage_settings` tests for secret write refusal, enum/integer coercion failures, structured-setting refusal, reset/default behavior, endpoint/model resolution, and tool enable/disable aliases. +- Add backup tests for secret-bearing export policy, owner-scoped exported sections, invalid import handling, skills dedup, settings/features merge, and admin gates. +- Add diagnostics tests for broader error redaction and sensitive output limits. +- Add admin wipe tests for every wipe kind, unknown-kind 400, rollback behavior, and admin gating. +- Add vault route tests for session omission, permission setting, login/unlock failures, lock/logout clearing, corrupt config, and admin gates. +- Add frontend tests for Settings/Admin panel save/load flows, vault password clearing, diagnostics buttons, cleanup/wipe confirmations, custom font/theme wiring, and tab state. +- Decide whether `user_templates` and `group_presets` should remain shared despite user-facing names. +- Decide whether backup/import should preserve explicit owner fields or force imported owner ownership. +- Decide whether a dedicated split is needed for the large `static/js/settings.js` and `static/js/admin.js` ownership boundary. diff --git a/specs/shell-mcp.md b/specs/shell-mcp.md new file mode 100644 index 000000000..6702d49f8 --- /dev/null +++ b/specs/shell-mcp.md @@ -0,0 +1,172 @@ +# Shell And MCP + +Last updated: dev@28d27ee | 2026-07-17 + +## Scope + +This spec covers shell and MCP behavior in: + +- shell routes in `routes/shell_routes.py`; +- the standalone shell helper in `services/shell/service.py`; +- agent shell/background execution in `src/tool_execution.py`, `src/agent_tools/subprocess_tools.py`, `src/bg_jobs.py`, and `src/bg_monitor.py`; +- app wiring and startup/shutdown in `app.py`; +- MCP configuration routes in `routes/mcp_routes.py`; +- MCP runtime state in `src/mcp_manager.py`; +- generic MCP OAuth helpers in `src/mcp_oauth.py`; +- built-in server registration in `src/builtin_mcp.py`; +- persisted `McpServer` config in `core/database.py`; +- MCP tool exposure in `src/agent_loop.py`, `src/tool_index.py`, `src/tool_schemas.py`, `src/tool_parsing.py`, `src/tool_implementations.py`, and `src/tool_security.py`; +- admin MCP/tool helpers in `src/agent_tools/admin_tools.py`; +- built-in servers in `mcp_servers/*.py`; +- Settings/Admin UI in `static/js/settings.js` and `static/js/admin.js`; +- CLI helper `scripts/odysseus-mcp`; +- Docker/native dependency context in `Dockerfile` and `docker-compose.yml`. + +Cookbook model-serving shell flows are covered in `cookbook-hwfit.md`; this spec owns the shared shell and MCP surfaces they reuse. + +## Shell Routes + +`routes.shell_routes` owns `/api/shell/exec` and `/api/shell/stream`. These routes are powerful by design and are admin-only. They execute admin-provided command strings through the host shell. + +Runtime behavior: + +- `/api/shell/exec` runs a bounded command and returns stdout, stderr, and exit code; +- `/api/shell/stream` streams SSE output through plain pipes, POSIX PTY, POSIX tmux log tailing, or a Windows detached-log fallback depending on request flags and platform; +- empty commands return an error result without spawning a shell; +- timeouts kill the subprocess where possible; +- disconnects can stop streaming subprocesses; +- POSIX PTY support is optional and reports an unsupported event when unavailable. + +`routes.shell_routes` also owns shell-adjacent Cookbook dependency endpoints: + +- `/api/cookbook/packages`; +- `/api/cookbook/packages/install`; +- `/api/cookbook/rebuild-engine`. + +Those endpoints probe local or SSH-remote packages, prepend user install bins for pip CLIs, validate SSH host/port through shared route validators, validate remote venv values, and restrict package installs to allowlisted dependencies. + +`services.shell.service.ShellService` is a small standalone subprocess abstraction with output caps. It does not own live route behavior, PTY/tmux paths, Windows shell selection, admin checks, or Cookbook package probes. + +## Agent Shell And Background Jobs + +`src.tool_execution` owns agent-side `bash` execution and the `#!bg` marker. A `bash` block whose first line is `#!bg` starts a detached background job instead of holding the chat stream open. + +`src.bg_jobs` owns disk-backed job state under `data/bg_jobs.json` and `data/bg_jobs/*`. It stores wrapper scripts, logs, exit-code files, timestamps, status, and capped result text. + +`src.bg_monitor` owns polling and auto-continuation. When a job finishes, it injects the job result into the session, drains the agent stream, persists only the assistant continuation plus `bg_result` metadata, and marks the job followed up. + +Runtime behavior: + +- background jobs are restart-tolerant while their state files remain; +- jobs have a maximum runtime and stale cleanup window; +- output is capped with head/tail retention; +- active sessions can defer follow-up until the next monitor pass. + +## Configured MCP Servers + +`routes.mcp_routes` owns admin HTTP configuration for MCP servers: + +- list/add/reconnect/enable/disable/delete servers; +- list tools and per-server tools; +- update per-server disabled tool lists; +- Google OAuth authorize/callback/manual exchange pages and generic Streamable HTTP OAuth redirect handling. + +`core.database.McpServer` persists transport, command, args, env, URL, enabled state, OAuth config, disabled tool names, and encrypted generic OAuth token/client state. `McpServer.env` is plaintext JSON in the database. + +`src.mcp_manager.McpManager` owns live connection state, stdio/SSE/Streamable HTTP transports, sessions, tool schemas, qualified names, and tool calls. HTTP route operations update both database state and live manager state where applicable. Streamable HTTP connects in a background task, can report `connecting` or `needs_auth`, and surfaces an authorization URL when the OAuth client flow redirects. Enabled configured servers connect concurrently at startup; each server has its own 20-second connection timeout and records `timeout` state without delaying siblings. The startup task has no second outer timeout. + +Stdio and SSE connection setup registers the session, exit stack, tool list, +and status as one completed unit. If initialization or tool discovery fails +before registration, the partial `AsyncExitStack` is closed so transports do +not leak into later reconnect attempts. + +`src.agent_tools.admin_tools.do_manage_mcp()` is the agent/admin tool path for MCP config and is re-exported lazily through `src.tool_implementations` for compatibility. It is narrower than the HTTP routes: add is stdio-only, command values are checked against an allowlist/denylist before persistence, and enable/disable primarily flips DB config. `scripts/odysseus-mcp` is config-only; it reads and mutates database rows, redacts env values by default, and does not report live manager connection state. + +## Built-In MCP Servers + +`src.builtin_mcp` owns startup registration of built-in MCP servers unless `ODYSSEUS_DISABLE_MCP` is enabled. + +Python stdio built-ins: + +- image generation; +- memory; +- RAG; +- email. + +The optional browser built-in uses `npx -y @playwright/mcp@latest --headless --caps vision`. It is cache-gated by checking npm's `_npx` cache for the requested package and falling back to `npx --no-install`; uncached/missing browser MCP is logged with install guidance and skipped rather than blocking startup or downloading packages at boot. Python built-ins are omitted from OpenAI function schemas because native/code-block paths already describe those capabilities; the browser built-in is exposed through MCP function schemas when connected. + +Built-in Python servers prepend the app root to inherited `PYTHONPATH` rather +than replacing the environment, so container/dev site-packages remain visible +on initial connect and automatic reconnect. They can be reconnected once on +tool-call failure. User-configured MCP servers return the call failure instead +of automatic reconnect. + +The built-in email MCP server is owner-aware when an owner is supplied by the +caller or configured through `ODYSSEUS_MCP_EMAIL_OWNER` / +`ODYSSEUS_EMAIL_OWNER`; if owner-scoped email accounts exist and no owner is +available, email MCP fails closed instead of exposing global accounts. Other +built-in servers remain process-global/admin trust-boundary tools unless their +own subsystem spec says otherwise. + +## Agent MCP Exposure + +`McpManager` owns raw qualified tool calls named `mcp__{server_id}__{tool_name}`. It does not own admin, owner, public-user, or disabled-tool policy; callers must enforce policy before dispatch. + +Current exposure path: + +- `routes.mcp_routes` stores disabled tool names; +- `src.agent_loop` loads disabled maps for prompts/schemas; +- `McpManager.get_all_openai_schemas()` and prompt descriptions filter disabled tools; +- `src.tool_index` indexes MCP prompt descriptions by manager generation; +- `src.tool_security` blocks all `mcp__*` tools for non-admin/public users; +- `src.tool_execution` dispatches received `mcp__*` calls to `McpManager.call_tool()`. + +Per-server disabled MCP tools currently hide tools from prompts/schemas while listings still return tools with disabled metadata. They are not a complete execution-time gate if a disabled qualified name reaches tool execution. Plan mode additionally asks `McpManager.plan_mode_blocked_mcp()` to hide write/unknown MCP tools and add qualified names to the runtime disabled set for that turn. + +## Degraded And Platform Behavior + +- `app.py` starts the background monitor and MCP startup tasks asynchronously; MCP startup is non-critical to app readiness. +- Configured MCP servers start concurrently with a per-server 20-second bound; + timeout state is stored per server and partial connection resources are + closed before returning. +- Missing Python `mcp` dependency degrades attempted MCP connections to error status. +- Missing or uncached browser NPX package is optional and log-only during built-in startup; startup should not perform an implicit package download. +- Windows does not support POSIX PTY/tmux paths; streaming falls back to pipes or detached logfile behavior. +- Docker images include selected shell dependencies and the Docker CLI, but host Docker socket access from inside the app container remains unavailable unless the operator explicitly enables `docker/host-docker.yml`/`ODYSSEUS_ENABLE_HOST_DOCKER=true` and mounts a real socket. +- OAuth supports Google `installed` or `web` key shapes, a remote paste-back exchange page, and generic Streamable HTTP OAuth token storage through encrypted `McpServer.oauth_tokens`. Valid JSON values that are not objects are treated as empty token state and replaced by an object on the next write. Google and generic MCP OAuth share `src.mcp_oauth.REDIRECT_URI`, built from `OAUTH_REDIRECT_BASE_URL`, `APP_PUBLIC_URL`, or `http://localhost:7000` plus `/api/mcp/oauth/callback`; `APP_PORT` is intentionally not part of this redirect calculation. +- `services.shell.service` remains a transitional/simple facade separate from route-level compatibility behavior. + +## Security And Provenance + +- Admin shell is intentional host command execution; do not expose shell routes or shell tools to regular users. +- `_require_admin()` gates shell routes and MCP config routes. The internal-tool loopback can be admin-equivalent only after auth middleware validates the internal token and loopback client. +- `_reject_cross_site()` currently applies to `/api/cookbook/packages`; `/api/shell/exec`, `/api/shell/stream`, package install, rebuild, and MCP write/OAuth routes do not call it directly. +- Shell helper paths use argv-based SSH, reject option-like hosts, validate SSH ports through shared helpers, restrict remote venv characters, and allowlist package installs. +- Non-admin/public tool policy blocks `bash`, `python`, file tools, `manage_mcp`, and all `mcp__*` tools. +- MCP stdio server registration is arbitrary host process execution and is admin-only. +- MCP OAuth key/token file paths supplied through routes are confined under `data/mcp_oauth`; generic Streamable HTTP OAuth token state is encrypted in the database. +- Built-in MCP servers are local/admin trust-boundary tools and are not + automatically equivalent to owner-scoped HTTP route behavior. Email MCP is + the current exception with explicit owner filtering; other built-ins need + their own owner policy before being treated as scoped surfaces. +- MCP output is untrusted tool output. Current MCP text output is not centrally capped before model re-entry. + +## Testing Notes + +Current targeted coverage includes Windows PTY import degradation, PTY unsupported stream events, the cross-site helper, `ShellService` stream deadline behavior, background store/monitor basics, concurrent MCP startup, per-server timeout isolation and cleanup, MCP manager cache/reconnect args, built-in `PYTHONPATH` preservation, non-object generic OAuth-token storage recovery, MCP CLI JSON/env serialization, MCP common truncation helper, action intent shell verbs, and public blocked-tool fail-closed behavior. + +The shell/MCP audit ran the targeted venv subset with 78 passing tests and one warning. + +## Current Gaps + +- Decide whether `/api/shell/exec`, `/api/shell/stream`, package install, rebuild, and MCP config/OAuth writes should call `_reject_cross_site()` directly. +- Add route-level shell exec/stream tests for admin gate, cross-site behavior, empty command, plain exec, timeout, PTY, tmux, and Windows detached fallback. +- Add background job tests for launch isolation, output truncation, done/failed/timeout/died states, pending follow-ups, and result text. +- Add route-level MCP CRUD/OAuth/disabled-tool tests with a fake manager and temp database. +- Add hard per-server disabled MCP execution checks or document disabled tools as prompt/schema filtering only. +- Make MCP tool indexing sensitive to disabled-map changes, not only manager generation. +- Fix stale outer prompt/cache behavior when MCP disabled tools change. +- Add central truncation and untrusted-context wrapping for MCP result text and images before model re-entry. +- Decide whether `McpServer.env` and OAuth key files need masking, encryption, and chmod beyond admin-only access. +- Decide whether built-in MCP servers should become owner-aware or remain documented as admin/global compatibility surfaces. +- Decide whether optional browser MCP cache misses should surface in `/api/mcp` status instead of startup logs only. diff --git a/specs/speech.md b/specs/speech.md new file mode 100644 index 000000000..8ac6657b4 --- /dev/null +++ b/specs/speech.md @@ -0,0 +1,130 @@ +# Speech + +Last updated: dev@df2fad2 | 2026-07-12 + +## Scope + +This spec covers speech behavior in: + +- app service initialization and route registration in `app.py`; +- `services/stt/stt_service.py`; +- `services/tts/tts_service.py`; +- `routes/stt_routes.py`; +- `routes/tts_routes.py`; +- `src/upload_limits.py`; +- settings defaults/cache in `src/settings.py`; +- settings routes in `routes/auth_routes.py`; +- model endpoint cleanup in `routes/model_routes.py`; +- settings/tool aliases in `src/tool_implementations.py`; +- frontend modules `static/js/voiceRecorder.js`, `static/js/tts-ai.js`, `static/app.js`, `static/js/chat.js`, `static/js/slashCommands.js`, `static/js/keyboard-shortcuts.js`, `static/js/settings.js`, and `static/index.html`; +- optional dependency declarations in `requirements-optional.txt`; +- runtime cache path `data/tts_cache/`; +- tests covering speech service toggles, TTS speed/cache, STT temp cleanup, upload limits, settings scrubbing, and model endpoint cleanup. + +## Current Call Sites Include + +- chat mic/send button behavior; +- browser and server STT recording paths; +- chat message read-aloud buttons and streaming TTS queueing; +- `/tts` slash command playback; +- keyboard shortcut TTS activation; +- admin/settings API writes and `manage_settings` aliases; +- model endpoint deletion cleanup for `endpoint:` speech providers. + +## STT + +`services.stt.STTService` owns speech-to-text provider behavior. `routes/stt_routes.py` owns `/api/stt/transcribe` and `/api/stt/stats`. `static/js/voiceRecorder.js` owns microphone capture, browser STT, server upload, and audio-attachment fallback. + +Provider runtime: + +- `disabled` returns unavailable and avoids provider calls; +- `browser` is client-side only through Web Speech API and does not call `/api/stt/transcribe`; +- `local` lazily imports `faster-whisper`, writes uploaded audio to a temporary WebM file, transcribes, and deletes the temp file in `finally`; +- `endpoint:` resolves a `ModelEndpoint` and posts `audio.webm` to `/audio/transcriptions` with model and optional language. + +Route behavior: + +- audio uploads are capped by the shared STT upload limit from `src.upload_limits`, including environment override validation; +- empty uploads return a route error; +- uploaded content type, extension, and magic bytes are not strongly validated today; +- endpoint providers report optimistic availability and fail at request time if offline/misconfigured. + +Frontend behavior: + +- browser recording needs secure context and microphone permissions; +- server transcription success inserts text into the input; +- failed server transcription can attach the recorded audio file to chat instead; empty transcription shows a no-speech message. + +## TTS + +`services.tts.TTSService` owns text-to-speech provider behavior, speed parsing, cache behavior, and local/provider-specific synthesis. `routes/tts_routes.py` owns `/api/tts/stats`, `/api/tts/synthesize`, and cache clearing. `static/js/tts-ai.js` owns frontend playback, client object-URL caching, browser TTS, queueing, and streaming button state. + +Provider runtime: + +- `disabled` returns unavailable and avoids provider calls; +- `browser` is client-side only through `speechSynthesis`; +- `local` currently means Kokoro and requires `torch`, `kokoro`, and CUDA/import availability; +- `endpoint:` resolves a `ModelEndpoint` and posts to `/audio/speech`. +- unknown or non-string `tts_provider` values are treated as unavailable rather + than being parsed as endpoint strings. + +Route behavior: + +- `/api/tts/synthesize` supports binary `audio` responses and JSON `base64` responses; +- binary responses choose WAV or MP3 MIME by audio magic bytes; +- synthesis input is passed to the service as submitted and capped there; +- malformed or nonpositive `tts_speed` falls back to `1.0`; +- provider unavailable returns 503; failed synthesis/transcription generally returns route-level failure. + +## Settings, Endpoints, And Cache + +Speech providers are global settings under `data/settings.json`, with defaults in `src/settings.py`. Settings reads are scrubbed for non-admin callers, writes are admin-only, and `manage_settings` can change non-secret speech settings through aliases. + +Visible UI state is not complete: backend and JS speech settings exist, the TTS settings card is currently hidden, and the STT settings JS exits when its removed DOM nodes are absent. + +`routes.model_routes` clears `tts_provider` and `stt_provider` references when a referenced model endpoint is deleted. + +TTS cache behavior: + +- server cache lives under `data/tts_cache/`; +- cache keys include provider, model, voice, safe speed, and text; +- cache files are stored as MP3 or WAV; +- route stats expose global cache state; +- cache clear is global; +- frontend TTS has a separate object-URL cache. + +## Security And Provenance + +Speech routes rely on app-wide authentication and do not implement route-local admin or scope checks. Bearer-token callers that pass app auth can reach speech stats/synthesis/transcription/cache-clear surfaces using global speech settings. + +Endpoint providers send user audio or assistant text to configured `ModelEndpoint` URLs with optional bearer keys. Endpoint lookup is by configured endpoint ID and currently does not enforce per-request owner filtering. `ModelEndpoint.api_key` is encrypted at rest and forwarded only process-side. + +Microphone audio, uploaded audio, endpoint transcripts, and assistant text sent to TTS are untrusted/user/provider-visible data flows. Transcripts become user input; they are not trusted system instructions. + +TTS cached audio can contain sensitive assistant text rendered as speech. The cache is global, has no owner partition or TTL, and is served inline/base64 by POST responses without a dedicated generated-file route. + +## Degraded Behavior + +- Optional local speech packages may be absent. +- Local STT can run CPU-only and tolerates missing/broken torch by falling back to CPU/int8 behavior. +- Local TTS/Kokoro is unavailable without CUDA/imports. +- External endpoint providers can be offline or misconfigured and may only fail at request time. +- Browser `speechSynthesis`, `SpeechRecognition`, `webkitSpeechRecognition`, secure context, and microphone permissions can be absent. +- Docker GPU overlays are passthrough-only and do not install speech engines by themselves. +- Optional dependency errors and route error wording are not fully consistent across STT and TTS. + +## Testing Coverage + +Existing coverage includes `test_speech_service_toggles`, malformed/non-string TTS provider and speed handling, `test_tts_cache_stats`, `test_stt_leak`, `test_direct_upload_limits`, `test_model_routes`, and settings scrub coverage. A focused audit run of those relevant tests passed. + +Missing coverage includes route-level STT/TTS success and failure shapes, auth/API-token behavior, endpoint owner isolation, STT type/magic rejection, TTS request-size/no-store/cache privacy behavior, degraded optional dependency paths, and frontend recorder/TTS fallback states. + +## Current Gaps + +- Visible speech settings UI is incomplete relative to backend settings. +- Optional Kokoro dependencies are mentioned by service errors but are not declared in `requirements-optional.txt`. +- Speech routes need a deliberate API-token/scope policy. +- Endpoint speech providers need owner-isolation or explicit global-settings documentation. +- TTS cache needs privacy policy: owner partition, TTL, no-store response headers, or accepted global cache semantics. +- STT upload validation needs content type/extension/magic-byte policy. +- Browser/compare STT mic behavior needs a product decision or regression test because compare can force send-button visuals while shared empty-input logic can start recording. diff --git a/specs/testing-devops.md b/specs/testing-devops.md new file mode 100644 index 000000000..14970da03 --- /dev/null +++ b/specs/testing-devops.md @@ -0,0 +1,210 @@ +# Testing And Devops + +Last updated: dev@e57f60b | 2026-07-20 + +## Scope + +This spec covers development and validation surfaces in: + +- `tests/`, `tests/conftest.py`, `tests/*.mjs`, and `tests/bombadil-spec.ts`; +- `tests/run_focus.py`, `tests/run_order_report.py`, `tests/_taxonomy.py`, `tests/TESTING_STANDARD.md`, and `tests/LAYOUT_INVENTORY.md`; +- `pyproject.toml`; +- `requirements.txt` and `requirements-optional.txt`; +- `package.json` and `package-lock.json`; +- `Dockerfile`, `docker-compose.yml`, `docker/gpu.nvidia.yml`, `docker/gpu.amd.yml`, `docker/host-docker.yml`, top-level standalone GPU compose files, and `docker/entrypoint.sh`; +- `scripts/`, `scripts/odysseus`, `scripts/_lib/cli.py`, `scripts/_completion/*`, `scripts/pr_blocker_audit.py`, and `scripts/odysseus-*`; +- GPU helper scripts `scripts/check-docker-gpu.sh` and `scripts/check-docker-amd-gpu.sh`; +- `.github/` templates, workflows, and description-check scripts; +- contributor workflow docs in `CONTRIBUTING.md` and `docs/pr-blocker-audit.md`; +- platform launchers `launch-windows.ps1`, `launcher.py`, `Odysseus.spec`, `build-windows-portable.ps1`, `start-macos.sh`, `build-macos-app.sh`, and `update_windows.bat`; +- setup/service files such as `setup.py`, `install-service.sh`, and `odysseus-ui.service`. + +## Test Runtime + +Pytest is configured in `pyproject.toml` with: + +- `testpaths = ["tests"]`; +- `asyncio_mode = "auto"`; +- marker and fast-lane/duration-reporting settings used by focused test runs. + +The expected local command uses the project venv: + +```bash +./venv/bin/pytest +``` + +Activated-venv `python -m pytest ` is equivalent. System/global `pytest` is not authoritative for this repo because installed versus stubbed dependencies can change collection behavior. + +`tests/conftest.py` inserts the repo root on `sys.path` and conditionally stubs missing heavy/runtime dependencies such as SQLAlchemy, FastAPI, Starlette, Pydantic, httpx, bcrypt, and pyotp. Tests that need real dependencies use explicit imports/skips. Tests that stub `sys.modules`, environment variables, globals, or parent packages must restore them with `monkeypatch` or an equivalent cleanup pattern. + +The suite currently contains roughly 728 `test_*.py` files. Treat that count as a moving source metric, not a target; focused regression tests are still preferred for narrow changes. + +Focused regression tests are preferred for narrow behavior changes. Broaden tests when touching shared contracts such as auth, owner filtering, OAuth/token custody, tool output, context building, provider calls, persistence, frontend rendering, or route/API shapes. + +`tests/run_focus.py` and `tests/_taxonomy.py` provide a local focused-run helper and category map. `.github/scripts/focused_test_guidance.py` and its focused-test guidance checks map changed files to suggested focused tests for PR review, while keeping the full pytest job informational. `tests/TESTING_STANDARD.md` documents expectations for targeted validation, and `tests/LAYOUT_INVENTORY.md` records the test-suite layout. CLI tests live under `tests/cli/`. + +## JS And UI Tests + +The repo has no frontend build pipeline, npm test script, or type-check script. `package.json` owns Node dependencies for Bombadil and the Anthropic SDK, and `package-lock.json` owns npm integrity/version state. + +Current frontend/JS validation includes: + +- pytest wrappers that run Node snippets and usually skip when `node` is missing; +- direct `.mjs` regressions under `tests/`; +- `tests/bombadil-spec.ts`, which requires npm-installed Bombadil dev dependencies and a running/browser-capable UI workflow when used. + +Use `node --check static/js/.js` for syntax checks on changed JS files when applicable. This is not a full module-graph, browser-global, or DOM integration check. + +## Dependencies + +`requirements.txt` owns core runtime and test dependencies, including pytest, pytest-asyncio, MCP, Chroma HTTP client, fastembed, qrcode, and core parsing/search/calendar dependencies. + +`requirements-optional.txt` owns optional feature dependencies: + +- `faster-whisper` for local STT; +- `ddgs` for DDG library support, while provider code can fall back to HTML scraping; +- `PyMuPDF` for PDF forms/rendering with AGPL implications for a network-served app; +- `markitdown[docx,pptx,xlsx,xls]` for Office/EPUB extraction, pinned to a release older than 30 days. + +Optional dependencies should produce clear degraded behavior when absent unless intentionally promoted to core. MarkItDown and PyMuPDF already have focused degraded-path coverage; local STT missing-`faster-whisper` behavior is a remaining coverage gap. Core runtime requirements include `httpx2` where compatibility tests depend on it. The official Docker image additionally installs `libmagic1` plus `python-magic==0.4.27` for content-based upload MIME sniffing; that pairing is image-owned because `python-magic` needs the system shared library at import time. + +Chroma has two compatibility modes: + +- Docker uses a separate `chromadb` service and core `chromadb-client`/`fastembed`; +- native macOS setup removes conflicting `chromadb-client` and installs full `chromadb`. + +Vector features should fail fast or degrade to unhealthy/keyword fallback when the service is unavailable. + +## Docker Runtime + +Docker Compose is the primary deployment path: + +```bash +docker compose up -d --build +docker compose ps +docker compose logs --tail=120 odysseus +``` + +`docker-compose.yml` starts Odysseus, ChromaDB, SearXNG, and ntfy. It binds services to loopback by default through `APP_BIND`, `CHROMADB_BIND`, and `NTFY_BIND`, persists configurable `APP_DATA_DIR`/`APP_LOGS_DIR`, SSH identity, HuggingFace cache, and user-local Python installs, and gives the Odysseus container host-loopback reachability through `host.docker.internal`. + +`Dockerfile` builds a Python 3.14 slim image with Node/npm, tmux, OpenSSH client, git/cmake, the pinned Docker CLI `29.6.2`, `gosu`, `libmagic1`, and the image-only `python-magic` wrapper. + +`docker/entrypoint.sh` owns writable path ownership repair, PUID/PGID user/group creation and privilege drop, optional host-Docker socket group handling, vLLM/CUDA environment defaults, idempotent `setup.py`, and final uvicorn execution. + +Docker does not mount the host Docker socket by default. Mounting it would grant powerful host access and is outside the default trust boundary. `docker/host-docker.yml` is the explicit opt-in overlay and sets `ODYSSEUS_ENABLE_HOST_DOCKER=true`; tests guard that the default and GPU compose files do not enable host Docker accidentally. + +## GPU And Platform + +Base `docker-compose.yml` plus `docker/gpu.nvidia.yml` or `docker/gpu.amd.yml` are the GPU source of truth. Top-level `docker-compose.gpu-nvidia.yml` and `docker-compose.gpu-amd.yml` are standalone mirrors for stack-management UIs that accept one compose file. `tests/test_gpu_compose_standalone.py` guards drift between those forms. + +GPU overlays pass host devices/runtime flags only. They do not install CUDA/ROCm userspace or serving engines; those are installed later through Cookbook/dependency flows. + +NVIDIA helper behavior: + +- `scripts/check-docker-gpu.sh` diagnoses passthrough; +- it is read-only by default; +- toolkit install and `.env` edits require explicit user flags and successful passthrough checks. + +AMD helper behavior: + +- `scripts/check-docker-amd-gpu.sh` is read-only; +- it prints expected `COMPOSE_FILE`/`RENDER_GID` values and verifies `/dev/kfd`/`/dev/dri` visibility. + +Native platform launchers: + +- `launch-windows.ps1` requires Python 3.11+, creates `venv`, installs `requirements.txt`, runs `setup.py`, discovers per-user Git Bash installs where possible, warns when Git Bash is missing, and starts uvicorn on port 7000 by default. +- `launcher.py`, `Odysseus.spec`, and `build-windows-portable.ps1` own the PyInstaller-style portable Windows launcher path, including app-root/data-dir differences covered by `src.runtime_paths`. +- `start-macos.sh` reads `.env`, defaults to port 7860 to avoid AirPlay conflicts, prefers Homebrew arm64 Python, installs/tolerates Homebrew Cookbook deps, handles Chroma package conflicts, starts ChromaDB for native runs, runs `setup.py`, and starts uvicorn. +- `build-macos-app.sh` builds a launcher app around the existing repo venv and logs to `logs/odysseus-app.log`. +- `update_windows.bat` owns the tested Windows Docker update flow. + +## Scripts And CLI + +`scripts/odysseus` is the umbrella dispatcher for executable `scripts/odysseus-*` commands. It discovers subcommands and executes them through the project venv Python when available. + +`scripts/_lib/cli.py` owns shared CLI behavior: + +- repo-root importability; +- quiet logging; +- JSON output and `--pretty`; +- `--version`; +- common parser scaffolding; +- exit handling. + +`LOG_LEVEL` is the shared process logging toggle. CLI helpers default it to +`WARNING` to keep JSON command output clean; the web app defaults it to `INFO` +and applies it to root, console, rotating-file, and direct-uvicorn logging. +Shell completions in `scripts/_completion/` introspect CLI `--help` output through the venv and cache subcommands. + +`scripts/odysseus-*` provide local CLI surfaces for backup, calendar, contacts, Cookbook, docs, gallery, logs, mail, MCP, memory, notes, personal docs, presets, research, sessions, signatures, skills, tasks, theme, and webhooks. + +When route/API behavior changes, check whether a matching CLI script depends on the old shape. There is no central CLI scrubber: each credential/log/mail/task/backup/MCP/webhook script owns its own sensitive-output behavior. + +## GitHub Metadata + +`.github/` owns issue/PR templates, a copyable PR review template, description-check workflows, security/governance workflows, Docker publishing, and a lightweight CI workflow. Current CI compiles Python with `python -m compileall`, syntax-checks first-party JS with `node --check`, emits focused-test guidance for changed code, and runs `python -m pytest -q` as an informational/non-blocking job; the pytest job skips documentation-only changes. + +`CONTRIBUTING.md` owns the branch model: PRs target `dev`; `main` is the curated user-running branch fast-forwarded from stable `dev` commits. Contributors who accidentally target `main` should retarget the PR base without rebasing. + +PR description checks: + +- run on `pull_request_target`; +- check out only base-branch `.github/scripts`; +- skip bot PRs; +- require Summary, Linked Issue, Type of Change, duplicate-search checklist, and substantive How to Test content; +- update a bot comment and swap `ready for review` / `needs work` labels. + +Issue description checks: + +- validate bug or feature sections based on labels; +- flag unfilled dropdown placeholders such as `-- Please Select --`; +- route public vulnerability reports toward GitHub Security Advisories; +- update a bot comment and swap status labels. + +Security metadata includes container Trivy SARIF upload, Dockerfile lint, dependency review, secret scan, workflow security linting, GitHub default-setup CodeQL, Dependabot metadata, and hardened PR/issue description checks that avoid unsafe head-branch execution. `docs/security-ci.md` documents CodeQL as a dynamic GitHub default-setup workflow; the repo should not add a checked-in CodeQL workflow while that default setup is active. + +`scripts/pr_blocker_audit.py` is a read-only maintainer/contributor triage helper documented in `docs/pr-blocker-audit.md`. It can fetch or ingest open PR metadata, estimate hot files and possible duplicate groups, and emit Markdown, JSON, or terminal reports. Its duplicate/blocker output is advisory, not an authority that a PR is blocked. + +Before posting PRs or issues, compare drafts against current templates on latest `main` or current `dev` as appropriate for the target. Keep unpublished drafts and raw related-search exports out of tracked implementation specs unless intentionally promoted. + +## Artifacts And Secrets + +- Do not read `.env*` files unless a user explicitly asks for a controlled setup/debug step; never print their values. +- Backup files, logs, CLI JSON, and raw issue/PR search exports can contain sensitive local data. +- Do not commit raw GitHub JSON unless there is an explicit maintainer reason. Prefer compact Markdown reports when publishing analysis. +- Specs are implementation truth. Planning, research, branch notes, and draft reports belong in tracked project docs when promoted. + +## Development Checks + +Common local checks: + +```bash +./venv/bin/pytest tests/path.py::test_name +./venv/bin/python -m py_compile app.py routes/*.py src/*.py +node --check static/js/changed-file.js +docker compose config +docker compose up -d --build +docker compose logs --tail=120 odysseus +``` + +Run the app for user-facing or integration changes. Unit tests and syntax checks do not replace end-to-end verification for UI, Docker, provider, auth, or routing behavior. + +## Shared Test Helpers + +`tests/helpers/` owns reusable test scaffolding. `cli_loader.load_script()` loads CLI files without running their `main()` entrypoint. `db_stubs` owns small DB stand-ins for tests that should not import a real app database. `import_state` owns conservative `sys.modules` and parent-module-attribute restoration for tests that install fake modules or import route files under alternate stubs. `tests/README.md` documents helper conventions and review expectations. + +## Current Gaps + +- Fresh install smoke coverage across Linux native, Docker, macOS native/app, Windows native, WSL/Git Bash, missing Node/npm, missing Chroma service, and GPU overlays remains a roadmap item. +- There is no frontend build/type-check/npm test pipeline. +- CI now covers Python compile, first-party JS syntax, focused-test guidance, + and pytest smoke; it does not cover Docker compose validation, launcher smoke + tests, browser/module-graph execution, or platform installs. +- Optional dependency behavior is broad; remaining gaps include local STT missing-`faster-whisper` behavior and provider/OAuth combinations not covered by focused tests. +- GitHub description-check scripts and `scripts/pr_blocker_audit.py` need continued local fixtures for section parsing, placeholder stripping, label swaps, workflow-safe behavior, and duplicate/hot-file heuristics. +- Spec bootstrap rules lack meta tests for reading `_readme.md`, spec shape, `.env*` handling, draft/report placement, and shared helper conventions. +- NVIDIA helper install/`.env` mutation paths and real Docker/GPU startup are not covered by local tests. +- Bash/Zsh completion behavior is not covered. +- There is no canonical full-suite known-failing/flaky ledger. +- There is no central CLI redaction/sensitive-output regression matrix across backup, logs, mail, MCP, tasks, and webhook scripts. +- Dependency/image pinning policy is mixed: Python requirements are mostly unpinned, SearXNG is pinned, Chroma image currently uses `latest`, npm uses a lockfile, and browser MCP uses cache-gated `@playwright/mcp@latest`.