odysseus/specs/shell-mcp.md

# Shell And MCP

Last updated: dev@28d27ee | 2026-07-17

## Scope

This spec covers shell and MCP behavior in:

- shell routes in `routes/shell_routes.py`;
- the standalone shell helper in `services/shell/service.py`;
- agent shell/background execution in `src/tool_execution.py`, `src/agent_tools/subprocess_tools.py`, `src/bg_jobs.py`, and `src/bg_monitor.py`;
- app wiring and startup/shutdown in `app.py`;
- MCP configuration routes in `routes/mcp_routes.py`;
- MCP runtime state in `src/mcp_manager.py`;
- generic MCP OAuth helpers in `src/mcp_oauth.py`;
- built-in server registration in `src/builtin_mcp.py`;
- persisted `McpServer` config in `core/database.py`;
- MCP tool exposure in `src/agent_loop.py`, `src/tool_index.py`, `src/tool_schemas.py`, `src/tool_parsing.py`, `src/tool_implementations.py`, and `src/tool_security.py`;
- admin MCP/tool helpers in `src/agent_tools/admin_tools.py`;
- built-in servers in `mcp_servers/*.py`;
- Settings/Admin UI in `static/js/settings.js` and `static/js/admin.js`;
- CLI helper `scripts/odysseus-mcp`;
- Docker/native dependency context in `Dockerfile` and `docker-compose.yml`.

Cookbook model-serving shell flows are covered in `cookbook-hwfit.md`; this spec owns the shared shell and MCP surfaces they reuse.

## Shell Routes

`routes.shell_routes` owns `/api/shell/exec` and `/api/shell/stream`. These routes are powerful by design and are admin-only. They execute admin-provided command strings through the host shell.

Runtime behavior:

- `/api/shell/exec` runs a bounded command and returns stdout, stderr, and exit code;
- `/api/shell/stream` streams SSE output through plain pipes, POSIX PTY, POSIX tmux log tailing, or a Windows detached-log fallback depending on request flags and platform;
- empty commands return an error result without spawning a shell;
- timeouts kill the subprocess where possible;
- disconnects can stop streaming subprocesses;
- POSIX PTY support is optional and reports an unsupported event when unavailable.

`routes.shell_routes` also owns shell-adjacent Cookbook dependency endpoints:

- `/api/cookbook/packages`;
- `/api/cookbook/packages/install`;
- `/api/cookbook/rebuild-engine`.

Those endpoints probe local or SSH-remote packages, prepend user install bins for pip CLIs, validate SSH host/port through shared route validators, validate remote venv values, and restrict package installs to allowlisted dependencies.

`services.shell.service.ShellService` is a small standalone subprocess abstraction with output caps. It does not own live route behavior, PTY/tmux paths, Windows shell selection, admin checks, or Cookbook package probes.

## Agent Shell And Background Jobs

`src.tool_execution` owns agent-side `bash` execution and the `#!bg` marker. A `bash` block whose first line is `#!bg` starts a detached background job instead of holding the chat stream open.

`src.bg_jobs` owns disk-backed job state under `data/bg_jobs.json` and `data/bg_jobs/*`. It stores wrapper scripts, logs, exit-code files, timestamps, status, and capped result text.

`src.bg_monitor` owns polling and auto-continuation. When a job finishes, it injects the job result into the session, drains the agent stream, persists only the assistant continuation plus `bg_result` metadata, and marks the job followed up.

Runtime behavior:

- background jobs are restart-tolerant while their state files remain;
- jobs have a maximum runtime and stale cleanup window;
- output is capped with head/tail retention;
- active sessions can defer follow-up until the next monitor pass.

## Configured MCP Servers

`routes.mcp_routes` owns admin HTTP configuration for MCP servers:

- list/add/reconnect/enable/disable/delete servers;
- list tools and per-server tools;
- update per-server disabled tool lists;
- Google OAuth authorize/callback/manual exchange pages and generic Streamable HTTP OAuth redirect handling.

`core.database.McpServer` persists transport, command, args, env, URL, enabled state, OAuth config, disabled tool names, and encrypted generic OAuth token/client state. `McpServer.env` is plaintext JSON in the database.

`src.mcp_manager.McpManager` owns live connection state, stdio/SSE/Streamable HTTP transports, sessions, tool schemas, qualified names, and tool calls. HTTP route operations update both database state and live manager state where applicable. Streamable HTTP connects in a background task, can report `connecting` or `needs_auth`, and surfaces an authorization URL when the OAuth client flow redirects. Enabled configured servers connect concurrently at startup; each server has its own 20-second connection timeout and records `timeout` state without delaying siblings. The startup task has no second outer timeout.

Stdio and SSE connection setup registers the session, exit stack, tool list,
and status as one completed unit. If initialization or tool discovery fails
before registration, the partial `AsyncExitStack` is closed so transports do
not leak into later reconnect attempts.

`src.agent_tools.admin_tools.do_manage_mcp()` is the agent/admin tool path for MCP config and is re-exported lazily through `src.tool_implementations` for compatibility. It is narrower than the HTTP routes: add is stdio-only, command values are checked against an allowlist/denylist before persistence, and enable/disable primarily flips DB config. `scripts/odysseus-mcp` is config-only; it reads and mutates database rows, redacts env values by default, and does not report live manager connection state.

## Built-In MCP Servers

`src.builtin_mcp` owns startup registration of built-in MCP servers unless `ODYSSEUS_DISABLE_MCP` is enabled.

Python stdio built-ins:

- image generation;
- memory;
- RAG;
- email.

The optional browser built-in uses `npx -y @playwright/mcp@latest --headless --caps vision`. It is cache-gated by checking npm's `_npx` cache for the requested package and falling back to `npx --no-install`; uncached/missing browser MCP is logged with install guidance and skipped rather than blocking startup or downloading packages at boot. Python built-ins are omitted from OpenAI function schemas because native/code-block paths already describe those capabilities; the browser built-in is exposed through MCP function schemas when connected.

Built-in Python servers prepend the app root to inherited `PYTHONPATH` rather
than replacing the environment, so container/dev site-packages remain visible
on initial connect and automatic reconnect. They can be reconnected once on
tool-call failure. User-configured MCP servers return the call failure instead
of automatic reconnect.

The built-in email MCP server is owner-aware when an owner is supplied by the
caller or configured through `ODYSSEUS_MCP_EMAIL_OWNER` /
`ODYSSEUS_EMAIL_OWNER`; if owner-scoped email accounts exist and no owner is
available, email MCP fails closed instead of exposing global accounts. Other
built-in servers remain process-global/admin trust-boundary tools unless their
own subsystem spec says otherwise.

## Agent MCP Exposure

`McpManager` owns raw qualified tool calls named `mcp__{server_id}__{tool_name}`. It does not own admin, owner, public-user, or disabled-tool policy; callers must enforce policy before dispatch.

Current exposure path:

- `routes.mcp_routes` stores disabled tool names;
- `src.agent_loop` loads disabled maps for prompts/schemas;
- `McpManager.get_all_openai_schemas()` and prompt descriptions filter disabled tools;
- `src.tool_index` indexes MCP prompt descriptions by manager generation;
- `src.tool_security` blocks all `mcp__*` tools for non-admin/public users;
- `src.tool_execution` dispatches received `mcp__*` calls to `McpManager.call_tool()`.

Per-server disabled MCP tools currently hide tools from prompts/schemas while listings still return tools with disabled metadata. They are not a complete execution-time gate if a disabled qualified name reaches tool execution. Plan mode additionally asks `McpManager.plan_mode_blocked_mcp()` to hide write/unknown MCP tools and add qualified names to the runtime disabled set for that turn.

## Degraded And Platform Behavior

- `app.py` starts the background monitor and MCP startup tasks asynchronously; MCP startup is non-critical to app readiness.
- Configured MCP servers start concurrently with a per-server 20-second bound;
  timeout state is stored per server and partial connection resources are
  closed before returning.
- Missing Python `mcp` dependency degrades attempted MCP connections to error status.
- Missing or uncached browser NPX package is optional and log-only during built-in startup; startup should not perform an implicit package download.
- Windows does not support POSIX PTY/tmux paths; streaming falls back to pipes or detached logfile behavior.
- Docker images include selected shell dependencies and the Docker CLI, but host Docker socket access from inside the app container remains unavailable unless the operator explicitly enables `docker/host-docker.yml`/`ODYSSEUS_ENABLE_HOST_DOCKER=true` and mounts a real socket.
- OAuth supports Google `installed` or `web` key shapes, a remote paste-back exchange page, and generic Streamable HTTP OAuth token storage through encrypted `McpServer.oauth_tokens`. Valid JSON values that are not objects are treated as empty token state and replaced by an object on the next write. Google and generic MCP OAuth share `src.mcp_oauth.REDIRECT_URI`, built from `OAUTH_REDIRECT_BASE_URL`, `APP_PUBLIC_URL`, or `http://localhost:7000` plus `/api/mcp/oauth/callback`; `APP_PORT` is intentionally not part of this redirect calculation.
- `services.shell.service` remains a transitional/simple facade separate from route-level compatibility behavior.

## Security And Provenance

- Admin shell is intentional host command execution; do not expose shell routes or shell tools to regular users.
- `_require_admin()` gates shell routes and MCP config routes. The internal-tool loopback can be admin-equivalent only after auth middleware validates the internal token and loopback client.
- `_reject_cross_site()` currently applies to `/api/cookbook/packages`; `/api/shell/exec`, `/api/shell/stream`, package install, rebuild, and MCP write/OAuth routes do not call it directly.
- Shell helper paths use argv-based SSH, reject option-like hosts, validate SSH ports through shared helpers, restrict remote venv characters, and allowlist package installs.
- Non-admin/public tool policy blocks `bash`, `python`, file tools, `manage_mcp`, and all `mcp__*` tools.
- MCP stdio server registration is arbitrary host process execution and is admin-only.
- MCP OAuth key/token file paths supplied through routes are confined under `data/mcp_oauth`; generic Streamable HTTP OAuth token state is encrypted in the database.
- Built-in MCP servers are local/admin trust-boundary tools and are not
  automatically equivalent to owner-scoped HTTP route behavior. Email MCP is
  the current exception with explicit owner filtering; other built-ins need
  their own owner policy before being treated as scoped surfaces.
- MCP output is untrusted tool output. Current MCP text output is not centrally capped before model re-entry.

## Testing Notes

Current targeted coverage includes Windows PTY import degradation, PTY unsupported stream events, the cross-site helper, `ShellService` stream deadline behavior, background store/monitor basics, concurrent MCP startup, per-server timeout isolation and cleanup, MCP manager cache/reconnect args, built-in `PYTHONPATH` preservation, non-object generic OAuth-token storage recovery, MCP CLI JSON/env serialization, MCP common truncation helper, action intent shell verbs, and public blocked-tool fail-closed behavior.

The shell/MCP audit ran the targeted venv subset with 78 passing tests and one warning.

## Current Gaps

- Decide whether `/api/shell/exec`, `/api/shell/stream`, package install, rebuild, and MCP config/OAuth writes should call `_reject_cross_site()` directly.
- Add route-level shell exec/stream tests for admin gate, cross-site behavior, empty command, plain exec, timeout, PTY, tmux, and Windows detached fallback.
- Add background job tests for launch isolation, output truncation, done/failed/timeout/died states, pending follow-ups, and result text.
- Add route-level MCP CRUD/OAuth/disabled-tool tests with a fake manager and temp database.
- Add hard per-server disabled MCP execution checks or document disabled tools as prompt/schema filtering only.
- Make MCP tool indexing sensitive to disabled-map changes, not only manager generation.
- Fix stale outer prompt/cache behavior when MCP disabled tools change.
- Add central truncation and untrusted-context wrapping for MCP result text and images before model re-entry.
- Decide whether `McpServer.env` and OAuth key files need masking, encryption, and chmod beyond admin-only access.
- Decide whether built-in MCP servers should become owner-aware or remain documented as admin/global compatibility surfaces.
- Decide whether optional browser MCP cache misses should surface in `/api/mcp` status instead of startup logs only.