odysseus/specs/testing-devops.md

# Testing And Devops

Last updated: dev@e57f60b | 2026-07-20

## Scope

This spec covers development and validation surfaces in:

- `tests/`, `tests/conftest.py`, `tests/*.mjs`, and `tests/bombadil-spec.ts`;
- `tests/run_focus.py`, `tests/run_order_report.py`, `tests/_taxonomy.py`, `tests/TESTING_STANDARD.md`, and `tests/LAYOUT_INVENTORY.md`;
- `pyproject.toml`;
- `requirements.txt` and `requirements-optional.txt`;
- `package.json` and `package-lock.json`;
- `Dockerfile`, `docker-compose.yml`, `docker/gpu.nvidia.yml`, `docker/gpu.amd.yml`, `docker/host-docker.yml`, top-level standalone GPU compose files, and `docker/entrypoint.sh`;
- `scripts/`, `scripts/odysseus`, `scripts/_lib/cli.py`, `scripts/_completion/*`, `scripts/pr_blocker_audit.py`, and `scripts/odysseus-*`;
- GPU helper scripts `scripts/check-docker-gpu.sh` and `scripts/check-docker-amd-gpu.sh`;
- `.github/` templates, workflows, and description-check scripts;
- contributor workflow docs in `CONTRIBUTING.md` and `docs/pr-blocker-audit.md`;
- platform launchers `launch-windows.ps1`, `launcher.py`, `Odysseus.spec`, `build-windows-portable.ps1`, `start-macos.sh`, `build-macos-app.sh`, and `update_windows.bat`;
- setup/service files such as `setup.py`, `install-service.sh`, and `odysseus-ui.service`.

## Test Runtime

Pytest is configured in `pyproject.toml` with:

- `testpaths = ["tests"]`;
- `asyncio_mode = "auto"`;
- marker and fast-lane/duration-reporting settings used by focused test runs.

The expected local command uses the project venv:

```bash
./venv/bin/pytest <test path>
```

Activated-venv `python -m pytest <test path>` is equivalent. System/global `pytest` is not authoritative for this repo because installed versus stubbed dependencies can change collection behavior.

`tests/conftest.py` inserts the repo root on `sys.path` and conditionally stubs missing heavy/runtime dependencies such as SQLAlchemy, FastAPI, Starlette, Pydantic, httpx, bcrypt, and pyotp. Tests that need real dependencies use explicit imports/skips. Tests that stub `sys.modules`, environment variables, globals, or parent packages must restore them with `monkeypatch` or an equivalent cleanup pattern.

The suite currently contains roughly 728 `test_*.py` files. Treat that count as a moving source metric, not a target; focused regression tests are still preferred for narrow changes.

Focused regression tests are preferred for narrow behavior changes. Broaden tests when touching shared contracts such as auth, owner filtering, OAuth/token custody, tool output, context building, provider calls, persistence, frontend rendering, or route/API shapes.

`tests/run_focus.py` and `tests/_taxonomy.py` provide a local focused-run helper and category map. `.github/scripts/focused_test_guidance.py` and its focused-test guidance checks map changed files to suggested focused tests for PR review, while keeping the full pytest job informational. `tests/TESTING_STANDARD.md` documents expectations for targeted validation, and `tests/LAYOUT_INVENTORY.md` records the test-suite layout. CLI tests live under `tests/cli/`.

## JS And UI Tests

The repo has no frontend build pipeline, npm test script, or type-check script. `package.json` owns Node dependencies for Bombadil and the Anthropic SDK, and `package-lock.json` owns npm integrity/version state.

Current frontend/JS validation includes:

- pytest wrappers that run Node snippets and usually skip when `node` is missing;
- direct `.mjs` regressions under `tests/`;
- `tests/bombadil-spec.ts`, which requires npm-installed Bombadil dev dependencies and a running/browser-capable UI workflow when used.

Use `node --check static/js/<changed-file>.js` for syntax checks on changed JS files when applicable. This is not a full module-graph, browser-global, or DOM integration check.

## Dependencies

`requirements.txt` owns core runtime and test dependencies, including pytest, pytest-asyncio, MCP, Chroma HTTP client, fastembed, qrcode, and core parsing/search/calendar dependencies.

`requirements-optional.txt` owns optional feature dependencies:

- `faster-whisper` for local STT;
- `ddgs` for DDG library support, while provider code can fall back to HTML scraping;
- `PyMuPDF` for PDF forms/rendering with AGPL implications for a network-served app;
- `markitdown[docx,pptx,xlsx,xls]` for Office/EPUB extraction, pinned to a release older than 30 days.

Optional dependencies should produce clear degraded behavior when absent unless intentionally promoted to core. MarkItDown and PyMuPDF already have focused degraded-path coverage; local STT missing-`faster-whisper` behavior is a remaining coverage gap. Core runtime requirements include `httpx2` where compatibility tests depend on it. The official Docker image additionally installs `libmagic1` plus `python-magic==0.4.27` for content-based upload MIME sniffing; that pairing is image-owned because `python-magic` needs the system shared library at import time.

Chroma has two compatibility modes:

- Docker uses a separate `chromadb` service and core `chromadb-client`/`fastembed`;
- native macOS setup removes conflicting `chromadb-client` and installs full `chromadb`.

Vector features should fail fast or degrade to unhealthy/keyword fallback when the service is unavailable.

## Docker Runtime

Docker Compose is the primary deployment path:

```bash
docker compose up -d --build
docker compose ps
docker compose logs --tail=120 odysseus
```

`docker-compose.yml` starts Odysseus, ChromaDB, SearXNG, and ntfy. It binds services to loopback by default through `APP_BIND`, `CHROMADB_BIND`, and `NTFY_BIND`, persists configurable `APP_DATA_DIR`/`APP_LOGS_DIR`, SSH identity, HuggingFace cache, and user-local Python installs, and gives the Odysseus container host-loopback reachability through `host.docker.internal`.

`Dockerfile` builds a Python 3.14 slim image with Node/npm, tmux, OpenSSH client, git/cmake, the pinned Docker CLI `29.6.2`, `gosu`, `libmagic1`, and the image-only `python-magic` wrapper.

`docker/entrypoint.sh` owns writable path ownership repair, PUID/PGID user/group creation and privilege drop, optional host-Docker socket group handling, vLLM/CUDA environment defaults, idempotent `setup.py`, and final uvicorn execution.

Docker does not mount the host Docker socket by default. Mounting it would grant powerful host access and is outside the default trust boundary. `docker/host-docker.yml` is the explicit opt-in overlay and sets `ODYSSEUS_ENABLE_HOST_DOCKER=true`; tests guard that the default and GPU compose files do not enable host Docker accidentally.

## GPU And Platform

Base `docker-compose.yml` plus `docker/gpu.nvidia.yml` or `docker/gpu.amd.yml` are the GPU source of truth. Top-level `docker-compose.gpu-nvidia.yml` and `docker-compose.gpu-amd.yml` are standalone mirrors for stack-management UIs that accept one compose file. `tests/test_gpu_compose_standalone.py` guards drift between those forms.

GPU overlays pass host devices/runtime flags only. They do not install CUDA/ROCm userspace or serving engines; those are installed later through Cookbook/dependency flows.

NVIDIA helper behavior:

- `scripts/check-docker-gpu.sh` diagnoses passthrough;
- it is read-only by default;
- toolkit install and `.env` edits require explicit user flags and successful passthrough checks.

AMD helper behavior:

- `scripts/check-docker-amd-gpu.sh` is read-only;
- it prints expected `COMPOSE_FILE`/`RENDER_GID` values and verifies `/dev/kfd`/`/dev/dri` visibility.

Native platform launchers:

- `launch-windows.ps1` requires Python 3.11+, creates `venv`, installs `requirements.txt`, runs `setup.py`, discovers per-user Git Bash installs where possible, warns when Git Bash is missing, and starts uvicorn on port 7000 by default.
- `launcher.py`, `Odysseus.spec`, and `build-windows-portable.ps1` own the PyInstaller-style portable Windows launcher path, including app-root/data-dir differences covered by `src.runtime_paths`.
- `start-macos.sh` reads `.env`, defaults to port 7860 to avoid AirPlay conflicts, prefers Homebrew arm64 Python, installs/tolerates Homebrew Cookbook deps, handles Chroma package conflicts, starts ChromaDB for native runs, runs `setup.py`, and starts uvicorn.
- `build-macos-app.sh` builds a launcher app around the existing repo venv and logs to `logs/odysseus-app.log`.
- `update_windows.bat` owns the tested Windows Docker update flow.

## Scripts And CLI

`scripts/odysseus` is the umbrella dispatcher for executable `scripts/odysseus-*` commands. It discovers subcommands and executes them through the project venv Python when available.

`scripts/_lib/cli.py` owns shared CLI behavior:

- repo-root importability;
- quiet logging;
- JSON output and `--pretty`;
- `--version`;
- common parser scaffolding;
- exit handling.

`LOG_LEVEL` is the shared process logging toggle. CLI helpers default it to
`WARNING` to keep JSON command output clean; the web app defaults it to `INFO`
and applies it to root, console, rotating-file, and direct-uvicorn logging.
Shell completions in `scripts/_completion/` introspect CLI `--help` output through the venv and cache subcommands.

`scripts/odysseus-*` provide local CLI surfaces for backup, calendar, contacts, Cookbook, docs, gallery, logs, mail, MCP, memory, notes, personal docs, presets, research, sessions, signatures, skills, tasks, theme, and webhooks.

When route/API behavior changes, check whether a matching CLI script depends on the old shape. There is no central CLI scrubber: each credential/log/mail/task/backup/MCP/webhook script owns its own sensitive-output behavior.

## GitHub Metadata

`.github/` owns issue/PR templates, a copyable PR review template, description-check workflows, security/governance workflows, Docker publishing, and a lightweight CI workflow. Current CI compiles Python with `python -m compileall`, syntax-checks first-party JS with `node --check`, emits focused-test guidance for changed code, and runs `python -m pytest -q` as an informational/non-blocking job; the pytest job skips documentation-only changes.

`CONTRIBUTING.md` owns the branch model: PRs target `dev`; `main` is the curated user-running branch fast-forwarded from stable `dev` commits. Contributors who accidentally target `main` should retarget the PR base without rebasing.

PR description checks:

- run on `pull_request_target`;
- check out only base-branch `.github/scripts`;
- skip bot PRs;
- require Summary, Linked Issue, Type of Change, duplicate-search checklist, and substantive How to Test content;
- update a bot comment and swap `ready for review` / `needs work` labels.

Issue description checks:

- validate bug or feature sections based on labels;
- flag unfilled dropdown placeholders such as `-- Please Select --`;
- route public vulnerability reports toward GitHub Security Advisories;
- update a bot comment and swap status labels.

Security metadata includes container Trivy SARIF upload, Dockerfile lint, dependency review, secret scan, workflow security linting, GitHub default-setup CodeQL, Dependabot metadata, and hardened PR/issue description checks that avoid unsafe head-branch execution. `docs/security-ci.md` documents CodeQL as a dynamic GitHub default-setup workflow; the repo should not add a checked-in CodeQL workflow while that default setup is active.

`scripts/pr_blocker_audit.py` is a read-only maintainer/contributor triage helper documented in `docs/pr-blocker-audit.md`. It can fetch or ingest open PR metadata, estimate hot files and possible duplicate groups, and emit Markdown, JSON, or terminal reports. Its duplicate/blocker output is advisory, not an authority that a PR is blocked.

Before posting PRs or issues, compare drafts against current templates on latest `main` or current `dev` as appropriate for the target. Keep unpublished drafts and raw related-search exports out of tracked implementation specs unless intentionally promoted.

## Artifacts And Secrets

- Do not read `.env*` files unless a user explicitly asks for a controlled setup/debug step; never print their values.
- Backup files, logs, CLI JSON, and raw issue/PR search exports can contain sensitive local data.
- Do not commit raw GitHub JSON unless there is an explicit maintainer reason. Prefer compact Markdown reports when publishing analysis.
- Specs are implementation truth. Planning, research, branch notes, and draft reports belong in tracked project docs when promoted.

## Development Checks

Common local checks:

```bash
./venv/bin/pytest tests/path.py::test_name
./venv/bin/python -m py_compile app.py routes/*.py src/*.py
node --check static/js/changed-file.js
docker compose config
docker compose up -d --build
docker compose logs --tail=120 odysseus
```

Run the app for user-facing or integration changes. Unit tests and syntax checks do not replace end-to-end verification for UI, Docker, provider, auth, or routing behavior.

## Shared Test Helpers

`tests/helpers/` owns reusable test scaffolding. `cli_loader.load_script()` loads CLI files without running their `main()` entrypoint. `db_stubs` owns small DB stand-ins for tests that should not import a real app database. `import_state` owns conservative `sys.modules` and parent-module-attribute restoration for tests that install fake modules or import route files under alternate stubs. `tests/README.md` documents helper conventions and review expectations.

## Current Gaps

- Fresh install smoke coverage across Linux native, Docker, macOS native/app, Windows native, WSL/Git Bash, missing Node/npm, missing Chroma service, and GPU overlays remains a roadmap item.
- There is no frontend build/type-check/npm test pipeline.
- CI now covers Python compile, first-party JS syntax, focused-test guidance,
  and pytest smoke; it does not cover Docker compose validation, launcher smoke
  tests, browser/module-graph execution, or platform installs.
- Optional dependency behavior is broad; remaining gaps include local STT missing-`faster-whisper` behavior and provider/OAuth combinations not covered by focused tests.
- GitHub description-check scripts and `scripts/pr_blocker_audit.py` need continued local fixtures for section parsing, placeholder stripping, label swaps, workflow-safe behavior, and duplicate/hot-file heuristics.
- Spec bootstrap rules lack meta tests for reading `_readme.md`, spec shape, `.env*` handling, draft/report placement, and shared helper conventions.
- NVIDIA helper install/`.env` mutation paths and real Docker/GPU startup are not covered by local tests.
- Bash/Zsh completion behavior is not covered.
- There is no canonical full-suite known-failing/flaky ledger.
- There is no central CLI redaction/sensitive-output regression matrix across backup, logs, mail, MCP, tasks, and webhook scripts.
- Dependency/image pinning policy is mixed: Python requirements are mostly unpinned, SearXNG is pinned, Chroma image currently uses `latest`, npm uses a lockfile, and browser MCP uses cache-gated `@playwright/mcp@latest`.