odysseus/specs/cookbook-hwfit.md

Cookbook And Hardware Fit

Last updated: dev@df2fad2 | 2026-07-12

Scope

This spec covers model setup/serving and hardware fit in:

• app route registration in app.py;
• routes/cookbook_routes.py;
• src/cookbook_serve_lifecycle.py;
• src/host_docker_access.py;
• Cookbook package/rebuild/shell integration in routes/shell_routes.py;
• routes/cookbook_helpers.py;
• routes/hwfit_routes.py;
• services/hwfit/* and services/hwfit/data/hf_models.json;
• durable Cookbook state through routes.cookbook_helpers.COOKBOOK_STATE_FILE;
• helper/CLI scripts scripts/odysseus-cookbook, scripts/add_hwfit_models.py, scripts/hf_download.py, and scripts/diffusion_server.py;
• Docker overlays docker-compose.gpu-*.yml, docker/gpu.*.yml, docker/host-docker.yml, scripts/check-docker-gpu.sh, and scripts/check-docker-amd-gpu.sh;
• frontend modules static/js/cookbook*.js, including Cookbook running, serve, download, diagnosis, progress, and HW Fit modules;
• tests covering Cookbook helpers, routes, CLI state, package detection, frontend progress, HW Fit services, serve profiles, Docker GPU overlays, and GPU diagnostic scripts.

Current Call Sites Include

• Cookbook modal and state modules in static/js/cookbook*.js;
• package readiness/install and rebuild flows through routes/shell_routes.py;
• direct shell exec/stream integration used by Cookbook task controls;
• model endpoint setup and serve flows;
• hardware-fit recommendations for model choices;
• image-model recommendations for diffusion serving;
• APFEL/local platform dependency paths where supported;
• Docker GPU helper scripts and compose overlays;
• the odysseus-cookbook CLI using the same Cookbook state file.

Cookbook Runtime

routes.cookbook_routes owns model download, setup, SSH key, cached model scan, serve, GPU state, kill-pid, state sync, Hugging Face latest lookup, vLLM recipe lookup, serve diagnosis, and task-status endpoints. src.cookbook_serve_lifecycle bridges scheduled cookbook_serve tasks into serve/stop behavior; task/calendar scheduling ownership stays in calendar-tasks-notes.md.

Access policy is split by surface:

• download/setup/SSH key/cache scan/serve/GPU/kill/state/task-status are admin/internal-tool surfaces;
• /api/cookbook/hf-latest is authenticated-user gated;
• HW Fit routes are authenticated read/probe routes through normal middleware, not admin-only operations;
• bearer API tokens do not satisfy Cookbook admin gates.

Runtime behavior:

• POSIX and most remote flows run detached through tmux;
• local Windows uses detached process/log/pid behavior under %TEMP%\\odysseus-tmux;
• remote Windows uses PowerShell runner scripts;
• missing tmux, docker, or serve-engine binaries return shaped errors where possible;
• local Docker inside the Odysseus container is available only when the Docker CLI exists, ODYSSEUS_ENABLE_HOST_DOCKER=true, and /var/run/docker.sock is actually mounted as a socket; otherwise Cookbook should show the host-Docker access hint and prefer remote SSH Docker workflows;
• model serve auto-registers LLM or image ModelEndpoint rows immediately, then frontend readiness probing can repair/create fallback endpoints;
• diffusion-server serves are registered as image endpoints;
• vLLM recipe routes fetch and cache model recipe manifests/YAML from vllm-project/recipes, normalize base args/env/dependencies/tool-calling/reasoning variants, and expose compatible strategy metadata for serve setup;
• Hugging Face download/setup paths can detect and persist encrypted HF tokens for later Cookbook/agent use;
• local and remote model paths can contain spaces or non-ASCII characters when helper validation/quoting accepts them;
• task status handles tmux, remote Windows logs, local Windows PID/log files, HF cache completion checks, stale browser-state download guards, pip dependency-install success sentinels, exit-code wrappers, serve diagnosis snapshots, and scheduled serve lifecycle hooks;
• scheduled serve lifecycle stop attempts only persist status=stopped, clear _scheduledStopAtMs, and delete auto-registered endpoints for sessions whose tmux/remote stop command succeeded or were already gone; failed stop attempts are logged without marking unrelated expired serves as stopped.

routes.cookbook_helpers owns validation and command construction:

• repository and model IDs;
• local directories, SSH hosts/ports, GPU selectors, and tokens;
• shell quoting for Bash and PowerShell;
• pip/install fallback chains;
• safe environment prefixes;
• serve command validation;
• user-shell PATH bootstrap, Git-Bash drive-path conversion, preflight, and exit-code helpers.

Cookbook routes request shell/SSH behavior; they do not relax shell security.

Shell Dependencies

routes.shell_routes.py owns Cookbook-adjacent package readiness/install, shell execution/streaming, and llama.cpp rebuild endpoints. The Cookbook UI calls these routes for dependency diagnosis, install/update actions, engine rebuilds, and tmux/reconnect/stop/kill flows. Windows uses detached log/PID wrappers where POSIX tmux is unavailable.

These are admin-only code-execution surfaces and should be reviewed with Cookbook changes even though they are implemented outside routes.cookbook_routes.py.

State, Secrets, And Provenance

Cookbook state lives under the shared data dir through the COOKBOOK_STATE_FILE constant, normally data/cookbook_state.json. Routes and the odysseus-cookbook CLI use the same state path.

State behavior:

• browser-facing state masks secrets;
• server-side env.hfToken is encrypted before storage;
• task payloads strip raw HF tokens;
• browser local storage strips HF token values;
• state POST has anti-wipe guards for server lists;
• state POST rejects stale done download state when the latest shard/cache markers still show an incomplete download;
• recent server-side tasks are preserved against stale browser overwrites;
• task-status validates saved shell-bound fields before SSH/tmux commands.

Cookbook auto-registered endpoints are currently shared/null-owner rows with no API key when created by backend serve registration. Browser fallback registration goes through the normal model-endpoint route. The desired ownership policy for Cookbook-created endpoints should remain explicit.

HW Fit is an MIT-licensed llmfit adaptation; attribution lives in project acknowledgments/licenses.

Hardware Fit

services/hwfit/hardware.py owns hardware detection across NVIDIA, AMD, Apple Silicon, Windows, CPU, RAM, available RAM, remote SSH, container/native probe context, and cached host detections.

services/hwfit/models.py, fit.py, profiles.py, image_models.py, and hf_discovery.py own model catalog loading, normalization, API-backed dynamic catalog refresh, memory estimates, quantization labels, fit scoring, serve profile computation, image model ranking, and backend/format servability filtering.

routes/hwfit_routes.py owns the HTTP surface and manual hardware override application.

Runtime behavior:

• hardware detection uses a cache with fresh=true bypass;
• probe results include scope/container visibility metadata, and containerized no-GPU/low-RAM states can return user-facing visibility warnings with rescan/manual/copy-diagnostics actions;
• manual hardware replacement is a what-if simulator, not additive hardware;
• manual hardware accepts cuda, rocm, metal, cpu_x86, and cpu_arm backends and must stay in lock-step with backend support in fit.py. Metal simulation marks unified memory and filters toward locally servable GGUF/MLX choices instead of CUDA/vLLM-only formats.
• ignore switches can drop detected GPU/RAM before ranking;
• homogeneous GPU grouping targets realistic multi-GPU pools;
• image model ranking normalizes to a single-GPU fit view;
• Metal/RDNA/backend restrictions can filter otherwise fit models.
• Apple Silicon bandwidth estimates use chip/core-specific tables for M-series Max/Pro/Ultra variants and avoid matching non-Apple GPU names.
• Windows and Apple/consumer-AMD paths filter toward GGUF/llama.cpp-compatible choices. On multi-GPU systems, fixed GGUF target quantization that cannot be served by the selected backend returns no_fit rather than None.

Platform And Degraded Behavior

• Linux, Windows/PowerShell, macOS, Docker, NVIDIA, AMD, Apple Silicon, and CPU-only systems have different command paths.
• Remote hosts are accessed through SSH helpers; Cookbook host/port/path inputs must be validated before command construction.
• HW Fit remote host/port query values currently do not share all Cookbook route-level validation before SSH probing.
• Missing local tools or failed installs should surface command/output/error detail where possible.
• GPU overlays remain optional and do not break CPU-only deployments.
• Docker GPU overlays pass host devices/env; they do not install CUDA/ROCm engines by themselves.
• Default Docker Compose intentionally does not mount the host Docker socket. docker/host-docker.yml is an explicit high-trust overlay for operators who accept broad host-Docker control from inside the container.
• NVIDIA Docker diagnostics are read-only by default, and .env edits/install actions require explicit flags.
• AMD Docker diagnostics are read-only and do not mutate .env.
• vLLM is rejected on unsupported Windows/macOS paths.
• llama.cpp CPU-only and GPU fallback scripts should preserve usable CPU paths.
• SSH probe failures, GPU driver errors, and no-GPU states should be distinguishable.
• Remote SSH host/port validation is shared through route validators for Cookbook/HWFit paths.
• Windows launcher/runtime Git Bash discovery includes per-user installs under %LocalAppData%\\Programs\\Git, and WSL/Git Bash detection shapes PATH handling for NVIDIA/remote flows.
• macOS startup helpers start ChromaDB alongside the app path.
• Ollama serve can auto-pick an available port, and scheduled task stop paths verify stop success before persisting a stopped state.

Model Catalog And Latest Lookup

HW Fit model scoring depends on bundled services/hwfit/data/hf_models.json, bundled services/hwfit/data/mlx_community_models.json, runtime dynamic caches under DATA_DIR/hwfit/, catalog normalization, and assumptions about model formats and quantization. scripts/add_hwfit_models.py updates the static HF catalog.

Hugging Face latest lookup and HW Fit dynamic refresh use external Hub metadata and can degrade to empty, unknown-size, partial, or malformed-result behavior. refresh_catalog=1 refreshes API-backed collection caches for MLX community and selected HF organization collections, with a 24-hour freshness guard and bundled JSON fallbacks when the network/cache is unavailable. HW Fit tolerates non-numeric gpu_count values from callers. Model normalization also treats non-string parameter_count and quantization fields as unknown rather than calling string methods and aborting the ranking pass. Catalog drift and dynamic latest-model metadata are separate sources of recommendation drift.

Security Policy

Admin gates must stay in place for install, serve, kill, setup, state mutation, and shell-like actions. /api/shell/exec is an admin primitive used by Cookbook task control and must stay in this review boundary. Scheduled cookbook_serve tasks are admin-only action tasks; task create/update/manual run/webhook/scheduler execution must all reject or pause them for non-admin owners.

Kill-pid guardrails:

• admin-only;
• PID floor;
• signal allowlist;
• validated remote host/port;
• frontend confirmation for TERM/KILL cleanup.

Shell-bound Cookbook inputs must pass helper validation before command construction. HF tokens, Cookbook state secrets, and endpoint API keys must remain encrypted or masked and must not be written back to clients in raw form. Host Docker socket access must stay opt-in and clearly distinguished from merely having a Docker CLI in the container.

Testing Coverage

Existing coverage is strongest for helper validation/quoting, SSH host validation, pip fallback and dependency-completion regressions, cached scan scripts, serve profile computation, scheduled serve lifecycle state persistence, hardware detection/ranking across AMD/NVIDIA/macOS/manual/container modes, MLX/Metal ranking, manual backend simulation, Docker GPU compose overlays, Cookbook CLI state, package detection, Windows path/task helpers, non-numeric GPU counts, non-string model catalog fields, and selected frontend progress regressions.

Route-level auth/security and degraded-return coverage is thinner for Cookbook admin routes, shell dependency routes, /api/cookbook/hf-latest, state/status edge cases, HW Fit routes, frontend JS behavior, and helper scripts such as hf_download.py, add_hwfit_models.py, and diffusion_server.py.

Current Gaps

• Cookbook-created model endpoint ownership/shared/null-owner policy needs a deliberate decision.
• /api/shell/exec and Cookbook package/rebuild routes need to remain cross-referenced with shell/admin specs because they are Cookbook-critical code-execution surfaces.
• Cookbook route auth/security and degraded-return behavior need route-level tests.
• /api/cookbook/hf-latest needs tests locking its user-authenticated access policy and failure behavior.
• HW Fit routes need route-level tests around missing catalogs, manual overrides, fit_only, profiles, and image-model cases.
• Dependency install/serve diagnosis remains split across Cookbook routes, shell routes, frontend diagnosis, optional binaries, and platform-specific scripts, even though longer serve-output tails are centralized through routes/cookbook_output.py.
• Model catalog, quantization, backend, and Hugging Face metadata drift need ongoing maintenance.