- Add fcntl.flock inter-process file lock shared by setup() and
create_user_oidc() so multi-worker first-admin bootstrap is
serialised across processes, not just threads within one worker.
Both methods reload auth.json inside the lock so the loser sees
the winner's write.
- _fetch_userinfo() now returns None (not {}) when discovery has
no userinfo_endpoint, and exchange_code() only sets
_userinfo_available=True when a live endpoint was reached.
Prevents the callback from treating 'no endpoint' as
authoritative group non-membership evidence.
- Rewrite _load_or_create_key() to write the Fernet key to a temp
file, fsync, then atomically os.link() into place. No reader
ever sees the final path before the complete key bytes are
available — a racing worker either sees no file or a complete
one, never an empty/partial file.
105 tests pass (97 existing + 8 new regressions covering the
three fixes).
Co-Authored-By: Kevin <holden093@users.noreply.github.com>
Three fixes from second review pass:
1. (security) Serialize the first-OIDC-user admin bootstrap inside
_config_lock. Previously the check and username
collision resolution happened outside the lock, so two concurrent
first-login callbacks could both observe an empty user map and
both persist as admin. Now idempotent lookup, bootstrap decision,
and collision resolution are all inside one critical section.
2. (auth) Make data/.app_key creation atomic via O_EXCL open so two
racing workers on a fresh deployment cannot generate different
keys. The loser reads the winner's key, guaranteeing every worker
shares the same Fernet key for OIDC state encryption.
3. (auth) Track whether UserInfo was successfully fetched
(_userinfo_available flag in claims). The callback now skips
admin group sync for existing users when UserInfo is unavailable
AND the id_token lacks a groups claim — a transient provider
failure no longer silently demotes existing OIDC admins. When
UserInfo succeeds or the id_token carries groups, admin status
syncs as before.
Regression tests: concurrent admin bootstrap, key-creation race,
UserInfo-unavailable preserves admin, UserInfo-available demotes,
id_token groups authoritative without UserInfo.
Two fixes from review feedback:
1. (security/availability) Record _last_jwks_refresh timestamp BEFORE
calling _refresh_jwks() so a failed JWKS fetch is also throttled
by the 60-second cooldown. Previously the timestamp was only set
after success, so an outage or key rotation would retry the IdP
on every login attempt.
2. (auth) Use the shared persistent app key from secret_storage
(_get_fernet) for OIDC state encryption instead of reading
data/.app_key directly and falling back to a per-process key.
This guarantees all uvicorn workers share the same Fernet key,
even on a fresh data directory — worker A's state is always
decryptable by worker B on the callback.
Add regression tests for both fixes.
1. JWKS fetch/parse errors now wrapped as OidcError instead of escaping
as raw exceptions. Transient network failures, HTTP errors, and bad
JSON responses from the JWKS endpoint all produce a controlled
OidcError, which the callback route already redirects to
/login?error=oidc_failed instead of a 500.
2. exchange_code() now extracts the stored redirect_uri from the
Fernet-encrypted state token and rejects any callback-derived
redirect_uri that differs. The token exchange POSTs the stored
value — the one the IdP saw in the authorization request — removing
the last callback dependence on request-derived URI behaviour.
Tests: 5 new (3 JWKS error wrapping + 2 redirect_uri binding)
Total: 130 passing (81 OIDC + 49 regression), 0 failures.
1. Discovery issuer mismatch now raises OidcError instead of logging
a warning (OIDC Discovery §1.1 requires mismatch abort).
2. Multi-audience ID tokens without azp are now rejected (OIDC Core
§2 requires azp when aud has multiple values).
3. /change-password, /2fa/setup, /2fa/confirm, and /2fa/disable now
reject OIDC users with a clear message. The frontend already hides
these cards, but the backend must also enforce the policy.
113 passing (76 OIDC + 37 regression), 0 failures.
Both vars were documented in .env.example and read by the application
code, but never forwarded to the container by any docker-compose file.
OIDC_REDIRECT_URI is essential for deployments behind a reverse proxy
to avoid deriving the wrong scheme from the inbound request.
1. Suppress first-user bootstrap admin when OIDC_ADMIN_GROUPS is
configured — a non-admin IdP user must not get admin just by being
the first to log in. Group membership is the only path when groups
are set.
2. Reject mismatched UserInfo sub: the UserInfo endpoint MUST NOT
overwrite the verified id_token subject. Also guard all verified
identity claims (sub, iss, aud, exp, iat, nonce, azp) from being
overwritten by UserInfo.
1. Fix bootstrap admin demotion: skip set_oidc_user_admin when
OIDC_ADMIN_GROUPS is unset so bootstrap/manual admin survives
subsequent logins.
2. Login CSRF: bind state to an HttpOnly cookie set at /login and
verified with constant-time compare at /callback.
3. JWKS cooldown: throttle refresh to once per 60s to prevent
attacker-triggered unbounded IdP fetches via random kid values.
4. Secure cookies: derive secure flag from request scheme when
SECURE_COOKIES is not explicitly set; OIDC session defaults
to secure on HTTPS connections.
5. Remove dead validation-shaped code: the jwt.decode block with
None key that swallowed all exceptions and discarded output.
1. aud validation: handle JSON array audiences per OIDC spec (check
membership, validate azp for multi-audience tokens).
2. JWKS caching: cache keys after first fetch; refresh only on unknown
kid — avoids live IdP round-trip on every login.
3. Algorithm pinning: restrict to RS256/ES256/etc from discovery doc,
never allow HS256 or 'none'.
4. OIDC_REDIRECT_URI: support a fixed redirect URI (proxy-safe,
doesn't trust Host header).
5. Stateless state: replace in-memory dict with Fernet-encrypted state
tokens — works across multiple uvicorn workers/processes.
6. OIDC_FIRST_USER_IS_ADMIN: bootstrap first OIDC user as admin when
no users exist and OIDC_ADMIN_GROUPS is unset — prevents zero-admin
lockout.
The base docker-compose.yml gained 6 OIDC_* env vars; the standalone
GPU files must mirror them or test_gpu_compose_standalone fails.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds OIDC authentication as an alternative to password login, enabling
sign-in via any standard provider (Authentik, Keycloak, Authelia, etc.).
New features:
- Generic OIDC provider support via .well-known discovery (authlib)
- Coexists with existing password auth — users choose at login
- Auto-creates local users on first OIDC login
- Admin group mapping: OIDC_ADMIN_GROUPS grants admin based on IdP groups
- Admin status syncs on every login (follows IdP membership)
- OIDC users cannot use password login or set up 2FA
- UI hides change-password and 2FA cards for OIDC users
New env vars:
- OIDC_ENABLED, OIDC_ISSUER, OIDC_CLIENT_ID, OIDC_CLIENT_SECRET
- OIDC_SCOPES, OIDC_ADMIN_GROUPS
New files:
- core/oidc.py — OidcManager (discovery, auth URL, code exchange,
id_token verification with JWT/JWKS)
- routes/oidc_routes.py — /api/auth/oidc/{login,callback,config}
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(rag): skip hidden and junk directories when indexing (#5559)
index_personal_documents walked the whole tree with no pruning, so
pointing RAG at a real-world folder silently swept in .obsidian/ plugin
JS, .git/ internals, node_modules/, and __pycache__/ — multiplying
indexing time and polluting retrieval with junk chunks.
Prune hidden directories and well-known junk directories from the walk,
and skip hidden files. The explicitly passed root is exempt, so a user
who deliberately indexes a hidden directory still gets its contents.
* fix(rag): prune hidden/junk dirs in the keyword index too, via a shared helper
The #5559 fix pruned only VectorRAG.index_personal_documents (the vector index).
The parallel keyword index built by PersonalDocsManager.refresh_index ->
load_personal_index walked the same tree unpruned, so .obsidian/, .git/,
node_modules/ etc. still swept into keyword retrieval and the file listing —
the 'end-to-end' guarantee was only half true.
Single-source the pruning policy in src/index_walk (prune_index_dirs +
is_indexable_file) and use it from both walkers so they cannot drift again.
The junk-dir match is now case-insensitive, so a Node_Modules on a
case-insensitive filesystem is pruned too.
Tests: keyword-path regressions covering hidden/junk dirs, hidden files, junk
at depth (not just top level), case-insensitive junk, and the explicit-hidden-
root exemption. The existing vector tests still pass against the shared helper.
* fix(email): use XOAUTH2 in test-connection for Google OAuth accounts
The test-connection endpoint was password-only and had no awareness of
OAuth accounts. For Google-connected accounts this caused two failures:
- IMAP: "Need IMAP host, username, and password" because imap_pass is
empty (no password is stored for OAuth accounts)
- SMTP: 535 BadCredentials because smtp.login() was called with an
empty password instead of an XOAUTH2 token
Fix: include oauth_provider and token fields in saved_body when hydrating
from the DB, then use conn.authenticate("XOAUTH2") / smtp.auth("XOAUTH2")
for Google accounts in both the IMAP and SMTP test paths, mirroring what
_send_smtp_message already does for real sends.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* test(email): add OAuth2 tests for test-connection endpoint
Covers the XOAUTH2 changes made to routes/email_routes.py:
- Google OAuth accounts must not be rejected with 'Need IMAP host,
username, and password' (no stored password for OAuth accounts)
- IMAP and SMTP test paths must use conn.authenticate('XOAUTH2')
for Google accounts
- Password accounts must still use conn.login() / smtp.login()
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(email): bind OAuth account tests to Google transport
---------
Co-authored-by: TNTBA <trynottobreakanything@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(config): forward Google OAuth env vars into Docker container and document setup
GOOGLE_OAUTH_CLIENT_ID, GOOGLE_OAUTH_CLIENT_SECRET, and GOOGLE_OAUTH_REDIRECT_URI
were read by the app but never forwarded through docker-compose.yml's explicit
environment allowlist, causing the "not set" error even when the vars existed in .env.
Also adds a documented section to .env.example with step-by-step GCP setup instructions
so users know where to get the credentials.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(config): cover OAuth in standalone compose files
* test(config): parse OAuth compose service env
* test(config): keep checkout skip wording neutral
---------
Co-authored-by: TNTBA <trynottobreakanything@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Slice 2i of the route-domain reorganization (#4082/#4071). Moves
compare_routes.py into routes/compare/, leaving a backward-compat
sys.modules shim at the old path. Pure file reorganization, no behavior
change.
The shim uses sys.modules replacement so the `import ... as cr` +
`monkeypatch.setattr(cr, "SessionLocal", ...)` / `"_owned_endpoint_by_url"`
/ `"_owned_endpoint_by_id"` pattern in test_endpoint_owner_scope_followup.py
reaches the canonical module.
Canonical module imports only from core/, src/, and routes.session_routes
(zero dependency on the legacy shim). One source-introspection test site
repointed: test_endpoint_owner_scope_followup.py (shared with other domains;
only the compare entry repointed here).
Adds tests/test_compare_routes_shim.py to pin the sys.modules shim
contract. Verified: compileall clean; targeted tests pass.
Slice 2h of the route-domain reorganization (#4082/#4071). Moves
admin_wipe_routes.py into routes/admin_wipe/, leaving a backward-compat
sys.modules shim at the old path. Pure file reorganization, no behavior
change.
The shim uses sys.modules replacement so the `import ... as
admin_wipe_routes` + `monkeypatch.setattr(admin_wipe_routes, "SessionLocal",
...)` / `"require_admin"` pattern in test_admin_wipe_gallery.py reaches
the canonical module.
Canonical module imports only from core/, src/, and stdlib (zero internal
routes/ coupling). Zero source-introspection landmines.
Adds tests/test_admin_wipe_routes_shim.py to pin the sys.modules shim
contract. Verified: compileall clean; targeted tests pass.
Slice 2g of the route-domain reorganization (#4082/#4071). Moves
cleanup_routes.py into routes/cleanup/, leaving a backward-compat
sys.modules shim at the old path. Pure file reorganization, no behavior
change.
The shim uses sys.modules replacement so string-targeted
monkeypatch.setattr("routes.cleanup_routes.*", ...) in
test_cleanup_owner_scope.py reaches the canonical module.
Canonical module imports only from src/ and stdlib (zero internal
routes/ coupling). Zero source-introspection landmines.
Adds tests/test_cleanup_routes_shim.py to pin the sys.modules shim
contract. Verified: compileall clean; targeted tests pass.
Slice 2f of the route-domain reorganization (#4082/#4071, per
specs/architecture-runtime-inventory.md §6.3). Moves note_routes.py into
routes/note/, leaving a backward-compat sys.modules shim at the old path.
Pure file reorganization, no behavior change.
The shim uses sys.modules replacement (same pattern as the merged gallery
#4903, research #4975, memory #5007, history #5090, and contacts #5227
slices) so that `import routes.note_routes`, `from routes.note_routes import
X`, `importlib.import_module(...)`, and the `import ... as note_routes` +
`monkeypatch.setattr(note_routes, "SessionLocal", ...)` pattern used by
test_note_reminder_fire_scope.py / test_notes_fail_closed_auth.py all
operate on the same module object the application uses.
The canonical module does NOT depend on the shim — routes/note/note_routes.py
imports only from core/, src/, and stdlib. The outbound email cross-domain
imports (routes.email_routes._get_email_config, routes.email_helpers.
_send_smtp_message) are function-local lazy imports that keep resolving
through the email module's own path (email is not yet migrated).
One source-introspection test site repointed to the new canonical path:
- test_model_helper_owner_scope.py (shared with history; history entry
already repointed in #5090, note entry repointed here)
Adds tests/test_note_routes_shim.py to pin the sys.modules shim contract
(legacy and canonical paths resolve to the same module object; monkeypatch
via legacy alias reaches the canonical module).
Verified: compileall clean; full suite 4487 passed, 3 skipped.
* security(url-safety): reject RFC 6598 shared address space in strict mode
Strict mode (block_private=True) is a full SSRF lockdown, but it only
rejected is_private and is_loopback targets. CPython does not classify RFC
6598 shared/CGNAT space (100.64.0.0/10) as is_private (it is "shared", not
"private"), so a public redirect into 100.64.0.1 passed the per-hop guard
and still issued the request to a potentially internal CGNAT service.
not is_global would also exclude it, but only on CPython 3.11.10+/3.12.4+/
3.13+; the CI matrix runs 3.11/3.12, so reject the range explicitly to stay
correct across patch levels and the 3.14 runtime image. Default local-first
mode is unchanged. Adds strict-mode coverage for shared, non-global, and
public targets.
* docs(url-safety): correct CGNAT is_global rationale in strict-mode comment
The prior comment claimed `not is_global` catches 100.64.0.0/10 only on
CPython 3.11.10+/3.12.4+/3.13+. That is inaccurate for CGNAT: is_global
is False for 100.64.0.1 on every supported version (verified 3.10-3.14).
The version-fragility applies to other ranges gh-113171 touched, not CGNAT.
The explicit range reject is still the right choice; restate the reason as
is_private not covering shared space, and not coupling strict mode to
is_global's broader, cross-version definition. No behavior change.
* feat(models): define capability schema and readers
* fix(models): harden Google catalog probing
Restrict native catalog probing to the Gemini host, keep provider keys out of request URLs, filter non-chat model resources, and preserve the manual refresh default in the built-in Google add flow.
Add Arch-specific package installation and NVIDIA runtime configuration
to the Docker setup guide. Cover passthrough verification, the NVIDIA
Compose overlay, and the distinction between GPU passthrough and
CUDA-backed model serving
Refs #831
* Harden skill importer against SSRF: block private targets + revalidate redirects per hop
The skill importer validated only the initial URL with the lenient SSRF guard
(block_private=False) and then fetched with follow_redirects=True, so a 3xx to
an internal/metadata address (169.254.169.254, 127.0.0.1, RFC-1918) was still
connected to — inconsistent with the hardened services/search/content.py
:_get_public_url path.
Add a _get_checked() helper that follows redirects manually and re-runs the
SSRF guard with block_private=True on every hop, and route all three fetch
sites (skills.sh unwrap, _fetch_bytes, _list_github_dir) through it. GitHub's
own redirects and the final-host _assert_github_url checks are preserved.
Adds hermetic regression tests (IP-literal hosts, faked HTTP layer) and updates
the existing mock signature for the new block_private kwarg.
Defense-in-depth: the endpoint is admin-gated (require_admin) and admins are
trusted per THREAT_MODEL.md, so this is not a cross-boundary vulnerability.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test: enforce follow_redirects=False invariant in mock client
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
chat_stream() references `_explicit_web_intent` in three places
(disabled-tools gating, global-disabled web allowance, and the
per-turn tool filter) but the assignment was dropped during a
branch merge. Every chat request raised
NameError: name '_explicit_web_intent' is not defined
at routes/chat_routes.py, surfacing to the client as a bare
"Internal Server Error" before any LLM call was made — chat was
fully broken on dev and main.
Restore the original definition, computed from the already-derived
tool intent, immediately before its first use:
_explicit_web_intent = bool(_tool_intent and _tool_intent.category == "web")
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(cherry picked from commit 30e87e3b82)
* fix(db): restrict data/app.db to 0600
app.db holds bearer-token hashes, bcrypt password hashes, and encrypted
provider keys but was created under the default umask (0644 -> world-readable),
unlike .app_key/vault/integrations which are already 0600 via safe_chmod.
init_db() now chmods the SQLite file to 0600 right after create_all (POSIX
only; no-op on Windows, skipped for Postgres / in-memory). Unconditional and
idempotent, so it also re-locks already-deployed 0644 installs on next
startup. The transient rollback journal inherits 0600 from the parent file at
creation - no sidecar handling needed; -wal/-shm don't exist until WAL is
enabled (#4409 C4) and inherit the same mode then.
Satisfies Rule B, unblocking #4413 and the vault/integration secret moves.
Mirrors src/secret_storage.py:43-45.
Verified: security + DB-permission suites pass; 6 pre-existing visual_report
failures (missing markdown/nh3 deps) are unrelated.
Closes#4407
* fix(db): harden SQLite path parsing and re-lock sidecars
Address review feedback on #4420.
P2: derive the file to chmod from engine.url (SQLAlchemy's parsed URL)
via _sqlite_db_path(), instead of DATABASE_URL.replace("sqlite:///", "").
A driver-qualified URL (sqlite+pysqlite://) or one carrying query args
(?cache=shared) previously slipped past the prefix check / string slice
and left the DB world-readable; the parsed path resolves correctly and
drops the query.
P3: re-lock stale -wal/-shm/-journal sidecars to 0o600 at startup. The
main file is chmod'd first, so any sidecar SQLite creates afterward
inherits 0o600, but a -wal/-shm left world-readable by an older 0o644
install (once WAL was enabled) could still expose DB pages. Absent
sidecars are the normal case, not an error.
Tests: unit-test _sqlite_db_path across driver/query/memory/postgres URL
forms, and a subprocess test asserting stale 0o644 -wal/-shm are
re-locked on startup.
* fix(db): handle sqlite file URI app db permissions
* fix(db): close remaining SQLite permission bypasses
---------
Co-authored-by: Ethan <23321960+0xLeathery@users.noreply.github.com>
Co-authored-by: Alexandre Teixeira <alexandremagteixeira@gmail.com>
* docs: update static/js/MODULE_SUMMARY.md to reflect current ES6 frontend
Rewrite the stale module summary to match the current no-build,
ES6-module frontend architecture. Adds coverage of app.js orchestration,
the chat/SSE pipeline (chat.js, chatStream.js, chatRenderer.js,
streamingRenderer.js), new subsystems (research/, compare/, document
streaming, cookbook*, skills.js), and removes the obsolete <script> load
order assumptions.
* cleanup: remove dead MEMORY_DOC / memory_doc paths (closes#4411)
Removes the unused MEMORY_DOC constant and the matching DataConfig
memory_doc field / set_data_paths entry. No runtime code imports or
references these paths, so this is a no-behavior-change dead-code
cleanup under the storage-architecture tracker #4377.
