mirror of
https://github.com/pewdiepie-archdaemon/odysseus.git
synced 2026-08-06 03:15:28 +00:00
cb4ec7942d
* Chat metrics: show backend's true generation t/s, not tokens÷wall-clock
The per-message tokens/sec read low and felt wrong because it was computed as
output_tokens / total_duration, where total_duration is wall-clock including
prefill, tool calls, and network — not pure decode time. llama.cpp already
reports the correct gen speed in its stream (timings.predicted_per_second), but
it was being dropped.
- llm_core.py: when parsing the OpenAI-compatible usage chunk, also read the
sibling `timings` block llama.cpp includes — pass predicted_per_second through
as gen_tps and prompt_per_second as prefill_tps on the usage event.
- agent_loop.py: capture backend_gen_tps/backend_prefill_tps from usage events;
in _compute_final_metrics prefer backend_gen_tps over the wall-clock division
when present (fall back to computed for cloud APIs that omit timings). Tag the
result with tps_source ("backend" vs "computed") and surface prefill_tps.
Result: the displayed t/s now matches the model's real decode speed and is
stable regardless of prompt length (a long prefill no longer deflates it).
Checks: py_compile passes; verified extraction against a real llama.cpp final
chunk (gen 79 t/s surfaced vs the deflated wall-clock figure shown before).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Chat metrics: surface true t/s on the direct-chat path too
Follow-up to the gen-tps work: the non-agent direct-chat stream path in
chat_routes turned the raw `usage` event straight into a metrics event but only
copied token counts — it never set tokens_per_second or response_time. So simple
(non-tool) replies showed "Speed: n/a" / "Time: undefineds" and the chip fell
back to a bare token count ("27 tok") instead of t/s.
Map the usage event's gen_tps (llama.cpp timings.predicted_per_second, added in
the prior commit) into tokens_per_second here too, tag tps_source=backend, and
set response_time from wall-clock for the stats popup.
Checks: py_compile passes; verified llama.cpp emits usage+timings on the final
stream chunk (gen ~90 t/s) that this path consumes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Tests: backend gen/prefill t/s passthrough and preference
Cover the two pieces of the true-t/s metric so it can be reviewed on its own:
- stream_llm surfaces llama.cpp's timings.predicted_per_second /
prompt_per_second as gen_tps / prefill_tps on the usage event (captured
llama.cpp final-chunk fixture), and omits them when the backend reports no
timings.
- _compute_final_metrics prefers backend_gen_tps over output/wall-clock,
tags tps_source ("backend" vs "computed"), and surfaces prefill_tps.
Reuses the fake-client stream harness from test_llm_core_streaming.py.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|---|---|---|
| .. | ||
| bombadil-spec.ts | ||
| conftest.py | ||
| test_action_intents.py | ||
| test_admin_wipe_gallery.py | ||
| test_agent_loop.py | ||
| test_api_key_manager_resilience.py | ||
| test_api_token_routes.py | ||
| test_app.py | ||
| test_app_static_mime.py | ||
| test_auth_event_loop.py | ||
| test_auth_regressions.py | ||
| test_auth_session_revocation.py | ||
| test_backup_cli_security.py | ||
| test_builtin_memory_consolidation.py | ||
| test_calendar_owner_scope.py | ||
| test_calendar_recurrence.py | ||
| test_chat_image_routing.py | ||
| test_chat_metrics.py | ||
| test_chat_stream_scope.py | ||
| test_chroma_client.py | ||
| test_cleanup_owner_scope.py | ||
| test_companion_pairing.py | ||
| test_companion_readonly.py | ||
| test_compare_js.py | ||
| test_context_compactor.py | ||
| test_cookbook_dependency_completion_regression.py | ||
| test_cookbook_helpers.py | ||
| test_ddg_redirect_resolution.py | ||
| test_deep_research_extraction_controls.py | ||
| test_deep_research_search_error.py | ||
| test_dialog_aria.py | ||
| test_document_deeplink.py | ||
| test_document_pdf_marker.py | ||
| test_document_tool_owner_scope.py | ||
| test_email_library_bulk_actions.py | ||
| test_email_smtp_security.py | ||
| test_endpoint_probing.py | ||
| test_endpoint_resolver.py | ||
| test_esc_menu_stack_js.py | ||
| test_forwarded_message_divider.py | ||
| test_gallery_endpoint_matching.py | ||
| test_gallery_image_privileges.py | ||
| test_history_topics_owner_scope.py | ||
| test_hwfit_macos.py | ||
| test_hwfit_quant_formats.py | ||
| test_keybind_altgr_js.py | ||
| test_llm_core_anthropic_cache.py | ||
| test_llm_core_concurrency.py | ||
| test_llm_core_fallback.py | ||
| test_llm_core_ollama.py | ||
| test_llm_core_reasoning.py | ||
| test_llm_core_sanitize_tool_calls.py | ||
| test_llm_core_streaming.py | ||
| test_local_endpoint_api_key_js.py | ||
| test_local_endpoint_js.py | ||
| test_markdown_rendering_js.py | ||
| test_markitdown_runtime.py | ||
| test_mcp_manager.py | ||
| test_memory_bullet_extraction.py | ||
| test_memory_extractor_vector_degraded.py | ||
| test_model_context.py | ||
| test_model_routes.py | ||
| test_notes_update_due_date.py | ||
| test_null_owner_gates.py | ||
| test_og_image_extraction.py | ||
| test_ollama_port_detection.py | ||
| test_pdf_runtime.py | ||
| test_personal_docs_exclusions.py | ||
| test_personal_docs_office_index.py | ||
| test_personal_docs_pdf_index.py | ||
| test_personal_upload_isolation.py | ||
| test_platform_compat.py | ||
| test_preset_fill_missing_defaults.py | ||
| test_provider_classification.py | ||
| test_provider_detection.py | ||
| test_provider_endpoints.py | ||
| test_rate_limiter.py | ||
| test_reply_recipients_js.py | ||
| test_research_owner_scope_routes.py | ||
| test_research_query_fallback.py | ||
| test_research_service.py | ||
| test_research_session_id_validation.py | ||
| test_research_utils.py | ||
| test_reserved_username_admin_escalation.py | ||
| test_resolve_endpoint_fallbacks.py | ||
| test_review_regressions.py | ||
| test_scheduler_restart_doublefire.py | ||
| test_search_cache_invalidation.py | ||
| test_search_query.py | ||
| test_search_ranking.py | ||
| test_security_regressions.py | ||
| test_serve_profiles.py | ||
| test_session_ghost_delete.py | ||
| test_session_mode_helpers.py | ||
| test_session_owner_attribution.py | ||
| test_settings_scrub.py | ||
| test_setup_admin_user.py | ||
| test_shell_routes.py | ||
| test_skill_index_prompt_injection.py | ||
| test_skills_delete_owner.py | ||
| test_skills_manager_owner_isolation.py | ||
| test_speech_service_toggles.py | ||
| test_sqlite_foreign_keys.py | ||
| test_strip_think.py | ||
| test_stt_leak.py | ||
| test_task_scheduler_cancel.py | ||
| test_task_scheduler_session_delivery.py | ||
| test_tool_index_keyword_boundaries.py | ||
| test_topic_analyzer.py | ||
| test_tts_cache_stats.py | ||
| test_unknown_tool_calls.py | ||
| test_upload_handler_atomicity.py | ||
| test_upload_routes_owner_scope.py | ||
| test_vault_password_not_in_argv.py | ||
| test_vision_model_detection.py | ||
| test_visual_report.py | ||
| test_webhook_ssrf_resilience.py | ||
| test_webhook_trigger_auth_exempt.py | ||
| test_windows_update_script.py | ||
| test_youtube_comments_timeout.py | ||