mirror of
https://github.com/pewdiepie-archdaemon/odysseus.git
synced 2026-08-11 05:45:32 +00:00
47 lines
1.8 KiB
Markdown
47 lines
1.8 KiB
Markdown
# llama.cpp Provider Shape
|
|
|
|
Last updated: dev@e57f60b | 2026-07-20
|
|
|
|
## Scope
|
|
|
|
Canonical provider ID `llamacpp`; OpenAI Chat/Responses and Anthropic Messages
|
|
compatibility plus native server metadata; reader
|
|
`src/model_capability_readers/llamacpp.py`.
|
|
|
|
## Metadata Shapes
|
|
|
|
`/v1/models` provides served identity and can include server model entries;
|
|
native `/props` is authoritative for the running model/server combination:
|
|
|
|
- `model_alias`/`model_path`;
|
|
- `default_generation_settings.n_ctx` and sampling `params`;
|
|
- `total_slots` and optional `/slots[].n_ctx` fallback;
|
|
- `chat_template_caps` for tools/system role;
|
|
- `modalities.vision|audio`;
|
|
- current server/build state.
|
|
|
|
Capability depends on weights, projection/model assets, chat template, parser,
|
|
and launch flags. It is endpoint evidence, not a checkpoint-name claim.
|
|
`/props` and `/v1/models` can be merged only for the same served identity.
|
|
|
|
## Request And Response Shape
|
|
|
|
llama-server supports several OpenAI-compatible tasks and native extensions.
|
|
Do not infer embeddings/rerank/chat solely from the OpenAI model card; use an
|
|
explicit server model capability field or endpoint configuration. Tool and
|
|
reasoning correctness can depend on selected chat template and parser.
|
|
|
|
## Fallback And Safety
|
|
|
|
The registry selects llama.cpp through an explicit vendor or endpoint kind; it
|
|
does not auto-detect `/props` from payload shape. Port 8000 currently maps to
|
|
the vLLM placeholder, while 8080 falls through to generic OpenAI-compatible.
|
|
llama.cpp-only `session_id` and `cache_prompt` affinity fields must remain local
|
|
endpoint behavior and never leak to strict cloud providers (#4640 and current
|
|
affinity tests).
|
|
|
|
## Current Gaps
|
|
|
|
- Multi-model routing requires per-served-model `/props` association.
|
|
- Parser/template configuration is not yet fully represented in canonical
|
|
endpoint metadata.
|