llama.cpp Provider Shape

Last updated: dev@e57f60b | 2026-07-20

Scope

Canonical provider ID llamacpp; OpenAI Chat/Responses and Anthropic Messages compatibility plus native server metadata; reader src/model_capability_readers/llamacpp.py.

Metadata Shapes

/v1/models provides served identity and can include server model entries; native /props is authoritative for the running model/server combination:

model_alias/model_path;
default_generation_settings.n_ctx and sampling params;
total_slots and optional /slots[].n_ctx fallback;
chat_template_caps for tools/system role;
modalities.vision|audio;
current server/build state.

Capability depends on weights, projection/model assets, chat template, parser, and launch flags. It is endpoint evidence, not a checkpoint-name claim. /props and /v1/models can be merged only for the same served identity.

Request And Response Shape

llama-server supports several OpenAI-compatible tasks and native extensions. Do not infer embeddings/rerank/chat solely from the OpenAI model card; use an explicit server model capability field or endpoint configuration. Tool and reasoning correctness can depend on selected chat template and parser.

Fallback And Safety

The registry selects llama.cpp through an explicit vendor or endpoint kind; it does not auto-detect /props from payload shape. Port 8000 currently maps to the vLLM placeholder, while 8080 falls through to generic OpenAI-compatible. llama.cpp-only session_id and cache_prompt affinity fields must remain local endpoint behavior and never leak to strict cloud providers (#4640 and current affinity tests).

Current Gaps

Multi-model routing requires per-served-model /props association.
Parser/template configuration is not yet fully represented in canonical endpoint metadata.

1.8 KiB Raw Blame History