LLM Router integration — spike results (Phase 0)

Covers the LLM Router instance the spike runs were executed against. Describes

only what was checked in HTTP requests or in the code of

/mnt/data2/dev/develop/llm-router and /mnt/data2/dev/develop/llm-router-plugins.

0. Instance under test

1. Healthcheck — verified

Only GET /health is usable as the benchmark healthcheck.

2. Model inventory — verified

GET /v1/models (code 200) returns 6 models:

google/gemma-4-12b-it
google/gemini-3.1-flash-lite-preview
google/gemma-4-12b-it-playground
qwen3-coder:30b
qwen/Qwen3.8-27B
qwen/Qwen3.8-Flash-Next

Note: auto_codex does not appear in this list even though it works as an

alias in requests. /v1/models is therefore not a complete picture of the

"model" dimension's value space.

3. Can the effective configuration be read? — NOT over HTTP

Checked and rejected (403 Access to this endpoint is denied by policy):

/config, /admin/config, /routing/config, /knowledge/stats, /utils,

/version, /api_version, /v1/api_version, /list_models.

The effective configuration is only reachable on the router host:

• model file: LLM_ROUTER_MODELS_CONFIG (default resources/configs/models-config.json),

the active_models key is required;

• plugin configuration: llm_router_plugins/resources/<type>/<plugin>.json

plus environment overrides.

Conclusion: in external mode AK-Bench cannot verify the effective

configuration of the router or its plugins. Per the plan's rule ("a

configuration is not considered active without verification"), experiments on a

shared router must be marked unverified unless file/log access on the host is

available.

4. Plugin pipeline — verified from code

5. The "Auto Codex" dimension (plugin agentic_routing_codex)

• Trigger: the auto_codex model alias backed by an api_type: "builtin" provider.
• ENV prefix: LLM_ROUTER_ROUTING_SEMANTIC_AGENTIC_CODEX_

(including …TRIGGER, …HEURISTIC_ENABLED, …SEMANTIC_ENABLED).

Names confirmed in code: AGENTIC_ROUTING_PREFIX = "LLM_ROUTER_ROUTING_SEMANTIC_AGENTIC_",

and the Codex prefix is AGENTIC_CODEX_ROUTING_PREFIX = f"{AGENTIC_ROUTING_PREFIX}CODEX_".

• Proof of life — verified: a request with model: "auto_codex" returned

"model": "qwen/Qwen3.8-Flash-Next" — routing replaced the model.

6. The "Knowledge" dimension (plugin agentic_knowledge)

Facts that matter for Knowledge/Replay experiments:

• the plugin changes payload["model"] only when escalation fires, and appends

a context block; the annotation lands in routing.knowledge **in the internal

payload**, not in the HTTP response the agent sees;

• the end of an agent session is not proof that the KB is ready — indexing is

asynchronous (worker), so Replay requires confirming that the write and index

steps have finished first;

• real trace of knowledge at work: the injected <agentic_knowledge> block was

visible in the context of the current operator session in this repository

(anecdotal evidence, but from real traffic).

7. Metrics and plugin activity signals — a significant gap

GET /metrics (code 200, Prometheus) contains, among others:

http_requests_total, http_request_duration_seconds_*, http_request_size_bytes_*
http_request_exceptions_total, http_requests_in_progress
llm_router_provider_calls_total, llm_router_provider_error_total
llm_router_provider_latency_seconds_*, llm_router_model_fallback_total
llm_router_lb_strategy_selected_total, llm_router_retry_total
llm_router_tokens_total, llm_router_response_format_total
llm_router_pipeline_stage_total{result="success",stage="provider_resolved"}
guardrail_incidents_total, masker_incidents_total

• llm_router_pipeline_stage_total has only the provider_resolved stage.
• There are no per-plugin counters: neither knowledge_retrieval_count, nor

knowledge_write_count, nor any agentic_routing_codex signal.

Conclusion: the current metric set is not enough to measure the Knowledge and

Auto Codex dimensions. One of the following is required:

1. enable router logging (LLM_ROUTER_LOG_TO_FILE, LLM_ROUTER_LOG_LEVEL) and

parse the plugins' decision lines,

1. a custom plugin export into /metrics,
2. reading KB/SSE state directly.

Until one of them exists, knowledge metrics must be reported as unavailable,

never as zero.

Measured indexing-lag observation (2026-10-30): while working in this

repository, the injected document registry covered only README.md and the plan

file, even though two new docs/*.md files had been on disk for several

minutes. This confirms the plan's thesis: **KB readiness must be verified

separately**, and the window between a write and knowledge becoming available is

a measurable metric (indexing_latency).

8. Router runtime environment variables (verified from docs/ENV_DEFINITIONS.md)

LLM_ROUTER_VERBOSE is the only verified way to see full payloads (and thus the

routing.knowledge annotation), but it logs PII — acceptable for the benchmark

only in an isolated environment and with secret filtering.

9. Managed vs shared mode — a design decision

Recommendation: experiments with the Knowledge and Auto Codex dimensions

require a managed instance (own container + own Redis/SSE + own configuration

file). The shared router is fine for integration and smoke tests, but its

results must be marked valid_with_warnings or invalid depending on the

capabilities an experiment requires.

10. Integration status