verdict
partial
runs
8
evaluated
8
success rate
0.0%
mean time
10 m 48 s
input tokens
4,541,888
generated tokens
149,945
tokens
4,691,833
tool calls
181
Experiments this subject took part in
| experiment | title | type | runs | evaluated | passed | success rate | mean time | input tokens | generated tokens | total tokens | tool calls |
|---|---|---|---|---|---|---|---|---|---|---|---|
| router-config-comparison@1.0.0 | Router models versus Auto Codex, with and without knowledge | configuration | 8 | 8 | 0 | 0.0% | 10 m 48 s | 4,541,888 | 149,945 | 4,691,833 | 181 |
Tasks this subject worked
| task | category | difficulty | runs | passed | success rate | mean time | input tokens | generated tokens | total tokens | tool calls |
|---|---|---|---|---|---|---|---|---|---|---|
| cadence-catchup-001@1.0.0 | new_feature | expert | 2 | 0 | 0.0% | 13 m 1 s | 1,782,999 | 51,862 | 1,834,861 | 60 |
| freight-shop-001@1.0.0 | new_feature | expert | 6 | 0 | 0.0% | 10 m 3 s | 2,758,889 | 98,083 | 2,856,972 | 121 |
Runs
in the order they started; the parameters column is the configuration each run was executed under
| run | parameters | started | task | agent | repetition | status | verdict | score | time | input tokens | generated tokens | total tokens | tool calls | failure class | error |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| run-20261102-134326-1f193cec | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 13:43:26 | cadence-catchup-001@1.0.0 | codex@codex-cli 0.166.0 | 2 | completed | partial | 0.960 | 13 m 32 s | 946,332 | 28,707 | 975,039 | 30 | ||
| run-20261102-145348-58a7a3f3 | knowledge_enabled=True, model=auto_codex | 2026-11-02 14:53:48 | cadence-catchup-001@1.0.0 | codex@codex-cli 0.166.0 | 1 | completed | partial | 0.920 | 12 m 29 s | 836,667 | 23,155 | 859,822 | 30 | ||
| run-20261102-153629-de5ac66d | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 15:36:29 | freight-shop-001@1.0.0 | codex@codex-cli 0.166.0 | 1 | completed | partial | 0.941 | 9 m 40 s | 514,930 | 12,666 | 527,596 | 25 | ||
| run-20261102-160115-cb48ab31 | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 16:01:15 | freight-shop-001@1.0.0 | codex@codex-cli 0.166.0 | 3 | completed | partial | 0.941 | 10 m 38 s | 402,646 | 15,902 | 418,548 | 17 | ||
| run-20261102-161153-16c04a4e | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 16:11:53 | freight-shop-001@1.0.0 | codex@codex-cli 0.166.0 | 1 | completed | partial | 0.941 | 11 m 36 s | 709,380 | 23,886 | 733,266 | 25 | ||
| run-20261102-164738-43ff8923 | knowledge_enabled=False, model=auto_codex | 2026-11-02 16:47:38 | freight-shop-001@1.0.0 | codex@codex-cli 0.166.0 | 1 | completed | partial | 0.941 | 6 m 29 s | 254,425 | 9,577 | 264,002 | 14 | ||
| run-20261102-165407-d907f577 | knowledge_enabled=False, model=auto_codex | 2026-11-02 16:54:07 | freight-shop-001@1.0.0 | codex@codex-cli 0.166.0 | 2 | completed | partial | 0.941 | 11 m 28 s | 361,986 | 19,008 | 380,994 | 17 | ||
| run-20261102-172849-b26fa228 | knowledge_enabled=True, model=auto_codex | 2026-11-02 17:28:49 | freight-shop-001@1.0.0 | codex@codex-cli 0.166.0 | 2 | completed | partial | 0.941 | 10 m 30 s | 515,522 | 17,044 | 532,566 | 23 |
Not verified
Configurations that are not verified. They stay in the report, because a result produced under them is only as trustworthy as they are.
| kind | subject | status | run | note |
|---|---|---|---|---|
| plugin | agentic_knowledge | unsupported | run-20261102-134326-1f193cec | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-134326-1f193cec | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-145348-58a7a3f3 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-145348-58a7a3f3 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-153629-de5ac66d | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-153629-de5ac66d | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-160115-cb48ab31 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-160115-cb48ab31 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-161153-16c04a4e | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-161153-16c04a4e | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-164738-43ff8923 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-164738-43ff8923 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-165407-d907f577 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-165407-d907f577 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-172849-b26fa228 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-172849-b26fa228 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-134326-1f193cec | |
| model | auto_codex | unverified | run-20261102-145348-58a7a3f3 | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-153629-de5ac66d | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-160115-cb48ab31 | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-161153-16c04a4e | |
| model | auto_codex | unverified | run-20261102-164738-43ff8923 | |
| model | auto_codex | unverified | run-20261102-165407-d907f577 | |
| model | auto_codex | unverified | run-20261102-172849-b26fa228 |