Variant
v-5dfe3345dc0f of router-config-comparison@1.0.0
runs
18
evaluated
17
success rate
94.4%
mean time
4 m 10 s
input tokens
3,307,047
generated tokens
77,381
tokens
3,384,428
tool calls
244
What this recording was
| field | value |
|---|---|
| variant | v-5dfe3345dc0f |
| experiment | router-config-comparison@1.0.0 |
| configuration | knowledge_enabled=False, model=qwen/Qwen3.8-27B |
| config hash | sha256:d5703fbe3c7ec9075f0ab4b8a548fc54171589d978b6492e14c31d13b681646b |
| baseline | yes |
In the configuration ranking
the ranking states one row per configuration, over all of its recordings; these are the numbers of that row, not of this recording alone
| place | configuration | recordings | runs | evaluated | success rate | mean time | mean input tokens | mean generated tokens | mean total tokens | mean tool calls | against the baseline |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 3 | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2 | 33 | 28 | 78.8% | 5 m 28 s | 284,222 | 6,408 | 290,630 | 16 |
The other recordings of this configuration
the same values, recorded under another variant uid; the ranking counts this page's runs and these together
| variant | baseline | runs | evaluated | passed | success rate | mean time | input tokens | generated tokens | total tokens | tool calls |
|---|---|---|---|---|---|---|---|---|---|---|
| v-4d3337664165 | yes | 15 | 11 | 9 | 60.0% | 7 m 1 s | 6,072,271 | 134,079 | 6,206,350 | 281 |
Runs of this recording
in the order they started; the parameters column is the configuration each run was executed under
| run | parameters | started | task | agent | repetition | status | verdict | score | time | input tokens | generated tokens | total tokens | tool calls | failure class | error |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| run-20261102-015302-e1907b59 | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 01:53:02 | bookshop-stock-002@1.0.0 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 3 m 59 s | 139,513 | 3,865 | 143,378 | 13 | ||
| run-20261102-015702-033aab98 | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 01:57:02 | bookshop-stock-002@1.0.0 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 2 m 59 s | 110,185 | 2,205 | 112,390 | 10 | ||
| run-20261102-020002-c8680ac1 | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 02:00:02 | bookshop-stock-002@1.0.0 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 3 m 1 s | 118,277 | 2,192 | 120,469 | 9 | ||
| run-20261102-024213-cd12e80f | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 02:42:13 | bookshop-tiers-001@1.0.0 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 1 m 60 s | 51,447 | 826 | 52,273 | 5 | ||
| run-20261102-024414-b4a4c0f1 | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 02:44:14 | bookshop-tiers-001@1.0.0 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 2 m 22 s | 73,253 | 1,407 | 74,660 | 7 | ||
| run-20261102-024637-153cbef0 | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 02:46:37 | bookshop-tiers-001@1.0.0 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 2 m 22 s | 84,790 | 1,310 | 86,100 | 8 | ||
| run-20261102-033745-147a6170 | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 03:37:45 | wordcount-001@1.0.0 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 3 m 10 s | 165,875 | 2,586 | 168,461 | 18 | ||
| run-20261102-034056-a469ad46 | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 03:40:56 | wordcount-001@1.0.0 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 1 m 35 s | 53,883 | 1,045 | 54,928 | 6 | ||
| run-20261102-034231-ceec4cf9 | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 03:42:31 | wordcount-001@1.0.0 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 1 m 16 s | 44,338 | 703 | 45,041 | 5 | ||
| run-20261102-050517-4824113c | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 05:05:17 | notes-search-001@1.0.0 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 6 m 23 s | 257,102 | 7,946 | 265,048 | 19 | ||
| run-20261102-051140-154e7756 | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 05:11:40 | notes-search-001@1.0.0 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 7 m 15 s | 322,255 | 9,563 | 331,818 | 21 | ||
| run-20261102-051856-7a01f2f0 | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 05:18:56 | notes-search-001@1.0.0 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 8 m 27 s | 334,137 | 11,345 | 345,482 | 20 | ||
| run-20261102-065152-be03d69a | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 06:51:52 | notes-sync-001@1.0.0 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 2 m 32 s | 68,691 | 2,284 | 70,975 | 7 | ||
| run-20261102-065424-0122485b | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 06:54:24 | notes-sync-001@1.0.0 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 3 m 58 s | 147,528 | 3,954 | 151,482 | 13 | ||
| run-20261102-065823-56d0518e | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 06:58:23 | notes-sync-001@1.0.0 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 4 m 1 s | 144,209 | 3,841 | 148,050 | 12 | ||
| run-20261102-080204-89ac58c6 | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 08:02:04 | slugify-001@1.0.0 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 2 m 5 s | 88,760 | 1,406 | 90,166 | 9 | ||
| run-20261102-080410-d2c89897 | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 08:04:10 | slugify-001@1.0.0 | codex@codex-cli 0.166.0 | 2 | timeout | not measured | 15 m 2 s | 989,571 | 18,787 | 1,008,358 | 51 | agent | I'm looking at the test results more carefully now—most of them are passing, but there are two specific cases where the hyphen handling is producing unexpected triple hyphens instead of the expected output. The issue seems to be that when I'm building the test strings with the hyphen character, something about how the input is being processed is causing the hyphens to multiply in the output. | |
| run-20261102-081913-0789e06a | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 08:19:13 | slugify-001@1.0.0 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 2 m 35 s | 113,233 | 2,116 | 115,349 | 11 |
Not verified
Configurations that are not verified. They stay in the report, because a result produced under them is only as trustworthy as they are.
| kind | subject | status | run | note |
|---|---|---|---|---|
| plugin | agentic_knowledge | unsupported | run-20261102-015302-e1907b59 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-015302-e1907b59 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-015702-033aab98 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-015702-033aab98 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-020002-c8680ac1 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-020002-c8680ac1 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-024213-cd12e80f | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-024213-cd12e80f | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-024414-b4a4c0f1 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-024414-b4a4c0f1 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-024637-153cbef0 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-024637-153cbef0 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-033745-147a6170 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-033745-147a6170 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-034056-a469ad46 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-034056-a469ad46 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-034231-ceec4cf9 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-034231-ceec4cf9 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-050517-4824113c | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-050517-4824113c | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-051140-154e7756 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-051140-154e7756 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-051856-7a01f2f0 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-051856-7a01f2f0 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-065152-be03d69a | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-065152-be03d69a | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-065424-0122485b | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-065424-0122485b | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-065823-56d0518e | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-065823-56d0518e | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-080204-89ac58c6 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-080204-89ac58c6 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-080410-d2c89897 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-080410-d2c89897 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-081913-0789e06a | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-081913-0789e06a | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-015302-e1907b59 | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-015702-033aab98 | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-020002-c8680ac1 | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-024213-cd12e80f | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-024414-b4a4c0f1 | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-024637-153cbef0 | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-033745-147a6170 | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-034056-a469ad46 | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-034231-ceec4cf9 | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-050517-4824113c | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-051140-154e7756 | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-051856-7a01f2f0 | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-065152-be03d69a | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-065424-0122485b | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-065823-56d0518e | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-080204-89ac58c6 | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-080410-d2c89897 | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-081913-0789e06a |