Variant
v-1bd895ea9679 of router-config-comparison@1.0.0
runs
18
evaluated
14
success rate
77.8%
mean time
7 m 59 s
input tokens
2,655,384
generated tokens
110,250
tokens
2,765,634
tool calls
235
What this recording was
| field | value |
|---|---|
| variant | v-1bd895ea9679 |
| experiment | router-config-comparison@1.0.0 |
| configuration | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next |
| config hash | sha256:18f8d0b48e31c8a666519c0994931d89919e7b85e8548a19eb7f9bbc601c7d71 |
| baseline | no |
In the configuration ranking
the ranking states one row per configuration, over all of its recordings; these are the numbers of that row, not of this recording alone
| place | configuration | recordings | runs | evaluated | success rate | mean time | mean input tokens | mean generated tokens | mean total tokens | mean tool calls | against the baseline |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 4 | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2 | 33 | 27 | 75.8% | 8 m 0 s | 278,068 | 10,337 | 288,405 | 16 | success -3%, time 1.46x, tokens 0.99x |
The other recordings of this configuration
the same values, recorded under another variant uid; the ranking counts this page's runs and these together
| variant | baseline | runs | evaluated | passed | success rate | mean time | input tokens | generated tokens | total tokens | tool calls |
|---|---|---|---|---|---|---|---|---|---|---|
| v-e816befad22e | no | 15 | 13 | 11 | 73.3% | 8 m 2 s | 6,520,869 | 230,871 | 6,751,740 | 295 |
Runs of this recording
in the order they started; the parameters column is the configuration each run was executed under
| run | parameters | started | task | agent | repetition | status | verdict | score | time | input tokens | generated tokens | total tokens | tool calls | failure class | error |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| run-20261102-020305-819a6a56 | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 02:03:05 | bookshop-stock-002@1.0.0 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 6 m 16 s | 254,599 | 10,261 | 264,860 | 20 | ||
| run-20261102-020922-c802a0ee | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 02:09:22 | bookshop-stock-002@1.0.0 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 3 m 56 s | 164,309 | 4,902 | 169,211 | 14 | ||
| run-20261102-021319-f8645ceb | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 02:13:19 | bookshop-stock-002@1.0.0 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 5 m 8 s | 223,668 | 7,523 | 231,191 | 17 | ||
| run-20261102-024900-b4e495b9 | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 02:49:00 | bookshop-tiers-001@1.0.0 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 4 m 48 s | 204,454 | 6,473 | 210,927 | 18 | ||
| run-20261102-025349-a41ee544 | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 02:53:49 | bookshop-tiers-001@1.0.0 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 4 m 48 s | 190,916 | 6,504 | 197,420 | 16 | ||
| run-20261102-025838-8e5d4349 | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 02:58:38 | bookshop-tiers-001@1.0.0 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 4 m 52 s | 185,126 | 6,992 | 192,118 | 17 | ||
| run-20261102-034347-4d78071a | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 03:43:47 | wordcount-001@1.0.0 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 6 m 8 s | 59,593 | 2,202 | 61,795 | 9 | ||
| run-20261102-034956-2f8213a2 | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 03:49:56 | wordcount-001@1.0.0 | codex@codex-cli 0.166.0 | 2 | timeout | not measured | 15 m 2 s | 83,828 | 2,874 | 86,702 | 12 | agent | The fix is on disk, but git doesn't report `text.py` as modified — verifying against HEAD. | |
| run-20261102-040500-60833b88 | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 04:05:00 | wordcount-001@1.0.0 | codex@codex-cli 0.166.0 | 3 | timeout | not measured | 15 m 2 s | 29,760 | 803 | 30,563 | 4 | agent | Found it — the working tree modified `text.split()` into `text.split(" ")`. Let me confirm which inputs misbehave. | |
| run-20261102-052723-7c46d753 | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 05:27:23 | notes-search-001@1.0.0 | codex@codex-cli 0.166.0 | 1 | timeout | not measured | 15 m 3 s | 25,668 | 632 | 26,300 | 6 | agent | I'll start by exploring the repo structure and reading the spec. | |
| run-20261102-054227-dc5f0497 | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 05:42:27 | notes-search-001@1.0.0 | codex@codex-cli 0.166.0 | 2 | timeout | not measured | 15 m 2 s | 68,159 | 4,703 | 72,862 | 9 | agent | Verifying against the spec rules with a throwaway script, plus the repo suite. | |
| run-20261102-055730-c94232f9 | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 05:57:30 | notes-search-001@1.0.0 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 5 m 28 s | 193,962 | 10,301 | 204,263 | 17 | ||
| run-20261102-070225-983f28a4 | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 07:02:25 | notes-sync-001@1.0.0 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 5 m 26 s | 113,987 | 7,555 | 121,542 | 10 | ||
| run-20261102-070752-ee5e374f | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 07:07:52 | notes-sync-001@1.0.0 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 7 m 42 s | 297,815 | 11,995 | 309,810 | 18 | ||
| run-20261102-071535-a34798d9 | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 07:15:35 | notes-sync-001@1.0.0 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 6 m 19 s | 194,746 | 9,719 | 204,465 | 13 | ||
| run-20261102-082149-abdd844b | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 08:21:49 | slugify-001@1.0.0 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 11 m 17 s | 147,994 | 7,191 | 155,185 | 12 | ||
| run-20261102-083306-81799937 | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 08:33:06 | slugify-001@1.0.0 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 2 m 6 s | 77,350 | 2,527 | 79,877 | 8 | ||
| run-20261102-083513-c8a810d4 | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 08:35:13 | slugify-001@1.0.0 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 9 m 13 s | 139,450 | 7,093 | 146,543 | 15 |
Not verified
Configurations that are not verified. They stay in the report, because a result produced under them is only as trustworthy as they are.
| kind | subject | status | run | note |
|---|---|---|---|---|
| plugin | agentic_knowledge | unsupported | run-20261102-020305-819a6a56 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-020305-819a6a56 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-020922-c802a0ee | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-020922-c802a0ee | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-021319-f8645ceb | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-021319-f8645ceb | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-024900-b4e495b9 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-024900-b4e495b9 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-025349-a41ee544 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-025349-a41ee544 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-025838-8e5d4349 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-025838-8e5d4349 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-034347-4d78071a | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-034347-4d78071a | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-034956-2f8213a2 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-034956-2f8213a2 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-040500-60833b88 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-040500-60833b88 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-052723-7c46d753 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-052723-7c46d753 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-054227-dc5f0497 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-054227-dc5f0497 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-055730-c94232f9 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-055730-c94232f9 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-070225-983f28a4 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-070225-983f28a4 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-070752-ee5e374f | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-070752-ee5e374f | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-071535-a34798d9 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-071535-a34798d9 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-082149-abdd844b | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-082149-abdd844b | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-083306-81799937 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-083306-81799937 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-083513-c8a810d4 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-083513-c8a810d4 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-020305-819a6a56 | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-020922-c802a0ee | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-021319-f8645ceb | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-024900-b4e495b9 | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-025349-a41ee544 | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-025838-8e5d4349 | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-034347-4d78071a | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-034956-2f8213a2 | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-040500-60833b88 | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-052723-7c46d753 | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-054227-dc5f0497 | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-055730-c94232f9 | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-070225-983f28a4 | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-070752-ee5e374f | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-071535-a34798d9 | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-082149-abdd844b | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-083306-81799937 | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-083513-c8a810d4 |