Variant
v-a0ed565bcc26 of router-config-comparison@1.0.0
runs
18
evaluated
18
success rate
100.0%
mean time
5 m 16 s
input tokens
1,952,135
generated tokens
56,493
tokens
2,008,628
tool calls
178
What this recording was
| field | value |
|---|---|
| variant | v-a0ed565bcc26 |
| experiment | router-config-comparison@1.0.0 |
| configuration | knowledge_enabled=True, model=auto_codex |
| config hash | sha256:9ffbba6932454dede1a53082a2faeb161bf15b8d342ee4237cc6bfdacb20e80c |
| baseline | no |
In the configuration ranking
the ranking states one row per configuration, over all of its recordings; these are the numbers of that row, not of this recording alone
| place | configuration | recordings | runs | evaluated | success rate | mean time | mean input tokens | mean generated tokens | mean total tokens | mean tool calls | against the baseline |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | knowledge_enabled=True, model=auto_codex | 2 | 33 | 30 | 84.8% | 6 m 25 s | 218,490 | 7,243 | 225,734 | 14 | success +6%, time 1.17x, tokens 0.78x |
The other recordings of this configuration
the same values, recorded under another variant uid; the ranking counts this page's runs and these together
| variant | baseline | runs | evaluated | passed | success rate | mean time | input tokens | generated tokens | total tokens | tool calls |
|---|---|---|---|---|---|---|---|---|---|---|
| v-dccd72463be1 | no | 15 | 12 | 10 | 66.7% | 7 m 47 s | 5,258,048 | 182,534 | 5,440,582 | 276 |
Runs of this recording
in the order they started; the parameters column is the configuration each run was executed under
| run | parameters | started | task | agent | repetition | status | verdict | score | time | input tokens | generated tokens | total tokens | tool calls | failure class | error |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| run-20261102-023143-af17689a | knowledge_enabled=True, model=auto_codex | 2026-11-02 02:31:43 | bookshop-stock-002@1.0.0 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 3 m 31 s | 124,678 | 3,136 | 127,814 | 10 | ||
| run-20261102-023515-f1dff616 | knowledge_enabled=True, model=auto_codex | 2026-11-02 02:35:15 | bookshop-stock-002@1.0.0 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 2 m 57 s | 107,513 | 2,363 | 109,876 | 11 | ||
| run-20261102-023813-b7d6ced3 | knowledge_enabled=True, model=auto_codex | 2026-11-02 02:38:13 | bookshop-stock-002@1.0.0 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 3 m 58 s | 180,199 | 3,237 | 183,436 | 13 | ||
| run-20261102-031245-d48061aa | knowledge_enabled=True, model=auto_codex | 2026-11-02 03:12:45 | bookshop-tiers-001@1.0.0 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 7 m 31 s | 65,070 | 1,527 | 66,597 | 6 | ||
| run-20261102-032017-d7e7635d | knowledge_enabled=True, model=auto_codex | 2026-11-02 03:20:17 | bookshop-tiers-001@1.0.0 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 6 m 36 s | 84,442 | 1,529 | 85,971 | 8 | ||
| run-20261102-032654-eef72cc9 | knowledge_enabled=True, model=auto_codex | 2026-11-02 03:26:54 | bookshop-tiers-001@1.0.0 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 10 m 50 s | 89,200 | 1,611 | 90,811 | 8 | ||
| run-20261102-044616-5ced7f24 | knowledge_enabled=True, model=auto_codex | 2026-11-02 04:46:16 | wordcount-001@1.0.0 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 5 m 20 s | 72,757 | 1,528 | 74,285 | 8 | ||
| run-20261102-045137-ae782a86 | knowledge_enabled=True, model=auto_codex | 2026-11-02 04:51:37 | wordcount-001@1.0.0 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 8 m 13 s | 57,880 | 1,598 | 59,478 | 6 | ||
| run-20261102-045951-9062760d | knowledge_enabled=True, model=auto_codex | 2026-11-02 04:59:51 | wordcount-001@1.0.0 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 5 m 25 s | 111,723 | 2,176 | 113,899 | 12 | ||
| run-20261102-063921-31a5efef | knowledge_enabled=True, model=auto_codex | 2026-11-02 06:39:21 | notes-search-001@1.0.0 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 4 m 21 s | 151,027 | 5,654 | 156,681 | 13 | ||
| run-20261102-064343-33d90543 | knowledge_enabled=True, model=auto_codex | 2026-11-02 06:43:43 | notes-search-001@1.0.0 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 2 m 18 s | 56,625 | 2,030 | 58,655 | 8 | ||
| run-20261102-064601-0d71057a | knowledge_enabled=True, model=auto_codex | 2026-11-02 06:46:01 | notes-search-001@1.0.0 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 5 m 50 s | 190,828 | 7,524 | 198,352 | 15 | ||
| run-20261102-074537-1abe2bd8 | knowledge_enabled=True, model=auto_codex | 2026-11-02 07:45:37 | notes-sync-001@1.0.0 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 7 m 49 s | 168,258 | 6,167 | 174,425 | 12 | ||
| run-20261102-075327-d3b09ae5 | knowledge_enabled=True, model=auto_codex | 2026-11-02 07:53:27 | notes-sync-001@1.0.0 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 4 m 8 s | 92,335 | 4,290 | 96,625 | 11 | ||
| run-20261102-075736-7dc8b5fa | knowledge_enabled=True, model=auto_codex | 2026-11-02 07:57:36 | notes-sync-001@1.0.0 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 4 m 27 s | 126,118 | 4,729 | 130,847 | 10 | ||
| run-20261102-085126-7081e5a7 | knowledge_enabled=True, model=auto_codex | 2026-11-02 08:51:26 | slugify-001@1.0.0 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 5 m 50 s | 82,112 | 2,404 | 84,516 | 8 | ||
| run-20261102-085717-2d40efc9 | knowledge_enabled=True, model=auto_codex | 2026-11-02 08:57:17 | slugify-001@1.0.0 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 3 m 35 s | 112,642 | 2,849 | 115,491 | 11 | ||
| run-20261102-090053-801966e8 | knowledge_enabled=True, model=auto_codex | 2026-11-02 09:00:53 | slugify-001@1.0.0 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 2 m 15 s | 78,728 | 2,141 | 80,869 | 8 |
Not verified
Configurations that are not verified. They stay in the report, because a result produced under them is only as trustworthy as they are.
| kind | subject | status | run | note |
|---|---|---|---|---|
| plugin | agentic_knowledge | unsupported | run-20261102-023143-af17689a | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-023143-af17689a | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-023515-f1dff616 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-023515-f1dff616 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-023813-b7d6ced3 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-023813-b7d6ced3 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-031245-d48061aa | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-031245-d48061aa | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-032017-d7e7635d | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-032017-d7e7635d | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-032654-eef72cc9 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-032654-eef72cc9 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-044616-5ced7f24 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-044616-5ced7f24 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-045137-ae782a86 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-045137-ae782a86 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-045951-9062760d | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-045951-9062760d | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-063921-31a5efef | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-063921-31a5efef | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-064343-33d90543 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-064343-33d90543 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-064601-0d71057a | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-064601-0d71057a | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-074537-1abe2bd8 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-074537-1abe2bd8 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-075327-d3b09ae5 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-075327-d3b09ae5 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-075736-7dc8b5fa | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-075736-7dc8b5fa | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-085126-7081e5a7 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-085126-7081e5a7 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-085717-2d40efc9 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-085717-2d40efc9 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-090053-801966e8 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-090053-801966e8 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| model | auto_codex | unverified | run-20261102-023143-af17689a | |
| model | auto_codex | unverified | run-20261102-023515-f1dff616 | |
| model | auto_codex | unverified | run-20261102-023813-b7d6ced3 | |
| model | auto_codex | unverified | run-20261102-031245-d48061aa | |
| model | auto_codex | unverified | run-20261102-032017-d7e7635d | |
| model | auto_codex | unverified | run-20261102-032654-eef72cc9 | |
| model | auto_codex | unverified | run-20261102-044616-5ced7f24 | |
| model | auto_codex | unverified | run-20261102-045137-ae782a86 | |
| model | auto_codex | unverified | run-20261102-045951-9062760d | |
| model | auto_codex | unverified | run-20261102-063921-31a5efef | |
| model | auto_codex | unverified | run-20261102-064343-33d90543 | |
| model | auto_codex | unverified | run-20261102-064601-0d71057a | |
| model | auto_codex | unverified | run-20261102-074537-1abe2bd8 | |
| model | auto_codex | unverified | run-20261102-075327-d3b09ae5 | |
| model | auto_codex | unverified | run-20261102-075736-7dc8b5fa | |
| model | auto_codex | unverified | run-20261102-085126-7081e5a7 | |
| model | auto_codex | unverified | run-20261102-085717-2d40efc9 | |
| model | auto_codex | unverified | run-20261102-090053-801966e8 |