Variant
v-17e7e69bf087 of router-config-comparison@1.0.0
runs
18
evaluated
16
success rate
88.9%
mean time
6 m 25 s
input tokens
2,371,680
generated tokens
70,279
tokens
2,441,959
tool calls
199
What this recording was
| field | value |
|---|---|
| variant | v-17e7e69bf087 |
| experiment | router-config-comparison@1.0.0 |
| configuration | knowledge_enabled=False, model=auto_codex |
| config hash | sha256:850faa471d75cee92cde85cb1090df5b7235eb3ae61d1d64e9bfd74648f89719 |
| baseline | no |
In the configuration ranking
the ranking states one row per configuration, over all of its recordings; these are the numbers of that row, not of this recording alone
| place | configuration | recordings | runs | evaluated | success rate | mean time | mean input tokens | mean generated tokens | mean total tokens | mean tool calls | against the baseline |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 2 | knowledge_enabled=False, model=auto_codex | 2 | 33 | 29 | 81.8% | 6 m 41 s | 196,230 | 6,933 | 203,163 | 13 | success +3%, time 1.22x, tokens 0.70x |
The other recordings of this configuration
the same values, recorded under another variant uid; the ranking counts this page's runs and these together
| variant | baseline | runs | evaluated | passed | success rate | mean time | input tokens | generated tokens | total tokens | tool calls |
|---|---|---|---|---|---|---|---|---|---|---|
| v-5f6f603add50 | no | 15 | 13 | 11 | 73.3% | 7 m 0 s | 4,103,911 | 158,523 | 4,262,434 | 227 |
Runs of this recording
in the order they started; the parameters column is the configuration each run was executed under
| run | parameters | started | task | agent | repetition | status | verdict | score | time | input tokens | generated tokens | total tokens | tool calls | failure class | error |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| run-20261102-021828-a3b13607 | knowledge_enabled=False, model=auto_codex | 2026-11-02 02:18:28 | bookshop-stock-002@1.0.0 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 4 m 46 s | 241,297 | 4,789 | 246,086 | 17 | ||
| run-20261102-022315-259a16a5 | knowledge_enabled=False, model=auto_codex | 2026-11-02 02:23:15 | bookshop-stock-002@1.0.0 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 3 m 59 s | 128,251 | 3,787 | 132,038 | 11 | ||
| run-20261102-022715-0f5b5375 | knowledge_enabled=False, model=auto_codex | 2026-11-02 02:27:15 | bookshop-stock-002@1.0.0 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 4 m 27 s | 173,650 | 3,854 | 177,504 | 15 | ||
| run-20261102-030330-6e680c03 | knowledge_enabled=False, model=auto_codex | 2026-11-02 03:03:30 | bookshop-tiers-001@1.0.0 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 2 m 49 s | 81,249 | 1,664 | 82,913 | 7 | ||
| run-20261102-030620-307d982c | knowledge_enabled=False, model=auto_codex | 2026-11-02 03:06:20 | bookshop-tiers-001@1.0.0 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 3 m 43 s | 62,588 | 1,462 | 64,050 | 6 | ||
| run-20261102-031005-8ee792e8 | knowledge_enabled=False, model=auto_codex | 2026-11-02 03:10:05 | bookshop-tiers-001@1.0.0 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 2 m 39 s | 52,158 | 1,256 | 53,414 | 5 | ||
| run-20261102-042003-4e77ac85 | knowledge_enabled=False, model=auto_codex | 2026-11-02 04:20:03 | wordcount-001@1.0.0 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 8 m 12 s | 65,143 | 1,868 | 67,011 | 7 | ||
| run-20261102-042815-65e2ad9b | knowledge_enabled=False, model=auto_codex | 2026-11-02 04:28:15 | wordcount-001@1.0.0 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 6 m 51 s | 44,291 | 729 | 45,020 | 5 | ||
| run-20261102-043507-07326b31 | knowledge_enabled=False, model=auto_codex | 2026-11-02 04:35:07 | wordcount-001@1.0.0 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 11 m 9 s | 91,894 | 2,139 | 94,033 | 10 | ||
| run-20261102-060259-4681bb55 | knowledge_enabled=False, model=auto_codex | 2026-11-02 06:02:59 | notes-search-001@1.0.0 | codex@codex-cli 0.166.0 | 1 | timeout | not measured | 15 m 3 s | 92,655 | 5,287 | 97,942 | 11 | agent | Found a real bug: with `for ... else`, my scoring block sits inside `if terms:`, so a blank query appended nothing and returned `[]`. Restructuring so an empty term list scores 0 and matches. | |
| run-20261102-061803-ca793f25 | knowledge_enabled=False, model=auto_codex | 2026-11-02 06:18:03 | notes-search-001@1.0.0 | codex@codex-cli 0.166.0 | 2 | timeout | not measured | 15 m 3 s | 364,794 | 13,619 | 378,413 | 24 | agent | The implementation is correct — the last "failure" was another typo in my hand-written expected list (I swapped `t2`/`t1`), and the final direct probe confirmed the spec order `['t0','d','tg','t2','t1','c','b','a','e','m2','m1']` and correct scoring. Cleaning up the stale `.pyc` churn so the diff is just the implementation: | |
| run-20261102-063306-62006f1c | knowledge_enabled=False, model=auto_codex | 2026-11-02 06:33:06 | notes-search-001@1.0.0 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 6 m 14 s | 127,093 | 4,757 | 131,850 | 12 | ||
| run-20261102-072154-ce0a96c9 | knowledge_enabled=False, model=auto_codex | 2026-11-02 07:21:54 | notes-sync-001@1.0.0 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 9 m 24 s | 292,586 | 6,876 | 299,462 | 21 | ||
| run-20261102-073119-046c63c5 | knowledge_enabled=False, model=auto_codex | 2026-11-02 07:31:19 | notes-sync-001@1.0.0 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 9 m 54 s | 186,195 | 7,893 | 194,088 | 14 | ||
| run-20261102-074114-fb309dd3 | knowledge_enabled=False, model=auto_codex | 2026-11-02 07:41:14 | notes-sync-001@1.0.0 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 4 m 22 s | 139,453 | 4,875 | 144,328 | 11 | ||
| run-20261102-084427-5c4e6c25 | knowledge_enabled=False, model=auto_codex | 2026-11-02 08:44:27 | slugify-001@1.0.0 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 2 m 54 s | 92,917 | 2,289 | 95,206 | 9 | ||
| run-20261102-084722-12cb8519 | knowledge_enabled=False, model=auto_codex | 2026-11-02 08:47:22 | slugify-001@1.0.0 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 2 m 3 s | 68,437 | 1,564 | 70,001 | 7 | ||
| run-20261102-084925-db233c9d | knowledge_enabled=False, model=auto_codex | 2026-11-02 08:49:25 | slugify-001@1.0.0 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 1 m 60 s | 67,029 | 1,571 | 68,600 | 7 |
Not verified
Configurations that are not verified. They stay in the report, because a result produced under them is only as trustworthy as they are.
| kind | subject | status | run | note |
|---|---|---|---|---|
| plugin | agentic_knowledge | unsupported | run-20261102-021828-a3b13607 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-021828-a3b13607 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-022315-259a16a5 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-022315-259a16a5 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-022715-0f5b5375 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-022715-0f5b5375 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-030330-6e680c03 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-030330-6e680c03 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-030620-307d982c | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-030620-307d982c | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-031005-8ee792e8 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-031005-8ee792e8 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-042003-4e77ac85 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-042003-4e77ac85 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-042815-65e2ad9b | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-042815-65e2ad9b | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-043507-07326b31 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-043507-07326b31 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-060259-4681bb55 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-060259-4681bb55 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-061803-ca793f25 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-061803-ca793f25 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-063306-62006f1c | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-063306-62006f1c | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-072154-ce0a96c9 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-072154-ce0a96c9 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-073119-046c63c5 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-073119-046c63c5 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-074114-fb309dd3 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-074114-fb309dd3 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-084427-5c4e6c25 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-084427-5c4e6c25 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-084722-12cb8519 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-084722-12cb8519 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-084925-db233c9d | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-084925-db233c9d | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| model | auto_codex | unverified | run-20261102-021828-a3b13607 | |
| model | auto_codex | unverified | run-20261102-022315-259a16a5 | |
| model | auto_codex | unverified | run-20261102-022715-0f5b5375 | |
| model | auto_codex | unverified | run-20261102-030330-6e680c03 | |
| model | auto_codex | unverified | run-20261102-030620-307d982c | |
| model | auto_codex | unverified | run-20261102-031005-8ee792e8 | |
| model | auto_codex | unverified | run-20261102-042003-4e77ac85 | |
| model | auto_codex | unverified | run-20261102-042815-65e2ad9b | |
| model | auto_codex | unverified | run-20261102-043507-07326b31 | |
| model | auto_codex | unverified | run-20261102-060259-4681bb55 | |
| model | auto_codex | unverified | run-20261102-061803-ca793f25 | |
| model | auto_codex | unverified | run-20261102-063306-62006f1c | |
| model | auto_codex | unverified | run-20261102-072154-ce0a96c9 | |
| model | auto_codex | unverified | run-20261102-073119-046c63c5 | |
| model | auto_codex | unverified | run-20261102-074114-fb309dd3 | |
| model | auto_codex | unverified | run-20261102-084427-5c4e6c25 | |
| model | auto_codex | unverified | run-20261102-084722-12cb8519 | |
| model | auto_codex | unverified | run-20261102-084925-db233c9d |