difficulty
hard
runs
24
evaluated
20
success rate
83.3%
mean time
6 m 44 s
input tokens
3,408,998
generated tokens
115,922
tokens
3,524,920
tool calls
286
Experiments this subject took part in
| experiment | title | type | runs | evaluated | passed | success rate | mean time | input tokens | generated tokens | total tokens | tool calls |
|---|---|---|---|---|---|---|---|---|---|---|---|
| router-config-comparison@1.0.0 | Router models versus Auto Codex, with and without knowledge | configuration | 24 | 20 | 20 | 83.3% | 6 m 44 s | 3,408,998 | 115,922 | 3,524,920 | 286 |
Tasks this subject worked
| task | category | difficulty | runs | passed | success rate | mean time | input tokens | generated tokens | total tokens | tool calls |
|---|---|---|---|---|---|---|---|---|---|---|
| bookshop-tiers-001@1.0.0 | bug_fix | hard | 12 | 12 | 100.0% | 4 m 37 s | 1,224,693 | 32,561 | 1,257,254 | 111 |
| notes-search-001@1.0.0 | new_feature | hard | 12 | 8 | 66.7% | 8 m 52 s | 2,184,305 | 83,361 | 2,267,666 | 175 |
Runs
in the order they started; the parameters column is the configuration each run was executed under
| run | parameters | started | task | agent | repetition | status | verdict | score | time | input tokens | generated tokens | total tokens | tool calls | failure class | error |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| run-20261102-024213-cd12e80f | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 02:42:13 | bookshop-tiers-001@1.0.0 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 1 m 60 s | 51,447 | 826 | 52,273 | 5 | ||
| run-20261102-024414-b4a4c0f1 | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 02:44:14 | bookshop-tiers-001@1.0.0 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 2 m 22 s | 73,253 | 1,407 | 74,660 | 7 | ||
| run-20261102-024637-153cbef0 | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 02:46:37 | bookshop-tiers-001@1.0.0 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 2 m 22 s | 84,790 | 1,310 | 86,100 | 8 | ||
| run-20261102-024900-b4e495b9 | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 02:49:00 | bookshop-tiers-001@1.0.0 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 4 m 48 s | 204,454 | 6,473 | 210,927 | 18 | ||
| run-20261102-025349-a41ee544 | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 02:53:49 | bookshop-tiers-001@1.0.0 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 4 m 48 s | 190,916 | 6,504 | 197,420 | 16 | ||
| run-20261102-025838-8e5d4349 | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 02:58:38 | bookshop-tiers-001@1.0.0 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 4 m 52 s | 185,126 | 6,992 | 192,118 | 17 | ||
| run-20261102-030330-6e680c03 | knowledge_enabled=False, model=auto_codex | 2026-11-02 03:03:30 | bookshop-tiers-001@1.0.0 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 2 m 49 s | 81,249 | 1,664 | 82,913 | 7 | ||
| run-20261102-030620-307d982c | knowledge_enabled=False, model=auto_codex | 2026-11-02 03:06:20 | bookshop-tiers-001@1.0.0 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 3 m 43 s | 62,588 | 1,462 | 64,050 | 6 | ||
| run-20261102-031005-8ee792e8 | knowledge_enabled=False, model=auto_codex | 2026-11-02 03:10:05 | bookshop-tiers-001@1.0.0 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 2 m 39 s | 52,158 | 1,256 | 53,414 | 5 | ||
| run-20261102-031245-d48061aa | knowledge_enabled=True, model=auto_codex | 2026-11-02 03:12:45 | bookshop-tiers-001@1.0.0 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 7 m 31 s | 65,070 | 1,527 | 66,597 | 6 | ||
| run-20261102-032017-d7e7635d | knowledge_enabled=True, model=auto_codex | 2026-11-02 03:20:17 | bookshop-tiers-001@1.0.0 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 6 m 36 s | 84,442 | 1,529 | 85,971 | 8 | ||
| run-20261102-032654-eef72cc9 | knowledge_enabled=True, model=auto_codex | 2026-11-02 03:26:54 | bookshop-tiers-001@1.0.0 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 10 m 50 s | 89,200 | 1,611 | 90,811 | 8 | ||
| run-20261102-050517-4824113c | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 05:05:17 | notes-search-001@1.0.0 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 6 m 23 s | 257,102 | 7,946 | 265,048 | 19 | ||
| run-20261102-051140-154e7756 | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 05:11:40 | notes-search-001@1.0.0 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 7 m 15 s | 322,255 | 9,563 | 331,818 | 21 | ||
| run-20261102-051856-7a01f2f0 | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 05:18:56 | notes-search-001@1.0.0 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 8 m 27 s | 334,137 | 11,345 | 345,482 | 20 | ||
| run-20261102-052723-7c46d753 | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 05:27:23 | notes-search-001@1.0.0 | codex@codex-cli 0.166.0 | 1 | timeout | not measured | 15 m 3 s | 25,668 | 632 | 26,300 | 6 | agent | I'll start by exploring the repo structure and reading the spec. | |
| run-20261102-054227-dc5f0497 | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 05:42:27 | notes-search-001@1.0.0 | codex@codex-cli 0.166.0 | 2 | timeout | not measured | 15 m 2 s | 68,159 | 4,703 | 72,862 | 9 | agent | Verifying against the spec rules with a throwaway script, plus the repo suite. | |
| run-20261102-055730-c94232f9 | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 05:57:30 | notes-search-001@1.0.0 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 5 m 28 s | 193,962 | 10,301 | 204,263 | 17 | ||
| run-20261102-060259-4681bb55 | knowledge_enabled=False, model=auto_codex | 2026-11-02 06:02:59 | notes-search-001@1.0.0 | codex@codex-cli 0.166.0 | 1 | timeout | not measured | 15 m 3 s | 92,655 | 5,287 | 97,942 | 11 | agent | Found a real bug: with `for ... else`, my scoring block sits inside `if terms:`, so a blank query appended nothing and returned `[]`. Restructuring so an empty term list scores 0 and matches. | |
| run-20261102-061803-ca793f25 | knowledge_enabled=False, model=auto_codex | 2026-11-02 06:18:03 | notes-search-001@1.0.0 | codex@codex-cli 0.166.0 | 2 | timeout | not measured | 15 m 3 s | 364,794 | 13,619 | 378,413 | 24 | agent | The implementation is correct — the last "failure" was another typo in my hand-written expected list (I swapped `t2`/`t1`), and the final direct probe confirmed the spec order `['t0','d','tg','t2','t1','c','b','a','e','m2','m1']` and correct scoring. Cleaning up the stale `.pyc` churn so the diff is just the implementation: | |
| run-20261102-063306-62006f1c | knowledge_enabled=False, model=auto_codex | 2026-11-02 06:33:06 | notes-search-001@1.0.0 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 6 m 14 s | 127,093 | 4,757 | 131,850 | 12 | ||
| run-20261102-063921-31a5efef | knowledge_enabled=True, model=auto_codex | 2026-11-02 06:39:21 | notes-search-001@1.0.0 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 4 m 21 s | 151,027 | 5,654 | 156,681 | 13 | ||
| run-20261102-064343-33d90543 | knowledge_enabled=True, model=auto_codex | 2026-11-02 06:43:43 | notes-search-001@1.0.0 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 2 m 18 s | 56,625 | 2,030 | 58,655 | 8 | ||
| run-20261102-064601-0d71057a | knowledge_enabled=True, model=auto_codex | 2026-11-02 06:46:01 | notes-search-001@1.0.0 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 5 m 50 s | 190,828 | 7,524 | 198,352 | 15 |
Not verified
Configurations that are not verified. They stay in the report, because a result produced under them is only as trustworthy as they are.
| kind | subject | status | run | note |
|---|---|---|---|---|
| plugin | agentic_knowledge | unsupported | run-20261102-024213-cd12e80f | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-024213-cd12e80f | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-024414-b4a4c0f1 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-024414-b4a4c0f1 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-024637-153cbef0 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-024637-153cbef0 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-024900-b4e495b9 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-024900-b4e495b9 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-025349-a41ee544 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-025349-a41ee544 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-025838-8e5d4349 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-025838-8e5d4349 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-030330-6e680c03 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-030330-6e680c03 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-030620-307d982c | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-030620-307d982c | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-031005-8ee792e8 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-031005-8ee792e8 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-031245-d48061aa | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-031245-d48061aa | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-032017-d7e7635d | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-032017-d7e7635d | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-032654-eef72cc9 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-032654-eef72cc9 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-050517-4824113c | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-050517-4824113c | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-051140-154e7756 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-051140-154e7756 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-051856-7a01f2f0 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-051856-7a01f2f0 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-052723-7c46d753 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-052723-7c46d753 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-054227-dc5f0497 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-054227-dc5f0497 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-055730-c94232f9 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-055730-c94232f9 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-060259-4681bb55 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-060259-4681bb55 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-061803-ca793f25 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-061803-ca793f25 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-063306-62006f1c | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-063306-62006f1c | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-063921-31a5efef | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-063921-31a5efef | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-064343-33d90543 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-064343-33d90543 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-064601-0d71057a | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-064601-0d71057a | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-024213-cd12e80f | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-024414-b4a4c0f1 | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-024637-153cbef0 | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-024900-b4e495b9 | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-025349-a41ee544 | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-025838-8e5d4349 | |
| model | auto_codex | unverified | run-20261102-030330-6e680c03 | |
| model | auto_codex | unverified | run-20261102-030620-307d982c | |
| model | auto_codex | unverified | run-20261102-031005-8ee792e8 | |
| model | auto_codex | unverified | run-20261102-031245-d48061aa | |
| model | auto_codex | unverified | run-20261102-032017-d7e7635d | |
| model | auto_codex | unverified | run-20261102-032654-eef72cc9 | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-050517-4824113c | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-051140-154e7756 | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-051856-7a01f2f0 | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-052723-7c46d753 | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-054227-dc5f0497 | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-055730-c94232f9 | |
| model | auto_codex | unverified | run-20261102-060259-4681bb55 | |
| model | auto_codex | unverified | run-20261102-061803-ca793f25 | |
| model | auto_codex | unverified | run-20261102-063306-62006f1c | |
| model | auto_codex | unverified | run-20261102-063921-31a5efef | |
| model | auto_codex | unverified | run-20261102-064343-33d90543 | |
| model | auto_codex | unverified | run-20261102-064601-0d71057a |