difficulty
easy
runs
24
evaluated
21
success rate
87.5%
mean time
6 m 11 s
input tokens
2,939,188
generated tokens
72,189
tokens
3,011,377
tool calls
258
Experiments this subject took part in
| experiment | title | type | runs | evaluated | passed | success rate | mean time | input tokens | generated tokens | total tokens | tool calls |
|---|---|---|---|---|---|---|---|---|---|---|---|
| router-config-comparison@1.0.0 | Router models versus Auto Codex, with and without knowledge | configuration | 24 | 21 | 21 | 87.5% | 6 m 11 s | 2,939,188 | 72,189 | 3,011,377 | 258 |
Tasks this subject worked
| task | category | difficulty | runs | passed | success rate | mean time | input tokens | generated tokens | total tokens | tool calls |
|---|---|---|---|---|---|---|---|---|---|---|
| slugify-001@1.0.0 | new_feature | easy | 12 | 11 | 91.7% | 5 m 5 s | 2,058,223 | 51,938 | 2,110,161 | 156 |
| wordcount-001@1.0.0 | bug_fix | easy | 12 | 10 | 83.3% | 7 m 17 s | 880,965 | 20,251 | 901,216 | 102 |
Runs
in the order they started; the parameters column is the configuration each run was executed under
| run | parameters | started | task | agent | repetition | status | verdict | score | time | input tokens | generated tokens | total tokens | tool calls | failure class | error |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| run-20261102-033745-147a6170 | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 03:37:45 | wordcount-001@1.0.0 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 3 m 10 s | 165,875 | 2,586 | 168,461 | 18 | ||
| run-20261102-034056-a469ad46 | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 03:40:56 | wordcount-001@1.0.0 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 1 m 35 s | 53,883 | 1,045 | 54,928 | 6 | ||
| run-20261102-034231-ceec4cf9 | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 03:42:31 | wordcount-001@1.0.0 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 1 m 16 s | 44,338 | 703 | 45,041 | 5 | ||
| run-20261102-034347-4d78071a | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 03:43:47 | wordcount-001@1.0.0 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 6 m 8 s | 59,593 | 2,202 | 61,795 | 9 | ||
| run-20261102-034956-2f8213a2 | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 03:49:56 | wordcount-001@1.0.0 | codex@codex-cli 0.166.0 | 2 | timeout | not measured | 15 m 2 s | 83,828 | 2,874 | 86,702 | 12 | agent | The fix is on disk, but git doesn't report `text.py` as modified — verifying against HEAD. | |
| run-20261102-040500-60833b88 | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 04:05:00 | wordcount-001@1.0.0 | codex@codex-cli 0.166.0 | 3 | timeout | not measured | 15 m 2 s | 29,760 | 803 | 30,563 | 4 | agent | Found it — the working tree modified `text.split()` into `text.split(" ")`. Let me confirm which inputs misbehave. | |
| run-20261102-042003-4e77ac85 | knowledge_enabled=False, model=auto_codex | 2026-11-02 04:20:03 | wordcount-001@1.0.0 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 8 m 12 s | 65,143 | 1,868 | 67,011 | 7 | ||
| run-20261102-042815-65e2ad9b | knowledge_enabled=False, model=auto_codex | 2026-11-02 04:28:15 | wordcount-001@1.0.0 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 6 m 51 s | 44,291 | 729 | 45,020 | 5 | ||
| run-20261102-043507-07326b31 | knowledge_enabled=False, model=auto_codex | 2026-11-02 04:35:07 | wordcount-001@1.0.0 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 11 m 9 s | 91,894 | 2,139 | 94,033 | 10 | ||
| run-20261102-044616-5ced7f24 | knowledge_enabled=True, model=auto_codex | 2026-11-02 04:46:16 | wordcount-001@1.0.0 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 5 m 20 s | 72,757 | 1,528 | 74,285 | 8 | ||
| run-20261102-045137-ae782a86 | knowledge_enabled=True, model=auto_codex | 2026-11-02 04:51:37 | wordcount-001@1.0.0 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 8 m 13 s | 57,880 | 1,598 | 59,478 | 6 | ||
| run-20261102-045951-9062760d | knowledge_enabled=True, model=auto_codex | 2026-11-02 04:59:51 | wordcount-001@1.0.0 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 5 m 25 s | 111,723 | 2,176 | 113,899 | 12 | ||
| run-20261102-080204-89ac58c6 | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 08:02:04 | slugify-001@1.0.0 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 2 m 5 s | 88,760 | 1,406 | 90,166 | 9 | ||
| run-20261102-080410-d2c89897 | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 08:04:10 | slugify-001@1.0.0 | codex@codex-cli 0.166.0 | 2 | timeout | not measured | 15 m 2 s | 989,571 | 18,787 | 1,008,358 | 51 | agent | I'm looking at the test results more carefully now—most of them are passing, but there are two specific cases where the hyphen handling is producing unexpected triple hyphens instead of the expected output. The issue seems to be that when I'm building the test strings with the hyphen character, something about how the input is being processed is causing the hyphens to multiply in the output. | |
| run-20261102-081913-0789e06a | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 08:19:13 | slugify-001@1.0.0 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 2 m 35 s | 113,233 | 2,116 | 115,349 | 11 | ||
| run-20261102-082149-abdd844b | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 08:21:49 | slugify-001@1.0.0 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 11 m 17 s | 147,994 | 7,191 | 155,185 | 12 | ||
| run-20261102-083306-81799937 | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 08:33:06 | slugify-001@1.0.0 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 2 m 6 s | 77,350 | 2,527 | 79,877 | 8 | ||
| run-20261102-083513-c8a810d4 | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 08:35:13 | slugify-001@1.0.0 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 9 m 13 s | 139,450 | 7,093 | 146,543 | 15 | ||
| run-20261102-084427-5c4e6c25 | knowledge_enabled=False, model=auto_codex | 2026-11-02 08:44:27 | slugify-001@1.0.0 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 2 m 54 s | 92,917 | 2,289 | 95,206 | 9 | ||
| run-20261102-084722-12cb8519 | knowledge_enabled=False, model=auto_codex | 2026-11-02 08:47:22 | slugify-001@1.0.0 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 2 m 3 s | 68,437 | 1,564 | 70,001 | 7 | ||
| run-20261102-084925-db233c9d | knowledge_enabled=False, model=auto_codex | 2026-11-02 08:49:25 | slugify-001@1.0.0 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 1 m 60 s | 67,029 | 1,571 | 68,600 | 7 | ||
| run-20261102-085126-7081e5a7 | knowledge_enabled=True, model=auto_codex | 2026-11-02 08:51:26 | slugify-001@1.0.0 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 5 m 50 s | 82,112 | 2,404 | 84,516 | 8 | ||
| run-20261102-085717-2d40efc9 | knowledge_enabled=True, model=auto_codex | 2026-11-02 08:57:17 | slugify-001@1.0.0 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 3 m 35 s | 112,642 | 2,849 | 115,491 | 11 | ||
| run-20261102-090053-801966e8 | knowledge_enabled=True, model=auto_codex | 2026-11-02 09:00:53 | slugify-001@1.0.0 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 2 m 15 s | 78,728 | 2,141 | 80,869 | 8 |
Not verified
Configurations that are not verified. They stay in the report, because a result produced under them is only as trustworthy as they are.
| kind | subject | status | run | note |
|---|---|---|---|---|
| plugin | agentic_knowledge | unsupported | run-20261102-033745-147a6170 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-033745-147a6170 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-034056-a469ad46 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-034056-a469ad46 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-034231-ceec4cf9 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-034231-ceec4cf9 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-034347-4d78071a | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-034347-4d78071a | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-034956-2f8213a2 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-034956-2f8213a2 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-040500-60833b88 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-040500-60833b88 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-042003-4e77ac85 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-042003-4e77ac85 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-042815-65e2ad9b | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-042815-65e2ad9b | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-043507-07326b31 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-043507-07326b31 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-044616-5ced7f24 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-044616-5ced7f24 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-045137-ae782a86 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-045137-ae782a86 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-045951-9062760d | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-045951-9062760d | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-080204-89ac58c6 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-080204-89ac58c6 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-080410-d2c89897 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-080410-d2c89897 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-081913-0789e06a | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-081913-0789e06a | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-082149-abdd844b | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-082149-abdd844b | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-083306-81799937 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-083306-81799937 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-083513-c8a810d4 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-083513-c8a810d4 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-084427-5c4e6c25 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-084427-5c4e6c25 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-084722-12cb8519 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-084722-12cb8519 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-084925-db233c9d | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-084925-db233c9d | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-085126-7081e5a7 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-085126-7081e5a7 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-085717-2d40efc9 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-085717-2d40efc9 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-090053-801966e8 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-090053-801966e8 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-033745-147a6170 | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-034056-a469ad46 | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-034231-ceec4cf9 | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-034347-4d78071a | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-034956-2f8213a2 | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-040500-60833b88 | |
| model | auto_codex | unverified | run-20261102-042003-4e77ac85 | |
| model | auto_codex | unverified | run-20261102-042815-65e2ad9b | |
| model | auto_codex | unverified | run-20261102-043507-07326b31 | |
| model | auto_codex | unverified | run-20261102-044616-5ced7f24 | |
| model | auto_codex | unverified | run-20261102-045137-ae782a86 | |
| model | auto_codex | unverified | run-20261102-045951-9062760d | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-080204-89ac58c6 | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-080410-d2c89897 | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-081913-0789e06a | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-082149-abdd844b | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-083306-81799937 | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-083513-c8a810d4 | |
| model | auto_codex | unverified | run-20261102-084427-5c4e6c25 | |
| model | auto_codex | unverified | run-20261102-084722-12cb8519 | |
| model | auto_codex | unverified | run-20261102-084925-db233c9d | |
| model | auto_codex | unverified | run-20261102-085126-7081e5a7 | |
| model | auto_codex | unverified | run-20261102-085717-2d40efc9 | |
| model | auto_codex | unverified | run-20261102-090053-801966e8 |