Task
wordcount-001@1.0.0
runs
12
success rate
83.3%
mean time
7 m 17 s
input tokens
880,965
generated tokens
20,251
tokens
901,216
tool calls
102
What the task asks
Word counting must ignore repeated whitespace
count_words splits on single spaces, so empty strings and runs of whitespace are counted wrong. The agent restores the documented behaviour.
full description (markdown source)
# Word counting must ignore repeated whitespace `count_words` in `src/wordcount/text.py` is documented to count whitespace-separated words, and an empty or whitespace-only string must count as zero words. Some inputs do not behave that way. Find out which, and fix the function so it matches its documentation. The public tests in `tests/` keep passing, so they are not enough on their own.
Constraints
- do not change the public API of the wordcount package
- do not modify the tests
Recorded identity
| field | value |
|---|---|
| uid | wordcount-001@1.0.0 |
| category | bug_fix |
| difficulty | easy |
| evaluator | pytest_evaluator |
| base commit | HEAD |
| repository | repo-df7b22d8342e498d |
| snapshot digest | sha256:2651055be454b53cf166c66ef56857397da9b43da426a17154e0c53de564edc7 |
| source | datasets/_sources/wordcount |
| ref | HEAD |
| mutation | none |
| timeout | 300 s |
| hidden tests | hidden_tests |
| public tests | public_tests |
| definition digest | sha256:2651055be454b53cf166c66ef56857397da9b43da426a17154e0c53de564edc7 |
| definition path | /mnt/data2/dev/develop/ak-bench/datasets/bug_fix/wordcount-001 |
Runs of this task
in the order they started; the parameters column is the configuration each run was executed under
| run | parameters | started | agent | repetition | status | verdict | score | time | input tokens | generated tokens | total tokens | tool calls | failure class | error |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| run-20261102-033745-147a6170 | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 03:37:45 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 3 m 10 s | 165,875 | 2,586 | 168,461 | 18 | ||
| run-20261102-034056-a469ad46 | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 03:40:56 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 1 m 35 s | 53,883 | 1,045 | 54,928 | 6 | ||
| run-20261102-034231-ceec4cf9 | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 03:42:31 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 1 m 16 s | 44,338 | 703 | 45,041 | 5 | ||
| run-20261102-034347-4d78071a | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 03:43:47 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 6 m 8 s | 59,593 | 2,202 | 61,795 | 9 | ||
| run-20261102-034956-2f8213a2 | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 03:49:56 | codex@codex-cli 0.166.0 | 2 | timeout | not measured | 15 m 2 s | 83,828 | 2,874 | 86,702 | 12 | agent | The fix is on disk, but git doesn't report `text.py` as modified — verifying against HEAD. | |
| run-20261102-040500-60833b88 | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 04:05:00 | codex@codex-cli 0.166.0 | 3 | timeout | not measured | 15 m 2 s | 29,760 | 803 | 30,563 | 4 | agent | Found it — the working tree modified `text.split()` into `text.split(" ")`. Let me confirm which inputs misbehave. | |
| run-20261102-042003-4e77ac85 | knowledge_enabled=False, model=auto_codex | 2026-11-02 04:20:03 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 8 m 12 s | 65,143 | 1,868 | 67,011 | 7 | ||
| run-20261102-042815-65e2ad9b | knowledge_enabled=False, model=auto_codex | 2026-11-02 04:28:15 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 6 m 51 s | 44,291 | 729 | 45,020 | 5 | ||
| run-20261102-043507-07326b31 | knowledge_enabled=False, model=auto_codex | 2026-11-02 04:35:07 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 11 m 9 s | 91,894 | 2,139 | 94,033 | 10 | ||
| run-20261102-044616-5ced7f24 | knowledge_enabled=True, model=auto_codex | 2026-11-02 04:46:16 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 5 m 20 s | 72,757 | 1,528 | 74,285 | 8 | ||
| run-20261102-045137-ae782a86 | knowledge_enabled=True, model=auto_codex | 2026-11-02 04:51:37 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 8 m 13 s | 57,880 | 1,598 | 59,478 | 6 | ||
| run-20261102-045951-9062760d | knowledge_enabled=True, model=auto_codex | 2026-11-02 04:59:51 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 5 m 25 s | 111,723 | 2,176 | 113,899 | 12 |
Not verified
Configurations that are not verified. They stay in the report, because a result produced under them is only as trustworthy as they are.
| kind | subject | status | run | note |
|---|---|---|---|---|
| plugin | agentic_knowledge | unsupported | run-20261102-033745-147a6170 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-033745-147a6170 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-034056-a469ad46 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-034056-a469ad46 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-034231-ceec4cf9 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-034231-ceec4cf9 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-034347-4d78071a | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-034347-4d78071a | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-034956-2f8213a2 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-034956-2f8213a2 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-040500-60833b88 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-040500-60833b88 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-042003-4e77ac85 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-042003-4e77ac85 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-042815-65e2ad9b | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-042815-65e2ad9b | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-043507-07326b31 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-043507-07326b31 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-044616-5ced7f24 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-044616-5ced7f24 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-045137-ae782a86 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-045137-ae782a86 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-045951-9062760d | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-045951-9062760d | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-033745-147a6170 | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-034056-a469ad46 | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-034231-ceec4cf9 | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-034347-4d78071a | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-034956-2f8213a2 | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-040500-60833b88 | |
| model | auto_codex | unverified | run-20261102-042003-4e77ac85 | |
| model | auto_codex | unverified | run-20261102-042815-65e2ad9b | |
| model | auto_codex | unverified | run-20261102-043507-07326b31 | |
| model | auto_codex | unverified | run-20261102-044616-5ced7f24 | |
| model | auto_codex | unverified | run-20261102-045137-ae782a86 | |
| model | auto_codex | unverified | run-20261102-045951-9062760d |