Task
bookshop-stock-002@1.0.0
runs
12
success rate
100.0%
mean time
4 m 5 s
input tokens
1,966,139
generated tokens
52,114
tokens
2,018,253
tool calls
160
What the task asks
Cancelling a confirmed order leaves the stock reserved
Cancelling an order that was already confirmed moves it to cancelled without returning the reserved stock to the catalogue. Restore the documented rule that a cancellation gives back what a confirmation took.
full description (markdown source)
# Cancelling a confirmed order leaves the stock reserved The catalogue is a Flask service in `src/bookshop`. Orders live in `src/bookshop/orders.py` and the stock they act on lives in `src/bookshop/catalog.py`; both modules document their rules, and the rules are the specification. Read them first. The invariant the service promises is: - **confirm** reserves the stock every line needs; - **ship** moves reserved stock out of the catalogue; - **cancel** returns reserved stock to the catalogue; - the available stock is always `stock_on_hand - reserved`. One of those promises is broken. Find which one, and repair it. ## What to check Write down the stock counts before and after each transition and compare them with what the docstrings promise. Do it for both kinds of cancellation: an order that was never confirmed, and an order that was. Then do it through the HTTP routes as well, because a route is allowed to be wrong in a way the service is not. ## Boundaries - Do not change the transition table: which statuses may follow which stays as documented. - Do not modify the repository test suite under `tests/`. - The public API of the `bookshop` package stays as it is. - The repository's own test suite has to stay green after your change.
Constraints
- do not change the public API of the bookshop package
- do not modify the repository test suite under tests/
- do not change the transition table
- do not add third-party dependencies
Recorded identity
| field | value |
|---|---|
| uid | bookshop-stock-002@1.0.0 |
| category | bug_fix |
| difficulty | expert |
| evaluator | pytest_evaluator |
| base commit | HEAD |
| repository | repo-8b7a77081db02132 |
| snapshot digest | sha256:b624c69ffb2fce8548877df78050047e50a7cf9860dc7cb726150fb707cac4e5 |
| source | datasets/_sources/bookshop |
| ref | HEAD |
| mutation | none |
| timeout | 900 s |
| hidden tests | hidden_tests |
| public tests | public_tests |
| definition digest | sha256:b624c69ffb2fce8548877df78050047e50a7cf9860dc7cb726150fb707cac4e5 |
| definition path | /mnt/data2/dev/develop/ak-bench/datasets/bug_fix/bookshop-stock-002 |
Runs of this task
in the order they started; the parameters column is the configuration each run was executed under
| run | parameters | started | agent | repetition | status | verdict | score | time | input tokens | generated tokens | total tokens | tool calls | failure class | error |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| run-20261102-015302-e1907b59 | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 01:53:02 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 3 m 59 s | 139,513 | 3,865 | 143,378 | 13 | ||
| run-20261102-015702-033aab98 | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 01:57:02 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 2 m 59 s | 110,185 | 2,205 | 112,390 | 10 | ||
| run-20261102-020002-c8680ac1 | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 02:00:02 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 3 m 1 s | 118,277 | 2,192 | 120,469 | 9 | ||
| run-20261102-020305-819a6a56 | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 02:03:05 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 6 m 16 s | 254,599 | 10,261 | 264,860 | 20 | ||
| run-20261102-020922-c802a0ee | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 02:09:22 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 3 m 56 s | 164,309 | 4,902 | 169,211 | 14 | ||
| run-20261102-021319-f8645ceb | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 02:13:19 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 5 m 8 s | 223,668 | 7,523 | 231,191 | 17 | ||
| run-20261102-021828-a3b13607 | knowledge_enabled=False, model=auto_codex | 2026-11-02 02:18:28 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 4 m 46 s | 241,297 | 4,789 | 246,086 | 17 | ||
| run-20261102-022315-259a16a5 | knowledge_enabled=False, model=auto_codex | 2026-11-02 02:23:15 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 3 m 59 s | 128,251 | 3,787 | 132,038 | 11 | ||
| run-20261102-022715-0f5b5375 | knowledge_enabled=False, model=auto_codex | 2026-11-02 02:27:15 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 4 m 27 s | 173,650 | 3,854 | 177,504 | 15 | ||
| run-20261102-023143-af17689a | knowledge_enabled=True, model=auto_codex | 2026-11-02 02:31:43 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 3 m 31 s | 124,678 | 3,136 | 127,814 | 10 | ||
| run-20261102-023515-f1dff616 | knowledge_enabled=True, model=auto_codex | 2026-11-02 02:35:15 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 2 m 57 s | 107,513 | 2,363 | 109,876 | 11 | ||
| run-20261102-023813-b7d6ced3 | knowledge_enabled=True, model=auto_codex | 2026-11-02 02:38:13 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 3 m 58 s | 180,199 | 3,237 | 183,436 | 13 |
Not verified
Configurations that are not verified. They stay in the report, because a result produced under them is only as trustworthy as they are.
| kind | subject | status | run | note |
|---|---|---|---|---|
| plugin | agentic_knowledge | unsupported | run-20261102-015302-e1907b59 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-015302-e1907b59 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-015702-033aab98 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-015702-033aab98 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-020002-c8680ac1 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-020002-c8680ac1 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-020305-819a6a56 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-020305-819a6a56 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-020922-c802a0ee | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-020922-c802a0ee | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-021319-f8645ceb | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-021319-f8645ceb | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-021828-a3b13607 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-021828-a3b13607 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-022315-259a16a5 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-022315-259a16a5 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-022715-0f5b5375 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-022715-0f5b5375 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-023143-af17689a | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-023143-af17689a | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-023515-f1dff616 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-023515-f1dff616 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-023813-b7d6ced3 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-023813-b7d6ced3 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-015302-e1907b59 | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-015702-033aab98 | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-020002-c8680ac1 | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-020305-819a6a56 | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-020922-c802a0ee | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-021319-f8645ceb | |
| model | auto_codex | unverified | run-20261102-021828-a3b13607 | |
| model | auto_codex | unverified | run-20261102-022315-259a16a5 | |
| model | auto_codex | unverified | run-20261102-022715-0f5b5375 | |
| model | auto_codex | unverified | run-20261102-023143-af17689a | |
| model | auto_codex | unverified | run-20261102-023515-f1dff616 | |
| model | auto_codex | unverified | run-20261102-023813-b7d6ced3 |