AK-Benchbenchmark report

Task

bookshop-stock-002@1.0.0

runs
12
success rate
100.0%
mean time
4 m 5 s
input tokens
1,966,139
generated tokens
52,114
tokens
2,018,253
tool calls
160

What the task asks

Cancelling a confirmed order leaves the stock reserved

Cancelling an order that was already confirmed moves it to cancelled without returning the reserved stock to the catalogue. Restore the documented rule that a cancellation gives back what a confirmation took.

full description (markdown source)
# Cancelling a confirmed order leaves the stock reserved

The catalogue is a Flask service in `src/bookshop`. Orders live in
`src/bookshop/orders.py` and the stock they act on lives in
`src/bookshop/catalog.py`; both modules document their rules, and the rules are
the specification. Read them first.

The invariant the service promises is:

- **confirm** reserves the stock every line needs;
- **ship** moves reserved stock out of the catalogue;
- **cancel** returns reserved stock to the catalogue;
- the available stock is always `stock_on_hand - reserved`.

One of those promises is broken. Find which one, and repair it.

## What to check

Write down the stock counts before and after each transition and compare them
with what the docstrings promise. Do it for both kinds of cancellation: an order
that was never confirmed, and an order that was. Then do it through the HTTP
routes as well, because a route is allowed to be wrong in a way the service is
not.

## Boundaries

- Do not change the transition table: which statuses may follow which stays as
  documented.
- Do not modify the repository test suite under `tests/`.
- The public API of the `bookshop` package stays as it is.
- The repository's own test suite has to stay green after your change.

Constraints

Recorded identity

fieldvalue
uidbookshop-stock-002@1.0.0
categorybug_fix
difficultyexpert
evaluatorpytest_evaluator
base commitHEAD
repositoryrepo-8b7a77081db02132
snapshot digestsha256:b624c69ffb2fce8548877df78050047e50a7cf9860dc7cb726150fb707cac4e5
sourcedatasets/_sources/bookshop
refHEAD
mutationnone
timeout900 s
hidden testshidden_tests
public testspublic_tests
definition digestsha256:b624c69ffb2fce8548877df78050047e50a7cf9860dc7cb726150fb707cac4e5
definition path/mnt/data2/dev/develop/ak-bench/datasets/bug_fix/bookshop-stock-002

Runs of this task

in the order they started; the parameters column is the configuration each run was executed under

runparametersstartedagentrepetitionstatusverdictscoretimeinput tokensgenerated tokenstotal tokenstool callsfailure classerror
run-20261102-015302-e1907b59knowledge_enabled=False, model=qwen/Qwen3.8-27B2026-11-02 01:53:02codex@codex-cli 0.166.01completedpassed1.0003 m 59 s139,5133,865143,37813
run-20261102-015702-033aab98knowledge_enabled=False, model=qwen/Qwen3.8-27B2026-11-02 01:57:02codex@codex-cli 0.166.02completedpassed1.0002 m 59 s110,1852,205112,39010
run-20261102-020002-c8680ac1knowledge_enabled=False, model=qwen/Qwen3.8-27B2026-11-02 02:00:02codex@codex-cli 0.166.03completedpassed1.0003 m 1 s118,2772,192120,4699
run-20261102-020305-819a6a56knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next2026-11-02 02:03:05codex@codex-cli 0.166.01completedpassed1.0006 m 16 s254,59910,261264,86020
run-20261102-020922-c802a0eeknowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next2026-11-02 02:09:22codex@codex-cli 0.166.02completedpassed1.0003 m 56 s164,3094,902169,21114
run-20261102-021319-f8645cebknowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next2026-11-02 02:13:19codex@codex-cli 0.166.03completedpassed1.0005 m 8 s223,6687,523231,19117
run-20261102-021828-a3b13607knowledge_enabled=False, model=auto_codex2026-11-02 02:18:28codex@codex-cli 0.166.01completedpassed1.0004 m 46 s241,2974,789246,08617
run-20261102-022315-259a16a5knowledge_enabled=False, model=auto_codex2026-11-02 02:23:15codex@codex-cli 0.166.02completedpassed1.0003 m 59 s128,2513,787132,03811
run-20261102-022715-0f5b5375knowledge_enabled=False, model=auto_codex2026-11-02 02:27:15codex@codex-cli 0.166.03completedpassed1.0004 m 27 s173,6503,854177,50415
run-20261102-023143-af17689aknowledge_enabled=True, model=auto_codex2026-11-02 02:31:43codex@codex-cli 0.166.01completedpassed1.0003 m 31 s124,6783,136127,81410
run-20261102-023515-f1dff616knowledge_enabled=True, model=auto_codex2026-11-02 02:35:15codex@codex-cli 0.166.02completedpassed1.0002 m 57 s107,5132,363109,87611
run-20261102-023813-b7d6ced3knowledge_enabled=True, model=auto_codex2026-11-02 02:38:13codex@codex-cli 0.166.03completedpassed1.0003 m 58 s180,1993,237183,43613

Not verified

Configurations that are not verified. They stay in the report, because a result produced under them is only as trustworthy as they are.
kindsubjectstatusrunnote
pluginagentic_knowledgeunsupportedrun-20261102-015302-e1907b59LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-015302-e1907b59descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-015702-033aab98LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-015702-033aab98descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-020002-c8680ac1LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-020002-c8680ac1descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-020305-819a6a56LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-020305-819a6a56descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-020922-c802a0eeLLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-020922-c802a0eedescriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-021319-f8645cebLLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-021319-f8645cebdescriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-021828-a3b13607LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-021828-a3b13607descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-022315-259a16a5LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-022315-259a16a5descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-022715-0f5b5375LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-022715-0f5b5375descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-023143-af17689aLLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-023143-af17689adescriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-023515-f1dff616LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-023515-f1dff616descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-023813-b7d6ced3LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-023813-b7d6ced3descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
modelqwen/Qwen3.8-27Bunverifiedrun-20261102-015302-e1907b59
modelqwen/Qwen3.8-27Bunverifiedrun-20261102-015702-033aab98
modelqwen/Qwen3.8-27Bunverifiedrun-20261102-020002-c8680ac1
modelqwen/Qwen3.8-Flash-Nextunverifiedrun-20261102-020305-819a6a56
modelqwen/Qwen3.8-Flash-Nextunverifiedrun-20261102-020922-c802a0ee
modelqwen/Qwen3.8-Flash-Nextunverifiedrun-20261102-021319-f8645ceb
modelauto_codexunverifiedrun-20261102-021828-a3b13607
modelauto_codexunverifiedrun-20261102-022315-259a16a5
modelauto_codexunverifiedrun-20261102-022715-0f5b5375
modelauto_codexunverifiedrun-20261102-023143-af17689a
modelauto_codexunverifiedrun-20261102-023515-f1dff616
modelauto_codexunverifiedrun-20261102-023813-b7d6ced3