AK-Benchbenchmark report

Task

stockpile-fefo-001@1.0.0

runs
12
evaluated
12
passed
12
success rate
100.0%
mean time
6 m 12 s
input tokens
3,412,912
generated tokens
103,428
tokens
3,516,340
tool calls
203

What the task asks

Implement FEFO allocation from its specification

allocate() in src/stockpile/allocate.py is declared, documented and tested, but its body raises NotImplementedError. Implement it from the written contract — which lots may give, in which order, how much each gives, what a plan is and what it is not — without touching the rest of the package.

full description (markdown source)
# Implement `allocate` from its specification

`allocate` in `src/stockpile/allocate.py` is declared, documented and tested,
but its body raises `NotImplementedError`. Write it.

The full contract is in the module's docstring: which lots may give, how a
lot that has already been promised to an earlier plan is treated, the FEFO
order and its two tie-breaks, how much is taken from each lot, and why a
partial plan is never returned. Read it before writing a line of code; it is
the specification, and the evaluation measures against it.

## The short version

- A lot may give when it is in the warehouse, not expired at `now` — a lot that
  expires on `now` is still eligible — and still has free quantity, counting
  whatever `already_planned` has promised from it.
- Earliest expiry first; ties go to the oldest receipt, then the smaller lot
  id. The earliest eligible lot is drained as far as it can go, then the next.
- If the eligible lots cannot cover the request, refuse the whole request with
  `InsufficientStock`; a plan that ships half its lines is worse than one that
  is refused up front.
- The result is a plan, not a movement: `picks` add up to the request and
  `to_reservation` turns the plan into something `Stock.reserve` can hold.

## Boundaries

- Implement it in `src/stockpile/allocate.py`; the public API of `stockpile`
  stays as it is, and nothing else in the package changes.
- The function is pure: it reads the stock and returns an `Allocation`;
  calling it twice on the same input returns the same plan.
- Do not add third-party dependencies — the standard library is enough.
- The repository's own test suite (`pytest`) has to stay green.

Constraints

Notes

The base state fails the hidden tests by design: the feature is missing, which is the opposite of the bug-fix convention where hidden tests pass on the base commit.

Recorded identity

fieldvalue
uidstockpile-fefo-001@1.0.0
categorynew_feature
difficultyexpert
evaluatorpytest_evaluator
base commitHEAD
repositoryrepo-747f8117bee5bab7
snapshot digestsha256:a321fefaf2ae1770dd37fe53e2b7d3b5fd8ef4a912b50a53d7832376154f2501
sourcedatasets/_sources/stockpile
refHEAD
mutationnone
timeout1,800 s
hidden testshidden_tests
public testspublic_tests
definition digestsha256:a321fefaf2ae1770dd37fe53e2b7d3b5fd8ef4a912b50a53d7832376154f2501
definition path/mnt/data2/dev/develop/ak-bench/datasets/new_feature/stockpile-fefo-001

Runs of this task

in the order they started; the parameters column is the configuration each run was executed under

runparametersstartedagentrepetitionstatusverdictscoretimeinput tokensgenerated tokenstotal tokenstool callsfailure classerror
run-20261102-200902-2be9dbd5knowledge_enabled=False, model=qwen/Qwen3.8-27B2026-11-02 20:09:02codex@codex-cli 0.166.01completedpassed1.00010 m 48 s536,45214,854551,30627none—
run-20261102-201952-bdfed954knowledge_enabled=False, model=qwen/Qwen3.8-27B2026-11-02 20:19:52codex@codex-cli 0.166.02completedpassed1.0004 m 57 s212,7845,246218,03014none—
run-20261102-202449-2aaf3e0fknowledge_enabled=False, model=qwen/Qwen3.8-27B2026-11-02 20:24:49codex@codex-cli 0.166.03completedpassed1.00010 m 20 s433,52915,274448,80323none—
run-20261102-203510-2919460cknowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next2026-11-02 20:35:10codex@codex-cli 0.166.01completedpassed1.0005 m 3 s287,1487,894295,04217none—
run-20261102-204014-9fa0f82eknowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next2026-11-02 20:40:14codex@codex-cli 0.166.02completedpassed1.0004 m 59 s256,1898,521264,71014none—
run-20261102-204514-4ab03ecbknowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next2026-11-02 20:45:14codex@codex-cli 0.166.03completedpassed1.0005 m 47 s271,5449,568281,11218none—
run-20261102-205102-77cec3caknowledge_enabled=False, model=auto_codex2026-11-02 20:51:02codex@codex-cli 0.166.01completedpassed1.0004 m 44 s165,9345,422171,35611none—
run-20261102-205547-64ffb964knowledge_enabled=False, model=auto_codex2026-11-02 20:55:47codex@codex-cli 0.166.02completedpassed1.0005 m 56 s195,3258,161203,48613none—
run-20261102-210144-40341822knowledge_enabled=False, model=auto_codex2026-11-02 21:01:44codex@codex-cli 0.166.03completedpassed1.0004 m 2 s137,8594,431142,29011none—
run-20261102-210547-94d6a5ffknowledge_enabled=True, model=auto_codex2026-11-02 21:05:47codex@codex-cli 0.166.01completedpassed1.0004 m 15 s159,1194,648163,76713none—
run-20261102-211003-7474c6a3knowledge_enabled=True, model=auto_codex2026-11-02 21:10:03codex@codex-cli 0.166.02completedpassed1.0007 m 3 s468,1798,967477,14625none—
run-20261102-211707-b1b04f82knowledge_enabled=True, model=auto_codex2026-11-02 21:17:07codex@codex-cli 0.166.03completedpassed1.0006 m 27 s288,85010,442299,29217none—

Not verified

Configurations that are not verified. They stay in the report, because a result produced under them is only as trustworthy as they are.
kindsubjectstatusrunnote
pluginagentic_knowledgeunsupportedrun-20261102-200902-2be9dbd5LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-200902-2be9dbd5descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-201952-bdfed954LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-201952-bdfed954descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-202449-2aaf3e0fLLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-202449-2aaf3e0fdescriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-203510-2919460cLLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-203510-2919460cdescriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-204014-9fa0f82eLLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-204014-9fa0f82edescriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-204514-4ab03ecbLLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-204514-4ab03ecbdescriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-205102-77cec3caLLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-205102-77cec3cadescriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-205547-64ffb964LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-205547-64ffb964descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-210144-40341822LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-210144-40341822descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-210547-94d6a5ffLLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-210547-94d6a5ffdescriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-211003-7474c6a3LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-211003-7474c6a3descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-211707-b1b04f82LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-211707-b1b04f82descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
modelqwen/Qwen3.8-27Bunverifiedrun-20261102-200902-2be9dbd5
modelqwen/Qwen3.8-27Bunverifiedrun-20261102-201952-bdfed954
modelqwen/Qwen3.8-27Bunverifiedrun-20261102-202449-2aaf3e0f
modelqwen/Qwen3.8-Flash-Nextunverifiedrun-20261102-203510-2919460c
modelqwen/Qwen3.8-Flash-Nextunverifiedrun-20261102-204014-9fa0f82e
modelqwen/Qwen3.8-Flash-Nextunverifiedrun-20261102-204514-4ab03ecb
modelauto_codexunverifiedrun-20261102-205102-77cec3ca
modelauto_codexunverifiedrun-20261102-205547-64ffb964
modelauto_codexunverifiedrun-20261102-210144-40341822
modelauto_codexunverifiedrun-20261102-210547-94d6a5ff
modelauto_codexunverifiedrun-20261102-211003-7474c6a3
modelauto_codexunverifiedrun-20261102-211707-b1b04f82