Task
stockpile-fefo-001@1.0.0
runs
12
evaluated
12
passed
12
success rate
100.0%
mean time
6 m 12 s
input tokens
3,412,912
generated tokens
103,428
tokens
3,516,340
tool calls
203
What the task asks
Implement FEFO allocation from its specification
allocate() in src/stockpile/allocate.py is declared, documented and tested, but its body raises NotImplementedError. Implement it from the written contract — which lots may give, in which order, how much each gives, what a plan is and what it is not — without touching the rest of the package.
full description (markdown source)
# Implement `allocate` from its specification `allocate` in `src/stockpile/allocate.py` is declared, documented and tested, but its body raises `NotImplementedError`. Write it. The full contract is in the module's docstring: which lots may give, how a lot that has already been promised to an earlier plan is treated, the FEFO order and its two tie-breaks, how much is taken from each lot, and why a partial plan is never returned. Read it before writing a line of code; it is the specification, and the evaluation measures against it. ## The short version - A lot may give when it is in the warehouse, not expired at `now` — a lot that expires on `now` is still eligible — and still has free quantity, counting whatever `already_planned` has promised from it. - Earliest expiry first; ties go to the oldest receipt, then the smaller lot id. The earliest eligible lot is drained as far as it can go, then the next. - If the eligible lots cannot cover the request, refuse the whole request with `InsufficientStock`; a plan that ships half its lines is worse than one that is refused up front. - The result is a plan, not a movement: `picks` add up to the request and `to_reservation` turns the plan into something `Stock.reserve` can hold. ## Boundaries - Implement it in `src/stockpile/allocate.py`; the public API of `stockpile` stays as it is, and nothing else in the package changes. - The function is pure: it reads the stock and returns an `Allocation`; calling it twice on the same input returns the same plan. - Do not add third-party dependencies — the standard library is enough. - The repository's own test suite (`pytest`) has to stay green.
Constraints
- do not change the public API of the stockpile package
- do not modify the repository test suite under tests/
- do not change the stock, the lots or the reservation lifecycle
- do not add third-party dependencies
- do not modify the allocation spec in the allocate module docstring
- an allocation is a plan: it reads the stock and never reserves anything
Notes
The base state fails the hidden tests by design: the feature is missing, which is the opposite of the bug-fix convention where hidden tests pass on the base commit.
Recorded identity
| field | value |
|---|---|
| uid | stockpile-fefo-001@1.0.0 |
| category | new_feature |
| difficulty | expert |
| evaluator | pytest_evaluator |
| base commit | HEAD |
| repository | repo-747f8117bee5bab7 |
| snapshot digest | sha256:a321fefaf2ae1770dd37fe53e2b7d3b5fd8ef4a912b50a53d7832376154f2501 |
| source | datasets/_sources/stockpile |
| ref | HEAD |
| mutation | none |
| timeout | 1,800 s |
| hidden tests | hidden_tests |
| public tests | public_tests |
| definition digest | sha256:a321fefaf2ae1770dd37fe53e2b7d3b5fd8ef4a912b50a53d7832376154f2501 |
| definition path | /mnt/data2/dev/develop/ak-bench/datasets/new_feature/stockpile-fefo-001 |
Runs of this task
in the order they started; the parameters column is the configuration each run was executed under
| run | parameters | started | agent | repetition | status | verdict | score | time | input tokens | generated tokens | total tokens | tool calls | failure class | error |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| run-20261102-200902-2be9dbd5 | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 20:09:02 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 10 m 48 s | 536,452 | 14,854 | 551,306 | 27 | none | — |
| run-20261102-201952-bdfed954 | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 20:19:52 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 4 m 57 s | 212,784 | 5,246 | 218,030 | 14 | none | — |
| run-20261102-202449-2aaf3e0f | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 20:24:49 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 10 m 20 s | 433,529 | 15,274 | 448,803 | 23 | none | — |
| run-20261102-203510-2919460c | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 20:35:10 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 5 m 3 s | 287,148 | 7,894 | 295,042 | 17 | none | — |
| run-20261102-204014-9fa0f82e | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 20:40:14 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 4 m 59 s | 256,189 | 8,521 | 264,710 | 14 | none | — |
| run-20261102-204514-4ab03ecb | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 20:45:14 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 5 m 47 s | 271,544 | 9,568 | 281,112 | 18 | none | — |
| run-20261102-205102-77cec3ca | knowledge_enabled=False, model=auto_codex | 2026-11-02 20:51:02 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 4 m 44 s | 165,934 | 5,422 | 171,356 | 11 | none | — |
| run-20261102-205547-64ffb964 | knowledge_enabled=False, model=auto_codex | 2026-11-02 20:55:47 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 5 m 56 s | 195,325 | 8,161 | 203,486 | 13 | none | — |
| run-20261102-210144-40341822 | knowledge_enabled=False, model=auto_codex | 2026-11-02 21:01:44 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 4 m 2 s | 137,859 | 4,431 | 142,290 | 11 | none | — |
| run-20261102-210547-94d6a5ff | knowledge_enabled=True, model=auto_codex | 2026-11-02 21:05:47 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 4 m 15 s | 159,119 | 4,648 | 163,767 | 13 | none | — |
| run-20261102-211003-7474c6a3 | knowledge_enabled=True, model=auto_codex | 2026-11-02 21:10:03 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 7 m 3 s | 468,179 | 8,967 | 477,146 | 25 | none | — |
| run-20261102-211707-b1b04f82 | knowledge_enabled=True, model=auto_codex | 2026-11-02 21:17:07 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 6 m 27 s | 288,850 | 10,442 | 299,292 | 17 | none | — |
Not verified
Configurations that are not verified. They stay in the report, because a result produced under them is only as trustworthy as they are.
| kind | subject | status | run | note |
|---|---|---|---|---|
| plugin | agentic_knowledge | unsupported | run-20261102-200902-2be9dbd5 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-200902-2be9dbd5 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-201952-bdfed954 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-201952-bdfed954 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-202449-2aaf3e0f | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-202449-2aaf3e0f | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-203510-2919460c | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-203510-2919460c | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-204014-9fa0f82e | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-204014-9fa0f82e | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-204514-4ab03ecb | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-204514-4ab03ecb | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-205102-77cec3ca | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-205102-77cec3ca | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-205547-64ffb964 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-205547-64ffb964 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-210144-40341822 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-210144-40341822 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-210547-94d6a5ff | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-210547-94d6a5ff | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-211003-7474c6a3 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-211003-7474c6a3 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-211707-b1b04f82 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-211707-b1b04f82 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-200902-2be9dbd5 | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-201952-bdfed954 | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-202449-2aaf3e0f | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-203510-2919460c | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-204014-9fa0f82e | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-204514-4ab03ecb | |
| model | auto_codex | unverified | run-20261102-205102-77cec3ca | |
| model | auto_codex | unverified | run-20261102-205547-64ffb964 | |
| model | auto_codex | unverified | run-20261102-210144-40341822 | |
| model | auto_codex | unverified | run-20261102-210547-94d6a5ff | |
| model | auto_codex | unverified | run-20261102-211003-7474c6a3 | |
| model | auto_codex | unverified | run-20261102-211707-b1b04f82 |