Task
freight-shop-001@1.0.0
runs
12
success rate
33.3%
mean time
11 m 12 s
input tokens
6,169,405
generated tokens
237,688
tokens
6,407,093
tool calls
262
What the task asks
Implement rate shopping from its specification
shop() in src/freight/shop.py is declared, documented and tested, but its body raises NotImplementedError. Implement it from the written contract — who gets quoted, per-carrier billable weight, the surcharge pass and its groups, what is rejected and why, and how the cheapest quote is chosen — without touching the rest of the package.
full description (markdown source)
# Implement `shop` from its specification `shop` in `src/freight/shop.py` is declared, documented and tested, but its body raises `NotImplementedError`. Write it. The full contract is in the module's docstring: which carriers get asked, why the others do not, how billable weight is computed per carrier, the order of the surcharge pass and how a group decides between its members, what a rejection looks like, and how the cheapest quote is chosen when the prices tie. Read it before writing a line of code; it is the specification, and the evaluation measures against it. ## The short version - Only carriers that hold a card for the band of the lane are quoted; the billable weight is the card's own divisor, minimum and step. - The line-haul comes from the card. Surcharges run in declaration order, each one against the quote as it looks at that moment; a group keeps its most expensive member and suppresses the rest, naming the winner. - A card that refuses the weight, or a carrier with no card for the band, is a rejection with a reason, not a price. - The cheapest total wins; ties go to the lower billable weight, then the alphabetical carrier name. A run with no price at all is not an empty result: it raises `NoQuotableCarrier`. ## Boundaries - Implement it in `src/freight/shop.py`; the public API of `freight` stays as it is, and nothing else in the package changes. - The function is pure: it reads the shipment and the tables and returns a `ShoppingResult`; calling it twice on the same input returns the same answer, in the same order. - Do not add third-party dependencies — the standard library is enough. - The repository's own test suite (`pytest`) has to stay green.
Constraints
- do not change the public API of the freight package
- do not modify the repository test suite under tests/
- do not change the zone table, the weight rules, the rate cards or the surcharges
- do not add third-party dependencies
- do not modify the shopping spec in the shop module docstring
- shopping is a pure function: nothing is booked and the result must not depend on card order
Notes
The base state fails the hidden tests by design: the feature is missing, which is the opposite of the bug-fix convention where hidden tests pass on the base commit.
Recorded identity
| field | value |
|---|---|
| uid | freight-shop-001@1.0.0 |
| category | new_feature |
| difficulty | expert |
| evaluator | pytest_evaluator |
| base commit | HEAD |
| repository | repo-19e3e7494c669757 |
| snapshot digest | sha256:37ebc3e1dc421131a508b71f99115c13191e2c1c4f6364e5c327bfcf0217bc8c |
| source | datasets/_sources/freight |
| ref | HEAD |
| mutation | none |
| timeout | 1,800 s |
| hidden tests | hidden_tests |
| public tests | public_tests |
| definition digest | sha256:37ebc3e1dc421131a508b71f99115c13191e2c1c4f6364e5c327bfcf0217bc8c |
| definition path | /mnt/data2/dev/develop/ak-bench/datasets/new_feature/freight-shop-001 |
Runs of this task
in the order they started; the parameters column is the configuration each run was executed under
| run | parameters | started | agent | repetition | status | verdict | score | time | input tokens | generated tokens | total tokens | tool calls | failure class | error |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| run-20261102-153629-de5ac66d | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 15:36:29 | codex@codex-cli 0.166.0 | 1 | completed | partial | 0.941 | 9 m 40 s | 514,930 | 12,666 | 527,596 | 25 | ||
| run-20261102-154610-09481f31 | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 15:46:10 | codex@codex-cli 0.166.0 | 2 | timeout | not measured | 15 m 3 s | 345,908 | 24,250 | 370,158 | 18 | agent | The code works correctly — my test had the wrong assumption. I forgot the default divisor is 5000, so 130×30×20 gives volumetric 15.6 kg (billed 16.0 kg), not 3.0 kg. The linehaul is 4200, surcharge 500, total 4700 — and the group logic worked perfectly: `oversize` won, `heavy` was suppressed, winner named. | |
| run-20261102-160115-cb48ab31 | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 16:01:15 | codex@codex-cli 0.166.0 | 3 | completed | partial | 0.941 | 10 m 38 s | 402,646 | 15,902 | 418,548 | 17 | ||
| run-20261102-161153-16c04a4e | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 16:11:53 | codex@codex-cli 0.166.0 | 1 | completed | partial | 0.941 | 11 m 36 s | 709,380 | 23,886 | 733,266 | 25 | ||
| run-20261102-162330-279c2d0e | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 16:23:30 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 11 m 50 s | 597,870 | 25,858 | 623,728 | 24 | ||
| run-20261102-163521-b51708f9 | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 16:35:21 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 12 m 16 s | 819,721 | 25,306 | 845,027 | 29 | ||
| run-20261102-164738-43ff8923 | knowledge_enabled=False, model=auto_codex | 2026-11-02 16:47:38 | codex@codex-cli 0.166.0 | 1 | completed | partial | 0.941 | 6 m 29 s | 254,425 | 9,577 | 264,002 | 14 | ||
| run-20261102-165407-d907f577 | knowledge_enabled=False, model=auto_codex | 2026-11-02 16:54:07 | codex@codex-cli 0.166.0 | 2 | completed | partial | 0.941 | 11 m 28 s | 361,986 | 19,008 | 380,994 | 17 | ||
| run-20261102-170536-da2abfe7 | knowledge_enabled=False, model=auto_codex | 2026-11-02 17:05:36 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 8 m 7 s | 389,741 | 14,953 | 404,694 | 20 | ||
| run-20261102-171345-07e43bdb | knowledge_enabled=True, model=auto_codex | 2026-11-02 17:13:45 | codex@codex-cli 0.166.0 | 1 | timeout | not measured | 15 m 3 s | 581,220 | 26,757 | 607,977 | 24 | agent | All 19 spec checks pass. Final verification — full suite plus a review of the finished body: | |
| run-20261102-172849-b26fa228 | knowledge_enabled=True, model=auto_codex | 2026-11-02 17:28:49 | codex@codex-cli 0.166.0 | 2 | completed | partial | 0.941 | 10 m 30 s | 515,522 | 17,044 | 532,566 | 23 | ||
| run-20261102-173920-82f7a48c | knowledge_enabled=True, model=auto_codex | 2026-11-02 17:39:20 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 11 m 43 s | 676,056 | 22,481 | 698,537 | 26 |
Not verified
Configurations that are not verified. They stay in the report, because a result produced under them is only as trustworthy as they are.
| kind | subject | status | run | note |
|---|---|---|---|---|
| plugin | agentic_knowledge | unsupported | run-20261102-153629-de5ac66d | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-153629-de5ac66d | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-154610-09481f31 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-154610-09481f31 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-160115-cb48ab31 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-160115-cb48ab31 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-161153-16c04a4e | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-161153-16c04a4e | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-162330-279c2d0e | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-162330-279c2d0e | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-163521-b51708f9 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-163521-b51708f9 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-164738-43ff8923 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-164738-43ff8923 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-165407-d907f577 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-165407-d907f577 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-170536-da2abfe7 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-170536-da2abfe7 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-171345-07e43bdb | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-171345-07e43bdb | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-172849-b26fa228 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-172849-b26fa228 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-173920-82f7a48c | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-173920-82f7a48c | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-153629-de5ac66d | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-154610-09481f31 | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-160115-cb48ab31 | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-161153-16c04a4e | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-162330-279c2d0e | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-163521-b51708f9 | |
| model | auto_codex | unverified | run-20261102-164738-43ff8923 | |
| model | auto_codex | unverified | run-20261102-165407-d907f577 | |
| model | auto_codex | unverified | run-20261102-170536-da2abfe7 | |
| model | auto_codex | unverified | run-20261102-171345-07e43bdb | |
| model | auto_codex | unverified | run-20261102-172849-b26fa228 | |
| model | auto_codex | unverified | run-20261102-173920-82f7a48c |