AK-Benchbenchmark report

Task

freight-shop-001@1.0.0

runs
12
success rate
33.3%
mean time
11 m 12 s
input tokens
6,169,405
generated tokens
237,688
tokens
6,407,093
tool calls
262

What the task asks

Implement rate shopping from its specification

shop() in src/freight/shop.py is declared, documented and tested, but its body raises NotImplementedError. Implement it from the written contract — who gets quoted, per-carrier billable weight, the surcharge pass and its groups, what is rejected and why, and how the cheapest quote is chosen — without touching the rest of the package.

full description (markdown source)
# Implement `shop` from its specification

`shop` in `src/freight/shop.py` is declared, documented and tested, but its
body raises `NotImplementedError`. Write it.

The full contract is in the module's docstring: which carriers get asked,
why the others do not, how billable weight is computed per carrier, the
order of the surcharge pass and how a group decides between its members,
what a rejection looks like, and how the cheapest quote is chosen when the
prices tie. Read it before writing a line of code; it is the specification,
and the evaluation measures against it.

## The short version

- Only carriers that hold a card for the band of the lane are quoted; the
  billable weight is the card's own divisor, minimum and step.
- The line-haul comes from the card. Surcharges run in declaration order,
  each one against the quote as it looks at that moment; a group keeps its
  most expensive member and suppresses the rest, naming the winner.
- A card that refuses the weight, or a carrier with no card for the band,
  is a rejection with a reason, not a price.
- The cheapest total wins; ties go to the lower billable weight, then the
  alphabetical carrier name. A run with no price at all is not an empty
  result: it raises `NoQuotableCarrier`.

## Boundaries

- Implement it in `src/freight/shop.py`; the public API of `freight` stays
  as it is, and nothing else in the package changes.
- The function is pure: it reads the shipment and the tables and returns a
  `ShoppingResult`; calling it twice on the same input returns the same
  answer, in the same order.
- Do not add third-party dependencies — the standard library is enough.
- The repository's own test suite (`pytest`) has to stay green.

Constraints

Notes

The base state fails the hidden tests by design: the feature is missing, which is the opposite of the bug-fix convention where hidden tests pass on the base commit.

Recorded identity

fieldvalue
uidfreight-shop-001@1.0.0
categorynew_feature
difficultyexpert
evaluatorpytest_evaluator
base commitHEAD
repositoryrepo-19e3e7494c669757
snapshot digestsha256:37ebc3e1dc421131a508b71f99115c13191e2c1c4f6364e5c327bfcf0217bc8c
sourcedatasets/_sources/freight
refHEAD
mutationnone
timeout1,800 s
hidden testshidden_tests
public testspublic_tests
definition digestsha256:37ebc3e1dc421131a508b71f99115c13191e2c1c4f6364e5c327bfcf0217bc8c
definition path/mnt/data2/dev/develop/ak-bench/datasets/new_feature/freight-shop-001

Runs of this task

in the order they started; the parameters column is the configuration each run was executed under

runparametersstartedagentrepetitionstatusverdictscoretimeinput tokensgenerated tokenstotal tokenstool callsfailure classerror
run-20261102-153629-de5ac66dknowledge_enabled=False, model=qwen/Qwen3.8-27B2026-11-02 15:36:29codex@codex-cli 0.166.01completedpartial0.9419 m 40 s514,93012,666527,59625
run-20261102-154610-09481f31knowledge_enabled=False, model=qwen/Qwen3.8-27B2026-11-02 15:46:10codex@codex-cli 0.166.02timeoutnot measured15 m 3 s345,90824,250370,15818agentThe code works correctly — my test had the wrong assumption. I forgot the default divisor is 5000, so 130×30×20 gives volumetric 15.6 kg (billed 16.0 kg), not 3.0 kg. The linehaul is 4200, surcharge 500, total 4700 — and the group logic worked perfectly: `oversize` won, `heavy` was suppressed, winner named.
run-20261102-160115-cb48ab31knowledge_enabled=False, model=qwen/Qwen3.8-27B2026-11-02 16:01:15codex@codex-cli 0.166.03completedpartial0.94110 m 38 s402,64615,902418,54817
run-20261102-161153-16c04a4eknowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next2026-11-02 16:11:53codex@codex-cli 0.166.01completedpartial0.94111 m 36 s709,38023,886733,26625
run-20261102-162330-279c2d0eknowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next2026-11-02 16:23:30codex@codex-cli 0.166.02completedpassed1.00011 m 50 s597,87025,858623,72824
run-20261102-163521-b51708f9knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next2026-11-02 16:35:21codex@codex-cli 0.166.03completedpassed1.00012 m 16 s819,72125,306845,02729
run-20261102-164738-43ff8923knowledge_enabled=False, model=auto_codex2026-11-02 16:47:38codex@codex-cli 0.166.01completedpartial0.9416 m 29 s254,4259,577264,00214
run-20261102-165407-d907f577knowledge_enabled=False, model=auto_codex2026-11-02 16:54:07codex@codex-cli 0.166.02completedpartial0.94111 m 28 s361,98619,008380,99417
run-20261102-170536-da2abfe7knowledge_enabled=False, model=auto_codex2026-11-02 17:05:36codex@codex-cli 0.166.03completedpassed1.0008 m 7 s389,74114,953404,69420
run-20261102-171345-07e43bdbknowledge_enabled=True, model=auto_codex2026-11-02 17:13:45codex@codex-cli 0.166.01timeoutnot measured15 m 3 s581,22026,757607,97724agentAll 19 spec checks pass. Final verification — full suite plus a review of the finished body:
run-20261102-172849-b26fa228knowledge_enabled=True, model=auto_codex2026-11-02 17:28:49codex@codex-cli 0.166.02completedpartial0.94110 m 30 s515,52217,044532,56623
run-20261102-173920-82f7a48cknowledge_enabled=True, model=auto_codex2026-11-02 17:39:20codex@codex-cli 0.166.03completedpassed1.00011 m 43 s676,05622,481698,53726

Not verified

Configurations that are not verified. They stay in the report, because a result produced under them is only as trustworthy as they are.
kindsubjectstatusrunnote
pluginagentic_knowledgeunsupportedrun-20261102-153629-de5ac66dLLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-153629-de5ac66ddescriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-154610-09481f31LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-154610-09481f31descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-160115-cb48ab31LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-160115-cb48ab31descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-161153-16c04a4eLLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-161153-16c04a4edescriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-162330-279c2d0eLLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-162330-279c2d0edescriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-163521-b51708f9LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-163521-b51708f9descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-164738-43ff8923LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-164738-43ff8923descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-165407-d907f577LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-165407-d907f577descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-170536-da2abfe7LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-170536-da2abfe7descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-171345-07e43bdbLLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-171345-07e43bdbdescriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-172849-b26fa228LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-172849-b26fa228descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-173920-82f7a48cLLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-173920-82f7a48cdescriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
modelqwen/Qwen3.8-27Bunverifiedrun-20261102-153629-de5ac66d
modelqwen/Qwen3.8-27Bunverifiedrun-20261102-154610-09481f31
modelqwen/Qwen3.8-27Bunverifiedrun-20261102-160115-cb48ab31
modelqwen/Qwen3.8-Flash-Nextunverifiedrun-20261102-161153-16c04a4e
modelqwen/Qwen3.8-Flash-Nextunverifiedrun-20261102-162330-279c2d0e
modelqwen/Qwen3.8-Flash-Nextunverifiedrun-20261102-163521-b51708f9
modelauto_codexunverifiedrun-20261102-164738-43ff8923
modelauto_codexunverifiedrun-20261102-165407-d907f577
modelauto_codexunverifiedrun-20261102-170536-da2abfe7
modelauto_codexunverifiedrun-20261102-171345-07e43bdb
modelauto_codexunverifiedrun-20261102-172849-b26fa228
modelauto_codexunverifiedrun-20261102-173920-82f7a48c