AK-Benchbenchmark report

Task

ledgerlite-rates-001@1.0.0

runs
12
success rate
100.0%
mean time
4 m 1 s
input tokens
2,088,806
generated tokens
58,729
tokens
2,147,535
tool calls
157

What the task asks

A conversion answers with the oldest rate in force

RateBook.rate_for returns the first published rate at or before the moment instead of the last, so once a currency's rate has been revised, every conversion answers with the superseded rate. Restore the documented rule: the rate published most recently at or before the moment is the one in force.

full description (markdown source)
# A conversion answers with the oldest rate in force

`RateBook.rate_for` in `src/ledgerlite/fx.py` answers one question: which rate
of a currency was in force at a moment. The rule, as documented in the module:

- a rate is in force from the instant it was published until the next one is;
- the rate published **most recently at or before** the moment is the one in
  force;
- a rate published later is knowledge the ledger did not have, so it is never
  the answer, and there is no answer before the first publication.

When a currency's rate is revised, the old rate stops being the answer from the
revision onward. The defect makes the book answer with the first rate it can
find at or before the moment, so after a revision every conversion is silently
valued at the superseded rate — no error, just wrong numbers.

The repository's own suite is green on the pinned commit, because it publishes
a single rate per currency, for which "first in force" and "last in force" are
the same rate. The defect only shows up once a currency has more than one
published rate.

Constraints

Recorded identity

fieldvalue
uidledgerlite-rates-001@1.0.0
categorybug_fix
difficultyexpert
evaluatorpytest_evaluator
base commitHEAD
repositoryrepo-c1fe8d57647b4a8a
snapshot digestsha256:acc3454a1ccd66fd62a937f91e7b9446687cd4721cb42b462c390e9405422882
sourcedatasets/_sources/ledgerlite
refHEAD
mutationnone
timeout900 s
hidden testshidden_tests
public testspublic_tests
definition digestsha256:acc3454a1ccd66fd62a937f91e7b9446687cd4721cb42b462c390e9405422882
definition path/mnt/data2/dev/develop/ak-bench/datasets/bug_fix/ledgerlite-rates-001

Runs of this task

in the order they started; the parameters column is the configuration each run was executed under

runparametersstartedagentrepetitionstatusverdictscoretimeinput tokensgenerated tokenstotal tokenstool callsfailure classerror
run-20261102-111029-a9c04726knowledge_enabled=False, model=qwen/Qwen3.8-27B2026-11-02 11:10:29codex@codex-cli 0.166.01completedpassed1.0002 m 57 s118,8372,616121,45312
run-20261102-111326-3c584bd3knowledge_enabled=False, model=qwen/Qwen3.8-27B2026-11-02 11:13:26codex@codex-cli 0.166.02completedpassed1.0002 m 10 s81,2801,90383,1837
run-20261102-111537-b9ab2fefknowledge_enabled=False, model=qwen/Qwen3.8-27B2026-11-02 11:15:37codex@codex-cli 0.166.03completedpassed1.0003 m 1 s130,3242,843133,16712
run-20261102-111839-c5f2e4c1knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next2026-11-02 11:18:39codex@codex-cli 0.166.01completedpassed1.0004 m 26 s197,7436,744204,48715
run-20261102-112306-6bc27784knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next2026-11-02 11:23:06codex@codex-cli 0.166.02completedpassed1.0006 m 46 s359,67811,459371,13720
run-20261102-112953-68b1af75knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next2026-11-02 11:29:53codex@codex-cli 0.166.03completedpassed1.0005 m 12 s266,6577,835274,49219
run-20261102-113507-5543ffe1knowledge_enabled=False, model=auto_codex2026-11-02 11:35:07codex@codex-cli 0.166.01completedpassed1.0006 m 49 s273,4338,650282,08317
run-20261102-114156-1ff0ba59knowledge_enabled=False, model=auto_codex2026-11-02 11:41:56codex@codex-cli 0.166.02completedpassed1.0002 m 49 s97,4732,29499,7679
run-20261102-114446-b3513377knowledge_enabled=False, model=auto_codex2026-11-02 11:44:46codex@codex-cli 0.166.03completedpassed1.0003 m 5 s137,6743,039140,71310
run-20261102-114752-9f3f4f92knowledge_enabled=True, model=auto_codex2026-11-02 11:47:52codex@codex-cli 0.166.01completedpassed1.0002 m 59 s109,7083,384113,09210
run-20261102-115051-39be5247knowledge_enabled=True, model=auto_codex2026-11-02 11:50:51codex@codex-cli 0.166.02completedpassed1.0005 m 19 s233,2265,848239,07416
run-20261102-115611-ee478152knowledge_enabled=True, model=auto_codex2026-11-02 11:56:11codex@codex-cli 0.166.03completedpassed1.0002 m 39 s82,7732,11484,88710

Not verified

Configurations that are not verified. They stay in the report, because a result produced under them is only as trustworthy as they are.
kindsubjectstatusrunnote
pluginagentic_knowledgeunsupportedrun-20261102-111029-a9c04726LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-111029-a9c04726descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-111326-3c584bd3LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-111326-3c584bd3descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-111537-b9ab2fefLLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-111537-b9ab2fefdescriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-111839-c5f2e4c1LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-111839-c5f2e4c1descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-112306-6bc27784LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-112306-6bc27784descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-112953-68b1af75LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-112953-68b1af75descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-113507-5543ffe1LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-113507-5543ffe1descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-114156-1ff0ba59LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-114156-1ff0ba59descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-114446-b3513377LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-114446-b3513377descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-114752-9f3f4f92LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-114752-9f3f4f92descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-115051-39be5247LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-115051-39be5247descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-115611-ee478152LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-115611-ee478152descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
modelqwen/Qwen3.8-27Bunverifiedrun-20261102-111029-a9c04726
modelqwen/Qwen3.8-27Bunverifiedrun-20261102-111326-3c584bd3
modelqwen/Qwen3.8-27Bunverifiedrun-20261102-111537-b9ab2fef
modelqwen/Qwen3.8-Flash-Nextunverifiedrun-20261102-111839-c5f2e4c1
modelqwen/Qwen3.8-Flash-Nextunverifiedrun-20261102-112306-6bc27784
modelqwen/Qwen3.8-Flash-Nextunverifiedrun-20261102-112953-68b1af75
modelauto_codexunverifiedrun-20261102-113507-5543ffe1
modelauto_codexunverifiedrun-20261102-114156-1ff0ba59
modelauto_codexunverifiedrun-20261102-114446-b3513377
modelauto_codexunverifiedrun-20261102-114752-9f3f4f92
modelauto_codexunverifiedrun-20261102-115051-39be5247
modelauto_codexunverifiedrun-20261102-115611-ee478152