Task
ledgerlite-rates-001@1.0.0
runs
12
success rate
100.0%
mean time
4 m 1 s
input tokens
2,088,806
generated tokens
58,729
tokens
2,147,535
tool calls
157
What the task asks
A conversion answers with the oldest rate in force
RateBook.rate_for returns the first published rate at or before the moment instead of the last, so once a currency's rate has been revised, every conversion answers with the superseded rate. Restore the documented rule: the rate published most recently at or before the moment is the one in force.
full description (markdown source)
# A conversion answers with the oldest rate in force `RateBook.rate_for` in `src/ledgerlite/fx.py` answers one question: which rate of a currency was in force at a moment. The rule, as documented in the module: - a rate is in force from the instant it was published until the next one is; - the rate published **most recently at or before** the moment is the one in force; - a rate published later is knowledge the ledger did not have, so it is never the answer, and there is no answer before the first publication. When a currency's rate is revised, the old rate stops being the answer from the revision onward. The defect makes the book answer with the first rate it can find at or before the moment, so after a revision every conversion is silently valued at the superseded rate — no error, just wrong numbers. The repository's own suite is green on the pinned commit, because it publishes a single rate per currency, for which "first in force" and "last in force" are the same rate. The defect only shows up once a currency has more than one published rate.
Constraints
- do not change the public API of the ledgerlite package
- do not modify the repository test suite under tests/
- do not change how rates are published or how a conversion rounds
- do not add third-party dependencies
Recorded identity
| field | value |
|---|---|
| uid | ledgerlite-rates-001@1.0.0 |
| category | bug_fix |
| difficulty | expert |
| evaluator | pytest_evaluator |
| base commit | HEAD |
| repository | repo-c1fe8d57647b4a8a |
| snapshot digest | sha256:acc3454a1ccd66fd62a937f91e7b9446687cd4721cb42b462c390e9405422882 |
| source | datasets/_sources/ledgerlite |
| ref | HEAD |
| mutation | none |
| timeout | 900 s |
| hidden tests | hidden_tests |
| public tests | public_tests |
| definition digest | sha256:acc3454a1ccd66fd62a937f91e7b9446687cd4721cb42b462c390e9405422882 |
| definition path | /mnt/data2/dev/develop/ak-bench/datasets/bug_fix/ledgerlite-rates-001 |
Runs of this task
in the order they started; the parameters column is the configuration each run was executed under
| run | parameters | started | agent | repetition | status | verdict | score | time | input tokens | generated tokens | total tokens | tool calls | failure class | error |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| run-20261102-111029-a9c04726 | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 11:10:29 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 2 m 57 s | 118,837 | 2,616 | 121,453 | 12 | ||
| run-20261102-111326-3c584bd3 | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 11:13:26 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 2 m 10 s | 81,280 | 1,903 | 83,183 | 7 | ||
| run-20261102-111537-b9ab2fef | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 11:15:37 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 3 m 1 s | 130,324 | 2,843 | 133,167 | 12 | ||
| run-20261102-111839-c5f2e4c1 | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 11:18:39 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 4 m 26 s | 197,743 | 6,744 | 204,487 | 15 | ||
| run-20261102-112306-6bc27784 | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 11:23:06 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 6 m 46 s | 359,678 | 11,459 | 371,137 | 20 | ||
| run-20261102-112953-68b1af75 | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 11:29:53 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 5 m 12 s | 266,657 | 7,835 | 274,492 | 19 | ||
| run-20261102-113507-5543ffe1 | knowledge_enabled=False, model=auto_codex | 2026-11-02 11:35:07 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 6 m 49 s | 273,433 | 8,650 | 282,083 | 17 | ||
| run-20261102-114156-1ff0ba59 | knowledge_enabled=False, model=auto_codex | 2026-11-02 11:41:56 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 2 m 49 s | 97,473 | 2,294 | 99,767 | 9 | ||
| run-20261102-114446-b3513377 | knowledge_enabled=False, model=auto_codex | 2026-11-02 11:44:46 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 3 m 5 s | 137,674 | 3,039 | 140,713 | 10 | ||
| run-20261102-114752-9f3f4f92 | knowledge_enabled=True, model=auto_codex | 2026-11-02 11:47:52 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 2 m 59 s | 109,708 | 3,384 | 113,092 | 10 | ||
| run-20261102-115051-39be5247 | knowledge_enabled=True, model=auto_codex | 2026-11-02 11:50:51 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 5 m 19 s | 233,226 | 5,848 | 239,074 | 16 | ||
| run-20261102-115611-ee478152 | knowledge_enabled=True, model=auto_codex | 2026-11-02 11:56:11 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 2 m 39 s | 82,773 | 2,114 | 84,887 | 10 |
Not verified
Configurations that are not verified. They stay in the report, because a result produced under them is only as trustworthy as they are.
| kind | subject | status | run | note |
|---|---|---|---|---|
| plugin | agentic_knowledge | unsupported | run-20261102-111029-a9c04726 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-111029-a9c04726 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-111326-3c584bd3 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-111326-3c584bd3 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-111537-b9ab2fef | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-111537-b9ab2fef | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-111839-c5f2e4c1 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-111839-c5f2e4c1 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-112306-6bc27784 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-112306-6bc27784 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-112953-68b1af75 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-112953-68b1af75 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-113507-5543ffe1 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-113507-5543ffe1 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-114156-1ff0ba59 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-114156-1ff0ba59 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-114446-b3513377 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-114446-b3513377 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-114752-9f3f4f92 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-114752-9f3f4f92 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-115051-39be5247 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-115051-39be5247 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-115611-ee478152 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-115611-ee478152 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-111029-a9c04726 | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-111326-3c584bd3 | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-111537-b9ab2fef | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-111839-c5f2e4c1 | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-112306-6bc27784 | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-112953-68b1af75 | |
| model | auto_codex | unverified | run-20261102-113507-5543ffe1 | |
| model | auto_codex | unverified | run-20261102-114156-1ff0ba59 | |
| model | auto_codex | unverified | run-20261102-114446-b3513377 | |
| model | auto_codex | unverified | run-20261102-114752-9f3f4f92 | |
| model | auto_codex | unverified | run-20261102-115051-39be5247 | |
| model | auto_codex | unverified | run-20261102-115611-ee478152 |