Task
ledgerlite-reconcile-001@1.0.0
runs
12
evaluated
8
passed
4
success rate
33.3%
mean time
11 m 29 s
input tokens
6,164,860
generated tokens
235,377
tokens
6,400,237
tool calls
259
What the task asks
Implement bank reconciliation from its specification
reconcile() in src/ledgerlite/reconcile.py is declared, documented and tested, but its body raises NotImplementedError. Implement it from the written contract — the window, the exact-amount rule, the two passes and their ordering, what counts as open — without touching the rest of the package.
full description (markdown source)
# Implement `reconcile` from its specification `reconcile` in `src/ledgerlite/reconcile.py` is declared, documented and tested, but its body raises `NotImplementedError`. Write it. The full contract is in the module's docstring: what a statement line is, what a candidate is, the two passes (the window and the reference), how ties are broken, what is `unmatched`, what is `open`, and what `difference_cents` means. Read it before writing a line of code; it is the specification, and the evaluation measures against it. ## The short version - A line claims exactly one candidate and a candidate is claimed by at most one line; an amount must be exactly equal, sign included. - Pass 1 looks only inside `window_days`; pass 2 is reference-driven and the window does not apply. - Lines are always considered oldest first; among candidates the closer, then earlier, then smaller-id booking wins. - `open_candidates` are entries that still carry an unclaimed candidate; `difference_cents` is the sum of the amounts the statement could not place. ## Boundaries - Implement it in `src/ledgerlite/reconcile.py`; the public API of `ledgerlite` stays as it is, and nothing else in the package changes. - The function is pure: it reads the journal and the statement and returns a `Reconciliation`; calling it twice on the same input returns the same answer. - Do not add third-party dependencies — the standard library is enough. - The repository's own test suite (`pytest`) has to stay green.
Constraints
- do not change the public API of the ledgerlite package
- do not modify the repository test suite under tests/
- do not change the journal, the chart, the rate book or the period close
- do not add third-party dependencies
- do not modify the reconciliation spec in the reconcile module docstring
- reconciliation is a pure function; nothing is marked or posted
Notes
The base state fails the hidden tests by design: the feature is missing, which is the opposite of the bug-fix convention where hidden tests pass on the base commit.
Recorded identity
| field | value |
|---|---|
| uid | ledgerlite-reconcile-001@1.0.0 |
| category | new_feature |
| difficulty | expert |
| evaluator | pytest_evaluator |
| base commit | HEAD |
| repository | repo-c1fe8d57647b4a8a |
| snapshot digest | sha256:8a516b1565f619f6b2fddfcc8fe062abdaf3658ec23485f6ec58f6d5cc378809 |
| source | datasets/_sources/ledgerlite |
| ref | HEAD |
| mutation | none |
| timeout | 1,800 s |
| hidden tests | hidden_tests |
| public tests | public_tests |
| definition digest | sha256:8a516b1565f619f6b2fddfcc8fe062abdaf3658ec23485f6ec58f6d5cc378809 |
| definition path | /mnt/data2/dev/develop/ak-bench/datasets/new_feature/ledgerlite-reconcile-001 |
Runs of this task
in the order they started; the parameters column is the configuration each run was executed under
| run | parameters | started | agent | repetition | status | verdict | score | time | input tokens | generated tokens | total tokens | tool calls | failure class | error |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| run-20261102-175103-7eadf2c3 | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 17:51:03 | codex@codex-cli 0.166.0 | 1 | timeout | not measured | not measured | 15 m 4 s | 590,243 | 23,356 | 613,599 | 26 | agent | The two identical entries are mapping to the same one due to how frozen dataclass equality works — both e1 and e2 have matching field values, so they're treated as equal. That's why the dict comprehension collapsed to a single key when it built the entry_of mapping. |
| run-20261102-180608-fb09d647 | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 18:06:08 | codex@codex-cli 0.166.0 | 2 | timeout | not measured | not measured | 15 m 4 s | 306,912 | 23,952 | 330,864 | 16 | agent | I need to fix my test expectations — the test had incorrect sign conventions. Let me trace through what the correct signs should be for each line: `e3` has a debit-normal account with a debit of 700, so it should be +700, meaning L3 should be +700; `e1` is a liability account with a credit of 1000, giving -1000 for L1. The implementation is actually working correctly — the test failures were in my own test expectations, not the code itself. I need to fix the test data: L3 should be +700 instead of -700, and I'll adjust the expected difference_cents and open_candidates accordingly. Then I'll re-run the full test suite to verify everything passes. |
| run-20261102-182113-fb3e7151 | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 18:21:13 | codex@codex-cli 0.166.0 | 3 | timeout | not measured | not measured | 15 m 4 s | 417,452 | 24,042 | 441,494 | 20 | agent | I'm checking the line 115 assertion to verify the entry ID matches "e-b", but I'm confused about which entries have the reference "ZZ" — both e-a and e-b do, so I need to figure out which one the assertion is actually pointing to. |
| run-20261102-183618-1b2f49a8 | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 18:36:18 | codex@codex-cli 0.166.0 | 1 | timeout | not measured | not measured | 15 m 4 s | 788,757 | 33,258 | 822,015 | 27 | agent | Tests are green and formatting is clean. Re-running my full spec harness once more and cleaning the `.pyc` noise my runs created. |
| run-20261102-185122-1a50269f | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 18:51:22 | codex@codex-cli 0.166.0 | 2 | completed | partial | 0.950 | 10 m 2 s | 641,240 | 20,168 | 661,408 | 24 | none | — |
| run-20261102-190125-007b2709 | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 19:01:25 | codex@codex-cli 0.166.0 | 3 | completed | partial | 0.950 | 9 m 34 s | 528,145 | 18,654 | 546,799 | 17 | none | — |
| run-20261102-191101-7b74b38f | knowledge_enabled=False, model=auto_codex | 2026-11-02 19:11:01 | codex@codex-cli 0.166.0 | 1 | completed | partial | 0.950 | 9 m 56 s | 392,283 | 14,473 | 406,756 | 20 | none | — |
| run-20261102-192057-a14677eb | knowledge_enabled=False, model=auto_codex | 2026-11-02 19:20:57 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 8 m 34 s | 439,895 | 15,003 | 454,898 | 19 | none | — |
| run-20261102-192932-7bb86699 | knowledge_enabled=False, model=auto_codex | 2026-11-02 19:29:32 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 8 m 49 s | 435,349 | 14,597 | 449,946 | 21 | none | — |
| run-20261102-193822-f24e2c68 | knowledge_enabled=True, model=auto_codex | 2026-11-02 19:38:22 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 10 m 41 s | 452,351 | 17,677 | 470,028 | 21 | none | — |
| run-20261102-194904-cb1d7f72 | knowledge_enabled=True, model=auto_codex | 2026-11-02 19:49:04 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 8 m 9 s | 439,702 | 12,057 | 451,759 | 21 | none | — |
| run-20261102-195713-1edd6faf | knowledge_enabled=True, model=auto_codex | 2026-11-02 19:57:13 | codex@codex-cli 0.166.0 | 3 | completed | partial | 0.950 | 11 m 48 s | 732,531 | 18,140 | 750,671 | 27 | none | — |
Not verified
Configurations that are not verified. They stay in the report, because a result produced under them is only as trustworthy as they are.
| kind | subject | status | run | note |
|---|---|---|---|---|
| plugin | agentic_knowledge | unsupported | run-20261102-175103-7eadf2c3 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-175103-7eadf2c3 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-180608-fb09d647 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-180608-fb09d647 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-182113-fb3e7151 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-182113-fb3e7151 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-183618-1b2f49a8 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-183618-1b2f49a8 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-185122-1a50269f | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-185122-1a50269f | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-190125-007b2709 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-190125-007b2709 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-191101-7b74b38f | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-191101-7b74b38f | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-192057-a14677eb | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-192057-a14677eb | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-192932-7bb86699 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-192932-7bb86699 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-193822-f24e2c68 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-193822-f24e2c68 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-194904-cb1d7f72 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-194904-cb1d7f72 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-195713-1edd6faf | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-195713-1edd6faf | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-175103-7eadf2c3 | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-180608-fb09d647 | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-182113-fb3e7151 | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-183618-1b2f49a8 | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-185122-1a50269f | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-190125-007b2709 | |
| model | auto_codex | unverified | run-20261102-191101-7b74b38f | |
| model | auto_codex | unverified | run-20261102-192057-a14677eb | |
| model | auto_codex | unverified | run-20261102-192932-7bb86699 | |
| model | auto_codex | unverified | run-20261102-193822-f24e2c68 | |
| model | auto_codex | unverified | run-20261102-194904-cb1d7f72 | |
| model | auto_codex | unverified | run-20261102-195713-1edd6faf |