AK-Benchbenchmark report

Task

ledgerlite-reconcile-001@1.0.0

runs
12
evaluated
8
passed
4
success rate
33.3%
mean time
11 m 29 s
input tokens
6,164,860
generated tokens
235,377
tokens
6,400,237
tool calls
259

What the task asks

Implement bank reconciliation from its specification

reconcile() in src/ledgerlite/reconcile.py is declared, documented and tested, but its body raises NotImplementedError. Implement it from the written contract — the window, the exact-amount rule, the two passes and their ordering, what counts as open — without touching the rest of the package.

full description (markdown source)
# Implement `reconcile` from its specification

`reconcile` in `src/ledgerlite/reconcile.py` is declared, documented and
tested, but its body raises `NotImplementedError`. Write it.

The full contract is in the module's docstring: what a statement line is, what
a candidate is, the two passes (the window and the reference), how ties are
broken, what is `unmatched`, what is `open`, and what `difference_cents`
means. Read it before writing a line of code; it is the specification, and the
evaluation measures against it.

## The short version

- A line claims exactly one candidate and a candidate is claimed by at most one
  line; an amount must be exactly equal, sign included.
- Pass 1 looks only inside `window_days`; pass 2 is reference-driven and the
  window does not apply.
- Lines are always considered oldest first; among candidates the closer, then
  earlier, then smaller-id booking wins.
- `open_candidates` are entries that still carry an unclaimed candidate;
  `difference_cents` is the sum of the amounts the statement could not place.

## Boundaries

- Implement it in `src/ledgerlite/reconcile.py`; the public API of
  `ledgerlite` stays as it is, and nothing else in the package changes.
- The function is pure: it reads the journal and the statement and returns a
  `Reconciliation`; calling it twice on the same input returns the same answer.
- Do not add third-party dependencies — the standard library is enough.
- The repository's own test suite (`pytest`) has to stay green.

Constraints

Notes

The base state fails the hidden tests by design: the feature is missing, which is the opposite of the bug-fix convention where hidden tests pass on the base commit.

Recorded identity

fieldvalue
uidledgerlite-reconcile-001@1.0.0
categorynew_feature
difficultyexpert
evaluatorpytest_evaluator
base commitHEAD
repositoryrepo-c1fe8d57647b4a8a
snapshot digestsha256:8a516b1565f619f6b2fddfcc8fe062abdaf3658ec23485f6ec58f6d5cc378809
sourcedatasets/_sources/ledgerlite
refHEAD
mutationnone
timeout1,800 s
hidden testshidden_tests
public testspublic_tests
definition digestsha256:8a516b1565f619f6b2fddfcc8fe062abdaf3658ec23485f6ec58f6d5cc378809
definition path/mnt/data2/dev/develop/ak-bench/datasets/new_feature/ledgerlite-reconcile-001

Runs of this task

in the order they started; the parameters column is the configuration each run was executed under

runparametersstartedagentrepetitionstatusverdictscoretimeinput tokensgenerated tokenstotal tokenstool callsfailure classerror
run-20261102-175103-7eadf2c3knowledge_enabled=False, model=qwen/Qwen3.8-27B2026-11-02 17:51:03codex@codex-cli 0.166.01timeoutnot measurednot measured15 m 4 s590,24323,356613,59926agentThe two identical entries are mapping to the same one due to how frozen dataclass equality works — both e1 and e2 have matching field values, so they're treated as equal. That's why the dict comprehension collapsed to a single key when it built the entry_of mapping.
run-20261102-180608-fb09d647knowledge_enabled=False, model=qwen/Qwen3.8-27B2026-11-02 18:06:08codex@codex-cli 0.166.02timeoutnot measurednot measured15 m 4 s306,91223,952330,86416agentI need to fix my test expectations — the test had incorrect sign conventions. Let me trace through what the correct signs should be for each line: `e3` has a debit-normal account with a debit of 700, so it should be +700, meaning L3 should be +700; `e1` is a liability account with a credit of 1000, giving -1000 for L1. The implementation is actually working correctly — the test failures were in my own test expectations, not the code itself. I need to fix the test data: L3 should be +700 instead of -700, and I'll adjust the expected difference_cents and open_candidates accordingly. Then I'll re-run the full test suite to verify everything passes.
run-20261102-182113-fb3e7151knowledge_enabled=False, model=qwen/Qwen3.8-27B2026-11-02 18:21:13codex@codex-cli 0.166.03timeoutnot measurednot measured15 m 4 s417,45224,042441,49420agentI'm checking the line 115 assertion to verify the entry ID matches "e-b", but I'm confused about which entries have the reference "ZZ" — both e-a and e-b do, so I need to figure out which one the assertion is actually pointing to.
run-20261102-183618-1b2f49a8knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next2026-11-02 18:36:18codex@codex-cli 0.166.01timeoutnot measurednot measured15 m 4 s788,75733,258822,01527agentTests are green and formatting is clean. Re-running my full spec harness once more and cleaning the `.pyc` noise my runs created.
run-20261102-185122-1a50269fknowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next2026-11-02 18:51:22codex@codex-cli 0.166.02completedpartial0.95010 m 2 s641,24020,168661,40824none—
run-20261102-190125-007b2709knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next2026-11-02 19:01:25codex@codex-cli 0.166.03completedpartial0.9509 m 34 s528,14518,654546,79917none—
run-20261102-191101-7b74b38fknowledge_enabled=False, model=auto_codex2026-11-02 19:11:01codex@codex-cli 0.166.01completedpartial0.9509 m 56 s392,28314,473406,75620none—
run-20261102-192057-a14677ebknowledge_enabled=False, model=auto_codex2026-11-02 19:20:57codex@codex-cli 0.166.02completedpassed1.0008 m 34 s439,89515,003454,89819none—
run-20261102-192932-7bb86699knowledge_enabled=False, model=auto_codex2026-11-02 19:29:32codex@codex-cli 0.166.03completedpassed1.0008 m 49 s435,34914,597449,94621none—
run-20261102-193822-f24e2c68knowledge_enabled=True, model=auto_codex2026-11-02 19:38:22codex@codex-cli 0.166.01completedpassed1.00010 m 41 s452,35117,677470,02821none—
run-20261102-194904-cb1d7f72knowledge_enabled=True, model=auto_codex2026-11-02 19:49:04codex@codex-cli 0.166.02completedpassed1.0008 m 9 s439,70212,057451,75921none—
run-20261102-195713-1edd6fafknowledge_enabled=True, model=auto_codex2026-11-02 19:57:13codex@codex-cli 0.166.03completedpartial0.95011 m 48 s732,53118,140750,67127none—

Not verified

Configurations that are not verified. They stay in the report, because a result produced under them is only as trustworthy as they are.
kindsubjectstatusrunnote
pluginagentic_knowledgeunsupportedrun-20261102-175103-7eadf2c3LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-175103-7eadf2c3descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-180608-fb09d647LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-180608-fb09d647descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-182113-fb3e7151LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-182113-fb3e7151descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-183618-1b2f49a8LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-183618-1b2f49a8descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-185122-1a50269fLLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-185122-1a50269fdescriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-190125-007b2709LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-190125-007b2709descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-191101-7b74b38fLLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-191101-7b74b38fdescriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-192057-a14677ebLLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-192057-a14677ebdescriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-192932-7bb86699LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-192932-7bb86699descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-193822-f24e2c68LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-193822-f24e2c68descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-194904-cb1d7f72LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-194904-cb1d7f72descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-195713-1edd6fafLLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-195713-1edd6fafdescriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
modelqwen/Qwen3.8-27Bunverifiedrun-20261102-175103-7eadf2c3
modelqwen/Qwen3.8-27Bunverifiedrun-20261102-180608-fb09d647
modelqwen/Qwen3.8-27Bunverifiedrun-20261102-182113-fb3e7151
modelqwen/Qwen3.8-Flash-Nextunverifiedrun-20261102-183618-1b2f49a8
modelqwen/Qwen3.8-Flash-Nextunverifiedrun-20261102-185122-1a50269f
modelqwen/Qwen3.8-Flash-Nextunverifiedrun-20261102-190125-007b2709
modelauto_codexunverifiedrun-20261102-191101-7b74b38f
modelauto_codexunverifiedrun-20261102-192057-a14677eb
modelauto_codexunverifiedrun-20261102-192932-7bb86699
modelauto_codexunverifiedrun-20261102-193822-f24e2c68
modelauto_codexunverifiedrun-20261102-194904-cb1d7f72
modelauto_codexunverifiedrun-20261102-195713-1edd6faf