AK-Benchbenchmark report

Task

notes-sync-001@1.0.0

runs
12
success rate
100.0%
mean time
5 m 50 s
input tokens
1,971,921
generated tokens
74,178
tokens
2,046,099
tool calls
152

What the task asks

Implement the note synchronisation merge from its specification

notesapi.merge is declared and specified in src/notesapi/sync.py but raises NotImplementedError. Implement it from the written contract, keeping the store and its tests intact.

full description (markdown source)
# Implement `merge` from its specification

`merge` in `src/notesapi/sync.py` is declared, documented and tested, but its
body raises `NotImplementedError`. Write it.

The full contract — what counts as a write, how a higher `updated_at` wins, how
tombstones behave, and what the result object must expose — is in the module's
docstring. Read it before writing a line of code; it is the specification, and
the evaluation measures against it.

## The short version

- A note the client has and the server does not is an **insert**.
- A note both have, where the client's `updated_at` is strictly newer, is an
  **update**; when the note is a tombstone, the update is a **delete**.
- A note both have with the same `updated_at`: identical bodies and tags is
  **unchanged**, anything else is a **conflict** — the merge does not guess.
- A note the server has and the client does not is left alone; the client copy
  is not a list of everything that exists.
- The result's `wins` holds the notes the server should now carry, and the five
  lists of ids never overlap.

## Boundaries

- Implement it in `src/notesapi/sync.py`; the public API of `notesapi` stays
  as it is.
- Do not modify the repository test suite under `tests/` and do not add
  third-party dependencies — the standard library is enough.
- The repository's own test suite (`pytest`) has to stay green, so the
  `store`, `errors` and `search` modules and their behaviour must keep working.

Constraints

Notes

The base state fails the hidden tests by design: the feature is missing, which is the opposite of the bug-fix convention where hidden tests pass on the base commit.

Recorded identity

fieldvalue
uidnotes-sync-001@1.0.0
categorynew_feature
difficultyexpert
evaluatorpytest_evaluator
base commitHEAD
repositoryrepo-5feced561a55273b
snapshot digestsha256:3c91d627a989a86913d9a4e02b4d19f293d45758109112b2399ea49f69a84f20
sourcedatasets/_sources/notesapi
refHEAD
mutationnone
timeout900 s
hidden testshidden_tests
public testspublic_tests
definition digestsha256:3c91d627a989a86913d9a4e02b4d19f293d45758109112b2399ea49f69a84f20
definition path/mnt/data2/dev/develop/ak-bench/datasets/new_feature/notes-sync-001

Runs of this task

in the order they started; the parameters column is the configuration each run was executed under

runparametersstartedagentrepetitionstatusverdictscoretimeinput tokensgenerated tokenstotal tokenstool callsfailure classerror
run-20261102-065152-be03d69aknowledge_enabled=False, model=qwen/Qwen3.8-27B2026-11-02 06:51:52codex@codex-cli 0.166.01completedpassed1.0002 m 32 s68,6912,28470,9757
run-20261102-065424-0122485bknowledge_enabled=False, model=qwen/Qwen3.8-27B2026-11-02 06:54:24codex@codex-cli 0.166.02completedpassed1.0003 m 58 s147,5283,954151,48213
run-20261102-065823-56d0518eknowledge_enabled=False, model=qwen/Qwen3.8-27B2026-11-02 06:58:23codex@codex-cli 0.166.03completedpassed1.0004 m 1 s144,2093,841148,05012
run-20261102-070225-983f28a4knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next2026-11-02 07:02:25codex@codex-cli 0.166.01completedpassed1.0005 m 26 s113,9877,555121,54210
run-20261102-070752-ee5e374fknowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next2026-11-02 07:07:52codex@codex-cli 0.166.02completedpassed1.0007 m 42 s297,81511,995309,81018
run-20261102-071535-a34798d9knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next2026-11-02 07:15:35codex@codex-cli 0.166.03completedpassed1.0006 m 19 s194,7469,719204,46513
run-20261102-072154-ce0a96c9knowledge_enabled=False, model=auto_codex2026-11-02 07:21:54codex@codex-cli 0.166.01completedpassed1.0009 m 24 s292,5866,876299,46221
run-20261102-073119-046c63c5knowledge_enabled=False, model=auto_codex2026-11-02 07:31:19codex@codex-cli 0.166.02completedpassed1.0009 m 54 s186,1957,893194,08814
run-20261102-074114-fb309dd3knowledge_enabled=False, model=auto_codex2026-11-02 07:41:14codex@codex-cli 0.166.03completedpassed1.0004 m 22 s139,4534,875144,32811
run-20261102-074537-1abe2bd8knowledge_enabled=True, model=auto_codex2026-11-02 07:45:37codex@codex-cli 0.166.01completedpassed1.0007 m 49 s168,2586,167174,42512
run-20261102-075327-d3b09ae5knowledge_enabled=True, model=auto_codex2026-11-02 07:53:27codex@codex-cli 0.166.02completedpassed1.0004 m 8 s92,3354,29096,62511
run-20261102-075736-7dc8b5faknowledge_enabled=True, model=auto_codex2026-11-02 07:57:36codex@codex-cli 0.166.03completedpassed1.0004 m 27 s126,1184,729130,84710

Not verified

Configurations that are not verified. They stay in the report, because a result produced under them is only as trustworthy as they are.
kindsubjectstatusrunnote
pluginagentic_knowledgeunsupportedrun-20261102-065152-be03d69aLLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-065152-be03d69adescriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-065424-0122485bLLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-065424-0122485bdescriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-065823-56d0518eLLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-065823-56d0518edescriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-070225-983f28a4LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-070225-983f28a4descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-070752-ee5e374fLLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-070752-ee5e374fdescriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-071535-a34798d9LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-071535-a34798d9descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-072154-ce0a96c9LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-072154-ce0a96c9descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-073119-046c63c5LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-073119-046c63c5descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-074114-fb309dd3LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-074114-fb309dd3descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-074537-1abe2bd8LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-074537-1abe2bd8descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-075327-d3b09ae5LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-075327-d3b09ae5descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-075736-7dc8b5faLLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-075736-7dc8b5fadescriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
modelqwen/Qwen3.8-27Bunverifiedrun-20261102-065152-be03d69a
modelqwen/Qwen3.8-27Bunverifiedrun-20261102-065424-0122485b
modelqwen/Qwen3.8-27Bunverifiedrun-20261102-065823-56d0518e
modelqwen/Qwen3.8-Flash-Nextunverifiedrun-20261102-070225-983f28a4
modelqwen/Qwen3.8-Flash-Nextunverifiedrun-20261102-070752-ee5e374f
modelqwen/Qwen3.8-Flash-Nextunverifiedrun-20261102-071535-a34798d9
modelauto_codexunverifiedrun-20261102-072154-ce0a96c9
modelauto_codexunverifiedrun-20261102-073119-046c63c5
modelauto_codexunverifiedrun-20261102-074114-fb309dd3
modelauto_codexunverifiedrun-20261102-074537-1abe2bd8
modelauto_codexunverifiedrun-20261102-075327-d3b09ae5
modelauto_codexunverifiedrun-20261102-075736-7dc8b5fa