Task
notes-sync-001@1.0.0
runs
12
success rate
100.0%
mean time
5 m 50 s
input tokens
1,971,921
generated tokens
74,178
tokens
2,046,099
tool calls
152
What the task asks
Implement the note synchronisation merge from its specification
notesapi.merge is declared and specified in src/notesapi/sync.py but raises NotImplementedError. Implement it from the written contract, keeping the store and its tests intact.
full description (markdown source)
# Implement `merge` from its specification `merge` in `src/notesapi/sync.py` is declared, documented and tested, but its body raises `NotImplementedError`. Write it. The full contract — what counts as a write, how a higher `updated_at` wins, how tombstones behave, and what the result object must expose — is in the module's docstring. Read it before writing a line of code; it is the specification, and the evaluation measures against it. ## The short version - A note the client has and the server does not is an **insert**. - A note both have, where the client's `updated_at` is strictly newer, is an **update**; when the note is a tombstone, the update is a **delete**. - A note both have with the same `updated_at`: identical bodies and tags is **unchanged**, anything else is a **conflict** — the merge does not guess. - A note the server has and the client does not is left alone; the client copy is not a list of everything that exists. - The result's `wins` holds the notes the server should now carry, and the five lists of ids never overlap. ## Boundaries - Implement it in `src/notesapi/sync.py`; the public API of `notesapi` stays as it is. - Do not modify the repository test suite under `tests/` and do not add third-party dependencies — the standard library is enough. - The repository's own test suite (`pytest`) has to stay green, so the `store`, `errors` and `search` modules and their behaviour must keep working.
Constraints
- do not change the public API of the notesapi package
- do not modify the repository test suite under tests/
- do not change the store or the error types
- do not add third-party dependencies
- do not modify the merge spec in the module docstring
- do not change the shape of MergeResult
Notes
The base state fails the hidden tests by design: the feature is missing, which is the opposite of the bug-fix convention where hidden tests pass on the base commit.
Recorded identity
| field | value |
|---|---|
| uid | notes-sync-001@1.0.0 |
| category | new_feature |
| difficulty | expert |
| evaluator | pytest_evaluator |
| base commit | HEAD |
| repository | repo-5feced561a55273b |
| snapshot digest | sha256:3c91d627a989a86913d9a4e02b4d19f293d45758109112b2399ea49f69a84f20 |
| source | datasets/_sources/notesapi |
| ref | HEAD |
| mutation | none |
| timeout | 900 s |
| hidden tests | hidden_tests |
| public tests | public_tests |
| definition digest | sha256:3c91d627a989a86913d9a4e02b4d19f293d45758109112b2399ea49f69a84f20 |
| definition path | /mnt/data2/dev/develop/ak-bench/datasets/new_feature/notes-sync-001 |
Runs of this task
in the order they started; the parameters column is the configuration each run was executed under
| run | parameters | started | agent | repetition | status | verdict | score | time | input tokens | generated tokens | total tokens | tool calls | failure class | error |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| run-20261102-065152-be03d69a | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 06:51:52 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 2 m 32 s | 68,691 | 2,284 | 70,975 | 7 | ||
| run-20261102-065424-0122485b | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 06:54:24 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 3 m 58 s | 147,528 | 3,954 | 151,482 | 13 | ||
| run-20261102-065823-56d0518e | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 06:58:23 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 4 m 1 s | 144,209 | 3,841 | 148,050 | 12 | ||
| run-20261102-070225-983f28a4 | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 07:02:25 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 5 m 26 s | 113,987 | 7,555 | 121,542 | 10 | ||
| run-20261102-070752-ee5e374f | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 07:07:52 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 7 m 42 s | 297,815 | 11,995 | 309,810 | 18 | ||
| run-20261102-071535-a34798d9 | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 07:15:35 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 6 m 19 s | 194,746 | 9,719 | 204,465 | 13 | ||
| run-20261102-072154-ce0a96c9 | knowledge_enabled=False, model=auto_codex | 2026-11-02 07:21:54 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 9 m 24 s | 292,586 | 6,876 | 299,462 | 21 | ||
| run-20261102-073119-046c63c5 | knowledge_enabled=False, model=auto_codex | 2026-11-02 07:31:19 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 9 m 54 s | 186,195 | 7,893 | 194,088 | 14 | ||
| run-20261102-074114-fb309dd3 | knowledge_enabled=False, model=auto_codex | 2026-11-02 07:41:14 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 4 m 22 s | 139,453 | 4,875 | 144,328 | 11 | ||
| run-20261102-074537-1abe2bd8 | knowledge_enabled=True, model=auto_codex | 2026-11-02 07:45:37 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 7 m 49 s | 168,258 | 6,167 | 174,425 | 12 | ||
| run-20261102-075327-d3b09ae5 | knowledge_enabled=True, model=auto_codex | 2026-11-02 07:53:27 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 4 m 8 s | 92,335 | 4,290 | 96,625 | 11 | ||
| run-20261102-075736-7dc8b5fa | knowledge_enabled=True, model=auto_codex | 2026-11-02 07:57:36 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 4 m 27 s | 126,118 | 4,729 | 130,847 | 10 |
Not verified
Configurations that are not verified. They stay in the report, because a result produced under them is only as trustworthy as they are.
| kind | subject | status | run | note |
|---|---|---|---|---|
| plugin | agentic_knowledge | unsupported | run-20261102-065152-be03d69a | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-065152-be03d69a | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-065424-0122485b | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-065424-0122485b | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-065823-56d0518e | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-065823-56d0518e | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-070225-983f28a4 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-070225-983f28a4 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-070752-ee5e374f | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-070752-ee5e374f | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-071535-a34798d9 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-071535-a34798d9 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-072154-ce0a96c9 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-072154-ce0a96c9 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-073119-046c63c5 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-073119-046c63c5 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-074114-fb309dd3 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-074114-fb309dd3 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-074537-1abe2bd8 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-074537-1abe2bd8 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-075327-d3b09ae5 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-075327-d3b09ae5 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-075736-7dc8b5fa | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-075736-7dc8b5fa | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-065152-be03d69a | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-065424-0122485b | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-065823-56d0518e | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-070225-983f28a4 | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-070752-ee5e374f | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-071535-a34798d9 | |
| model | auto_codex | unverified | run-20261102-072154-ce0a96c9 | |
| model | auto_codex | unverified | run-20261102-073119-046c63c5 | |
| model | auto_codex | unverified | run-20261102-074114-fb309dd3 | |
| model | auto_codex | unverified | run-20261102-074537-1abe2bd8 | |
| model | auto_codex | unverified | run-20261102-075327-d3b09ae5 | |
| model | auto_codex | unverified | run-20261102-075736-7dc8b5fa |