Task
notes-search-001@1.0.0
runs
12
success rate
66.7%
mean time
8 m 52 s
input tokens
2,184,305
generated tokens
83,361
tokens
2,267,666
tool calls
175
What the task asks
Implement search_notes from its specification
search_notes is declared and specified in src/notesapi/search.py but raises NotImplementedError. Implement it from the written contract, keeping the store and its tests intact.
full description (markdown source)
# Implement `search_notes` from its specification `search_notes` in `src/notesapi/search.py` is declared, documented and tested, but its body raises `NotImplementedError`. Write it. The full contract — matching, ranking, filtering and paging, in the order the rules are applied — is in the module's docstring. Read it before writing a line of code; it is the specification, and the evaluation measures against it. ## The short version - A query is whitespace-separated terms; a note matches when **every** term occurs (case-insensitively, as a substring) in its body or a tag. - An empty or blank query matches every live note. - Score: one point per term found in the body, two per term found in a tag. - Order: score descending, then `updated_at` descending, then `note_id` ascending. - Deleted notes never match. With `tags`, a note must carry every one of them (case-insensitively) to be considered. `offset` skips, `limit` caps (`None` = no cap); a negative `offset` or `limit` raises `ValueError`. - Return the notes, not their ids. Never modify the store. ## Boundaries - Implement it in `src/notesapi/search.py`; the public API of `notesapi` stays as it is. - Do not modify the repository test suite under `tests/` and do not add third-party dependencies — the standard library is enough. - The repository's own test suite (`pytest`) has to stay green, so the `store`, `errors`, `sync` modules and their behaviour must keep working.
Constraints
- do not change the public API of the notesapi package
- do not modify the repository test suite under tests/
- do not change the store or the error types
- do not add third-party dependencies
- do not modify the search spec in the module docstring
Notes
The base state fails the hidden tests by design: the feature is missing, which is the opposite of the bug-fix convention where hidden tests pass on the base commit.
Recorded identity
| field | value |
|---|---|
| uid | notes-search-001@1.0.0 |
| category | new_feature |
| difficulty | hard |
| evaluator | pytest_evaluator |
| base commit | HEAD |
| repository | repo-5feced561a55273b |
| snapshot digest | sha256:b4d8c715c5f3cb5a585fb10473c530f21b74e6405d23158d68b324e1ea21315b |
| source | datasets/_sources/notesapi |
| ref | HEAD |
| mutation | none |
| timeout | 600 s |
| hidden tests | hidden_tests |
| public tests | public_tests |
| definition digest | sha256:b4d8c715c5f3cb5a585fb10473c530f21b74e6405d23158d68b324e1ea21315b |
| definition path | /mnt/data2/dev/develop/ak-bench/datasets/new_feature/notes-search-001 |
Runs of this task
in the order they started; the parameters column is the configuration each run was executed under
| run | parameters | started | agent | repetition | status | verdict | score | time | input tokens | generated tokens | total tokens | tool calls | failure class | error |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| run-20261102-050517-4824113c | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 05:05:17 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 6 m 23 s | 257,102 | 7,946 | 265,048 | 19 | ||
| run-20261102-051140-154e7756 | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 05:11:40 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 7 m 15 s | 322,255 | 9,563 | 331,818 | 21 | ||
| run-20261102-051856-7a01f2f0 | knowledge_enabled=False, model=qwen/Qwen3.8-27B | 2026-11-02 05:18:56 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 8 m 27 s | 334,137 | 11,345 | 345,482 | 20 | ||
| run-20261102-052723-7c46d753 | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 05:27:23 | codex@codex-cli 0.166.0 | 1 | timeout | not measured | 15 m 3 s | 25,668 | 632 | 26,300 | 6 | agent | I'll start by exploring the repo structure and reading the spec. | |
| run-20261102-054227-dc5f0497 | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 05:42:27 | codex@codex-cli 0.166.0 | 2 | timeout | not measured | 15 m 2 s | 68,159 | 4,703 | 72,862 | 9 | agent | Verifying against the spec rules with a throwaway script, plus the repo suite. | |
| run-20261102-055730-c94232f9 | knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next | 2026-11-02 05:57:30 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 5 m 28 s | 193,962 | 10,301 | 204,263 | 17 | ||
| run-20261102-060259-4681bb55 | knowledge_enabled=False, model=auto_codex | 2026-11-02 06:02:59 | codex@codex-cli 0.166.0 | 1 | timeout | not measured | 15 m 3 s | 92,655 | 5,287 | 97,942 | 11 | agent | Found a real bug: with `for ... else`, my scoring block sits inside `if terms:`, so a blank query appended nothing and returned `[]`. Restructuring so an empty term list scores 0 and matches. | |
| run-20261102-061803-ca793f25 | knowledge_enabled=False, model=auto_codex | 2026-11-02 06:18:03 | codex@codex-cli 0.166.0 | 2 | timeout | not measured | 15 m 3 s | 364,794 | 13,619 | 378,413 | 24 | agent | The implementation is correct — the last "failure" was another typo in my hand-written expected list (I swapped `t2`/`t1`), and the final direct probe confirmed the spec order `['t0','d','tg','t2','t1','c','b','a','e','m2','m1']` and correct scoring. Cleaning up the stale `.pyc` churn so the diff is just the implementation: | |
| run-20261102-063306-62006f1c | knowledge_enabled=False, model=auto_codex | 2026-11-02 06:33:06 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 6 m 14 s | 127,093 | 4,757 | 131,850 | 12 | ||
| run-20261102-063921-31a5efef | knowledge_enabled=True, model=auto_codex | 2026-11-02 06:39:21 | codex@codex-cli 0.166.0 | 1 | completed | passed | 1.000 | 4 m 21 s | 151,027 | 5,654 | 156,681 | 13 | ||
| run-20261102-064343-33d90543 | knowledge_enabled=True, model=auto_codex | 2026-11-02 06:43:43 | codex@codex-cli 0.166.0 | 2 | completed | passed | 1.000 | 2 m 18 s | 56,625 | 2,030 | 58,655 | 8 | ||
| run-20261102-064601-0d71057a | knowledge_enabled=True, model=auto_codex | 2026-11-02 06:46:01 | codex@codex-cli 0.166.0 | 3 | completed | passed | 1.000 | 5 m 50 s | 190,828 | 7,524 | 198,352 | 15 |
Not verified
Configurations that are not verified. They stay in the report, because a result produced under them is only as trustworthy as they are.
| kind | subject | status | run | note |
|---|---|---|---|---|
| plugin | agentic_knowledge | unsupported | run-20261102-050517-4824113c | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-050517-4824113c | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-051140-154e7756 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-051140-154e7756 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-051856-7a01f2f0 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-051856-7a01f2f0 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-052723-7c46d753 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-052723-7c46d753 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-054227-dc5f0497 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-054227-dc5f0497 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-055730-c94232f9 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-055730-c94232f9 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-060259-4681bb55 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-060259-4681bb55 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-061803-ca793f25 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-061803-ca793f25 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-063306-62006f1c | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-063306-62006f1c | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-063921-31a5efef | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-063921-31a5efef | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-064343-33d90543 | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-064343-33d90543 | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| plugin | agentic_knowledge | unsupported | run-20261102-064601-0d71057a | LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, … |
| plugin | agentic_routing_codex | unsupported | run-20261102-064601-0d71057a | descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp … |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-050517-4824113c | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-051140-154e7756 | |
| model | qwen/Qwen3.8-27B | unverified | run-20261102-051856-7a01f2f0 | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-052723-7c46d753 | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-054227-dc5f0497 | |
| model | qwen/Qwen3.8-Flash-Next | unverified | run-20261102-055730-c94232f9 | |
| model | auto_codex | unverified | run-20261102-060259-4681bb55 | |
| model | auto_codex | unverified | run-20261102-061803-ca793f25 | |
| model | auto_codex | unverified | run-20261102-063306-62006f1c | |
| model | auto_codex | unverified | run-20261102-063921-31a5efef | |
| model | auto_codex | unverified | run-20261102-064343-33d90543 | |
| model | auto_codex | unverified | run-20261102-064601-0d71057a |