AK-Benchbenchmark report

Task

notes-search-001@1.0.0

runs
12
success rate
66.7%
mean time
8 m 52 s
input tokens
2,184,305
generated tokens
83,361
tokens
2,267,666
tool calls
175

What the task asks

Implement search_notes from its specification

search_notes is declared and specified in src/notesapi/search.py but raises NotImplementedError. Implement it from the written contract, keeping the store and its tests intact.

full description (markdown source)
# Implement `search_notes` from its specification

`search_notes` in `src/notesapi/search.py` is declared, documented and tested,
but its body raises `NotImplementedError`. Write it.

The full contract — matching, ranking, filtering and paging, in the order the
rules are applied — is in the module's docstring. Read it before writing a line
of code; it is the specification, and the evaluation measures against it.

## The short version

- A query is whitespace-separated terms; a note matches when **every** term
  occurs (case-insensitively, as a substring) in its body or a tag.
- An empty or blank query matches every live note.
- Score: one point per term found in the body, two per term found in a tag.
- Order: score descending, then `updated_at` descending, then `note_id`
  ascending.
- Deleted notes never match. With `tags`, a note must carry every one of them
  (case-insensitively) to be considered. `offset` skips, `limit` caps
  (`None` = no cap); a negative `offset` or `limit` raises `ValueError`.
- Return the notes, not their ids. Never modify the store.

## Boundaries

- Implement it in `src/notesapi/search.py`; the public API of `notesapi` stays
  as it is.
- Do not modify the repository test suite under `tests/` and do not add
  third-party dependencies — the standard library is enough.
- The repository's own test suite (`pytest`) has to stay green, so the
  `store`, `errors`, `sync` modules and their behaviour must keep working.

Constraints

Notes

The base state fails the hidden tests by design: the feature is missing, which is the opposite of the bug-fix convention where hidden tests pass on the base commit.

Recorded identity

fieldvalue
uidnotes-search-001@1.0.0
categorynew_feature
difficultyhard
evaluatorpytest_evaluator
base commitHEAD
repositoryrepo-5feced561a55273b
snapshot digestsha256:b4d8c715c5f3cb5a585fb10473c530f21b74e6405d23158d68b324e1ea21315b
sourcedatasets/_sources/notesapi
refHEAD
mutationnone
timeout600 s
hidden testshidden_tests
public testspublic_tests
definition digestsha256:b4d8c715c5f3cb5a585fb10473c530f21b74e6405d23158d68b324e1ea21315b
definition path/mnt/data2/dev/develop/ak-bench/datasets/new_feature/notes-search-001

Runs of this task

in the order they started; the parameters column is the configuration each run was executed under

runparametersstartedagentrepetitionstatusverdictscoretimeinput tokensgenerated tokenstotal tokenstool callsfailure classerror
run-20261102-050517-4824113cknowledge_enabled=False, model=qwen/Qwen3.8-27B2026-11-02 05:05:17codex@codex-cli 0.166.01completedpassed1.0006 m 23 s257,1027,946265,04819
run-20261102-051140-154e7756knowledge_enabled=False, model=qwen/Qwen3.8-27B2026-11-02 05:11:40codex@codex-cli 0.166.02completedpassed1.0007 m 15 s322,2559,563331,81821
run-20261102-051856-7a01f2f0knowledge_enabled=False, model=qwen/Qwen3.8-27B2026-11-02 05:18:56codex@codex-cli 0.166.03completedpassed1.0008 m 27 s334,13711,345345,48220
run-20261102-052723-7c46d753knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next2026-11-02 05:27:23codex@codex-cli 0.166.01timeoutnot measured15 m 3 s25,66863226,3006agentI'll start by exploring the repo structure and reading the spec.
run-20261102-054227-dc5f0497knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next2026-11-02 05:42:27codex@codex-cli 0.166.02timeoutnot measured15 m 2 s68,1594,70372,8629agentVerifying against the spec rules with a throwaway script, plus the repo suite.
run-20261102-055730-c94232f9knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next2026-11-02 05:57:30codex@codex-cli 0.166.03completedpassed1.0005 m 28 s193,96210,301204,26317
run-20261102-060259-4681bb55knowledge_enabled=False, model=auto_codex2026-11-02 06:02:59codex@codex-cli 0.166.01timeoutnot measured15 m 3 s92,6555,28797,94211agentFound a real bug: with `for ... else`, my scoring block sits inside `if terms:`, so a blank query appended nothing and returned `[]`. Restructuring so an empty term list scores 0 and matches.
run-20261102-061803-ca793f25knowledge_enabled=False, model=auto_codex2026-11-02 06:18:03codex@codex-cli 0.166.02timeoutnot measured15 m 3 s364,79413,619378,41324agentThe implementation is correct — the last "failure" was another typo in my hand-written expected list (I swapped `t2`/`t1`), and the final direct probe confirmed the spec order `['t0','d','tg','t2','t1','c','b','a','e','m2','m1']` and correct scoring. Cleaning up the stale `.pyc` churn so the diff is just the implementation:
run-20261102-063306-62006f1cknowledge_enabled=False, model=auto_codex2026-11-02 06:33:06codex@codex-cli 0.166.03completedpassed1.0006 m 14 s127,0934,757131,85012
run-20261102-063921-31a5efefknowledge_enabled=True, model=auto_codex2026-11-02 06:39:21codex@codex-cli 0.166.01completedpassed1.0004 m 21 s151,0275,654156,68113
run-20261102-064343-33d90543knowledge_enabled=True, model=auto_codex2026-11-02 06:43:43codex@codex-cli 0.166.02completedpassed1.0002 m 18 s56,6252,03058,6558
run-20261102-064601-0d71057aknowledge_enabled=True, model=auto_codex2026-11-02 06:46:01codex@codex-cli 0.166.03completedpassed1.0005 m 50 s190,8287,524198,35215

Not verified

Configurations that are not verified. They stay in the report, because a result produced under them is only as trustworthy as they are.
kindsubjectstatusrunnote
pluginagentic_knowledgeunsupportedrun-20261102-050517-4824113cLLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-050517-4824113cdescriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-051140-154e7756LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-051140-154e7756descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-051856-7a01f2f0LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-051856-7a01f2f0descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-052723-7c46d753LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-052723-7c46d753descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-054227-dc5f0497LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-054227-dc5f0497descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-055730-c94232f9LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-055730-c94232f9descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-060259-4681bb55LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-060259-4681bb55descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-061803-ca793f25LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-061803-ca793f25descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-063306-62006f1cLLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-063306-62006f1cdescriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-063921-31a5efefLLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-063921-31a5efefdescriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-064343-33d90543LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-064343-33d90543descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-064601-0d71057aLLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-064601-0d71057adescriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
modelqwen/Qwen3.8-27Bunverifiedrun-20261102-050517-4824113c
modelqwen/Qwen3.8-27Bunverifiedrun-20261102-051140-154e7756
modelqwen/Qwen3.8-27Bunverifiedrun-20261102-051856-7a01f2f0
modelqwen/Qwen3.8-Flash-Nextunverifiedrun-20261102-052723-7c46d753
modelqwen/Qwen3.8-Flash-Nextunverifiedrun-20261102-054227-dc5f0497
modelqwen/Qwen3.8-Flash-Nextunverifiedrun-20261102-055730-c94232f9
modelauto_codexunverifiedrun-20261102-060259-4681bb55
modelauto_codexunverifiedrun-20261102-061803-ca793f25
modelauto_codexunverifiedrun-20261102-063306-62006f1c
modelauto_codexunverifiedrun-20261102-063921-31a5efef
modelauto_codexunverifiedrun-20261102-064343-33d90543
modelauto_codexunverifiedrun-20261102-064601-0d71057a