AK-Benchbenchmark report

Task

slugify-001@1.0.0

runs
12
success rate
91.7%
mean time
5 m 5 s
input tokens
2,058,223
generated tokens
51,938
tokens
2,110,161
tool calls
156

What the task asks

Implement slugify from the specification

textkit.slugify is declared and specified but raises NotImplementedError. The agent implements it from the written contract, keeping casefold_word intact.

full description (markdown source)
# Implement `slugify` from the specification

`textkit.slugify` in `src/textkit/slugify.py` is declared, documented and
tested, but its body raises `NotImplementedError`. Write it.

## Contract

`slugify(text: str, separator: str = "-") -> str` builds a URL slug:

1. The result is lowercase.
2. ASCII letters (`a-z`, `A-Z`) and digits (`0-9`) are kept.
3. Every other character — including spaces, punctuation and non-ASCII letters
   such as `ż` or `é` — acts as a separator. There is no transliteration.
4. A run of one or more separator characters collapses to a single `separator`.
5. The result never starts or ends with `separator`.
6. An input with no letters or digits yields an empty string.
7. `separator` is a single character and is inserted verbatim.

## Examples

| call | result |
|---|---|
| `slugify("Hello World")` | `"hello-world"` |
| `slugify("Hello, World!")` | `"hello-world"` |
| `slugify("one -- two ___ three")` | `"one-two-three"` |
| `slugify("  --Trimmed--  ")` | `"trimmed"` |
| `slugify("AK-Bench 2026")` | `"ak-bench-2026"` |
| `slugify("Zażółć")` | `"za"` |
| `slugify("Zażółć gęślami")` | `"za-g-lami"` |
| `slugify("!!!")` | `""` |
| `slugify("")` | `""` |
| `slugify("Hello World", separator="_")` | `"hello_world"` |

Applying `slugify` to its own result must not change it again.

## Boundaries

- Implement it in `src/textkit/slugify.py`; the public API of the `textkit`
  package stays as it is.
- Do not modify the tests and do not add third-party dependencies — the standard
  library is enough.
- The repository's own test suite (`pytest`) has to stay green, so
  `textkit.casefold_word` must keep working.

Constraints

Notes

The base state fails the hidden tests by design: the feature is missing, which is the opposite of the bug-fix convention where hidden tests pass on the base commit.

Recorded identity

fieldvalue
uidslugify-001@1.0.0
categorynew_feature
difficultyeasy
evaluatorpytest_evaluator
base commitHEAD
repositoryrepo-004eb785a5e88d8a
snapshot digestsha256:2c39bf1195b0a52e05d98d63da379bdb54eb4cf185e30c2eb24e81eb42230177
sourcedatasets/_sources/textkit
refHEAD
mutationnone
timeout300 s
hidden testshidden_tests
public testspublic_tests
definition digestsha256:2c39bf1195b0a52e05d98d63da379bdb54eb4cf185e30c2eb24e81eb42230177
definition path/mnt/data2/dev/develop/ak-bench/datasets/new_feature/slugify-001

Runs of this task

in the order they started; the parameters column is the configuration each run was executed under

runparametersstartedagentrepetitionstatusverdictscoretimeinput tokensgenerated tokenstotal tokenstool callsfailure classerror
run-20261102-080204-89ac58c6knowledge_enabled=False, model=qwen/Qwen3.8-27B2026-11-02 08:02:04codex@codex-cli 0.166.01completedpassed1.0002 m 5 s88,7601,40690,1669
run-20261102-080410-d2c89897knowledge_enabled=False, model=qwen/Qwen3.8-27B2026-11-02 08:04:10codex@codex-cli 0.166.02timeoutnot measured15 m 2 s989,57118,7871,008,35851agentI'm looking at the test results more carefully now—most of them are passing, but there are two specific cases where the hyphen handling is producing unexpected triple hyphens instead of the expected output. The issue seems to be that when I'm building the test strings with the hyphen character, something about how the input is being processed is causing the hyphens to multiply in the output.
run-20261102-081913-0789e06aknowledge_enabled=False, model=qwen/Qwen3.8-27B2026-11-02 08:19:13codex@codex-cli 0.166.03completedpassed1.0002 m 35 s113,2332,116115,34911
run-20261102-082149-abdd844bknowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next2026-11-02 08:21:49codex@codex-cli 0.166.01completedpassed1.00011 m 17 s147,9947,191155,18512
run-20261102-083306-81799937knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next2026-11-02 08:33:06codex@codex-cli 0.166.02completedpassed1.0002 m 6 s77,3502,52779,8778
run-20261102-083513-c8a810d4knowledge_enabled=False, model=qwen/Qwen3.8-Flash-Next2026-11-02 08:35:13codex@codex-cli 0.166.03completedpassed1.0009 m 13 s139,4507,093146,54315
run-20261102-084427-5c4e6c25knowledge_enabled=False, model=auto_codex2026-11-02 08:44:27codex@codex-cli 0.166.01completedpassed1.0002 m 54 s92,9172,28995,2069
run-20261102-084722-12cb8519knowledge_enabled=False, model=auto_codex2026-11-02 08:47:22codex@codex-cli 0.166.02completedpassed1.0002 m 3 s68,4371,56470,0017
run-20261102-084925-db233c9dknowledge_enabled=False, model=auto_codex2026-11-02 08:49:25codex@codex-cli 0.166.03completedpassed1.0001 m 60 s67,0291,57168,6007
run-20261102-085126-7081e5a7knowledge_enabled=True, model=auto_codex2026-11-02 08:51:26codex@codex-cli 0.166.01completedpassed1.0005 m 50 s82,1122,40484,5168
run-20261102-085717-2d40efc9knowledge_enabled=True, model=auto_codex2026-11-02 08:57:17codex@codex-cli 0.166.02completedpassed1.0003 m 35 s112,6422,849115,49111
run-20261102-090053-801966e8knowledge_enabled=True, model=auto_codex2026-11-02 09:00:53codex@codex-cli 0.166.03completedpassed1.0002 m 15 s78,7282,14180,8698

Not verified

Configurations that are not verified. They stay in the report, because a result produced under them is only as trustworthy as they are.
kindsubjectstatusrunnote
pluginagentic_knowledgeunsupportedrun-20261102-080204-89ac58c6LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-080204-89ac58c6descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-080410-d2c89897LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-080410-d2c89897descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-081913-0789e06aLLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-081913-0789e06adescriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-082149-abdd844bLLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-082149-abdd844bdescriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-083306-81799937LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-083306-81799937descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-083513-c8a810d4LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-083513-c8a810d4descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-084427-5c4e6c25LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-084427-5c4e6c25descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-084722-12cb8519LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-084722-12cb8519descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-084925-db233c9dLLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-084925-db233c9ddescriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-085126-7081e5a7LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-085126-7081e5a7descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-085717-2d40efc9LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-085717-2d40efc9descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
pluginagentic_knowledgeunsupportedrun-20261102-090053-801966e8LLM_ROUTER_KNOWLEDGE_AGENTIC_ENABLED is not set, so the plugin keeps its built-in default; verification methods effective_router_config, plugin_runtime_events, …
pluginagentic_routing_codexunsupportedrun-20261102-090053-801966e8descriptor declares no plugin-level toggle (benchmark.enabled_when), so activation is not observable and enabled is reported as false; verification methods resp …
modelqwen/Qwen3.8-27Bunverifiedrun-20261102-080204-89ac58c6
modelqwen/Qwen3.8-27Bunverifiedrun-20261102-080410-d2c89897
modelqwen/Qwen3.8-27Bunverifiedrun-20261102-081913-0789e06a
modelqwen/Qwen3.8-Flash-Nextunverifiedrun-20261102-082149-abdd844b
modelqwen/Qwen3.8-Flash-Nextunverifiedrun-20261102-083306-81799937
modelqwen/Qwen3.8-Flash-Nextunverifiedrun-20261102-083513-c8a810d4
modelauto_codexunverifiedrun-20261102-084427-5c4e6c25
modelauto_codexunverifiedrun-20261102-084722-12cb8519
modelauto_codexunverifiedrun-20261102-084925-db233c9d
modelauto_codexunverifiedrun-20261102-085126-7081e5a7
modelauto_codexunverifiedrun-20261102-085717-2d40efc9
modelauto_codexunverifiedrun-20261102-090053-801966e8