AK-Bench

Agentic Knowledge Benchmark — an extensible platform for repeatable

evaluation of coding agents, LLM routers, models, and the plugins that extend

their capabilities (knowledge bases, routing, optimizations).

Overriding principle: the agent under test is a replaceable component.

AK-Bench does not implement an agent, a memory, or routing. It orchestrates

task execution through a chosen agent and verifies the outcome independently.

Status

The full plan lives in .plans/AK-Bench-implementation-plan.md (Polish).

Requirements

• Python 3.12+ (easiest through uv)
• Git
• Docker — optional, only for sandboxing runs

Quick start

uv venv --python 3.12 .venv
uv pip install -e ".[dev]"

akbench init          # directories + configuration + database (migrations to head)
akbench doctor        # environment readiness
akbench config show   # effective configuration and paths
akbench descriptor validate  # component descriptors
akbench dataset validate     # tasks (add --verify-mutations to prove defects)

Commands

Global options: --config/-c, --project-root, --verbose/-v, --quiet/-q, --version.

Configuration

configs/defaults.yaml, overridable through AKBENCH__SECTION__FIELD

environment variables:

export AKBENCH__LOGGING__LEVEL=DEBUG
export AKBENCH__DATABASE__URL=sqlite:////tmp/akbench.sqlite

Configuration lookup: --config → AKBENCH_CONFIG →

<project root>/configs/defaults.yaml → built-in defaults.

Project root: AKBENCH_HOME → the nearest ancestor whose pyproject.toml

declares the project name ak-bench → the current directory.

Unknown keys are rejected: an option that is silently ignored looks effective

while it is not.

Layout

src/akbench/
  cli/          # CLI layer (typer)
  config/       # Pydantic models + YAML/ENV loading
  core/         # errors, paths, time, logging
  descriptors/  # descriptor models, ENV allowlist, registry
  agents/       # adapter contract, agent registry, catalog, Codex adapter
  runtime/      # runtime manager contract, LLM Router adapter, execution modes
  evaluators/   # evaluator engine, pytest + command evaluators, isolation
  orchestration/  # run lifecycle state machine, the run orchestrator
  experiments/  # experiment descriptor, dimension registry, matrix strategies, planner
  storage/      # SQLAlchemy, models, Alembic migrations, migration runner
  datasets/     # task descriptors, dataset validation, repositories, workspaces
  bootstrap.py  # `akbench init` logic
  doctor.py     # environment diagnostics
tests/
  unit/         # paths, configuration, storage, doctor, descriptors, agents
  integration/  # CLI
  contract/     # shared adapter contract suite, fake agent, CLI stub
descriptors/    # component definitions (agents, plugins, models, ...)
datasets/       # task definitions, hidden tests and mutation patches
configs/        # configuration and experiment definitions
data/           # run directories, repository cache and database (git-ignored)
docs/           # integration and development documentation
spike/          # Phase 0 code (flow validation)

Documentation

• docs/AGENT_INTEGRATION.md — verified Codex CLI capabilities, including what

its interface does not provide (cost, cancellation, session reset)

• docs/ROUTER_INTEGRATION.md — LLM Router: healthcheck, models, plugins,

metrics and verification limits

• docs/DESCRIPTOR_SYSTEM.md — descriptor kinds, registries, ENV allowlist, dimension rules
• docs/DATASETS.md — tasks, repositories, repeatable workspaces, mutation verification
• docs/AGENT_ADAPTERS.md — the adapter contract, agent registry, capability gating, contract tests
• docs/RUNTIME_MANAGER.md — execution modes, the runtime manager, the LLM Router adapter, the configuration triad
• docs/EVALUATORS.md — the evaluator engine, verdicts, hidden-test isolation, raw results
• docs/RUN_ORCHESTRATOR.md — the run lifecycle, isolation, retry policy and run records
• docs/EXPERIMENTS.md — experiment descriptors, validation, matrix strategies and planning
• docs/PLUGINS.md — the plugin adapter contract, verification and what a plugin can honestly report
• docs/METRICS.md — the metric catalog, per-run collection, aggregation and baseline deltas
• docs/KNOWLEDGE_REPLAY.md — the memory experiment: arms, scenarios, the readiness gate and isolation
• docs/STORAGE.md — the experiment history: tables, artifacts, the reproducibility manifest, verification
• docs/REPORTING.md — the static HTML reports: layout, the rules the numbers obey, cost and verification
• docs/EXPORT.md — the JSON, CSV and HTML export and the secret screening that precedes it
• docs/REPRODUCIBILITY.md — what "the same conditions" means here and where it is tested
• docs/CLI.md — every implemented command and the interface facts that go with it
• docs/DEVELOPMENT.md — development environment, tests, migrations

Conventions

Code, comments, docstrings, docs and commit messages are in English. Only the

implementation plan (.plans/) is written in Polish.

License

MIT — see LICENSE.