Tasks
every recorded task
| task | category | difficulty | evaluator | runs | evaluated | passed | success rate | mean time | input tokens | generated tokens | total tokens | tool calls |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bookshop-stock-002@1.0.0 | bug_fix | expert | pytest_evaluator | 12 | 12 | 12 | 100.0% | 4 m 5 s | 1,966,139 | 52,114 | 2,018,253 | 160 |
| bookshop-tiers-001@1.0.0 | bug_fix | hard | pytest_evaluator | 12 | 12 | 12 | 100.0% | 4 m 37 s | 1,224,693 | 32,561 | 1,257,254 | 111 |
| cadence-catchup-001@1.0.0 | new_feature | expert | pytest_evaluator | 12 | 3 | 1 | 8.3% | 14 m 26 s | 9,168,585 | 308,855 | 9,477,440 | 335 |
| cadence-takeover-001@1.0.0 | bug_fix | expert | pytest_evaluator | 12 | 12 | 12 | 100.0% | 3 m 60 s | 2,182,492 | 53,408 | 2,235,900 | 169 |
| freight-shop-001@1.0.0 | new_feature | expert | pytest_evaluator | 12 | 10 | 4 | 33.3% | 11 m 12 s | 6,169,405 | 237,688 | 6,407,093 | 262 |
| ledgerlite-rates-001@1.0.0 | bug_fix | expert | pytest_evaluator | 12 | 12 | 12 | 100.0% | 4 m 1 s | 2,088,806 | 58,729 | 2,147,535 | 157 |
| ledgerlite-reconcile-001@1.0.0 | new_feature | expert | pytest_evaluator | 12 | 8 | 4 | 33.3% | 11 m 29 s | 6,164,860 | 235,377 | 6,400,237 | 259 |
| notes-search-001@1.0.0 | new_feature | hard | pytest_evaluator | 12 | 8 | 8 | 66.7% | 8 m 52 s | 2,184,305 | 83,361 | 2,267,666 | 175 |
| notes-sync-001@1.0.0 | new_feature | expert | pytest_evaluator | 12 | 12 | 12 | 100.0% | 5 m 50 s | 1,971,921 | 74,178 | 2,046,099 | 152 |
| slugify-001@1.0.0 | new_feature | easy | pytest_evaluator | 12 | 11 | 11 | 91.7% | 5 m 5 s | 2,058,223 | 51,938 | 2,110,161 | 156 |
| stockpile-fefo-001@1.0.0 | new_feature | expert | pytest_evaluator | 12 | 12 | 12 | 100.0% | 6 m 12 s | 3,412,912 | 103,428 | 3,516,340 | 203 |
| stockpile-writeoff-001@1.0.0 | bug_fix | expert | pytest_evaluator | 12 | 12 | 12 | 100.0% | 3 m 40 s | 2,345,811 | 47,327 | 2,393,138 | 156 |
| wordcount-001@1.0.0 | bug_fix | easy | pytest_evaluator | 12 | 10 | 10 | 83.3% | 7 m 17 s | 880,965 | 20,251 | 901,216 | 102 |