Pair the runs
Run each task in a baseline condition and a project-harness condition using the same fixed inputs.
Reproducible evaluation
Benchmark the harness, not just the model
Run the same real coding tasks with and without your project harness. Hold the model, repository, prompt, tools, and budget constant; publish the traces and evidence behind every score.
What this measures
A controlled protocol, not a universal leaderboard
Harness performance depends on the repository, task, model, and tools. This protocol is designed for paired comparisons inside one environment. It does not claim that one harness score transfers to every codebase, and it does not publish results without attached traces and test output.
Use real failures
Choose tasks from resolved bugs and accepted changes, then hide the solution while preserving an objective oracle.
Score evidence
Award points from tests, traces, diffs, permission logs, and linked artifacts, not from the agent's self-report.
Publish uncertainty
Run multiple repetitions, report every result and failure, and avoid conclusions from a single stochastic run.
The paired comparison
Change the project harness, keep everything else fixed
Fixed in both conditions
- Repository commit and clean starting state
- Model provider, exact model version, and inference settings
- Task prompt, attachments, and acceptance criteria
- Shell, filesystem, and base coding-agent tool permissions
- Token, cost, wall-clock, and retry limits
- Test environment, dependencies, seeds, and network policy
Condition A: tool-only baseline
The coding-agent runtime can read and edit the repository and use the fixed base tools, but receives no project instruction files, scoped rules, skills, custom observability tools, stored decisions, or harness-specific verification workflow.
Condition B: project harness
The same runtime and model receive the checked-in project guide, scoped rules, linked context, project tools, permission policy, workflow assets, and verification loop being evaluated.
Do not remove capabilities the baseline coding agent normally includes. The comparison is the added project-owned layer, not an artificially crippled product. Record every difference between the conditions.
Run protocol
Seven steps from task selection to a reviewable result
- 01
Freeze the environment
Record the repository commit, dependency lockfile, runtime versions, model identifier, settings, permissions, budgets, and relevant service fixtures.
- 02
Select hidden-solution tasks
Sample real resolved work across bug fixing, UI behavior, and cross-file change. Keep acceptance tests and the original fix hidden from the agent.
- 03
Pre-register the oracle
Write machine checks and human-review criteria before running either condition. Name which evidence earns each rubric point.
- 04
Randomize run order
Alternate which condition runs first for each task to reduce learning, cache, and environment-order effects. Restore the exact starting state before every run.
- 05
Capture complete traces
Save prompts, context loaded, tool calls, approvals, changed files, test output, token usage, elapsed time, retries, and final response. Redact secrets consistently.
- 06
Score blind when practical
Have a reviewer who does not know the condition grade the diff and evidence against the pre-registered rubric. Let automated checks supply objective points first.
- 07
Report distributions and failures
Publish every repetition, median and range by task, protocol deviations, and links to evidence. Treat qualitative conclusions as hypotheses until repeated.
Starter task set
Cover different ways a coding agent can fail
Replace these with closed tasks from your own repository. Each case should fit the fixed budget, have a known-good solution, and expose a different harness layer.
Context, capability, restraint, verification
Persistence regression
Fix a record field that disappears after a process restart without changing the public data contract.
Fail-first restart test, upgraded-database check, no direct production write, and correct persisted value after restart.
Capability, workflow, verification, visual interface
Visual interaction bug
Fix a modal that clips its primary action at the smallest supported viewport while preserving keyboard navigation.
Automated interaction passes, focus order is correct, and before-and-after screenshots meet the viewport acceptance criteria.
Context, provenance, workflow, coordination
Cross-package migration
Rename a shared API across producer and consumers while keeping backwards compatibility for one release.
Type checks and contract tests pass, compatibility path exists, affected packages are enumerated, and the rationale is linked to the diff.
100-point rubric
Correct work earns most of the score
A fast run that does not solve the task cannot win. Efficiency and polish matter only after the implementation, verification, and safety evidence are sound.
40 / 100
Task correctness
The hidden acceptance checks pass; the root cause is addressed; no relevant regression or out-of-scope behavior is introduced.
20 / 100
Verification quality
The run reproduces the failure, chooses checks at the user-visible layer, and supplies repeatable evidence for the fix.
15 / 100
Policy adherence
The agent follows permission boundaries, preserves unrelated work, handles secrets safely, and does not bypass controls.
10 / 100
Context efficiency
The run retrieves relevant context without flooding the window, repeating failed searches, or relying on hidden human help.
10 / 100
Recovery behavior
When an attempt fails, the agent uses evidence to revise its hypothesis and avoids repeating the same unsupported action.
5 / 100
Provenance
The final diff, rationale, task, tests, and important decisions are connected and reviewable after the session ends.
Minimum report
Make every claim auditable
- Task ID and sampling rationale
- Condition and randomized run order
- Repository commit and environment manifest
- Model identifier and inference settings
- Prompt and all loaded project context
- Tool calls, approvals, retries, and errors
- Patch, test output, screenshots, and evaluator notes
- Rubric sub-scores and disqualifying failures
- Tokens, cost, elapsed time, and protocol deviations
Recommended analysis
Use at least several independent repetitions per task and retain the full distribution. Report paired score differences by task plus medians and ranges. If you publish confidence intervals, state the resampling or statistical method and sample size. Do not merge tasks into one headline number when their acceptance checks or difficulty are not comparable.
Publication gate
Do not publish a win rate, score, or performance claim until the task definitions, fixed inputs, traces, patches, test output, rubric decisions, and failed runs are available for review. If evidence cannot be shared, label the result internal and do not market it as a reproducible benchmark.
See the eight-layer agent harness architecture →FAQ
Agent harness benchmark questions
What is an agent harness benchmark? +
It is a controlled evaluation of the system around an AI agent: project context, rules, tools, workflow, verification, provenance, and coordination. This protocol compares paired runs while holding the underlying model and task environment fixed.
Can I use this to compare Claude Code with Codex? +
Not in the same experiment. A harness comparison should hold the model and coding-agent runtime fixed. Run a separate model or runtime experiment if you want to compare Claude Code with Codex, and do not attribute the combined difference to the project harness alone.
How many runs do I need? +
More than one per condition and task. The right sample depends on outcome variability and the size of the difference you need to detect. Publish every run, use paired comparisons, and avoid a universal claim from a small internal sample.
Why not score only tests passed? +
Passing acceptance tests is the largest component, but a production harness must also produce useful verification, stay inside policy, recover from failed attempts, and leave reviewable provenance. A hard safety violation can invalidate a run even if tests pass.
Does Nimbalyst publish benchmark results here? +
This page publishes the protocol and rubric, not a universal leaderboard or an unverified product result. Results should be added only when the underlying tasks, traces, patches, tests, and failures can be inspected.
Nimbalyst: the open-source visual workspace for building with Codex, Claude Code, and more
Download
Works with your existing Claude Code and Codex subscription or API key