Don't wait for the next model

On our AutomationBench agent, GPT-5.2 fully completes 30% of held-out tasks. After Beaker tuned the agent's harness around that model, it completes 48%. GPT-6 Luna came out nine months later and completes 39% in the same untuned harness. If you're an AI engineer building agents, the capability you are waiting for may already be in today's models. Tuning the harness so it works with the model, rather than against it, gets you that capability now instead of at the next release.

Your harness might be fighting the model

OpenAI took GPT-5.6 Sol's ARC-AGI-3 score from 13.3% to 38.3%, with about 6x fewer output tokens, by keeping its reasoning between turns and compacting old context. The official harness had thrown that reasoning away and truncated the context. LangChain took its coding agent from 52.8 to 66.5 on Terminal Bench 2.0 by changing only its prompt, tools and middleware. Databricks ran the same models at the same reasoning effort through different coding harnesses on its own codebase, and cost per task differed by more than 2x in some cases, mostly because one harness sent about 3x less context per turn.

The experiment

Our agent is the AutomationBench recipe from the Beaker cookbook. It handles Zapier-style business tasks across Gmail, Slack, Google Sheets, Salesforce and other apps. A task counts as completed only if every check on the final state passes. We picked four OpenAI models released between August 2025 and July 2026 and had Beaker tune the harness separately for each one on 120 training tasks. Our reference is one of the latest available models, GPT-6 Luna (medium reasoning), in the same untuned harness.

Results are reported on 104 held-out tasks that Beaker never saw during training, averaged over two runs per model.

Nine months of releases, or one tuning run

In the untuned harness, the share of tasks fully completed rose by about one point per month of release date, from 30% for GPT-5.2 (December 2025) to 39% for GPT-6 Luna (September 2026). GPT-5 Mini (17%) is left out of that trend; with it, the slope is about 1.5 points a month. Tuning took GPT-5.2, GPT-5.4 Mini and GPT-5.6 Luna to about 48%, and GPT-5 Mini to 31%.

Beaker found these improvements autonomously. For each model, it ran the agent on the training tasks, analyzed the mistakes and failures in the traces, and wrote hypotheses about how to fix them. It tested each hypothesis independently, kept the ones the results validated and dropped the rest.

The case viewer compares a hypothesis with the one it was built on, sample by sample, showing which samples improved, which regressed, and the agent's output before and after.

One fix helped every model, and it was most of the gain

In AutomationBench the rules for a task are often not in the request. They sit in an email from finance, a Slack thread or a policy sheet, and every model acted on the request alone. This reflects many enterprise agent use cases, where standard operating procedures are rarely written down properly. Beaker implicitly infers these procedures from data and experiments.

The first change Beaker kept, found separately in each model's experiment, was to look up those rules before writing anything. On the training tasks it accounted for +14 of GPT-5.2's +27 points, +22 of GPT-5.4 Mini's +23 (over two changes), +19 of GPT-5.6 Luna's +22 and +11 of GPT-5 Mini's +13. That is about three quarters of the training-set gain (66 of 85 points). Two other fixes were tool bugs that every model ran into: the harness sent every tool definition twice, which pushed GPT-5.4 Mini past its context window, and an Airtable search returned sample IDs for an empty table.

The other changes added +13 for GPT-5.2 and +1 to +3 for the others, and the model-specific ones depended on how each model behaved:

• GPT-5.2 gave up early. When a title search found no Pipefy card in a vendor-approval task, it asked procurement for the card ID and updated nothing, scoring 16% where GPT-5.4 Mini and GPT-5.6 Luna scored 100%. Its tuned harness recovers missing cards and ranks queues by priority before deadline, and its score on that task rose to 72%.
• GPT-5.6 Luna guessed spreadsheet tab names. 45% of its runs hit a "worksheet not found" error, the most of any model. Its tuned harness has it list the real tabs first, and fill in a private table of each record's conditions and exceptions before acting.

Newer models make intelligence cheaper

Tuning shows how much more the older models could do, but that doesn't automatically make them the most appropriate pick for production. Tuned, the three newer models score the same, and each newer one costs less per task. Tuned GPT-5.6 Luna matches tuned GPT-5.2 at a thirtieth of the cost per task. Each model release unlocks more intelligence per dollar spent.

So don't wait for a new model to improve your agent. Tune the harness now, and when a new model ships, re-test and tune for it so you get the most out of each release. For GPT-5.6 Luna, Beaker took 2.5 hours and the experiment cost $46.

Try Beaker for free → to tune the harness around your own agent's model, or talk to us and we'll set up your first experiment with you.