GPT-5.6 Sol Max vs. xhigh: Why Max Isn't My Default

I assumed the Codex effort selector was a simple intelligence dial: turn it up, wait longer, get a better answer.

The data does not behave that way.

Across 112 real repository tasks, GPT-5.6 Sol passed 77 tasks at xhigh and 75 at max. The higher setting took 35% longer and cost 46% more on average. ultra eventually pulled ahead, but at more than twice the runtime of xhigh.

Higher effort helps on this benchmark overall, but the gains are uneven and expensive. That is why max is not my default.

That result also explains why I do not treat the model’s internal allowance as a performance score. My earlier GPT-5.6 Juice value observations showed a much larger reported value for max than xhigh, yet the repository benchmark still moved backward.

Where the data comes from

I used the public data behind Distributed Radar, a community-run coding-agent benchmark. Volunteers run DeepSWE tasks on their own machines, then the service grades each patch again in a clean container. Client-reported success does not count by itself. The server-side verifier decides whether the task passed. The DRadar client is open source.

I froze the data at August 2, 2026, 02:38 UTC. At that point, the public dataset contained:

• 112 repository-level tasks
• 21 model-and-effort combinations
• 2,352 task cells
• 19,210 graded runs

The source is live, so the numbers will move. The exact inputs for this post were the public task table and leaderboard.

For ability, I used the leaderboard’s per-task majority result. Each task therefore contributes one pass or fail. Heavily repeated tasks cannot dominate the total.

For time and cost, I used the latest valid run in each task cell and averaged across all 112 tasks. The dollar value is a token-price equivalent reported by the dataset, not necessarily the amount a subscriber saw on a credit-card bill. Hardware and local conditions can also affect runtime.

The Sol effort curve

Here is the complete GPT-5.6 Sol curve:

The endpoints follow the expected pattern: low is weakest and ultra is strongest.

The middle of the curve is less tidy.

From high to xhigh, the model gained eight net tasks while adding only 3.2 minutes and $1.28 to the average run. That is a useful upgrade.

From xhigh to max, the model lost two net tasks while adding nine minutes and $3.05. That is an expensive regression.

From max to ultra, it gained four net tasks, but average runtime rose from 34.4 to 52.4 minutes and cost rose from $9.62 to $14.98.

If effort were a clean intelligence dial, I would expect diminishing gains. I would not expect the middle of the curve to move backward.

Max did not simply fail on the same tasks

The drop from 77 to 75 hides more movement than it appears to.

When I compared the per-task majority results for xhigh and max:

• max solved four tasks that xhigh failed
• max failed six tasks that xhigh solved
• the remaining 102 tasks kept the same outcome

max was an expensive version of xhigh that traded one set of successes for another and ended two tasks behind.

That matters in practice. A one-point aggregate difference can look like noise, but your repository is not an aggregate. If it happens to resemble one of those six regressions, paying for the higher setting gives you a worse outcome.

The non-monotonic behavior was not limited to one adjacent pair. Across the six Sol effort levels, 28 of the 112 tasks had at least one point where a lower effort achieved a majority pass and a higher effort later failed.

This does not prove that extra reasoning caused the failures. Volunteers ran the settings at different times, with model randomness and environmental variation. A controlled experiment would rerun every task multiple times under matched conditions.

But it does disprove the operational shortcut I started with: a higher effort label does not guarantee a more reliable result on every task.

The last few passes cost a lot

Suppose I ignore the dip at max and compare high directly with ultra.

ultra passed ten more tasks out of 112. To get those ten additional majority passes:

• average runtime increased from 22.2 to 52.4 minutes
• average cost increased from $5.29 to $14.98
• the task-level pass rate rose from 61.6% to 70.5%

That can be a good trade when failure is expensive. If the task is a difficult migration, a subtle concurrency bug, or a change that would otherwise consume an engineer’s afternoon, another $9.69 of equivalent compute may be trivial.

It is a poor default for every task in a queue.

Background refactors, repetitive edits, and work with strong automated tests have cheap failure recovery. For those jobs, throughput matters. Running every task on ultra means spending more than twice as long for a gain that appears on only part of the workload.

The right effort setting depends on the cost of being wrong, not only the probability of being right.

What I would use instead

Based on this snapshot, my practical defaults would be:

Use medium for cheap, recoverable work. It handled 66 tasks, close to high’s 69, while using less time and equivalent cost. Good tests can catch the misses.

Use xhigh for serious repository work. It gives a clear ability gain over high with modest extra runtime, and costs much less than max or ultra.

Do not use max as an automatic upgrade. In this dataset, max is slower, more expensive, and slightly worse than xhigh. A future snapshot may change that, but the label alone is not a reason to select it.

Reserve ultra for expensive failures or a second attempt. It had the highest total, so there are tasks where the extra compute pays off. I would reach for it when xhigh fails, or when validating the result is unusually difficult.

This produces a simple escalation policy:

recoverable task  →  medium
serious task      →  xhigh
xhigh failed      →  ultra

max is the awkward step I would currently skip.

What this dataset cannot tell us

This snapshot has several limits, so it should not become a universal rule.

First, this is community benchmark data, not a controlled lab experiment. Machines, timing, and model randomness vary. Server-side verification makes the pass/fail result trustworthy, but it does not remove every source of variance.

Second, the dataset labels 105 of the 112 tasks as feature requests. The curve describes repository-level feature implementation much better than it describes small bug fixes, greenfield applications, or ordinary chat.

Third, majority voting compresses uncertainty. A task that passes two out of three runs and a task that passes every run both count as one majority pass here.

Fourth, the database is live. This article is a dated photograph of a moving system, not a permanent model card.

Those limits also make the result useful. Public model summaries usually give one score per model. This dataset shows which tasks change, what the extra reasoning costs, and where a setting that sounds stronger stops being a straightforward upgrade.

Effort is a budget, not a quality preset

The effort selector does not behave like Low, Medium, and High image quality. It changes how the agent explores a problem, and more exploration creates more opportunities both to find the right path and to wander away from it.

On this snapshot, xhigh is the practical sweet spot for Sol. ultra buys the best overall result at a steep premium. max sits between them without earning its place.

I still want higher effort available, but not as the default.