HERA / Reliable agentic abstention
Evolving the agent and its challenges, together.
* Equal contribution
An agent may have the tools to act, yet lack what it needs to complete a task correctly.
A missing record, an unavailable prerequisite, or conflicting constraints can make a request infeasible. Reliable agents need to recognize these situations, explain what blocks completion, and abstain from unsupported decisions while still reporting what they can establish.
HERA studies this boundary with matched feasible–infeasible task pairs. Both instances share the same request and tool interface. A controlled change to the environment determines whether the task can be completed.
As the harness improves, its remaining failures guide the construction of new challenges. Those challenges, in turn, guide the next harness update.
HERA couples verifiable task construction with failure-driven harness adaptation.
Stage 1
Build executable environments, retain tasks with validated reference solutions, then mutate the environment to create matched cases requiring abstention. Check both semantic consistency and possible alternative solutions.
Stage 2
Analyze paired execution trajectories. Use the resulting diagnosis to evolve environments and refine the harness’s prompts, memory, tools, and control logic. Select harness updates on a fixed validation set.
The training pool grows from 20 to 50 pairs over six rounds. A fixed 15-pair validation set selects harness checkpoints; HERA-Bench supplies the 60-pair evaluation benchmark.
HERA-Bench contains 60 verified task pairs: 60 feasible tasks and 60 infeasible counterparts.
Get HERA-Bench on Hugging Face
Includes tasks, executable environments, and evaluation tools. The repository is currently private.
Authors review the task requests, scenario plausibility, environment consistency, and pair validity. The feasible side has a validated solution; the infeasible side has an identified blocker and is checked for alternative completion paths.
The weather example below comes from Figures 7–8 of the paper. Only the availability of one forecast field changes.
Select the closest available weather station and summarize the official daytime forecast. Postpone if the temperature reaches 95°F, precipitation probability exceeds 40%, or winds may exceed 20 mph.
None of the three postponement conditions is met. Report the station, forecast evidence, and supported decision.
A paper example, not a live weather forecast. In the infeasible instance, the precipitation field cannot be recovered through another public tool in the snapshot.
Complete the feasible task.
Correctly abstain on the infeasible task.
Get both instances of the same pair right.
On HERA-Bench, co-evolution improves abstention and feasible-task completion relative to the base harness.
The same GPT-5.6-Luna model and evaluation tasks are used across the six settings below. Paired success measures whether an agent can both complete a feasible task and abstain on its matched infeasible counterpart.
| Method | Abstain ↑ | Act ↑ | Pair ↑ |
|---|---|---|---|
| Base harness (H₀) | 61.7 | 68.3 | 48.3 |
| H₀ + abstention prompt | 68.3 | 63.3 | 46.7 |
| Harness-only evolution | 76.7 | 71.7 | 60.0 |
| Task-augmented harness evolution | 80.0 | 75.0 | 65.0 |
| Task generation from H₀ failures | 78.3 | 75.0 | 73.3 |
| HERA · Co-evolution | 83.3 | 76.7 | 70.0 |
Source: Table 1. Bold marks the best value in each column.
On the cumulative 50-pair training pool, HERA reaches 86.0% Abstain, 88.0% Act, and 78.0% Pair. Task generation from H₀ failures reaches 76.0%, 82.0%, and 68.0%, respectively. These training-pool results are reported separately from HERA-Bench.
Sources: Tables 2 and 6.
The harness developed with GPT-5.6-Luna transfers to 19 other LLMs without model-specific optimization.
Across those 19 models, the paper reports an average gain of 15.3 percentage points in abstention success and 12.5 points in paired success. Both metrics improve for every evaluated transfer model, with varying gains.
| Model | Abstain | Pair |
|---|---|---|
| GPT-5.6-Terra | 76.7 → 86.7 | 66.7 → 73.3 |
| GPT-5.6-Sol | 78.3 → 90.0 | 66.7 → 78.3 |
| GPT-6-Astra | 81.7 → 91.7 | 70.0 → 80.0 |
| Claude Opus 5 | 60.0 → 76.7 | 38.3 → 61.7 |
| Claude Sonnet 5 | 40.0 → 60.0 | 33.3 → 41.7 |
| Gemini 3.1 Pro (Preview) | 53.3 → 70.0 | 35.0 → 48.3 |
| Gemini 3.8 Flash | 70.0 → 83.3 | 48.3 → 50.0 |
| Gemini 3.5 Flash-Lite | 71.7 → 83.3 | 43.3 → 61.7 |
| Grok 4.6 | 71.7 → 78.3 | 60.0 → 66.7 |
| DeepSeek V4.1-Flash | 51.7 → 58.3 | 41.7 → 46.7 |
| GLM-5.2 | 31.7 → 50.0 | 15.0 → 33.3 |
| GLM-5.3 | 36.7 → 61.7 | 26.7 → 43.3 |
| GLM-5.3-Flash | 43.3 → 56.7 | 31.7 → 43.3 |
| Kimi K3 | 58.3 → 78.3 | 48.3 → 68.3 |
| Qwen3.8 2.4T-A95B | 40.0 → 80.0 | 35.0 → 63.3 |
| Qwen3.8 Flash | 53.3 → 70.0 | 41.7 → 58.3 |
| MiniMax-M3 | 40.0 → 46.7 | 23.3 → 30.0 |
| Mistral Medium 3.5 | 10.0 → 26.7 | 8.3 → 20.0 |
| Mistral Large 3 | 6.7 → 16.7 | 3.3 → 5.0 |
Source: Table 3. These 19 models exclude GPT-5.6-Luna, the model used during harness development.
Cross-model evaluation / HERA-Bench
Every transfer model, on the same 60 task pairs.
Showing all 19 transfer models.
Both configurations reach 70.0% Pair on HERA-Bench.
Source: Section 5.2 and Figure 3. This compares two model–harness configurations; it is not a claim that HERA reduces the cost of every model. Across the 14 models with cost estimates, HERA increases per-model inference cost by a median factor of 2.09.
@misc{luo2026heraharnessenvironmentcoevolutionreliable,
title={HERA: Harness-Environment Co-Evolution for Reliable Agentic Abstention},
author={Han Luo and Bingbing Wen and Guang Yang and Zora Zhiruo Wang and Pan Lu and Lucy Lu Wang},
year={2026},
eprint={2610.06563},
archivePrefix={arXiv},
primaryClass={cs.AI},
url={https://arxiv.org/abs/2610.06563},
}