Harness scaling for generalizable agentic reinforcement learning

The problem

Agent harnesses shape how models use tools, manage context, and coordinate long-horizon interaction (arxiv.org). These harnesses are a central component of modern agentic systems (arxiv.org). However, existing approaches typically train language models through a single fixed harness (arxiv.org). According to the authors, this can lead to specialization to a particular execution configuration and weaken transfer across other harnesses (arxiv.org). While task and environment scaling expand what agents solve or where they operate, they often retain a fixed execution harness (arxiv.org).

The idea

The authors introduce Harness Scaling to extend policy optimization from a single harness to diverse execution configurations (arxiv.org). To model harness variation, the authors factorize the harness into five components: system prompt, tool interface, observation, context management, and workflow (arxiv.org). Because exhaustive training of all possible combinations introduces redundancy, the authors construct a compact training subset using a strength-2 orthogonal design (arxiv.org). This design preserves individual component representations and balanced pairwise combinations (arxiv.org). During reinforcement learning, the same task is paired with structurally nearby harness configurations to provide controlled local variation (arxiv.org).

The results

The authors report that across multiple benchmarks, model scales, and reinforcement learning algorithms, Harness Scaling consistently improves generalization to unseen white-box configurations (arxiv.org). Furthermore, models trained through structured white-box variation transfer effectively to unseen external black-box harnesses (arxiv.org). The authors state that these models outperform those trained through a single complex black-box harness (arxiv.org).

Why it matters

These results establish harness diversity as a meaningful scaling dimension for agentic reinforcement learning (arxiv.org). The findings highlight that effective scaling depends critically on how this diversity is organized during training (arxiv.org). By avoiding specialization to a single execution setup, models can develop more robust and transferable agent behavior (arxiv.org).

Caveats

This paper is a preprint (arxiv.org). The authors note that their evaluation focuses on constructed white-box configurations and specific external black-box harnesses, and they detail constraints on design sizes determined by pairwise coverage and the Rao bound (arxiv.org). The sources do not provide exact numerical performance metrics or benchmark names beyond broad qualitative and comparative descriptions.