OpenAgent formalizes the open-world generalization gap in tool-use LLM agents across four non-stationary dimensions: queries, tool schemas, feedback dynamics, and task domains. Diagnostic experiments reveal that SFT agents suffer from brittle symbolic anchoring and open-loop inertia, while RL agents display boundary blindness due to outcome-reward biases. To solve this, the authors propose Perturbation-Augmented Fine-Tuning (PAFT), which injects trajectory-level disturbances during training to restore closed-loop recovery and active refusal.
Diagnosing Distributional Shifts in Agentic Tool Use
While LLM agents achieve near-perfect success on closed, static benchmarks, real-world deployment presents non-stationary environments involving deprecated APIs and ambiguous instructions. This paper formalizes the OpenAgent problem setting, systematically probing distributional shifts across four key dimensions: User Queries (Δ&mathcal{Q}), Tool Sets (Δ&mathcal{A}), Interaction Dynamics (Δ&mathcal{O}), and Task Domains (Δ&mathcal{D}).
The authors built a controlled sandbox environment to evaluate agents across a 4-tier hierarchy, uncovering critical structural failure points:
Tier 1 (Perception): SFT suffers from Symbolic Anchoring acting as a superficial pattern matcher that collapses when tool names drift or descriptions contain semantic adversarial traps.
Tier 2 (Interaction): SFT exhibits open-loop trajectory inertia, frequently ignoring corrective feedback or hallucinating success. RL adapts dynamically under explicit redirection but degrades heavily under ambiguous null returns.
Tier 3 (Reasoning): While RL excels at local path shortcuts (Trace Simplification), both SFT and RL collapse under Logic Inversion when tool dependencies are topologically reversed (e.g. shifting A → B to B → A).
Tier 4 (Internalization & Boundary Awareness): SFT exhibits complete open-loop blindness to fatal errors. RL detects the errors but suffers from Boundary Blindness a teleological bias induced by outcome-based rewards that forces the agent to fabricate answers rather than issue an honest refusal.
To bridge the gap between clean training manifolds and open-world anomalies, the authors introduce PAFT, injecting trajectory-level perturbations during SFT (optimal ratio α = 0.3):
Environmental Feedback Perturbation (EFP): Injects mid-trajectory observation anomalies (like value deprecation) to convert open-loop patterns into closed-loop recovery behaviors.
Solvability Boundary Perturbation (SBP): Injects fatal errors with no fallback tools, teaching explicit refusal actions (aend) and successfully pushing Tier-4 refusal rates from 0.0% to 99.6%.
Symbolic Representation Perturbation (SRP): Applies synonymous rewriting and noise to tool definitions, forcing the model to ground decisions in invariant functional semantics rather than memorized surface tokens.
One-Line Takeaway: Current agent post-training creates brittle closed-loop shortcuts; PAFT restores open-world agent robustness by training over trajectory-level feedback anomalies and fatal execution boundaries.