Article

Mutation Testing of Task-Scoped State Oracles in Software-Agent Benchmarks: A Cross-Benchmark Empirical Study

2026-08-18

Abstract excerpt

<title>Abstract</title> <p>Stateful software-agent benchmarks increasingly use executable evaluators to decide whether an agent produced the requested persistent effects. Yet a high task score is informative only if the evaluator rejects harmful extra effects without rejecting equivalent representations. We introduce a deterministic mutation-testing protocol for calibrating such task-scoped state oracles. Startin...

Topics

Open a Topic to create a Post that cites this publication.

Genomics and Rare Diseases

Identifiers and source

Literature Corpus work: a2eff2ad-8323-52d2-b63c-f902d9ecc15c
DOI: 10.21203/rs.3.rs-10665114/v1

Open publication

Related research

Semantic proximity does not establish scientific evidence.

Click a neighbor to travelStep 1 · 12 closest

Mutation Testing of Task-Scoped State Oracles in Software-Agent Benchmarks: A Cross-Benchmark Empirical StudyDOI 10.21203/rs.3.rs-10665114/v1Read

Execution Grounded Multiagent Systems for Reliable Backend Code Generation with Large Language Models'Semantic proximity Read

Autonomous AI and Agentic Testing Agents: A Multi-Agent Architecture for Self-Directed Software Quality AssuranceSemantic proximity Read

Specification-first convergence with an AI coding agent: a case study of dismantling a core architectural invariant across 189 files in a 717k-line codebase with no test oracle and no human code reviewSemantic proximity Read

Who Broke the Pipeline? Traceability andAccountability in Multi-Agent LLM SoftwareEngineering PipelinesSemantic proximity Read

The World Is a Hypothesis: Epistemic Control for Executable World-Model Agents in Novel Interactive EnvironmentsSemantic proximity Read

AnnCoder: A mti-Agent-Based Code Generation and Optimization ModelSemantic proximity Read

SAMF: SAWANT (Structured Agentic Workflow for Alignment, Validation, and Negotiated Testing) for Reliable, Safe, and Verifiable LLM PromptingSemantic proximity Read

Agentic systems are adept at solving well-scoped, verifiable problems in computational biologySemantic proximity Read

Trustworthy agentic genomics through versioned skill librariesSemantic proximity Read

Protocol-Dependent Resolution in Finite-Sample Model SelectionSemantic proximity Read

Multi-Agent Human-AI Systems with Low-Code Platforms Enabling Adaptive Web Services and Real-Time Anomaly Remediation in Distributed ArchitecturesSemantic proximity Read

From Model Assurance to Causal Answerability: A Multi-Timescale Information Architecture for Governed Agentic AISemantic proximity Read