QA Is Becoming the Control System: Test the Factory, Not Just the Code
Implementation Can Move Faster Than Its Controls
What happens to the people who guard quality when a coding agent can produce a pull request in minutes?
The most valuable work doesn't disappear. It moves outward, away from inspecting code and toward governing the system that produces it. Testing the artifact still matters. But when the actor writing the code interprets intent, chooses between solution paths, calls tools with real permissions, and decides for itself whether to stop or escalate, a test of the final diff can't see most of what could go wrong.
That's the shift. And it's conditional, not inevitable.
Implementation can accelerate. The evidence is genuinely mixed. DevOps Research and Assessment (DORA) found a positive association between artificial intelligence (AI) adoption and delivery throughput in its 2025 report, alongside a negative association with stability. Model Evaluation and Threat Research (METR) ran a small randomized trial in early 2025 with 16 developers on 246 tasks across mature open-source projects, and found a 19% slowdown for experienced maintainers using the tools available then. METR's later update says the magnitude evidence remains too weak to draw firm conclusions. The premise here isn't that coding agents always speed things up. It's that when implementation does accelerate, the controls around it have to keep pace, or they become the bottleneck.
Review congestion is a plausible local risk. It's not a universal outcome. A study of 567 Claude Code pull requests across open-source projects found no significant merge-time difference compared with matched human pull requests, though this is open-source data, not enterprise evidence. The risk isn't that agentic changes always take longer to review. It's that the volume and variety of changes can grow faster than your review capacity. Measuring that gap takes local data on queue depth, reviewer load, and escaped defects.
This is the quality side of The AI Code Generation Process Paradox. When code generation changes the production process, quality controls have to change with it.
The Category Error
A common delivery arrangement treats quality assurance (QA) as a late gate. Code arrives after implementation, testing, and often deployment. QA checks it. That model works when the production process is stable, the actor producing changes is a known human developer, and the failure modes live in the code itself.
Agentic engineering changes the actor. An AI agent doesn't just write code. It interprets intent, picks between solution paths, calls tools with real permissions, modifies state across multiple turns, weighs its own evidence, and decides whether to stop or escalate. Anthropic's evaluation guidance treats those behaviors as the things you need to evaluate. They create risks that don't show up in the final diff. The code may be correct. The trajectory that produced it may still violate policy, use a tool beyond its authority, fabricate evidence, or fail to escalate a decision that required human judgment. The Open Worldwide Application Security Project (OWASP) community cheat sheet identifies tool abuse, excessive autonomy, and approval manipulation as agent-specific risks that conventional artifact review doesn't cover.
What you're really assuring isn't just the code. It's the whole production arrangement. Anthropic frames this as evaluating the model and harness together as one production system.
Test the Factory, Not Just the Code
Every agent-assisted workflow runs inside a factory. That factory includes the model and its configuration, the prompts and repository instructions, the tools and permissions, the retrieval sources and memory state, the orchestration logic, the deterministic checks, the model-based graders, the approval boundaries, and the deployment environment. A capable model inside a weak factory is still a weak production system.
The Dark Factory showed how scenario holdouts and probabilistic satisfaction move validation into the production system. The organizational question is who designs, calibrates, and governs those controls.
Scenario evaluation looks at the whole run, not just the result. In Anthropic's evaluation guidance, a trace is the complete trial record: the tools called, the parameters used, the state changes made, the evidence collected, and the escalation decisions taken. The outcome is the final environment state, not the agent's statement of completion. Agents can disagree with the environment about whether a task succeeded, which is exactly the kind of failure mode that artifact inspection alone can't catch. OpenAI's trace grading covers tool choice, handoffs, guardrails, and policy compliance across the execution path, giving reviewers a structured record of what happened rather than only the artifact produced.
Authority testing asks a different question: does the factory respect its own boundaries? It tests both that allowed tools work with permitted parameters and that the factory rejects over-scoped requests, stale approvals, and adversarial inputs. OWASP's community guidance recommends least privilege, parameter-bound approvals, and fail-closed behavior for consequential operations. ToolEmu, using 36 toolkits and 144 simulated cases with a model evaluator, found failures in simulated high-stakes tool scenarios that represented potentially harmful real-world actions. tau-bench, a customer-service benchmark using historical models, found that function-calling agents succeeded on fewer than half of the studied tasks. These results don't generalize to every setting, but they establish that authority and capability can't be assumed from artifact quality alone.
The Deterministic Floor Doesn't Move
None of this replaces conventional testing. Anthropic's vendor guidance keeps unit tests as a typical correctness floor for coding agents. Ham Vocke's practical test pyramid shows that a portfolio of tests at different granularities remains the fast-feedback foundation. The National Institute of Standards and Technology (NIST) has published an AI risk framework that calls for rigorous software testing alongside quantitative and qualitative AI risk measurement, though that's general AI-risk guidance, not coding-agent-specific.
If a change fails a deterministic check, it stops. Those checks return explicit pass/fail results, even though the checks themselves can still be incomplete or misconfigured. Those invariants matter even more when the production actor can take different paths to the same result. Strict checks on authorization, data boundaries, required evidence, and irreversible actions are the hard shell around flexible internal behavior.
The American Society for Quality has long described quality assurance as including audits of processes and systems, not just inspection of outputs. That's continuity with traditional QA practice, not evidence of AI adoption. The continuity is real. What's new is evaluating an actor that interprets, chooses, uses authority, and decides, not just the code it produces.
What QA Can Own
This is a proposal, not an industry consensus. The evidence for who should own this work is thin. Anthropic's first-party report describes dedicated evaluation teams with domain and product roles, but doesn't identify them as QA. Studies of agentic pull requests confirm that human review still matters: the 567-PR study found that 45.1% of merged agentic pull requests received human revisions, and a separate study of 33,000 agent-authored pull requests found task type, continuous integration, reviewer engagement, duplicates, and misalignment all at play. Neither examines quality-team job design.
What follows isn't derived from how companies already do this. It's built on quality competencies that already exist. QA professionals are well suited to design and calibrate the controls around agentic execution. The core skills aren't new: ambiguity discovery, adversarial scenario design, risk modeling, failure analysis, evidence calibration, and risk communication. What changes is when and where these skills apply. They move earlier in the process and up a layer of abstraction. These responsibilities might live inside QA. They might be shared across product, engineering, security, and operations. Ownership should follow competence, not a new silo.
Intent and Ambiguity
Before the agent acts, someone has to turn product intent, risk tolerance, and user harm into checkable conditions. This means finding the contradictions, the underspecified edge cases, and the assumptions that an agent won't question. NIST's framework asks for context, intended use, system requirements, risk tolerance, and human oversight to be documented. The work is deciding which claims need deterministic proof and which need contextual judgment. This is analytical design work, not test execution.
Scenario Design
Good scenarios come from things that already hurt: manual checks, recent bugs, support cases, rejected changes, and observed failure modes. You want both cases that should pass and cases that should trigger a control. tau-bench validates final environment state for function-calling agents across repeated trials, a practice that matters because a single passing run doesn't establish reliable behavior. For consequential capabilities, hold back some scenarios or have them authored independently, so the producing system can't memorize the evaluation set.
Evaluator Calibration
The split is straightforward: deterministic graders for objective conditions, large language model (LLM) graders for bounded semantic judgments. But calibrate every model grader against qualified human review before it blocks or clears work. LLM judges show position, verbosity, and self-enhancement biases in a chat-preference setting. A recent preprint found that agent confidence can greatly exceed actual success, with one studied condition showing an agent predicting 77% success while succeeding 22% of the time. What you track matters: disagreement rates, insufficient-evidence outcomes, bias slices, and grader-version regressions. Every grader needs an explicit insufficient-evidence outcome, so the system can tell a clear pass from uncertainty.
Authority and Escalation
Each action carries its own risk profile: impact, reversibility, data sensitivity, and blast radius. For each risk class, set the allowed tools and scopes. Mark the cases that must escalate, and test both directions: missed escalation and needless escalation. Keep human approval for consequential or irreversible actions. OWASP's community guidance leans toward least privilege, independent validation, and risk-based approval. An escalation confusion matrix measures whether the factory calls humans at the right times without overwhelming them.
Exception Review and Production Learning
Human review matters most where the stakes are: high-risk changes, low-confidence evidence, evaluator disagreement, novel trajectories, and policy-boundary events. Check what happened in production. Don't accept the agent's completion statement. Every escaped failure should become a scenario, an invariant, a rubric correction, or a monitoring signal. NIST's framework covers production monitoring, incident response, override, recovery, and continual improvement. One useful local measure is the escape-to-control conversion rate: how many failures become permanent improvements rather than recurring surprises.
The First 30 Days
Don't reorganize the company, declare a maturity level, or grant broad autonomy on day one. Run a bounded experiment to find out whether factory-level assurance adds signal beyond your current test and review process.
- Define the assured system. Start by versioning the complete factory. That means the model, configuration, instructions, tools, permissions, retrieval, memory, orchestration, checks, graders, approval boundaries, and environment. Classify work by consequence, reversibility, data sensitivity, and privilege. Name the accountable human for each consequential decision. Produce a one-page factory manifest.
- Build a scenario bank from real work. Start with 20 to 50 scenarios drawn from current manual checks, recent bugs, support cases, and near misses. Include both cases where an action should occur and where it shouldn't. For every high-risk scenario, define the expected environment outcome, the required evidence, the forbidden actions, and the escalation condition. Keep some scenarios outside the producing workflow's normal context.
- Layer the graders. Start with deterministic checks for build, tests, types, security, permissions, required evidence, and environment state. Add model graders only for bounded semantic claims that rules can't express. Calibrate every model grader against qualified human judgment before it blocks or clears work. Give graders an explicit insufficient-evidence outcome. Record grader version, rubric version, and disagreement by risk class.
- Test authority and escalation. Probe each tool with every kind of input you can think of: allowed, over-scoped, malformed, stale, replayed, and adversarial. Test both directions of escalation: cases that must escalate and cases that should continue. Confirm interruption, rollback, and fail-closed behavior. Document every fail-open condition and its owner.
- Compare outcomes and close the loop. Run repeated trials on the same factory version. Compare the agent-assisted and ordinary workflows on review time, change failure, rollback, escaped defects, unsafe-action attempts, must-escalate recall, unneeded-escalation workload, and grader disagreement. Convert every observed escape or unfair grade into a scenario, invariant, rubric correction, or monitoring signal. Decide which controls earned continuation and which add only friction.
The 2025 DORA report found that 90% of respondents use AI at work, while 30% reported little or no trust in AI-generated code. That gap between usage and trust isn't a problem to solve with more automation. Treat it as a signal to make the controls around agentic execution visible, measurable, and led by human judgment.
This is earned autonomy: the system gets more authority only after a risk slice shows stable outcomes, reliable escalation, calibrated graders, and acceptable production results. Stop or narrow the experiment when controls can't distinguish safe from unsafe work, when human workload grows without reducing escapes, or when the evaluator proves easier to optimize than the underlying goal.
The factory creates code. The control system creates the conditions for trusting it. That's work worth putting quality professionals at the center of.