AI is starting to change quality assurance in a meaningful way, but the most useful question is not whether AI can replace QA. A better question is where AI can expand the depth and breadth of testing while humans remain responsible for judgment, context, and final approval.
That distinction matters, especially for data products. Many QA activities in data products are relatively objective: does the transformation follow the defined logic, do filters behave correctly, do calculations reconcile, does the result remain stable over time, and are anomalies appearing where they should not? These are areas where AI agents can do a surprising amount of useful work.
The opportunity is also much bigger than automating a few test cases. With the right setup, AI can participate across the full lifecycle: reviewing requirements, identifying gaps and edge cases, generating and refining test cases, locating test data, validating implementation against requirements, troubleshooting failures, and helping teams prioritize what actually matters. The human still stays in the loop throughout, but the nature of that human involvement changes.
Start QA before QA starts
One of the biggest benefits of AI-assisted QA is that the process can begin before development is finished. A useful starting point is to give the agent access to the requirement or PRD and ask it to perform a structured review for gaps, conflicting logic, undefined assumptions, likely edge cases, and places where the proposed solution may behave differently under unusual conditions.
This can be especially useful for data products, where edge cases may come from missing values, historical data issues, unusual distributions, timing differences, or combinations of conditions that are difficult to anticipate manually. The important part is that every issue the agent identifies does not automatically become a requirement. A PM may deliberately decide that an edge case is too rare, too low impact, or too expensive to address in the first version, and that decision should still be explicit and documented.
That feedback also helps the agent improve. Sometimes the team has to “teach” the model why a technically valid concern is not material in the business context. This is a recurring pattern in human-in-the-loop QA: the agent surfaces possibilities; the team decides what matters.
Turn the finalized requirement into a living test plan
Once the requirement is stable enough, AI can generate test cases directly from the PRD, acceptance criteria, and expected behavior. This should not be a one-time exercise. If the requirement changes, the test cases should be regenerated, and the team should review what changed. Showing the delta between the previous and updated test plan can make that review much faster.
Historical manual test cases can also be useful as examples. Feeding the agent representative prior tests helps it understand the level of detail, coverage, and style expected by the team. Depending on the nature of the product change, the resulting test plan may span everything from spelling and naming consistency to business logic validation, ETL checks, filtering interactions, historical trend validation, anomaly detection, boundary conditions, and regression coverage.
The goal is not to eliminate human review of the test plan. It is to reduce the manual effort of producing it while dramatically increasing the number of scenarios the team can consider.
Use AI to find the right data to test
Another practical use is test-data identification. Instead of asking a human tester to manually search through a large dataset for records that satisfy a specific condition, the agent can help locate examples that fit the scenario being tested. That is particularly helpful when validating edge cases, unusual distributions, or specific business-rule combinations.
Synthetic test-data generation is another possible use case, especially when real examples do not exist or should not be used because of privacy concerns. But even without synthetic data, simply helping the team find the right records can remove a significant amount of friction from the testing process.
Validate implementation against intent
Once development is complete, the agent can evaluate the implementation against the requirement, acceptance criteria, and test cases. A mature workflow can bring together the PRD, severity levels, source code, raw data, expected result datasets, and test cases so the agent can assess whether the implementation behaves as expected.
The output can be structured as Pass, Conditional Pass, or Fail, with the agent explaining why a condition passed with caveats, identifying the specific logic or code associated with the concern, and suggesting possible remediation. Confidence can also be included, but this requires calibration. A confidence score may sound objective while actually depending heavily on the type of feature, the quality of the requirement, and the amount of context available to the agent.
That is why confidence and severity should be treated as things the team refines over time rather than universal numbers that can be defined once and reused everywhere.
AI makes previously impractical QA possible
One of the most meaningful changes is not simply faster testing. It is that AI can make a level of testing economically possible that would have been unrealistic with manual QA.
Consider a data product where the team wants to compare month-over-month behavior across more than five years, segmented by entity group, and flag anomalies across more than 10,000 individual comparisons. A human could theoretically do that work, but in practice the effort would be enormous and the consistency difficult to maintain.
An AI agent can run those checks quickly, identify unusual patterns, group issues into higher-level root-cause categories, and assign an initial severity level. The human team can then focus on the most important findings, troubleshoot representative cases, challenge severity when needed, and refine the logic used to classify future issues.
The workflow becomes:
Large-scale validation → anomaly detection → root-cause grouping → severity classification → human investigation → feedback
That is where the model becomes especially powerful. AI does the broad scan; humans focus their time where judgment is required.
Human-in-the-loop is more than approval
It is easy to describe human-in-the-loop as “AI does the work and a person signs off,” but that is too simplistic. Humans play a much more active role. They provide business history that may not be fully documented, explain why a backward-compatibility rule exists, clarify why a specific workflow has to be accommodated, and help the agent understand when a historical data issue is expected rather than defective.
They also challenge the agent’s recommendations. For example, an agent may identify three possible exclusion rules to address a data issue. On paper, all three may sound reasonable. But when the team asks the agent to quantify the impact of each rule individually and in combination, it may become clear that one of them changes almost nothing.
At that point, the best decision may be to skip the extra rule rather than add unnecessary complexity. That is a good example of why AI-generated recommendations should not be treated as gospel. The model can surface options, but the team still has to weigh impact, complexity, and materiality.
Severity and confidence need to be taught
Severity is another area where human judgment matters. Unless the organization has a very explicit definition, labels like P0, P1, or P2 can be surprisingly subjective, and the same is true for confidence scores.
A technically unusual condition may look severe to the agent while having almost no real business impact. Another issue may affect only a small number of records but still matter significantly because of customer impact, revenue, compliance, or trust. Teams therefore need to give the agent feedback when they disagree with a severity or confidence assessment and explain why.
Over time, that feedback can make the agent better aligned with how the organization evaluates risk.
Human-in-the-loop is not just review. It is how the system learns how the business thinks.
Build trust progressively
Teams should not move from fully manual QA to fully AI-assisted QA overnight. A safer approach is to start with parallel testing: run human QA and AI QA side by side for a defined set of tickets, compare the findings, and examine false positives, false negatives, severity alignment, and whether the agent is consistently interpreting the requirement correctly.
At the beginning, this may actually take more time than manual QA alone. That is normal. The team is learning where the agent is strong, where it is weak, how much context it needs, how to structure the PRD, how to define acceptance criteria, and how to calibrate severity and confidence.
Only after enough evidence has accumulated should the team reduce manual execution. Trust should be earned through validation, not assumed.
Keep human controls around change
Even in a highly AI-driven QA model, some controls are worth keeping. The agent can identify problematic code, explain why a test failed, suggest changes, and even propose replacement code, but it should not automatically change production logic without human approval.
Engineering should still approve code changes, product should still approve requirement and business-logic changes, and human peer review and unit testing should remain in place. Using separate models or agents for different stages can also reduce the risk of one system validating its own assumptions. For example, engineering and PM/UAT can use separate agents or models to provide an additional layer of independence.
The objective is not to remove accountability. It is to remove unnecessary manual execution while preserving ownership of change.
AI QA is not equally useful everywhere
This model is particularly well suited to data products because much of the expected behavior can be expressed in relatively objective terms: calculations, transformations, filters, logic, distributions, thresholds, and expected trends.
It is less suited to areas where quality is inherently subjective. For a workflow or consumer experience, questions such as “Is this intuitive?”, “Does the layout feel confusing?”, or “Would a new user know what to do next?” still require meaningful human judgment and real usability feedback.
AI can help identify possible usability issues or apply heuristics, but it should not be treated as a replacement for user research or UX validation.
Use AI most aggressively where quality can be evaluated against explicit logic, data, and expected outcomes. Keep humans closer to the loop where quality depends on perception, judgment, or user experience.
If you want to pilot this, start narrow
For a product leader trying to pilot AI-assisted QA, the best approach is not to transform the entire QA organization at once. Start with one narrow class of tickets where expected behavior is relatively objective. Run AI and human QA in parallel, compare the results, and refine the workflow based on what the agent gets right, what it misses, and where it overstates risk.
Different types of development work may require different acceptance criteria for AI output, so avoid forcing one universal QA workflow too early. Give the team room to experiment and expect some false starts. The first few weeks may take more effort, not less, because the team may need to refine prompts, improve documentation, adjust test cases, calibrate severity, and learn which types of work are best suited for the model.
That learning period is part of the investment.
Strong AI QA Starts With Strong Inputs
AI-assisted QA works best when the inputs are clear and well defined. Requirements need to be specific, acceptance criteria need to reflect the intended behavior, test cases need to cover the right scenarios, and historical context or special treatment rules need to be documented where they matter.
AI can help improve and organize all of those artifacts, but it cannot reliably compensate for ambiguity that the team itself has not resolved. In practice, the quality of AI-assisted testing is closely tied to the quality of the product definition it receives.
One useful side effect of this model is that it creates pressure for better requirements earlier in the development lifecycle. When an agent is expected to interpret the requirement, generate test cases, validate implementation, and explain failures, vague logic becomes much harder to ignore. Gaps that might previously have surfaced during manual QA—or even after release—can be challenged much earlier.
That makes stronger inputs more than a prerequisite for AI QA. Improving those inputs becomes part of the quality process itself.
I think this fits the rest of the article much better. I’d also keep the closing section immediately after it:
The goal is not zero humans. It is better use of human time.
The most compelling outcome of AI-assisted QA is not simply fewer manual test steps. It is broader coverage, deeper analysis, faster troubleshooting, stronger requirements, more systematic edge-case discovery, and a better use of human attention.
AI can scan thousands of comparisons, identify patterns, generate scenarios, and suggest root causes far faster than a person can. Humans still decide which findings matter, which tradeoffs are acceptable, and whether the product is ready.
Use AI for scale, breadth, and repetition. Keep humans responsible for context, judgment, and change.