A new study by VentureBeat Pulse Research, conducted in June 2026 among 157 technology leaders in enterprises employing 100 or more workers, reveals a significant gap between the degree of autonomy organizations grant to AI agents and their level of trust in the evaluation systems meant to test them. The report, titled "The agent evaluation gap," shows that half of organizations have already deployed an agent or a large language model (LLM) feature that successfully passed their internal evaluations but subsequently failed before customers in a production environment. Despite this, two-thirds of organizations currently allow, or are actively working to allow, the deployment of agent changes to production based solely on automated evaluations, without any human-in-the-loop involvement.
The Evaluation Gap: Agents Pass Internal Tests but Fail Before Customers
The central finding of the study defines the "evaluation gap"—the distance between the autonomy organizations grant to their AI agents and the trust they place in the testing designed to prevent failures. According to the data, 50% of organizations that perform evaluations deployed an agent or LLM feature during the past 12 months that successfully passed internal testing but experienced a direct failure in front of customers in production. These failures included incorrect outputs, broken workflows, or other quality incidents. Within this group, approximately a quarter of the organizations experienced such an incident more than once. Only 36% of respondents reported experiencing no such failures, while the remaining respondents reported that they do not perform pre-deployment evaluations (8%) or do not closely track the root causes of failures (6%). These figures indicate that passing an internal evaluation test does not guarantee the proper functioning of an agent in practice.
Trust in automated testing itself is exceptionally low: only 5% of organizations state that they fully trust automated evaluations today. This means that 95% of organizations point to specific limitations preventing them from placing full trust in these tests. The most common limitation, cited by 29% of respondents, is a lack of alignment between evaluation results and real-world performance (results that pass testing but fail in practice). Other cited limitations include bias or inconsistency in testing (21%), a lack of explainability in evaluation outcomes (18%), and concerns regarding data leakage or privacy violations within the evaluation process itself (17%). Many organizations are finding that passing an evaluation is not a guarantee that the agent will operate properly.
Automation Advances Without Human Oversight in Production
Despite the prominent lack of trust in automated testing systems, the study shows that the autonomy ceiling for agents continues to rise rapidly. Two-thirds of organizations (66%) already permit fully automated deployment without human intervention for agents defined as low-risk (34%), or are actively developing their deployment pipelines to allow this within 12 months (33%). Only 22% of organizations rule out deployment without human oversight in the foreseeable future. The trend is clear: organizations are moving toward granting automated deployment approval based on evaluations, thereby removing human control, precisely at the point in time when they admit that these evaluations do not align with real-world reality.
This trend is not unique to small companies. When segmented by company size, larger enterprises are further along the path to zero-human-review deployment compared to smaller companies (70% versus 64%). Additionally, large companies were slightly more likely to have shipped an agent to customers that passed internal tests but failed in practice (54% versus 48%). The report notes that these figures are directional, as the sample included 57 respondents from companies with over 2,500 employees and 100 respondents from smaller companies.
The Evaluation Tooling Market: Segmentation and Platform Switching Intentions
The evaluation layer for AI agents is characterized by heavy fragmentation and early-stage development. The most common tools today are the native tools provided by model vendors: OpenAI's evaluation platform (with 17%) and Anthropic's evaluation tools in the Claude Console (with 13%). However, a particularly striking figure shows that 17% of organizations do not use any dedicated agent evaluation tools at all. Independent and specialized evaluation platforms are scattered in relatively low percentages: DeepEval leads with 12%, Braintrust with 8%, while other vendors (including LangSmith, Weave, Promptfoo, Langfuse, and Arize) are scattered across single digits. Around 11% of organizations have developed their own in-house tools.
No independent platform has yet emerged as the accepted industry standard, but the evaluation tooling market is facing a significant shift. A clear majority of 64% of organizations plan to adopt a new, additional, or replacement platform within 12 months, with 31% planning to do so as early as the upcoming quarter. In terms of evaluating new tools, Confident AI's DeepEval leads the consideration list with 20%, ahead of OpenAI (with 13%) and Braintrust (with 9%). This trend represents a first wave of dedicated tool adoption among organizations that have previously relied on solutions from model providers or have not used dedicated tools at all.
Real-Time Monitoring and Procurement Decisions
In the monitoring of AI agents in production, there is a clear distinction between operational monitoring (whether the system is running, response times, costs, and system errors) and output quality monitoring (real-time automated validation of answer correctness and policy compliance). This distinction is critical because an incorrect but "confident" answer from an agent will not trigger an alert in a standard operational monitoring system. The study reveals a significant gap in monitoring: 51% of organizations monitor only system functioning, while only 23% monitor the correctness of answers and outputs in real time. Roughly 75% of organizations do not implement automated monitoring of output correctness in production, relying instead on manual evaluations or not monitoring it at all.
When it comes to selecting evaluation tools, organizations act based on practical and financial considerations. The cost of testing is the most influential factor in selection (28%), followed closely by ease of integration (27%) and evaluation accuracy (24%). Factors such as the breadth of observability capabilities (at 13%) and the vendor's roadmap (4%) have a lower impact. The primary success metric for organizations using these tools is evaluation consistency (36%)—meaning obtaining the same result for the same behavior over time—which is significantly higher than experimentation speed (19%), failure reduction (18%), production visibility (13%), and regulatory compliance (11%). Overall satisfaction with existing tools is moderate, averaging 3.8 out of 5.
Future Investments and Research Methodology
Despite the drive toward full automation in deployment, enterprise budgets indicate they plan to backstop the process using human involvement and tighter monitoring. The second-largest investment area for the coming year (behind operational monitoring and production observability) is human review workflows, cited by 26% of respondents. This figure exceeds the rate of organizations planning to increase investment in automated evaluation pipelines (16%). Only 8% of organizations reported that their budget for this area is not expected to increase. These data suggest that organizations are hedging their risk: they are building pathways to autonomy, but simultaneously investing significant resources in human reviewers and monitoring systems to catch failures that automated evaluations miss.
The research methodology is based on a targeted survey conducted in June 2026 as part of VentureBeat's Pulse Research series. Respondents were filtered to include only technology leaders from organizations employing 100 or more people (n=157). The sample consists of senior executives and purchasing influencers: 38% are final decision-makers for AI procurement, and 34% are recommenders or influencers. Roles include product and program managers (15%), consultants (10%), engineering and IT directors (8%), and Chief Technology, Security, and Information Officers (CTO/CIO/CISO) (8%), alongside other roles (37%). The breakdown by organization size was weighted toward mid-market companies: 100-499 employees (37%), 500-2,499 employees (27%), 2,500-9,999 employees (20%), 10,000-49,999 employees (10%), and giant enterprises of 50,000 or more employees (6%). The leading industry in the sample is technology and software (23%), followed by retail and consumer goods (15%), healthcare and life sciences (12%), and manufacturing (10%). The report serves as a directional indication only and does not constitute a precise probability sample. The survey was rebuilt in June 2026, and therefore no comparisons were made to prior data from April and May of the same year.