At Waymo, an AI project isn't ready until its evals are — not when the model performs well

Few companies face higher stakes when deploying AI than Waymo, the self-driving car company under Alphabet that spun out of Google. Its models do not merely generate text or automate back-office tasks: They help vehicles navigate unpredictable streets, respond to human drivers and make split-second decisions in the physical world.But the methods Waymo uses to manage those risks — continuous evaluation, carefully curated data, human oversight and clearly defined business outcomes — offer a broader playbook for enterprises deploying AI agents in nearly any industry.Manasi Joshi, Waymo’s director of engineering for systems intelligence and machine learning, explained at VB Transform 2026 how the autonomous vehicle company trains, tests and deploys AI at scale. To date, Waymo has driven more than 220 million fully autonomous, or "rider-only," miles, with 17 times fewer serious crash injuries than human drivers over the same distance, according to the company. To achieve these impressive results, Joshi said Waymo has adopted what she called “eval-forced development” or “eval-centric development,” making evaluation a core part of engineering rather than a final check performed before deployment.“The stage at which our projects are maturing can be easily kind of transpired based on the eval maturity that they showcase,” Joshi said.In practice, Waymo assesses a project’s readiness partly by examining the maturity of the tests surrounding it. That approach has clear implications for enterprises building customer service agents, coding assistants, financial systems or other AI applications: If a company cannot reliably measure a system’s performance, it may not be ready to place that system into production.Evals must continue after launchJoshi said much of Waymo’s quality work has shifted toward evaluations, including tests conducted during model training, after training and inside open-loop and closed-loop simulations.“Eval is not a one-time task to launch a model,” she said.Waymo instead treats evaluation as a continuous process spanning driving, simulation and validation. Its methodology combines datasets, performance metrics and infrastructure capable of operating efficiently at scale.For enterprises, that means testing an agent before launch is insufficient. Teams must continue evaluating it as underlying models, business processes, user behavior and incoming data change. Those evaluations should also connect to actual business outcomes rather than relying solely on broad industry benchmarks.Joshi cautioned that model-quality measurements are only as trustworthy as the evaluation data behind them. Waymo therefore pairs its performance claims with information about the properties of the datasets used to test its systems.Testing the rare and dangerous casesWaymo’s evaluation hierarchy remains grounded in one overriding objective: safety.The company draws on first-party driving logs, some third-party data and realistic simulations that expose its systems to scenarios spanning billions of synthetic miles. Task owners choose specialized data and metrics for situations involving vulnerable road users, railroad crossings, construction zones and other complex environments.The same principle applies outside autonomous driving. Enterprises need to test not only the routine requests their agents handle successfully, but also uncommon situations where errors could create financial, legal, security or reputational damage.Joshi emphasized that Waymo does not leave release decisions entirely to automated systems. Its production-readiness reviews include extensive human oversight, while internal safety leaders approve software releases and service-area expansions.“This is not AI-driven and completely automated and zero human oversight,” she said. “Human lives are at stake.”Efficiency cannot come at the expense of reliabilityWaymo faces another problem familiar to enterprise AI teams: Demand for compute, storage, memory and network capacity is grow