Rand analysts warned that AV platform developers weren’t performing enough other types of testing prior to putting their AVs on the road.
Rand analysts warned that AV platform developers weren’t performing enough other types of testing prior to putting their AVs on the road.
( Source: Copyright (C) Andrey Popov)

AV Platform Development AV Platforms - Simulation testing vs. real-world testing

Author / Editor: Seth Lambert / Nicole Kareta

In the world of autonomous vehicle (AV) technology companies, two radically different approaches to AV platform development are slowly emerging. Which concept is more recommendable?

As demonstrated by companies like Waymo and Cruise, one approach to AV platform development is to commit to extremely onerous amounts of real-world testing of AVs on public and private roads in order to learn about rare “edge cases”—driving scenarios that are seldom encountered in real life, yet still theoretically possible. An opposing approach, as taken by startup firms such as FiveAI in the UK, Ascent Robotics in Japan, and others, is to restrict most testing to AV driving simulators that are run on computers. Not only is the latter approach far less costly and painstaking, but a growing number of believers are convinced this is the better approach as the number of miles “driven” by an AV platform (which is always greater when done in simulation) will directly correlate to its preparedness for utilization in the real world.

When enterprises such as Google sister company Waymo and General Motors subsidiary Cruise each set out to build their respective AV platforms, the conventional wisdom was that a huge amount of road testing of AVs equipped with their technology would be necessary to “learn” how these platforms would respond to real-world driving situations.

Waymo was one of the first companies to dedicate a fleet of AVs to the laborious task of driving physically from one location to another, on many different types of roads, in assorted weather conditions, and with varying amounts of traffic. The company began these real-world driving tests all the way back in 2009 when the company was still under the Google corporate umbrella. Since then, Waymo (the company was made a separate entity and renamed in 2016) has driven its AVs more than 20 million real-world miles—far more than any other competing AV platform company. For Waymo, this record is an achievement that the company has been reiterating for years.

For its part, Cruise has driven AVs nearly a million miles in the real world in 2019 alone, mostly in and around its San Francisco headquarters—in “the most dense part” of the city, according to Cruise founder Kyle Vogt. Cruise claims that testing in a much more challenging urban location gives the company an edge over Waymo, whose majority of tests in the nearly-always-sunny, flat, and more sparsely populated suburbs of Phoenix, Arizona do not present enough driving variety for the latter firm’s platform. (Cruise says its AVs encounter pedestrians, cyclists, construction and six-way intersections up to 47 times more often than Phoenix-based Waymo AVs in their San Francisco surroundings.) Between the two companies, in the state of California, Waymo and Cruise accounted for 79% of the nearly three million miles driven by the 60+ AV developers registered to operate on public roads in 2019.

Simulation testing

Both Waymo and Cruise state that in addition to real-world testing, they also record and store their AVs’ operational driving data. They then employ simulators to play back the data and alter variables iteratively in order to fine-tune their platforms’ algorithms via processes of experimentation and machine learning (ML). Like real-world tests, simulation testing like this is particularly valuable for teaching platforms how to handle “edge cases”—driving scenarios that are rare, but still possible, meaning that the risk of a driver one day encountering them is not zero. In fact, many academic members of the AV development community have stated that it’s these “edge cases” that will ultimately determine whether an AV platform can truly guarantee safety for human occupants of a vehicle.

It should also be said that while real life presents “unpredictable” driving circumstances on a daily basis, simulators are capable of coming up with much more complex driving situations than drivers could ever encounter on real roads (for instance, a case of 10 cars entering an intersection all at once, or 50 pedestrians all crossing the same street from many different directions). Thus, simulator experience can potentially be claimed to be more valuable for ML than real-world driving.

Waymo says that one day in its simulators is equivalent to 100 years of driving experience in the physical world. The company says its AV platform has logged more than 15 billion virtual miles “driven” via simulation. At Cruise, the firm claims its simulators generate the equivalent of 200,000 miles daily.

Prioritizing simulation

While both Waymo and Cruise have viewed simulators as complementary to real-world testing, AV startup firms like the UKs FiveAI and Japan’s Ascent Robotics have chosen them as the primary source for their platforms’ ML. In the UK, FiveAI was one of the first companies to eschew the need for pre-made, high-resolution 3D environmental maps in favor of on-the-fly map generation and in-the-lab iterative simulation-based testing. The company’s goal is to be a B-to-B solution provider to other AV technology companies. “We’re able to explain how we view the problems [of real-world trials] and how we turn that into software that’s useful for other people,” says FiveAI CEO Stan Boland.

In Japan, Ascent Robotics has shown off a videogame-like, AI-enhanced virtual-reality (VR) AV simulator called Atlas. Atlas was built specifically to mimic the real-world driving environments of Japan, where traffic can be denser and streets can be narrower than in other places. Playstation steering-wheel controllers and VR headsets allow testers to interact with Atlas. Perhaps not coincidentally, Japan’s “Father of the Playstation”—former Sony Computer Entertainment President Ken Kutaragi—has joined the firm as an outside director.

Ultimately, however, the devil of simulation testing may be in the details. Critics say that simulators are only as good as the verisimilitude with which they can duplicate real environments—which includes the accuracy of positioning, trajectories, speeds, and tactile qualities of both moving objects, like pedestrians, cyclists, and other vehicles, and stationary ones, like buildings and fire hydrants.

While some companies like Ascent Robotics and NVIDIA have built their own simulator software from scratch or partnered with others to do so, other firms, such as Toyota, have made use of off-the-shelf 3D computer gaming engines for their simulators, customizing them with their own data or, in some cases, with information gleaned from real-world driving tests.

How much testing is enough?

But despite so many collective tests, some observers are convinced they’re not enough. According to analysts at Rand Corporation, most AV platforms will not be qualified to universally drive on all roads in all types of weather and traffic conditions until they’ve journeyed the equivalent of hundreds of millions of miles in the worst of cases, and possibly hundreds of billions of miles in the best cases.

In a report from several years ago, Rand analysts warned that AV platform developers weren’t performing enough other types of testing prior to putting their AVs on the road—even if the latter action is merely for platform makers to acquire further experience in the real world. “Developers cannot simply drive their way to safety”, claimed the report, as the number of miles of real-world tests generally do not approach the number needed to completely avoid deaths and injuries.

Using formulas integrating failure rates, reliability rates, and fatality rates, the Rand analysts effectively stated that “on-the-job” training of AV platforms using human drivers—whether on closed private roads or in the context of operating robo-taxi services—presents unacceptable risks to vehicle occupants. By extension, simulation testing is therefore not an option but a necessity for AV developers. But even with exhaustive testing, claimed the Rand report, the safety of AV occupants may not be guaranteed.

“The self-driving teams I’ve led over the past 17 years have championed various attitudes toward on-road testing,” says Chris Urmson, the CEO of California-based AV company Aurora and the former CTO of Waymo when it was still under the Google moniker. “While huge numbers of on-road miles may initially seem impressive, experience has taught me which approaches merely look like progress and which can actually move the needle toward real progress.” For Aurora, its simulators conduct 735,000 tests virtually per day, which is 100 times the rate the company was running a year ago. Also Waymo has admitted in company tweets that “most of the development, learning, and validation of the Waymo Driver [AV platform] comes through [the] billions of miles driven within our simulation environments.”

Whether large companies such as Waymo or Cruise or small startups like FiveAI and Ascent Robotics will be able to achieve the truly staggering number of test miles proposed by the Rand report that would be necessary to avoid fatalities and injuries completely is questionable. There are some AV industry specialists, like MobilEye President and CEO Amnon Shashua, who believe that each time a platform is “tweaked”—even if the change is to just one line of software code—test mileage counts need to be reset to zero.

Financial pressures

For companies like Waymo, continued pressure from parent organizations to make progress toward profitability has raised questions about how much value robust real-world testing is producing. In its most recent financing round, Waymo took money from outside investors such as Silver Lake Partners, AutoNation, and AV platform developer/auto part firm Magna International for the first time in its history, in an indication that its parent Alphabet is not prepared to keep throwing money at the AV enterprise forever.

In 2018, Waymo started its Waymo One robo-taxi service in the suburbs of Phoenix in order to subsidize further ongoing testing efforts. Ireland-based Aptiv is another company to have taken this route, operating a fully automated robo-taxi service in Las Vegas in association with ridesharing firm Lyft. Still other companies such as Cruise, Toyota, and Hyundai have indicated they intend to get into the robo-taxi business as well in order to finance their AV platform programs as disillusionment about quick achievement of Society of Automotive Engineers (SAE) Level 4 or 5 (“high automation” or “full automation”) functionality spreads among investors.