When Simulation Isn’t a Game
The vital role of simulators in creating and assessing self driving cars
I played Flight Simulator on a Commodore 64 when I was young. My plane crashed (a lot!) but it didn’t matter because it was just a simulator. The game tested my flying skills by arbitrarily changing the weather or having an instrument fail so I could learn how to react to unusual circumstances. Now the same principles are at the heart of the trillion dollar self driving car goldrush.
Despite the billions of dollars already invested in the effort to create self driving cars, the consensus among experts is that we are still years away from widespread deployment of verifiably safe, fully autonomous cars that can cope with all conditions. Given the nearly 40,000 US road deaths last year, technology to reduce the toll can’t come fast enough. But people (and regulators) are understandably nervous about sharing their roads and towns with early versions of 2-ton machines that may act unpredictably, and lack common sense, conscience or accountability.
Testing, Testing and More Testing
Because the AI systems powering self driving cars learn by experiencing situations, autonomous vehicles would have to be driven hundreds of millions of miles and, under some scenarios, hundreds of billions of miles to create enough data to clearly demonstrate their safety. Americans drive nearly 3 trillion miles every year, resulting in about 1 fatality per 100 million miles driven. According to ABI Research, “accruing the tens of billions of miles necessary to have confidence in autonomous systems would require the deployment of at least three million still unproven autonomous vehicles over the course of 10 years.”
Closed course testing is a valuable tool; a private facility such as Waymo’s ‘Castle’ testing ground, or shared resources, such as Mcity at the University of Michigan or K-City in South Korea, enable testing in controlled environments that resemble streets, but without the risk to the public. However, closed course testing only tests for things you can think of — self driving cars need more data (experience) than can be gained on a test track.
Real world testing is expensive, potentially dangerous and very slow. The sheer scale of the task also goes against exclusively real-world development — it’s just not practical to accumulate all of the testing experience necessary to develop an artificial driver.
Experimental self driving cars can’t be allowed to learn solely by trial and error on public roads. They will inevitably be hesitant, fallible and even dangerous in their early stages. Current development work on public roads still has a human operator present to take over in case of a problem. So while the most obvious development effort is visible in the sensor-laden test vehicles on selected public roads, the majority of the work to solve the challenge of driverless cars now happens out of sight, out of the physical world: Simulators are key.
The Simulation Shortcut
Driverless car simulations are sophisticated: autonomous vehicle companies use highly detailed computer simulations to tweak their vehicles safely before putting them on the road. This software-based technology allows engineers to drive many more miles — potentially millions of miles a day — and try out many more scenarios than they could ever hope to go through in the real world, however big a fleet of vehicles they might have.
There’s plenty of precedent for using simulations in high-stakes situations. Pilots learn to fly in simulators. Surgeons learn techniques using simulators. And now, increasingly, self driving car companies are using simulators to train their “drivers” of the future.
Simulation provides an instant means to train and assess performance in a risk-free environment, on a massive scale. It plays a critical role in helping validate new software releases before they are installed on test vehicles. It allows automakers to try out scenarios they cannot or do not want to recreate in the real world. A simulation lets autonomous cars experience and learn from situations that are too dangerous to try in reality. Dangerous situations, real or contrived, can be scripted in simulation to test the autonomous car’s ability to react, without ever putting anyone in harm’s way.
Real world testing may encounter difficult scenarios from time to time, but not frequently enough to learn how to handle them. For example, in the real world, you only have a few minutes every day to drive a particular road as the sun goes down. In simulation it’s possible to “drive” every road 24 hours a day at sunset, test for those lighting conditions, and then layer-in different weather, traffic, etc.
Carcraft
“Our progress on public roads is made possible by our deep investment in simulation”
John Krafcik, CEO, Waymo
Leading self driving car firm Waymo announced towards the end of 2018 that their fleet of self driving vehicles has driven more than 10 million miles, which far exceeds the distances driven by their competitors. But as impressive as that real-world mileage may seem, in their simulation software called Carcraft, 25,000 virtual self-driving cars collectively drive about 10 million miles every day, running in Google’s vast datacenter computers.
In their most recent Safety Report, Waymo outlined how their development process flows from Simulation to Closed-Course Testing before being introduced to their on-road fleet. Waymo’s simulator can replay the real-world miles driven by their test fleet, but also can build completely new realistic virtual scenarios for software to be tested against. In order to ensure the fidelity of the outputs, the simulator starts with a highly-detailed virtual replica of the roads and intersections, complete with identical dimensions, lanes, curbs, and traffic lights. Each driving maneuver, be it a turn, merge or obstacle avoidance, can be undertaken thousands of times over to practice and eventually master a skill. To make a scene more complex, the simulator can add vehicles, pedestrians and cyclists that never existed in the original scene. Through a process called fuzzing, they can alter the speed, trajectory and position of objects on these virtual streets, creating hundreds of variations from any one scene.
Engineers can then isolate combinations of factors that create problems for the robocars, and concentrate on fixing those issues. New skills learned in the simulator can be tested and then shared with every vehicle across the fleet. Following that, the private closed course testing facility and real-world driving can be used to validate the simulated experience. Waymo now say that since the basic driving software works well in daily situations, the vast majority of improvements and refinements made are based on things picked up in simulation.
Not All Miles Are Equal
For firms developing self driving cars, testing in the real world follows a law of diminishing returns — the more advanced your autonomous driving system becomes, the more miles you actually need to drive to have an impact, because you’ve handled the basics and are moving on to edge cases. By definition, it’s hard to have training data for rare events.
In simulation, firms can focus on the most challenging interactions which they are unlikely to face in their daily road testing. The difficulty of the miles tested is crucial — this is one reason why some commentators are sceptical about published industry claims regarding miles tested as a barometer of overall progress. If firms are testing only on a small subset of roads, while they can refine their performance within that domain, it remains a limited dataset, lacking in variety of examples.
Lots of Simulators
GM’s self driving car unit, Cruise, has developed its own simulation tools and runs about 200,000 hours of compute jobs each day on Google Cloud, creating around 30,000 virtual cars. Chip maker NVIDIA, which develops self driving computers for car manufacturers such as Toyota, also now offers its simulation set of tools, called Drive Constellation, which provides photo-realistic simulation and even allows third-party developers to add cityscapes, vehicle dynamics models and traffic scenarios.
Until recently, Simulators were developed as part of the self-driving teams’ work at companies like Waymo or Zoox, or by automotive suppliers like ANSYS, Siemens, Mobileye, dSPACE or IPG Automotive. Increasingly, however, simulators are also being built by startups, such as Metamoto. While bigger players have their own simulations, “Simulators as a Service” makes sense for companies who don’t have the resources to both build self driving cars and accurate simulators. There are even several open source solutions should you wish to try creating your own simulations. (https://github.com/carla-simulator/carla or https://github.com/lgsvl/simulator)
The use of simulation to supplement real world testing is not limited to self driving cars. Amazon is using similar technology to recreate the sidewalk environments to speed the development of their delivery robot, known as Scout. The custom-built environment includes the position of curbstones and driveways and street furniture within centimeters, while different surface textures and varying weather conditions can be tested without blocking sidewalks with too many physical robots.
Fidelity
Simulator results don’t always play out as expected in the real world — ask the Formula 1 teams who regularly develop parts that are faster in simulation but are slower when they are tried on track! Waymo’s CTO identifies fidelity as the number one requirement for a simulator -
“I think what makes it a good simulator, and what makes it powerful is two things: “One, fidelity. And by fidelity, I mean, not how good it looks. It’s how well it behaves, and how representative it is of what you will encounter in the real world. And then second is scale.”
Unlike games, the power is that they mirror the real world in the ways that are significant to the self-driving car and allow it to get billions more miles than physical testing would allow. The self driving car simulators owe more to the world of aerospace and defense than Grand Theft Auto. The simulation has to be a faithful representation of what would have happened in the actual vehicle. If the timing is off by even a fraction of a second, it can change the outcome. Firms need to constantly assess the correlation between simulator and real world testing. They need to create a virtuous cycle of data where simulation becomes more sophisticated and more accurate as you accumulate real-world driving miles, which means the results of your virtual testing are more reliable for use back in your cars driving on actual roads.
The New Driver Test?
One of the biggest areas for debate in self driving cars is regulation — how do we determine when they are safe enough? How do you compare systems from different providers? Can you have confidence in how each system will cope in varying environments and conditions?
Today in most jurisdictions, for human drivers, we test for knowledge of prescribed behaviours, and we take learner drivers on a test drive with an examiner to assess how well they seem to control the car. Then, with the omnipresent threat of police enforcement, a driving license is issued in the hope drivers will act in a responsible and compliant matter, knowing they should, in most cases, have the common sense to respond to unfamiliar situations in a generally safe way. So how should we license robodrivers?
Just as all the major players in the development of self driving cars are relying on simulation to do the bulk of the work alongside the more visible test vehicles, regulators are also considering if simulation is a viable way to assess objectively the capabilities of robodrivers and to certify their proficiency. Validation will likely require an extensive mix of simulation and physical testing procedure in order to cover all known, including rare, critical road traffic scenarios.
With so many different simulation platforms in operation, there’s a need for a standard independent system to evaluate objectively. The complexity of testing many vehicle types in multiple driving scenarios poses a major challenge for those tasked with ensuring autonomous vehicles are safe. Many of those involved with development and validation believe that some common tests will be needed as a foundation to determine whether vehicle systems perform safely.
Unlike human drivers, autonomous cars rely on training data rather than real knowledge of the world, so they can easily be confused by unfamiliar scenarios. Only simulation, along with closed circuit and real world testing, can significantly reduce the time to market and the cost of testing and certification of self driving cars. As Huei Peng, Director of Mcity said, any system that works for self driving cars will be “a combination of more than 99 percent simulation plus some carefully designed structured testing plus some on-road testing.”
