Training Autonomous Vehicle Systems in Virtual Worlds

The real world can be a complicated landscape to capture in entirety, especially for engineers working to uncover quality data to train autonomous vehicle (AV) systems.

While real-world testing and training is important for AV system development, testing and training in the real world to develop enough quality data for all scenarios needed can be time-consuming and costly for organizations hoping to innovate quickly. And many automated-driving engineers often ask how they can analyze and synthesize scenarios once created. This is when the conversation typically turns to the role of modeling and simulation.

Three different approaches to modeling are: first principles, behavioral and data-driven. With the data-driven approach there are large amounts of data where artificial intelligence (AI) and AI modeling can be leveraged.

AV systems comprising a combination of these models are used for testing to assess how the vehicle’s software reacts to dozens of pedestrians, other vehicles and traffic scenarios over hundreds of miles of road, thousands of landscape features and just about any weather condition imaginable. 

That’s where simulation comes in as it can help build quality data for engineers to input into AV systems, training them for all conditions and scenarios. A simulated “virtual world” spans multiple areas of an environment, such as scenes, scenarios, vehicles and dynamics.

ADAS algorithms (MathWorks)

By developing a virtual world that accurately reflects the real world, engineers can run AV software through hours of simulation prior to testing AV systems on the road.

Virtual worlds require multidisciplinary skills from engineers for both developing algorithms using multiple tools and deploying these algorithms as software applications.

Despite the many disciplines and considerations engineers must keep in mind for AV system development, the key element for success in practical development of AV applications is uncovering issues early on and knowing what aspects of the workflow on which to focus time and resources for the best results. Here’s how AV engineers can build a reliable virtual world.

Start With a Scenario

As anticipated, engineering simulations need to start with a real-world scene to emulate before running scenarios to create data. The scene can be simple at first, such as a standard roundabout, but needs to be scalable to, for example, the 12-entry Arc de Triomphe in Paris.

The scene is just the starting point and requires engineers to incorporate other elements such as vehicles, pedestrians, scenery and even weather into the simulation before testing how the AI-based perception system, which in turn drives the motion planning and control system in an AV, reacts to the scenario.

Determine What’s Being Tested in the Virtual World

There’s also the question of what specifically is being tested, which can determine the fidelity of a component or subsystem with respect to the real world.

If it’s vehicle dynamics, it’s critical that the scenario includes models of cars, trucks, trailers and other multi-axle vehicles, and that their wheels, steering and suspension mirror their real-world counterparts as closely as possible.

If the scenario is testing radar sensors, it’s less important that each object be photorealistic (as would be the case if camera and lidars were being tested), but critical that the number of objects surrounding the simulated vehicle corresponds with the real-world environment.

Scaling Virtual Worlds

Developing virtual worlds needs to be a scalable process that gives engineers the flexibility to apply their domain expertise as well as ease their transition to new domains.

Such a complex simulation requires multidisciplinary skills, with separate algorithms required for features such as automatic emergency braking, adaptive cruise control and automated parking – and each system must interact with other systems encompassing perception, planning and controls.

Fortunately, to create virtual worlds that include vehicles and other objects, engineers have access to tools to design simulated environments and modeling tools to test simulated vehicles in the virtual world.

Govind Malleichervu_MathWorks1

Accurate simulation and testing are key to ensuring that an AV system is working properly, and that its individual components work well with each other before deployment in the real world. At every stage, engineers must verify that each feature and its associated algorithm is optimized and working properly and that it performs as expected in each scenario and use case.

It’s only once each algorithm is functionally, verifiably correct in the virtual world that it can be implemented as embedded software capable of powering AV systems on the road.

Govind Malleichervu (pictured, above left) is Automotive Industry Manager with engineering software developer MathWorks.