Augmented Reality on the Road: Cars With a Vision

Augmented reality and the superimposition of real images with virtual animations is generally known from games like Pokémon Go.

But this technology also is finding its way into our cars through various exciting applications: wearable technology, head-up displays visible only to the driver, or even the windshield of the vehicle as a real, semitransparent display that both driver and passengers can view and interact with.

Combined with various modalities such as speech, gaze tracking, and gesture recognition, AR improves usability and provides easy access to a variety of information, from basic needs like our phone’s contact list, to more complex information like diagnostics from the car’s sensors.

Portable or Integrated?

There are typically two basic types of displays that can be used for AR applications: wearables and displays integrated in the car. Wearable AR leverages glasses that can provide additional information about the world we see. The glasses seamlessly switch between different scenarios and add context to the user’s surroundings, regardless of their location. For example, a user could park their car in a parking lot, and the glasses will guide them directly to the destination via projected maps.

But inside the vehicle, wearables can cause distractions and interfere with driving. This creates  the need for technologies that are integrated with the car itself, such as HUD or windshield-based AR. HUD and smart windshields are screens in the driver's natural field of vision that can provide a variety of information without taking eyes off the road. Further, video-based AR systems use the vehicle's forward-facing camera and lay an additional layer of information over it. A famous recent example is the new Mercedes-Benz A-Class, which offers an AR-based navigation application with the vehicle's front camera.

Ease of Use is Critical

Most recent AI-based advancements for the in-vehicle experience are aimed at providing additional ways for the car to collect and provide information, whether via speech-, gaze-, or gesture-based interaction; cabin surveillance systems; or advanced sensor fusion functions. AR is another way the car can provide its drivers and passengers with additional, complex information in an easy-to-use format.

An important aspect in this context is the display surface, which significantly influences the choice of feasible applications and user-friendliness. For example, new technologies such as the smart windshield from Saint-Gobain Sekurit use the entire windshield as a transparent display. This allows the user experience to be enriched by the addition of highly targeted and intelligent content offerings shown directly on the windshield – without compromising visibility.

Use Cases

AR technologies can be used for a variety of applications, including infotainment, safety or convenience features. When the focus is on infotainment, the windshield can be used to provide the user with information about the visible outdoor environment, interesting restaurants or the offerings of a newly opened cafe.

For convenience applications, the windshield can be used as an output channel when the user interacts with on-board systems or online content, keeping their eyes off the instrument panel and on the road.

Safety is not limited to minimizing distractions. For example, AR can be used in combination with state-of-the-art sensor fusion and eye tracking technologies to direct the driver’s attention to exactly where it’s needed, like another vehicle or a hazard on the road.

Autonomous Driving

The more autonomous driving develops, the more important AR will become. On one hand, there are considerably more possibilities for entertainment and infotainment functions if the driver no longer must concentrate on traffic. On the other hand, AR can play an interesting role in providing information about maneuvers and a general understanding of the vehicle. This can increase the feeling of safety for passengers who are critical or unsure about riding in an autonomous vehicle.

The applications for AR, both in the cars of today and of the future, are plentiful. In the future, it will be crucial for us to consider how we can best benefit drivers and passengers as we implement these applications, making the driving experience more entertaining, productive, and safe.

Nils Lenke Headshot