While we’ve previously covered the car industry’s plans for applying augmented reality to head-up displays projected onto cars’ windshields, an alternative solution would be for drivers to wear augmented reality glasses like Microsoft’s HoloLens 2.
Volkswagen thought that a great application for the HoloLens 2 was a driving program where drivers on a race track receive steering and braking cues through augmented reality. When VW went to the track to try the system in 2015, however, nothing happened.
It turns out that when the HoloLens 2 goes into a moving vehicle its sensors lose tracking, so the holograms it normally displays disappear. That’s when VW engineers made a call to Microsoft for some very intensive tech support.
Investigation revealed that HoloLens uses two main types of sensors to measure its motion — visible light cameras and an inertial measurement unit. The IMU measures acceleration and rotational speed.
Put this system into a moving car and it suffers the electronic equivalent of motion sickness. That happens when the motion we see is decoupled from the motion we feel, and when a HoloLens goes into a moving vehicle, the very same thing happens as the processor tries to reconcile what it’s seeing through the cameras with the motion its accelerometer detects.
Just as the problem is similar to human motion sickness, so is the solution. Looking out the window to get your bearings is often helpful for preventing motion sickness. For the HoloLens, that means connecting a GPS on the car to the glasses so that they have a firmer understanding of their position compared to their surroundings.
Joshua Elsdon, a Microsoft senior software engineer who worked on the project, had to find solutions from his Zurich apartment during the Covid shutdown. He mocked up a solution using a plastic box, sticking bits of tape inside to add visual texture and give the HoloLens cameras elements to track.
He rode trams and buses around Zurich wearing a HoloLens headset, making sure its holograms held up as the vehicles moved. At night, Elsdon even rode up and down elevators in his apartment building to keep testing the technology.
“We had to do a lot of testing in my apartment,” Elsdon said. “These aren’t ideal development conditions. All of this stuff was done remotely and distributed across different countries, which was interesting.”
Now VW has a prototype of a system that could aid drivers. “We think mixed reality information is the most intuitive information we could provide to enhance our customers’ user experience,”, said the head of the data science team at Volkswagen Group Innovation, Andro Kleen. “Because what you see there, and what you need to process, is very close to what humans normally see and process. It’s not so abstract.”
HoloLens 2 glasses look like regular glasses, with a few extra features attached.
The company says that it is expecting to use the HoloLens 2 in several primary areas. One is the fairly conventional application of augmented reality for engineers doing prototyping in R&D, where they can do the iterative development of head-up displays and underlying functions or for sensor data for automated driving.
There are also two new use cases for mobile applications of HoloLens technology now that the problem of its motion sickness has been solved. One is to support professional drivers of heavy-duty and transport vehicles. The goal is to help them control technical systems in familiar and comprehensible ways and to drive in complex environments like mines and dirty roads, where the glasses might display hidden road hazards.
Another use is for vehicle passengers, providing seamless user experiences for automated driving and for passenger entertainment by displaying navigation information, enabling gesture control of car functions, or by projecting additional information such as points of interest in the real dimension.
Microsoft has its own applications in mind. One is to provide help to maintenance technicians aboard ships at sea to perform necessary repairs. Previously, they were able to get assistance walking them through the procedures to do work while ships were docked. Now the system can be used on ships at sea.
“The more remote the equipment or machine is, the harder it is to get the expert on site,” says Marc Pollefeys, Microsoft director of science and an expert in 3D computer vision and machine learning who serves as a professor of computer science at ETH Zurich, a public research university. “This feature turned out to be critical to unlock HoloLens 2 for the maritime space.”
So far, HoloLens’ moving platform feature only works on large ships, but Microsoft says it will refine HoloLens 2 for use in elevators, trains, and other moving environments.