In recent years the acceptance of augmented reality (AR) glasses in production and maintenance applications has been limited. Users complained that the headsets were too bulky, emitted too much heat and had a restricted field of view. In addition, the high development costs were a disincentive for small and medium-sized enterprises. In the meantime Microsoft has made numerous improvements. The HoloLens 2 has been available since the autumn. Research now centres on platforms, predictive systems and applications in forklift systems.

The HoloLens 2 deploys a new time-of-flight depth sensor, combined with integrated AI and semantic understanding. It facilitates the direct manipulation of holograms with the same instinctive interaction that users experience with physical objects in the real world. This feature has been praised by testers. Gripping experiments function without any problems. In addition to the improvements in the display and the direct manipulation of holograms, the HoloLens 2 boasts two eye-tracking sensors which are intended to make interaction with holograms even more natural.

The research community has embarked on a new approach to smart glasses. Working together with three major forklift truck manufacturers and other industrial companies, scientists from the Institute for Integrated Production Hannover (IPH) and the Institute for Transport and Automation Technology (ITA) at Hannover University aim to integrate AR into everyday forklift operations. In future the truck drivers will wear AR glasses that enable them to see through obstacles. In addition, important information and warnings can be displayed in the field of view. The camera image is superimposed on the driver's real field of vision. If, for example, he looks directly ahead, he sees the lift mast and the load as well as the image from the front camera. When the driver engages reverse gear and looks over his shoulder, he sees the image from the rear-view camera and at the same time his actual field of vision. No matter in which direction he looks, it is as if he can see through obstacles.

Over the next two years the research partners will have to overcome a number of challenges. Firstly, they will have to find suitable locations for cameras on forklift trucks so as to create a virtual panoramic view. This is the main responsibility of the scientists at the ITA. They are also responsible for the algorithm of image superimposition: all obstacles that restrict visibility will be overlaid by camera images. The scientists at the IPH are working on a so-called ‘context-based information display‘. The question is: What additional information does the forklift driver need at what time, and how can this be displayed in his field of vision? For example, it would be possible to display order data so that the driver can dispense with printed lists. Warnings could also be displayed, such as a low battery charge or a blocked path. To avoid overtaxing the driver and possibly endangering road safety, scientists at IPH are investigating which information is really needed in which scenario. They are also developing gesture control so that the driver can interact intuitively with the AR glasses. With simple hand movements the driver will be able to cancel screen messages ‒ for example, if he has taken note of a warning ‒ or display additional information such as order data.

Predictive head-mounted display

Bielefeld University and the local University of Applied Sciences are working on a new type of head-mounted display – one with predictive capabilities. “The defining feature of our assistance system is that it does not simply give instructions. It knows the user, comprehends the current situation, independently identifies objects and courses of action, and adjusts its support accordingly,” explains Professor Thomas Schack, who is coordinating the Avikom project. Schack's group belongs to the Faculty of Psychology and Sports Science and is a participant in the Excellence Cluster for Cognitive Interaction Technology (CITEC).

Avikom stands for “Audio-visual Support with a Cognitive and Mobile Assistance System”. To permit the system to adapt to the respective users, their skills are assessed in advance using software-based diagnostics. In this way the software diagnoses in advance which difficulties the users will encounter during various work processes. On this basis the system can provide individualized information to support users in a targeted and motivating manner. “The Avikom headset thus offers an excellent opportunity to adapt technical support to the individual needs of employees,” explains Prof. Dr. Günter Maier from the Faculty of Psychology and Sports Science. He is contributing to the project via his research group “Work and Organizational Psychology.”

AR platform

The researchers have combined text and audio in a single head-mounted device. The Avikom system can talk to the user in a way similar to a car navigation system. “In addition, employees in noisy production zones can also talk to each other via the device without being disturbed by ambient noise,” assures Professor Dr. Joachim Waßmuth of the Institute for System Dynamics and Mechatronics (ISyM) at Bielefeld University of Applied Sciences. “For this purpose the headset is equipped with a smart noise cancellation system.”

But how can SMEs integrate AR into everyday mechanical engineering applications? “Augmented Reality is already used successfully in many areas – for example, video game and sales support. So far SMEs have failed to profit from these possibilities due to the high development costs. Our AcRoSS platform is one way to reduce these costs. We are demonstrating exciting potential for additional services and business models with AR,” explains Daniel Röltgen, AcRoSS project manager at Fraunhofer Institute for Mechatronic Systems Design (IEM). The basic idea is that companies can create customized AR solutions and services from finished modules and operate them via the platform.

Today, repair and assembly instructions are frequently provided in the form of smart data-based services. Via the AcRoSS platform, these services could be combined with various pre-fabricated augmented reality components such as position recognition and gesture control. Communication and data transmission take place centrally via the platform software. In companies where costly and time-consuming in-house developments are not feasible, the platform could provide easy access to augmented reality technology.