Human–robot collaboration for improved welding seam inspection

Robots are being deployed in more and more situations, many of which involve collaboration between humans and robots — for example, relieving humans of onerous tasks in the workplace. The challenge now is how best to integrate the robot into the working environment and how to operate it. In a joint project with Volkswagen, the Fraunhofer Institute for Telecommunications, Heinrich Hertz Institute, HHI will demonstrate the advantages that the use of human–robot collaboration (HRC) can bring to the inspection of welding seams in the automotive industry.

Flawless welding seams are a critical quality criterion in automotive engineering. In the future, welds will be inspected by a human–robot team, with each contributing its specific skills and expertise. Controlled by gesture and voice commands, the robot will hold and manoeuvre the specific components into position, while the employee marks and records any defects in weld quality.

The joint research project EASY COHMO (Ergonomics Assistance Systems for Contactless Human-Machine Operation) draws on the many years of experience that Fraunhofer HHI has accumulated in the fields of 3D capture, 3D information processing and 3D visualisation. This system for the visual inspection of welding seams on key components in automobile production offers a great example of how HRC can function in the industry. In the coming years, this technology will be providing concrete assistance with inspection procedures at Volkswagen.

The inspection process has remained unchanged for decades. First of all, each component has to be manually fixed in a rotary positioner so that it can be inspected from all angles. Inevitably, this involves the employee having to assume unergonomic positions, which may lead to repetitive strain injuries. Furthermore, only a short time is set aside for this complex procedure, which can have a negative impact on the quality of inspection.

At present, employees are free to conduct the inspection in their own manner. This inevitably results in a non-standardised procedure that makes it more difficult to recognise systematic defects on the basis of different inspectors’ observations. In addition, any such defects often go unrecorded or have to be laboriously entered into a separate system via a keyboard and mouse.

In the future, this inspection process will be interactive, with robots used to hold heavy components and manoeuvre them into a position in which employees can inspect them in an ergonomic manner. Such robots will have at least six axes and will therefore be able to move, turn and tilt a component in every possible direction. The robot will automatically remove a component from the line and present it for inspection. The employee will interact with the robot by means of explicit and implicit gestures, thereby steering the component into the desired position. Employees can concentrate their attention on identifying defects and, as a result, will overlook fewer defects.

The middleware developed by Fraunhofer HHI coordinates the various sensors that serve to capture the overall work situation. Based on the employee’s position and gestures, the software calculates the required movement of the robotic arm. This also ensures user safety: for example, whenever the employee ceases to look directly at the component, the software halts the robotic arm as a precaution.

An augmented hand gesture-based user interface is projected onto the component and workbench. Image ©Fraunhofer HHI.

“The robot can also be set to respond to personalised instructions,” said Paul Chojecki, project manager at Fraunhofer HHI. “Our new perceptual interface is able to process a user’s individual gestures and voice commands. This means the system can be quickly customised to a workstation’s specific requirements.”

Gestures are used to mark, categorise and confirm defects on the component. By the means of precise object tracking and projector-based augmentation, employees are provided with an interface directly on the component in their working space. This offers an efficient and intuitive way of generating a digital 3D record of any defects in the welding seams. By contrast, the current inspection procedure lacks any systematic record of defects, because there is very little formalised exchange of information between different employees. With the new procedure, any defects can be immediately entered into the system and then statistically evaluated. This means that systematic defects are more rapidly identified and can thereby be eliminated at the welding stage.

The position of weld seam defects identified by the user can be determined by a combination of finger tracking and 3D seam tracking. Image ©Fraunhofer HHI.

The system features a large number of sensors combined with multimodal controls based on an enhanced middleware, along with customised operating instructions and machine learning. It is an approach that could well bring the breakthrough for further HRC applications — and also expand the scope of collaboration and interaction between humans and robots. Further areas of use include milling machines, for instance, or interactive robotic assistants in non-industrial settings — such as in the healthcare or services sectors.

The joint project EASY COHMO is part of the 3Dsensation research alliance and is financed by the German Federal Ministry of Education and Research (BMBF) as part of the government funding program Zwanzig20 – Partnerschaft für Innovationen.

Top image: The robot presents the workpiece in an ergonomic position to enable easy inspection. Image ©Fraunhofer HHI.