New system translates individual-specific gestures into speech for those with limited communication

A team of researchers at Pennsylvania State University has developed a prototype system that uses wearable sensors and artificial intelligence to convert gestures to speech. This system converts body movements that are unique to individuals into speech, differentiating it from generic systems.

This new approach — published in the journal Augmentative and Alternative Communication — is designed to help individuals with motor or visual impairments communicate more independently and effectively. The key innovation of this system is the individual-level customizability.

Unlike conventional gesture-to-speech systems trained on large datasets, the system is designed to learn idiosyncratic movements — gestures that have a specific meaning unique to a person. This allows the technology to adapt to the specific movements the user finds easy to make, thereby reducing the physical stress on the user.

The system works by having a person wear a sensor on their wrist and repeating a gesture about three times. Using an AI algorithm, it measures the motion characteristics of the gesture, learning its unique pattern. This pattern or gesture is then assigned to a spoken phrase like “come here” or “stop it”. A connected smartphone app speaks the phrase aloud whenever the user makes the gesture.

The team worked closely with people who had limited speech in the course of the development. One notable participant was Emma Elko, who has a cortical visual impairment. The system successfully learned her personal gestures, allowing her to communicate without the help of her mother — her primary communication partner.

The researchers say the next step is to test the prototype with more people to refine its ability to distinguish between similar gestures and ignore involuntary movements. They also plan to add cameras to the existing sensor, for a better level of accuracy.