Advance marks critical step toward brain-computer interfaces that hold immense promise for those with limited or no ability to speak.
In a scientific first, Columbia neuroengineers have created a system that translates thought into intelligible, recognizable speech. By monitoring someone’s brain activity, the technology can reconstruct the words a person hears with unprecedented clarity. This breakthrough, which harnesses the power of speech synthesizers and artificial intelligence, could lead to new ways for computers to communicate directly with the brain. It also lays the groundwork for helping people who cannot speak, such as those living with amyotrophic lateral sclerosis (ALS) or recovering from stroke, regain their ability to communicate with the outside world.
These findings were published today in Scientific Reports.
“Our voices help connect us to our friends, family and the world around us, which is why losing the power of one’s voice due to injury or disease is so devastating,” said Nima Mesgarani, PhD, the paper’s senior author and a principal investigator at Columbia University’s Mortimer B. Zuckerman Mind Brain Behavior Institute. “With today’s study, we have a potential way to restore that power. We’ve shown that, with the right technology, these people’s thoughts could be decoded and understood by any listener.”
This would be a game changer. It would give anyone who has lost their ability to speak, whether through injury or disease, the renewed chance to connect to the world around them.
Decades of research has shown that when people speak — or even imagine speaking — telltale patterns of activity appear in their brain. Distinct (but recognizable) pattern of signals also emerge when we listen to someone speak, or imagine listening. Experts, trying to record and decode these patterns, see a future in which thoughts need not remain hidden inside the brain — but instead could be translated into verbal speech at will.
But accomplishing this feat has proven challenging. Early efforts to decode brain signals by Dr. Mesgarani and others focused on simple computer models that analyzed spectrograms, which are visual representations of sound frequencies.
But because this approach has failed to produce anything resembling intelligible speech, Dr. Mesgarani and his team, including the paper’s first author Hassan Akbari, turned instead to a vocoder, a computer algorithm that can synthesize speech after being trained on recordings of people talking.
“This is the same technology used by Amazon Echo and Apple Siri to give verbal responses to our questions,” said Dr. Mesgarani, who is also an associate professor of electrical engineering at Columbia Engineering.
To teach the vocoder to interpret to brain activity, Dr. Mesgarani teamed up with Ashesh Dinesh Mehta, MD, PhD, a neurosurgeon at Northwell Health Physician Partners Neuroscience Institute and co-author of today’s paper. Dr. Mehta treats epilepsy patients, some of whom must undergo regular surgeries.
“Working with Dr. Mehta, we asked epilepsy patients already undergoing brain surgery to listen to sentences spoken by different people, while we measured patterns of brain activity,” said Dr. Mesgarani. “These neural patterns trained the vocoder.”
Next, the researchers asked those same patients to listen to speakers reciting digits between 0 to 9, while recording brain signals that could then be run through the vocoder. The sound produced by the vocoder in response to those signals was analyzed and cleaned up by neural networks, a type of artificial intelligence that mimics the structure of neurons in the biological brain.
The end result was a robotic-sounding voice reciting a sequence of numbers. To test the accuracy of the recording, Dr. Mesgarani and his team tasked individuals to listen to the recording and report what they heard.
“We found that people could understand and repeat the sounds about 75% of the time, which is well above and beyond any previous attempts,” said Dr. Mesgarani. The improvement in intelligibility was especially evident when comparing the new recordings to the earlier, spectrogram-based attempts. “The sensitive vocoder and powerful neural networks represented the sounds the patients had originally listened to with surprising accuracy.”
Dr. Mesgarani and his team plan to test more complicated words and sentences next, and they want to run the same tests on brain signals emitted when a person speaks or imagines speaking. Ultimately, they hope their system could be part of an implant, similar to those worn by some epilepsy patients, that translates the wearer’s thoughts directly into words.
“In this scenario, if the wearer thinks ‘I need a glass of water,’ our system could take the brain signals generated by that thought, and turn them into synthesized, verbal speech,” said Dr. Mesgarani. “This would be a game changer. It would give anyone who has lost their ability to speak, whether through injury or disease, the renewed chance to connect to the world around them.”
The Latest on: Brain-computer interfaces
via Google News
The Latest on: Brain-computer interfaces
- Will we ever control the world with our minds?on August 14, 2019 at 6:24 pm
You might also like: Scheuermann can use this brain-computer interface to control a robotic arm with her thoughts, well enough to feed herself chocolate. Three years later she successfully flew a ...
- Neuralink guide: How wearables will play their part in Elon Musk's brain techon August 13, 2019 at 2:30 pm
At an event in San Francisco, the Neuralink team revealed it has been developing a brain-computer interface (BCI) made of thin, thread-like implants. This could one day work with (or, more ...
- Elon Musk’s Neuralink: Both an evolution and a plan for radical changeon August 13, 2019 at 4:45 am
When Elon Musk first started talking about launching a brain-computer interface company, he made a number of comments that set expectations for what that idea might entail. The company ...
- General Counsel Joins Elon Musk's Human Brain-Machine Interface Companyon August 12, 2019 at 5:18 pm
That’s the same month the San Francisco-based company announced it developed electrode “threads” that could advance human brain-computer machine interface technology. If he stays on board, Feerst ...
- How a Brain-Computer Interface Could Help Paralyzed Veterans at Homeon August 12, 2019 at 11:34 am
Over the course of a five-year study, a paralyzed man was able to regain conscious control of his hand, wrist, and fingers by using a brain-computer interface technology from Battelle called NeuroLife ...
- Brain Computer Interface Devices - Growing Popularity and Emerging Trends in the Market: NeuroPace, MindMotion, Emotiv Systemson August 12, 2019 at 6:03 am
Edison, NJ -- (SBWIRE) -- 08/12/2019 -- Global Brain Computer Interface Devices Market Report 2019 is latest research study released by HTF MI evaluating the market, highlighting opportunities, risk ...
- Wireless Brain Sensors Market Industry Review, Key Players Profile, Statistics and Growth to 2027on August 9, 2019 at 11:56 am
Significant technological advances in brain-computer interfaces have been leading to continuous evolution of brain sensors. The sensors used in currently are remotely accessible through wireless ...
- Tricky interfaces: Brain-computer interfaces are still a long way offon August 9, 2019 at 4:52 am
It will still be a long time before brain-computer interfaces are able to read thoughts. And many limitations need to be considered, according to Roger Gassert. Brain-computer interfaces (BCIs ...
- Comparison of Two-Talker Attention Decoding from EEG with Nonlinear Neural Networks and Linear Methodson August 8, 2019 at 2:22 am
Auditory attention decoding (AAD) through a brain-computer interface has had a flowering of developments since it was first introduced by Mesgarani and Chang (2012) using electrocorticograph ...
- After Elon Musk, Facebook now investing in wearable mind-reading machineson August 6, 2019 at 6:19 am
At its annual F8 summit in 2017, the social media giant had announced its intentions of creating a Brain-Computer Interface (BCI) and transforming that into a wearable tech that would let people write ...
via Bing News