Neural Prosthesis: Using Brain Activity to Decode Speech
Author: National Research University Higher School of Economics | Contact: https://www.hse.ru/en/
Peer-Reviewed Publication: Yes
Journal Reference: DOI Link to the Study Paper
Library of Related Papers: Electronics/Software Publications
Synopsis: Speech decoding from a small set of spatially segregated minimally invasive intracranial EEG electrodes with a compact and interpretable neural network. Millions of people worldwide are affected by speech disorders limiting their communication ability. Causes of speech loss can vary and include stroke and certain congenital conditions. Speech neuroprostheses-brain-computer interfaces capable of decoding speech based on brain activity can provide an accessible and reliable solution for restoring communication to such patients.
Neuroprosthetics (neural prosthetics) is a discipline in neuroscience and biomedical engineering concerned with developing neural prostheses. They are sometimes contrasted with a brain-computer interface, which connects the brain to a computer rather than a device to replace missing biological functionality.
Researchers from HSE University and the Moscow State University of Medicine and Dentistry have developed a machine learning model that can predict the word about to be uttered by a subject based on their neural activity recorded with a small set of minimally invasive electrodes. The paper 'Speech decoding from a small group of spatially segregated minimally invasive intracranial EEG electrodes with a compact and interpretable neural network' has been published in the Journal of Neural Engineering. A grant from the Russian Government financed the research as part of the 'Science and Universities' National Project.
Millions of people worldwide are affected by speech disorders limiting their communication ability. Causes of speech loss can vary and include stroke and certain congenital conditions.
Technology is available today to restore such patients' communication function, including 'silent speech' interfaces which recognize speech by tracking the movement of articulatory muscles as the person mouths words without making a sound. However, such devices help some patients but not others, such as people with facial muscle paralysis.
Speech neuroprostheses-brain-computer interfaces capable of decoding speech based on brain activity-can provide an accessible and reliable solution for restoring communication to such patients.
Unlike personal computers, devices with a brain-computer interface (BCI) are controlled directly by the brain without needing a keyboard or a microphone.
A major barrier to the wider use of BCIs in speech prosthetics is that this technology requires highly invasive surgery to implant electrodes in the brain tissue.
The most accurate speech recognition is achieved by neuroprostheses with electrodes covering a large cortical surface area. However, these solutions for reading brain activity are not intended for long-term use and present significant risks to the patients.
Researchers of the HSE Centre for Bioelectric Interfaces and the Moscow State University of Medicine and Dentistry have studied the possibility of creating a functioning neuroprosthesis capable of decoding speech with acceptable accuracy by reading brain activity from a small set of electrodes implanted in a limited cortical area. The authors suggest that in the future, this minimally invasive procedure could even be performed under local anesthesia. In the present study, the researchers collected data from two patients with epilepsy who had already been implanted with intracranial electrodes for presurgical mapping to localize seizure onset zones.
The first patient was implanted bilaterally with five sEEG shafts with six contacts in each. The second patient was implanted with nine electrocorticographic (ECoG) strips with eight contacts. Unlike ECoG, electrodes for sEEG can be implanted without a full craniotomy via a drill hole in the skull. In this study, only the six contacts of a single sEEG shaft in one patient and the eight contacts of one ECoG strip in the other were used to decode neural activity.
The subjects were asked to read aloud six sentences, each presented 30 to 60 times in a randomized order. The sentences varied in structure, and most words within a single sentence started with the same letter. The sentences contained a total of 26 different words. As the subjects were reading, the electrodes registered their brain activity.
This data was then aligned with the audio signals to form 27 classes, including 26 words and one silence class. The resulting training dataset (containing signals recorded in the first 40 minutes of the experiment) was fed into a machine-learning model with a neural network-based architecture. The learning task for the neural network was to predict the next uttered word (class) based on the neural activity data preceding its utterance.
In designing the neural network's architecture, the researchers wanted to make it simple, compact, and easily interpretable. They devised a two-stage architecture that first extracted inner speech representations from the recorded brain activity data, producing log-mel spectral coefficients. Then they predicted a specific class, i.e., a word or silence.
Thus trained, the neural network achieved 55% accuracy using only six channels of data recorded by a single sEEG electrode in the first patient and 70% accuracy using only eight channels of data recorded by a single ECoG strip in the second patient. Such accuracy is comparable to that demonstrated in other studies using devices that required electrodes to be implanted over the entire cortical surface.
The resulting interpretable model makes it possible to explain in neurophysiological terms which neural information contributes most to predicting a word about to be uttered. The researchers examined signals from different neuronal populations to determine which were pivotal for the downstream task. Their findings were consistent with the speech mapping results, suggesting that the model uses neural signals which are pivotal and can therefore be used to decode imaginary speech.
Another advantage of this solution is that it does not require manual feature engineering. The model has learned to extract speech representations directly from the brain activity data. The interpretability of results also indicates that the network decodes signals from the brain rather than from any concomitant activity, such as electrical signals from the articulatory muscles arising due to a microphone effect.
The researchers emphasize that the prediction was always based on the neural activity data preceding the utterance. This, they argue, makes sure that the decision rule did not use the auditory cortex's response to speech already uttered.
"Using such interfaces involves minimal risks for the patient. If everything works out, it could be possible to decode imaginary speech from neural activity recorded by a small number of minimally invasive electrodes implanted in an outpatient setting with local anesthesia" - Alexey Ossadtchi, leading author of the study, director of the Centre for Bioelectric Interfaces of the HSE Institute for Cognitive Neuroscience.
Resources That Provide Relevant Information
- Brain-Computer Interface Augmented With Virtual Walking Avatar Can Control Gait
- Findings from SCI Brain Computer Interface Study
- Shared Control Brain Computer Interfaces
- Revolutionary EEG Brain Computer Interface
- Brain Interface Device Helps Stroke Patients Retrain Brains to Move Paralyzed Hands
- Man with Quadriplegia Uses Brain Interface Technology to Move Again
- Brain Computer Interfaces for People with Spinal Cord Injuries
- Brain to Machine Interface Gets People with Paralysis Back on Their Feet
This peer reviewed article relating to our electronics/software section was selected for publishing by the editors of Disabled World due to its likely interest to our disability community readers. Though the content may have been edited for style, clarity, or length, the article "Neural Prosthesis: Using Brain Activity to Decode Speech" was originally written by National Research University Higher School of Economics. Should you require further information or clarification, they can be contacted at https://www.hse.ru/en/ Disabled World makes no warranties or representations in connection therewith.
Disabled World is an independent disability community established in 2004 to provide disability news and information to people with disabilities, seniors, their family and/or carers. See our homepage for informative news, reviews, sports, stories and how-tos. You can also connect with us on Twitter and Facebook or learn more on our about us page.
Disabled World provides general information only. The materials presented are never meant to substitute for professional medical care by a qualified practitioner, nor should they be construed as such. Financial support is derived from advertisements or referral programs, where indicated. Any 3rd party offering or advertising does not constitute an endorsement.
• Cite This Page (APA): National Research University Higher School of Economics. (2023, January 19). Neural Prosthesis: Using Brain Activity to Decode Speech. Disabled World. Retrieved June 2, 2023 from www.disabled-world.com/assistivedevices/computer/neural-prosthesis.php
• Permalink: <a href="https://www.disabled-world.com/assistivedevices/computer/neural-prosthesis.php">Neural Prosthesis: Using Brain Activity to Decode Speech</a>