Scientists create first ‘brain-to-text’ system: System combines cortical information with linguistic knowledge and machine-learning algorithms to extract most obvious word sequence.

Speech is produced in the cerebral cortex of the human brain. The brain waves associated with this phenomenon can be recorded directly by connecting electrodes to the surface of the cortex.

Researchers at KIT and Wadsworth Center, US have, for the first time, demonstrated the possibility of reconstructing the basic units of continuous speech from these brain waves – generating complete sentences and corresponding texts. The study was published in the scientific journal Frontiers in Neuroscience.

“It has long been speculated whether humans may communicate with machines via brain activity alone,” explained Tanja Schultz, who conducted the research along with her team at the Cognitive Systems Lab of KIT.

“As a major step in this direction, our recent results indicate that both single units in terms of speech sounds as well as continuously spoken sentences can be recognized from brain activity.”

The study was a collaboration between researchers from informatics, neuroscience, and medicine. The brain activity of seven epileptic patients was recorded, who voluntarily participated in the study. Apart from their routine clinical treatment, neurological treatment was administered by placing an electrode array on the surface of the cerebral cortex (electrocorticography (ECoG)).

As the participants read samples of texts out loud, their ECoG signals were recorded using high resolution in time and space. This data was then analysed to create ‘Brain-to-Text’ system. In Karlsruhe, techniques for signal processing and automatic speech detection have been developed and put into practical application.

“In addition to the decoding of speech from brain activity, our models allow for a detailed analysis of the brain areas involved in speech processes and their interaction,” explained Christian Herff and Dominic Heger, the researchers behind the Brain-to-Text system.

The system created is the first to decode continuous speech and represent it in a textual context. It combines cortical information with linguistic knowledge and machine-learning algorithms – precisely extracting the most obvious word sequence. The results obtained are a preliminary step towards understanding the extremely complex speech processes that occur in the brain.

At present, the system converts speech into text. In the future, ‘brain-to-text’ system might allow speech recognition from mere thoughts as well. Moreover, it may be a building block for developing a system of speech communication for patients suffering from locked-in syndrome.