Neuroscientists at the University of Toronto Scarborough have for the first time used brain activity gathered by EEG to reconstruct images of what people perceive.
The technique, developed by Dan Nemrodov, a postdoctoral fellow in the lab of Assistant Professor Adrian Nestor, digitally reconstructs images seen by test subjects based on electroencephalography (EEG) data.
When we see something, our brain creates a mental percept, which is essentially a mental impression of that thing. We were able to capture this percept using EEG to get a direct illustration of what’s happening in the brain during this process.
For the study, images of faces were shown to test subjects hooked up to EEG equipment. Their recorded brain activity was then used to digitally recreate the image in the subject’s mind using machine learning algorithms.
It’s not the first time researchers have used neuroimaging techniques to reconstruct images from visual stimuli. The current method was pioneered by Nestor who successfully reconstructed facial images from functional magnetic resonance imaging (fMRI) data. This is, however, the first time this has been done using EEG.
And while techniques like fMRI — which measures brain activity by detecting changes in blood flow — can grab finer details of what’s going on in specific areas of the brain, EEG has greater practical potential — it’s more common, portable, and inexpensive. EEG also has greater temporal resolution, which means it can measure with detail how a percept develops in time.
fMRI captures activity at the time scale of seconds, but EEG captures activity at the millisecond scale. So we can see with very fine detail how the percept of a face develops in our brain using EEG.
In fact, the researchers estimate that it takes our brain about 170 milliseconds (0.17 seconds) to form a good representation of a face we see. The study shows that EEG has potential for this type of image reconstruction notes Nemrodov, something many researchers doubted.
Using EEG data for image reconstruction has great theoretical and practical potential from a neurotechnological standpoint, given its advantages. Work is now underway in Nestor’s lab to test how image reconstruction based on EEG data could be done using memory and applied to objects other than faces. This could eventually have wide-ranging clinical applications.
It could provide a means of communication for people who are unable to verbally communicate. Not only could it produce a neural-based reconstruction of what a person is perceiving, but also of what they remember and imagine, of what they want to express.
It could also have forensic uses for law enforcement in gathering eyewitness information on potential suspects rather than relying on verbal descriptions provided to a sketch artist.
What’s really exciting is that we’re not reconstructing squares and triangles but actual images of a person’s face, and that involves a lot of fine-grained visual detail,’ adds Nestor.
The fact we can reconstruct what someone experiences visually based on their brain activity opens up a lot of possibilities. It unveils the subjective content of our mind and it provides a way to access, explore and share the content of our perception, memory and imagination.