Updated: fMRI Based Visual Stimulus Reconstruction

11 12 2008

A simple view of what the brain does is acquire input, process it, then produce output. One strategy for understanding what processing takes place is to record the patterns of brain activity while showing many patterns of input, then see if you can use the information gained to predict a novel input, given the pattern of brain activity. The canonical example of this approach is visual input reconstruction based on recorded spike trains in the visual system of the blowfly.

The blowfly is a relatively simple system (though quite efficient) with a tiny brain. Could a similar approach work in humans?  Although we can’t drop electrodes into the visual cortex (usually), we can put people in fMRI scanners to visualize the pattern of blood oxygenation, which is correlated with neural activity.

In Visual Image Reconstruction from Human Brain Activity using a Combination of Multiscale Local Image Decoders, Miyawaki et al demonstrate visual input prediction using fMRI responses. Using 3mm3 voxels, the group measured the activity level across early visual cortex (V1-V4) for numerous 10×10 binary patterns of visual stimuli. They looked at correlations in 1×1, 1×2, 2×1 and 2×2 bins of voxel activity to hundreds of visual test patterns. The activity represented local image elements. Then they displayed novel visual input and used a linear combination of the local image element responses to predict the visual input from the brain activity alone. It is noteworthy that they only required several hundred training images before visual input prediction was possible.

Predicted visual input from fMRI activity in V1 and V2

Predicted visual input from fMRI activity in V1 and V2

Note that a rentinotopic map, where the relative spatial position of visual input is reflected in the activity across the visual cortex, is not strictly required for this technique to work. What is required is that the response of each local element is consistent across similar patterns of input in the element’s receptive field. Furthermore, the spatial scale of pattern representation in early processing regions of human visual cortex is broad enough to be picked up by the fMRI scanner.

It would be interesting to see how much higher visual resolution could be predicted with an fMRI approach. Could this approach be adapted to predict input from the responses of cells with more complex receptive fields in higher cortical areas? Or, are those cells too intermingled with neighbors with vastly different response properties to be separable by fMRI?  Higher areas are vital for our own brains to rapidly perceive the contours of complex images. I’d also like to see how well non-contiguous images are predicted.

Cellular resolution calcium imaging with bulk loaded dyes has been used to map fine-grained detail of receptive fields in lower animals visual and somatosensory cortex. Is input prediction possible from these recordings? Is the input training set too limited? Could more complex input be perceived using a fewer number of complex cells from higher visual areas (V2 and above)?



13 responses

11 12 2008
Preview : fMRI Based Visual Stimulus Reconstruction « Brain Windows

[…] see the updated post on this […]

11 12 2008

Wow. Just wow.

11 12 2008
Coming Soon: Dream Recorders « Robot Pirate Ninja

[…] via Updated: fMRI Based Visual Stimulus Reconstruction « Brain Windows. […]

11 12 2008

OK… Got massive incoming links from metafilter. Plus Matt Drudge has just linked one of the most distorted science stories of all time from his homepage that references this work.

NO. This paper does not decode dreams.

NO. It doesn’t even come close.

It’s total speculation at this point to be able to decode dreams.

However, if V1 accurately reports the visions we see while in REM sleep, then this paper, combined with the results from Kay et al, Nature 2008 does get us ONE step closer to that.

Remember, this is very low resolution reconstruction of a single visual stimulus that is fixated on for seconds. Natural visual stimuli are much more complex. Dream images likely move rapidly. Current fMRI technology is at least an order of magnitude away from natural scene reconstruction in both temporal and spatial scales.

12 12 2008
Images reconstructed from human brain using fMRI | Think Gene

[…] Very interesting news today, as Japanese scientists published a paper in Neuron discussing the reconstruction of images directly from the human brain using fMRI.  It does seem like science is making a great deal of progress for the human brain. The future consequences of research like this may be controversial, but it’s the inevitable price of progress. Pink Tentacle covered this story along with a more technical overview by Brain Windows. […]

12 12 2008

The “dream thing” is really the fault of one of the researchers…

“By applying this technology, it may become possible to record and replay subjective images that people perceive like dreams.” The scientists, lead by chief researcher Yukiyaso Kamitani, focused on the image recognition procedures in the retina of the human eye.

It’s that “may become possible” which some people read as “When can I buy one at Wal-Mart?” which is causing the disconnect.

BTW, stick with that “post first, explain later” model, if you would. As one who runs into similar issues with composing thoughtful posts, sometimes it’s just better to get the basic information out there and then do the follow-up later. It also helps the search engines find stuff if you link the two posts (as wordpress does so well).

12 12 2008

There might be a translation issue as well…

Definitely sticking with the post first then update model for hot off the press sort of things. Broke 1000 hits/day for the first time today mostly referred from the metafilter post. More than when my thesis advisor won the Nobel Prize. Fun! I wonder what would happen if it was directly dugg rather than having come through pinktentacle/metafilter.

12 12 2008

MeFi is one of the quicker aggregation sites to find the “Best of the Web”, and a lot of people who read it run their own sites (hi!), so I’d expect a bit more traffic. Good work, keep it up.

It looks like Digg picked a different write-up to feature on the topic. I like your picture better than the canned one of people waking up.

13 12 2008
Scientists reconstruct visual image from fMRI scan - Sherdog Mixed Martial Arts Forums

[…] of US science journal Neuron. AFP: Dreams may no longer be secret with Japan computer screen A slightly more technical article. Courtesy of ****filter. __________________ "If only the fundies interpreted the […]

13 12 2008
Scientists reconstruct visual images from fMRI scans - Sherdog Mixed Martial Arts Forums

[…] of US science journal Neuron. AFP: Dreams may no longer be secret with Japan computer screen A slightly more technical article. Courtesy of m e t a filter.[/quote] __________________ "If only the fundies interpreted […]

13 12 2008
Top Posts « WordPress.com

[…] Updated: fMRI Based Visual Stimulus Reconstruction A simple view of what the brain does is acquire input, process it, then produce output. One strategy for understanding […] […]

13 12 2008

Your brain is now their open book….

Scientists claim to have extracted images directly from the brain. They trained people to read the word “neuron” with their eyes really really hard until the scientists’ local decoders decoded a visual image reconstruction of their subjects’ retino…

7 07 2009
Mike Stay

I imagine that they would need to retrain on each individual in order to get pixel-level detail.

How about for colors? Would fMRI be able to distinguish, say, a solid field of red from green from blue? Would the response be consistent enough across people to use it to determine what color a red-green colorblind person actually sees?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: