MindEye2: Shared-Subject Models Enable fMRI-to-Image with 1 Hour of Data

MindEye2: Shared-Subject Models Enable fMRI-to-Image with 1 Hour of Data(arxiv.org)

77 points by xkgt 2 years ago | 30 comments

thorum 2 years ago |

From the article,

> The present work demonstrates that it is now practical for patients to undergo a single MRI scanning session and produce enough data to perform high-quality reconstructions of their visual perception.

> Such image reconstructions from brain activity are expected to be systematically distorted due to factors including mental state, neurological conditions, etc.

> This could potentially enable novel clinical diagnosis and assessment approaches, including applications for improved locked-in (pseudocoma) patient communication (Monti et al., 2010) and brain-computer interfaces if adapted to real-time analysis (Wallace et al., 2022) or non-fMRI neu-roimaging modalities.

> As technology continues to improve, we note it is important that brain data be carefully protected and companies collecting such data be transparent with their use.

flurb 2 years ago |

While decoding dreams, as some here suggested, would certainly be an intriguing venue of research, I believe, decoding the minds eye of people who lack the minds eye would be an even more interesting topic. How accurate are their "view", for a lack of a better word? Would there even be an image to capture?

kordlessagain 2 years ago | |

It's an interesting set of questions to explore. Having thought about this for many years now (since Blake Ross' post on Aphantasia) my hypothesis is that there wouldn't be any decoding because there is no attention on a visual based internal mind object. As a comparative, if we could decode audio from a person's mind and then you took someone who didn't do echoic recall, then you would expect to "hear" static. The percentage of the population who doesn't do echoic recall is, in some references, estimated to be about 60%.

monkpit 2 years ago |

The fact that this is possible is so bizarre to me. Impressive work.

wiz21c 2 years ago |

What about doing the opposite: get an image and stimulate the brain to perceive it ?

rl3 2 years ago | |

For that you’ll need a first class airline ticket with Tom Hardy and Leo.

GaggiX 2 years ago |

Looking at the result (figure 4), the model looks more like a good classifier than an image retriever, the generated images have the same subject as the original but are very different, even the colors of the subjects.

teaearlgraycold 2 years ago | |

Exactly. The whole image generation bit, at this stage, is noise. But you could easily describe what someone is looking at. Even that is very interesting. And perhaps this will scale up to get high quality, accurate, outputs.

srslyskptcl 2 years ago |

Why does anyone think this type of scientific inquiry should be pursued at all.

extheat 2 years ago | |

Same reason we study and do research in biology and fields in general. Understanding more about biology, whether in neuroscience or elsewhere, is the fundamental backbone of medicine. Do you want to live in a world where we don't pursue scientific research into things because they seem scary to you? Does it not concern you us building AI smarter than humans, while knowing little to nothing about our own brains? Before we start fading in relevance, we ought to understand how our own bodies (and minds) work! So in fact I think the exact opposite. We should be doing much more work in understanding our own biology (as long as it's ethical).

thorum 2 years ago | | |

I’d be fine living in a world where nobody bothered to invent non-consensual mind reading tech. Or, if that’s too much to ask, where we invent everything else that needs inventing first.

teaearlgraycold 2 years ago | |

I definitely believe that this technology will get misused. But I expect the researchers are thinking of helpful medical applications. Perhaps to aid people with locked-in syndrome?

Edit: More interestingly, it would be amazing if this could show us someone’s perception rather than what they’re looking at. That would be tremendously useful to therapists.

reliablereason 2 years ago | |

Cause it has huge medical implications?

ilaksh 2 years ago |

I hope someone will do a similar experiment with visualization (imagination) only. Some people are better at this than others so maybe focus on those people who easily create vivid images in their mind's eye.

Or maybe text.

dewarrn1 2 years ago |

It's remarkable that this works, and it's equally remarkable that the authors succeeded using a (deliberately) naive approach with the fMRI data. I was just teaching the visual system to students over the last few weeks: cognitive neuroscience/neuropsychology can tell us a lot about what brain systems should be involved here. It's such a tempting research domain to dive into using a more sophisticated approach, but then again, maybe the authors' simpler data-driven method is a strength.

shrubble 2 years ago |

So here is a question: what would happen if this process faced a person who was specifically hostile to it? Could you "trick" the process by intensely thinking of, for instance, a yellow ball while viewing a cat picture, as one example?

keenmaster 2 years ago | |

I assume the two would be discernible from each other in your brain data to a sufficiently advanced AI. One is your visual stream and one is your visual imagination. On top of that would be signatures of intention to deceive, which are also detectable in theory :)

jojobas 2 years ago | | |

All your brainwaves are belong to us.

rossant 2 years ago | |

From the article:

> fMRI is extremely sensitive to movement and requires subjects to comply with the task: decoding is easily resisted by slightly moving one’s head or thinking about unrelated information (Tang et al., 2023)

wiz21c 2 years ago | | |

So maybe that's why magicians are so good at illusion: as we think about what we expect to see, we don't process information of our eyes correctly...

ukuina 2 years ago |

This is insane! If capture latency is reduced, can this be used to visualize dreams?

boardwaalk 2 years ago |

As someone with damage to my visual cortex with a very specific effect I’m very curious what this would come up with for me. Not something useful on its own, but more understanding means more possibilities for treatments.

nunodonato 2 years ago | |

aphantasia?

kordlessagain 2 years ago | | |

In discussions about neurological diversity and the potential for treatments or adaptations, recognizing and respecting individual experiences while avoiding overgeneralization is key. Conversations around aphantasia, brain injuries, and visual processing offer opportunities for learning and exploration but should be navigated with care to ensure they are inclusive and sensitive to all involved.

Aphantasia is the term attributed to the inability to willfully create internal visual objects in the mind that can be perceived consciously. So, that leaves an open unconscious "will" still being able to create them. Studies, and personal experience, shows people considered to have this "condition" called Aphantasia, are able to see hypnogogic images during transition to sleep. And most of us have dreams too.

Drawing a conclusion of that term applied to someone who clearly states they have damage to a certain part of their brain and then correlating that an estimated 1-2% of the population is a bit of an insensitive thing, maybe.

Then again, I saw a lot of this while exploring the topic a few years back, so it's understandable that people who visualize think it's some sort of "problem" instead of a super power (which it is).

artninja1988 2 years ago |

Very cool work. Can something similar be done for audio?

GoblinSlayer 2 years ago | |

https://www.nytimes.com/2023/08/15/science/music-brain-pink-...

swagatkonchada 2 years ago |

very exciting tech. also very scary, if we imagine how authority can misuse this.

mt_ 2 years ago |

Impressive