Movie reconstruction from human brain activity
Why Hacker News recommends it
HN cites it as the landmark brain-decoding demo and compares newer reconstruction work against it.
What it is
Jack Gallant's lab demonstrates reconstructing movie trailer clips a subject watched from fMRI brain activity, using regression models trained on hours of viewing data. The video shows the original clip beside the reconstruction.
Recommended by 1 HN commenters across 1 different months, 2013 to 2024.
What Hacker News said
Recommends Houshalter · 2013-08 · on “Computer can read letters directly from the brain”
This is a much cooler example [this video] If I'm understanding the description correctly, they are just training it to recognize what the image is closest to and taking an slice of a youtube video that most closely matches it. I imagine if they used a more efficient method or trained it more, they could do way better. It seems like most of the data to build an accurate picture of what they are seeing is already there.
Cites runn1ng · 2013-04 · on “Reconstruction Of Visual Images Using Brain Activity Patterns”
This is one year and something old and returns better results, in my opinion. [this video] On the other hand, they presented only the best results in that video.
Cites creamyhorror · 2013-04 · on “Reconstruction Of Visual Images Using Brain Activity Patterns”
According to the abstract: Here, we present a neural decoding approach in which machine learning models predict the contents of visual imagery during the sleep onset period given measured brain activity, by discovering links between human fMRI patterns and verbal reports with the assistance of lexical and image databases. Decoding models trained on stimulus-induced brain activity in visual cortical areas showed accurate classification, detection, and identification of contents. So it sounds like the fMRI patterns are associated with words via verbal reports from humans, and the video is simply generated from image databases by using those words. In other words, they classify the fMRI pattern
Cites chch · 2014-03 · on “Yale researchers reconstruct facial images locked in a viewer’s mind”
Although it's hard to tell from the images presented with the article, the face generation looks like it could be similar to the techniques used in Nishimoto et al., 2011, which used a similar library of learned brain responses, though for movie trailers: [this video] Their particular process is described in the YouTube caption: The left clip is a segment of a Hollywood movie trailer that the subject viewed while in the magnet. The right clip shows the reconstruction of this segment from brain activity measured using fMRI. The procedure is as follows: [1] Record brain activity while the subject watches several hours of movie trailers. [2] Build dictionaries (i.e., regression models) that translat
Cites Houshalter · 2014-12 · on “Introduction to Modern Brain-Computer Interface Design”
MRI technology is improving over time, but I don't know how practical it would be in everyday usage. fNIR can potentially do the same thing much cheaper. I believe it's restricted to only the first few cm of brain tissue but that might be plenty. With fMRI you can even reconstruct entire videos of what a person is seeing (e.g. [this video] ). Controlling something simpler should be easier.
Cites zach · 2016-06 · on “VOX.com – A guy trained a machine to “watch” Blade Runner”
Extremely interesting to compare with Jack Gallant's reconstruction of video from the brain MRI of a viewer: [this video]
Cites xamuel · 2018-09 · on “Deep image reconstruction from human brain activity (2017)”
Reminds me of this project from 2011: [this video] If I were a billionaire, one of the projects I'd do would be to apply this sort of technology at scale, upon volunteers who were witness to poorly recorded historic events. WWII, Beatles concerts, JFK assassination, 9/11... all of them have been recorded from angles/perspectives that exist only in peoples' memories, and the longer we wait, the more of those memories will be lost.
Cites SmokeGS · 2019-06 · on “Evidence of a nocebo response following a nationwide antidepressant drug switch”
I feel like we are so close to wirelessly monitoring all brain waves (next 50 years) we will probably be able to, in real time, recognize and diagnose abnormal brain patterns. I think it would be promising if we get this sensor tech up and then perhaps deploy nanotech to interfere with maladaptive brain patterns. The ethics of such a device is considerably sticky. TED talk by Jack Gallant [LINK] recreating vision: [this video] another ted talk by John-Dylan Haynes [LINK] paraphrased: "wireless non-contact mind reading tech may be cheap and mass producible within the next 50 years." the only rate limiting step with his method of decoding thoughts is that it takes time under a high energy device to