Constructing artificial systems that see and recognize the world similarly to human visual systems is a key goal of computer vision. Recent advancements in population brain activity measurement, together with improvements within the implementation and design of deep neural network models, have made it possible to directly compare the architectural features of artificial networks to those of biological brains’ latent representations, revealing crucial details about how these systems work. Reconstructing visual images from brain activity, corresponding to that detected by functional magnetic resonance imaging (fMRI), is certainly one of these applications. That is a captivating but difficult problem since the underlying brain representations are largely unknown, and the sample size typically used for brain data is small.

Deep-learning models and techniques, corresponding to generative adversarial networks (GANs) and self-supervised learning, have recently been utilized by academics to tackle this challenge. These investigations, nevertheless, call for either fine-tuning toward the actual stimuli utilized within the fMRI experiment or training latest generative models with fMRI data from scratch. These attempts have demonstrated great but constrained performance by way of pixel-wise and semantic fidelity, partially resulting from the small amount of neuroscience data and partially resulting from the multiple difficulties related to constructing complicated generative models.

Diffusion Models, particularly the less computationally resource-intensive Latent Diffusion Models, are a recent GAN substitute. Yet, as LDMs are still relatively latest, it’s difficult to have an entire understanding of how they work internally.

Through the use of an LDM called Stable Diffusion to reconstruct visual images from fMRI signals, a research team from Osaka University and CiNet attempted to deal with the problems mentioned above. They proposed an easy framework that may reconstruct high-resolution images with high semantic fidelity without the necessity for complex deep-learning models to be trained or tuned.

The dataset employed by the authors for this investigation is the Natural Scenes Dataset (NSD), which offers data collected from an fMRI scanner across 30–40 sessions during which each subject viewed three repeats of 10,000 images. 

To start, they used a Latent Diffusion Model to create images from text. Within the figure above (top), z is defined because the generated latent representation of z that has been modified by the model with c, c is defined because the latent representation of texts (that describe the photographs), and zc is defined because the latent representation of the unique image that has been compressed by the autoencoder.

To investigate the decoding model, the authors followed three steps (figure above, middle). Firstly, they predicted a latent representation z of the presented image X from fMRI signals inside the early visual cortex (blue). z was then processed by a decoder to supply a rough decoded image Xz, which was then encoded and passed through the diffusion process. Finally, the noisy image was added to a decoded latent text representation c from fMRI signals inside the higher visual cortex (yellow) and denoised to supply zc. From, zc a decoding module produced a final reconstructed image Xzc. It’s necessary to underline that the one training required for this process is to linearly map fMRI signals to LDM components, zc, z and c.

Ranging from zc, z and c the authors performed an encoding evaluation to interpret the inner operations of LDMs by mapping them to brain activity (figure above, bottom). The outcomes of reconstructing images from representations are shown below.

Images that were recreated using simply z had a visible consistency with the unique images, but their semantic value was lost. Alternatively, images that were only partially reconstructed using c yielded images that had great semantic fidelity but inconsistent visuals. The validity of this method was demonstrated by the flexibility of images recovered using zc to supply high-resolution images with great semantic fidelity.

The ultimate evaluation of the brain reveals latest details about DM models. In the back of the brain, the visual cortex, all three components achieved great prediction performance. Particularly, z provided strong prediction performance within the early visual cortex, which lies at the back of the visual cortex. Also, it demonstrated strong prediction values within the upper visual cortex, which is the anterior a part of the visual cortex, but smaller values in other regions. Alternatively, within the upper visual cortex, c led to the perfect prediction performance.

Try the Paper and Project Page. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to affix our 26k+ ML SubReddit, Discord Channel, and Email Newsletter, where we share the most recent AI research news, cool AI projects, and more.