Deep Learning for Natural Image Reconstruction from Electrocorticography Signals

Hiroto Date, Keisuke Kawasaki, Isao Hasegawa, Takayuki Okatani

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Several recent studies proposed various methods for reconstructing natural images from human functional magnetic resonance imaging (fMRI) data. However, few studies have proposed reconstruction methods for electrophysiolgical brain activities such as electroencephalography (EEG) and electrocorticography (ECoG). To investigate whether natural images can be reconstructed from electrophysiological brain activities, we conducted a large-scale experiment on natural image reconstruction from ECoG signals using deep learning. We first recorded ECoG signals from two macaque monkeys while presenting diverse natural images. Then, we trained several deep learning models for reconstructing presented images from ECoG signals. Comparing reconstruction models, we find that models trained with an adversarial loss produced reconstructions that contain visible features in presented images. Furthermore, our results with downsampled ECoG signals show the importance of rich temporal dynamics in ECoG signals for image reconstruction. Our results indicate the possibility of reconstructing diverse natural images from electrophysiological brain activities using deep learning.

Original languageEnglish
Title of host publicationProceedings - 2019 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2019
EditorsIllhoi Yoo, Jinbo Bi, Xiaohua Tony Hu
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages2331-2336
Number of pages6
ISBN (Electronic)9781728118673
DOIs
Publication statusPublished - 2019 Nov
Event2019 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2019 - San Diego, United States
Duration: 2019 Nov 182019 Nov 21

Publication series

NameProceedings - 2019 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2019

Conference

Conference2019 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2019
CountryUnited States
CitySan Diego
Period19/11/1819/11/21

Keywords

  • Brain modeling
  • computer vision
  • deep learning
  • image reconstruction

ASJC Scopus subject areas

  • Biochemistry
  • Biotechnology
  • Molecular Medicine
  • Modelling and Simulation
  • Health Informatics
  • Pharmacology (medical)
  • Public Health, Environmental and Occupational Health

Fingerprint Dive into the research topics of 'Deep Learning for Natural Image Reconstruction from Electrocorticography Signals'. Together they form a unique fingerprint.

Cite this