Super resolution for multi frames with 3D feature extraction and RNN prediction

Xi Huang, Yoshihiro Sugaya, Tomo Miyazaki, Shinichiro Omachi

Research output: Chapter in Book/Report/Conference proceedingConference contribution


Super-resolution research has received extensive attention in recent years. With the rise and maturity of deep learning technology, learning-based super-resolution (SR) method has gradually become the mainstream. Its main idea is to directly learn the mapping between low-resolution images and highresolution images by constructing a corresponding neural network. Different from the SR method only for a single image, the study of video SR is developing at the same time. For videos, in addition to considering the spatial characteristics of each frame, the variation of between frames in the time dimension is also a vital element affecting the video SR performance. However, the use of inter-frame motion characteristics remains a problem. Due to the characteristic limitations of the convolutional neural networks (CNN), it is not an efficient way to model the dependency in time dimension. There are also some SR methods using the optical flow to help with the modeling, but they still struggle with a problem of coping with complex large-scale motions. In order to deal with the above problems, we propose a network utilizing 3D convolution and recurrent neural network (RNN). In the proposed method, we use 3D convolution to extract the intra-frame and inter-frame features, then combine it with our original RNN to model the long-term dependence and short-term dependence in the video sequence. Furthermore, for reducing the huge amount of computational cost caused by the 3D convolution, we used the structure of the Laplacian pyramid to balance the calculation of the network. Finally, we conducted an experimental evaluation of our model and confirmed its effectiveness.

Original languageEnglish
Title of host publicationSSPS 2019 - 2019 International Symposium on Signal Processing Systems
PublisherAssociation for Computing Machinery
Number of pages5
ISBN (Electronic)9781450362412
Publication statusPublished - 2019 Sep 20
Event2019 International Symposium on Signal Processing Systems, SSPS 2019 - Beijing, China
Duration: 2019 Sep 202019 Sep 22

Publication series

NameACM International Conference Proceeding Series


Conference2019 International Symposium on Signal Processing Systems, SSPS 2019


  • 3D convolution
  • Multi-frame SR
  • Recurrent neural network
  • Super resolution

ASJC Scopus subject areas

  • Software
  • Human-Computer Interaction
  • Computer Vision and Pattern Recognition
  • Computer Networks and Communications


Dive into the research topics of 'Super resolution for multi frames with 3D feature extraction and RNN prediction'. Together they form a unique fingerprint.

Cite this