TY - GEN
T1 - Super resolution for multi frames with 3D feature extraction and RNN prediction
AU - Huang, Xi
AU - Sugaya, Yoshihiro
AU - Miyazaki, Tomo
AU - Omachi, Shinichiro
N1 - Publisher Copyright:
© 2019 Association for Computing Machinery.
PY - 2019/9/20
Y1 - 2019/9/20
N2 - Super-resolution research has received extensive attention in recent years. With the rise and maturity of deep learning technology, learning-based super-resolution (SR) method has gradually become the mainstream. Its main idea is to directly learn the mapping between low-resolution images and highresolution images by constructing a corresponding neural network. Different from the SR method only for a single image, the study of video SR is developing at the same time. For videos, in addition to considering the spatial characteristics of each frame, the variation of between frames in the time dimension is also a vital element affecting the video SR performance. However, the use of inter-frame motion characteristics remains a problem. Due to the characteristic limitations of the convolutional neural networks (CNN), it is not an efficient way to model the dependency in time dimension. There are also some SR methods using the optical flow to help with the modeling, but they still struggle with a problem of coping with complex large-scale motions. In order to deal with the above problems, we propose a network utilizing 3D convolution and recurrent neural network (RNN). In the proposed method, we use 3D convolution to extract the intra-frame and inter-frame features, then combine it with our original RNN to model the long-term dependence and short-term dependence in the video sequence. Furthermore, for reducing the huge amount of computational cost caused by the 3D convolution, we used the structure of the Laplacian pyramid to balance the calculation of the network. Finally, we conducted an experimental evaluation of our model and confirmed its effectiveness.
AB - Super-resolution research has received extensive attention in recent years. With the rise and maturity of deep learning technology, learning-based super-resolution (SR) method has gradually become the mainstream. Its main idea is to directly learn the mapping between low-resolution images and highresolution images by constructing a corresponding neural network. Different from the SR method only for a single image, the study of video SR is developing at the same time. For videos, in addition to considering the spatial characteristics of each frame, the variation of between frames in the time dimension is also a vital element affecting the video SR performance. However, the use of inter-frame motion characteristics remains a problem. Due to the characteristic limitations of the convolutional neural networks (CNN), it is not an efficient way to model the dependency in time dimension. There are also some SR methods using the optical flow to help with the modeling, but they still struggle with a problem of coping with complex large-scale motions. In order to deal with the above problems, we propose a network utilizing 3D convolution and recurrent neural network (RNN). In the proposed method, we use 3D convolution to extract the intra-frame and inter-frame features, then combine it with our original RNN to model the long-term dependence and short-term dependence in the video sequence. Furthermore, for reducing the huge amount of computational cost caused by the 3D convolution, we used the structure of the Laplacian pyramid to balance the calculation of the network. Finally, we conducted an experimental evaluation of our model and confirmed its effectiveness.
KW - 3D convolution
KW - Multi-frame SR
KW - Recurrent neural network
KW - Super resolution
UR - http://www.scopus.com/inward/record.url?scp=85077179692&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85077179692&partnerID=8YFLogxK
U2 - 10.1145/3364908.3364909
DO - 10.1145/3364908.3364909
M3 - Conference contribution
AN - SCOPUS:85077179692
T3 - ACM International Conference Proceeding Series
SP - 82
EP - 86
BT - SSPS 2019 - 2019 International Symposium on Signal Processing Systems
PB - Association for Computing Machinery
T2 - 2019 International Symposium on Signal Processing Systems, SSPS 2019
Y2 - 20 September 2019 through 22 September 2019
ER -