TY - JOUR
T1 - Extending information maximization from a rate-distortion perspective
AU - Zhang, Yan
AU - Hu, Junjie
AU - Okatani, Takayuki
N1 - Funding Information:
This work was partly supported by CREST , JST Grant Number JPMJCR14D1 and KAKENHI 19H01110.
Publisher Copyright:
© 2020
PY - 2020/7/25
Y1 - 2020/7/25
N2 - In this paper, we propose a new interpretation of the information maximization method (InfoMax) from a perspective of the rate distortion theory. We show that under specific conditions, InfoMax is equivalent to the minimization of a compression rate under the constraint of zero distortion. Zero distortion, or equivalently, zero reconstruction error between the input and its reconstruction, does not provide meaningful solutions in many cases. Based on the new interpretation, we extend InfoMax to be able to deal with non-zero distortion and also to learn under/over-complete representations. Experimental results on synthetic as well as real data show the effectiveness of our method.
AB - In this paper, we propose a new interpretation of the information maximization method (InfoMax) from a perspective of the rate distortion theory. We show that under specific conditions, InfoMax is equivalent to the minimization of a compression rate under the constraint of zero distortion. Zero distortion, or equivalently, zero reconstruction error between the input and its reconstruction, does not provide meaningful solutions in many cases. Based on the new interpretation, we extend InfoMax to be able to deal with non-zero distortion and also to learn under/over-complete representations. Experimental results on synthetic as well as real data show the effectiveness of our method.
KW - Information maximization
KW - Rate distortion
KW - Unsupervised representation learning
UR - http://www.scopus.com/inward/record.url?scp=85081206654&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85081206654&partnerID=8YFLogxK
U2 - 10.1016/j.neucom.2020.02.061
DO - 10.1016/j.neucom.2020.02.061
M3 - Article
AN - SCOPUS:85081206654
VL - 399
SP - 285
EP - 295
JO - Neurocomputing
JF - Neurocomputing
SN - 0925-2312
ER -