TY - JOUR
T1 - Statistical-mechanical analysis of pre-training and fine tuning in deep learning
AU - Ohzeki, Masayuki
N1 - Publisher Copyright:
©2015 The Physical Society of Japan.
Copyright:
Copyright 2015 Elsevier B.V., All rights reserved.
PY - 2015/3/15
Y1 - 2015/3/15
N2 - In this paper, we present a statistical-mechanical analysis of deep learning. We elucidate some of the essential components of deep learning - pre-training by unsupervised learning and fine tuning by supervised learning. We formulate the extraction of features from the training data as a margin criterion in a high-dimensional feature-vector space. The self-organized classifier is then supplied with small amounts of labelled data, as in deep learning. Although we employ a simple single-layer perceptron model, rather than directly analyzing a multi-layer neural network, we find a nontrivial phase transition that is dependent on the number of unlabelled data in the generalization error of the resultant classifier. In this sense, we evaluate the efficacy of the unsupervised learning component of deep learning. The analysis is performed by the replica method, which is a sophisticated tool in statistical mechanics. We validate our result in the manner of deep learning, using a simple iterative algorithm to learn the weight vector on the basis of belief propagation.
AB - In this paper, we present a statistical-mechanical analysis of deep learning. We elucidate some of the essential components of deep learning - pre-training by unsupervised learning and fine tuning by supervised learning. We formulate the extraction of features from the training data as a margin criterion in a high-dimensional feature-vector space. The self-organized classifier is then supplied with small amounts of labelled data, as in deep learning. Although we employ a simple single-layer perceptron model, rather than directly analyzing a multi-layer neural network, we find a nontrivial phase transition that is dependent on the number of unlabelled data in the generalization error of the resultant classifier. In this sense, we evaluate the efficacy of the unsupervised learning component of deep learning. The analysis is performed by the replica method, which is a sophisticated tool in statistical mechanics. We validate our result in the manner of deep learning, using a simple iterative algorithm to learn the weight vector on the basis of belief propagation.
UR - http://www.scopus.com/inward/record.url?scp=84925003591&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84925003591&partnerID=8YFLogxK
U2 - 10.7566/JPSJ.84.034003
DO - 10.7566/JPSJ.84.034003
M3 - Article
AN - SCOPUS:84925003591
VL - 84
JO - Journal of the Physical Society of Japan
JF - Journal of the Physical Society of Japan
SN - 0031-9015
IS - 3
M1 - 034003
ER -