Training RBMs based on the signs of the CD approximation of the log-likelihood derivatives

Asja Fischer; Christian Igel

Training RBMs based on the signs of the CD approximation of the log-likelihood derivatives

Department of Computer Science

1 Citation (Scopus)

Abstract

Contrastive Divergence (CD) learning is frequently applied to Restricted Boltzmann Machines (RBMs), the building blocks of deep believe networks. It relies on biased approximations of the log-likelihood gradient. This bias can deteriorate the learning process. It was claimed that the signs of most components of the CD update are equal to the corresponding signs of the log-likelihood gradient. This suggests using optimization techniques only depending on the signs. Resilient backpropagation is such a method and we combine it with CD learning. However, it does not prevent divergence caused by the approximation bias.

Original language	English
Title of host publication	19th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN 2011)
Editors	M. Verleysen
Number of pages	6
Publisher	ESANN
Publication date	2011
Pages	495-500
ISBN (Print)	978-2-87419-044-5
ISBN (Electronic)	978-2-87419-057-5
Publication status	Published - 2011
Event	19th European Symposium On Artificial Neural Networks, Computational Intelligence and Machine Learning - Bruges, Belgium Duration: 27 Apr 2011 → 27 Apr 2011 Conference number: 19

Conference

Conference	19th European Symposium On Artificial Neural Networks, Computational Intelligence and Machine Learning
Number	19
Country/Territory	Belgium
City	Bruges
Period	27/04/2011 → 27/04/2011

Access to Document

Training RBMs Based on the Signs of the CD Approximation of the Log-likelihood DerivativesFinal published version, 0.98 MB

https://www.elen.ucl.ac.be/Proceedings/esann/esannpdf/es2011-79.pdf

Cite this

Training RBMs based on the signs of the CD approximation of the log-likelihood derivatives. / Fischer, Asja; Igel, Christian.

19th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN 2011). ed. / M. Verleysen. ESANN , 2011. p. 495-500.

Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review

Fischer, A & Igel, C 2011, Training RBMs based on the signs of the CD approximation of the log-likelihood derivatives. in M Verleysen (ed.), 19th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN 2011). ESANN , pp. 495-500, 19th European Symposium On Artificial Neural Networks, Computational Intelligence and Machine Learning, Bruges, Belgium, 27/04/2011. <https://www.elen.ucl.ac.be/Proceedings/esann/esannpdf/es2011-79.pdf>

@inproceedings{e737025b121e44969e36ffbcc5469e82,

title = "Training RBMs based on the signs of the CD approximation of the log-likelihood derivatives",

abstract = "Contrastive Divergence (CD) learning is frequently applied to Restricted Boltzmann Machines (RBMs), the building blocks of deep believe networks. It relies on biased approximations of the log-likelihood gradient. This bias can deteriorate the learning process. It was claimed that the signs of most components of the CD update are equal to the corresponding signs of the log-likelihood gradient. This suggests using optimization techniques only depending on the signs. Resilient backpropagation is such a method and we combine it with CD learning. However, it does not prevent divergence caused by the approximation bias.",

author = "Asja Fischer and Christian Igel",

year = "2011",

language = "English",

isbn = "978-2-87419-044-5 ",

pages = "495--500",

editor = "M. Verleysen",

booktitle = "19th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN 2011)",

publisher = "ESANN ",

note = "19th European Symposium On Artificial Neural Networks, Computational Intelligence and Machine Learning, ESANN 2011 ; Conference date: 27-04-2011 Through 27-04-2011",

}

TY - GEN

T1 - Training RBMs based on the signs of the CD approximation of the log-likelihood derivatives

AU - Fischer, Asja

AU - Igel, Christian

N1 - Conference code: 19

PY - 2011

Y1 - 2011

N2 - Contrastive Divergence (CD) learning is frequently applied to Restricted Boltzmann Machines (RBMs), the building blocks of deep believe networks. It relies on biased approximations of the log-likelihood gradient. This bias can deteriorate the learning process. It was claimed that the signs of most components of the CD update are equal to the corresponding signs of the log-likelihood gradient. This suggests using optimization techniques only depending on the signs. Resilient backpropagation is such a method and we combine it with CD learning. However, it does not prevent divergence caused by the approximation bias.

AB - Contrastive Divergence (CD) learning is frequently applied to Restricted Boltzmann Machines (RBMs), the building blocks of deep believe networks. It relies on biased approximations of the log-likelihood gradient. This bias can deteriorate the learning process. It was claimed that the signs of most components of the CD update are equal to the corresponding signs of the log-likelihood gradient. This suggests using optimization techniques only depending on the signs. Resilient backpropagation is such a method and we combine it with CD learning. However, it does not prevent divergence caused by the approximation bias.

M3 - Article in proceedings

SN - 978-2-87419-044-5

SP - 495

EP - 500

BT - 19th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN 2011)

A2 - Verleysen, M.

PB - ESANN

T2 - 19th European Symposium On Artificial Neural Networks, Computational Intelligence and Machine Learning

Y2 - 27 April 2011 through 27 April 2011

ER -

Training RBMs based on the signs of the CD approximation of the log-likelihood derivatives

Abstract

Conference

Access to Document

Fingerprint

Cite this