Training RBMs based on the signs of the CD approximation of the log-likelihood derivatives

Asja Fischer; Christian Igel

Training RBMs based on the signs of the CD approximation of the log-likelihood derivatives

Datalogisk Institut

1 Citationer (Scopus)

Abstract

Contrastive Divergence (CD) learning is frequently applied to Restricted Boltzmann Machines (RBMs), the building blocks of deep believe networks. It relies on biased approximations of the log-likelihood gradient. This bias can deteriorate the learning process. It was claimed that the signs of most components of the CD update are equal to the corresponding signs of the log-likelihood gradient. This suggests using optimization techniques only depending on the signs. Resilient backpropagation is such a method and we combine it with CD learning. However, it does not prevent divergence caused by the approximation bias.

Originalsprog	Engelsk
Titel	19th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN 2011)
Redaktører	M. Verleysen
Antal sider	6
Forlag	ESANN
Publikationsdato	2011
Sider	495-500
ISBN (Trykt)	978-2-87419-044-5
ISBN (Elektronisk)	978-2-87419-057-5
Status	Udgivet - 2011
Begivenhed	19th European Symposium On Artificial Neural Networks, Computational Intelligence and Machine Learning - Bruges, Belgien Varighed: 27 apr. 2011 → 27 apr. 2011 Konferencens nummer: 19

Konference

Konference	19th European Symposium On Artificial Neural Networks, Computational Intelligence and Machine Learning
Nummer	19
Land/Område	Belgien
By	Bruges
Periode	27/04/2011 → 27/04/2011

Adgang til dokumentet

Training RBMs Based on the Signs of the CD Approximation of the Log-likelihood DerivativesForlagets udgivne version, 0,98 MB

https://www.elen.ucl.ac.be/Proceedings/esann/esannpdf/es2011-79.pdf

Citationsformater

Training RBMs based on the signs of the CD approximation of the log-likelihood derivatives. / Fischer, Asja; Igel, Christian.

19th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN 2011). red. / M. Verleysen. ESANN , 2011. s. 495-500.

Publikation: Bidrag til bog/antologi/rapport › Konferencebidrag i proceedings › Forskning › peer review

Fischer, A & Igel, C 2011, Training RBMs based on the signs of the CD approximation of the log-likelihood derivatives. i M Verleysen (red.), 19th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN 2011). ESANN , s. 495-500, 19th European Symposium On Artificial Neural Networks, Computational Intelligence and Machine Learning, Bruges, Belgien, 27/04/2011. <https://www.elen.ucl.ac.be/Proceedings/esann/esannpdf/es2011-79.pdf>

@inproceedings{e737025b121e44969e36ffbcc5469e82,

title = "Training RBMs based on the signs of the CD approximation of the log-likelihood derivatives",

abstract = "Contrastive Divergence (CD) learning is frequently applied to Restricted Boltzmann Machines (RBMs), the building blocks of deep believe networks. It relies on biased approximations of the log-likelihood gradient. This bias can deteriorate the learning process. It was claimed that the signs of most components of the CD update are equal to the corresponding signs of the log-likelihood gradient. This suggests using optimization techniques only depending on the signs. Resilient backpropagation is such a method and we combine it with CD learning. However, it does not prevent divergence caused by the approximation bias.",

author = "Asja Fischer and Christian Igel",

year = "2011",

language = "English",

isbn = "978-2-87419-044-5 ",

pages = "495--500",

editor = "M. Verleysen",

booktitle = "19th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN 2011)",

publisher = "ESANN ",

note = "19th European Symposium On Artificial Neural Networks, Computational Intelligence and Machine Learning, ESANN 2011 ; Conference date: 27-04-2011 Through 27-04-2011",

}

TY - GEN

T1 - Training RBMs based on the signs of the CD approximation of the log-likelihood derivatives

AU - Fischer, Asja

AU - Igel, Christian

N1 - Conference code: 19

PY - 2011

Y1 - 2011

N2 - Contrastive Divergence (CD) learning is frequently applied to Restricted Boltzmann Machines (RBMs), the building blocks of deep believe networks. It relies on biased approximations of the log-likelihood gradient. This bias can deteriorate the learning process. It was claimed that the signs of most components of the CD update are equal to the corresponding signs of the log-likelihood gradient. This suggests using optimization techniques only depending on the signs. Resilient backpropagation is such a method and we combine it with CD learning. However, it does not prevent divergence caused by the approximation bias.

AB - Contrastive Divergence (CD) learning is frequently applied to Restricted Boltzmann Machines (RBMs), the building blocks of deep believe networks. It relies on biased approximations of the log-likelihood gradient. This bias can deteriorate the learning process. It was claimed that the signs of most components of the CD update are equal to the corresponding signs of the log-likelihood gradient. This suggests using optimization techniques only depending on the signs. Resilient backpropagation is such a method and we combine it with CD learning. However, it does not prevent divergence caused by the approximation bias.

M3 - Article in proceedings

SN - 978-2-87419-044-5

SP - 495

EP - 500

BT - 19th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN 2011)

A2 - Verleysen, M.

PB - ESANN

T2 - 19th European Symposium On Artificial Neural Networks, Computational Intelligence and Machine Learning

Y2 - 27 April 2011 through 27 April 2011

ER -

Training RBMs based on the signs of the CD approximation of the log-likelihood derivatives

Abstract

Konference

Adgang til dokumentet

Fingeraftryk

Citationsformater