Bounding the bias of contrastive divergence learning

TY - JOUR

T1 - Bounding the bias of contrastive divergence learning

AU - Fischer, Anja

AU - Igel, Christian

PY - 2011/3

Y1 - 2011/3

N2 - Optimization based on k-step contrastive divergence (CD) has become a common way to train restricted Boltzmann machines (RBMs). The k-step CD is a biased estimator of the log-likelihood gradient relying on Gibbs sampling. We derive a new upper bound for this bias. Its magnitude depends on k, the number of variables in the RBM, and the maximum change in energy that can be produced by changing a single variable. The last reflects the dependence on the absolute values of the RBM parameters. The magnitude of the bias is also affected by the distance in variation between the modeled distribution and the starting distribution of the Gibbs chain.

AB - Optimization based on k-step contrastive divergence (CD) has become a common way to train restricted Boltzmann machines (RBMs). The k-step CD is a biased estimator of the log-likelihood gradient relying on Gibbs sampling. We derive a new upper bound for this bias. Its magnitude depends on k, the number of variables in the RBM, and the maximum change in energy that can be produced by changing a single variable. The last reflects the dependence on the absolute values of the RBM parameters. The magnitude of the bias is also affected by the distance in variation between the modeled distribution and the starting distribution of the Gibbs chain.

U2 - 10.1162/NECO_a_00085

DO - 10.1162/NECO_a_00085

M3 - Journal article

C2 - 21162669

SN - 0899-7667

VL - 23

SP - 664

EP - 673

JO - Neural Computation

JF - Neural Computation

IS - 3

ER -