In All Likelihood, Deep Belief Is Not Enough


deepbelief.tar.gz (45.5 MB)
deepbelief_moec.tar.gz (45.5 MB)



The archive contains Python code for training and evaluating deep belief networks (DBNs) using the estimator described in this paper, as well as sets of natural image patches (van Hateren) and handwritten digits (MNIST) for running the experiments. The parameters of the experiments were changed so as to reduce their running time while still giving similar results to the results reported in the paper. Note that an experiment can still easily take a few hours, depending on your hardware.

A minimal example for training a stack of two RBMs with data points stored in the columns of a NumPy array data:
dbn = DBN(RBM(num_visibles=data.shape[0], num_hiddens=100))
dbn.add_layer(RBM(num_visibles=100, num_hiddens=200))
To evaluate a model, first estimate its partition function, then estimate its likelihood.
logptf = dbn.estimate_log_partition_function(num_ais_samples=100, beta_weights=arange(0, 1, 1E-3))
loglik = dbn.estimate_log_likelihood(data)
The choice of training and evaluation parameters is crucial and likely has to be adjusted to achieve good results.

The archive deepbelief_moec.tar.gz additionally contains code for training mixtures of elliptically contoured distributions or, more specifically, mixtures of Gaussian scale mixtures.


  author = "L. Theis and S. Gerwinn and F. Sinz and M. Bethge",
  title = "In All Likelihood, Deep Belief Is Not Enough",
  year = 2011,
  journal = "Journal of Machine Learning Research",
  pages = "3071-3096",
  month = "Nov",
  keywords = "natural image statistics, deep belief networks, boltzmann machines"
University of Tuebingen BCCN CIN MPI