# Deep Belief Networks Stack [[Boltzmann Machines#Restricted Boltzmann Machines|RBM]] layers assuming conditional independence $ p\left(x, v_{1}, v_{2}\right)=p\left(x \mid v_{1}\right) \cdot p\left(v_{1} \mid v_{2}\right) $ Deep Belief Networks are directed models Dense layers with single forward flow - As RBM is directional: $p\left(x_{i} \mid v, \boldsymbol{\theta}\right)=\sigma\left(\boldsymbol{W}_{\cdot i} \boldsymbol{x}+c_{i}\right)$ ## Deep Boltzmann Machines ![[deep-boltzmann.jpg]] Stacking RBM layers from *above* and *below* layers - Markov model Energy function: $ \begin{array}{l} E\left(x, v_{1}, v_{2} \mid \theta\right)=x^{T} W_{1} v_{1}+v_{1}^{T} W_{2} v_{2}+v_{2}^{T} W_{3} v_{3} \\ p\left(v_{2}^{k} \mid v_{1}, v_{3}\right)=\sigma\left(\sum_{j} W_{1}^{j k} v_{1}^{j}+\sum_{l} W_{3}^{k l} v_{3}^{k}\right) \end{array} $ Computing gradients is intractable Instead, variational methods (mean-field) or sampling methods are used --- ## References 1. Lecture 8.5 UvA DL Course 2020