Lecture 12 | Machine Learning (Stanford)
Why Cross Validated Stack Exchange recommends it
Stats Stack Exchange users link it when asking about or explaining EM, since the lecture derives the algorithm through a lower bound on the log likelihood.
Recommended by 1 people across 1 different months, 2015 to 2019.
What it is
Lecture 12 of Stanford's CS 229 Machine Learning course, taught by Andrew Ng in 2008. It covers unsupervised learning through clustering, Jensen's inequality, mixtures of Gaussians and the expectation-maximization (EM) algorithm.
What people said
Recommends user5016984 · 2015-07 · Cross Validated Stack Exchange · on “Expectation Maximization clarification questions”
I know there are many questions related to EM, but I still could not find the answer I seek. I have learned Expectation Maximization from this turtorial: Andrew Ng [this video] . The lecture is easy to follow. To my understanding, Andrew's intuition is that when doing maximum likelihood for latent variable model, the log likelihood function can be difficult to maximize due to that the latent variable is unobserved. Therefore, we can find a tight lower bound of the log likelihood function by using Jensen's inequality, and maximize this lower bound function, which is easier to maximize than the original log likelihood function. Suppose ${\vec x}^{(i)}$ is the observed vector for sample i, ${\vec z}
Cites Lenar Hoyt · 2016-01 · Cross Validated Stack Exchange · on “What is the relationship between generative models and density estimation?”
Thanks. Here Prof. Ng introduces EM as a way of doing density estimation, wouldn't that be a parametric goal? [this video]
Cites olooney · 2019-04 · Cross Validated Stack Exchange · on “EM algorithm for mixture of Gaussians - is it ok to use my updated mu's in my new estimate of Sigma, within a single M-step?”
Yes, and it's easy to see why if we look at it the right way. Forget about the M-step; in fact, forget the fact that we're inside an iterative algorithm at all. Forget about the past and that you ever had other estimates for $\vec{\mu}$ or $\mathbb{\Sigma}$ . And forget that there are other classes, and just focus on one class. And forget that the weights can be interpreted as $p(m|\vec{x}_i;\theta)$ and just think of them as sample weights, origin unknown, called $w_i$ . For convenience let's also assume that these weights are normalized, e.g. $\sum w_i = 1$ . What remains is a sample of $n$ (instead of $T$ so I can reserve $T$ for "transpose") observations $\vec{x}_i$ with weights $w_i$ wh