Saturday, April 4, 2009

latent variable text models

I've been reading about latent variable text models. I'm trying to make sense of them by reading a lot of different papers. I got started reading Blei, et Al.,'s paper on the Latent Dirichlet Allocation. This went a bit over my head, especially with respect to the variational methods. By reading the Grifith, Steyvers, and Tenenbaum paper, I got a taste of how the problem would be solved using Markov Chain Monte Carlo methods and Gibbs sampling. This gave me a broader idea of different approaches to the parameter estimation aspect, but I'm still a little bit in terra incognito. I went back and read the Hoffmann paper on probablistic LSA, which brings me back on my radar. The previous method, LSA used SVD, which I'm familiar with, more or less. The Griffiths, Steyvers, and Tenenbaum paper did a good job of motivating topic models over LSA with arguments from psychological literature. But they did less well of motivating topic models over PLSA (or aspect model, as Hoffmann calls it). Also, for PLSA, the parameter estimation is done via EM, which is also in my charted territory. He does move to a variational method ("Tempered EM"), which seems pretty cool-- it uses the entropy/Helmholtz free energy idea from chemistry--Pretty cool! If only I remembered everthing from chemistry.