Evaluation Methods for Topic Models
A natural evaluation metric for statistical topic models is the probability of held-out documents given a trained model. While exact computation of this probability is intractable, several estimators for this probability have been used in the topic modeling literature, including the harmonic mean method and empirical likelihood method. In this paper, we demonstrate experimentally that commonly-used methods are unlikely to accurately estimate the probability of held-out documents, and propose two alternative methods that are both accurate and efficient.
The above code release includes fixes for bugs in the Chib-style implementation reported by Matthew Willson. The buggy original version used for the paper is available for reference. That version is still consistent for long Markov chains but is not expected to work as well. See the README in the new release for details. As a result of this mistake, we probably under-reported the performance of the Chib-style method.
We have now also released most of the data used in the paper. You could download everything with a command like:
wget -erobots=off -r -np -p -k -nH --cut-dirs=4 http://homepages.inf.ed.ac.uk/imurray2/pub/09etm/data/