Abstract

Vector-based representations have long been a popular choice for researchers interested in unsupervised learning of word meaning. With the 2013 release of word2vec, the use of shallow neural language models (sometimes called word embeddings or prediction-based models) for constructing such vectors has become extremely popular. In the past year, however, several researchers have demonstrated that traditional distributional models (sometimes called count-based models) are capable of similar levels of performance when properly parameterized. Furthermore, count-based models give rise to more easily interpretable lexical representations, making them preferable to neural models in certain use cases. I give examples of several areas in which simple co-occurrence based models demonstrate surprisingly high levels of utility or performance: predicting human judgments of semantic relatedness and similarity, estimation of geographic locations, extraction of semantic topics, and a toy question-answering task and dataset recently proposed by researchers at Facebook AI Research as a possible step towards human-level natural language understanding. Applying a very simple, interpretable model to this dataset highlights benefits and shortcomings of the proposed task, and points the way to improved training and testing environments for natural language understanding systems.Abstract not available