Natural Language Understanding Wiki

From (Roth & Woodsend, 2014)[1]

"In general, there are two major ways of applying distributed representations to NLP tasks. First, they can be fed into existing supervised NLP sys- tems as augmented features in a semi-supervised manner. This kind of approach has been adopted in a variety of applications (Turian et al., 2010). Despite its simplicity and effectiveness, it has been shown that the potential of distributed representations cannot be fully exploited in the generalized linear models which are adopted in most of the ex- isting NLP systems (Wang and Manning, 2013). One remedy is to discretize the distributed feature representations, as studied in Guo et al. (2014). However, we believe that a non-linear system, e.g. a neural network, is a more powerful and effec- tive solution. Some decent progress has already been made in this paradigm of NLP on various tasks (Collobert et al., 2011; Chen and Manning, 2014; Sutskever et al., 2014)."

Analogy by arithmetic operation[]

  • Word embedding: Mikolov et al. (2013)[2]
  • Visual embedding: Reed et al. (2015)[3]


  1. Roth, M., & Woodsend, K. (2014). Composition of word representations improves semantic role labelling. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 407-413).
  2. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic Regularities in Continuous Space Word Representations. In Proceedings of NAACL HLT, 2013
  3. Scott Reed, Yi Zhang, Yuting Zhang, Honglak Lee. 2015. Deep Visual Analogy-Making. Advances in Neural Information Processing Systems (NIPS), Montreal.