From 1970s to 2010s, we can observe a paradigm shift from algorithms to machine learning, from programming to modeling, from imperative to declarative, from deterministic to probabilistic.

In syntactic parsing, hand-written grammars are replaced by data-driven learned grammars. (TODO: cite) Most well-known is perhaps the debate between Chomsky and Norvig (a starting point: [1]).

In coreference resolution: the same trend. From Ng (2005): "Recent research in coreference resolution [...] has exhibited a shift from knowledge-based approaches to data-driven approaches, yielding learning-based coreference systems that rival their hand-crafted counterparts in performance (e.g., Soon et al. (2001), Ng and Cardie (2002b), Strube et al. (2002), Yang et al. (2003), Luo et al. (2004))."

Scripts are defined in Schank & Abelson (1977)[1] as curated structures on a limited set of primitives. In 2016, people talk about statistical scripts (Pichotta & Mooney, 2016)[2].

In many subtasks of NLP, decades were spent on devising one features after another. Nowadays people favour automatic discovery of features via models such as convolutional neural networks (Collobert et al. 2011)[3].

Same story in machine vision. (todo: cite)

See also: Machine learning is the new algorithms

Restaurant script Edit

A restaurant script created by Schank & Abelson (1977) (see figure) lists as entry conditions "hungry" and "has money" and as results "not hungry" and "has less money". Though the authors have made it clear that the script isn't meant to be exhaustive or infallible, the limit of the model is obvious.

Screen Shot 2016-04-27 at 16.29.10

From Schank & Abelson (1977)

References Edit

  1. Schank, Roger C., and Robert P. Abelson. 1977. Scripts, plans, goals, and understanding: An inquiry into human knowledge structures. Psychology Press.
  2. Pichotta, K., & Mooney, R. J. (2016). Learning Statistical Scripts With LSTM Recurrent Neural Networks. In Proceedings of the 30th AAAI Conference on Artificial Intelligence (AAAI-16), Phoenix, Arizona.
  3. Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa, P. (2011). Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12, 2493–2537.