Natural Language Understanding Wiki

Narrative cloze is a task proposed by Chambers and Jurafsky (2008)[1] It is widely used to evaluate models of script knowledge (Pichotta & Mooney, 2016a[2]; Pichotta & Mooney, 2016b[3]; Jans et al., 2012[4]; Rudinger et al., 2015a[5]; Rudinger et al. (2015b)[6])

From Pichotta & Mooney (2016b): "The exact definition of the Narrative Cloze evaluation depends on the formulation of events used in a script system. For example, Cham- bers and Jurafsky (2008), Jans et al. (2012), and Rudinger et al. (2015) evaluate inference of held- out (verb, dependency) pairs from documents; Pi- chotta and Mooney (2014) evaluate inference of verbs with coreference information about multi- ple arguments; and Pichotta and Mooney (2016) evaluate inference of verbs with noun informa- tion about multiple arguments. In order to gather human judgments of inference quality, the latter also learn an encoder-decoder LSTM network for transforming verbs and noun arguments into En- glish text to present to annotators for evaluation."


  1. Chambers, N., and Jurafsky, D. 2008. Unsupervised learning of narrative event chains. In Proceedings of ACL, 789– 797. Association for Computational Linguistics.
  2. Pichotta, K., & Mooney, R. J. (2016a). Learning Statistical Scripts With LSTM Recurrent Neural Networks. In Proceedings of the 30th AAAI Conference on Artificial Intelligence (AAAI-16), Phoenix, Arizona.
  3. Pichotta, K., & Mooney, R. J. (2016b). Using Sentence-Level LSTM Language Models for Script Inference. Retrieved from
  4. Bram Jans, Steven Bethard, Ivan Vulic, and Marie Francine Moens. 2012. Skip n- grams and ranking functions for predicting script events. In Proceedings of the 13th Conference of the European Chapter of the Association for Com- putational Linguistics (EACL-12), pages 336–344.
  5. Rachel Rudinger, Pushpendre Rastogi, Francis Ferraro, and Benjamin Van Durme. 2015a. Script induction as language modeling. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP-15).
  6. Rudinger, Rachel, Vera Demberg, Ashutosh Modi, Benjamin Van Durme, and Manfred Pinkal. 2015b. "Learning to predict script events from domain-specific text." Lexical and Computational Semantics (* SEM 2015): 205.