Diverse Natural Language Inference


Inference Is Everything (zip), DNC v0.2 (url)


Natural Language Inference (NLI) is the task of determining whether a human would likely infer a textual hypothesis from a context, or premise (Dagan et al., 2006, 2013). Many NLI datasets do not clearly test what types of semantic reasoning a model can perform. We address this issue by recasting semantic annotations from existing NLP datasets into labelled NLI examples. We present a large-scale collection of recast NLI datasets that evaluates how well models perform distinct types of reasoning. We refer to this collection as the DNC: the Diverse Natural Language Inference Collection.

For a detailed description of the datasets and the item construction and collection methods as well as models of these data, please see the following papers:

White, A. S., P. Rastogi, K. Duh, & B. Van Durme. 2017. Inference is Everything: Recasting Semantic Resources into a Unified Evaluation Framework. Proceedings of the The 8th International Joint Conference on Natural Language Processing, pages 996–1005, Taipei, Taiwan, November 27 – December 1, 2017.

Poliak, A., A. Haldar, R. Rudinger, J.E. Hu, E. Pavlick, A.S. White, & B. Van Durme. 2018. Collecting Diverse Natural Language Inference Problems for Sentence Representation Evaluation. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31-November 4, 2018.

Kim, N., R. Patel, A. Poliak, P. Xia, A. Wang, T. McCoy, I. Tenney, A. Ross, T. Linzen, B. Van Durme, B. Bowman, E. Pavlick. 2019. Probing What Different {NLP} Tasks Teach Machines about Function Word Comprehension, Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*{SEM}), Minneapolis, Minnesota, June 6 - 7, 2019.

If you make use of these datasets in a presentation or publication, we ask that you please cite the corresponding papers. Additionally, if you use any recast NLI dataset, we ask that you please cite the original dataset that we previously recast (we provide a bibtex entry for each dataset online).


Benjamin Van Durme bio photo
Benjamin Van Durme
Aaron Steven White bio photo
Aaron Steven White
Rachel Rudinger bio photo
Rachel Rudinger
Adam Poliak bio photo
Adam Poliak
Edward Hu bio photo
Edward Hu