Diverse Natural Language Inference

Data

Inference Is Everything (zip), DNC v0.1 (url)

About

Natural Language Inference (NLI) is the task of determining whether a human would likely infer a textual hypothesis from a context, or premise (Dagan et al., 2006, 2013). Many NLI datasets do not clearly test what types of semantic reasoning a model can perform. We address this issue by recasting semantic annotations from existing NLP datasets into labelled NLI examples. We present a large-scale collection of recast NLI datasets that evaluates how well models perform distinct types of reasoning. We refer to this collection as the DNC: the Diverse Natural Language Inference Collection.

For a detailed description of the datasets and the item construction and collection methods as well as models of these data, please see the following papers:

White, A. S., P. Rastogi, K. Duh, & B. Van Durme. 2017. Inference is Everything: Recasting Semantic Resources into a Unified Evaluation Framework. Proceedings of the The 8th International Joint Conference on Natural Language Processing, pages 996–1005, Taipei, Taiwan, November 27 – December 1, 2017.

Poliak, A., A. Haldar, R. Rudinger, J.E. Hu, E. Pavlick, A.S. White, & B. Van Durme. 2018. Collecting Diverse Natural Language Inference Problems for Sentence Representation Evaluation. To appear in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31-November 4, 2018.

If you make use of these datasets in a presentation or publication, we ask that you please cite both of these papers. Additionally, if you use any recast NLI dataset, we ask that you please cite the original dataset that we previously recast (we provide a bibtex entry for each dataset online).

Researchers

Benjamin Van Durme bio photo
Benjamin Van Durme
Aaron Steven White bio photo
Aaron Steven White
Rachel Rudinger bio photo
Rachel Rudinger
Adam Poliak bio photo
Adam Poliak