Overview of the NTCIR-11 Recognizing Inference in TExt and Validation (RITE-VAL) Task

Abstract

This paper describes an overview of Recognizing Inference in TExt and Validation (RITE-VAL) task in NTCIR-11. We evaluated systems that automatically recognize semantic re lations between sentences such as entailment, contradiction and independence in Japanese (JA), English (EN), Simpli fied Chinese (CS) and Traditional Chinese (CT). RITE-VAL task has the following two subtasks: Fact Validation subtask (FV) and System Validation subtask (SV). SV consists of bi nary classification subtask (SVBC) and multi-classification subtask (SVMC). We had 23 active participating teams, and received 170 formal runs (59 Japanese runs, 9 English runs, 53 Simplified Chinese runs and 49 Traditional Chinese runs). This paper also describes how the datasets for RITE-VAL had been developed, how the systems were evaluated, and reports RITE-VAL formal run results.