NTCIR-7 ACLIA IR4QA Results based on Qrels Version 2

Abstract

This document is a postscript to the Overview of the NTCIR-7 ACLIA IR4QA Task [2]. At the NTCIR 7 Workshop Meeting (December 2008), participating systems of IR4QA were evaluated based on “qrels ver sion 1,” which covered the depth-30 pool for every topic and went further down the pool for a limited number of topics. Here, we report on revised results based on “qrels version 2” which covers the depth-100 pool for every topic. While the version 1 and version 2 results are generally in agreement, some differences in system rankings and significance test results suggest that the additional effort was worthwhile.