Notetaking as validity evidence: A mixed-methods investigation of question preview in EAP listening assessment

Abstract

Recent scholarship has questioned the cognitive validity of listening tests with preview, in which test-takers can see test questions before listening. This study mined student notes for evidence of cognitive processes in listening tests with and without preview, using a mixed-methods design that explored the effect of test format on notetaking behaviors. Qualitative analysis indicated that students who previewed items were more likely to systematically omit information, highlight previewed keywords, and engage in shallower structural representation. Conversely, KruskalWallis tests revealed that students who listened without preview took more notes, especially of main ideas and details, and had better coverage of the lecture. However, correlation and hierarchical linear regression analyses found these notetaking achievements did not predict higher scores in the no-preview condition, while in the preview condition, only note quantity and focus on minor ideas predicted scores. Both strands of data suggest that students' cognitive processes were shaped by the format of the exam they experienced. These findings may bear on validity arguments for listening assessment and inform the way that language instructors prepare their students for academic listening.