Cognitive validity in listen-to-write summary tasks: A mixed-methods analysis of notetaking data

Authors

• Rebecca Yeager
• GoMee Park, The University of Texas Rio Grande ValleyFollow
• Junhee Park

Document Type

Article

Publication Date

2026

Abstract

This study examines the cognitive validity of a listen-to-summarize task on a university placement test. Cognitive validity is underinvestigated in listening assessment, especially for higher-order listening processes (Holzknecht et al., 2017; Rukthong, 2021). However, student notes from listening tests can record evidence of these processes (Yeager et al., 2024; Cushing, 1991). This study employed an exploratory sequential mixed-methods design to investigate the higher-order listening processes elicited by an integrated listen-to-summarize task. First, notes from two administrations of the same test form – one with notetaking scaffolding (n = 25) and one without (n = 25) – were analyzed qualitatively. Second, qualitative codes informed the development of a rubric to quantify higher-order listening processes across administrations, enabling comparison with summary and multiple-choice scores on another test form (n = 118). Qualitative analysis revealed substantial evidence of higher-order discourse construction processes in student notes, including Selecting, Integrating, Monitoring, and Structure-Building (Field, 2013), especially when scaffolding was provided. Linear regression models indicated that Selecting processes significantly predicted test scores for both the integrated task and the multiple-choice task, while Structure-Building processes were significant only for the integrated task. Results support the cognitive validity of the listen-to summarize task and inform improved test design and score interpretation.

Comments

© The Author(s) 2026. This is an open access article distributed under the terms of the Creative Commons Attribution 4.0 International License, which permits the user to copy, distribute, and transmit the work provided that the original authors and source are credited. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Recommended Citation

Yeager, R., Park, G., & Park, J. (2026). Cognitive validity in listen-to-write summary tasks: A mixed-methods analysis of notetaking data. Studies in Language Assessment.

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

Publication Title

Studies in Language Assessment