Title: A multifaceted approach to investigating the equivalence of computer-based and paper-and-pencil assessments: an example of reading diagnostics

Authors: Hong Jiao, Shudong Wang

Addresses: Department of Measurement, Statistics and Evaluation, University of Maryland, 1230B Benjamin Building, College Park, MD 20742, USA. ' Shudong Wang, Northwest Evaluation Association, 5885 SW Meadows Road, Suite 200, Lake Oswego, OR 97035-3256, USA

Abstract: The purpose of this study was to present a multifaceted approach to investigating the equivalence of the computer-based and the paper-and-pencil tests and exemplify it using the Stanford Diagnostic Reading Tests. The multifaceted approach includes construct equivalence evaluation, test-level comparison and item-level comparisons, especially differential functioning of items across the two modes. The results show that less than 1% of items in grades 3 and 4 were flagged with DIF, though about 5% of the items in grade 2 were flagged with DIF. The CBT and the PPT for the same grade test shared similar test characteristics. The mode did not affect the raw score to scale score conversion relationship significantly. The underlying constructs between the two modes of the same grade test satisfied the three levels of construct equivalence, i.e., congeneric, τ-equivalent and parallel equivalent.

Keywords: differential item functioning; test characteristics; construct equivalence; computer-based tests; paper-and-pencil tests; multifaceted approaches; Stanford diagnostic reading tests; test-level comparisons; item-level comparisons; DIF; test grades; raw scores; scale scores; conversion relationships; congeneric equivalents; τ-equivalents; parallel equivalents; learning technology.

DOI: 10.1504/IJLT.2010.037307

International Journal of Learning Technology, 2010 Vol.5 No.3, pp.264 - 288

Published online: 02 Dec 2010 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article