Calculating Predictive Accuracy

How accurate are your tests in predicting outcomes? If you use predictive exit exams, then you should evaluate the accuracy of those tests for your students. One very effective way is to use a 2x2 contingency table to evaluate test characteristics. This will only work if you have 2 dichotomous variables (e.g., pass/fail, true/false, etc.). It will also only work if everyone who takes one test also takes the other - that is, if you restrict graduation or progression using a test, you cannot truly evaluate the predictive accuracy of an exit exam because not all students were able to take the criterion test (usually the NCLEX).

Using this model, you can evaluate various test characteristics, especially how accurate the test is in classifying students as either NCLEX failers or NCLEX passers.

I have put together a simple Excel spreadsheet here that will help you compute the values you need. There is an explanation on the 2nd sheet of the file, so take a look at that too.

Using this model is not a new idea, and has been done for quite some time to evaluate how well dichotomous predictions and outcomes compare. See the list below.

____________________________________

Demakis, G. J. (2004). Application of clinically-derived malingering cutoffs on the California Verbal Learning Test and the Wechsler Adult Intelligence Test-Revised to an Analog Malingering Study. Applied Neuropsychology, 11(4), 220-226.

Dent, H. E. (1976). Assessing black children for mainstream placement. Retrieved April 2, 2008, from ERIC online.

Fuchs, L. S., Fuchs, D., Compton, D. L., Bryant, J. D., Hamlett, C. L., & Seethaler, P. M. (2007). Mathematics screening and progress monitoring at first grade: Implications for responsiveness to intervention. Exceptional Children, 73(3), 311-330.

Hintze, J. M., Ryan, A. L., & Stoner, G. (2003). Concurrent validity and diagnostic accuracy of the dynamic indicators of basic early literacy skills and the Comprehensive Test of Phonological Processing. School Psychology Review, 32(4), 541-556.

Kahle-Wrobleski, K., Corrada, M. M., Li, B., & Kawas, C. H. (2007). Sensitivity and specificity of the Mini-Mental State Examination for identifying dementia in the oldest-dld: The 90+ study. Journal of the American Geriatrics Society, 55(2), 284-289.

Langendyk, V. (2006). Not knowing that they do not know: Self-assessment accuracy of third-year medical students. Medical Education, 40(2), 173-179.

Mazzocco, M. M. M., & Thompson, R. E. (2005). Kindergarten predictors of math learning disability. Learning Disabilities Research and Practice, 20(3), 142-151.

Pena, E. D., Gillam, R. B., Malek, M., Ruiz-Felter, R., Resendiz, M., Fiestas, C., et al. (2006). Dynamic sssessment of school-age children's narrative ability: An experimental investigation of classification accuracy. Journal of Speech, Language and Hearing Research, 49(5), 1037-1057.

R. Antonelli Incalzi, M. Cesari, C. Pedone, L. Carosella, & P.U. Carbonin. (2003). Construct validity of the Abbreviated Mental Test in older medical inpatients. Dementia & Geriatric Cognitive Disorders, 15(4), 199-206.

Raghoebar-Krieger, H. M. J., Sleijfer, D., Bender, W., Stewart, R. E., & Popping, R. (2001). The reliability of logbook data of medical students: An estimation of interobserver agreement, sensitivity and specificity. Medical Education, 35(7), 624-631.

Riedel, B. W. (2007). The relation between DIBELS, reading comprehension, and vocabulary in urban first-grade students. Reading Research Quarterly, 42(4), 546-562.

Scott, M. S., Delgado, C. F., Shihfen Tu, & Fletcher, K. L. (2005). Selecting and validating tasks from a kindergarten screening battery that best predict third grade educational placement. Education & Training in Developmental Disabilities, 40(4), 377-389.

Smith, C. B., & Watkins, M. W. (2004). Diagnostic utility of the Bannatyne WISC-III Pattern. Learning Disabilities Research and Practice, 19(1), 49-56.

Solan, H. A., Shelley-Tremblay, J. F., Hansen, P. C., & Larson, S. (2007). Is there a common linkage among reading comprehension, visual attention, and magnocellular processing? Journal of Learning Disabilities, 40(3), 270-278.

Tombaugh, T., & McDowell, I. (1996). Mini-Mental State Examination (MMSE) and the modified MMSE (3MS): A psychometric comparison. Psychological Assessment, 8(1), 48-57.

Wood, F. B., Hill, D. F., Meyer, M. S., & Flowers, D. L. (2005). Predictive assessment of reading. Annals of Dyslexia, 55(2), 193-216.