The Need for Incremental Validity

Incremental validity refers simply to how much a new tool adds to the predictive power of existing tools when predicting some criterion outcome (Hunsley & Meyer, 2003)⁠. For example, if a new test for intelligence were developed, it should be compared to tests currently available in order to evaluate the incremental increase in predictive power the new tool provides. Does the new tool more accurately predict the criterion outcome? When added to other tools, is the increase in predictive accuracy practically important?

In a recent study, the HESI Admission Assessment® was evaluated for how well it predicted student success in nursing programs (Murray, Merriman, & Adamson, 2008).⁠ The authors calculated correlation coefficients for a group of ADN (N = ~68) and BSN (N = ~69) students, using the Admission Assessment and the course grade to calculate the correlations. It is not clear whether continuous data were used for course grades (e.g., 74%) or if grade letters were used. Since the correlation coefficients are referred to as Pearson coefficients, I will assume continuous data were used. Also, many of a larger sample of ADN students did not complete their programs, and a t-test was used to compare Admission Assessment scores between completers and non-completers.

The authors found correlation coefficients between admission test scores and course grades that ranged from .442 at the largest to -.015 at the smallest. The admission tests scores were statistically significantly different in the ADN completer vs. non-completer groups. The authors conclude that the admission test scores were "...valid predictors of student success and can therefore assist faculty in the selection process as well as assist students in identifying their academic weaknesses so that these weaknesses can be remediated before they enter the nursing program" (p. 171). There are several problems with this conclusion.

1. The correlations were small, and only a few even reached a medium effect size, when considering the r². This spreadsheet shows the calculation of the r²s - over 80% of them were small or nill in size. Statistical significance does not infer validity, especially incremental validity.

2. All of the BSN students completed their academic programs, yet the correlations for the BSN students were smaller than for the ADN students, where 80/217 didn't complete the program. BSN students likely had at least 1 year of college work prior to taking the admission test. The admission test assesses for competency in several areas (math, reading, grammar, etc.). It is even more puzzling then why the BSN students, who had successfully made it through the first year of college where many general education courses are taken, had lower overall correlations than did the associate degree students.

3. The authors suggest that the admission test can be used to help faculty and students before entry into the nursing program, but the BSN students took the exam after admission to their program. We don't even know if remediation increases achievement or success later.

4. Students who scored low on the admission test were referred for remediation (this remediation was not described), but this was not accounted for in the relationship between the admission test and course grades. Clearly if the remediation intervention had any positive effect, academic achievement would be increased, and any resulting correlation analysis would be affected as well.

Aside from these methodological questions, there remains an overall incremental validity question. It seems that the admission test used in this study is similar in many ways to other more well-known exams such as the Scholastic Aptitude Test (SAT). A majority of students going to college in this country take either the SAT or the ACT before entering college. Data available on the SAT specifically, which is designed to predict college student academic success early in the college career, show consistently higher correlation coefficients than was presented in the study reviewed here (Burton & Ramist, 2001)⁠.

In addition, Kuncel and Hezlett (2007)⁠ present a nice overview showing correlations between several standardized tests and various criterion outcomes - which are consistently higher than correlations reviewed here. Even high school grades have been shown to be more related to college academic performance than standardized tests (Data on Student Preparation, College Readiness, and Achievement in College, 2007; Kirby, White, & Aruguete, 2007; Weissberg, Owen, Jenkins, & Harburg, 2003)⁠.

The question that then emerges is this: Why do we have a separate "nursing" admission test which tests for basic academic skills when so many other larger, well-studied, and more widely deployed and understood tests are available? The answer isn't clear.

References


Burton, N. W., & Ramist, L. (2001). Predicting success in college: SAT® studes of classes graduating since 1980. College Board. Retrieved May 22, 2008, from http://professionals.collegeboard.com/data-reports-research/cb/sat-classes-graduating-since-1980.

Data on Student Preparation, College Readiness, and Achievement in College. (2007). Peer Review, 9(1), 24-25.

Hunsley, J., & Meyer, G. J. (2003). The incremental validity of psychological testing and assessment: Conceptual, methodological, and statistical issues. Psychological Assessment, 15(4), 446-455.

Kirby, E., White, S., & Aruguete, M. (2007). Predictors of white and minority student success at a private women's college. College Student Journal, 41(2), 460-465.

Kuncel, N. R., & Hezlett, S. A. (2007). ASSESSMENT: Standardized tests predict graduate students' success. Science, 315(5815), 1080-1081. doi: 10.1126/science.1136618.

Murray, K. T., Merriman, C. S., & Adamson, C. (2008). Use of the HESI Admission Assessment to predict student success. Computers, Informatics, Nursing, 26(3), 167-72. doi: 10.1097/01.NCN.0000304781.27070.a7.

Weissberg, N. C., Owen, D. R., Jenkins, A. H., & Harburg, E. (2003). The Incremental variance problem: Enhancing the predictability of academic success in an urban, commuter institution. Genetic, Social & General Psychology Monographs, 129(2), 153-180.