Is Cultural Competence the Answer to Health Disparities?

The whole issue of the May-June 2008 Journal of Professional Nursing is devoted to cultural competency in nursing education, and largely contains work from faculty at The University of Pennsylvania. It is a very interesting read, as the authors describe their journey to being more culturally competent, aware, and diverse as a school. I whole-heartedly support this kind of movement in nursing education, but continue to get the sense that cultural competency is somehow tied to better patient outcomes (not necessarily from this special issue of JPN, but in general).

Drevdahl and colleages¹ provided a very good review of the assumption that cultural competency is the answer to health disparities. Their conclusions, which are supported by the available data, are that little evidence exists that cultural competency (either the training or being "competent" itself) are effective means at reducing health disparities. Certainly, there has been an increasing focus on cultural competency in the last 10 years, but little has changed with health disparities.

Recent work^3-6 has illustrated just how poor the current situation is. Essentially, cultural competency has not been shown to improve patient outcomes when the data are considered in whole. A recent publication in nursing, however, continues to link the two issues.⁷

Certainly a lack of awareness of issues surrounding appropriate culturally-sensitive care could lead to unequal treatment for some persons, but disparities still exist on many fronts. ^8-10 The source of these disparities is more than individuals, and so nursing's focus on the individual and on cultural competence will be insufficient to address the problems. More attention must be paid the social, structural, and political forces in our society that perpetuate inequality on all fronts. Specifically, greater attention needs to be paid to the antecedents of poor health, such as poverty, low-quality educational systems, and crime.

These are all nursing problems because they are public health problems. Viewing them as such puts the problems back in the lap of nursing.

1. Drevdahl D, Canales M, Dorcy K. Of goldfish tanks and moonlight tricks: Can cultural competency ameliorate health disparities? Adv Nur Science. 2008;31(1):13-27.

2. Beach M, Price E, Gary T, et al. Cultural competence: A systematic review of health care provider educational interventions. Med Care. 2005;43(4):356-373.

3. Capell J, Veenstra G, Dean E. Cultural competence in healthcare: Critical analysis of the construct, its assessment and implications. J Theory Construct Test. 2007;11(1):30-37.

4. Jones M, Cason C, Bond M. Cultural attitudes, knowledge, and skills of a health workforce. J Transcult Nurs. 2004;15(4):283-290.

5. Price EG, Beach MC, Gary TL, et al. A systematic review of the methodological rigor of studies evaluating cultural competence training of health professionals. Acad Med. 2005;80(6):578-586.

6. Scrimshaw SC, Fullilove MT, Fielding JE, Normand J, Anderson LM. Culturally competent healthcare systems: A systematic review. Amer J Prev Med. 2003;24(Suppl3):68-79.

7. Giger J, Davidhizar R, Purnell L, et al. American Academy of Nursing Expert Panel report: Developing cultural competence to eliminate health disparities in ethnic minorities and other vulnerable populations. J Transcult Nurs. 2007;18(2):95-102.

8. Blendon RJ, Buhr T, Cassidy EF, et al. Disparities in health: perspectives of a multi-ethnic, multi-racial America. Health Aff (Millwood). 2007;26(5):1437-1447.

9. Snowden LR. Bias in mental health assessment and intervention: Theory and evidence. Amer J Pub Health. 2003;93(2):239-243.

10. Trivedi AN, Zaslavsky AM, Schneider EC, Ayanian JZ. Trends in the quality of care and racial disparities in Medicare Managed Care. New Eng J Med. 353(7):692-700.

The Need for Incremental Validity

Incremental validity refers simply to how much a new tool adds to the predictive power of existing tools when predicting some criterion outcome (Hunsley & Meyer, 2003)⁠. For example, if a new test for intelligence were developed, it should be compared to tests currently available in order to evaluate the incremental increase in predictive power the new tool provides. Does the new tool more accurately predict the criterion outcome? When added to other tools, is the increase in predictive accuracy practically important?

In a recent study, the HESI Admission Assessment® was evaluated for how well it predicted student success in nursing programs (Murray, Merriman, & Adamson, 2008).⁠ The authors calculated correlation coefficients for a group of ADN (N = ~68) and BSN (N = ~69) students, using the Admission Assessment and the course grade to calculate the correlations. It is not clear whether continuous data were used for course grades (e.g., 74%) or if grade letters were used. Since the correlation coefficients are referred to as Pearson coefficients, I will assume continuous data were used. Also, many of a larger sample of ADN students did not complete their programs, and a t-test was used to compare Admission Assessment scores between completers and non-completers.

The authors found correlation coefficients between admission test scores and course grades that ranged from .442 at the largest to -.015 at the smallest. The admission tests scores were statistically significantly different in the ADN completer vs. non-completer groups. The authors conclude that the admission test scores were "...valid predictors of student success and can therefore assist faculty in the selection process as well as assist students in identifying their academic weaknesses so that these weaknesses can be remediated before they enter the nursing program" (p. 171). There are several problems with this conclusion.

1. The correlations were small, and only a few even reached a medium effect size, when considering the r². This spreadsheet shows the calculation of the r²s - over 80% of them were small or nill in size. Statistical significance does not infer validity, especially incremental validity.

2. All of the BSN students completed their academic programs, yet the correlations for the BSN students were smaller than for the ADN students, where 80/217 didn't complete the program. BSN students likely had at least 1 year of college work prior to taking the admission test. The admission test assesses for competency in several areas (math, reading, grammar, etc.). It is even more puzzling then why the BSN students, who had successfully made it through the first year of college where many general education courses are taken, had lower overall correlations than did the associate degree students.

3. The authors suggest that the admission test can be used to help faculty and students before entry into the nursing program, but the BSN students took the exam after admission to their program. We don't even know if remediation increases achievement or success later.

4. Students who scored low on the admission test were referred for remediation (this remediation was not described), but this was not accounted for in the relationship between the admission test and course grades. Clearly if the remediation intervention had any positive effect, academic achievement would be increased, and any resulting correlation analysis would be affected as well.

Aside from these methodological questions, there remains an overall incremental validity question. It seems that the admission test used in this study is similar in many ways to other more well-known exams such as the Scholastic Aptitude Test (SAT). A majority of students going to college in this country take either the SAT or the ACT before entering college. Data available on the SAT specifically, which is designed to predict college student academic success early in the college career, show consistently higher correlation coefficients than was presented in the study reviewed here (Burton & Ramist, 2001)⁠.

In addition, Kuncel and Hezlett (2007)⁠ present a nice overview showing correlations between several standardized tests and various criterion outcomes - which are consistently higher than correlations reviewed here. Even high school grades have been shown to be more related to college academic performance than standardized tests (Data on Student Preparation, College Readiness, and Achievement in College, 2007; Kirby, White, & Aruguete, 2007; Weissberg, Owen, Jenkins, & Harburg, 2003)⁠.

The question that then emerges is this: Why do we have a separate "nursing" admission test which tests for basic academic skills when so many other larger, well-studied, and more widely deployed and understood tests are available? The answer isn't clear.

References


Burton, N. W., & Ramist, L. (2001). Predicting success in college: SAT® studes of classes graduating since 1980. College Board. Retrieved May 22, 2008, from http://professionals.collegeboard.com/data-reports-research/cb/sat-classes-graduating-since-1980.

Data on Student Preparation, College Readiness, and Achievement in College. (2007). Peer Review, 9(1), 24-25.

Hunsley, J., & Meyer, G. J. (2003). The incremental validity of psychological testing and assessment: Conceptual, methodological, and statistical issues. Psychological Assessment, 15(4), 446-455.

Kirby, E., White, S., & Aruguete, M. (2007). Predictors of white and minority student success at a private women's college. College Student Journal, 41(2), 460-465.

Kuncel, N. R., & Hezlett, S. A. (2007). ASSESSMENT: Standardized tests predict graduate students' success. Science, 315(5815), 1080-1081. doi: 10.1126/science.1136618.

Murray, K. T., Merriman, C. S., & Adamson, C. (2008). Use of the HESI Admission Assessment to predict student success. Computers, Informatics, Nursing, 26(3), 167-72. doi: 10.1097/01.NCN.0000304781.27070.a7.

Weissberg, N. C., Owen, D. R., Jenkins, A. H., & Harburg, E. (2003). The Incremental variance problem: Enhancing the predictability of academic success in an urban, commuter institution. Genetic, Social & General Psychology Monographs, 129(2), 153-180.

Fair Testing

National standards for educational measurement, assessment, and evaluation leave little room for some practices. While there are several really important standards that faculty should be aware of, perhaps the most important is this: Important educational decisions should not be made on the basis of a single test score. Faculty would never think of having a course where the only evaluation method in the course was a single test. Why are progression or exit tests used in this way, then?

Faculty need look no further than the national debate around the No Child Left Behind Act. While billed as an accountability tool, clearly there are still children being left behind. Part of NCLB has been the creation and promotion of testing programs. Along with the advent of such testing programs has come inappropriate uses of tests for the purposes of promotion, tracking, and graduation. Because every test score contains within it an error component, no one test score should be used to make important educational decisions.

Short Bibliography

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.

Applied Research Center. (1999). No exit? Testing, tracking, and students of color in U.S. Public Schools. Applied Research Center. Retrieved May 8, 2008, from http://www.arc.org/pdf/262bpdf.pdf.

Heubert, J.P., & Hauser, R.M. (Eds.). (1999). High stakes: Testing for tracking, promotion, and graduation. Washington, DC: National Academy Press. Retrieved May 8, 2008, from the National Academy Press Web site: http://www.nap.edu/openbook.php?isbn=0309062802

Joint Committee on Testing Practices. (2004). Code of fair testing practices in education. Retrieved May 8, 2008, from the American Psychological Association Web site: http://www.apa.org/science/FinalCode.pdf

Spurlock, D., Jr. (2006). Do no harm: Progression policies and high-stakes testing in nursing education. Journal of Nursing Education, 45, 297-302.

Calculating Predictive Accuracy

How accurate are your tests in predicting outcomes? If you use predictive exit exams, then you should evaluate the accuracy of those tests for your students. One very effective way is to use a 2x2 contingency table to evaluate test characteristics. This will only work if you have 2 dichotomous variables (e.g., pass/fail, true/false, etc.). It will also only work if everyone who takes one test also takes the other - that is, if you restrict graduation or progression using a test, you cannot truly evaluate the predictive accuracy of an exit exam because not all students were able to take the criterion test (usually the NCLEX).

Using this model, you can evaluate various test characteristics, especially how accurate the test is in classifying students as either NCLEX failers or NCLEX passers.

I have put together a simple Excel spreadsheet here that will help you compute the values you need. There is an explanation on the 2nd sheet of the file, so take a look at that too.

Using this model is not a new idea, and has been done for quite some time to evaluate how well dichotomous predictions and outcomes compare. See the list below.

____________________________________

Demakis, G. J. (2004). Application of clinically-derived malingering cutoffs on the California Verbal Learning Test and the Wechsler Adult Intelligence Test-Revised to an Analog Malingering Study. Applied Neuropsychology, 11(4), 220-226.

Dent, H. E. (1976). Assessing black children for mainstream placement. Retrieved April 2, 2008, from ERIC online.

Fuchs, L. S., Fuchs, D., Compton, D. L., Bryant, J. D., Hamlett, C. L., & Seethaler, P. M. (2007). Mathematics screening and progress monitoring at first grade: Implications for responsiveness to intervention. Exceptional Children, 73(3), 311-330.

Hintze, J. M., Ryan, A. L., & Stoner, G. (2003). Concurrent validity and diagnostic accuracy of the dynamic indicators of basic early literacy skills and the Comprehensive Test of Phonological Processing. School Psychology Review, 32(4), 541-556.

Kahle-Wrobleski, K., Corrada, M. M., Li, B., & Kawas, C. H. (2007). Sensitivity and specificity of the Mini-Mental State Examination for identifying dementia in the oldest-dld: The 90+ study. Journal of the American Geriatrics Society, 55(2), 284-289.

Langendyk, V. (2006). Not knowing that they do not know: Self-assessment accuracy of third-year medical students. Medical Education, 40(2), 173-179.

Mazzocco, M. M. M., & Thompson, R. E. (2005). Kindergarten predictors of math learning disability. Learning Disabilities Research and Practice, 20(3), 142-151.

Pena, E. D., Gillam, R. B., Malek, M., Ruiz-Felter, R., Resendiz, M., Fiestas, C., et al. (2006). Dynamic sssessment of school-age children's narrative ability: An experimental investigation of classification accuracy. Journal of Speech, Language and Hearing Research, 49(5), 1037-1057.

R. Antonelli Incalzi, M. Cesari, C. Pedone, L. Carosella, & P.U. Carbonin. (2003). Construct validity of the Abbreviated Mental Test in older medical inpatients. Dementia & Geriatric Cognitive Disorders, 15(4), 199-206.

Raghoebar-Krieger, H. M. J., Sleijfer, D., Bender, W., Stewart, R. E., & Popping, R. (2001). The reliability of logbook data of medical students: An estimation of interobserver agreement, sensitivity and specificity. Medical Education, 35(7), 624-631.

Riedel, B. W. (2007). The relation between DIBELS, reading comprehension, and vocabulary in urban first-grade students. Reading Research Quarterly, 42(4), 546-562.

Scott, M. S., Delgado, C. F., Shihfen Tu, & Fletcher, K. L. (2005). Selecting and validating tasks from a kindergarten screening battery that best predict third grade educational placement. Education & Training in Developmental Disabilities, 40(4), 377-389.

Smith, C. B., & Watkins, M. W. (2004). Diagnostic utility of the Bannatyne WISC-III Pattern. Learning Disabilities Research and Practice, 19(1), 49-56.

Solan, H. A., Shelley-Tremblay, J. F., Hansen, P. C., & Larson, S. (2007). Is there a common linkage among reading comprehension, visual attention, and magnocellular processing? Journal of Learning Disabilities, 40(3), 270-278.

Tombaugh, T., & McDowell, I. (1996). Mini-Mental State Examination (MMSE) and the modified MMSE (3MS): A psychometric comparison. Psychological Assessment, 8(1), 48-57.

Wood, F. B., Hill, D. F., Meyer, M. S., & Flowers, D. L. (2005). Predictive assessment of reading. Annals of Dyslexia, 55(2), 193-216.

Exist Tests and What They Mean

While I have been most concerned with exit tests in higher education, there is not so much difference in the underlying issues between higher education and secondary education.

An interesting story on the Chronicle of Higher Ed site yesterday reported that a soon-to-be published study evaluated the effectiveness of high-school exit tests on improving academic achievement among graduates. The study¹ found that there was no such improvement; when compared to the "gold standard" national test, exit test scores did not predict achievement on the gold standard test. Interesting. A similar story was reported last year² in the Chronicle.

What do these tests mean if they have little relationship to the criterion test (i.e., the gold standard)?

1. Schmidt, P. (2008, May 13). State high-school exit tests do not improve academic achievement, study finds. Chronicle of Higher Education. Retrieved May 14, 2008, from http://chronicle.com/daily/2008/05/2820n.htm.

2. Basken, P. (2007, September 14). High-school exit tests don't mean readiness. Chronicle of Higher Education. Retrieved May 14, 2008, from http://chronicle.com/weekly/v54/i03/03a01804.htm.

Progression Policies

I continue to be contacted by students, lawyers, and (rarely) faculty members about progression policies. I am actually quite happy when students call because that tells me these folks are critically thinking. Most of them tell me they have found my articles in research databases and want to talk about the results. I am very happy that students, when they have questions, are going to the literature to look for answers. This is a great change from what we see reported in the literature that nurses most frequently ask other colleagues when they have questions in the clinical setting.¹ I do hope this spirit of inquiry follows these students into their careers.

I continued to be troubled, however, that so many students are having trouble with graduating, being certified for graduation, etc. - based on the fact that they can't achieve a cut-score on some end-of-program predictive test. I put together an explanation of why this is not a good idea and posted a link on this page.

As can be seen on this page, many schools are being involved in court actions, protests, and other negative situations over their use of progression policies and high-stakes testing. If anyone has news links to other sites with related news, please email me.

1.  Pravikoff, D. S., Tanner, A. B., & Pierce, S. T. (2005). Readiness of U.S. nurses for evidence-based practice. The American Journal of Nursing, 105(9), 40-51.