Commentary, critique, and ideas on nursing and health by Dr. Darrell Spurlock, Jr.

The NLN Education Summit 2008 has drawn to a close and today will be busy with folks traveling back home after several very full days of activities.  There were many great sessions and the Presidental Banquet and Awards Dinner was accented by a local Mexican music and dance ensemble, Fiesta Mexicano.  Here are some additional pics of the events.

In the evening before a nice dinner on the San Antonio Riverwalk.

A little later in the same general area.

Fiesta Mexicano at the concluding Banquet and Awards Ceremony.

I am writing live from the NLN Education Summit 2008.  Things so far are going very well.  We were treated to a keynote speech from Dr. Gloria Smith, long-time leader in nursing and famous (or as she might have it, infamous) speaker on things regarding diversity, difference, race, and culture.  Wednesday evening was capped off by a nice reception sponsored by the College Network, where nurse educators from nearly every state in the U.S. as well as some from Japan and Mexico were able to mingle and fellowship.

Today (Thursday), there has been today already a very nice breakfast, a panel discussion on increasing diversity in nursing education, and two breakout sessions with a wide range of topics.  Currently, everyone is gearing up for the third session of the day, to start at 2:30.  Over lunch, attendees were able to look at the various posters on display, visit with exhibitors, and just meet and network with other nurse educators from across the country (really, the globe).

Here are some selected images thus far:

Attendees taking in the sights and sounds.

Dr. Terry Valiga (formerly of the NLN, now at Duke SON) doing a signing for her upcoming new book.

Dr. Linda Carl discussing her poster on program improvement and student success

Yesterday, Dr. Randy Pausch died after a long and heroic battle with pancreatic cancer.  He became famous for his “Last Lecture” soon after being diagnosed and given 3-6 months to live.  As a nurse, I know the grave prognosis that exists for most folks diagnosed with pancreatic cancer.  I once new an emergency physician that summed it up nicely: Hearing about an otherwise health someone being diagnosed with pancreatic cancer makes you just want to go and get a CT scan.  Dr. Pausch was a picture of health before his diagnosis and his extreme health probably let him survive nearly 2 years after being diagnosed.

I will take away from reading Dr. Pausch’s story a renewed comittment to not just interact with students and colleagues, but to (at least attempt to) inspire them.  I will also seek to find inspiration in more places than I have in the past.  Certainly, Dr. Pausch will be greatly missed by his family and friends, and for those of us who have been following his profound story of life that gained attention only at the news of an imminent death, we can only hope to lead a life as inspirational as his.

As I continue to be contacted weekly by students, faculty, parents, etc., about the use of progression policies in nursing programs across the country, something interesting has come to light.  Schools are using varying cut scores in their progression policies.  Indeed Nibert, Young and Britt (2003) noted that of the schools surveyed in thier study, scores that schools used for “benchmarking” (i.e., a “cut-score”) ranged from 77 to 90 (corrected for today’s scores, that would be 770 to 900).

Why should the scores schools use be different from place to place?  Standard setting for educational tests is somewhat of a science (Broadfoot, 2002; Karantonis & Sireci, 2006), but it doesn’t appear that most schools are using any empirical data to support the cut scores they are using.  This can be inferred in that schools have varying cut scores in place for their progression policies.  How are decisions on cut scores made?  What evidence supports one score over another?

For example, Lewis (2006) reported on data from a HESI Exit Exam® dataset with N = 8,009, that students scoring in the 800-849 range on the Exit Exam passed the NCLEX-RN® 93.3% of the time.  Students scoring in the 700-799 range pased the NCLEX-RN 85.3% of the time (see the Figure).

Why then, would schools have a cut score of 850 on the Exit Exam when data from a large N suggests that students scoring less than 850 might have more than a 9/10 chance of passing the NCLEX-RN?

Zieky and Perie (2006) suggest that when setting a cut score, the harm that will be done if students are misclassified must be considered.  If students are misclassified as “likely to fail” and are therefore prevented from graduating and taking the NCLEX-RN, then there can be profound consequences for that student’s life.  Their job plans, financial status, personal relationships, and many other dimensions of life are instantly in jeopardy.  There is a serious risk of harm, therefore, if students are misclassified as unprepared for the licensure exam when they in fact are.

On the other hand, if students are classified as likely to pass, and are therefore allowed to graduate and sit for the licensure exam, the risk of harm in that misclassification is less harmful for the student (who can take the exam again in a matter of weeks), but more important for the school, which seeks to have a high NCLEX-RN pass rate.

The only logical conclusion then is that when cut scores are chosen non-empirically (i.e., not based on available data), and possibly even arbitrarily, the risk of harm is shifted significantly to the student with schools erring on the side of not allowing qualified students to graduate and sit for the licensure exam.

The school’s licensure pass rate may be protected, but the lives of many students are profoundly and negatively impacted.  Many, many qualified nurses are then kept from the workforce because they are unable to test for a nursing license due to a progression policy that prevents their graduate on the basis of a score from one test.

References

Broadfoot, P. (2002). Dynamic versus arbitrary standards: Recognising the human factor in assessment. Assessment in Education: Principles, Policy & Practice, 9(2), 157-159.

Karantonis, A., & Sireci, S. G. (2006). The bookmark standard-setting method: A literature review. Educational Measurement: Issues and Practice, 25(1), 4-12.

Lewis, C. (2006). Predictive accuracy of the HESI Exit Exam on NCLEX-RN pass rates and effects of progression policies on nursing student exit exam scores. Dissertation Abstracts International, 66(11), B. (UMI No. 3195986)

Nibert, A. T., Young, A., & Britt, R. (2003). The HESI Exit Exam: Progression benchmark and remediation guide. Nurse Educator, 28(3), 141-145.

Zieky, & Perie (2006). A primer on setting cut scores on tests of educational achievement. Retrieved June 22, 2008 from http://www.ets.org/Media/Research/pdf/Cut_Scores_Primer.pdf.

Using tests in appropriate ways can be a very powerful tool for learning. Specifically, there is an effect, the testing effect, which basically changes the knowledge one has by the simple act of testing (Roediger & Karpicke, 2006a)⁠. This effect has been seen in much basic research (in the laboratory) and in applied research. For quite some time, some educators have promoted the use of tests not only for assessment purposes (i.e., summative assessment), but also for learning (formative assessment). These educators can now rest assured there is some empiric backing to their positions, that is, that testing can be used for learning, too.

Additional work by the authors (Roediger & Karpicke, 2006b)⁠ showed that repeated testing was more effective among samples of college students than was repeated studying. The authors provide several potential theoretical reasons for the results, including some based on basic understanding of human memory and some based upon the idea that “testing as learning” provides practice to the test-taker, thereby increasing recall and performance on future tests.

Another important bit of research is on providing feedback to students after multiple-choice tests (Butler & Roediger, 2008)⁠. Using a 3×3x3×2 experimental factorial design, the researchers investigated how the amount of study, number of multiple-choice alternatives, feedback condition, and report option (forced vs. free report) interacted to influence participant performance on test performance. Results showed that prior testing and studying both resulted in improved later performance on the study measures, but that prior testing had a larger effect than studying. Also, feedback given to the participants reduced the number of errors they made on future tests, likely due to them correcting misinformation in their knowledge, allowing for more accurate recall remembered materials.

There are frequently questions about the value of testing in student learning, and the research discussed here provides some evidence toward that point. Tests don’t have to be only for summative assessment - that is, to assess what students have learned. Tests can also be used to assist in students’ learning. Providing frequent testing in courses, along with feedback on the tests with a focus on correcting mis-remembered information may be an effective strategy to enhance student learning.

Butler, A., & Roediger, H. (2008). Feedback enhances the positive effects and reduces the negative effects of multiple-choice testing. Memory & Cognition, 36(3), 604-616.

Roediger, H. L., & Karpicke, J. D. (2006a). The power of testing memory: Bbasic research and implications for educational practice. Perspectives on Psychological Science, 1(3), 181-210. doi: 10.1111/j.1745-6916.2006.00012.x.

Roediger, H. L., & Karpicke, J. D. (2006b). Test-enhanced learning. Psychological Science, 17(3), 249-255. doi: 10.1111/j.1467-9280.2006.01693.x.

The nursing education literature is full of research (of varying qualities) on predicting NCLEX-RN® passing and failure by nursing students.  It has been a dominant topic in the literature since a national test became available.  It seems that nearly every month, in one nursing education publication or another, there is some new report on predicting NCLEX-RN outcomes, programs to help students “at-risk” of failure on the NCLEX-RN, or about remediation of low-scoring students on some academic skill set.

What effect does this preoccupation by nursing faculty have on addressing other problems in nursing education?  Most of the studies and reports put out on this topic focus on student-level variables.  GPA. Test scores. Test anxiety. Course failures.

What about curriculum evaluation?  There isn’t much about that.  If students get to the end of an academic program and are unprepared for the licensure exam, is that a student problem, or is that a curricular problem?  Students don’t pass themselves.  Faculty pass students on to the next course.  So, what are students to think when they get to the end of a program yet are un(der)prepared to take the NCLEX-RN?  Is it really their fault?  I think not.  In the age of public school accountability, it is not only students who pay, but also faculty when student performance is sub-par.  In nursing education, however, it is much easier to shift the burden to students and make the issue one of student preparation, rather than one of the systems and processes that get students to the end of their programs, un(der)prepared, in the first place.

Calibration is key.  Curricula must be calibrated to the test.  This does not mean that one cannot teach more than is on the test (NCLEX-RN), but you certainly cannot teach less than is on the test and expect graduates to pass.  The NCLEX-RN blueprint changes periodically, and every 2-3 years it seems the passing standard on the NCLEX-RN is raised (the test becomes more difficult to “pass”).  Do faculty stay on top of these changes and re-calibrate their curricula to meet the dynamic nature of the licensure exam?  If the passing standard increased periodically, are nursing education curricula adjusted accordingly in difficulty level?  This could be part of the seemingly omnipresent problem of NCLEX-RN pass rates.

I won’t discuss whether or not the NCLEX-RN should drive nursing education (at least at the pre-licensure level) like it does.  It truly does, no questions asked.  If it was announced tomorrow that the NCSBN was changing the content of the licensure exam to include a significant focus on genetic therapies for developmental disorders, schools would be compelled to increase or add this content to their current curricula.  What I think is happening is that less distinct changes, such as a .07 logit increase in the passing standard, are not being followed by schools as closely as they should be.  True, the national pass rate doesn’t “plummet” when the passing standard is changed, but NCLEX pass rates seem to be a nagging problem for schools, and this could reflect the underlying, always dynamic nature of the test graduates take to become licensed.  This idea is not to forsake issues of quality and consistency in curricula themselves, but it is a reasonable proposition, given the gravity of the data present on the problem.

The whole issue of the May-June 2008 Journal of Professional Nursing is devoted to cultural competency in nursing education, and largely contains work from faculty at The University of Pennsylvania. It is a very interesting read, as the authors describe their journey to being more culturally competent, aware, and diverse as a school. I whole-heartedly support this kind of movement in nursing education, but continue to get the sense that cultural competency is somehow tied to better patient outcomes (not necessarily from this special issue of JPN, but in general).

Drevdahl and colleages¹ provided a very good review of the assumption that cultural competency is the answer to health disparities. Their conclusions, which are supported by the available data, are that little evidence exists that cultural competency (either the training or being “competent” itself) are effective means at reducing health disparities. Certainly, there has been an increasing focus on cultural competency in the last 10 years, but little has changed with health disparities.

Recent work^3-6 has illustrated just how poor the current situation is. Essentially, cultural competency has not been shown to improve patient outcomes when the data are considered in whole. A recent publication in nursing, however, continues to link the two issues.⁷

Certainly a lack of awareness of issues surrounding appropriate culturally-sensitive care could lead to unequal treatment for some persons, but disparities still exist on many fronts. ^8-10 The source of these disparities is more than individuals, and so nursing’s focus on the individual and on cultural competence will be insufficient to address the problems. More attention must be paid the social, structural, and political forces in our society that perpetuate inequality on all fronts. Specifically, greater attention needs to be paid to the antecedents of poor health, such as poverty, low-quality educational systems, and crime.

These are all nursing problems because they are public health problems. Viewing them as such puts the problems back in the lap of nursing.

1. Drevdahl D, Canales M, Dorcy K. Of goldfish tanks and moonlight tricks: Can cultural competency ameliorate health disparities? Adv Nur Science. 2008;31(1):13-27.
 

2. Beach M, Price E, Gary T, et al. Cultural competence: A systematic review of health care provider educational interventions. Med Care. 2005;43(4):356-373.
 

3. Capell J, Veenstra G, Dean E. Cultural competence in healthcare: Critical analysis of the construct, its assessment and implications. J Theory Construct Test. 2007;11(1):30-37.
 

4. Jones M, Cason C, Bond M. Cultural attitudes, knowledge, and skills of a health workforce. J Transcult Nurs. 2004;15(4):283-290.
 

5. Price EG, Beach MC, Gary TL, et al. A systematic review of the methodological rigor of studies evaluating cultural competence training of health professionals. Acad Med. 2005;80(6):578-586.
 

6. Scrimshaw SC, Fullilove MT, Fielding JE, Normand J, Anderson LM. Culturally competent healthcare systems: A systematic review. Amer J Prev Med. 2003;24(Suppl3):68-79.
 

7. Giger J, Davidhizar R, Purnell L, et al. American Academy of Nursing Expert Panel report: Developing cultural competence to eliminate health disparities in ethnic minorities and other vulnerable populations. J Transcult Nurs. 2007;18(2):95-102.
 

8. Blendon RJ, Buhr T, Cassidy EF, et al. Disparities in health: perspectives of a multi-ethnic, multi-racial America. Health Aff (Millwood). 2007;26(5):1437-1447.
 

9. Snowden LR. Bias in mental health assessment and intervention: Theory and evidence. Amer J Pub Health. 2003;93(2):239-243.
 

10. Trivedi AN, Zaslavsky AM, Schneider EC, Ayanian JZ. Trends in the quality of care and racial disparities in Medicare Managed Care. New Eng J Med. 353(7):692-700.

Incremental validity refers simply to how much a new tool adds to the predictive power of existing tools when predicting some criterion outcome (Hunsley & Meyer, 2003)⁠. For example, if a new test for intelligence were developed, it should be compared to tests currently available in order to evaluate the incremental increase in predictive power the new tool provides. Does the new tool more accurately predict the criterion outcome? When added to other tools, is the increase in predictive accuracy practically important?

In a recent study, the HESI Admission Assessment® was evaluated for how well it predicted student success in nursing programs (Murray, Merriman, & Adamson, 2008).⁠ The authors calculated correlation coefficients for a group of ADN (N = ~68) and BSN (N = ~69) students, using the Admission Assessment and the course grade to calculate the correlations. It is not clear whether continuous data were used for course grades (e.g., 74%) or if grade letters were used. Since the correlation coefficients are referred to as Pearson coefficients, I will assume continuous data were used. Also, many of a larger sample of ADN students did not complete their programs, and a t-test was used to compare Admission Assessment scores between completers and non-completers.

The authors found correlation coefficients between admission test scores and course grades that ranged from .442 at the largest to -.015 at the smallest. The admission tests scores were statistically significantly different in the ADN completer vs. non-completer groups. The authors conclude that the admission test scores were “…valid predictors of student success and can therefore assist faculty in the selection process as well as assist students in identifying their academic weaknesses so that these weaknesses can be remediated before they enter the nursing program” (p. 171). There are several problems with this conclusion.

1. The correlations were small, and only a few even reached a medium effect size, when considering the r². This spreadsheet shows the calculation of the r²s - over 80% of them were small or nill in size. Statistical significance does not infer validity, especially incremental validity.

2. All of the BSN students completed their academic programs, yet the correlations for the BSN students were smaller than for the ADN students, where 80/217 didn’t complete the program. BSN students likely had at least 1 year of college work prior to taking the admission test. The admission test assesses for competency in several areas (math, reading, grammar, etc.). It is even more puzzling then why the BSN students, who had successfully made it through the first year of college where many general education courses are taken, had lower overall correlations than did the associate degree students.

3. The authors suggest that the admission test can be used to help faculty and students before entry into the nursing program, but the BSN students took the exam after admission to their program. We don’t even know if remediation increases achievement or success later.

4. Students who scored low on the admission test were referred for remediation (this remediation was not described), but this was not accounted for in the relationship between the admission test and course grades. Clearly if the remediation intervention had any positive effect, academic achievement would be increased, and any resulting correlation analysis would be affected as well.

Aside from these methodological questions, there remains an overall incremental validity question. It seems that the admission test used in this study is similar in many ways to other more well-known exams such as the Scholastic Aptitude Test (SAT). A majority of students going to college in this country take either the SAT or the ACT before entering college. Data available on the SAT specifically, which is designed to predict college student academic success early in the college career, show consistently higher correlation coefficients than was presented in the study reviewed here (Burton & Ramist, 2001)⁠.

In addition, Kuncel and Hezlett (2007)⁠ present a nice overview showing correlations between several standardized tests and various criterion outcomes - which are consistently higher than correlations reviewed here. Even high school grades have been shown to be more related to college academic performance than standardized tests (Data on Student Preparation, College Readiness, and Achievement in College, 2007; Kirby, White, & Aruguete, 2007; Weissberg, Owen, Jenkins, & Harburg, 2003)⁠.

The question that then emerges is this: Why do we have a separate “nursing” admission test which tests for basic academic skills when so many other larger, well-studied, and more widely deployed and understood tests are available? The answer isn’t clear.

References
Burton, N. W., & Ramist, L. (2001). Predicting success in college: SAT® studes of classes graduating since 1980. College Board. Retrieved May 22, 2008, from http://professionals.collegeboard.com/data-reports-research/cb/sat-classes-graduating-since-1980.

Data on Student Preparation, College Readiness, and Achievement in College. (2007). Peer Review, 9(1), 24-25.

Hunsley, J., & Meyer, G. J. (2003). The incremental validity of psychological testing and assessment: Conceptual, methodological, and statistical issues. Psychological Assessment, 15(4), 446-455.

Kirby, E., White, S., & Aruguete, M. (2007). Predictors of white and minority student success at a private women’s college. College Student Journal, 41(2), 460-465.

Kuncel, N. R., & Hezlett, S. A. (2007). ASSESSMENT: Standardized tests predict graduate students’ success. Science, 315(5815), 1080-1081. doi: 10.1126/science.1136618.

Murray, K. T., Merriman, C. S., & Adamson, C. (2008). Use of the HESI Admission Assessment to predict student success. Computers, Informatics, Nursing, 26(3), 167-72. doi: 10.1097/01.NCN.0000304781.27070.a7.

Weissberg, N. C., Owen, D. R., Jenkins, A. H., & Harburg, E. (2003). The Incremental variance problem: Enhancing the predictability of academic success in an urban, commuter institution. Genetic, Social & General Psychology Monographs, 129(2), 153-180.

There are many different software products available to help with relaying information in online formats. I have a personal preference for TechSmith’s products, Camtasia Studio and SnagIt. Though neither of these is available right now for Mac, they are both under development for Mac according to the TechSmith site. I find SnagIt particularly helpful at doing screen captures and then making a few notes on the image with arrows, text, “stamps,” and so forth. I use both products under Parallels for Mac, and I also use Jing, another TechSmith project that is intended more for on-the-fly screencasting. With Jing, there is a 5-minute limit on recording, but the sharing feature allows quick uploading to FTP sites, screencast.com, or now, you can just save the file to your local system.

All of these products help provide a richer online communication experience. Some folks like Adobe’s Captivate product, but I just haven’t been able to get on board. While it is true that if you are truly seeking to create step-by-step instructional videos, like for how to use a certain software program, then Captivate might be good. I find Camtasia Studio, and especially the PowerPoint plugin is superior for recording of lectures or discussion for online audiences.

For the Mac users out there, SnapZ Pro is a good screen recording and capture tool, but it just doesn’t have the integrated annotation and production tools that Camtasia has. I use them all for their intended purposes but could easily reduce down to using just Camtasia and SnagIt when they become available for Mac.

Just as a side note, I found it comical that while using Windows Live Writer to make this posting, the word “PowerPoint” was deemed by Live Writer as a misspelling. Now it is clear why switching to a Mac is a good idea!