Development of Valid and Reliable Case Studies for Teaching, Diagnostic Reasoning, and Other Purposes

1. Development of Valid and Reliable Case Studies for Teaching, Diagnostic Reasoning, and Other Purposes Margaret Lunney, RN, PhD Professor College of Staten Island, The City University of New York

2. What Are Case Studies? • Simulations of patients’ stories: • Written, Video-taped, Computer-based • Designed for specific purpose(s) • Variety of designs: • Length • Content • Style: Linear, branching

3. When Are Case Studies Used? • Practice e.g., orientation of new nurses, discussion of complex cases • Education e.g., teaching aids, testing • Research e.g., measurement of accuracy, evaluation of knowledge

4. Advantages: Simulations of Patients’ Stories • Standardized • Represent important, usual, familiar, & challenging situations • Restricts the complexity of practice • User “gets involved” • Reasonable cost

5. Why Is It Difficult to Develop Good Case Studies? • Case studies are tools • Quality= Validity & Reliability • Development takes time, energy and commitment

6. Case Studies as Tools • Goal • Strengthen the link between knowledge and application • How • Operationalize complex abstract concepts and “real” patients’ stories, e.g., • Hope • Caregiver Stress • Ineffective Self Health Management

7. Case Studies as Tools to Measure Nursing Concepts • Nursing Concepts • nursing diagnoses • Capture key elements of a real patient situation • Use principles of measurement • Instrumentation-Method to measure concepts

8. Case Studies as Tools (cont.) • Measurement: Definition • See Waltz, Strickland & Lenz, 2005 • Importance of conceptual frameworks • Challenges of measurement

9. Case Studies as Tools (cont.) • The dilemma & challenge: • Patients’ stories are complex • Overlapping variables • Biases in interpretation • Use of heuristics • Many types of thinking • Goal • Reduce ambiguous, abstract ideas to concrete behavioral indicators

10. Measurement Frameworks • Norm-referenced • Evaluate performance relative to the performance of others • Criterion-referenced • Evaluate performance based on pre-determined standards

11. Case Studies as Criterion-Referenced Tools: Reliability • Definition: • Consistency of scoring • Range of variability is reduced, use nonparametric procedures • Test-Retest • Parallel forms • Interrater & intrarater agreement

12. Case studies as Criterion-Referenced Tools: Validity • Definition: • Measures what was intended, systematic error is reduced • Types: • Content validity • Criterion-related validity • Construct validity

13. Case Studies as Criterion-Referenced Tools: Tasks • Precisely specify target behaviors • Identify standards for target behavior • Discriminate who did and did not acquire the target behavior • Compare subjects’ performance to the standards

14. Case Studies as Tools (cont.) • Types of written simulations • Linear • Branching • Free branching • Modified free branching • Forced branching

15. Linear Technique • Subjects follow same sequence • One or more sections • If more than one section, students receive specific instructions, and • Sections are appropriate for all subjects • One section does not influence other sections • Each section samples inquiries or actions

16. Guidelines for Case Study Development: Linear Format • Identify the overall purposes: • What is the general topic? • What kind of problem solving will be represented? • How important are the responses, e.g., learning process, grade, research data?

17. Guidelines (cont.) 2. Specify objectives, e.g., to • Measure accuracy of diagnosis • Illustrate relation of cues to inferences • Facilitate planning • Orient new nurses • Teach sequential decision making

18. Guidelines (cont.) 3. Decide the complexity: • Length • Number of diagnoses, interventions, and/or outcomes • Amounts of high, moderate & low relevance data • Types of data

19. Guidelines (cont.) 4. Obtain literature sources & nurse experts to support validity: • Select authoritative sources: actual cases, literature sources • Decide importance of each source • Identify expert judges, e.g., may ask colleagues • Use well-known experts for research tools

20. Guidelines (cont.) 5. Formulate case studies, directions, & scoring manual • Directions must be explicit, comprehensive, and clearly written • Scoring manual provides scores for possible answers

21. A) Create Blueprint • Choose a general problem area • What should this exercise teach or test? • What kinds of problems need to be solved? • What kinds of knowledge needs to be used? • Define the objectives to be sampled • Select concrete problem situation(s) • Outline and diagram the exercise

22. B) Prepare Specifications • Method of administration • Proportion of content for each objective • Style of writing case(s) • Restrictions, e.g. time • Describe scoring procedures

23. C) Construct Case Studies • Develop pool of data/items to match objectives • Review to determine content validity and appropriateness • Edit, delete, change as indicated • Assemble the case(s)

24. D) Set Standards or Cut Scores • What types of answers are “correct” • What answers/scores are acceptable or unacceptable • Classification of various answers, # of points toward a grade

25. Guidelines (cont.) 6. Obtain content validity ratings from experts: • Select content validity method Item-objective congruence Interrater agreement Average congruency percentage • Send with explicit instructions • Be prepared to make changes

26. Guidelines (cont.) 7. Evaluate with a pilot test: • Consider demographics, e.g., Experience Knowledge • Evaluate directions, restrictions, scoring methods, interrater reliability

27. Conclusion • Case studies are fun to use • The effort is “worth it” • New case study feature in IJNTC, send submissions to Margaret Lunney • Suggested website: http://ublib.buffalo.edu/libraries/projects/cases/case.htm