Naturalistic Evaluation

1. Aaron D. Schroeder Ph.D. Basic Tenets and Case Study Naturalistic Evaluation

2. Naturalistic Evaluation • An approach to assessment that evolved from the work of researchers at the Indiana Center for Evaluation • Wolf • Tymitz • Guba • Lincoln • Has been used in numerous applications including schools, social programs, museums, health care

3. Two Streams • Naturalistic Evaluation has evolved from two parallel streams • ‘Responsive evaluation’ from the evaluation field • Naturalistic methodologies from the area of qualitative ‘inquiry’ used by anthropologists and sociologists practicing ‘ethnography’ • Combining the two makes Naturalistic Evaluation an intensive endeavor

4. Most Salient Feature • Emergent Design • The investigator enters a new situation with the intent of uncovering the needs of the clients and participants in the study • In contrast to ‘preordinate’ designs (hypotheses generated before hand and tested by measuring predetermined variables) • What is valued (and measured) is generated pluralistically • Audience concerns and issues organize the design • Methods are interactive and qualitative • Feedback is continuous and suited to audience needs

5. Viewpoint: Major Role of Evaluation • Guba and Lincoln believe it is to respond to audience requirements for information in ways that take account of the different value perspectives of its members • Naturalistic Evaluation puts the evaluator in the role of learner – the informants teach • However, criteria for scientific rigor still apply!

6. Purpose • Generally used to discover what is wrong (generally) and how to fix it (in a manner that will last), or how something is working (generally) and why (so you know how to keep doing it, or for possible use by others) • This is more rigorous than finding out what is wrong (according to the audience) and deciding what to do (according to the audience)

7. Requirements • The findings need to be • Credible (rather than Internally Valid) • Corroboration of data through cross checking and triangulation (all data sources considered, qualitative and quantitative) • Applicable (rather than Externally Valid) • ‘Working Hypotheses’ generated for testing in other situations • Accompanied by ‘Thick Descriptions’ (a very literal description of the entity being evaluated) • Auditable (rather than Reliable) • A second evaluator should be able to look at the same documentation and reasoning and understand the logic • Confirmable (rather than Objective) • The data being collected (e.g. opinions on operations) can be confirmed with other evidence

8. Difference from Other Participant Oriented • While the Naturalistic Evaluator is looking for the ‘truth’ of the situation, (s)he is generally doing so via unobtrusive measures and triangulation with other sources of data (existing documents, surveys, etc.)

9. General Phases • Familiarization phase • Gaining entry • Understanding the Circumstance and the players • Three C’s • Collection of Data • Classification and Analysis of Data • Confirmation of Propositions • Synthesis • Presentation of findings and debriefing

10. Case Study • An Enhanced Assessment and Support Team (EAST) for Dementing Elders – review of a Scottish Regional Initiative • Gary S. Stevenson, Heather Ewing, Joyce Herschell & Dennis Keith • Journal of Mental Health April 2006; 15(2) 251-258

11. Background • There is increasing acceptance of the benefits of community assessment and support for older people with mental health issues but there remains a lack of evaluation about community teams for this patient population

12. Study Aims • Specifically, to determine whether a community-based mental health team for elders with dementia could evaluate patients in a timely manner, address their care requirements and thereby av0id admission to either local psychogeriatric day hospital or inpatient care services Secondary aim was to ascertain the acceptability of the service to patients, caregivers, and professionals

13. Method • Naturalistic Evaluation • Surveyed participants • Conducted Structured Interviews with participants • Collected standardized data on patient tests for dementia levels, quality of life and visits to the hospital including miles travelled • Collected same data on a comparison group • Created very ‘Thick Description’ of the program including • Every participant (nurses, caregivers, doctors evaluators) • Population being served (demographics) • Funding for the program • Types and methods of assessment tests • Description of the EAST program

14. Findings Working Hypotheses • Integrated multiagency community mental health teams for the older population can be effective, meeting the needs and wishes of both patients and caregivers • Integrated multiagency community mental health teams for the older population can reduce the requirement for centralized hospital-based services

15. Naturalistic Evaluation • Pros • When done correctly, it can provide a very complete picture of a program’s context, stakeholders, operations, effects (qualitative and quantitative), and systems of relationship • A participant-oriented ‘gold standard’ • Cons • When done correctly, can • Take a prohibitively long time • Cost a lot • Be very labor intensive • When done incorrectly, can give a false sense of knowledge

16. Can take some time • Not unlike the work of Jane Goodall and her discovery of the social systems and organization of Chimpanzee societies • Only took her 43 years!