Leveraging Competing and Complementary Roles for Success in R & D

Questions to Ponder… • What are the bridges evaluators must cross in the R & D world? • What are the roles at play in the R & D continuum? • How can evaluators respond to different (sometimes conflicting) expectations around issues of the validity of evidence? © 2005

The Implementation Phase • The Practitioner/Implementer • LOI High • LOA High • ER Will it work in context? • The Conceptualizer – Can it stand the adaptations? • The Funder – What is the market niche? © 2005

Overview • Concerns about the quality of evidence and claims underlie all social science • Such concerns have been punctuated by the increased interest in evidence-based inquiry and evidence-based practice… © 2005

Emerging Needs, Differing Perspectives • Increased pressure on Practitioners to select and engage in only those practices that are evidence-based, elevates considerations of what constitutes “evidence” and “evidence based” • Increased pressure on R&D Organizations to collect evidence for their products and services to satisfy practitioner requirements © 2005

Varied Perspectives on Validity • There exist varied perspectives regarding how external readers approach or engage in evaluation documents and varied responses from evaluation community regarding how to appropriately deal with this © 2005

House • 1985: Decisions about the data to collect are intertwined with prospectively considering the rhetorical power of statements one wishes to issue relative to audience... • …regardless of the veracity of the claim(s) being made, evaluators must attend to audience — if evaluation fails to provide audience with acceptable explanation / fails to enhance understanding of some phenomenon, findings may not be considered adequate • Thus, persuasion plays a role in evaluative claims and the perceived validity of the inferences and the extent to which the evaluator herself is able to craft a compelling rhetorical argument is partially a product of audience. Validity is therefore not merely about reaching “true” assertions. © 2005

Patton • 2002: goal of ensuring evaluative validity should not be to reach technical standards but rather, to determine whether appropriate methods and measures have been utilized for the particular evaluation purpose(s) and relative to the intended users of the evaluation findings © 2005

Lincoln • 2003: validity is not simply a matter of determining which data collection efforts lead to better information "…but rather, which kinds of evidence will best address certain questions, and, at a foundational level, which kinds of literary-rhetorical devices are being employed, and which kinds of symbolic-interpretive processes are being brought to bear in the mounting of a persuasive argument?" (italics in original). © 2005

Up the Ladder • Context: A state department of education • State interested in documenting accountability for the state funding of teacher professional development • Participants in the experience expressed interest in “telling their stories,” and resisted the state department data collection efforts © 2005

Ready, Set…Ready, Set… • Context: Proprietor of online professional development courses • While proprietor interested in collecting evidence of success of product, timing issues (evaluability) precluded the collection of meaningful data • In rush to advance evidence that program “works,” organization began to inappropriately utilize student data to make claims about program efficacy © 2005

Hurry, hurry! • Context: Textbook publisher interested in examining curricular materials with eye on textbook adoption • While product development pressed to “rush to market,” organization continued to stress need for the collection of “rigorous evidence” © 2005

All That Glitters • Context: State department and University interested in outcomes of systemic school reform model • State department interested in “bottom line accountability” student achievement • Participants interested in having their stories heard and having individual school successes and obstacles documented through case studies. © 2005

Evaluator Responsibilities • What are the responsibilities of the evaluator regarding evidence? • How does the evaluator navigate between competing demands for evidence? • At what point does the evaluator need to intervene with client(s) to ensure that claims being made are adequately supported? © 2005

Crossing Boundaries into the Future • Serve the needs of a broad base of stakeholders to protect against bias • Anticipate informational needs • Preserve credibility while remaining flexible © 2005

Leveraging Competing and Complementary Roles for Success in R & D