COE 501 Introduction to Research and Evaluation in Education

1. COE 501Introduction to Research and Evaluation in Education Dr. Tirupalavanam G. Ganesh tganesh@asu.edu

2. Introduction to ResearchEvaluation Discussion of Ideas in Chapters 1 and 2

3. Educational Research • An overview of educational research • Scientific and disciplined inquiry • Functions of educational research • Specific approaches • Quantitative designs • Qualitative designs • Guidelines for determining the appropriateness of an approach • Limitations of scientific and disciplined inquiry

4. Educational Research • Formal definition • The application of a scientific and disciplined inquiry approach to the study of educational problems • Goal • To explain or help understand educational issues, questions, or problems

5. Educational Research • Secondary purposes • Help others understand research results • Use results to improve teaching and learning • Raise new topics for study • Educational research as an evolving, ongoing process

6. Educational Research • Six ways we can knowsomething • Tradition • Expert opinion • Personal experience • Intuition • Logic • Inductive • Deductive • Research

7. Educational Research • Tradition • Doing things as they have always been done • Limitations • Often based on an idealized past • Can be distant from current realities and the complexities associated with them • Experts or authorities • Relying on the expertise or authority of others • Limitations • Experts can be wrong • Experts can disagree among themselves as ina “second opinion”

8. Educational Research • Personal experience • Relying on one’s knowledge of prior experiences • Limitations • How I am affected by an event depends on who I am • But we frequently need to know something that cannot be learned through experience • Intuition • Relying on your “gut” feeling • Limitations • Difficulty verifying results

9. Educational Research • Inductive reasoning • Reasoning from the specific to the general • Limitations • In order to be certain of a conclusion one must observe all examples • All examples can be observed only in very limited situations where there are few members of the group

10. Educational Research • Deductive reasoning • Reasoning from the general to the specific • Limitations • You must begin with true premises in order to arrive at true conclusions • Only organizes what is already known

11. Educational Research • Research • Systematically studying problems using a scientific and disciplined inquiry approach • Limitations • Difficulty removing errors related to the complexity of human behavior in varying contexts • Difficulty controlling researcher bias

12. Educational Research • Research provides the most unbiased and verifiable understanding • Some decisions require such evidence, others do not • Class size and retention policies need to be based on evidence from research given the importance of such policies • Basing these policies on tradition, experts, personal experience, intuition, or logic subject them to criticism related to the limitations of each source of knowledge

13. Educational Research • What are some questions related to your professional work, and what sources of information would you rely on to reflect on these questions? • Why are some sources of information appropriate and others not?

14. Scientific and Disciplined Inquiry • A systematic approach to examining issues and questions that combines features of inductive and deductive reasoning with other characteristics to produce a reliable approach to understanding • Characteristics • Systematic nature • Detailed descriptions of procedures

15. Scientific and Disciplined Inquiry • Four general steps • Identify a topic • Collect data • Analyze data • Report the results and implications • Flexibility of these steps to incorporate a range of purposes and methods

16. Functions of Research • Basic research • Conducted to develop, test, or refine theory • Examples related to learning theory • Piaget • Constructivism • Mastery learning • Gardner’s multiple intelligences

17. Functions of Research • Applied research • Conducted to examine the usefulness of theory in solving practical educational problems • Examples • Developing seventh grade social studies curriculum around a problem-solving approach to learning • Examining the effectiveness of a computer-based algebra program developed around a mastery learning approach • Accommodating varied learning styles when teaching lessons in modern literature

18. Functions of Research • Evaluationresearch • Conducted to assess the merit or worth of a specific practice in terms of the values operating at a specific site • Merit is exemplified by a program accomplishing what it was supposed to accomplish • Worth is exemplified by the value attached to a program by those using it

19. Functions of Research • Evaluation (continued) • Examples • The computerized algebra program being used in Williams Middle School has been installed properly, is being used properly, and student achievement is increasing as a result of its use • The computerized algebra program being used in Williams Middle School is perceived to be an efficient and effective expenditure of district funds

20. Specific Approaches to Research • Lack of a single, appropriate method to study education • Family of research methods • Quantitative • Qualitative

21. Specific Approaches • Differentiating characteristics • Underlying assumptions of the researcher • Purpose of the research • Research questions • Research designs

22. Specific Approaches • Differentiating characteristics • Interaction between the researcher and subjects/participants • Variables/Context • Data collection and analysis • Reports

23. Specific Approaches • Complementary nature of quantitative and qualitative approaches • Different purposes of research • Explanatory • Exploratory • Consideration of the strengths and weaknesses of different approaches for specific purposes • Qualitative versus Quantitative Research

24. Quantitative Designs • Purposes • Describe current conditions • Investigate relationships • Study causes and effects • Four major designs • Descriptive/survey • Correlational • Causal comparative • Experimental

25. Quantitative Designs • Descriptive/survey • Purpose – to describe current conditions • Examples • How many students drop out of school in Arizona? • What are the attitudes of parents, students, and teachers to an extended school year? • What kinds of activities typically occur in sixth-grade art classes, and how frequently does each occur? • What have been the reactions of school administrators to innovations in teaching physical science? • To what extent are elementary teachers using math manipulatives?

26. Quantitative Designs • Descriptive/survey (continued) • Characteristics • Use of large samples • Use of tests, questionnaires, and surveys • Focused on information related to preferences, attitudes, practices, concerns, or interests • Statistical analysis of numerical data • Potential problems • Instrument development • Low response rates • Honest responses from subjects

27. Quantitative Designs • Correlational • Purpose – to ascertain the extent to which two or more variables are statistically related • Examples • What is the relationship between ACT scores and freshmen grades? • Is a teacher’s sense of efficacy related to their effectiveness? • Do significant relationships exist between the types of activities used in math classrooms and student achievement?

28. Quantitative Designs • Correlational (continued) • Characteristics • Measurement with a correlation coefficient • One group of subjects measured on two variables • Use of instruments to measure variables • Focused on the direction and nature of the relationship

29. Quantitative Designs • Correlational (continued) • Do not establish cause-effect relationships between variables • Potential problems • Instrument development • Inferring cause and effect relationships

30. Quantitative Designs • Causal-comparative • Purpose – to explore relationships among variables that cannot be actively manipulated or controlled by the researcher • Examples • What is the effect of part-time employment on the achievement of high school students? • What characteristics differentiate students who dropout from those who do not? • What is the effect of attending a “magnet” school on student attitude?

31. Quantitative Designs • Causal comparative (continued) • Characteristics • Selection of subjects from at least two groups in which the cause (i.e., the independent variable) has already occurred, cannot/should not be manipulated • Statistical comparisons of the effect (i.e., the dependent variable) using at least two groups • Potential problems • Inferring cause and effect relationships

32. Quantitative Designs • Experimental • Purpose – to establish cause and effect relationships between variables • Examples • Examine the effect of teaching with a 1) co-operative groups strategy or 2) traditional lecture approach on student’s achievement • Examine the effect of teaching with manipulatives or a traditional algorithm approach on the test scores of algebra students

33. Quantitative Designs • Characteristics • Stringent procedures for selecting subjects and assigning them to groups • Manipulation of the causal variable • Control of extraneous variables • Statistical analysis of numerical data

34. Quantitative Designs • Potential problems • Inability of researcher to adequately control extraneous variables • Use of complicated research designs • Complex statistical analyses of data

35. Qualitative Designs • Purpose – provide field focused, interpretative, detailed descriptions and interpretations of participants and their settings • Four designs • Action research • Historical research • Ethnography • Grounded theory

36. Qualitative Designs • Action research • Purposes • To provide a solution to an educator’s problem in their own school or organization • To improve practice or understand issues • Examples • How can our college move to a performance based model for undergraduate teacher preparation programs? • How can disciplinary policies be enforced consistently in our school?

37. Qualitative Designs • Action research (continued) • Characteristics • Cyclical nature of data collection and analysis • Four basic steps – identify a problem, collect data, analyze data, and take action to resolve the problem • Typically the educator “owns” the problem in that they carry out the research themselves • Potential problem • Insular nature of the process can affect the rigor of the study

38. Qualitative Designs • Historical research • Purpose – to gain insight into past events, issues, of personalities to better understand the current situation • Examples • The difficulties being experienced while implementing a standards based curriculum can be understood more completely if one recognizes the historical top-down control of curriculum imposed on teachers by the State Department of Education • Current parochial school policies can be better understood with knowledge of the role these schools have played in the education of students in the community for the last fifty years

39. Qualitative Designs • Historical research (continued) • Characteristics • Focus on specific individuals, social issues, events, or policies • Documents and artifacts are the primary sources of data • Data is already available and is complied, presented, and interpreted • Data is examined carefully for authenticity and truthfulness

40. Qualitative Designs • Historical research (continued) • Potential problems • Authenticity • Truthfulness • Reliance on secondary sources • Values of researcher can affect interpretation

41. Qualitative Designs • Ethnography • Purpose – to obtain an understanding of the shared beliefs and practices of a particular group or culture • Examples • What is the nature of the problems a teacher encounters when she begins using a constructivist approach to instruction after having taught for ten years using a very traditional approach? • What does “inclusion” mean to a special needs child who is placed in an inclusionary classroom?

42. Qualitative Designs • Ethnography (continued) • Characteristics • The study is conducted in the natural setting for a lengthy period of time • Emerging research design • Participants are observed in naturally occurring activities • Researchers develops trust with participants • Cyclical nature of data collection and analysis

43. Qualitative Designs • Ethnography (continued) • Characteristics (continued) • Observation and interviews are the dominate data collection strategies • Inductive nature of the data analysis • Potential problems • Insufficient time spent in the field • Poor data collection efforts or insufficient data collected • Poor data analysis • Researcher bias

44. Qualitative Designs • Grounded theory • Purpose – to derive theory from the analysis of identified patterns, themes, and categories emerging from data • Examples • What theories underlie the school change efforts of teachers in a parochial elementary school? • What underlying theory explains teacher’s changing from traditional assessment beliefs and practices to alternative, performance-based assessment beliefs and practices?

45. Qualitative Designs • Grounded theory (continued) • Characteristics • Respect for participant’s beliefs and views • Qualitative data collection using analytic strategies • Inductively reasoned synthesis of data through the use of constant comparison analysis • Conceptual nature of the process

46. Qualitative Designs • Grounded theory (continued) • Potential problems • Researcher bias • Poor data collection strategies • Difficulty analyzing data

47. Guidelines for Choosing A Design • Problems dictate methods • Each design has particular characteristics that coincide with different types of problems

48. Limitations of Scientific and Disciplined Inquiry Approaches • Four limitations • Value-based, philosophical, or ethical problems or questions cannot be solved • These approaches provide a potentially overly simplified views of reality • Methodological concerns • Access to subjects • Data collection strategies • Data analysis • Limitations of research designs • Legal and ethical responsibilities of the researcher