Project Design and Data Collection Methods:

1. SoTL Fellows 2005-2006 Project Design and Data Collection Methods: A quick and dirty introduction to classroom research Margaret Waterman September 21, 2005

2. Where are we inthe Action Research Cycle? Decide when and how to collect data Identify Innovation ☻ Develop Action Plan Gather Data Analyze Data Interpret Data

3. Design vs. Method • Design is about when and from whom you are going to collect data so that the data can be interpreted. On Tuesday With Colonel Mustard • Methods are about where and how you are going to collect that data. In the Library With a candlestick With an interview

4. Designs: Experimental • Experimental: strongest for drawing inferences about cause and effect. • Control vs. experimental groups, randomly assigned, one variable at a time • Diagram as follows: O1 X1 O2 O1 O2 O = observation, X = treatment; Each row is a group

5. Experimental designs • Problems: • Not always possible in education to have two or more matched groups, • Random assignment usually not feasible, • May not be suitable for small-scale projects • Usually not for classroom research problems

6. Designs: Pre-Experimental • Weakest designs: Post test only X1 O1 Slightly stronger: Post-test with comparison group X1 O1 O1 • NOT RECOMMENDED

7. Pre-Experimental: Post Test Only Problems: • Too many other reasons for observed conditions • Cannot draw valid inferences about causality because you don’t know what pre-treatment condition is. • In comparison group: may have differences in incoming ability in the two groups • Good for: preliminary observation and development of hypotheses, not showing cause and effect

8. Designs: Pre-Experimental • One group, Pre-test Post-test O1 X1 O2 • Uses: Can make some inferences about cause and effect, but only when discuss possible intervening variables. • Useful in classrooms when trying new techniques or materials.

9. Pre-Experimental: Pre-test, Post-test • Problems: • Can’t discount all intervening variables: e.g., history, awareness of being in an experiment, selection effects on group membership. • Generalizability is limited • May sensitize students with the pre-test

10. Designs: Quasi-Experimental • Uses an experimental design, but groups are not randomly assigned. O1 X1 O2 O1 O2 Effect of treatment isolated X1 O2 Use two more groups to O2 isolate effect of pre-test. • Can make inferences with more confidence than with one-group designs Problem: intervening variables again.

11. Design: Time Series • One group, multiple observations before and after treatment. Can be two groups or two treatments. O1 O2 O3 O4 X1 O5 O6 O7 O8 OR O1 O2 O3 O4 X1 O5 O6 O7 X2 O8 O9 O10 Look for BIG differences after treatment as compared to before treatment. Problem: The intervening variables

12. Linking Design and Methods Suppose the design is one-group, pre-test, post-test: O1 X1 O2 How, where, and exactly when will the data be gathered at the two points of observation?

13. Action Research Often Uses These Kinds of Methods to Gather Data Observations Surveys Interviews Artifact Analysis

14. Methods of Gathering Data • Make observations • Structured observations • Unstructured observations • Ask questions • Surveys • Interviews • Examine and Score Artifacts • Tests, portfolios, treatment plans, student comments • Must design scoring guide

15. Structured Observations • Examples: • Use a class map to keep track of who asks questions, answers questions, makes comments. • Have a list of behaviors. Make simple hash marks for each type of behavior when it occurs. • Make a list, from a recorded class discussion, of the kinds of questions you asked.

16. Methods: Unstructured Observations • Examples • A journal that a teacher keeps to record personal impressions of how a class is going. • A written remembrance of interactions with one or a few students that are being tracked over time. • A written set of impressions made while watching a videotape of a class.

17. Methods: Unstructured Observations • Think of these as similar to ethnographic “field notes,” a place to record reactions, impressions, planned actions, responses and thoughts. • Can be useful to see trends in how YOU are responding to the innovation you are attempting. • Can be useful when studying one or a few students, again, over time to see trends.

18. Methods of Gathering Data • Make observations • Structured observations • Unstructured observations • Ask questions • Surveys • Interviews • Examine and Score Artifacts • Tests, portfolios, treatment plans, student comments • Must design scoring guide

19. Methods: Questioning (survey/interview) • Uses a prepared set of questions. • May have open-ended or close-ended questions, or a combination. • Good for getting background info. • Can be used pre and post treatment. • Can address things that may not be visible, such as attitudes, motivation. • Use with individuals or groups.

20. Methods: Questioning with Surveys • Examples: • A survey of attitudes toward science • Rankings of importance • Student ratings of instruction • May include open-ended questions, e.g., what element of this course most helped you to learn? • May include some content, but if entirely content, it’s a test and is an artifact of the course.

21. Sample from an Survey

22. Methods: Questioning with Interviews • Advantage over surveys: can ask follow-ups, more personal contact. • Examples: • Sort readings by usefulness, telling why • Solve a genetics problem aloud, explaining thinking • A focus group • Interview of team members in small groups

23. Sample from Interview

24. Methods of Gathering Data • Make observations • Structured observations • Unstructured observations • Ask questions • Surveys • Interviews • Examine and Score Artifacts • Tests, portfolios, treatment plans, student comments • Must design scoring guide

25. Method: Examine and Score Artifacts • Examples of artifacts: • Diagrams of cells before and after instruction • Wear on computer keys to see which are hit most • Answers to a test question • Portfolios • Term papers • Case analyses • Management plans • Treatment plans

26. Methods: Artifact Analysis • Decide what you kinds of materials you want to collect • Justify why the artifact you are choosing is a good choice given your research question • Create a scoring rubric (guide) to assign points • Good for pre-post designs

27. A KEY TO SUCCESS: Pilot Test Your Instruments • Give a small group of people (not in your class if you can) your survey or interview • Collect sample artifacts to see if your grading scheme works • Try out your observation scheme to see if it needs to be tweaked.

28. Designs and Methods: Summary • Many of these approaches are familiar to we who teach • Much of this is used in research in many disciplines. • Talk about your design and instruments with Teaching Associates, Fellows, other colleagues. • Keep your data collection focused. • Try to keep the project reasonable for the time you have available.

29. To create your own plans answer the questions below: • From whom are you gathering data? More than one class, subgroups? • When will you gather data? All one term, over several courses, in three weeks? • How will you gather data? Questions, artifacts, observations? • Where will you gather data? Classroom, online forum, mailed survey, telephone?