The Participant Perception Indicator: finding out what they learn…as they learn it!

1. The Participant Perception Indicator:finding out what they learn…as they learn it! Carl Berger University of Michigan carl.berger@umich.edu

2. What’s a PPI? • The Participant Perception Indicator (PPI) is a tool to help evaluate and/or assess instructional projects or courses that may have technology components or be technology enabled. • The PPI contains a questionnaire of one page length, or a short web survey, that measures perceptions of knowledge, experience and confidence of 3-5 themes determined important for learning.

3. Problems with pre and post tests... • Measures only twice • What’s happening during… • Often focuses on knowledge only • Some times measures general satisfaction • Almost always multiple choice • One right answer • Doesn’t look for profiles.. • gain, no significant differences, even worse • Looks at entire class.

4. The Participant Perception Indicator... • Can measure multiple times: before, during and after • A quick and short measure • Looks at themes as well as combined measures • Can be used to develop clusters of similar learners or find if there are differing profiles for different learners • Measures cognitive, behavioral and affective • Lends to repeated measures test, profile analysis, cluster analysis, and (gasp) multiple dimensional scaling

5. What is it… • A single page or short web page. • A series of Likert scales • Parallel statements of perception of knowledge, experience and confidence based on themes with questions reflecting themes.

6. Background • Developed in 1974 to examine pre-service and in-service teachers competence in teaching • Used in many studies including • Examining teacher computer literacy • Middle manager experience in distance independent learning • Collaborative chemistry experience

7. The basic form…directions √ √ √

10. Example • This PPI was used in ED727: Research Methods in Cognition and Technology • Wanted to look for growth in four themes: analysis, skills, collaboration, creation • Created 20 statements related to the four themes

11. The first four statements of the ED727 PPI(You may take the survey at http://FreeOnlineSurveys.com/rendersurvey.asp?id=27880) (continued on to 20 questions around four themes of Analysis, skills, collaboration and creation)

12. Statements related to themes • Theme: Analysis • 1.Analyze data and use to reach conclusions • 8. Search for information on the internet to learn to do complex analysis • 11 Evaluate the information and conclusions presented in graphs and charts • 12. Use stats to tease our changes in user perceptions of knowledge, experience and confidence • 15. Use case studies and projects to identify and learn evaluation

13. Theme: Skills • 2. Use Excel to create charts and graphs • 3. Search for evidence to improve your understanding of analysis • 5. Obtain information from the web • 13. Use surveys, stats, qualitative analysis to report results • 14. Post work in CourseTools

14. Theme: Collaboration • 4. Contribute ideas to group discussion • 7. Use classmates as co-designers • 10. Contribute to questions posted by my colleagues • 17. Suggest information sources that are not included in course materials • 18. Participate in class discussions online

15. Theme: Creation • 6. Develop themes from qualitative data • 9. Develop problem solving inventories • 16. Develop skills in evaluation project management • 19. Develop evaluation studies with my colleagues • 20. Design and produce an effective articulation and report of my conclusions

16. ED 727 Class PPI • The purpose of this study was to find if an interactive research methods course had any impact on perceptions of analysis, skills, collaboration and creation and how students perceptions changed as they worked though the course

17. Advanced research methods class Graduate course Wide range of entering skills Presentation 1/3 of class was computer lab work Case studies started 2/3rds through the course Through lab and case study, students covertly and overtly were encouraged to collaborate with peers All students do similar but varied labs Students work in groups Results shared with all in class Emphasis on making sense of all data Presentation of results by team to entire class Context

18. Analysis • Averaged the results for each theme. • For this study also averaged the thee measure of knowledge, behavior and confidence. • Graphs developed for the themes showing beginning, middle and end of the course. (The error bars are equal to the 95% confidence intervals.)

19. Results High perception Low perception

20. Results • Perceptions of all themes increased significantly from the beginning to the end of the course. • Perceptions of analysis and collaboration increased linearly throughout the course. • Perceptions of skill and creation increased more during the second half of the course (Where collaboration and creation activities took place). • Perceptions for all themes were closer together at the end of the course.

21. Conclusions • The Participant Perception Indicator easily discriminated among beginning, middle and ending student perceptions, was easy to use and to interpret. • For this Study, student perception of knowledge, experience and confidence increased for all themes throughout the course. Collaboration and Creation perceptions increased more during the second half of the course. While Creation perception started with the lowest perception and remained so at the middle of the course, it gained sufficiently during the second half to place well within the all final perceptions. • The goals of faculty member for increasing student perceptions of skills, analysis, collaboration and creation were met in the course and evidenced by the results from the Participant Perception Indicator.

22. Example 2: ColLabNet Chemistry • Would an introductory chemistry collaborative laboratory course have any impact on collaboration, laboratory experience and inquiry skills and how students changed as they worked though the course

23. The first seven statements

24. What we looked for... • Differences in Laboratory work, Collaboration and Inquiry • Differences in Schools and Colleges • Differences in Gender

25. Results High perception Low perception

26. Limitations • Measures perception rather than observation • Should be part of a broader assessment or evaluation package • Works best with clear themes and statements

27. References to other uses of the PPI All references are hyperlinked to full reports • Berger, C., Kerner, N., & Lee, J. (1999) Understanding Student Perceptions of Collaboration, Laboratory and Inquiry Use in Introductory Chemistry, Paper presented at the meeting of the National Association for Research in Science Teaching, Long Beach, CA. • Berger, C., & Barritt, M. (1997) Evaluation of the Location Independent Learning and Teaching Course:Economics of Networks, Process Survey, University of Michigan, Ann Arbor, MI. • Berger, C. and E. Carlson (1988). "Measuring computer literacy of teacher trainers." Journal of Educational Computing Research4(9): 289-303.