1 / 19

Evaluation Plan

An Evaluation Plan and Tool Kit for the Archibald Bush Innovative Teaching and Technology Strategies Grant Valerie Ruhe and J.D. Walker, Center for Teaching and Learning Services and Digital Media Center University of Minnesota. Evaluation Plan.

mikaia
Download Presentation

Evaluation Plan

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. An Evaluation Plan and Tool Kit for the Archibald Bush Innovative Teaching and Technology Strategies GrantValerie Ruhe and J.D. Walker, Center for Teaching and Learning Services andDigital Media CenterUniversity of Minnesota

  2. Evaluation Plan • Evaluation: A formal appraisal of the quality of an educational phenomenon (Popham, 1993) • Gathering evidence to determine how well the innovation is “working”. • What does the grant say about the evaluation plan?

  3. Foci of the Bush Grant Course-level Evaluation • Student Outcomes • Learning Process: Student Engagement Reflective Learning Responsible Learning

  4. The Bush grant re: Evidence • Grades—ABC and DFW rates (outcomes). • Faculty Reflection Logs and conversations with consultants (process) • Surveys e.g. compare engagement across groups: with/without innovation and/or: • Surveys to compare across time, e.g. before/after the innovation.

  5. Survey Items: Closed-ended (Examples) • Student Engagement • Mostly, I come to class because I want the certificate/degree. • Most of the time, I enjoy this class. • Reflective Learning • I enjoy applying theories or concepts from this class to new situations. • I sometimes find myself thinking about the lecture after it has ended. • Responsible Learning • I usually come to class prepared. • I usually review my lecture notes after class.

  6. Survey Items: Open-ended • What are the most important benefits of this innovation for you? • What are the drawbacks? • What kind of problems have you had with the technology? Be specific. • What suggestions do you have for improvement?

  7. Tailoring the Evaluation Plan to Meet Your Needs • What do you want to know? • Collect the kind of evidence to find out what you want to know, for example: • Business: experimental design with two treatment groups and long pre- and post-intervention surveys. • Architecture: a series of short surveys, interviews and observations, then analyze them for recurring themes.

  8. Optional Evaluation Methods • Scoring Rubrics for student assignments • Class Observation: take notes, analyze them for themes, can inform surveys. • Narrative: To learn about learners’ reflective practices. Triangulation is important. • Interviews: Open-ended items, can deviate, tape-record and analyze for themes. • Focus groups: Validity checks.

  9. Implementing the Plan • Decide what you want to know • Choose your preferred methods/tools • Write surveys and/or interview protocols • Collect data • Data entry: training? • Data analysis

  10. Tools: Scoring Rubrics • “Scoring rubrics…guide the analysis of the products and/or processes of students' efforts…[and] provide a description of what is expected at each score level.” (Moskal, 2000) • a systematic way of evaluating qualitative data; a way of reducing subjectivity

  11. Tools: Scoring Rubrics Development process: • break general concepts down into evaluation criteria • develop descriptions of degrees to which, and ways in which, the criteria can be satisfied • conduct pilot testing on sample data

  12. Tools: Surveys Survey questions should: • be interpreted by respondents in the same way; • ask for information that respondents are able to provide; • provide information that is interpretable by you.

  13. Tools: Questionnaires Time estimates:How many hours per day do you typically study?  Less than 1 hour  1 – 1.5 hours  1.5 – 2 hours  2 - 2.5 hours  More than 2.5 hours  Less than 2.5 hours  2.5 – 3 hours  3 – 3.5 hours  3.5 – 4 hours  More than 4 hours

  14. Tools: Questionnaires Scale issues:How would you rate your instructor’s knowledge of the subject matter of this class?  poor  fair  adequate  good  very good  exceptional

  15. Tools: Questionnaires Scale issues:How effective were the small group discussions in helping you to learn the course material?  very effective  effective  don’t know  ineffective  very ineffective

  16. Tools: Questionnaires Item order effects:Which of the following activities helped you to learn the course material?  Lectures  Large group discussions  Small group discussions  Group assignments  Course readings  Studying for quizzes

  17. Tools: Questionnaires Open versus closed-ended questions:How easy or difficult to use did you find each of the components of our course website?

  18. Tools: Questionnaires Open versus closed-ended questions:How easy or difficult to use did you find these components of our course website? Discussions  very easy Quizzes  easy Content modules  difficult Audio files  very difficult PowerPoint files etc.

  19. Tools: Questionnaires Interpretability: Which of these statements best describes your professor?  very good teacher, but not very approachable  about the same on approachability and teaching quality  very approachable, but not a very good teacher  a born teacher  extremely approachable

More Related