Evaluation Test

1. Evaluation Test Justin K. ReeveEDTECH 505-4173 Dr. Ross Perkins

2. Reflection It's useful to know the different models that are available for evaluation, and being able to identify which one is best for a situation. The goals, activities, and evaluation procedures in the programs play a crucial role in how the evaluation is conducted. Efficiency, efficacy, and impact should be measured across any program. There are a number of programs in our district that currently lack any evaluation, or for which we collect a minimal amount of information that doesn't follow a defined, systematic process. At best, a simple survey is supplied at the end of the programs (for example, see http://wsd.net/evolve-evaluation), but no formative evaluation is conducted during the training sessions themselves, and the survey results are never reviewed by the training coordinator. Until these deficiencies are resolved, our evaluation process is ineffective. The knowledge I've learned in the class so far has given me the ability to identify these weak points.

4. Rationale for Evaluation An evaluation produces statistical data that helps sponsors and staff members choose appropriate resources, identify areas in need of adjustment, become aware of program strengths and weaknesses, and understand the outcome of actions conducted in a program. Photo Source: http://bit.ly/aFQPDC

6. Evaluation and EdTech Evaluation enables educational technologists to choose, improve, and measure the instruction and learning process. Adopting a defined, systematic process gives the evaluator a vision of the aspects of a program working together. Photo Source: http://www.flickr.com/photos/17258892@N05/2588347668/

8. Research and Evaluation Research uses systematic processes to determine general knowledge, whether through controlled variables and the scientific method, or qualitative methods that have a rigorous base in scientific inquiry. Evaluation allows the evaluator and program’s stakeholders to better understand the internal processes of the program. Exterior of Watch (Research) Photo Source: http://bit.ly/dkUdot Interior of Watch (Evaluation) Photo Source: http://bit.ly/cTrz6z

10. Efficiency, Efficacy, and Impact Efficiency refers to the balance between time and resources spent on a program, and how well the returns exceed the investment. Efficacy refers to how well objectives and desired needs are attained. Impact refers to how participants change their behavior on a permanent basis. Scales Photo Source: http://bit.ly/bd3bmw Energy-Efficient Light Bulb Photo Source: http://bit.ly/a6tYvN Change Photo Source: http://bit.ly/9tqUUv

12. Impact of Evaluation An evaluation can identify connections between goals and needs and implement focused activities to these ends. Formative and summative evaluation results identify processes that work effectively, and decisions that can be made for necessary program modifications. Photo Source: Created by Justin K. Reeve

14. Goal-Free Model A data collection- and analysis-oriented model, ideal for qualitative evaluations, in which program aspects come from observation of the program, without regard to goals and objectives. The evaluator uses the data to infer conclusions of the program's impact on clients' needs. Photo Source: http://arcmit01.uncw.edu/jonesi/Images/ScrivenGoalFreeEvaluation.png

16. Kirkpatrick's Four-Level Model A model in which students evaluate the program's objectives, value, and relevance (Reaction), and participate in objective-based assessments (Learning). The model evaluates the impact of the program on participants (Behavior) and the organization for the decision-makers (Results). Photo Source: Created by Justin K. Reeve

18. Qualitative vs. Quantitative Qualitative data are collected with observations, interviews, surveys, and case studies. Data are subjectively interpreted. Quantitative data are collected with tests, instruments, and other quantifiable measuring tools, and clearly state numerical facts and may be used to predict outcomes. Photo Source: Created by Justin K. Reeve

20. Levels of Data Nominal data consists of mutually exclusive unordered categories (black/white). Ordinal data consists of ranked, ordered items, without numerical differences between the items (top 10 list). Interval data is ranked numerically without an absolute zero on the scale (temperature). Ratio data is ranked numerically with an absolute zero on the scale (length). Ordinal Photo Source: http://bit.ly/9QSadT Ratio Photo Source: http://bit.ly/aDQ5pM Nominal Photo Source: Created by J. Reeve Interval Photo Source: http://bit.ly/90pIEc

22. Data Instruments Interview: A conversation which obtain considerable information, but with poor comparison across instances. Scale: A response in ranked intervals; a scale should be well-organized to prevent evaluator bias. Test: An assessment tool which produces numerical data. Observation: Constructing narratives of perception from predetermined criteria. Interview Photo Source: http://bit.ly/ddPXQx Scale Photo Source: Personal Collection Test Photo Source: http://bit.ly/bfrQbW Observation Photo Source: http://bit.ly/aXQawR

24. Formative vs. Summative Formative evaluation tracks a program as it happens. Real-time adjustments can be made to the program from these results. Summative evaluation occurs after the program and gauges the overall effectiveness and objective attainment. Parent Assisting with Homework Photo Source: http://is.gd/frHYn Parent Reviewing Homework Photo Source: http://is.gd/frHRF

26. Samples and Populations The population represents all potential participants that meet the criteria for the intended evaluation. The sample is a selection of participants, typically random, that is representative of the target population, or represents a desired section of the population. Photo Source: http://is.gd/frvgb

28. Validity and Reliability Reliability refers to data consistency, and the ability to produce the same results when an analysis is repeated. Validity refers to data where an accurate conclusion follows from the premises. Evaluation data should be both reliable and valid. Photo Source: Created by Justin K. Reeve

30. Independent and Dependent Variables are elements of an evaluation that may take on different characteristics or values. Dependent variables are uncontrolled, and strictly observed or studied. Independent variables are controlled by the evaluator or participants, and manipulated when necessary. Photo Source: http://www.flickr.com/photos/tentangqatar/4385556714/

32. Criterion- vs. Norm-Referenced Criterion-referenced tests measure individual learning and personal achievement against curricular standards, with clear expectations of participant performance. Norm-referenced tests rank how students compare to others in the population, without setting goals or using standards of achievement. Photo Sources: http://bit.ly/b2Tv07     http://bit.ly/c2CQgf

33. Activity Evaluation The idea of using visual metaphors as an assessment is an interesting one. I'd actually like to see some data comparing the benefits and disadvantages of this form of assessment vs. a traditional test. I personally struggle a little with coming up with actual metaphors and not just visual examples of the concepts being assessed (hopefully this isn't too evident in these slides), but that gives me a chance to improve my abstract thinking skills. I would like it if the activity's instructions explicitly gave extra allowed word space to explain each visual metaphor on the following slide, so I could better justify my choice of images (though maybe this defeats the point of coming up with a strong visual metaphor).