280 likes | 453 Views
OKAIRP Fall Conference. NSU @ Broken Arrow November 4, 2005. Designing Methods for Determining Educational Gain: Involving Faculty in Assessment. Mark L. Giese Northeastern State University Tahlequah, Oklahoma. Measuring Program Dispositions as a Model. Knowledge Skills Dispositions.
E N D
OKAIRP Fall Conference NSU @ Broken Arrow November 4, 2005
Designing Methods for Determining Educational Gain: Involving Faculty in Assessment Mark L. Giese Northeastern State University Tahlequah, Oklahoma
Measuring Program Dispositions as a Model • Knowledge • Skills • Dispositions
What are dispositions? • What is the big deal about dispositions? • Dispositions are: • Concepts we inherently value as a profession. • Esoteric to the discipline.
What are dispositions in law? • Ethical • Passion for service • Accessible
What are dispositions in Education? In Health and Kinesiology, dispositions may include: • Hard work • Teamwork • Our students or clients can improve with practice
Who is involved? • Use all persons with a stake: • Faculty • Administrators • Let faculty make mistakes
Why measure? • Looks at the Affective Learning Domain. • In the past, typical areas of measurement have only included knowledge and skills. • Accrediting agencies (learned societies) require measurement in this domain.
Steps in developing a measurement tool for your unit • Planning • Review the literature and ask your friends what they are doing. • Decide on which dispositions to measure (committee). • Decide if you intend to use this exercise as an exit competency or if you are looking for a change from entry to exit. • Determine an a priori standard.
Steps in developing a measurement tool for your unit • Implementation • Devise a measurement tool (rubric) • Collect your data • Analyze your data • Determine if an intervention is necessary • REMEASURE!
Examples • Example of a college rubric for measuring dispositions in pre-service teacher education candidates in the COE at NSU.
Examples • Example of a Sociology department rubric for measuring dispositions: • Attempt to measure “Social Distance”
Importance • Measuring dispositions is important because it gives us another piece of information to determine our effectiveness and quality.
Examples of Presentations on Social Distance • “Social Distance as a Disposition of Teacher Candidates.” Presented at AACTE, Chicago, Illinois, February 8, 2004.
Examples of Presentations on Social Distance • “Attitudes Towards Social Distance of Pre-Service Teacher Education Students.” Presented at Higher Education Research Day, Edmond, Oklahoma, November 2003.
Examples of Presentations on Social Distance • “Attitudes Towards Social Distance of Pre-Service Teacher Education Students.” Presented at the Oklahoma Association of Colleges for Teacher Education Conference, Norman, Oklahoma, November 9, 2000.
Two Types of Measurement • One-time (barrier) • Demonstrate program effectiveness (gain)
Measurement Goals • One Time: Graduation requirement etc. • Using a pre/post test design in an attempt to determine program gain. May be done at entry and exit of any major.
Does your data fit the model? • Parametric (t, ANOVA) • Non-parametric
Determining Program Gain • Measure at pretest. • Measure at post test. • Determine the difference. • Use hypothesis testing to determine if a significant difference exists.
How ANCOVA works • The variable under study is the dependent variable and the covariate is the “other” data point. Both must be known for each subject (matched).
How ANCOVA works • The influence of the covariate is removed using a straight-forward linear regression method. • The ANCOVA produces an adjusted post test score for each subject and an adjusted mean as well.
ANCOVA Example • Three Methods of teaching French: • Pretest……..instruction……Post test • Measures of intelligence available • Remove the effect of intelligence so the French method may be the cause of the change from pre/post score.
ANCOVA Group 1 Pre Post . . . . . . . . . . . . . . . . X X Group 3 Pre Post . . . . . . . . . . . . . . . . X X Group 2 Pre Post . . . . . . . . . . . . . . . . X X
Assumptions of ANOVA • Homogenity of variance • Equal distribution • Random sampling • Additive effects
Extra Assumptions of ANCOVA • IV does not influence the covariate variable • Equal regression slopes • Linearity • r of .60 (Shavelson, 1996)
Review • Ask faculty if they need/desire to measure dispositions. • Assist with choosing an instrument and measurement model. • Assist in determining gain and the magnitude of the effect.
Questions ??QUESTIONS??