290 likes | 478 Views
Modeling Education with Work Domain Analysis: A Work in Progress. Lee Nickles September 11, 2002. Outline. Defining Educational Evaluation Obstacles to Performing an Evaluation Work Domain Analysis as Solution Process of Building the Model. What is Educational Evaluation?.
E N D
Modeling Education with Work Domain Analysis: A Work in Progress Lee Nickles September 11, 2002
Outline • Defining Educational Evaluation • Obstacles to Performing an Evaluation • Work Domain Analysis as Solution • Process of Building the Model
What is Educational Evaluation? “Evaluation in the context of educational systems is briefly defined as examining the effectiveness of an educational system (or component of that system) in meeting learning and teaching goals.” Based on: (Bloom et al, 1971; Stevens, et al [NSF], 1993; Walker, 1997)
Formative vs. Summative Evaluation • Formative • During development and implementation • Examines system to reveal problems and improvements • Summative • At the end of the system’s life • Examines overall success of the system • My focus: Formative
Formative Evaluation is a “Good Thing™” • Similar to Quality Control, leads to continuous improvement • Changes can be made in the current life cycle of the system
Evaluation Obstacles (1) • Education systems have multiple, interacting components • Work together to produce learning • Isolating which component produced what or (how much) learning is difficult • Testing each individually is too great a burden • Components build on each other...
Evaluation Obstacles (2) • There is a lack of validated measurement instruments for evaluation • There may not be a test that analyzes whether or not your specific web site produced learning • You may not have the time or resources to generate a valid measurement
Validation Solution • Measurement Triangulation • Use multiple measurements of different types to measure the thing you want to evaluate • Subjective Opinion/Attitude • Performance • Process/Behavior • If they give similar/confirming readings, you can have greater confidence in your result
Triangulation Problem • How do we apply this practically? • That is, how do you know your measurements are triangulating properly? • They are measuring slightly different things • Need a structured way to apply triangulation
Proposed Solution: Work Domain Analysis • Interacting components • Specifically examines tightly coupled systems, with many relationships between the components • Structure to apply triangulation • Provides the abstraction hierarchy framework to guide measurement triangulation
Work Domain Analysis - Abstraction Hierarchy • Rasmussen, 1985 • Modeling tool • Broad and deep view of the system • Way to represent the levels of abstraction of a system together and how components, functions, and goals connect to each other
Concrete Abstract Physical Objects System Functions Goals and Objectives Levels of System Abstraction Can describe a system at levels of abstraction from the physical
Means-Ends Relationships Items in one level of abstraction are related to others by means-ends relationships Ends: Level 1 tells WHY Level 2 is performed 1: Goals 2: Functions Means: Level 3 tells HOW Level 2 is accomplished 3: Physical Form
Education and the AH • Problem: the AH hasn’t really been applied thoroughly to intentional systems • Intentional systems: systems where actions are based on the user’s intentions more than physical constraints • Some say it can’t be done (Wong et al, 1998) • Same say it can (Hajdukiewicz et al, 1999) • No really good examples in the literature
Building an Abstraction Hierarchy to Describe Education • I’ve gone through this process for some time (too long?) • Many iterations, back and forth with Dr. Pritchett • NOT Done! • Process may be as informative as the product, so here is (approximately) the process...
Early Versions • Considered various ways an AH has been represented • Separating very different domains that are part of the same system • Think about how cognitive functions, social activities, and information flow should be represented • Systems based on user intention rather than physical constraints
Early Framework - Domains Difficult to Separate/Categorize things
Process Control Ongoing process Goal: keep process stable Change is bad Education Defined start/end of educational system Goal: produce learning in the students Change is desired I noticed something... AH has typically been applied to process control
Education as Pragmatic Change Over Time Is it this simple? Not very general!
Education as Pragmatic Domains Interesting, but is the domain only interesting at the top level? Did we lose the element of change?
Three Dimensional AH? • Maybe this was going to take multiple dimensions of domains to represent • About this time, I switched from pen and paper to Excel to create AHs and used the IT Web site for ISyE 4009 as a case
Multi-Dimensional, “Unit” Based What is a unit?
Multi-Dimensional, Week Based This is getting somewhere! But weeks are not general.
Multi-Dimensional, Assessment Based An assessment (homework, exam, project, etc.) as an essential part of course structure
Where do I go now? • Maybe stop now and model an entire course with this method • Are there things we’re leaving out? • What can we leave out? • Consider, how useful is this for evaluation? • Can we easily generate an AH for a course? • Is it general enough to apply broadly? • Is it specific enough to be useful?