CAREER-TECHNICAL EDUCATION AND ASSESSMENT

1. CAREER-TECHNICAL EDUCATION AND ASSESSMENT PAST, PRESENT, POSSIBLE

2. James T. Austin Assessment and Evaluation Services Center on Education & Training for Employment College of Education The Ohio State University: Big 10 Champions

3. AGENDA Assessment Defined Two Worlds: Large-Scale & Classroom Perspectives Past: Getting To The Way We Were Present: High Stakes Testing Possible: Expansions In Multiple Directions Example Applications Using Graphic Designer

4. TWO CULTURES Large Scale Assessment (district level & up) What states/districts/nations do Prototype of assessment to public Few players (NCS, Harcourt, CTB/McGraw-Hill) Norm- and/or criterion-referenced assessments Quality control known and used (reliability, validity) Classroom Assessment What teachers do Phye�s handbook covers this domain thoroughly Standards of assessment competence are available

5. ASSESSMENT TRIANGLE Cognition: a feature of thinking or performing Knowledge representation, skill development EX: mental models of software & project, artistic competence How do mental models change as novices become proficient? Observation: a situation and stimulus Task situations that require observable responses EX: entry-level tasks as a part of graphic design projects Design & creation of brochures for an ad campaign Interpretation: inference linking cognition-observation Methods for scoring thinking/performing from observations EX: statistical model for performance assessments

6. PURPOSES SERVED Assessment conducted to Assist learning Measure individual achievement Evaluate programs

7. SAMPLES OF ASSESSMENT Think of CTE students � 1. in class in the �graphic design� lab What is the error in this brochure layout? Can you fix it? 2. in teams at SkillsUSA/VICA competition Design a logo for Organization X under constraints. 3. answering high stakes writing prompt Persuade school board to adopt a stand on X. 4. taking a graphics certification exam on-line Modify given artwork according to change ticket.

8. ASSESSMENT PAST:HOW DID WE GET HERE �? Tests used throughout history Biblical & Chinese examples Binet, Army alpha/beta tests marked start Expansion of �testing as technology� 1920-2000 Military testing for recruit selection & placement Educational testing to assess achievement (Resnick) Business testing for personnel selection Certification testing for a range of professions Testing is a social institution (Hanson, 1993)

9. CTE TESTING PAST � Trade tests developed during WW I differentiated novices, apprentices, & experts Permitted evaluation of claims of recruits Prosser and Quigley well aware of testing Vocational Education in Democracy (1949) Intelligence, trade, and entry tests defined Testing varies for entry vs experienced

10. O.T.A. 1994 REPORT Comprehensive report in parts Classified state policies for skills assessment 18 states mandated local written testing 15 states mandated testing without guidance 10 states encouraged testing guidance 8 states had no specific policy Assessment targets: Academic & occupational (vocational, generic workplace, & broad technical skills)

11. O.T.A. CONCLUSIONS (1) Assessment in secondary CTE differed � best practices like alternative forms of assessment just being explored for rest of education � but quality varies greatly. (2) In CTE, testing integral to learning (3) Critical issues in performance assessment: Comparability of judgments Judgment-standard correspondence (4) Critical issues in written testing: Relevance of items to capabilities for performance Effects of testing on teaching & learning

12. ASSESSMENT PRESENT � Testing now major part of accountability Systems concepts key to appropriate high stakes testing Linn (2001) models design and evaluation of such systems: reliable, valid, & fair Standards for Educational Accountability criteria National Board for Educational Testing & Public Policy as an independent auditor

13. ASSESSMENT PRESENT � Testing increasing for some time & into future No Child Left Behind Act: Annual testing Grades 3-8, interventions/consequences HST increasing for students and teachers Making Standards Matter (AFT, 2001) documents prevalence Miller et al. (2001) review teacher testing for NRC Threat or opportunity? Tradeoffs? Opposition, but Cizek & Phelps believe HST is an opportunity

14. PRESENT � Alternative models seek full status in both worlds Stecher et al. (1996) reviewed for CTE field Custer (2000) also surveyed for CTE field Vermont, Maryland, Kentucky Koretz et al. (1991) reviewed Vermont large-scale experience with portfolios Herman (1997) reviewed large scale alternative assessment

15. PRESENT � Technology: always present, increasing Testing in general Bennett (1999) Distance assessment Kingsbury & Houser (1999) discuss one system Major testing firms rolling out (Sylvan Prometric) Much more � Scoring & reporting facilities Roll-up for Perkins III Consolidated Annual Report

16. LARGE-SCALE CTE PRACTICES HSTW ASSESSMENT Used to monitor progress of this reform since 1987 Operated by ETS for SREB (given in even years) Modeled on standards in Math, Language, & Science NOCTI Long-time player (since 1973) Combines written & performance formats Student and teacher competency assessments Now evaluating on-line assessment systems

17. TYPICAL CTE PRACTICES DISTANCE DELIVERY Mostly first generation, meaning NOT adaptive testing Several systems use QuestionMark software OKLAHOMA www.okcareertech.org/testing/online.htm OHIO Tech-Prep modular assessment under development ITWorks standards represent collaboration between government & business (www.itworks-ohio.org) Modular organization, ~30 items per module �Core� modules cut across 4 pathways or strands IM, ISS, NS, & PSD are the occupational areas Other module content is specific or cuts across a subset (1-3)

18. THEMES: PAST & PRESENT Dissent since 1920s (Madaus, 1995, 1997) �Tyranny of testing� in 1962 Protests increasing (Sacks, Kohn, Popham) Reforms Competency testing: 1970s (Haney, 1978) Standards to fore: 1990s (Resnick & Wirt, 1996) High Stakes Testing (Heubert & Hauser, 1999)

19. THEMES � Standards for educational & psychological testing (1999) Just revised by 3 professional groups Ch 13 addresses educational applications 19 standards (validation for each purpose) Validation as consequences: intended/unintended High stakes tests & retention Industry certification tests & student engagement

20. ASSESSMENT: POSSIBLE What might 21st century bring to CTE? Possible futures applied to graphic design rethinking & redesigning CTE assessment alternative assessments computerized distance assessment

21. Scenario #1: RETHINK Represents most change in �way things are done� Has the most scientifically-based support, also costs the most National Research Council review panel Pellegrino et al. (2001) Knowing what students know Merges cognition, measurement, & technology Cognition, observation, interpretation form assessment triangle Called for 3 types of alignment vertical (levels of educational system) horizontal (across curriculum, instruction, & assessment) temporal (across student�s �educational� career) Tennessee VAAS is a model, with some flaws

22. APPLICATION Need assessment triangle: C-O-I COGNITION: Schemas, mental models, expert-novice OBSERVATION Create situations & responses to tap INTERPRETATION Select models to link C & O

23. APPLICATION TO CTE Details require an occupation I chose graphic design Fortuitously, Techniques March issue arrived Consciously looking for a sample occupation Emerging, entrepreneurial Traditional (advertising) and IT (website design) Blend of artistic & technology inputs Next slides provide background

24. GRAPHIC DESIGNERS Design/create graphics to meet specific commercial or promotional needs, such as displays or logos � use a variety of mediums to achieve artistic or decorative effect (O*Net-SOC Code = 27.1024.00) Work domains: prepress, press, binding

25. GRAPHIC DESIGN � National Council for Skill Standards (www.ncss.org) Used �common framework� of National Skill Standards Board Standards & certifications available Electronic Imaging (pre-press) Sheetfed and Web Offset Press (press) Flexographic Press (press) Finishing and Distribution (binding)

26. SOME PROGRAMS Secondary: Metro Tech, OK; Thomas Edison, MD Post-secondary: Tunxis CC, CT Extended: Cuyahoga Valley Career Center, OH Digital Design Program (Exemplary program, 2000) Interactive Media pathway of Tech-Prep IT program 2 years (48 students), screened by guidance/recruiting team Art plus computers = powerful combination Look at the waiting lists!! 100% retention rate, 76% placement rate

27. POSSIBLE � �Microworlds" built for the graphic designer (i.e., a simulated work environment with individual & team elements that an entire class would enter for a semester or so) Portfolios of student work Across the (usually) 2 years of a program Performance assessments start with smaller tasks of entry-level professional & synthesize to integrated project level

28. POSSIBLE � Some drawbacks: Many models are fairly domain-specific Will require extensive occupational analyses Transitions in Work and Learning (NRC, 1997) Construct/validate mental models, contextual situations Development of the C-O-I assessment triangle elements Creation of standards & curriculum

29. Scenario #2: ALTERNATIVE ASSESSMENT Wiggins has been a consistent advocate Authentic (1989 book) & educative (1998 book) Some alternative assessments in CTE (1) Career-Technical Assessment Program (2) Kentucky Instructional Results Information System (3) Laborers-AGC Environmental Training & Certification (4) National Board for Professional Teaching Standards (5) Oklahoma Competency-Based Testing Program (6) VICA National Competition (7) NOCTI TOCT & SOCAT

30. WIGGINS� AUTHENTICITY Authenticity derives from match of assessments to: (1) "worthy problems or questions of importance" (2) faithful representation of contexts facing workers in a field (3) non-routine, multi-stage tasks (4) tasks that require generating a quality product, performance (5) transparent or demystified criteria (6) interactions between assessor and assessee (7) effect of process & product/performance determines quality (8) trained assessor judgment (9) the search for patterns of response in diverse settings Tradeoffs associated with achieving authenticity

31. AUTHENTIC � Applied Psychological Measurement Performance assessment special issue (Hambleton, 2000) Miller & Linn on validation Hambleton, Jaeger, & Plake on standards Clauser on developing & scoring

32. POSSIBLE � Graphic Design uses portfolios Would portfolios change? Additional assessments? Performance assessments Projects Presentations (to advisory committee??)

33. Scenario #3COMPUTER-DISTANCE ASSESSMENT Computers used to deliver assessments since 1960s, especially influential in adaptive testing and format expansions (multimedia) Emergence of networks brings distance delivery to the forefront: RAND Issue Paper (Klein & Hamilton, 1999) NGA position paper on distance assessment (2002)

34. COMPUTERIZED � Curriculum & instruction: Evolution from correspondence and TV/video courses to distance education Assessment: Evolution from individual & group paper-pencil to authentic, computer networked Computer performance assessment of problem-solving O�Neil (1999) special issue of Computers in Human Behavior Combines authentic & computer/distance assessments Consider graphic design scenario

35. POSSIBLE � Graphics Design profession SkillsUSA evaluates 5 areas: layout and design; film image assembly; plate making; offset duplication; written test of common graphic comm issues Contestants compose final art using a desktop computer and software system

36. POSSIBLE � Graphic Design application Extracted from British distance course, web-site design (http://www.it.bton.ac.uk/staff/lp22/CP303) Deliverable 1: an initial requirements description Deliverable 2: brief analysis of comparable sites Deliverable 3: user class analysis, in the form of personas Deliverable 4: task scenarios Deliverable 5: user object model Deliverable 6: content structure diagram Deliverable 7: site structure diagram Deliverable 8: initial evaluation results Deliverable 9: storyboards Deliverable 10: lo-fi prototype Deliverable 11: evaluation report Deliverable 12: maintenance document Deliverable 13: design document

37. HOW IS ASSESSMENT DONE? WestEd distinction from CTAP Cumulative = developed over time On-demand = real-time, �in the spotlight� Portfolios are cumulative Useful internally (student evaluation) Useful externally (job searches) Other formats can be on-demand Sudden presentation to client

38. HOW COULD WE ASSESS ? Three Possible Futures Rethinking & redesigning Alternative assessments Computerized & distance assessment Apply to Both Worlds of Assessment

39. CONCLUSIONS Assessment for CTE viewed from several perspectives: Two worlds of classroom and large-scale assessment History: past, present, potential Exemplary programs of assessment What are some important goals to be debated? Can/should the two worlds be blended? What constitutes scientifically-based research support? How can tradeoffs be considered? Balance claimed gains from new assessments against costs What are actions needed to reach consensual goals? Assessment literacy for CTE teachers and administrators

40. ASSESSMENT LITERACY Concept articulated by Richard Stiggins Lacking for many educators (1991), but an immense potential upside (2000) for both worlds Large scale buy-in and preparation for students Classroom assessment improvements Teachers: what capabilities are required? Leading edge Administrators: what capabilities are required Support and guidance