David G. Payne, PhD Senior Executive Director, Client Relations Higher Education Division

1. SUNY Conference Syracuse, NY April 27-28, 2005 SUNY General Education Learning Outcomes: Strengthened Campus-based Assessment David G. Payne, PhDSenior Executive Director, Client Relations Higher Education Division Patricia Connor Director of Higher Education Initiatives Educational Testing Service

2. Overview • ETS • Who we are • What we bring to the table • Academic Profile and SUNY General Education Learning Outcomes Assessment • Customized SUNY General Education Learning Outcomes Assessment Modules • Key Features of the Customized SUNY Assessments

3. Educational Testing Service • Founded in 1947 by • American Council on Education • Carnegie Foundation for the Advancement of Teaching • College Entrance Examination Board. • Today, as the world’s premier educational measurement and research organization, ETS develops and annually administers more than 11 million tests. • As a Not for Profit organization, ETS has a specific Mission.

4. The ETS Mission • To help advance quality and equity in education by providing fair and valid assessments, research and related services. • Our products and services measure knowledge and skills, promote learning and performance, and support education and professional development for all people worldwide.

5. ETS’s Largest Exams and Annual Volumes • TOEFL – 780,000 • GRE – 500,000 • GMAT – 229,000 • SAT I – 3.1 million • AP – 1.4 million • PRAXIS – 770,000

6. What else does ETS Provide? • Products and services to support Institutional and Curriculum Development for Higher Education • Academic Profile: General Education Test • Major Field Tests: Outcomes Assessments in 16 fields of study • ICT Literacy: Information and Communication Technology Assessment • Criterion: On-line Writing Evaluation • Free or Low-cost Test Preparation Materials for all Examinees • Mission-related services

7. Academic Profile and SUNY Learning Outcomes

8. Academic Profile Test of general education that measures reading, writing, math and critical thinking in the context of the humanities, social sciences and natural sciences.

9. Academic Profile Skill Scores (Norm-Referenced Scaled Scores) • 4 Skill Areas in Standard Form • Critical Thinking • Reading • Writing • Mathematics • Optional 40 minute Essay

10. Academic Profile:Standard Form • Covers all four skill areas in one test • Optional Essay • 108 multiple choice questions • 120 minutes • Generates total and content area scores for both individuals and groups • Provides demographic data • Allows for locally-written questions

11. Academic Profile and SUNY Learning Outcomes • Mathematics • 5 specific skill areas • Basic Written Communication • 3 specific skill areas • Critical Thinking • 2 specific skill areas

12. Mapping Academic Profile to SUNY Learning Outcomes • Academic Profile Standard Form • 108 questions • Reading • Writing • Mathematics • Critical Thinking • SUNY General Education • Assessment Modules • 40 minutes • 36 Questions • OR • 40 minutes Essay • Paper and Pencil or • Online Delivery • Flexible Score Reporting • Academic Profile Optional Essay • 40 minutes • Range of Topics • Online Delivery • Holistic Scoring

13. Academic Profile and SUNY Learning Outcomes • Mathematics • 4 specific skill areas • Basic Written Communication • 3 specific skill areas • Critical Thinking • 2 specific skill areas

14. Academic Profile for SUNY General Education Learning Outcomes Assessment • Tests Available January 2006 • Report Formats • Paper • Electronic • Normative data • National Norms • SUNY Campuses (depending on volumes)

15. Advantages In use at many institutions nationwide Nationally normed Valid Reliable Paper and on-line versions Web-based score reporting Group and subgroup reports Disadvantages Not precisely mapped to SUNY Learning Outcomes Academic Profile for SUNY

16. Customized SUNY Learning Outcomes Assessment

17. Four Guiding Principles for Customized SUNY Assessments • Customized solution to match SUNY General Education Learning Outcomes • Subscores for as many skill/competencies as possible • Ease and efficiency of administration and reporting • Students • Faculty • Administrators • Simple, clear score reports that mirror the SUNY outcomes • ETS quality and experience applied to the SUNY needs

18. SUNY Mathematics Module: Skills Assessed • Interpret and draw inferences from mathematical models such as formulas, graphs, tables and schematics • Represent mathematical information symbolically, visually, numerically and verbally • Use arithmetical, algebraic, geometric and statistical methods to solve problems • Estimate and check mathematical results for reasonableness, determine alternatives and select optimal results • Recognize the limits of mathematical and statistical methods

19. Customized SUNY Mathematics Module:Key Features • Careful Mapping of Skills to Assessment • Format • 36 Multiple Choice Items • Paper and Pencil Format • 40 Minutes • Scores Reported • Individual Student Overall Scores • 4 Subscores

20. Sample Mathematics Question Pat purchased 4 large picture frames for $20.00 each, 100 medium picture frames for $10.00 each, and 8 small picture frames for $5.00 each. What was the average (arithmetic mean) price per picture per frame? • $7.50 • 10.00* • 11.67 • 15.00

21. Sample Mathematics Question According to the information in the table, what is the range of the cost per night for a room for 1 person? • $25 • $50 • $62 • $79*

22. Produce coherent texts within common college-level written forms Demonstrate the ability to revise and improve such texts Research a topic, develop an argument, and organize supporting details Customized SUNY Basic Written Communication Module: Skills Assessed

23. Customized SUNY Basic Written Communication Module: Key Features • Careful Mapping of Skills to Assessment • Tasks and Format • Short essay (Produce Text) • Extended revising and editing of written work tasks • Paragraphs and Short Essays • Students will do corrections and make revisions • Paper and Pencil Format • 45-50 Minutes • Scores Reported • Individual Student Overall Scores • 2 Subscores • Text Production • Revisions and Corrections

24. Sample Short Essay Task Some people have suggested that everyone between the ages of 18 and 21 should be required to perform one year of community or government service, such as in the Peace Corps, Environmental Conservancy Corps, a hospital, the military, a rural or inner-city school, or other community outreach projects. Should government or community service be required of 18-21 year olds? Support your point of view with reasons from your own experience, observation, or reading.

25. Sample Revision Question “Being a female jockey, she was often interviewed.” Rewrite, beginning with She was often interviewed The next words will be • “on account of she was” • “because she was”* • “by her being” • “being as she was”

26. Critical thinking and analytical writing skills, including the ability to Identify, analyze, and evaluate arguments as they occur in their own or other’s work; and Develop well-reasoned arguments Customized SUNY Critical Thinking Module:Skills Assessed

27. Customized SUNY Critical Thinking Module: Key Features • Careful Mapping of Skills to Assessment • Tasks and Format • 2 Short Essays (20 - 25 minutes each) • “Develop an Argument” • “Evaluate an Argument” • Task Characteristics • Paper and pencil format • No specialized knowledge required

28. Sample Develop Argument Task Directions: Present your perspective on the issue below, using relevant reasons and/or examples to support your views. The media (books, film, music, television, for example) tend to create rather than reflect the values of society.

29. Sample Evaluate Argument Task Directions: Discuss how well reasoned you find this argument.Hospital statistics regarding people who go to the emergency room after roller-skating accidents indicate the need for more protective equipment. Within this group of people, 75 percent of those who had accidents in streets or parking lots were not wearing any protective clothing (helmets, knee pads, etc.) or any light-reflecting material (clip-on lights, glow-in-the-dark wrist pads, etc.). Clearly, these statistics indicate that by investing in high-quality protective gear and reflective equipment, roller skaters will greatly reduce their risk of being severely injured in an accident.

30. Customized SUNY Critical Thinking Module: Key Features (continued) • Scoring • Essays scored on holistic basis • Scoring can be either human or computer (to be determined based on input from SUNY) • Scores Reported • Individual overall score (based on both essays) • 2 Subscores • Develop an Argument • Evaluate an Argument

31. Score Reporting

32. Sample Individual Student Score Distribution

33. Sample Individual Student Score Distribution

34. Sample MathematicsSubscore Distribution

35. Customized SUNY Mathematics Module: Subscores • Interpret and draw inferences from mathematical models such as formulas, graphs, tables and schematics • Represent mathematical information symbolically, visually, numerically and verbally • Use arithmetical, algebraic, geometric and statistical methods to solve problems • Estimate and check mathematical results for reasonableness, determine alternatives and select optimal results • Recognize the limits of mathematical and statistical methods

36. Produce coherent texts within common college-level written forms Demonstrate the ability to revise and improve such texts Research a topic, develop an argument, and organize supporting details Customized SUNY Basic Written Communication Module: Skills Assessed

37. Assesses critical thinking and analytical writing skills, including the ability to Identify, analyze, and evaluate arguments as they occur in their own or other’s work Develop well-reasoned arguments Customized SUNY Critical Thinking Module:Skills Assessed

38. SummaryandSupplemental Information

39. Customized SUNY General Education Learning Outcomes Assessment • 3 Customized Assessments • Mapped to SUNY Learning Outcomes • Each module is independent • Overall Scores and Subscores • Easily administered within a class period • Paper and Pencil Format • Demographic data options • Platform that can support future addition of new modules • Option for locally developed questions (to be determined with input from SUNY Central Administration and and individual campuses)

40. Customized SUNY General Education Learning Outcomes Assessment • SUNY Outcome Measures will add value beyond Board of Trustees Mandate • Accreditation (e.g., MSA) • Curriculum development • Benchmarking and Quality Improvement • Assist in the development of a culture of evidence-based decision making through campus-based assessment • Respond proactively to public demand for increased accountability in Higher Education

41. Customized SUNY General Education Learning Outcomes Assessment • Reporting Timetable: 10-20 work days • Report Formats • Paper • Electronic • Data reported directly to campus • Individually reliable scores AND subscores • Normative data • SUNY Campuses (dependent upon volume) • National Comparisons in the future • Demographic breakdowns available (e.g., Major, Year in School)

42. SUNY General Education Learning Outcomes Assessment Modules • ETS can also provide assistance with • Sampling plans • Data interpretation • Assisting campuses in linking results with curriculum • Support for motivating students in learning outcomes assessments • Development of additional assessments to meet individual campus needs • Test Development Workshops • Development of Standards Based Assessments

43. We welcome feedback! • Questions/Comments from audience OR • E-mail David Payne at: Dpayne@ets.org • E-mail Trish Connor at: Pconnor@ets.org