1 / 50

ChemEd 2003 Auburn University, July 27-31, 2003

ChemEd 2003 Auburn University, July 27-31, 2003. A Methodology for the Simultaneous Assessment of Cognitive and Content Domains. Ray A. Gross, Jr. Prince George’s Community College Largo, Maryland. My Final Objective for Today.

chase-carson
Download Presentation

ChemEd 2003 Auburn University, July 27-31, 2003

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ChemEd 2003Auburn University, July 27-31, 2003

  2. A Methodology for the Simultaneous Assessment of Cognitive and Content Domains Ray A. Gross, Jr. Prince George’s Community College Largo, Maryland

  3. My Final Objective for Today • To provide a methodology for identifying student weaknesses--both in content areas and cognitive skills • So that you may use the methodology to find weaknesses and then devise strategies to overcome the weaknesses

  4. My Intermediate Objective • To show how to develop a set of desired learning outcomes (DLOs) that reflect what is actually tested and can also be assessed

  5. Outline of Presentation 1. Define terms 2. Develop content categories 3. Marry content and cognitive categories in a matrix 4. Write DLOs from the matrix 5. Code exam questions for assessment 6. Prepare an assessment matrix

  6. What is a DLO? A statement written by an instructor to a student telling the student by an action verb (related to a cognitive skill) what the instructor expects the student to be able to demonstrate during a semester for a specific topic (related to a content area).

  7. What is a DLO? • A statement the instructor can use to code exam questions so that the questions can be assessed by content area and cognitive skill in a systematic and consistent fashion.

  8. Example of a DLO • Predictproducts forname reactionssuch as the Grignard and Wittig reactions. • A DLO completes the phrase—“Upon successful completion of this course, a student should (or will) be able to:”

  9. A Set of DLOs • The number of DLOs determined by the instructor to cover the content domain of the course.

  10. Two Elements in a DLO • Cognitive Skill—A mental process used by a person in problem solving (general– applies to all courses) • Content Area—A portion of a course defined by the instructor (specific – applies to a given course) • Time—A semester

  11. What are the Cognitive Skills? • Some person’s idea of the mental processes humans use to solve problems (i.e., learn). • Bloom’s Taxonomy: Six ways of thinking or six cognitive skills humans use to solve problems, rank ordered by complexity. These six skills span all disciplines and are adopted for use herein.

  12. Bloom’s Cognitive Domain A. Knowledge--rote B. Comprehension—grasp meaning C. Application—use material in new situations D. Analysis—break material into components E. Synthesis—put parts together F. Evaluation—use judgment

  13. Content Domain • A list of content areas that can be subdivided into course topics • The content domain is course specific and must be determined by the instructor.

  14. Content Domain • Course topics are derived from content areas in the same way action verbs are derived from cognitive areas. • An example of a content domain is shown on the next slide for organic chemistry.

  15. A Content Domain (Orgo) 1. Nomenclature 2. Equations 3. Mechanisms 4. Problems 5. Synthesis 6. Explanations 7. Correlations 8. Classifications 9. Definitions

  16. Your Content Domain • Determined initially by what you have tested in the past

  17. Limits of Assessment • The cognitive domain (six alphabetic items) and content domain (nine numeric items) define the level to which the course is subdivided for assessment purposes. • For the example, a 6 x 9 matrix defines the parameters for assessment and the corresponding DLOs.

  18. What is a Cognitive-Content Matrix? • A way of showing how the content and cognitive domains are interrelated • A way of showing how the two independent variables that make up a set of DLOs are related • A tool for qualitatively or quantitatively assessing an evaluation or combined evaluations

  19. Before Developing Content Domain--we will use Organic Nomenclature to: • Explore how a given content area may be assessed by all six cognitive skills • Learn how to dissect a question into its cognitive and content components • Learn how to code a question by its content and cognitive components

  20. Six Cognitive Codes for each Content Code • Content area 1 = Nomenclature • DLOs may be coded from 1-A to 1-F

  21. Knowledge—rote Q1. Write the name of CH4? Code = 1. Nomenclature A. Knowledge • The student is intimately familiar with the item.

  22. B. Comprehension—graspmeaning Q2. Give the IUPAC name of CH3CH(CH3)CH3. Code = 1. Nomenclature B. Comprehension • The formula is not new to the student; the instructor discussed it prior to the exam.

  23. C. Application—usematerial in new situations Q3. Give the IUPAC name for CH3C(CH3)2CH(CH3)CH3. Code = 1. Nomenclature C. Application • Student has never seen this structure before; nomenclature rules are applied.

  24. D. Analysis—breakmaterial into components Q4. What is the parent hydrocarbon of 3-isopropyl-4-methylheptane. Code = 1. Nomenclature D. Analysis • The name is broken into its component parts to find parent name.

  25. E. Synthesis—putparts together • Q5. What is the IUPAC name of the structure on your right? Code = 1. Nomenclature E. Synthesis • Component parts must be assembled into a name; requires analysis then synthesis.

  26. F. Evaluation—use judgment • A research group studied the mechanism for the dehydration and rearrangement of the aglucone 2,4-dihydroxy-7-methoxy-2H-1,4-benzoxazin-3(4H)-one (8) into 3-formyl-6-methoxybenzoxazolin-2(3H)-one (9) by the reagent N-ethoxycarbonyltrichloroacetaldimine.

  27. F. Evaluation—use judgment(continued) • Q6. The carbon atom in 8, indicated by the question mark in Figure 1, should be given what number by nomenclature rules? • Code = 1. Nomenclature F. Evaluation • Judgment is required in comparing two ring systems, which students haven’t seen before.

  28. Summary • All six cognitive skills (codes A-F) might be tested against nomenclature (code 1). • In theory, 54 (6 x 9) question types might be formulated for organic chemistry. • For any course in which Bloom’s taxonomy is adopted, a similar matrix is theoretically possible. • In practice, only a fraction of the possible codes is needed.

  29. How is the Content Domain Developed? • By analysis not synthesis • Analyze previous exams, categorize each Q by content • Consolidate the content in the questions into a content domain • Also, categorize each Q by cognitive category as you go (Bloom’s A-F)

  30. Content Development Process 1. Make a content domain of about 8 categories. 2. Code previous questions into one of the categories by number. 3. Add categories if necessary. 4. Consolidate categories if necessary.

  31. The Process is a Retro-Analysis of each Question 5. Concurrently, code each Q by letter—a Bloom cognitive skill (e.g., C). 6. Use the codes to make a qualitative content—cognitive matrix (see slide 18).

  32. Build a Qualitative Cognitive-Content Matrix • Lay out a content vs cognitive matrix by number and letter codes. • Each dual-coded Q (e.g., 7D, 4C, and 6F) equates to a box in the matrix. • Place an X in each box that is represented by a dual-coded Q. • The result is a qualitative matrix • It shows what you actually tested by content and cognitive categories.

  33. Qualitative Matrix • Shows which content and cognitive categories are to be assessed or were assessed

  34. Subordinate Cognitive Domain to Content Domain • In order to minimize the number of DLOs, the cognitive domain is subordinated to the content domain, so that only nine DLOs are needed for the 54 possibilities. • The cognitive skill for a specific question will be evident by the letter code you assign when the question is formulated.

  35. Use the Qualitative Matrix to Develop DLOs

  36. Consolidation of Cognitive Skills • One or more verbs that encompass all of the cognitive skills tested are married with the content category to arrive at a single DLO for nomenclature.

  37. Construct DLO • Name chemical species from structures and draw structures from names. • This is the DLO for nomenclature. • Develop DLOs for each content category in the same way.

  38. Cognitive-Content Linkage

  39. Process Marries Domains • For the next exam, dual code each Q. • Assign points for each Q and develop a quantitative matrix. • Determine max points for each box. • Calculate student percentages for each box. • Both content and cognitive skills are assessable.

  40. Dual Coding Process • Allows instructor to predetermine the mix of content and cognitive skills to be assessed for each evaluation

  41. Coding Example 1 • Which compound, ethanol, ethane or diethyl ether has the highest boiling point? • Content = 7. Correlation (property—structure) • Cognitive = D. Analysis or B. Comprehension

  42. Note • In the previous example, an instructor who covered the specific question in class would code the question as a comprehension, whereas an instructor who covered the principles involved but not the specific question would code the question as an application. • (Added per ChemEd 2003 participation!)

  43. Coding Example 2 • Calculate the degrees of unsaturation in C5H10O. • Content = 4. Problem • Cognitive = C. Application

  44. Coding Example 3 • Explain why the nitration of bromobenzene is slower than that of benzene yet produces an ortho,para product. • Cognitive = 6. Evaluation • Content = F. Explanation

  45. A Partial Quantitative Matrix

  46. Quantitative Matrix • See the Word document (in the folder with these PowerPoint slides) for an actual quantitative matrix. • The quantitative matrix identifies student strengths and weaknesses. • A strategy for improving weaknesses can be developed from the identified weaknesses.

  47. Summary • Content areas may be assessed against each applicable cognitive skill by dual coding test questions. • Content-cognitive weaknesses can be identified and actions formulated to improve student learning in those areas. • The skill levels (i.e., degree of higher-order thinking) for a given exam can be determined in advance by systematically coding exam questions as they are prepared.

  48. Comments • The DLOs provide a methodology for doing a macro-assessment that is useful for instructors. • For the DLOs to be meaningful for students, examples of coded test questions for each teaching block are probably necessary.

More Related