Using a Diagnostic Test in the First Intermediate Accounting Course (Year 3): Outcomes Assessment in Comparison to

1. Using a Diagnostic Test in the First Intermediate Accounting Course (Year 3): Outcomes Assessment in Comparison to Bloom’s Taxonomy Michael J. Krause Le Moyne College, Syracuse NY INTRODUCTION On the Monday of Thanksgiving week, in the years 2004-2006, Intermediate I students took the same diagnostic test on the “accounting cycle”. Test used all 40 multiple choice questions found in Spiceland’s Intermediate Accounting (3rd edition, chapter 2) test bank. Students got advanced notice about test topic but told not to prepare for the exam. Upon completing test students responded to a survey designed to measure attitudes about the experience. Using a second separate answer sheet, students got immediate feedback on their performance. Selection of diagnostic test topic based upon this author’s belief that a sustained competency about the “accounting cycle” is a prerequisite for further study of advanced theories and skills related to financial accounting. In 2006 and 2005 nearly all students participated because five points automatically added to attendee’s first exam grade where topic initially tested six weeks earlier. Without such an incentive, 29 out of 43 students voluntarily attended in 2004. While author taught 65% of 2004 class as sophomores, all 2006 and 2005 students had other professors for introductory accounting courses. 2006 and 2005 students used the Norton Intermediate text. The 2004 class however, used Spiceland text, the source for test material. Entry into Intermediate II course restricted to students who earn a grade of “C” or better in Intermediate I. Over the three years observed, students who failed to qualify for second course proved to be: 2006 (17%); 2005 (26%); and 2004 (21%). Since 2004 this author categorized diagnostic test questions as: concepts; examples; and calculations. Upon participation in an August 2006 AAA panel discussion in Washington DC (on course-embedded outcomes assessment), this author first considered a connection of his question categorization to the first three fundamental levels of cognitive development found in Bloom’s Taxonomy of Educational Objectives (1946). LITERATURE REVIEW (Bloom’s Taxonomy Applied) Diverse previous studies used Bloom’s Taxonomy (1956) as framework to organize analysis and to further discussion. Bloom’s educational objectives listed in progressive cognitive order are: knowledge; comprehension; application; analysis; synthesis; and evaluation. Karns, Burton, and Martin (1983) analyzed six Principles of Economics texts. They evaluated how well exam questions coordinated with stated text learning objectives using Bloom’s Taxonomy to categorize text material. Hampton and Krentler (1993) used Bloom’s work to classify multiple choice questions found in four Introductory Management and four Introductory Marketing texts. Davidson and Baldwin (2005) looked at end of chapter materials in forty-one Intermediate Accounting texts from 1934 to 2004. They then related their analysis to Bloom’s six educational objectives to identify trends in learning goals over seventy years. Athanassiou, McNett & Harvey (2003) provide excellent background about Bloom’s theory as they shared their attempts to shift learning management to students as a way to promote critical thinking skills. Reeves (1990) also clearly puts Bloom’s work into context as he explains its relevance to teaching Business Ethics. Perhaps most creatively, Jui-Hung Ven and Chien-Pen Chuang (2005) developed a lexicon of action verbs related to Bloom’s six levels of cognitive behavior by analyzing job descriptions for information occupations in USA, Australia and Taiwan. The six studies mentioned generated some common findings. 1.) Text books fail to provide evaluation materials beyond basic cognitive levels (Karns et al, 1983), (Hampton and Krentler, 1993) and (Davidson and Baldwin, 2005). 2.) Best opportunities to evaluate at highest levels of cognitive development provided by creative personal pedagogies (Reeves, 1990) and (Athanassiou et al, 2003). CONCLUSIONS 1. Multiple choice questions from text book materials can be used to assess Bloom’s three basic levels of cognitive development – knowledge, comprehension and application. This author’s classification of multiple choice questions as concepts, examples and calculations reflects Bloom’s taxonomy. This study also affirms findings from previous research that text materials do an excellent job of assessing the most basic levels of cognitive development. 2. This study verifies that Bloom’s taxonomy is indeed a progressive listing. Over the three years observed, a comparison of a cognitive category’s error rate (based on all observed answers for the entire diagnostic test) versus the percentage of questions a category holds in total test population (the existence rate) showed the following trends: a.) The most basic cognitive level, knowledge, showed an error rate consistently below the existence rate. b.) Comprehension, the next higher cognitive level, demonstrated an error rate remarkably similar to the existence rate. c.) Application, the highest of the fundamental cognitive levels, displayed an error rate consistently above the existence rate. The calculation category (application) generated the most consistent standard deviation for observed incorrect answers. Despite the fact that the overall test performance for the three class years was clearly different, all three class years struggled uniformly with the exam’s highest level of cognitive development assessed. 3. Using the current diagnostic test as a prototype, a future exam construct could reduce total questions asked and thus become more proportionately reflective of the three cognitive categories assessed. Available research on “action verbs” can guide new questions creation that would minimize or eliminate dependence upon text book materials. Such an independent test instrument could facilitate research establishing that students have truly obtained a sustained competency over an assessed topic. 4. Further research needed to link multiple choice diagnostic testing results to other assessment vehicles designed to measure more advanced cognitive development. Such research could be designed to establish that a competency at lower levels of cognitive development is a prerequisite to obtaining skills reflective of the highest levels of cognitive development. 5. Survey results over the three years observed indicate that students had a positive reaction to the diagnostic test assessment experience. In particular, the timing of the experience was favorably rated. REFERENCES Athanassiou, N., J.M. McNett, and C. Harvey, 2003. Critical Thinking in the Management Classroom: Bloom’s Taxonomy as a Learning Tool. Journal of Management Education 27 (5): 533-555. Davidson, R.A., and B.A. Baldwin, 2005. Cognitive Skills Objectives in Intermediate Accounting Textbooks: Evidence from End-of-Chapter Material. Journal of Accounting Education 23: 79-95. Hampton, D.R., and K.A. Krentler, 1993. The Use of Management and Marketing Textbook Multiple Choice Questions: A Case Study. Journal of Education for Business 69 (1): 40-48. Karns, J.M.L, G.E. Burton, and G.D. Martin, 1983. Learning Objectives and Testing: An Analysis of Six Principles of Economics Textbooks, Using Bloom’s Taxonomy. The Journal of Economic Education 14 (3): 16-20. Reeves, M.F., 1990. An Application of Bloom’s Taxonomy to the Teaching of Business Ethics. Journal of Business Ethics 9 (7): 609-616. Ven, J-H., and C-P Chuang, 2005. The Comparative Study of Information Competencies-Using Bloom’s Taxonomy. The Journal of American Academy of Business, Cambridge 7 (1): 136-143. 3.) Bloom’s cognitive levels are progressive. Higher levels of cognitive development requires mastery of more basic levels (Reeves, 1990) and (Davidson & Baldwin, 2005). 4.) Action verbs can best describe (frame) assessment activities at the six levels of cognitive development and can guide new assessment development (Reeves, 1990), (Athanassiou et al, 2003) and definitively so (Jui-Hung Ven & Chien-pen Chuang, 2005). OBSERVATIONS Exhibit 1 - Test Results: Author’s % of Errors as a % of Total Test Bloom’s ObjectiveCategoryTotal Test200620052004 Knowledge (recall) Concept 27.5% 17.3% 20.0% 17.3% Comprehension (understanding) Example 42.5% 42.0% 43.2% 42.2% Application (use) Calculation 30.0%40.7%36.8%40.5% 100.0%100.0%100.0%100.0% Student Population………………………………… 52 44 29 Total Incorrect Answers………………………….. 715 732 348 Errors as a % of a Question Exhibit 2 – Mean Error Rate - Questions in a Category: 200620052004 Concept (n = 11).……..….. 21.7% 30.4% 18.8% Example (n = 17)……..….. 33.9% 42.3% 29.8% Calculation (n = 12)………. 46.6% 50.9% 40.5% - Standard Deviation - Question Error Rate: 2006 20052004 Concept (n = 11)……..….… 10.4% 13.5% 11.2% Example (n = 17)….………. 14.5% 18.8% 18.2% Calculation (n = 12)……….. 12.5% 12.6% 12.8% Exhibit 3 - Overall Comparative Performance2006 20052004 Median Correct Test Score………… 66% 60% 70% Exhibit 4 – Average Absolute Value of Difference in Performance (by categories tested) Test Category2006 v. 20052006 v. 2004 Concept (n = 11) 8.7% 5.2% Example (n = 17) 11.4% 8.4% Calculation (n = 12) 13.2% 13.4% Exhibit 5 – Comparative Survey Results (Strongly Agree assigned “1”; Strongly Disagree assigned “5”) 200620052004 Questions Asked: Mean / Std. Dev.Mean / Std. Dev.Mean / Std. Dev. 1. Test was a good experience………… 2.04 / .80 1.81 / .55 1.79 / .48 2. Test helped me understand Account- ing basics that I need to know………..... 1.78 / .70 1.86 / .56 1.59 / .49 3. Test helped me measure the progress that I made this semester in compre- hending Financial Accounting……….... 1.98 / .65 1.88 / .54 1.86 / .63 4. The Thanksgiving week class is a good time to take a diagnostic test….... 1.68 / .73 1.52 / .73 1.86 / .82