“Strategies for On-Line Assessment” Presented by: JulieAnne Adamich Ph. D., CPA

1. Florida Association of Accounting Educators Fall 2004 “Strategies for On-Line Assessment” Presented by: JulieAnne Adamich Ph. D., CPA St. Petersburg College E-mail address: adamichj@spcollege.edu

2. Define the purpose of the Assessment Process • It is a Formative evaluation process • based on theContinuous collection and analysis of data • with the primary purpose of emphasizing Improvement of: - student learning - student development - the program within the institution

3. “Improvements in learning will depend on how well assessment, curriculum, and instruction are aligned and reinforce a common set of learning goals” Pellegrino, et al (2001,p. 25) It also is a way of ensuring that students have been exposedto and have “acquired” the major learning outcomes …. essential for successful job placement. Assessment accomplishes two important goals

4. Assessment is an integral part of the Planning and EvaluationProcess Formative Goals: Continuous Improvement Both Formative and Summative Goals: Focus: Allocating Resources Summative Goals: Accountability, Performance Evaluation

5. Another reason-“Accreditation”Requirements for SACS Reaffirmation • “The institution engages in ongoing, integrated, and institution-wide research-based planning and evaluation processes that incorporate a systematic review of programs and services that (a) results in continuing improvement and (b) demonstrates that the institution is effectively accomplishing its mission.” (Core Requirement 5) • “The institution identifies expected outcomes for its educational programs and its administrative and educational support services; assesses whether it achieves these outcomes; and provides evidence of improvement based on analysis of those results.” (Comprehensive Standard 16)

6. Goalsfor Developing On-Line Assessments • Select questions that assess learning objectives taught in a course/program. - Textbook End of Chapter assignments and test banks are typically categorized by learning objective. • Select questions that match the taxonomy level of the student’s learning development. • Utilize technology incorporated into Learning Management Systems (such as WebCT) which include quizzing modules to minimize instructor intervention during the assessment process to permit time for instructor focus on the analysis of detailed reports available to the instructor and the provision of student feedback. • Incorporate quiz features/settings that discourage or prevent cheating.

7. On-LineAssessment Resources Quizzing Assessments using WebCT (which is similar to other Learning Management Systems such as Blackboard, eCollege, ANGEL etc.)

8. Pre-formatted matching and multiple-choice quizzesare provided to assess each chapter inThomson Learning’s Course E-Packs.

9. Quizzes can also be created directly in WebCT (or imported from a Test Bank, Respondus etc.)

10. Students know the status of each question at all times. For matching questions, select the correct answer by clicking on the drop down box. Answers are saved by clicking on “Save Answer”

11. For multiple choice questions, students click on the button to the left of the correct answer.

12. On-Line Quizzing Security Features • Develop question sets and randomize answers to generate a “unique” quiz for each student. • Deliver questions one at a time to discourage students from printing the entire quiz. • Limit the quiz duration to discourage students from using/sharing resources. • Passwords, quiz availability, selective release settings limit the availability of the quiz.

13. There are a variety of question delivery options to promote a secured environment.

14. Question sets can also be developed in WebCT. For example, the system randomly selects either MC#1 OR MC#2. Each exam is therefore unique to discourage cheating.

15. There are also settings for each question, where the answer order can be randomized.

16. … and upon completion of the quiz click “Finish” Students verify that all questions have been answered …

17. Webct provides the opportunity for students to receive immediate feedback.

18. There are a variety of options for student feedback.

19. Scores can be released at various stages Question feedback (such as the correct answer, the total score, and grader’s comments) are available options.

20. Instructors can also view statistical information regarding the performance of the entire class on each quiz.

21. Test Item AnalysisReporting Features available in WebCT * Item Difficulty Index * Item Discrimination Index • Group Performance Indices: - Standard Deviation - Mean

22. Item Difficulty Index Upper 25%: the percentage of students in the upper 25% of quiz scorers who answered this question correctly. Whole Group: the percentage of students who answered the question correctly. Lower 25%: the percentage of students in the lower 25% of quiz scores who answered this question correctly.

23. Item Discrimination Index Discrimination: an indication as to how well the question relates between high versus low scorers. Higher values indicate better discrimination. A negative value indicates that the lower-scoring portion of the class performed better on this question than the high-scorers. Good Discrimination LowDiscrimination

24. SD:the standard deviation of the results from the average score. Group Performance N:the total number of all student responses for the question. Mean:the average score of all students for the question.

25. Analyze assessment results and prepare an assessment report Analyze each major learning outcome, based on the statistical results. Include the following: • Criteria for Success, i.e., expected statistical results • Summary of Assessment Findings, i.e., actual statistical results • Use of Results, i.e., analyze results to determine if planned program improvements are deemed necessary • Action Plan and Timetable for Implementation • Budgetary and Planning Implications

26. AICPA Educational Competency Assessment (ECA) Web Site The Educational Competency Assessment (ECA) Web site is an online resource for educators. The software is designed to guide accounting educators through the process of assessing a program or course. The purpose of the site is to help educators develop or reform curricula and courses to support the developmentof a set of competencies, consistent with the findings of the CPA Vision. Register free at: http://eca.aicpaservices.org

27. Organizers(A series of templates to develop a database for your institution with the capability to sort by selected course and/or program) • Evaluate Competency Performance (ECC) Organizer: • Documents your comparison of the actual level to your desired level of coverage of competencies to determine the degree to which the AICPA Core Competencies are covered in your courses and programs. • Organized by Competency Categories, evaluate your perceived degree of: * Actual Competency Coverage * Desired Competency Coverage * Desired Learning Outcome * Overall Conclusion • Assess Student Performance (ASP) Organizer: • Documents the process of gathering, analyzing and using assessment information to determine how well students are developing the AICPA Core Competencies. • Areas of input include: * Describing the Assessment Activity * Documenting Findings * Drawing Conclusions * Recommending Improvements

28. Quizzing AssessmentsMajor Pros • Explicit demonstration of knowledge • Efficiently examines a wide range of content knowledge • Can use well-established practices to evaluate reliability and validity • Inexpensive to administer and evaluate for a large number of students • If embedded in courses, allow for more representative or complete student samples • If test is used multiple times, can obtain longitudinal/trend or pre/post data Source: AICPA Educational Competency Assessment (ECA) Web Site Register free at: http://eca.aicpaservices.org

29. Major Cons – Mitigated using On-line Assessments • Labor-intensive to develop valid and reliable questions – Test Bank questions can be imported into WebCT. • May be unable to generalize performance to other tasks/ Less “authentic” because they do not simulate conditions of work – Personal Trainer “simulates” detailed exercises and problems. • Student participation rates and motivation levels may be low for non-graded examinations. Limited feedback to students if they are not allowed access to results for individual questions– Quiz settings provide options of hints, solution, and IMMEDIATE feedback upon completion of the quiz. • Standardized tests may encourage instructors to ‘teach to the test’ and direct student focus away from competencies– Question sets may be developed to randomize questions based on criteria (such as learning outcome, competency level, etc.).

30. Resources for Faculty * BEEP – Best Educational E-Practices http://www.spcollege.edu/eagle/research/beep/beep38.htm Archive includes information associated with the following resource categories: • Assessment and Evaluation • Innovative Technologies • Instructional Resources • Merlot. Well-known and growing collection of peer-reviewed online learning materials and more. Officially titled Multimedia Educational Resource for Learning and Online Teaching. http://www.merlot.org