Online and Computer-based Assessment Tools

1. Online and Computer-based Assessment Tools Presented by Jennifer Fager For University of Wisconsin-Superior Enhancement Day 1/19/2011

2. Advice for Assessing Student Learning in an Online Environment • Multiple Choice and Short Answer Questions • Not always the best technique in either online or ground formats • Lend themselves to cheating more so in online • Questions, regardless of ground or online, must be written well • The “test” has been overused and relied upon unless the instructor is an expert in writing good test items

3. Advice continued • Use multiple forms of assessment • Has corroborating evidence of student achievement • More personalized assessments reduce the likelihood of plagiarism • “Term paper” vs. application paper using personal examples • General research paper vs. select topics focus • Use examples more “authentic” in format • Consider problems of the field • Engage in “conversation’s” online

4. Advice continued • Security options • Lockdown browser • Student can’t go to another website during the test • NOTE: Most students now have multiple devices so they can go to their IPhones, blackberries, laptops, IPads, etc. to research the “answers”

5. Advice Continued • Set test time limits • Even if the student gets help, time restricts the amount • Can punish the slower readers/processors • What do you want in terms of meaningful, thoughtful responses

6. Advice continued • Assume any online test is, by definition, open book • And perhaps open notes, and open neighbors • Don’t try to make it otherwise

7. Advice continued • Quiz/test software products are available • Can generate multiple versions of tests • Each student can receive a “different” test • http://www.fau.edu/irm/instructional/respondus.php • Have students explain their answers • 25% have to explain ‘A’ answers, 25% ‘B’ answers, etc.

8. Advise continued • Other security ideas • “thumb” identification • Webcams • Face-to-face testing at a testing center

9. Online Assessment Resources • Commercial Assessment Data Management Tools – • E.g. TracDat, WeaveOnline, Chalk & Wire • Curriculum Design Tools • E.g. WIDS – link outcomes to Performance Assessment Tasks • Survey Instruments • Survey Monkey • Zoomerang • Portfolios • Course Management Systems Quiz and Analysis Functions • Blackboard and others

10. Resources Continued • Google Docs • Jing • Reflective Instruments • Leadership • Personality Style • Teamwork • Merlot (has content template builder) • Clickers • AdobeConnect with polling features • Lock-down Browsers for online testing

12. Fundamental Questions • What evidence do you have that students achieve your stated learning outcomes? • How have you articulated these outcomes to students and adjunct faculty? • In what ways do you analyze and use evidence of student learning?

13. Assessment Process • The same process is used for online programs and courses as for ground (traditional): • Needs Assessment • Needs of the profession • Needs of the learners • Articulate your expectations • Measure achievement of expectations • Collect and analyze data • Use evidence to improve learning • Assess effectiveness of improvement

14. Measure Online Learning • Direct Methods • Mostly the same as ground • Can use any student work products that can be saved in an electronic form • Some challenges exist with lab work, oral performances, etc. • Indirect Methods • Again, same as ground • Survey response rates are notoriously low

15. Advantages of Online Assessment • Artifact collection can be automated through the course management software • Can work for ground as well • Assignments, rubrics can be standardized and mapped to specific competencies/outcomes • Again, can work for ground as well

16. New Opportunities with Online • Assess quality of discussions, group work • Particularly effective with asynchronous • Assess individual learning in more depth • Student advising can be more robust • Use of standardized rubrics can result in greater consistency in grading • Particularly effective when combined with periodic instructor norming and use of anchor samples

17. Discussion & Group Work • Faculty can isolate and organize contributions to a threaded discussion and give feedback on how to improve • Can archive individual contributions and group discussions • Formative and summative • Course, program, and institutional levels • Create a record of instructor effectiveness • Useful for annual review, tenure

18. Individual Student Learning • Individual learning/achievements can be tracked across the educational experience • E-format allows for wide dissemination of collected data • Current software packages provide various levels of data collection & dissemination

19. Rubrics • Clear articulation of expectations that are linked to specific course & program outcomes • A means through which correspondence of online & ground rigor can be ensured (alignment) • Increases consistency in grading across sections, courses, programs, even colleges • Holistic vs. Analytic

20. Example of a Successful Online Assessment Strategy • Develop course/program maps which articulate learning outcomes and link them to specific courses and course assignments • Collect and archive artifacts through the course management software • Post rubrics through course management software for student and faculty use

21. Ensure Validity, Reliability • Tracking inter-rater reliability helps achieve consistency in grading • Post rubrics, anchor samples for 24/7 faculty use • Full-time/core faculty should formulate competencies, achievement standards & interpret results

22. Sample Assessments • Review the Video Gaming Assignment included at the end of this packet. • Determine how this assignment could be adapted for online learning

23. General Education Outcomes Example • The ability and inclination to think and make connections across academic disciplines • How would you assess this outcome in ground or online courses?

24. Websites and Resources • http://fod.msu.edu/oir/TeachWithTech/online-courses.asp • http://www.uwstout.edu/soe/profdev/assess.cfm • http://www.fgcu.edu/onlinedesign/index.html • http://www.pvc.maricopa.edu/~lsche/resources/onlineteaching.htm • http://sloanconsortium.org/

25. Questions and Concerns • What issues have you encountered when assessing online learning? • What are your concerns regarding the use of online assessment techniques? • What policies and/or procedures need to be in place to reduce or alleviate these concerns?

26. Questions Still Burning? • What else do you need to know?

27. The Bottom Line? • Assessment in online learning can be the same as assessment on ground—artifacts and tools can be the same even if the medium is different!