1 / 26

Pros and Cons of on-line testing using Moodle

TEL@York2006 April 25, 2006. Pros and Cons of on-line testing using Moodle. A case study of the elementary Japanese course. by Norio Ota, Noriko Yabuki-Soh, Gergana Ivanova (DLLL), Mike Street (IT Consultant, ATS). Preamble. TEL initiatives by the Japanese Section of DLLL

zora
Download Presentation

Pros and Cons of on-line testing using Moodle

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. TEL@York2006 April 25, 2006 Pros and Cons of on-line testing using Moodle A case study of the elementary Japanese course by Norio Ota, Noriko Yabuki-Soh, Gergana Ivanova (DLLL), Mike Street (IT Consultant, ATS)

  2. Preamble TEL initiatives by the Japanese Section of DLLL • Server-based web course development • Developing interactive instructional materials for self-study • Developed a distance education course for the elementary level Japanese language course (tested at Glendon for 4 yrs) using videoconferencing • Introducing Media Site Live for video-streamed lectures • Developing web-based on-line tests for the elementary Japanese course assisted by ATS (2005-06) • Developing a distance education course for the advanced Japanese course for St. Mary’s University in Halifax (to be implemented in September 2006)

  3. Web-based Testing (WBT) Computer-based Testing (CBT) vs. WBT On-line course delivery software products • WebCT • Hot Potatoes • Sakai • FLE3 (Future Learning Environment 3) • Moodle (Modular Object-Oriented Dynamic Learning Environment) Moodle • Free • Open source – customizable • Non-proprietary • User-friendly • Ease of installation • Flexible • Language support • Modular & Comprehensive Why on-line testing? • Reducing marking time • Developing distance education courses

  4. Challenges for on-line testing for languages Concerns by language teaching professionals • limited types of questions • lack of analytical tools (natural language parsing) • lack of qualitative evaluation • lack of evaluation for communicative competence • security issues

  5. Implementation and Objectives • On-line testing is NOT comprehensive • On-line testing is to access each learner’s knowledge and recognition of: • Vocabulary • Expressions • Conjugations • Sentence structures • Basic kana characters and basic sino-Japanese characters (kanji) • Simple context (communicative understanding) • Sociolinguistic and pragmatic aspects

  6. Developing tests • Transforming paper-based tests into web-based versions • Modifying types of questions • Developing new types of questions • Reviewing students’ answers and modifying possible answers • Readjusting answers to introduce partial marking system • Developing a questionnaire for students’ feedback

  7. Overview of Moodle • Course Management System (CMS aka LMS) • Similar to WebCT, or Blackboard • Our Moodle runs on Linux OS using PHP and a MySQL database backend. • Open-source, modular and flexible • Large, growing community • Community-driven support

  8. Overview of the Japanese Tests • Tests run in the Multimedia Language Lab (MLC) • 50 mins per test • 2 to 5 tests run a day, over 2 days, 5 mins between tests • Students self-register for accounts using Moodle • Ran an Orientation Session at beginning to create accounts and try out sample test • Tests were originally developed in MS Word. • Imported into Moodle via web interface

  9. Question Types • Aside from the normal types of questions, the tests also included: • Fill-in-the-blanks (multiple blanks within a paragraph, using Cloze or Embedded Answers type questions) • Wildcard characters in the answers (for partial marks) • Custom question type: • 50 Japanese characters in a large image (png), each given a unique number • Students answered the question by typing the number of the character into the box, in the correct order. (able to use wildcards for partial marks here too).

  10. Security Measures • IP-restricted (MLC only) • Each test has unique password • Time-based release of each test • During the allowed test time, students can review answers • Allowed a maximum of 2 attempts • Moodle ‘secure window’ test

  11. Technical Challenges • Accurate representation of Japanese characters • Clarity of images • Import not functioning in Moodle, so questions were cut/pasted • Time-consuming to enter questions initially • Cloze type question still sensitive to illegal characters • Various minor software glitches (eg. Half marks getting rounded down)

  12. For the Future • Moodle has proven to be a successful way of deploying language tests online • All of the major/minor software problems are going to be fixed over the summer 2006 • Eventually will be bringing Language Placement Tests on board (French and Italian)

  13. What is tested? Evaluation criteria for JP1000 • Attendance & Participation (Sept.-April): 10% • Oral Presentation (4 times a year): 25% • Quizzes (dictation, every week) & Homework (practice of Japanese characters): 15% • Tests (4 times a year, on-line testing): 50% -grammar & structure; vocabulary; idiomatic expressions -reading of Japanese scripts

  14. Types of questions Pen & Paper vs. On-line • Reading words • “What would you say when…?” • Translation (E→J) -Complete a dialogue -Short answers vs. jumbled words • Kanji(sino-Japanese characters) • Translation (J→E) -Short answers vs. multiple-choice

  15. New attempts • Short answers • Multiple-choice questions • Questions using a table • Questions using pictures • “Read a passage and answer questions”

  16. ‘Cons’ for Students • Less lenient marking compared to paper-based tests • Stressful for those uncomfortable with computers and bad eye-hand coordination; displaying time left in the corner • “Save without Submitting” function tedious and demanding • Difficult to return to an unanswered or problematic question, slow in viewing answers • Easier to cheat • Blurry screens

  17. ‘Pros’ for Students • Quiet atmosphere, easier to focus • Easier to answer questions (multiple-choice, kanji recognition, re-attempts) • Easier to correct answers • Immediate feedback • No concern about bad handwriting; hands do not get tired from writing; no need to use pencils • Less stress for those familiar with technology, more fun • Gain more knowledge about computer technology - important for academic and professional life

  18. On-line vs. PaperTest Average (%) Test 1 Test 2 Test 3 Test4 Paper 78.44 55.16 64.90 66.98 (2004-05) On-line64.5759.0959.5262.50 (2005-06)

  19. Students’ Self-reflection (1) Q: How satisfied are you with the time you have allotted to prepare for the tests? A: dissatisfied 6.67% somewhat dissatisfied 46.67% satisfied34.67% very satisfied 12%

  20. Students’ Self-reflection (2) Q: How satisfied are you with the efforts you have allotted to prepare for the tests? A: dissatisfied 8.79% somewhat dissatisfied44.59% satisfied38.51% very satisfied 8.11%

  21. ‘Pros’ for Instructors (1) • Less time spent on marking, less tired, improved quality of teaching • A test can be automatically re-graded for the entire class if a mistake is discovered in a question or more possible answers are added to the system • Easier to deliver the correct answers to students • Typed text easier to read than unintelligible handwriting

  22. ‘Pros’ for Instructors (2) • Easier coordination between instructors regarding reviewing students’ answers, modifying possible answers, marking • Faster processing of the test results • Objective assessment(not a comprehensive approach) • Increased marking consistency

  23. ‘Cons’ for Instructors • Technical problems unsolvable by instructors • Too sensitive to spaces, hyphens, unpredictable bugs in the program • Location and scheduling restrictions • Limitations regarding the types of questions • Academic honesty issues

  24. Cheating…  Weaker classroom control • Assigned seats, gaps in the seating  Inability to cut off the Internet  ‘Monitored activities’  Multiple attempts, higher test scores  Limited attempts (two attempts only)  Easy to duplicate the test and export it to Word  Question pools, randomized questions, different order of answers; run Moodle from a CD

  25. Outcome • Learned more about limitations, bugs and positive features re Moodle • Why were test scores lower this year? • Academic honesty issues • Learning process for faculty • Faculty’s willingness to learn and no quick resistance to Moodle • Cooperation between IT consultant and faculty with good working relationship and initiatives • Tech problems are hard to deal with by faculty alone: requires a tech support person at the test site. • Limitations re the test site

  26. References • Abhijeet Chavan (2004) Open-Source Learning Management with Moodle http://www.linuxjournal.com/article/7478 • Moodle (2006) Moodle for Language Teaching http://moodle.org/course/view.php?id=31 • Aditya Nag (2005) Moodle: An open source learning management system http://business.newsforge.com/article.pl?sid=05/05/09/2117200 • Den Pain and Judy Le Heron (2003) WebCT and Online Assessment:  The best thing since SOAP? http://www.ifets.info/journals/6_2/7.html • Röver, C. (2000)Web-Based Language Testing: Opportunities and Challenges http://www2.hawaii.edu/~roever/wbt.htm • _________(2001) WEB-BASED LANGUAGE TESTING http://llt.msu.edu/vol5num2/roever/default.html • Sabine Siekmann (2006) CALICO Software Report Which Web Course Management System is Right for Me?A Comparison of WebCT 3.1 and Blackboard 5.0 http://calico.org/CALICO_Review/review/webct-bb00.htm • University of Ontario (2006) WebCT http://www.uoit.ca/EN/main/11258/12122/17767/learning_webct.html

More Related