1 / 85

Effective? use of VITAL (in Engineering)

Effective? use of VITAL (in Engineering). Tim Bullough, Engineering with thanks to: John Marsland, Electrical Engineering and Electronics Mark Johnson and Richard Dodds, Engineering. University of Liverpool L&T Conference June 06. QUESTION What Faculty are you from? A. Arts B. SES

estelle-genero
Download Presentation

Effective? use of VITAL (in Engineering)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Effective? use of VITAL (in Engineering) Tim Bullough, Engineering with thanks to: John Marsland, Electrical Engineering and Electronics Mark Johnson and Richard Dodds, Engineering University of Liverpool L&T Conference June 06

  2. QUESTION What Faculty are you from? A. Arts B. SES C. Science D. Medicine/Vets E. Engineering F. Teaching Support (CLL…) G. Other Press your response, then press SEND 1/1

  3. Blackboard (VITAL) is a VLE (virtual learning environment) • Module management • student registration • student tracking • resource control • group-work • Learning resources • weblinks • interactive self-study • lecture handouts/notes Module information • Communications • discussion forums • file exchange • online chat • Online assessment • tests and surveys • formative and summative • feedback to students • automated marking and analysis • student evaluation

  4. In theory a VLE should: • allow individual and self-paced learning • direct students to lots of data/information • encourage student-student interaction (peer learning) • allow testing of student understanding • provide guidance when errors or misconceptions are found • be cheaper and less time-consuming than conventional teaching Still not clear to e-learning designers or teaching staff: what students expect from e-learning how to motivate students to learn using the technologies how to bring in human interaction. how to blend with conventional teaching

  5. QUESTION Do you use VITAL for teaching? A. Yes (a lot) B. Yes (moderate) C. Yes (very occasionally) D. Not at all E. don’t teach Press your response, then press SEND 1/4

  6. QUESTION What does VITAL stand for? A. Very Irritating Test for fAiling Lecturers B. Virtual Interactive Teaching at Liverpool C. Version One To Aid Learners D. Vital Initiation Tool for Archiving Lectures E. Don’t Know Press your response, then press SEND 2/4

  7. QUESTION What software does VITAL use? A. WebCT B. Whiteboard C. Blackboard D. LUSID E. Access Press your response, then press SEND 3/4

  8. QUESTION What topic are you most interested in? A. Formative assessment B. PBL C. Group/teamwork D. Student feedback/evaluation E. All of the above Press your response, then press SEND 4/4

  9. (Very) short history of VITAL/Blackboard at Liverpool • WebCT trail in 02/03 • Launched VITAL(Blackboard) in May 03 • Started properly in 03/04 session Peak monthly usage: SessionActive users (unique) Modules 03/04 9572 (Mar 04) 3512 04/05 12453 (Nov 05) 4181 05/06 14329 (Jan 06) 5006 (24.5million hits) Staff using VITAL: BioSci Chem Engg Geog Health Law Sociol Mangt Vet Mar 04: 25 20 23 23 - 25 22 41 21 Mar 05: 48 28 32 21 47 20 24 44 27 Mar 06: 63 45 54 32 60 26 26 58 48

  10. VLEs: A National Picture UCISA/JISC national survey 2005 into VLEs in HE: • VLEs in ~95% of institutions • Blackboard (43%) and WebCT (37%) dominate • Subject take-up (pre-92 HE): Business/Law (85%) English/History/Psychol/Sociol (72%) Languages (70%)…….. • Medicine/Engineering/Phys.Sci (52%) • 46% of respondents do more than just post lecture notes and weblinks

  11. VLEs: A National Picture UCISA/JISC national survey 2005 into VLEs in HE: Barriers to development of e-learning: 1. lack of time 2. lack of money 3. lack of support staff 4. lack of academic staff development 5. lack of incentives/career development Very little concern about technical issues, student demand, institutional strategy etc.

  12. A little bit about Engineering……. Engineering in HE: Applied Maths/Physics topics Engineering Design Industry skills/issues….& Management Teamwork & Problem solving + • Engineering students…. in Yr 1: • Don’t know how to self-learn • Don’t know how to access/evaluate information • Not comfortable mixing maths and physics • Have no idea at all about how well they are learning at University - but they really want to know! • Engineering programmes: • High direct contact hours (20 lectures/week + labs) • Lecture-based modules compete with 100%-coursework modules for the attention of students

  13. Effective? use of VITAL (in Engineering): • Formative assessment • Developing a Student Case-Study: Sustainable Development • PBL and group-work • Collecting student Feedback & Evaluation • Other things: recorded lectures, e-lectures,summative assessment

  14. QUESTION Which quotation do you think best describes “Formative assessment” …? A. A range of …. assessment procedures ….. to modify and enhance learning and understanding. B. A process to recognise and respond to student learning in order to enhance that learning, during the learning. C. The provision of information as feedback to modify the teaching and learning activities. D. Providing the capacity for students to monitor the quality of their own work during its production, so they can improve it. E. Any assessment designed to promote further improvement of student attainment Press one response, then press SEND 1/2

  15. QUESTION Have you used the VITAL assessment tool in your teaching? A. Yes (auto marked, formative) B. Yes (auto marked, summative) C. Yes (manually marked, text responses), D. No (but have been considering it), E. No (can’t see it is relevant to my teaching) F. Don’t teach Press response, then press SEND 2/2

  16. Formative assessment (me and John Marsland)

  17. What is formative assessment? Feedback-assisted learning and self-diagnostic: • scores do not contribute (much) to grades • smaller, more frequent assessments • detailed (immediate) feedback to every student • tutor and students can monitor progress • students can take the tests as often as they wish to assist learning and revision

  18. Computer-aided assessment • Benefits (in theory) • anywhere anytime • students work at their own pace • enables frequent small tests • provides rapid feedback • can include multimedia etc • analysis of results automated • eliminates marking burden/errors • scales well for large groups • Limitations • investment of time & effort • cannot test “higher order” knowledge • requires access to networked PC • staff need to learn new skills • for summative assessments: reliability, security and validation

  19. Assessment in Blackboard (VITAL) Types of questions: True/False(easy, but of limited use) Multiple Choice (good for simple recall of facts, and calculations) Multiple Answer (tests recall and relationships) Ordering (tests recall of sequences, but difficult to think of questions) Matching(same as mcq. Can be used for feedback) Fill in the Blank (facts and spellings, but need to be careful to include all variants) Calculated formula (question includes an equation with different random variables for each student) Hot Spot (student has to point to specific area on image) Essay/Short Answer(text box - needs manual marking) • Include figures, attached documents, web-links, videos, etc….. • Randomise questions (from pool?)?

  20. Formative assessment using VITAL: what’s been tried in Engineering? (Me and John Marsland) • Both taught traditional Yr 1 engineering lecture-modules, with one “class-test” or paper-based “home-work” problem sheet around wk8 • Major coursework marking burden for 100-300 students in Yr1 classes • More importantly, wanted to improve student engagement with taught material during the semester, not just in the week before the exam. • + Give new Yr 1 students a better idea of their understanding? BUT How much time & effort to set-up, mark and maintain? How do you get students to do the assessments? Do you provide academic support to any students who have problems?

  21. Year 1 Structural Materials (265 students) (me) • Pre-2003/4: • No VITAL. Single mid-semester paper class-test, and mcq exam. • 2003/4: • ~100 mcq’s put into VITAL as “Practice Questions” But only the best students used practice questions! • Single mid-semester class-test, and mcq exam  no better than before • 2004/5 onwards: • Replaced paper class-test with fourVITAL assessments (5% each). • Tests 1 and 2: knowledge/understanding of technical information. • Tests 3 & 4 include some real engineering problems. • Retained Practice Questions, but linked explicitly to the formative tests

  22. Create content area for the assessments

  23. Give clear instructions • Attempt ALL questions (negative marking?)? • Scaling? ~2 weeks to complete test encourage peer discussion Ask for feedback Delay score-return by 1 week Save and resume?

  24. Types of questions: factual knowledge….. • Effective questions? • clear, succinct questions and responses • don’t give away the answer (to other questions) • avoid ‘negative’ questions? • are the wrong responses plausible (esp for calculations)? avoid negative questions?! decide a point value for each question

  25. Types of questions: simple numerical questions…. Effective questions? Are the wrong responses plausible

  26. Include images and links to websites….

  27. Language subtleties and over-complicated questions can be impossible for non-native English speakers:

  28. Practice Questions…. to replace direct tutor support • Optional and instantly marked • ~50% of practice questions are "directed related" to test and/or exam questions • >120 questions • don’t count towards module mark • marked instantly

  29. Practice Questions (cont.) Attempt rate for Practice Questions >60-70%

  30. tutorial support (cont.) Detailed worked solutions to numerical questions provided after test marked

  31. MARKING automatic and IMMEDIATE when test submitted: returns “score only”, “score and correct answers”, “score and feedback”, UNLESS…… Include a (0-point) short-answer text feedback question: delays disclosure of score and provides feedback Release marks manually later on…

  32. Gradebook: Alphabetic “Grade List” shows all students’ scores for all tests: Summary of all responses to each question in Assessment Attempt Details ….. Missing marks are often students who have left the Univ, NOT non-submissions individual student's responses Submission rate for formative VITAL tests 90-95%

  33. Assessment Attempt Details: provides feedback about level of whole class understanding:

  34. Did it work? • Good test submission (95% in 05/06) and Practice Question usage (~70%) • Most usage just before test deadlines: Test 4 Test 3 Test 1&2 Exam Start of sem • Regular feedback comments VERY informative: discussed in class • Response data indicates topics they are having problems with

  35. Did it work? 03/04: 1 class-test + optional VITAL practice questions Exam 41.5% 04/05: 4 VITAL tests + linked Practice Questions Exam 49.5% Exactlythe same exam! …..Probably added about 5% to average exam performance EFFORT:2-3 weeks to set up questions, and ~2 days each year to maintain • Problems: • Need to be very careful with language, esp for Yr1 sem1 OS students • “work-around” to temporarily withhold marks on test submission • (VITAL Equation Editor not very good for Engg)

  36. Year 1 Digital Electronics 250 students Year 1 Audio Electronics 50 students Year 1 Electronic Circuits and Devices 80 students (John Marsland) • 2003/4: • One in-term paper-based homework assignment - lots of marking. • Relatively poor exam performance: little student engagement with lecture material. • Now: • 8 x weekly VITAL mcq-type 10-question assessments • Each test worth 2-3% • Best to allow each test to be taken at any time over 3 week period • Students told to work alone, but working together ok • Return “score only” instantly on submission of test. Correct responses available 3 weeks after deadline. No other feedback/help given/needed.

  37. Year 1 Digital Electronics 250 students Year 1 Audio Electronics 50 students Year 1 Electronic Circuits and Devices 80 students (John Marsland) • Did it Work? • Excellent completion rate: ~90% submission • VERY positive student feedback • Interaction with students probably greater than if no VITAL for large classes: students view it as active participation • Excellent for tracking student retention/drop-out in Yr1 sem1 EFFORT:about 1 week/module to set up questions, ~1 day/session to maintain • Problems: • Late/incorrect registrations in year 1 sem 1. • (Some Equation Editor problems).

  38. …. a few final comments on using Blackboard (VITAL) for (formative) assessment: • are there educational benefits? [yes] • are there any other advantages (financial, time)? [yes] • are there problems? [yes] • does it integrate with the rest of the module? [yes] • how much additional effort is required... • to create the online assessments and surveys? [quite a lot] • to manage their use? [a little] “Engineering students are fundamentally lazy”. Do the formative assessments really improve student performance, or is it the threat of failure in any assessment that motivates and rewards them so they attend more lectures and labs, and do a bit more private study?

  39. 2. Developing a Student Case-Study: Sustainable Development (Richard Dodds and me)

  40. QUESTION In what year did the first definition of sustainable development appear? “to ensure that it meets the needs of the present without compromising the ability of future generations to meet their own needs.” (The Brundtland Report) A. 1887 B. 1922 C. 1964 D. 1976 E. 1987 Press response, then press SEND 1/3

  41. QUESTION Which one of the following is not one of the "Five Principles" of sustainable development (UK Govt 2005 Strategy, Securing the future) A. Living within environmental limits B. Achieving a sustainable economy C. Promoting good governance D. Using sound science responsibly E. Ensuring a strong, healthy and just society F. Minimising pollution and waste Press response, then press SEND 2/3

  42. QUESTION Have you ever used the VITAL Assignment Tooland/or Word templates? A. Only the Assignment Tool B. Only Word templates C. Both Assignment Tools and Word templates D. Neither (but I know what they are) E. Neither (don’t know what they are) Press response, then press SEND 3/3

  43. Teaching of Sustainable Development in Design to all Year 1 Engineers (250 students) Pre-2005/6 • Individual paper-based “structured essay”, based on four lectures. • Only about 75% submission (typically 4-10 pages). • VERY disappointing report (both technical content and style). • VERY heavy marking load (6 academic staff). The Challenge for 2005/6 (I took over as coordinator) • Retain individual assessment • Wanted on-going individual formative feedback to improve quality of final document • Marking load limit set at ~10 mins/student, with just one pg student and 1 academic staff.

  44. Teaching of Sustainable Development in Design to all Year 1 Engineers (250 students) Approach taken • VITAL Assignment ToolandWORD Template (drop-down menus and limited text areas  standard format) • Word template issued via VITAL in wk1 and resubmitted part-filled using Assignment Tool in 3 weekly stages, before FINAL version submitted. • Nominal marks (5%, 3% or 0) at each stage • Optional feedback comments - provided for all at penultimate version • Electronic + hard-copy retained for marking FINAL version

  45. word template downloaded by student Submit part-completed case-study each week……..

  46. part-completed word template doc uploaded as attached file

  47. Assignment grading tool opened from gradebook….. part-completed word doc for marking mark feedback

  48. final doc submitted after 4 weeks

  49. Advantages of process • 80-90% submission at each stage: easy and quick for pg to mark • Feedback comments possible (but takes time!). Used for penultimate version or when student wildly off-track. • Much higher quality FINAL case-study (technically and structurally). • Download all submitted work as a record (and for safekeeping). • Weekly mark in VITAL (+comments) useful feedback to students: but also discuss general issues in subsequent lecture. Suggested Improvements/problems • Not all students use feedback comments before doing current version? • Easy to accidentally delete all submitted documents! • Chinese character filenames

  50. 3. PBL and group-work in VITAL

More Related