Assessments Through The Learning Process

1. Assessments Through The Learning Process October 2013Eric Shepherd. Questionmark, CEOeric@questionmark.comwww.questionmark.com

2. Thank you

3. Using the cards C A D B • You have colored cards on your chairs/tables • Get one of each color, lettered A, B, C and D • Let’s practice holding up cards • During this presentation we’ll be using questions to show you the power of questions • Please answer quickly and don’t be embarrassed – we are here to learn!

4. Research Suggests… A B C D • Learners generally prefer response cards to simple hand-raising • Response Cards Shown to Improve • classroom engagement • amount they learn • amount they retain -- Dr. Will Thalheimer, citing multiple examples of research on the use of response cards

5. How much tea was dumped into Boston Harbor at the Boston Tea Party? A B C D 100 chests of tea 34 chests of tea 342 chests of tea 342 tea bags

6. Answer: C • 342 chests of tea • Over 46 tons of tea • Worth (at the time) over £10,000 • £678,827 using the retail price index • £866,694 using the GDP deflator • £7,422,564 using average earnings • £10,025,165 using per capita GDP • £25,350,172 using the GDP Citation: Lawrence H. Officer, "Five Ways to Compute the Relative Value of a UK Pound Amount, 1830 - 2005" MeasuringWorth.Com, 2006.

7. Learning Without Questions zzzzzzzzzz Content zzzzzzzzzz zzzzzzzzzz zzzzzzzzzz zzzzzzzzzz

8. Learning Benefits of Questions Content Questions

9. We are using answer cards to… A B C D …expose poor presentation skills …stimulate thinking and keep you awake …frustrate our color blind friends All of the above

10. Questionmark • Provide Assessment Management Systems • To create, deliver and report on assessments • For exams, tests, quizzes, and surveys • More than 1,500 customers have purchased Questionmark • Corporate, Government, and Education markets • More than 14,000 authoring systems installed • Last month users created 40,000+ questions in Questionmark • Offices in US, UK, Canada, Germany, Belgium • Available in numerous languages

11. Worldwide Practical Experience

12. Sample List of Canadian Customers • Loyalist College • Manitoba Hydro • Memorial University of Newfoundland • Moneris Solutions • National Association of Pharmacy Regulatory Authorities • Northern Ontario School of Medicine • Omnivex Corporation • Plains Midstream • Safety Codes Council • Scotiabank • Telus : Telus Supporting Units • University of Calgary : Faculty of Veterinary Medicine - University of Calgary • University of Manitoba - Faculty of Medicine • University of Ottawa • University of Ottawa - Faculty of Medicine • University of Saskatchewan • Alberta Health Services • Aluminerie Alouette Inc. • Apache Corporation • ATB Financial • British Columbia Hydro • Canada Border Services Agency - CBSA • CMLTO - College of Medical Laboratory Technologists of Ontario • Department of National Defence • Everest Colleges Canada Inc • ExxonMobil • Flakeboard Company Limited • Fortis BC Inc • Grant MacEwan University • International Competency Assessment Board • Invesco Canada Ltd. • Justice Institute of British Columbia

13. Financial Services Pharma / Health High Tech / Manufacturing Telcom Energy / Utilities Public Sector Education Transportation Prof. Services Retail Who Else Uses Questionmark?

14. What would you like from this session? A B C D …brain dump; tell me everything …I don’t know; say something interesting …hurry up; I am only here for the networking …why so many questions?

15. Increase Revenue = € £ $ ¥ Reduce Costs = € £ $ ¥ Reduce Time to Market Things that Drive What-To-Do

16. Health &Safety Money = € £ $ ¥ Laws People or Talent How Long Do We Have? Resources Practical Limitations for Planning

17. What are the business drivers that drive the use of learning and assessments? A B C D Increase revenue, time-to-market, laws Increase revenue, reduce costs, safety Increase revenue, reduce costs, time-to-market Money, time-to-market, laws and talent

18. What are the practical limitations of using assessments? A B C D Money, health and safety Money, laws, health and safety Money, laws, resources, and safety Money, laws, time-to-market, health and safety and, access to talent and resources

20. What To Do How To Do It Who Will Do It Knowledge, Skills and Abilities Required What Will They Do What Will They Use

21. Competencies What To Do How To Do It Who Will Do It Knowledge, Skills and Abilities Required What Will They Do What Will They Use

22. Mental Check to Confirm a Competency “Given What Do What”

23. Competency is the “Ability To Do”(or “Ability To Perform”) and requires…. Cognitive Skills

24. Example - Attitudes and Behaviors • Customer Results • Consistently meets Customer Needs. Provides solutions for external and internal customers that meet and/or exceeds customer expectations. Identifies opportunities that will benefit the customer and works in partnership with the customer to drive the business forward. • Results Oriented • Ability to meet and exceed goals, make difficult decisions, act on opportunities that create value for the company. • Demonstrate ability to exceed stated job responsibilities, take informed risks, make principled decisions, and faces high-stakes issues head on. • Create a work environment where others feel it is safe to take risks.

25. Competencies Required Knowledge, Skills and Abilities Required

26. Knowledge, Skills and Abilities Required Assessment • How do we know the Knowledge, Skills and Abilities required? • Job Task Analysis • Survey people who do the job • MoSCoW analysis • Knowledge, Skills and Abilities that we: • Must have • Should have • Could have • Would be nice to have

27. Knowledge, Skills and Abilities Required Skills Knowledge Knowledge Abilities Abilities Skills What We Need What We Have Skill Survey, Quiz, or Test, Assessment

28. What We Have Knowledge, Skills and Abilities Required Skills Knowledge Knowledge Abilities Abilities Skills What We Need Topics To Train Knowledge - = Skills Abilities

29. Which is the best answer to the question… Why are skill surveys useful? A B C D Helps us survey skills Helps us know what actions to take Know what knowledge we have Determine qualified candidates

30. Assessment Thru Learning Process Learning Work Place Mentoring Collaborative & E-learning Instructor In/As LearningAssessments Prepare For Learning Of-LearningAssessments Evidence Accumulated

31. Assessment Thru Learning Process Learning Work Place Mentoring Collaborative & E-learning Instructor Job Task Analysis Needs Analysis In/As LearningAssessments Prepare For Learning Slow Forgetting Curve Of LearningAssessments Course Evaluations Evidence Accumulated

32. Pre-Learning Assessments • Purpose: • Create intrigue • Route to course • Inform instructors • Establish benchmarks Learning Work Place Mentoring Collaborative & E-learning Instructor In/As LearningAssessments Prepare For Learning Of LearningAssessments Evidence Accumulated

33. In-Learning Assessments • Purpose: • Encourages focus on learning • Provide instant feedback to instructor • Stimulate and strengthen memory • Correct misconceptions Learning Work Place Mentoring Collaborative & E-learning Instructor In/As LearningAssessments Prepare For Learning Of LearningAssessments Evidence Accumulated

34. In a Perfect World

35. In the Actual WorldThey don’t get it All!

36. Post-Learning Assessments • Purpose: • Was learning environment okay? • Knowledge transfer occurred? • Reduce Forgetting Curve • Certifications Learning Work Place Mentoring Collaborative & E-learning Instructor In/As LearningAssessments Prepare For Learning Of LearningAssessments Evidence Accumulated

37. Learning Curve • Purpose: • Content repetition • Memory retrieval practice • Strengthens memory recall Ouch! The Forgetting Curve Questions Questions Knowledge/Skills Learning Experience Learning Experience Learning Experience Time

38. In the Actual WorldThey Forget It!

39. Slow the Forgetting Curve • Purpose: • Memory retrieval practice • Strengthens memory recall • Reduce Forgetting Curve Questions Questions Questions Questions Knowledge/Skills Learning Experience Learning Experience Learning Experience Time

40. Questions Thru the Learning Process Reduce Forgetting Create Intrigue Questions Questions Questions Questions Questions Strengthen Memory Knowledge/Skills Time

41. Which of the following combination of instructional activities will increase learning and reduce forgetting?Question borrowed from the work of Dr. Will Thalheimer A B C 30 minutes of instruction and 30 minutes of questioning 30 minutes of instruction and 30 minutes of review and study time Both will produce similar results

42. Why Don’t They Get It?

43. 0 Dr. Will Thalheimer research:

44. Which of the following combination of instructional activities will increase learning and reduce forgetting?Question borrowed from the work of Dr. Will Thalheimer A B C 30 minutes of instruction and 30 minutes of questioning 30 minutes of instruction and 30 minutes of review and study time Both will produce similar results

45. Tests and Exams That Measure

46. Tests and Exams That Measure • Regulatory • Governments • Health and Safety • Examples • Driving Tests • Pilots Exams • Doctor Certification • Non-Regulatory, certification • Vendor Specific • Microsoft • Oracle • SAP • Vendor Neutral • ECDL/ICDL • Regulatory Compliance • Demonstrating compliance to government • Health/Safety/Environmental • Financial Services

47. Setting a Pass/Fail score? Norm Referenced (pass/fail determined after test) Criterion Referenced (pass/fail score determined before test) 250 people 250 people 3rdgroup of People 75 pass 1stgroup of people 65 pass Only 50 people can pass 2ndgroup of people 45 pass

48. How each type of assessment measures the participant…

49. Are Regulatory Compliance tests Norm or Criterion referenced? A B C D Norm referenced Criterion referenced Neither of A nor B Both A and B

50. How Do I Know It’s a Good Test