Issues in Assessment when Teaching Statistics

1. Issues in Assessment when Teaching Statistics Penny Bidgood, Kingston University, UK Assessment Matters – original assessment for original student work, HEA York 2011

2. Issues of assessment • Why assess? • Who is being assessed/ assessing? • What should be assessed? • Where is the assessment taking place? • How is the assessment done? HEA, York, 2011

3. Why assess? Code of practice for .. Assessment of students Assessment –processes that appraise knowledge, understanding, abilities or skills • Promoting learning by providing feedback • Evaluating knowledge, understanding, abilities, skills • Providing a grade • Enabling public and HE providers to know attainment level “Diversity of assessment practice between and within different subjects is to be expected and welcomed” (http://www.qaa.ac.uk/academicinfrastructure/codeOfPractice/) HEA, York, 2011

4. Why assess? Code of practice for .. Assessment of students • In implementing assessment policies consult subject bench marks and professionals • Types of assessment should be appropriate for subject, mode of learning and the student • Promote effective learning • Effective and appropriate measurement but avoid excessive burden for students • Provide appropriate and timely feedback HEA, York, 2011

5. Who do we Assess? Specialist statistics or Service modules Similarity: analyse data appropriately and report results effectively (exploratory data analysis / statistical modelling) Differences: class size depth of mathematics computer package used HEA, York, 2011

6. Who do we Assess? • Students have different strengths and approaches to learning and may perform differently in various types of assessment • Student Voice • A variety of assessment tools are required HEA, York, 2011

7. Who is assessing? Learning outcomes:- “Communicate technical ideas in writing” Lecturers:- • workloads • Research takes priority • Re-assessment issues Design different types of assessment HEA, York, 2011

8. Who? Variety in assessment Four Themes:- • Relating assessment to real world problems • Assessing statistical thinking • Individualised assessment methods • Assessing problem solving HEA, York, 2011

9. What do we Assess? Statistics as “mathematics” • Formal examinations and tests Statistics as an “applied subject” • Assignments and projects Statistics in a consultancy • Oral and written reports • portfolios HEA, York, 2011

10. What do we Assess? Understand • the purpose and logic of statistical investigations • the process of statistical investigations • mathematical relationships • probability and chance • Master procedural skills HEA, York, 2011

11. What do we Assess? Develop • interpretive skills and statistical literacy • ability to communicate statistically • useful statistical dispositions • “The Assessment Challenge in Statistics Education” (1997) Gal I and Garfield J B • “Assessment Methods in Statistical Education: An International Perspective” (2010) Bidgood P, Hunt N & Jolliffe F (eds) HEA, York, 2011

12. Guidelines for Assessment and Instruction in Statistics Education (GAISE) 2005 • Emphasise statistical literacy and develop statistical thinking; • Use real data; • Stress conceptual understanding rather than mere knowledge of procedures; • Foster active learning in the classroom; • Use technology for developing conceptual understanding and analyzing data; • Use assessments to improve and evaluate student learning. HEA, York, 2011

13. What do we Assess? • Garfield (1994, 1995) stressed the need for assessments that measure the understanding of a problem solving approach • The Mathematics, Statistics and OR Overview Report (2000) stated “Student engagement and performance has often been greatest when dealing with well-focused problems of a practical nature” HEA, York, 2011

14. Problem Solving approach • Using real, or at least realistic, datasets in an appropriate context of real problems. • Led by reforms in statistical education that emphasises statistical thinking, reasoning and conceptual understanding • Demands from employers – graduates with technical skills and the ability to communicate findings appropriately HEA, York, 2011

15. Data Handling Cycle English National Curriculum HEA, York, 2011

16. What do we Assess? HEA, York, 2011

17. Where? • Computer lab • On-line testing from a large databank • Using a computer package to analyse data • Practical assessment • Formal examination • In the lecture room • Often difficult to supervise adequately • In a seminar HEA, York, 2011

18. How ? (from Peter Holmes, 2004) • Examination • Quizzes • Multiple choice test (question bank) • “Take-home” assignments • Coursework – apply specific techniques to particular problem • Analyse computer output • Group work assessment • Design and carry out a statistical investigation • Prepare a report and present it • Oral presentations • Case Studies/Projects HEA, York, 2011

19. How? Seen examination • One week before students are given a numerical question with a small amount of data – just to illustrate the data and their layout. • In the examination proper students are given the full set of data and access to SPSS. • They are required to explain their choice of test, briefly describe the SPSS commands, report summary statistics and draw appropriate conclusions. C. Dracup, Northumbria (PiSA project) HEA, York, 2011

20. How? Multiple Choice Test Which of the following statements about the application of one-way ANOVA is false ? • The null hypothesis being tested is that all of the underlying means for all groups are identical • The F-statistic compares the variability between the sample means with the variability within samples • Large values of the F-statistic provide evidence of a difference between the underlying true means • If the P-value is large, this implies that all the means are the same • A small P-value suggests that the data provides evidence of differences in the underlying mean between some of the groups Assessment on a Budget Wild et al in Gal and Garfield HEA, York, 2011

21. How? Examination/Tests • Issue a published paper for students to review in advance and answer question(s) on it in a formal exam setting • Give students case studies throughout the year, which they are free to discuss with each other, but the assessment on the case studies is in the form of a supervised test HEA, York 2011

22. How?“Take-home” Assignments The standard statistics assignment • analyse of a set of data, using a suitable package, and submit a written report Plagiarism concerns can have a strong influence on assessment strategies Group or individual assessment? Move away from “take-home” assignments, particularly in large service modules. Plagiarism in Statistics Assessment (PiSA) http://www.jiscpas.ac.uk/documents/pisa.pdf HEA, York, 2011

23. How? Plagiarism in Statistics Assessment (PiSA) Project The RSSCSE and the MSOR Network jointly funded the PiSA project, which aimed to: • survey HE lecturers in Statistics to find out what methods of assessment and strategies to deter plagiarism are being employed currently; • to identifyand synthesise elements of good practice; • to disseminate findings widely. HEA, York, 2011

24. How? Plagiarism issues • Plagiarism is “To take and use as one’s own the thoughts, writings or inventions of another” (OED) • Collaboration is to work together for mutual benefit • Collusion is to work together for mutual benefit with the intention to deceive a third party HEA, York 2011

25. How? Plagiarism issues • Is coursework important in Statistics? • Are Statistics lecturers alert to plagiarism? • Is plagiarism causing a reduction in coursework? • How are Statistics lecturers tackling plagiarism? • What good practice can we share? • Not prevalence • Not case history • Deterrence is the key HEA, York, 2011

26. How?Deterrring Plagiarism • Institutional procedures • Organisational measures • Supervised assessments • Individualised assessments • Student-centred assessments • Electronic submission HEA, York 2011

27. How? Plagiarism Key Findings • Plagiarism, or more specifically collusion, is a significant problem within large Statistics service modules and all lecturers need to give serious attention to anti-plagiarism assessment strategies. • The majority of Statistics lecturers are well aware of plagiarism issues and are taking action, however small, to combat it. • It is quite common for Statistics lecturers to fail to apply institutional procedures in “minor” cases of plagiarism. In contrast, some lecturers make every effort to demonstrate how the regulations and penalties might apply to Statistics assessments, giving examples of cases detected in previous years. HEA, York, 2011

28. How? Plagiarism Key Findings • Plagiarism often goes undetected on large service modules due to a multiplicity of assessors. It is most likely to be detected when one person assesses all the students. • There is much innovative work taking place in the area of individualised assessment, but also some duplication of effort. • Assessments that require students to collect their own data, either individually or in small groups, are widely employed. • Many lecturers have moved away from take-home assignments to in-class supervised computer-based assessments. HEA, York, 2011

29. How? Plagiarism Key Findings • In-class tests can be exposed to a high risk of cheating by unsuitable accommodation, inadequate invigilation, failure to check student identities, and naïve organisation. • TURNITIN is being used increasingly. • In projects/case studies it is good practice to include an element that assesses the student’s working method and, ideally, an oral to check that it is genuinely the student’s own work. • Online cheating companies openly offer an easily accessible way for students to obtain professional individual help with Statistics assignments. HEA, York, 2011

30. How?Individualised assignments • Randomise elements or parameters • Allocate different subset of a large dataset e.g. ISCUS – Individualised Student Coursework Using Spreadsheets. This is based on Excel and allows you to use your own dataset. (Developed by Neville Hunt, Coventry University) • Students allocated subset of same source data based on ID number ABCDEFG • e.g. delete rows C, D and E • or calculate a 9G% confidence interval HEA, York, 2011

31. How? Statistical Investigation Students find own data, from journals, internet sources, about themselves • Allocate a particular periodical – data from any issue in current year • Medical statistics – find own example of a medical case study • Time series modules, find own data from “Economagic” • Sports science – collect data on fellow students e.g. heart rates HEA, York, 2011

32. How? Reporting conclusions • Report in form of poster, oral presentation, written as for a newspaper article etc • Vary format of the submission • Posters, typically produced by a group of students, although each should be able to “defend” the content • Written report, but vary the “client” – newspaper article, research paper, briefing document for local MP etc HEA,York, 2011

33. How? Group work assessment • Assignment of sub tasks within a group • Degree of input from each participant • Quality of final product from group • Individual learning which has taken place Peer Assessment • Some element of marks might be how they rate others’ contribution and their own HEA,York, 2011

34. How? Case Studies/Projects • Need to include an element that assesses the student’s working method • Ideally an oral exam or presentation • “4P model” • Project log • Project report • Practical development • Presentation (Sue Starkings, reported in Gal and Garfield, 1997) HEA, York, 2011

35. How? Creating Statistical Resources from Real Datasets (STARS) Aims • To make available real datasets and scenarios of relevance to Business, Health and Psychology • To develop web-accessible statistics worksheets using these datasets and various software (Excel, MINITAB, SPSS) • To develop resources for producing individualised datasets and assignments • Funded by the Higher Education Funding Council for England October 2002-January 2006 HEA, York, 2011

36. ISI, Durban, 2009

37. Final thoughts • Chatfield (2005) – difficult to teach using a problem-solving approach Problem Solving: A Statistician’s Guide • Rossman and Chance (2002) developed materials, motivated with real data and scenarios, using various problem-solving skills. ICOTS Proceedings • Jolliffe (2007) “Asking students to do real statistics on real data and to report on the results, is now feasible in a way that it was not in the past” (due to huge expansion in technology) http://www.stat.auckland.ac.nz/~iase • Marriott et al (2009) developed assessment regimes that correspond to the problem-solving approach in teaching and learning www.amstat.org/publications/jse/v9n3/ HEA, York 2011

38. References • http://www.qaa.ac.uk/academicinfrastructure/ • http://stars.ac.uk • http://app.gen.umn.edu/artist/ • http://www.jiscpas.ac.uk/documents/pisa.pdf • http://www.amstat.org/education/gaise/GAISEcollege.htm HEA, York 2011

39. References • Bidgood P, Hunt N & Jolliffe F (eds) (2010) “Assessment Methods in Statistical Education: An International Perspective” • Gal I and Garfield J B (1997) “The Assessment Challenge in Statistics Education” • Starkings, S. Assessing Student Projects • Wild et al Assessment on a Budget • Garfield J B (1994) “Beyond Testing and Grading: Using Assessment to improve Student Learning” Journal of Statistics Education • Garfield JB (1995) “How Students Learn Statistics” International Statistical Review 63 1 • Holmes P (2004) “Assessment in Statistics: a two-edged sword” in Assessment with a Purpose Conference • Hunt D N “Individualized Statistics Coursework Using Spreadsheets” Teaching Statistics 29 2 MSOR Overview Report (2000) Quality Assurance Agency for HE HEA, York, 2011

43. Assessment Tools Generators for individualised datasets, assignments and solutions • ISCUS – Individualised Student Coursework Using Spreadsheets. This is based on Excel and allows you to use your own dataset as well as those from STARS. (Developed by Neville Hunt, Coventry University) • DRUID – Dynamic Resources Using Interesting Data This is not tied to any statistics package but uses specific datasets. (Developed at the RSS Centre for Statistical Education, Plymouth University) HEA, York, 2011

44. Example from a Charts worksheet in Minitab using the Obesity dataset Q5. Why is the clustered column chart unsuitable for the age data? • We are now going to draw histograms of the ages for each of the treatment groups so that we can compare them • From the main menu select Graph > Histogram • Highlight Simple and click on OK • Enter Age in the Graph variables: box • Under Multiple Graphs > Multiple Variables choose In separate panels of the same graph and under Multiple Graphs > By Variables enter Treatment group in the By variables with groups in separate panels: box • Click on OK • Under Scale> Y-Scale Type choose Percent • Click on OK • Enter an appropriate title as before • Click on OK • Click on OK again to produce the chart below. There are so many different ages they become cluttered rather than clustered! The ages need to be grouped into intervals. HEA, York, 2011

45. Carlow, 2010

46. Q6. Is there an evident difference in the distribution of patients’ ages between the two groups? Height and baseline weight Try replacing C2 (Age) by C4 (Height) in the analysis above. Q7. Does the distribution of patients’ heights differ between groups? Repeat the analysis using the baseline weights in C9. Q8. Does the distribution of patients’ weights differ between groups? Both distributions appear quite similar although there is a wider spread of ages for the Placebo group. The most common age for the Placebo group is 40-44, whilst for the New drug group both 36 -40 and 52-56 have the same frequencies. Height has an almost uniform distribution in the Placebo group, but a more uneven distribution in the New drug group. Weight has a slightly negatively skewed distribution in the Placebo group, while the distribution in the New drug group is bi-modal. HEA, York, 2011