1 / 35

Assessing Confidence in the Chinese Learner

Assessing Confidence in the Chinese Learner. Stephen Bruce Napier University, Edinburgh. Assessing Confidence in the Chinese Learner. Chinese students at Napier University Rationale for confidence-based assessment Student responses to the test

aaron
Download Presentation

Assessing Confidence in the Chinese Learner

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Assessing Confidence in the Chinese Learner Stephen Bruce Napier University, Edinburgh

  2. Assessing Confidence in the Chinese Learner • Chinese students at Napier University • Rationale for confidence-based assessment • Student responses to the test • Academic self-concept, confidence and achievement research

  3. English Foundation Programme • Set up in 2001 to prepare Chinese students for entry to Napier University, both in terms of linguistic and study skills. • The needs of these students were complex, the “learning curve” for them was extremely steep, and that the demands they faced on a linguistically complex and culturally unfamiliar one-year Masters would be considerable, even given the requisite IELTS pass. • Proportion of Chinese students on Napier Business Masters programmes has increased in recent years, which has resulted in staff and the institution facing a very different cohort with different strengths, and also needs, to their predecessors. • Presents a major challenge to lecturers as they struggle to adapt their teaching approaches to suit both the increased number of overseas students and also the home students on their modules. Dr Lesley Gourlay, Napier University (2004). Crossing Boundaries: A Case Study Masters Level Chinese Students, LTSN (In Press).

  4. Linguistic and Cultural Issues Challenges for Students LANGUAGE • Extreme difficulties in understanding / taking notes in lectures • Difficulties with tutorial participation due to lack of confidence in English • Some difficulties with tutorial tasks due to slow reading speeds • Fear of failing exams due to difficulties in writing English in a time limit EDUCATIONAL CULTURE • Feeling “lost” in semester 1 • Some students unclear about expectations in UK-style coursework & reading • Experience extremely stressful for some Lesley Gourlay, Napier University (l.gourlay@napier.ac.uk)

  5. Linguistic and Cultural Issues Challenges for Staff LANGUAGE • Unsure if students are understanding lectures • Required to spend extra time explaining lecture content at the end • Some difficulties with tutorial participation and integration EDUCATIONAL CULTURE • Some staff faced with numerous examples of plagiarism in coursework • Some difficulties with tasks involving critical appraisal • Some students struggled with application of theory to practice Lesley Gourlay, Napier University (l.gourlay@napier.ac.uk)

  6. Potential responses by the university Lesley Gourlay, Napier University (l.gourlay@napier.ac.uk)

  7. Confidence and achievement • Chinese learners show a preference for rote-learning and memorisation. However their responses to learning strategy questionnaires indicate preference for deep learning. • Learner self-esteem has been found to be positively associated with academic achievement (Brookover et al., 1964; Prendergast & Binder, 1975; Song & Hattie, 1984) • Comparative studies support a view that the self-esteem of young Chinese students is lower than UK and American (Chan, 2000). • self-effacing and modest values in Chinese culture, strongly influenced by the Confucian tradition of a ‘humble’ character • traditional authoritarian style of education or the highly competitive pressures created by schools, families and society • gap in living standards in the UK and Chinese cultures • Recent advances in academic self-concept and achievement research

  8. Assessing Confidence in the Chinese Learner This short pilot study is interested in two questions: • Would the Chinese students responses to a confidence-based test differ from UK students ? • A good cultural test of this format • Would their use of confidence levels in the test relate to their academic self-concept, or attitudes to academic life ? • Improving academic self-concept is often posited as mediating other desirable attributes

  9. Rationale for confidence-based scoring • A student’s ability in answering may fall into a number of categories • I know it • I’m not quite sure, but I think I know it • Perhaps I can identify the answer by a deductive process on the distracters • If I guess I’ve a 25% chance of being correct (for 4 answer choices) • I really haven’t a clue and perhaps worst of all • I really know it …… what do you mean I’ve got it wrong !! Davies (2002) • The standard MCQ cannot distinguish between the above • possible to pass exams with knowledge only half learned or poorly understood • the student certain of their knowledge should be rewarded • the student should not be rewarded for guesswork • encourage students to reflect on the reliability of their answers • Would the student use this knowledge to make a decision or perform an action (usable knowledge) ? Hassmen and Hunt (1994)

  10. Confidence-based MCQ • Select an answer in the usual way for a multiple choice question • Indicate your confidence that you are correct. C=1 (low) C=2 (mid) C=3 (high) Gardner-Medwin and Gahan (2003)

  11. Scoring confidence-based MCQ What is my score if my answer is or ? C = 3 (High) C = 2 (Mid) C = 1 (Low) Confidence level 1 3 2 -4 0 -1 • Confidence level decision is governed by 2 judgements: • estimated probability that the chosen answer will be correct • the impact of the reward / benefit for a right / wrong answer • Students rarely discuss their decisions in terms of explicit probabilities Gardner-Medwin and Gahan (2003)

  12. Encouraging good confidence judgement Students are encouraged to reflect on the reliability of their answers. Good confidence judgement means more marks ! • If you are sure that your answer is correct …. • select C=2 (mid) or C=3 (high) confidence level • get the marks your confidence deserves! • If your are unsure (or guessing !) …. • select C=1 (low) confidence level • don’t lose marks through misplaced confidence!

  13. Confidence-based MCQ feedback A 100% confidence-based score is 40 answers at C=2 (mid) confidence …. so that 40 correct at C=3 would be 150% Gardner-Medwin and Gahan (2003) A summary of confidence levels….

  14. Confidence-based MCQ feedback Is the confidence-based score higher than the % correct score? No • Check the instances of INCORRECT answers at C=3 (high) and C=2 (mid). These can indicate areas of knowledge where you are misinformed. A large number of these perhaps indicates general overconfidence. • Check your number of CORRECT answers at C=1 (low). A large number indicates that you know more than you are willing to admit and are perhaps underconfident.

  15. Confidence-based MCQ feedback Is the confidence-based score higher than the % correct score? Yes You have shown good confidence judgement! You are willing to express an appropriate level of confidence when considering your knowledge. This is important when making decisions and performing actions based on that knowledge.

  16. The student group • 31 Chinese students (54% sample) • 18 male, 13 female • 24 postgraduate and 7 undergraduate • Ages: 18-25 (19), 26-32 (8), 33-40 (4) • Current on English as a Foreign Language (EFL) Foundation Programme • 35 weeks of 20 hours of EFL study • Gateway and Starter routes dependant upon English proficiency • Students recently completed their IELTS exam (International English Language Testing System). • Entry on to a Napier Programme conditional on a student’s IELTS score • At the time the students did not know their results • Students complete a confidence-based academic vocabulary test (40 Qs)

  17. Chinese student responses to confidence-based MCQ test • No indications that Chinese students (red) behave differently from home based students (blue) • Questions appear quite difficult, since low marks are more represented. • 930 ‘Blue’ data from UCL including 40 first time responses.

  18. Chinese student responses to confidence-based MCQ test • No indications that Chinese students are reluctant to select high confidence levels (C=3) • on average, high confidence was selected most often • Higher test scorers showed better confidence judgement • Lowest test-scorers tended to be over-confident • No significant difference between genders observed.

  19. Chinese student responses to confidence-based MCQ test • Instances of C=3 (high) confidence increased as the question difficulty decreased. • ‘Difficult’ language comprehension test as some small grammatical differences separated answers and distracters. Harder Easier

  20. Correlating confidence-based score and final exam (IELTS) • only 7 students (of 31) improved on their % correct score with good confidence judgement. • 6 of these students were the highest test scorers in the class. • no apparent correlation between IELTS exam and confidence-based test • IELTS exam measuring larger scope of competencies

  21. Student opinions of the test • Strongly agree • Agree • Neutral • Disagree • Strongly disagree

  22. Student comments “It is a real good interesting test and I can receive different aspects of my knowledge. However, it will cost lots of time to finish this exam. As a result, I suggest we can do the exam at a regular time such an once a week.” “It is an interesting test and I would like to do it at my university. I think that is obviously fair for students.” “It is interesting and a little difficult to understand the result. I think I should get a higher confidence-based score because I chose (c3) nine times, and the number answered correct is 7. Maybe I have not understood this well.” “I think it is quite interesting and helpful.it is also a good way to show me the link between confidence and academic study.”

  23. Academic self-concept and confidence judgement • Improving academic self-concept is often posited as mediating other desirable attributes such as persistence on academic tasks, motivation, and self-efficacy. • Is good confidence judgement a measure and a positive mediator • an improved measure than the standard % correct score • informed interventions for enhancing academic self-concept can make use of recent advances in theory Craven (1996) • The locus of control refers to how people explain events that happen to themselves and others. • internal locus - guided by his/her personal decisions and efforts. • external locus - guided by fate, luck, or other external circumstances • a more 'surface' approach to learning is associated with an external locus of control, Watkins & Biggs (1996) • Scoring scheme & feedback helps to equate internal expectation (probability answered correctly) with external performance (score) • desirable attribute that is associated with an internal locus of control

  24. A model for self-concept • Posited to be multifaceted and hierarchical in nature • Shavelson, Hubner and Stanton (1976) • Numerous studies now based or extended from this model Hattie (1992)

  25. Self-concept and academic achievement • From the literature… • Support for the multifaceted nature…. • academic self-concept and academic achievement strongly correlated • Marsh, Byrne and Shavelson (1988), Hattie (1992) • little correlation between social self-concepts and academic achievementSong & Hattie (1984), Waugh (1999) • Support for the hierarchical nature less clear cut…. • more support for a hierarchical model for adolescents and • a unitary structure for younger children Hattie (1992) • achievement and academic self-concept deserve special interpretations at the level of specific subjects Marsh (1990) • standard self-description questionnaires based on the Shavelson model for preadolescents, adolescents and late adolescents Marsh (1992a, b, c)

  26. Measuring academic self-concept • Your perceptions of your ability and achievement • adapted from Marsh(1992), Song & Hattie (1984) and Waugh (2001) • indicate your relative agreement with each of the 20 statements: • All the time, or nearly all the time • Most of the time • Some of the time • None of the time, or almost none of the time • 1st 10 general university experience • eg. I am capable of getting good marks at university • I am proud of my achievements at university • 2nd 10 experience of English Language classes only • eg. I am sure of myself in English Language classes • I am achieving at a high level in English Language classes Questionnaire feedback For your general university experience, your responses total 18 points (30 maximum). For your English Language classes experience, your responses total 19 points (30 maximum).

  27. Measuring academic self-concept • Your attitudes to academic life • locus of control inventory adapted from Trice (1985) • select True or False to indicate your agreement with each statement: • eg. My academic marks most often reflect the effort I put into classes. • I came to university because it was expected of me Questionnaire feedback

  28. Correlating academic self-concept, confidence and achievement • perception of ability and achievement scores (general and EFL scales), increase in tandem. • as locus scores become increasingly internal (towards 0), perception of ability and achievement scores increase accordingly

  29. Gender observations in the questionnaire responses Scores from questionnaire: Females tended to respond with lower scores than males for the perception of ability and achievement (general and EFL) scales. The responses indicating the most internal of locus of control scores, were male students.

  30. Correlating academic self-concept, confidence and achievement • Student’s perception of ability and achievement in EFL shows a moderate correlation with their IELTS exam result. • in line with academic self-concept and achievement research – strongly subject based: support for the hierarchical structure. • No significant gender differences in the confidence-based test scores. • Females attained higher scores in the IELTS exam.

  31. Correlating academic self-concept, confidence and achievement • Perception of EFL ability & achievement scores are moderately correlated with the number of correct answers at high confidence. • These figures may indicate a trend, but may be noise • one confidence-based test is insufficient to identify genuine correlations

  32. Conclusions • Indications that Chinese students perform in a similar manner to confidence-based assessment as their UK counterparts. • Provide a scheduled series of confidence-based tests for Chinese students on the English Foundation programme • Examine academic self-concept and confidence-based assessment • potentially valuable role for the enhancement of academic self-concept, and the development of other positive academic behaviours • Effective with younger learners who may enjoy a game perspective • The scoring scheme and test feedback seems applicable to recent research that posits a reciprocal relationship between academic self-concept and academic achievement (Marsh, 2003). Acknowledgments • Lecturing staff in the Centre for Business Languages, Napier University • Nicola Beasley (Napier University) for initial software development • Tony Gardner-Medwin (UCL) for valued discussions and analysis

  33. Bibliography Brookover, W.B., Thomas, S. and Paterson A. (1964) Self-concept of ability and school achievement, Sociology of Education, 37, pp. 271-279. Craven, R. (1996) Enhancing Academic Self-Concept: A Large-Scale Longitudinal Study in an Educational Setting, PhD thesis, University of Sydney Davies, P. (2002) There’s no confidence in multiple-choice testing, Proceedings of the 6th International CAA conference, Loughborough, pp. 119-130. Echternacht, G.J. (1972) The use of confidence testing in objective tests, Review of Educational Research, 42:2, pp. 217-237. Gardner-Medwin, A.R. and Gahan M. (2003) Formative and summative confidence-based assessment, Proceedings of the 7th International CAA conference, Loughborough, pp. 147-155. Gourlay, L. (2004) Crossing Boundaries: A Case Study Masters Level Chinese Students, LTSN (In Press) Hassmen, P. and Hunt, D.P. (1994) Human self-assessment in multiple-choice testing, Journal of Educational Measurement, 31, pp. 149-160.

  34. Bibliography Hattie, J. (1992) Self-Concept, Lawrence Erlbaum Associates, New Jersey Khan, K.S., Davies, D.A. and Gupta, J.K. (2001) Formative self assessment using multiple true-false questions on the Internet: feedback according to confidence about correct knowledge. Medical Teacher, 23, pp. 158-163. Marsh, H.W (1992a) Self-Description Questionnaire (SDQ)III: A theoretical and empirical basis for the measurement of multiple dimensions of preadolescent self-concept: A test manual and research monograph, MacArthur, New South Wales, Australia, University of Western Sydney, Faculty of Education. Marsh H.W., Byrne, B.M. and Shavelson, R.J. (1988) A multifaceted academic self-concept; Its hierarchical structure and its relation to academic achievement. Journal of Educational Psychology, 80, 366-380. Marsh H.W. (1990b) The structure of academic self-concept: The Marsh/Shavelson model, Journal of Educational Psychology, 82, 623-636. Marsh, H.W. (2003) A reciprocal effects model of the causal ordering of academic self-concept and achievement, Paper presented at NZARE AARE, Auckland, New Zealand.

  35. Bibliography Prendergast, M.A. and Binder D.M. (1975) Relationships of selected self-concept and academic achievement measures, Measurement and Evaluation Guidance, 8, pp. 92-95. Song, I.S. and Hattie, J. (1984) Home environment, self-concept and academic achievement measures, Journal of Educational Psychology, 76, pp. 1269-1281. Watkins, D.A. and Biggs, J.B. (Eds, 1996) The Chinese Learner: Cultural, Psychological and Contextual Influences, CERC & ACER, (ISBN 0 86431 182 6). Waugh, R.F. (2001) Measuring ideal and real self-concept on the same scale, based on a multifaceted, hierarchical model of self-concept, Educational and Psychological Measurement, 61:1, pp. 85-101.

More Related