1 / 12

Using Verbal Reports for Data Collection and Analysis

Using Verbal Reports for Data Collection and Analysis. Maja Gjurovikj STANAG 6001 testing team Ministry of Defence Republic of Macedonia. Why trialling?. Collect data to be analyzed: Do items test what they were meant to test? Do they distinguish between masters or non-masters?

melissagrant
Download Presentation

Using Verbal Reports for Data Collection and Analysis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Using Verbal Reports for Data Collection and Analysis Maja Gjurovikj STANAG 6001 testing team Ministry of Defence Republic of Macedonia

  2. Why trialling? • Collect data to be analyzed: • Do items test what they were meant to test? • Do they distinguish between masters or non-masters? • Are the ambiguous? • Inform about test administration procedures: • time • instructions • answer keys • Bank items to generate new tests: • determine characteristics of items • keep/revise/reject

  3. Test item characteristics

  4. Psychometric approaches vs. mentalistic approaches empirical validity theoretical validity concurrent validity predictive validity QUANTITATIVE APPROACHES EXAMINE PSYCHOMETRIC PROPERTIES OF THE TEST face validity content validity construct validity QUALITATIVE APPROACHES PROVIDE MORE INFORMATION ON TEST CONTENT, TEST TASK PROPERTIES AND TEST-TAKING PROCESSES

  5. What is verbal report analysis? • a special label used to describe the verbal data gathered • a technique that draws inferences about the cognitive processes that produced the verbalization • comprised of ‘protocols’ - utterances made by an individual • invaluable at the trialling and pretesting stage • supplement other quantitative data

  6. Variations on verbal report procedure Form of Report TALK ALOUD THINK ALOUD Temporal Concurrent Retrospective Concurrent Retrospective Variations Procedural Mediated Non-med. Mediated Non-med. Mediated Non-med. Mediated Non-med. Variations

  7. Stages in collecting and analyzing verbal reports DATA COLLECTION STAGES • Task identification • Task analysis • Selecting an appropriate procedure • Selecting subjects • Training subjects • Collecting verbal reports • Collecting supplementary data • Transcribing verbal reports DATA ANALYSIS STAGES • Developing an encoding scheme • Segmenting protocols • Encoding protocols • Calculating encoder reliability • Analyzing data Source: Green, A. (1998). Verbal Protocol Analysis in Language Testing Research: A Handbook. Cambridge: Cambridge University Press and University of Cambridge Local Examinations Syndicate

  8. Examples of verbal report methodology • Cohen (1994) • five test-takers • a two-part test • concurrent reports • observe test-takers during the test-taking process • give test-takers a questionnaire • conduct in the test-takers’ L1 • Buck (1994) • six test-takers • 54-item test based on a single listening text • 13 short sections, one at a time • attend a post-test interview • take the test items again

  9. Advantages vs. disadvantages • reveals test validity • provides direct information • has wide application • commitment in time • disruption of behaviour • individual differences

  10. Some qualitative data analysis softwares • Coding Analysis Toolkit (CAT;  https://cat.texifter.com) • Weft QDA (http://www.pressure.to/qda) • Text Analysis Markup System (TAMS; http://tamsys.sourceforge.net) • Transana (https://www.transana.com) • QDA Miner Lite (https://provalisresearch.com/products/qualitative-data-analysis-software)

  11. Points to note • reveals the dynamics of the test taking processes • uncovers students’ unobservable behavior • resourceful feedback concerning the mental processes or test-taking strategies • complements the analysis generated from quantitative investigation • a developing and maturing method

  12. References: • Green, A. (1998). Verbal Protocol Analysis in Language Testing Research: A Handbook. Cambridge: Cambridge University Press and University of Cambridge Local Examinations Syndicate. • Grotjahn, R. (1986). Test validation and cognitive psychology: Some methodological considerations. Language Testing 3: 159–185. • Zheng, Y. (2009) Protocol Analysis in the Validation of Language Tests: • Potential of the Method, State of the Evidence. International Journal of Pedagogies and Learning 5: 124–137 • Council of Europe (2009). Relating Language examinations to the Common European Framework of Reference for Languages: learning, teaching, assessment Cambridge: Cambridge University Press. (https://www.youtube.com/watch?v=BwpPIiBK0cA)

More Related