1 / 13

Christian Gütl, IICM, TU-GRAZ Instructions for Experiment

Christian Gütl, IICM, TU-GRAZ Instructions for Experiment. „Natural Language Processing“. The Experiment Procedure. Topic: “ Natural Language Processing (NLP) ” Objective of the Experiment Gain insight about extracting and evaluating concepts and generating questions Informed consent

phelan-mcmillan
Download Presentation

Christian Gütl, IICM, TU-GRAZ Instructions for Experiment

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Christian Gütl, IICM, TU-GRAZInstructions for Experiment „Natural Language Processing“

  2. The Experiment Procedure • Topic: “Natural Language Processing (NLP)” • Objective of the Experiment • Gaininsightaboutextractingandevaluatingconceptsandgeneratingquestions • Informed consent • Please fill in the informed consent Christian Gütl, IICM, TU-GRAZ

  3. The Experiment Procedure • Step 1: Learning the text • Step 2: Questionnaire 1 • Step 3: Learning Activity I (Questionnaire 2) • Step 4: Test (Questionnaire 3) • Step 5: Learning Activity II (Questionnaire 4) • Step 6: Questionnaire 5a • Please allow yourself breaks between tasks! Christian Gütl, IICM, TU-GRAZ

  4. Step 1 Please learn the text (max. 30 minutes)  See email (Subject: „NLP – text“) Christian Gütl, IICM, TU-GRAZ

  5. Step 2 Please fill in Questionnaire 1 (max. 10 min) • See email (Subject „phd Questionnaire 1“) • Please give all answers in English! • Please do not use the text while answering the questions! • Click on the „Submit“-Button once you have finished the task! Christian Gütl, IICM, TU-GRAZ

  6. Step 3a: Extracting concepts • Please extract the main concepts of the NLP-text, using Questionnaire 2 (max. 10 min) • (Example: Speed of light; theory of relativity, Einstein). • Please separate your concepts with semicolons (;) or commas (,) •  See email (Subject „ phd Questionnaire 2“) • After you have finished this step, please continue working on Questionnaire 2 (Step 3b) as described on the next slide Christian Gütl, IICM, TU-GRAZ

  7. Step 3b: Generating questions • Please generate eight questions from the NLP-text, using Questionnaire 2. • Construct two Open Ended questions using the text (Example: What do you know about theory of light?) •  Construct two Fill In The Blank questions using the text (Example: The founder of this university was ________.) •  Construct two Single Choice questions using the text. Their answer should be “no”! (Example: Albert Einstein was born in Norway) •  Construct two Multiple Choice questions with three distractors for each chapter.Onlyoneofthedistractorsshouldbecorrect! (Example: The sun is _____. blue, green, yellow, red) Note thatyouhavetogenerateonequestionforeachcategoryfirstandthenanotheroneafterwards. Christian Gütl, IICM, TU-GRAZ

  8. Step 4 Please fill in Questionnaire 3 (max 15 Minutes) and allow yourself a break afterwards. • See email (Subject „ phd Questionnaire 3“). • Please do not use the text while answering the questions! Christian Gütl, IICM, TU-GRAZ

  9. Step 5a: Evaluation of concepts Evaluation Task I (max. 10 min.) • Please evaluate the concepts you will find in Questionnaire 4. Do you think the concept is relevant for the text? • 1 = not relevant; 5 = very relevant • See email (Subject „ phd Questionnaire 4“). • After you have finished this task, please continue working on Questionnaire 4 (Step 5b) as described on the next slide Christian Gütl, IICM, TU-GRAZ

  10. Step 5b: Evaluation of questions Evaluation Task II (max. 35 min) Please evaluate the questions provided in Questionnaire 4, using the criteria as described below. For each criterion described below, please assign a score between 1 ( = very bad) and 5 (very good) to each question. Pertinence: Relevancy of the formulated questions. Has the question a precise and logical connection to the topic? Terminology: Appropriateness of the words chosen to formulate the question. Level: Extent in approaching the topic. Is the question trivial or expresses it a significant meaning? (1 = trivial, easy; 5 = difficult, significant meaning) Answer (only if provided!): Is the provided answer relevant? Distractors (only for multiple choice!): Quality of the computed distractors of the multiple choice questions Christian Gütl, IICM, TU-GRAZ

  11. Step 5c General Survey Please fill in the last section of Questionnaire 4 (max. 15 min) Christian Gütl, IICM, TU-GRAZ

  12. Step 6 Evaluating questions Evaluation Task III Please evaluate the questions provided in Questionnaire 5a (see email), using again the criteria as described below. For each criterion, please assign a score between 1 ( = very bad) and 5 (very good) to each question. Pertinence: Relevancy of the formulated questions. Has the question a precise and logical connection to the topic? Terminology: Appropriateness of the words chosen to formulate the question. Level: Extent in approaching the topic. Is the question trivial or expresses it a significant meaning? (1 = trivial, easy; 5 = difficult, significant meaning) Answer (only if provided!): Is the provided answer relevant? Distractors (only for multiple choice!): Quality of the computed distractors of the multiple choice questions Christian Gütl, IICM, TU-GRAZ

  13. Thank you for participating in this Experiment! Christian Gütl, IICM, TU-GRAZ

More Related