Hidenobu KUNICHIKA, Minoru URUSHIMA, Tsukasa HIRASHIMA and Akira TAKEUCHI

1. A Computational Method of Complexity of Questions on Contents of English Sentences and its Evaluation Hidenobu KUNICHIKA, Minoru URUSHIMA, Tsukasa HIRASHIMA and Akira TAKEUCHI Dept. of AI, Kyushu Institute of Technology, Japan IEEE Proceedings of the International Conference on Computers in Education (ICCE). Presenter: Hsiao-Pei Chang

2. Outline • Introduction • The outline of the QA function • The definition of difficulty of questions • Evaluation • Conclusions

3. Introduction • QA (Question and Answer) about the contents of a story iswidely used in language learning. • QA in a target language iseffective for acquiring practical skills because learners useplural language skills to answer questions. • When a teacher and a learner practice such QA, the teacher will give suitable questions for the learner. • Most of computer assisted language learning systems use questions prepared beforehand, so teachers are burdened with preparing questions and answers for every learning material.

4. Introduction • The target of our study is to realize a QA function which provides suitable questions for each learner and tailored advice according to the learner’s answers. • To achieve the target, we need the following functions: (1) to understand English sentences (2) to generate various kinds of question sentences automatically (3) to select suitable questions for each learner from a set of generated question sentences (4) to analyze learners’ answer sentences and to diagnose errors

5. The outline of the QA function - Processing flow • the QA function generates question sentences aboutthe story as many as possible from syntactic and semanticinformation extracted by natural language processing. • the function calculates difficulty of each question for a learner and then, selects a suitable and purposive question for the learner by referring to the difficulty of questions, educational intentions and information about a story. • Learner's answers are analyzed by natural language processing, and syntactic and semantic information are extracted. The QA function also identifies syntactical errors. After that it judges the answers semantically by comparing semantic information of the answers with semantic information of the story. • Results of the judgment are stored into a student model and then used for selecting the next question.

6. The outline of the QA function – Controlling difficulty of questions • In order to keep learners’ high motivation, it is important to give suitable questions for each learner. • The QA function selects suitable questions by not only considering the learning targets but also referring to difficulty of questions for a learner. • For example, when a learning target is to master the use of grammatical knowledge and a learner has a poor record, the function selects questions which have low value of difficulty and use particular grammatical knowledge.

7. The outline of the QA function – Information to calculate difficulty • Our method of calculating difficulty of questions uses syntacticand semantic information generated by natural languageprocessing. We have proposed a method of extracting syntacticand semantic information of stories based on DCG(Definite Clause Grammar). • Syntactic information consists of a syntactic tree, whichexpresses parts of speech and modification relationships ofwords and phrases, and a feature structure, which expressesboth grammatical functions of words and phrases and grammaticalinformation such as sentence structure and idioms. • Semantic information consists of time and space informationand information about verbs, nouns and modifiers.

9. The definition of difficulty of questions • To answer questions, learners execute some processes. Thus difficulty of answering questions depends on difficulty of tasks executing the processes. • Factors that influence the difficulty are classified into two categories; • one is dependent on only sentence structure and independent of a learner's understanding state. → "complexity" of questions • The other is the learner's understanding state. • By combining complexity and the learner's understanding state, we can evaluate difficulty of questions for each learner. This paper mainly focuses on the definition of complexity, because available information about an understanding state depends on implementation of student models and it may differ system by system.

10. Complexity along the answering process • Process of answering questions consists of the followingthree sub processes; • understanding the contents of a story, • understandingthe content of a question sentence, • composingan answer sentence. • The complexity of a questions is representedby a triplet (D1, D2, D3). Each element correspondsto the complexity of each process. • we assume thateach complexity is proportional to its factors, and each complexityis calculated by weighted sum of the factors.

11. (1) Complexity of understanding the contents of sentences • we define the complexity of understandingthe meaning of each sentence in the story as the sum of“the complexity of a sentence” D1a and “the complexity ofthe paragraph which includes the sentence” D1b as follows. → D1 = D1a + D1b (a) Complexity of a sentence • The complexity of a sentence is defined as follows. D1a = w1a1 * f1a1 + w1a2 * f1a2 + w1a3 * f1a3+ w1a4 * f1a4 + w1a5 * f1a5 • “f”s are the factors of the complexity and “w”s are theirweights.

12. D1a = w1a1 * f1a1 + w1a2 * f1a2 + w1a3 * f1a3+ w1a4 * f1a4 + w1a5 * f1a5 • Factors of complexity of a sentence are as follows. f1a1) The number of propositions: • A proposition is thesmallest unit of meaning in sentences. Oneproposition consists of one predicate such as a verb, adjectiveor adverb, and one or more variables such as nounsor noun phrases. • The number of propositions is countedby referring to the syntactic tree. f1a2) Degree of conflict on identifying a referent: • It is necessaryto identify anaphoric relation in order to understand the contents of sentences. Thus the number of candidates for a referent influences on complexity of the process.

13. D1a = w1a1 * f1a1 + w1a2 * f1a2 + w1a3 * f1a3+ w1a4 * f1a4 + w1a5 * f1a5 f1a3) Distance between anaphoric expression and its referent: • Complexity of identifying anaphoric relation is alsoinfluenced by distance between anaphoric expression andits referent . Our method uses the number of prepositionbetween anaphoric expression and its referent as thedistance. f1a4) Complexity of modifying relation: • We assumethat it is complex to understand the content of a sentencein which modifying relation exists across some words. • Complexity of modifying relation of words influences complexity ofunderstanding the content of a sentence. f1a5) The number of clauses:

14. (b) Complexity of a paragraph • The complexity of a paragraph is defined as follows. D1b = w1b1 * f1b1 f1b1) Scatter of objects: • The complexity of understanding the contents of sentences depends on the degree of coherence of propositions [5]. • Generally, people think that there are some relationships between sentences if thesame object appears in all of the sentences. Therefore, we focus on objects in sentences and use scatter of objects defined as follows as a factor on the degree of coherence. Scatter of objects = (the number of appearance of objects /the number of different objects) -1

15. (2) Complexity of understanding the content of a question sentence • Complexity of understanding the content of a question sentence is the same as the complexity of understanding a sentence. The formula is as follows. D2 = w21 * f21 + w22 * f22 + w23 * f23 + w24 * f24 + w25 * f25 • f21) The number of propositions. • f22) Degree of conflict on identifying a referent. • f23) Distance between anaphoric expression and its referent. • f24) Complexity of modifying relation. • f25) The number of clauses.

16. (3) Complexity of composing an answer sentence • Generally, a variety of correct answers can be considered for each question. To calculate the complexity, we have to know the learner's answer sentence before asking the question, but it is impossible. • So, our method uses “Yes, ...” or “No, ...”, e.g. “Yes, he is.” for alternative questions and a sentence in a story used for question generation for questions using interrogative pronouns as an assumed answer. D3 = w31 * f31 + w32 * f32 + w33 * f33 + w34 * f34 +w35 * f35 + w36 * f36 • f31) The query form: • The complexity of composing an answersentence is influenced by the query form of a question. • To answer questions using interrogative pronounsis higher level because a learner must have the necessarywriting skill to compose an answer sentence.

17. Customizing complexity • Difficulty of understanding the contents of sentences dependson learners’ knowledge. So, in order to calculatedifficulty of questions, it is necessary to refer to informationabout learners’ understanding states. • A student model of HELEN has ability to assessa learner’s understanding statesabout vocabulary and grammars. • HELEN represents influence of alearner’s understanding states as the following triplet. → ((f1w, f1g), (f2w, f2g), (f3w, f3g))

18. f1w) The number of unfamiliar vocabularies in sentencesin a story used for question generation. f1g) The number of unfamiliar grammars in sentences in astory used for question generation. f2w) The number of unfamiliar vocabularies in a questionsentence. f2g) The number of unfamiliar grammars in a question sentence. f3w) The number of unfamiliar vocabularies in an assumedanswer. F3g) The number of unfamiliar grammars in an assumedanswer. • D1a = w1a1 * f1a1 + w1a2 * f1a2 + w1a3 * f1a3 + w1a4* f1a4 + w1a5 * f1a5 + w1w * f1w + w1g * f1g • D2 = w21 * f21 + w22 * f22 + w23 * f23 + w24 * f24 +w25 * f25 +w2w * f2w + w2g * f2g • D3 = w31 * f31 + w32 * f32 + w33 * f33 + w34 * f34 +w35 * f35 + w36 * f36 + w3w * f3w + w3g * f3g

19. Examples of calculating complexity of questions • We present examples of complexity of questions about thecontents of the following sentences which correspond to oneparagraph in a textbook “New Horizon” [12] for Japanesejunior high school students. This is my brother Fred. He goes to college in San Francisco. He is studying space science. He likes rock climbing. • The following question sentences are generated automaticallyfrom the above sentences by the automated questiongeneration function [8]. (4, 2, 5) Where does Fred go? (4, 3, 5) Who goes to college in San Francisco? (4, 3, 1) Does Fred go to college in San Francisco? (4.5, 1, 1) Does Fred love rock climbing? (9, 5, 1) Is Fred who goes to college in San Francisco studyingspace science?

20. Evaluation • To evaluate the appropriateness of our definition of complexity,we compared the difficulty with humans' judgment. • We gave subjects two questions, and asked them to judgewhich is difficult. Then, we evaluated both the recall rateand the precision. • In this evaluation,we define complexity of a question as the sum of threeelements of the triplet. • Complexityof understanding the contents of sentences • Complexityof understanding the content of a question • Complexityof composing an answer sentence.

21. The method of experimentation • Steps : • selected three stories in New Horizon and sevenparagraphs in the stories at random. • generated questionsentences from each paragraph by the automated questiongeneration function. • randomly selectedfour question sentences for each paragraph from the generatedquestions. → "questionset". • gave paragraphs and question sets to 16 subjects whoare undergraduate students, and asked the subjects to sort thequestion sentences of each question set in order of difficulty. • In this evaluation, we assumed that all factors affect equallyto the complexity. That is to say, all weights of these factorsare 1, and we calculated complexity of each question by addingup D1, D2 and D3.

22. The results and consideration (1) The recall rate • Each question set consists of four questions, sothere are 6 pairs of questions in each question set. And thereare 7 paragraphs. Thus there are 42 pairs of questions in total. • As the result,we have found that there are 23 pairs in whichalmost subjects have made the same judgment, and there are19 pairs out of the 23 pairs in which our method have madethe same judgment. Therefore, the recall rate is about 83%.

23. The results and consideration (2) The precision rate In order to select difficult or easy questions, the QA functioncompares values of difficulty of questions. Thus, it isnecessary to define a threshold value used for judgment ofsignificant difference. A covering rate is a rate of number of the pairs in which difference of values of difficulty is bigger than the threshold value.

24. Conclusions • This paper has described the definition of difficulty ofquestions for selecting adaptive questions for learners‘ understandingstates and the mechanism of its calculation. • Wehave compared judgments by humans and our method. Asthe result, we have seen that our method is useful if the thresholdvalue is decided by considering the balance of the precisionrate and the covering rate.

25. Difficulty of articles – Readability features • Lexical features • Word frequency • BNC&Google count • 專業分級指標 • 教育部-詞彙分級表 (6級) • 英檢單字分級 (3級) • Average number of syllables per word • Grammatical features • Sentence length • Word count • 文法難度分級 (遠東課本1~6冊) • Parse features • Average parse tree height • Average number of noun phrases • Average number of verb phrases • Average number of SBARs

27. Difficulty of questions • 單字考題 • 正確答案單字在大考中心list的level (distractors也會選相同level的單字) • 文法考題 • 題目要考的文法pattern的難度level • 閱讀考題 • 原本文章的難度level