Matthieu Hermet, Stan Szpakowicz

1. Matthieu Hermet, Stan Szpakowicz Automated Analysis of Students’ Free-text Answers for Computer-Assisted Assessment University of Ottawa, Canada

2. CAA for CALL • To address the specificity of CALL… → where student material contains syntactic and orthographic errors • …with minimal pre-encoded material : • Content validation : simple • Form validation : difficult → A good case for automating based on Natural Language Processing

3. Text comprehension • The uOttawa project: CALL solutions for helping French-as-a-Second-Language students to enhance autonomous reading comprehension → master the structure of text in order to understand the author’s discursive intention → guess the meaning of unknown words → develop reformulation and synthesis capabilities

4. DidaLect …is a FSL tool aimed at teaching autonomous reading skill (designed for intermediate- and advanced-level students) • Is an instance of eLearning Intelligent Tutoring System: • adaptation to individual student’s skills and agenda • access to external resources (dictionaries) • built to reflect the cognitive concerns such as matching feedback to the student’s behaviour

5. Intelligence in DidaLect • DidaLect begins its operation with a placement test to determine a student’s initial level: • varying order of questions to pick up the best of a student’s skill • the implementation includes fuzzy logic methods • A separate element of DidaLect is the processing of free-text answers: • need of a robust CAA component • a trade-off between symbolic processing and Machine Learning techniques

6. Free-text answer assessment • The problem is to know in advance what material to expect in student answers. • Usually implemented as a classification problem: a student answer must match reference answer(s). → Size and form of reference material affects the process • Here, a reference answer is the text itself → A case for trying symbolic processing using techniques of Computational Linguistics

7. Expected limitations • No possibility of modifying the size and form of the reference material, except by automatic processing to control reformulation. • Therefore, this only works for limited forms of questions. • Strong need to ground selection of questions in a firm didactic theory. • Questions on texts for Text Comprehension (didactics offers a classification of question types).

8. Text-Explicit: based on a single sentence « d'habitude, l'hermaphrodisme frappe surtout les mâles, qu'elle dote de simulacres d'appareils génitaux féminins. » Q : Qu’arrive-t-il aux ours mâles lorsqu’ils sont frappés d’hermaphrodisme ? Ex R : « Ils ont les génitaux féminins » Text-Implicit : based on two co-referenced sentences « le détecteur de décélération situé à l'avant du véhicule génère instantanément un courant électrique, qui déclenche une amorce, qui elle-même enflamme un mélange allumeur. Ce dernier met finalement le feu à l'agent propulseur responsable du gonflement du coussin. » Q : Quelle est la réaction en chaîne qui se produit lorsque survient un impact ? Question types 1

9. Question types 2 • Identification, cause-effect, goal, comparison, definition, instrumental… • These categories express linguistically through lexical connectors • Goal : for, so that, in order to… • Cause–Effect : because, therefore… • So, the control of reformulation can be automated

10. Processing • Find lexical differences between the student’s answer S and reference R • Parse S and R, produce dependency relations • Process different words (using a dictionary) to detect synonyms • Control of syntax in S • Control of reformulation in S wrt R

11. Tools • A robust parser that enables partial recovery from errors in student’s answer • A dictionary of synonyms • A derivational dictionary • Locally derived resources: • State and action verbs • Ensemble of typical errors, set of syntactic and reformulation structures

12. Semantics and synonyms • Examine word set differences and commonalities in search for: • Common words • Reformulated words • Different words • Detect synonyms accross parts of speech: • Derive forms for a word lemma • Search synonyms for each form and look for a match in Word Sets

13. Syntax and reformulation • Correct syntactical structures to verify syntax of student’s answer • Lexicalized reformulation structures to verify discourse conformity Ex : pollution has increased with the rise of transportation Q : Why has pollution increased ? Ans : With the rise of transportation is partially wrong → Because of the rise of transportation OR it has increased due to the rise of transportation ETC.

14. Parsing and tree-building S : Le cardio-vasculaire d'un rat s'approche à une personne humain. SUBJ(<approche^approcher:53>, <cardio-vasculaire^cardio-vasculaire:48>) OBJ(<approche^approcher:53>, <personne^personne:56>) VMOD_POSIT1(<approche^approcher:53>, <une^un:55>) NMOD_POSIT1(<cardio-vasculaire^cardio-vasculaire:48>, <rat^rat:51>) PREPOBJ(<une^un:55>, <à^à:54>) PREPOBJ(<rat^rat:51>, <d'^de:49>) DETERM(<rat^rat:51>, <un^un:50>) The above are incrementally recomposed, based on lexical selection that maximizes promise of discovering material which diverges from R. That material is processed in parallel, in a similar fashion: SUBJ( <OBJ(<approcher>, <NMOD(<être>, <humain>)>)>, <NMOD(<NMOD(<système>, <cardio-vasculaire>)>, <rat>)>)

15. Main types of heuristics • To address syntactic correctness and/or equivalence between S and R: the same sense but different structures → bank of typical errors and correct structures • To address discursive variations, detected as supplementary material → bank of state and action verbs: action verbs must be present, possibly reformulated • To address, partially, errors in S → word replacement to relaunch parsing when stopped due to lexical mistakes

16. Reformulation rules • Examples : • Abstraction : incidence sur le temps de gestation → incidence sur la possibilité d’avoir une gestation écourtée (words like fact, chance, etc.) • Cause-Effect : Le plasma augmente et dilue les paramètres chimiques → L’augmentation du plasma dilue les paramètres chimiques • Is-A : Le rat est un animal qui + S → Le rat + S • Attribute : Le rat possède un système cardio-vasculaire → Le système C-V du rat

17. Assessment • Must give student feedback on: • Agreement and orthography • Syntax: signal errors and provides correction via display of a correct structure • Semantics: signals error and provides admissible words • Completeness of content with respect to R

18. R: Et puisque le rat est un animal qui possède un système cardio-vasculaire très semblable à celui de l’humain, il est donc permis de tirer les mêmes conclusions pour l’humain. Q: Pourquoi peut-on tirer les mêmes conclusions pour l'humain et pour le rat ? S: Le cardio-vasculaire d’un rat s’approche à une personne humain. Start by creating wordlists : Words of S absent in R → s’approcher, personne Words of R absent in S → animal, posséder, système, semblable Common words → rat, cardio-vasculaire, humain Example

19. S: SUBJ(<approche^approcher:53>, <cardio-vasculaire^cardio-vasculaire:48>) OBJ(<approche^approcher:53>, <personne^personne:56>) VMOD_POSIT1(<approche^approcher:53>, <une^un:55>) NMOD_POSIT1(<cardio-vasculaire^cardio-vasculaire:48>, <rat^rat:51>) PREPOBJ(<une^un:55>, <à^à:54>) PREPOBJ(<rat^rat:51>, <d'^de:49>) DETERM(<rat^rat:51>, <un^un:50>) R: SUBJ(<est^être:2>,<rat^rat:1>) OBJ_SPRED(<est^être:2>, <animal^animal:4>) OBJ(<possède^posséder:6>, <système^système:8>) COREF_REL(<animal^animal:4>, <qui^qui:5>) NMOD_POSIT1(<système^système:8>, <cardio-vasculaire^cardio-vasculaire:9>) NMOD_POSIT1(<système^système:8>, <semblable^semblable:11>) NMOD_POSIT1(<celui^celui:13>, <humain^humain:16>) ADJMOD(<semblable^semblable:11>, <celui^celui:13>) PREPOBJ(<humain^humain:16>, <de^de:14>) PREPOBJ(<celui^celui:13>, <à^à:12>) DETERM(<système^système:8>, <un^un:7>) DETERM(<animal^animal:4>, <un^un:3>) DETERM_DEF(<rat^rat:1>, <le^le:0>) CONNECT_REL(<possède^posséder:6>, <qui^qui:5>) Parse (partial output)

20. Comparison (partial) SUBJ(< OBJ(<approche^approcher:53>,<personne^personne:56>) >,<NMOD_POSIT1(<cardio-vasculaire^cardio-vasculaire:48>,<rat^rat:51>)>) SUBJ(<OBJ_SPRED(<est^être:2>,< COREF_REL(<animal^animal:4>,< CONNECT_REL(< OBJ(<possède^posséder:6>,< NMOD_POSIT1(< NMOD_POSIT1(<système^système:8>,<cardio-vasculaire^cardio-vasculaire:9>) >,<ADJMOD(<semblable^semblable:11>,< NMOD_POSIT1(<celui^celui:13>,<humain^humain:16>) >)>)>) >,<qui^qui:5>) >) >)>,<rat^rat:1>) • Consider structures to assess for syntactic correctness • Heuristics to put some structures into equivalence : → Here, «Le SCV du rat» and «Le rat est un animal qui possède un SCV» are equivalent, but expressed in different syntactic structures

21. - <RESULTS> - <DEF> - <E L="fr"> <W>semblable</W> <SW ENC="n">00730065006D0062006C00610062006C0065</SW> <P>adj.</P> </E> - <DF N="1"> <W>qui ressemble à; comparable, similaire.</W> </DF> </DEF> - <DEF> ….. - <WD L="fr" W="2"> <W L="fr">approchant</W> <SW ENC="n">0061007000700072006F006300680061006E0074</SW> </WD> To retrieve a synonymy relation : Produce derivations for all words in List 1 Find matches in a synonymy basis under entries for words of List 2 The search process can be repeated at most once, using <DEF> lexemes → semblable = approchant OR → ressemblerà = s’approcher de 4. In this way we can catch both synonyms and attached prepositions. Synonyms

22. Conclusions and future work • Automation is possible, with 2 main restrictions : • « bad faith » answers • Lexical errors based on homonymy → as long as S contains elements of answer, S can be evaluated • Future Work : to assemble the parts through software engineering !