URL : cslu.ogi/~sproatr/Courses /AAC/

1. URL: http://www.cslu.ogi.edu/~sproatr/Courses/AAC/ CS506/606: Seminar on Speech and Language Processing for Augmentative and Alternative CommunicationSpring 2010Symbol Systems & Even More Symbol Systems

2. SLP for AAC The papers… • Schlosser, R. W. and Sigafoos, J., 2002. Selecting graphic symbols for an initial request lexicon: Integrative review. Augmentative and Alternative Communication. 18(June), 102-123. • P. L. Albacete, S. K. Chang and G. Polese. 2006. Iconic language design for people with significant speech and multiple impairments. Assistive Technology and Artificial Intelligence. Lecture Notes in Computer Science, 1458. Springer. • Binger, C., Light, J., 2007. The effect of aided AAC modeling on the expression of multi-symbol messages by preschoolers who use AAC. Augmentative and Alternative Communication 23 (1), 30-43. • Patel, R., Pilato, S., & Roy, D. (2004). Beyond Linear Syntax: An Image-Orientation Communication Aid. Assistive Technology Outcomes and Benefits, 1(1), 57-67. • HamedH. Sad and Franck Poirier. Using Pictographic Representation, Syntactic Information and Gestures in Text Entry. Human-Computer Interaction, Part II, HCII 2009, LNCS 5611, pp. 735–744, 2009. • Sutton, A., Soto, G., Blockberger, S., 2002. Grammatical issues in graphical symbol communication. Augmentative and Alternative Communication 18 (September), 192-204.

3. SLP for AAC Schlosser et al. • Review of literature on selecting graphic symbols for initial request lexicons • Evidence-based practice: • Theory based • Research based • Anecdote based

4. SLP for AAC Issues • Iconicity & realism • How much the symbol resembles its referent • Speech output • Role of speech output is equivocal • Spoken language comprehension of the referent • Knowing a word referent seems to assist in learning opaque symbols • Concreteness of the referent • Role of concreteness of the referent has not been sufficiently studied • Reinforcement value of the referent • Translucency: “relation between iconicity and reinforcement value of the referent is still largelyunknown.” • Correspondence • Is there a correspondence between the symbol and the associated action?

5. SLP for AAC Directions for Future Research • Examine the role of iconicity (opaque versus iconic) on requesting referents of varying reinforcement status (preferred versus nonpreferred). • Effects of iconicity (opaque versus iconic) on requesting referents that are receptively known by the learner versus referents that are unknown. • Evaluate the role of iconicity in requesting referents that are concrete versus those that are abstract. • Investigate whether symbols of varying degrees of iconicity for referents yield differential requesting as a function of the mode of transmission (i.e., speech output versus no speech output).

6. SLP for AAC Implications for Clinical Practice

7. SLP for AAC Albacete et al • “Icon Algebra” + Conceptual Dependency • Introduces iconic languages • “Finally, there are "natural" iconic languages such as the Chinese ideographs, the Mayan glyphs and the Egyptian pictograms.” • This is 牛屎 • Semantic compaction – Minspeak • Map concepts onto multi-meaning icon sentences

8. SLP for AAC Minspeak iconic keyboard

9. SLP for AAC Minspeak formalization • APPLE+VERB eat • APPLE+NOUN food • Each of these icons is polyvalent • This paper provides a formalization of the system in terms of iconic algebra • Icon is a pair (Xm, Xi), where Xm is the meaning and Xi is the image • (This is just Saussure’s notion of signifier and signified)

10. SLP for AAC Iconic operators

11. SLP for AAC Iconic operators • MARker: • Marks the image of icon Y with the image of icon X to emphasize a local feature. 本 • CONtextual interpretation • Meaning of X considered in the context of Y • ENHancement • Enhances contextual richness of X by adding attributes of Y • INVersion • Meaning of X is inverted

12. SLP for AAC Conceptual dependency • Due to Schank • Basic principle: • For any two sentences that are identical in meaning, regardless of the underlying language, there should be only one representation. • Conceptual structure is a network of concepts

13. SLP for AAC Conceptual dependency

14. SLP for AAC CD for iconic languages

15. SLP for AAC Inference algorithm • Basic module builds a skeleton frame for the iconic sentence by searching for common attribute values in the frames of the single icons in the sentence, according to three match criteria. • Second module of the inference algorithm tries to extend the skeleton frame for the iconic sentence by applying iconic operators • Third module runs a brute force algorithm to provide further textual descriptions for the iconic sentence, aiming to further enrich its semantic frame.

16. SLP for AAC Example

17. SLP for AAC Example

18. SLP for AAC Binger & Light • Tested aided AAC in effecting multi-symbol messages with 5 preschoolers who used AAC • Children who use AAC frequently use one-symbol messages: • Lack of communicative efficiency • Symbols may encode multiple concepts (ice-cream  “I want an icecream”) • Communication partners often ask questions that require only one-word answers • There may be other factors due to AAC use • Asymmetry between input (speech) and output (AAC) modalities

19. SLP for AAC Aided AAC “Essentially, all that is required to convert a spoken model into an aided AAC model is to use natural speech while pointing toand labeling key graphic symbols on the AAC device (e.g., the child points to the symbol MILK, then the adult says More milk while pointing to MOREand MILK).”

20. SLP for AAC Subjects

21. SLP for AAC DiGeorge Syndrome (Wikipedia) Cognitive impairments Children with 22q11.2 have a specific profile in neuropsychological tests. They usually have a borderline normal IQ with most individuals having higher scores in the verbal than the nonverbal domains. Cognitive functioning when processing information involving space and time usually shows significant impairment and this generally slows the development of numerical and arithmetical knowledge and skills. Noteworthy is that these patients are a specifically high-risk group for developing schizophrenia. 30% have at least one incident of psychosis and about a quarter develop actual schizophrenia.

22. SLP for AAC Method • Seven play scenarios • Photographs + PCS • Independent variable was use of aided AAC

23. SLP for AAC Results

24. SLP for AAC Results

25. SLP for AAC Results

26. SLP for AAC Prader-Willi Syndrome (Wikipedia) Individuals with PWS are at risk of learning and attention difficulties. Curfs and Frym (1992) conducted research into the varying degrees of learning disability found in PraderWilli Syndrome (PWS).[12] Their results were as follows: * 5%: IQ above 85 (average to low average intelligence) * 27%: IQ 70 – 85 (borderline intellectual functioning) * 39%: IQ 50 – 70 (mild intellectual disability) * 27%: IQ 35 – 50 (moderate intellectual disability) * 1%: IQ 20 – 35 (severe intellectual disability) * <1%: IQ <20 (profound intellectual disability)

27. SLP for AAC Patel et al. • Proposes an AAC communication aid that is based on “semantic rather than syntactic schema” • Relates to the problem of linearization that we discussed in the introductory lecture. • Mention a boy named Paul who has problems with his current “linear composition system”

28. SLP for AAC Example:Element Cues (Howard Shane)

29. SLP for AAC Teaching Language Concepts (TLC)

30. SLP for AAC Image-oriented messaging interface

31. SLP for AAC Semantic schema

32. SLP for AAC Linearization

33. SLP for AAC Sad & Poirier • Aimed at text entry on mobile devices: • “We propose an interface to enter text that bypasses letter encoding. It’s based on the mental models of meanings already existing in the human mind, and can be processed at very high speed.” • Combines graphical elements with gestures

34. SLP for AAC Example

35. SLP for AAC Gestures • Down-right: verbs in the future tense • Down: verbs in the present tense • Down-left: verbs in the past tense • Left: delete the last typed word • Up-right: adjectives • Up: nouns • Up-left other parts of speech or grammatical categories

36. SLP for AAC Stats • Gives direct access to about 1500 common French words • 450 words cover about 50% of written/spoken French (they claim)

37. SLP for AAC Key disabling

38. SLP for AAC Reminds one a bit of science fiction Since, however, it is inherently impossible in a society based on speech to indicate truly the method of communication of Second Foundationers among themselves, the whole matter will be hereafter ignored. The First Speaker will be represented as speaking in ordinary fashion, and if the translation is not always entirely valid, it is at least the best that can be done under the circumstances. It will be pretended therefore, that the First Speaker did actually say, "First, I must tell you why you are here," instead of smiling just so and lifting a finger exactly thus.

39. SLP for AAC Sutton et al. • Examines the effects of graphic symbol communication on syntactic and morphological acquisition

40. SLP for AAC Production issues

41. SLP for AAC Other issues • Comprehension issues • Acquisition issues It is hard to underspecify in a graphical representation