GRYPHON

1. GRYPHON Taylor Curtis Eric Lantz Kate Nelson Gio Messner Michelle Phillips

2. General Framework Text-to-speech Speech-to-text Parser DM DB Sphinx Phoenix Dialogue Manager Database Festival

3. General Framework Text-to-speech Speech-to-text Parser DM DB Sphinx Phoenix Dialogue Manager Database Festival

4. General Framework Text-to-speech Speech-to-text Parser DM DB Sphinx Phoenix Dialogue Manager Database Festival

5. General Framework Text-to-speech Speech-to-text Parser DM DB Sphinx Phoenix Dialogue Manager Database Festival

6. General Framework Text-to-speech Speech-to-text Parser DM DB Sphinx Phoenix Dialogue Manager Database Festival

7. Galaxy Communicator Parser Speech-to-Text DM Server HUB Text Input Database Speech and Text Output

8. General Framework Text-to-speech Speech-to-text Parser DM DB Sphinx Phoenix Dialogue Manager Database Festival

9. Speech-to-text Real-time CMU Partial Hypotheses Vocabulary / .corpus File Hand dictionary / .handdict Phoneme Set SPHINX "KWON " "KWON CARON " "PAUL BIOLOGY GROW " "PAUL BIOLOGY COURSES " "PAUL BIOLOGY COURSES ARE AND " "PAUL BIOLOGY COURSES ARE THE " "WHAT BIOLOGY COURSES ARE THERE" TEXT

10. Phoneme Set Phoneme Example Translation AA odd AA D AE at AE T AH hut HH AH T AO ought AO T AW cow K AW AY hide HH AY D B be B IY CH cheese CH IY Z D dee D IY DH thee DH IY EH Ed EH D ER hurt HH ER T EY ate EY T F fee F IY G green G R IY N HH he HH IY IH it IH T IY eat IY T JH gee JH IY Phoneme Example Translation K key K IY L lee L IY M me M IY N knee N IY NG ping P IH NG OW oat OW T OY toy T OY P pee P IY R read R IY D S sea S IY SH she SH IY T tea T IY TH theta TH EY T AH UH hood HH UH D UW two T UW V vee V IY W we W IY Y yield Y IY L D Z zee Z IY ZH seizure S IY ZH ER

11. Text Input • Sphinx can limit system • Recognizes some voices poorly • Text input option increases reliability

12. General Framework Text-to-speech Speech-to-text Parser DM DB Sphinx Phoenix Dialogue Manager Database Festival

13. Phoenix • University of Colorado, Boulder • Semantic Frame Parser • No predefined grammar • Grammars are defined in Backus-Naur Form

14. Traditional syntax parse Our Advisor, Jeff Ondich Keyword parse Animal ActionFood moose ate leaves

15. Our Grammar Is Somewhere In Between "What Computer Science courses does Jeff Ondich teach?" [Department] [Professor] [_Cs] [First_name] [Last_name] Computer Science Jeff Ondich

16. Format of a Phoenix Grammar File Frame: Courses Nets: [Department] [Professor] and elsewhere in the file: [Department] ([_Biol]) ([_Cs]) ; [_Cs] (CS) (Computer Science) ; [Professor] (*[First_name] [Last_name]) ; [First_name] (Jeff) (David) ; [Last_name] (Ondich) (Musicant) ; Net definitions are in brackets Slots (subnets) Strings in parentheses without brackets are terminals [Professor] [First_name] [Last_name] Jeff Ondich

17. An Example Parse “Show me all 5As in English and Philosophy” 1 [Class_period] [_And] [Department] [Department] 2 [_5a] [_Engl] And [_Phil] 5As English Philosophy Courses:[Class_Period].5a Courses:[Department].Engl Logic:And Courses:[Department].Phil 3 “Courses” and “Logic” are frames (“Respond” is the third one we use)

18. General Framework Text-to-speech Speech-to-text Parser DM DB Sphinx Phoenix Dialogue Manager Database Festival

19. One Huge Table? • Bad • Redundant • Repetitive +----+---------+------+--------------------------------------------------+------------+-------+------+-------------+-----------+ | id | syn_num | dept | title | build_name | room1 | day1 | start_time1 | end_time1 | +----+---------+------+--------------------------------------------------+------------+-------+------+-------------+-----------+ | 16 | 6398 | ocp | s i t australia:environment | | | | | | | 17 | 6399 | ocp | a c m florence | | | | | | | 18 | 6400 | astr | introduction to astronomy | olin | 149 | m | 01:50pm | 03:00pm | | 19 | 6401 | astr | observational and laboratory astronomy | good | 104 | t | 07:00pm | 10:00pm | | 20 | 6402 | biol | conservation biology | olin | 149 | m | 11:10am | 12:20pm | | 21 | 6402 | ents | conservation biology | olin | 149 | m | 11:10am | 12:20pm | | 22 | 6403 | astr | special project | | | | | | | 23 | 6405 | astr | special project | | | | | | | 24 | 6412 | biol | genes, evolution, and development | olin | 141 | m | 08:30am | 09:40am | | 25 | 6415 | phys | complexity and chaos | olin | 302 | m | 09:50am | 11:00am | | 26 | 6417 | phys | cellular automata:new science | olin | 302 | m | 08:30am | 09:40am | | 27 | 6418 | amst | introduction to u.s. latino/a studies | good | 3 | m | 01:50pm | 03:00pm | | 28 | 6419 | phys | newtonian mechanics | olin | 2 | m | 12:30pm | 01:40pm | | 29 | 6420 | biol | genes and evolution lab | hul | 206 | t | 01:00pm | 05:00pm | | 30 | 6421 | phys | newtonian mechanics lab | olin | 301 | t | 01:00pm | 05:00pm | | 31 | 6422 | biol | genes and evolution lab | hul | 206 | w | 02:00pm | 06:00pm | | 32 | 6423 | biol | genes and evolution lab | hul | 206 | th | 01:00pm | 05:00pm | | 33 | 6425 | phys | newtonian mechanics lab | olin | 301 | th | 01:00pm | 05:00pm | | 34 | 6426 | amst | the sublime in america | boli | 161 | t | 01:15pm | 03:00pm |

20. The Database Structure DAY TIMESPAN TIMEANDPLACE ROOM id start_time id id id room_id day_id end_time building_id day course_id timespan_id room COURSE BUILDING FULFILLS id id id syn_num build_name course_id title dist_id min_cred max_cred scrch_only COURSEINDEPT id course_num course_id section DISTRO dept_id id dist_name PROFESSOR TAUGHTBY id id DEPARTMENT first_name course_id id last_name professor_id dept_name

21. General Framework Text-to-speech Speech-to-text Parser DM DB Sphinx Phoenix Dialogue Manager Database Festival

22. Dialogue Manager • Query Manager • Logic Parser • SQL Query Builder • Output Director

23. Some Queries Can’t Stand Alone Can I see Geology as well? Where are they held? Show me the ones in the morning I said 3A What math classes are there? What’s in Sayles 251? Where are they held? Who teaches those? Who teaches those?”

24. A Query is a Continuation if… • It contains a continuation keyword • It contains no question words • It contains only question words “Show me the RAD classes too.” “And Physics.” “When is it?”

25. Combining the Old With the New OLD QUERY NEW QUERY Courses:[Class_Period].5a Courses:[Department].Engl Logic:And Courses:[Department].Phil Courses:[Class_Period].5a Courses:[Department].Engl Logic:And Courses:[Department].Phil Courses:[Class_Period].5a Courses:[Class_Period].5a Courses:[Department].Engl Courses:[Class_Period].5a Courses:[Department].Engl Logic:And Courses:Correction Courses:[Class_Period].3a Courses:Correction Courses:[Class_Period].3a “No, 3A” Combined Query Courses:[Class_Period].5a Courses:[Class_Period].3a Courses:[Department].Engl Logic:And Courses:[Department].Phil

26. Logic Parser “Real” Logic vs. Human Logic “Show me all the English and History courses.” If we interpret this as a logical AND, we won’t return any results. “Show me the courses that are English and RAD.” In this case, we do want to find courses that fulfill both of these criteria. • How can we get most cases right? • Change any occurrence of “and” to “or” unless it comes after the word “course” or “class”.

27. Resolving Ambiguity “Classes that are RAD and English or History.” This has two possible interpretations, because English uses infix ordering. and or RAD or and History English History RAD English Postfix order is unambiguous, so each of the above trees is mapped to a different postfix statement: RAD English History or and RAD English and History or

28. Which interpretation to use? • Precedence rules: • apply NOTs. • apply ORs between elements of the same type. • apply ANDs. • apply all other ORs • if a NOT is inside a “homogenous” clause, apply it to the whole clause

29. So, for example… Courses:[Class_Period].5a Courses:[Department].Engl Logic:And Courses:[Department].Phil Courses:[Class_Period].5a Courses:[Department].Engl Logic:Or Courses:[Department].Phil Courses:[Class_Period].5a Logic:And Courses:[Department].Engl Logic:Or Courses:[Department].Phil Courses:[Class_Period].5a Logic:And Courses:[Department].Engl Courses:[Department].Phil Logic:Or Courses:[Class_Period].5a Courses:[Department].Engl Courses:[Department].Phil Logic:Or Logic:And

30. Make SQL Query The Problem: Phoenix Output Database Courses:[Course_Period].5a Courses:[Department].Engl Courses:[Department].Phil Logic:Or Logic:And SELECT course.name FROM course WHERE course.id = 7; ? DAY TIMESPAN TIMEANDPLACE ROOM BUILDING DISTRO FULFILLS COURSE COURSEINDEPT PROFESSOR TAUGHTBY DEPARTMENT

31. SQL Syntax • SELECT the information (columns) we want to show • FROM the tables that have the information we want • WHERE certain things are true • JOINS connecting two tables • VALUES specified by user “What are the names of all courses in the English department?” SELECT course.nameFROM course, courseindept, department WHEREcourse.id = courseindept.course_id AND courseindept.dept_id = department.idAND department.name = ‘Engl’;

32. The Reason For Joins DAY TIMESPAN TIMEANDPLACE ROOM id start_time id id id room_id day_id end_time building_id day course_id timespan_id room Course names are here COURSE BUILDING FULFILLS id id id syn_num build_name course_id title dist_id min_cred max_cred scrch_only COURSEINDEPT We need to connect these tables together id course_num course_id section DISTRO dept_id id dist_name Dept names are here PROFESSOR TAUGHTBY last_name id DEPARTMENT first_name course_id id id professor_id dept_name SELECT course.nameFROM course, courseindept, department WHEREcourse.id = courseindept.course_id AND courseindept.dept_id = department.idAND department.name = ‘Engl’;

33. Getting Back to Our Problem • Endless possibilities for Phoenix output • But many outputs have similar form Courses:[Department].Geol Courses:[Department].Engl Courses:[Department].Soan Constant Portion Variable Portion • We want to look for constant portions and map the variable portions to appropriate SQL statements department.name = ‘Engl’ Courses:[Department].Engl

34. The Patterns File: Our New Best Friend Variable Portion (wildcard) Courses:[Department].$1 | course.name | course, courseindept, department | JOIN_COURSE_DEPARTMENT | department.name = $1 Add to SELECT field Add to FROM field Add to JOINS Add to VALUES Set equal to variable portion # JOIN_COURSE_DEPARTMENT = course.id = courseindept.course_id AND courseindept.dept_id = department.id SELECT course.nameFROM course, courseindept, department WHEREcourse.id = courseindept.course_id AND courseindept.dept_id = department.idAND department.name = ‘Engl’; Courses:[Department].Engl

35. Putting It All Together • For SELECT, FROM, and JOINS, we can simply concatenate the components for different lines • For VALUES, we need to take logic into account • (This is why we did all that crazy postfix stuff) Courses:[Class_Period].5a Courses:[Department].Engl Courses:[Department].Phil Logic:Or Logic:And 5a Engl Phil Phil (Phil Or Engl) Engl Or ((Phil Or Engl) And 5a) _______ Stack 5a And • If it’s not logic, put it on a stack • If it is logic, pop things off the stack, match to patterns file, combine, and put back on

36. Preparing Output • Send the formed SQL query to the database, and process the results Database Results status results Festival Mozilla

37. Why Bimodal Output • The project was originally envisioned with all-speech output • Searches can return hundreds of results • Festival speaks quite slowly • Not designed for telephone use • Text output of results allows user to find relevant results quickly • Mozilla provided a predefined user interface

38. General Framework Text-to-speech Speech-to-text Parser DM DB Sphinx Phoenix Dialogue Manager Database Festival

39. Festival • Multi-Lingual • Text-to-Speech • British and American English • Initial plan vs. actual implementation • Response

40. Thank You from G R Y P H O N raciously earranging our emes libly eimagining our emes o eady our arser ey