1 / 100

Spoken Language Support for Software Development

Spoken Language Support for Software Development. Andrew Begel Advisor: Susan L. Graham Computer Science Division, EECS University of California, Berkeley. Human-Computer Interaction. Programming Languages. Human-Centric Software Tools. Software Engineering.

devin
Download Presentation

Spoken Language Support for Software Development

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Spoken Language Support for Software Development Andrew Begel Advisor: Susan L. Graham Computer Science Division, EECS University of California, Berkeley

  2. Human-Computer Interaction Programming Languages Human-Centric Software Tools Software Engineering Interactive Software Development

  3. while (counter < limit) {  } Motivation • Programmers conventionally use keyboard • Long hours at keyboard leads to higher risk of RSI • Can speech-based programming be an alternative? • Combines an unambiguous domain (programming) with an inherently ambiguous input modality (speech) • Great for exploring ambiguity handling in a new context

  4. while counter is lessthan limit do ... Programming by Voice • My Goal • Find out how developers use code verbally. Use this to develop a naturally verbalizable input form. • Build development environment that supports verbal authoring, navigation, modification. • Extend conventional compiler analyses to support ambiguities generated by speech. • Learn how developers can use voice-based programming, and iterate design.

  5. Challenges • Programming languages were not designed to be spoken. • Speech is inherently ambiguous. Programming tools were not designed for ambiguity. • Speech tools are poorly suited for programming tasks. • Programmers are not used to verbal software development.

  6. Talk Outline • Introduction and Motivation • Programming by Voice • Program Analyses for Ambiguous Inputs • Program Navigation and Editing • Conclusion

  7. Programming by Voice for (int i = 0; i < 10; i++ ) { ▌ }

  8. Current Tools are Awkward! VoiceCode [Desilets 2004] for loop … after left paren … declare india of type integer … assign zero … after semi … recall one … less than ten … after semi … recall one … increment … after left brace for (▌ ;; ) { }

  9. i 1 2 3 4 Current Tools are Awkward! VoiceCode [Desilets 2004] for loop … after left paren … declare india of type integer … assign zero … after semi … recall one … less than ten … after semi … recall one … increment … after left brace for (int i = 0;▌ ; ) { } i

  10. i 1 2 3 4 Current Tools are Awkward! VoiceCode [Desilets 2004] for loop … after left paren … declare india of type integer … assign zero … after semi … recall one … less than ten … after semi … recall one … increment … after left brace for (int i = 0; i < 10;▌ ) { } i

  11. i i 1 1 2 2 3 3 4 4 Current Tools are Awkward! VoiceCode [Desilets 2004] for loop … after left paren … declare india of type integer … assign zero … after semi … recall one … less than ten … after semi … recall one … increment … after left brace for (int i = 0; i < 10; i++ ) { ▌ } i

  12. Programming by Voice Related Work Multiple Tasks Begel ‘05 Arnold ‘00 Snell ‘00 Desilets ‘01 ‘04 Price ‘00 ‘02 AuthoringOnly Gray ‘03 Computer-Centric Human-Centric

  13. How do Programmers Speak Code? • 10 programmers read Java code out loud (Begel ‘05) • Most programmers spoke the same way

  14. How do Programmers Speak Code? for (int i = 0; i < 10; i++ ) { ▌ } for int i equals zero i less than ten i plus plus

  15. How do Programmers Speak Code? • Awkwardness by Design (Structural) • (int)foo • (3 + 5) * 7

  16. How do Programmers Speak Code? • Individual InconsistencySystem.out.println vs. System out println • bar sub i vs. bar of i vs. i from bar

  17. How do Programmers Speak Code? Native English speakers vs. non-native speakers (Pronunciation) tur vs. t u r println array[i++]vs.array[i]++

  18. A Natural Way to Code for (int i = 0; i < 10; i++ ) { ▌ } for int i equals zero i less than ten i plus plus

  19. KW or ID? for int i equals zero i less than ten i plus plus Spelling of ID? KW or #? 4inteyeequals0 ayelessthenten i plus plus Too Many Ambiguities for (int i = 0; i < 10; i++ ) { ▌ }

  20. for times equals 8 file 2 load times equals one Sometimes It’s Non-Obvious fore *= 8; file.tooLode.times = won ▌ for (times = 8; file(2, load); times == one) { ▌ } 4; times = ate(file).to(load).equals(1) ▌

  21. Command Language Easy to analyze, but prescriptive NaturalLanguage Flexible, but ambiguous Programming by Voice Design Tradeoffs

  22. Spoken Java • Semantically identical to Java • Syntactically easier to say than Java • Methodology generalizable to any computer language • All punctuation has English equivalents • Open Brace, End For Loop • Most punctuation is optional • Provide verbalization for all abbreviations • Relaxed phrasing for better fit with English • (int)foo “cast foo to integer” • foo = 6 “set foo to 6” • foo[i]++ “increment the ith element of array foo”

  23. SPEED: Speech Editor • Build an editor that supports naturally verbalized programs • SPEED: SPEech EDitor • Based on IBM ViaVoice, Eclipse IDE, Harmonia • Spoken Java Language for Composition • Spoken Command language for Navigation, Editing, Template instantiation, Refactorings, Search • Audible and visual feedback • Similar to JavaSpeak (Smith 2000)

  24. Harmonia Analysis Framework • Framework to support interactive editors • Language-based, programmer-oriented tools • Incremental analyses • Lexing (Wagner ‘97), GLR Parsing (Wagner ‘97, Begel ‘04), Static Semantics (Garrison ‘87, Begel, Jamison) • C, Java, Titanium, Cool, Flex, Bison • Also, languages where indentation and CRs are significant • Interactive Program Transformations (Java) • CodeLink (Toomim et. al. ‘04) • Shorthand Editing

  25. Talk Outline • Introduction and Motivation • Programming by Voice • Program Analyses for Ambiguous Inputs • Program Navigation and Editing • Conclusion

  26. Traditional Compiler Analyses for (i = 0; i < 10; i++ ) {  } Programming languages are designed to be unambiguous LexicalAnalysis Parsing SemanticAnalysis For Loop Assign Expr FOR I FOR I = 0

  27. 4 EYE Ambiguity-Aware Analyses for i equals zero ... Handles input stream, syntactic and semantic ambiguities LexicalAnalysis AmbiguousParsing SemanticAmbiguityResolution Ambig Stmt  FOR I For Loop Assign Expr FOUREYE = 0  FOR Assign Expr I = 0

  28. Scan Input Stream Homophone Dictionary LexicalAnalysis CommercialSpeechRecognizer

  29. 4 eye fore = four aye == Concatenated words cause them too for i = foreeye == foriequals 4 i equals fore ayeequals foureyeequals Homophones Cause Ambiguities for i equals

  30. 4 EYE Ambiguity-Aware Analyses for i equals zero ... LexicalAnalysis XGLRAmbiguousParsing SemanticAmbiguityResolution Ambig Stmt  FOR I For Loop Assign Expr FOUREYE = 0  FOR Assign Expr I = 0

  31. Expr Expr IF IF FOO BAR ELSE KW KW ID ID KW GLR Parsing [ Tomita 85 ]

  32. Expr Expr IF IF FOO BAR ELSE KW KW ID ID KW GLR Parsing [ Tomita 85 ]

  33. Expr Expr IF IF FOO BAR ELSE KW KW ID ID KW GLR Parsing [ Tomita 85 ]

  34. Expr Expr IF IF FOO BAR ELSE KW KW ID ID KW GLR Parsing [ Tomita 85 ]

  35. Expr Expr IF IF FOO BAR ELSE KW KW ID ID KW GLR Parsing [ Tomita 85 ]

  36. Expr Expr IF IF FOO BAR ELSE KW KW ID ID KW GLR Parsing [ Tomita 85 ]

  37. Expr Expr Expr Expr IF IF IF IF BAR FOO FOO ELSE KW KW KW KW ID ID ID KW GLR Parsing [ Tomita 85 ]

  38. Expr Expr Expr Expr IF IF IF IF BAR FOO FOO ELSE KW KW KW KW ID ID ID KW GLR Parsing [ Tomita 85 ] Stmt

  39. Expr Expr Expr Expr IF IF IF IF BAR FOO FOO ELSE ELSE KW KW KW KW ID ID ID KW KW GLR Parsing [ Tomita 85 ] Stmt

  40. Expr Expr Expr Expr IF IF IF IF BAR BAR FOO FOO ELSE ELSE KW KW KW KW ID ID ID ID KW KW GLR Parsing [ Tomita 85 ] Stmt

  41. Expr Expr Expr Expr IF IF IF IF FOO FOO BAR BAR ELSE ELSE KW KW KW KW ID ID ID ID KW KW GLR Parsing [ Tomita 85 ] Stmt Stmt

  42. Expr Expr Expr Expr IF IF IF IF FOO FOO BAR BAR ELSE ELSE KW KW KW KW ID ID ID ID KW KW GLR Parsing [ Tomita 85 ] Stmt Stmt Stmt

  43. Expr Expr Expr Expr IF IF IF IF FOO FOO BAR BAR ELSE ELSE KW KW KW KW ID ID ID ID KW KW GLR Parsing [ Tomita 85 ] Stmt Stmt Stmt Stmt

  44. GLR Ambiguity Support • Fork when there is more than one possible interpretation (tree)

  45. XGLR Ambiguity Support • Fork when there is more than one possible interpretation (tree) • Fork when the lexical input is ambiguous

  46. XGLR Parsing [Begel 04] IF FIFTY FIVE < X

  47. XGLR Parsing [ Begel 04 ] IF FIFTY FIVE < X KW

  48. IF KW XGLR Parsing [ Begel 04 ] FIFTY FIVE < X

  49. XGLR Parsing [ Begel 04 ] 55 < X # 50 5 < X # # 50 FIVE < X # ID FIFTY 5 < X ID # IF FIFTY FIVE < X ID KW ID

  50. 55 # 50 # IF IF IF IF IF KW KW KW KW KW XGLR Parsing [ Begel 04 ] < X Op 5 < X # 50 FIVE < X # ID FIFTY 5 < X ID # FIFTY FIVE < X ID ID

More Related