1 / 60

Tutorial T4 - Programming by Example Techniques

Tutorial T4 - Programming by Example Techniques. Brad A. Myers Human Computer Interaction Institute School of Computer Science Carnegie Mellon University Pittsburgh, PA  15213-3891 (412) 268-5150 bam+@cs.cmu.edu http://www.cs.cmu.edu/~bam. Contents:. Abstract & Bio Slides from Talk

fawzi
Download Presentation

Tutorial T4 - Programming by Example Techniques

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Tutorial T4 - Programming by Example Techniques Brad A. MyersHuman Computer Interaction InstituteSchool of Computer ScienceCarnegie Mellon UniversityPittsburgh, PA  15213-3891(412) 268-5150bam+@cs.cmu.edu http://www.cs.cmu.edu/~bam

  2. Contents: • Abstract & Bio • Slides from Talk • Sources for Further Information • Articles • Early VP & PBE Taxonomy • Demonstrational Interfaces paper • Gamut Paper • Intelligence in Demonstrational Interfaces

  3. Abstract This tutorial covers the broad range of techniques used in programming systems that use "Programming By Example (PBE)." Sometimes called "Programming By Demonstration" or "Demonstrational Interfaces", this refers to systems that allow the user to give examples of the desired behaviors and contents, and the system generalizes from those examples so the result will work in different contexts. Some PBE systems use sophisticated artificial intelligence techniques to generalize, others use simple rule-based heuristics, and still others require the user to perform the generalization. Applications of PBE techniques have been to general-purpose programming, programming of user interfaces, defining animations, creating macros or scripts for text editing, designing business charts, desktop file manipulation, hypertext browsing, programming for kids, etc. The tutorial will include many demonstrations and videos of research and commercial uses of programming-by-example techniques.

  4. About the Presenter Brad Myers is a Senior Research Scientist in the Human-Computer Interaction Institute in the School of Computer Science at Carnegie Mellon University, where he is the principal investigator for various projects, including Demonstrational Interfaces, User Interface Software, Natural Programming, and the Pebbles Palm Pilot Project. He and his students have created about a dozen systems that use programming-by-example techniques. He is the author or editor of over 190 publications, including "Creating User Interfaces by Demonstration" and "Languages for Developing User Interfaces," and he is on the editorial board of five journals, including the Journal of Visual Languages and Computing. His research interests include User Interface Development Systems, user interfaces, Programming by Example, programming languages for kids, Visual Programming, interaction techniques, window management, and programming environments. He belongs to SIGCHI, ACM, IEEE Computer Society, IEEE, and Computer Professionals for Social Responsibility. He will also be a keynote speaker for VL'2000.

  5. Topics • What is Programming by Example? • Applications of Programming by Example • Overview of PBE Systems • Issues in PBE Systems • User Interfaces for giving examples • User Interface for feedback to show the program • Implementation architectures

  6. What is Programming by Example? • Direct Manipulation allows properties to be set by directly moving objects with the mouse and setting properties • Example: interface builders, Visual Basic • Limited to static parts of the interface • No way to point at objects that will be drawn by the user • How set the color of an object in Visual Basic at run time?

  7. Demonstrational Tools • Demonstrational Tools allow the user to operate on example objects which represent objects that are created at run-time. • Size of the boxes depends on labels • Demonstrational systems allow the user to operate on examples, and then generalize to produce a general-purpose procedure or prototype. 2000 Seattle VL’2000 IEEE

  8. "Examples" • Draw an example of the objects that will be created at run time • Draw objects in approximately the right places, and systems creates general constraints • Give examples of the steps of the algorithm 2000 Prototype Seattle VL’2000 IEEE

  9. Generalization • Is a hard problem, which is why you don't see many commercial products that do this. • Research on: • Algorithms for generalization • “Artificial Intelligence” • User Interfaces to help generalization

  10. Motivations • Bring programming to more people • Automate repetitive tasks • Expand how much of the interface can be specified interactively. • Interactive editors are much faster to use than programming with toolkits • Frameworks improve productivity by factors of 3 to 5, interactive tools by factors of 10 to 50! • It might take an hour to draw an interface interactively, compared to days to program it. • It is much more natural to specify the graphical parts of applications using a graphical editor.

  11. Example in Emacs • Useful to create scripts for repetitive actions: • Rename all gif files to have OLD at the front of their names archesonly.gif circuitafter.gif circuitbefore.gif emacsrecord.gif generalizeloc.gif generalizeobj.gif heightgen.gif invocationdb.gif leftgen.gif moveselection.gif mv archesonly.gif OLDarchesonly.gif mv circuitafter.gif OLDcircuitafter.gif mv circuitbefore.gif OLDcircuitbefore.gif mv emacsrecord.gif OLDemacsrecord.gif mv generalizeloc.gif OLDgeneralizeloc.gif mv generalizeobj.gif OLDgeneralizeobj.gif mv heightgen.gif OLDheightgen.gif mv invocationdb.gif OLDinvocationdb.gif mv leftgen.gif OLDleftgen.gifmv moveselection.gif OLDmoveselection.gif

  12. Example in Microsoft Word • Thefillparagraphs macro changes RETURNs into spaces, unless there are 2 in a row. Lots of short ¶lines¶ can be filled and ¶ combined. ¶ ¶ But two RETURNS¶signal a new ¶ paragraph. ¶ Lots of short lines can be filled and combined.¶ But two RETURNS signal a new para-graph.

  13. Applications of Programming by Example • General-purpose programming • Programming for kids • Programming of user interfaces • Creating macros or scripts for text editing • Designing business charts • Desktop file manipulation • Hypertext browsing

  14. General Issues for All PBE Systems • Amount of Inferencing • None -> Lots • Predictability vs. amount of help • If Inferencing: • Number of examples • Negative Examples • Hints or other help from user • Feedback on the Inferencing • Control structures: loops, conditionals • Variables • How show user the resulting program?

  15. Overview of Some PBE Systems • Chronological • Bias towards my systems • Easier to get pictures / video! • There are many other systems not covered here • See references

  16. Pygmalion • David C. Smith’s PhD thesis • Watch What I Do, chapter 1. • 1975 in Smalltalk • No inferencing • Work out program using icons and example values • Primary contributions: • Programming with icons • Working out program using examples • General motivation

  17. Pygmalion Picture

  18. Emacs • R. Stallman, 1987 in Teco, Lisp, C, etc. • Easily record a sequence of keystrokes • No inferencing • Very light-weight and extremely useful

  19. Emacs Example archesonly.gif circuitafter.gif circuitbefore.gif … mv archesonly.gif OLDarchesonly.gif mv circuitafter.gif OLDcircuitafter.gif mv circuitbefore.gif OLDcircuitbefore.gif … ^x( ^a ^k "mv " ^y " OLD" ^y ^n ^x) ^u1000 ^xe

  20. Emacs Contributions • Primary contributions: • Clearly demonstrate benefits of macros by example • Quick and easy macro creation for “keyboard macros” • Fully programmable using Lisp for further End-User-Programming (EUP)

  21. Rehearsal World • William Finzer and Laura Gould, 1984 in Smalltalk • Watch What I Do, chapter 4. • No inferencing • Metaphor of “actors” on a “stage” • Wings for off-screen computation • PBE by opening an “eye” icon • Primary Contributions: • Stage metaphor: on-stage, wings, etc. • Eye metaphor: for easy controlling of when recording

  22. Rehearsal World Pictures

  23. SmallStar • Daniel Halbert, 1980-1984, in Smalltalk • Watch What I Do, chapter 5. • Create programs in the Star office system • Desktop (file) manipulations • No inferencing • Introduced notion of “data descriptions” • Specify what data using dialog boxes • Edit straight-line program to add conditionals and loops • video (6 min)

  24. SmallStar Contributions • Primary contributions • Data descriptions concept and UI • UI Techniques for control structures • Demonstration in a complex environment

  25. Peridot • Brad Myers, 1987, in Lisp • Watch What I Do, chapter 6. • Create user interface widgets (buttons, scroll bars) • Inferred: • Graphical constraints among the objects. • Control structures such as iteration over all the items in a menu • How the mouse affects the graphics, such as that the check mark should follow the mouse. • Feedback using question and answer • No representation of the program • video (8 min)

  26. Peridot Contributions • Primary Contributions • Inferencing using condition-action rules • Question-and-answer feedback technique shown not successful • Demonstrational technique for graphical positions • Coverage of UI behaviors

  27. Eager • Allen Cypher, 1990 in HyperCard • Watch What I Do, chapter 9. • Inferred loops from repeated sequence of actions • Had to match actions and look for generalizations of values • “Anticipatory Feedback” of what it thought the user would do next • User can say “do the rest” • No way to know what the step after would be • Users weren’t comfortable without knowing the stopping condition • video (4 min)

  28. Eager Contributions • Primary Contributions • Anticipatory Feedback • Algorithms for matching actions where parameters differ • Showed that people were not comfortable telling it to go without knowing what it would do and when it would stop.

  29. Garnet • Brad Myers, 1988 - 1992 in Lisp • Watch What I Do, chapter 10. • User interface development environment • Lapidary: Create UI elements, less inferencing • video (2.5 min) • Contributions: • UI for defining graphical constraints • More sophisticated behaviors by example

  30. Garnet, cont. • C32 • video (2 min) • Contributions: • Simple generalizations on copy-paste • Gilt: Graphical styles for widgets • video (1.5 min) • Contributions: • Extend text metaphor for tabs and • Gilt: transformations of widget values • video (5:15 min) • Contributions: • Inferring parameters of procedures from examples

  31. Chimera • David Kurlander, 1990, Lisp+C+Postscript • Watch What I Do, chapter 12. • Graphical search and replace • Graphical “comic strip” representation for all actions • Generalization of macros selected from the transcript • Explicit specification of control structures • video (15 min)

  32. Chimera Contributions • Primary Contributions: • Graphical Search and Replace • Inferring more sophisticated constraints from examples • Comic-book metaphor for command history & programs • Editing command history as undo/redo

  33. DEMO • David A. Wolber and Gene L. Fisher, 1991, 1992 • David A. Wolber and Gene L. Fisher, "Demonstrational Technique for Developing Interfaces with Dynamically Created Objects," Proceedings UIST'91, Nov, 1992, Monterey, CA, pp. 89-97. • Gene L. Fisher, Dale E. Busse, and David A. Wolber, "Adding Rule-Based Reasoning to a Demonstrational Interface Builder," Proceedings UIST'92, Nov, 1991, pp. 221-230. • Create (pieces of) user interfaces • First system to support dynamic creation of objects. • Infers graphical relationships by examples of edits • Only simple relationships provided • No feedback of code

  34. DEMO Contributions • Primary Contributions • Concept of “Stimulus-Response” programming • Dynamic creation of objects

  35. Tourmaline • Brad Myers and Andrew Werth, 1990-1992. Lisp, Ness, WordBasic • Watch What I Do, chapter 14. • Infer composite styles for text formatting • E.g., more than one format in the header • Infer roles for each piece of text, and formatting for that role • Infer tables from a drawing • video (7:22 min) • Contributions • Roles

  36. Marquise • Brad Myers, Rich McDaniel, Dave Kosbie, 1993 in Lisp (part of Garnet) • Myers B., McDaniel, R. and Kosbie, D.. "Marquise: Creating Complete User Interfaces by Demonstration," CHI'94. Amsterdam, Netherlands, April 24-29, 1993. pp. 293-300. • Do more by demonstration • just show the way that the interface should operate. • Demonstrate when the behaviors start & feedback • Mouse button does one of 10 things • Demonstrate both behavior and conditions • Built-in support for palettes and modes. • Feedback using sentences with clickable words • video (3 min)

  37. Marquise Contributions • Primary Contributions • Four modes • Ability to demonstrate mouse actions by changing modes in the middle using the keyboard • Introduction of icons for “mouse droppings”  • Demonstration that text with pop-up embedded menus was not a great idea

  38. Pursuit • Francemary Modugno, 1993-1995 in Lisp • Watch What I Do, chapter 20. • Generalize desktop programs • Inferred generalizations of data descriptions • Comic book representation for programs • video (6:43 min)

  39. Pursuit Contributions • Primary Contributions • Generalizing the comic-strip metaphor • Handling of errors, pop-ups in the macro • Meta-dialog boxes • Showing the importance of the representation for the program

  40. Gold • Brad Myers, 1994 in Lisp • CHI’94, pp. 106-111 • Generalize business charts • Give one or two examples of desired elements • video (7:00 min) • Primary Contributions: • Generalizing graphics with no “program” • Feedback and editing by labeling • Handling of special items

  41. InferenceBear & Grizzly Bear • Martin Frank, 1994-1996 • Martin Frank and Jim Foley, "Model-Based User Interface Design by Example and By Interview,” UIST'93. Atlanta, GA, Nov. 3-5, 1993, pp. 129-137. • Create UIs by demonstration and direct edit • User control through dialog boxes, edit using textual language: EET • First UI builder with multiple examples • Snapshots of before and after • Multiple examples • Add more positive examples to cause generalization • Negative examples to specify exceptions

  42. InferenceBear & Grizzly Bear

  43. InferenceBear Pictures

  44. InferenceBear Contributions • Primary Contributions • Inferencing algorithm which can handle linear equations • Use of negative examples • Studies showing negative examples are difficult

  45. Pavlov • David Wolber, 1995-present • David Wolber, "Pavlov: Programming by Stimulus-Response Demonstration," CHI'96. April 1996. pp. 252-259 • Single positive example • Little generalization • Score editor for feedback and editing • Stimulus from mouse or time-based • video

  46. Pavlov Contributions • Primary Contributions • Score editor as representation • Mouse-based stimulus

  47. StageCast Creator • (formerly called KidSim, Cocoa) • Allen Cypher & David C. Smith;1994-present • http://www.stagecast.com • Formerly research at Apple, now commercial company: • Create animations and rough simulations • Program using before and after pictures • Can add additional control to rules • No automatic generalization • Video – on computer (3 min)

  48. StageCast Contributions • Primary Contributions • Before-After rules by example • Grouping of rules into sets: • Random ordered, first to succeed • Bringing PBD to wider audience

  49. Gamut • Rich McDaniel, 1996 - 1999 • R.G. McDaniel and B.A. Myers, "Getting More Out Of Programming-By-Demonstration.” CHI'99. Pittsburgh, PA, May 15-20, 1999. pp. 442-449. • Domain: "board games" and educational software • Goal: new interaction techniques so can infer more complex behaviors • E.g., how a piece can move in Monopoly / Chess • Reduce number of modes in UI • “Do Something”, “Stop That” • New interaction techniques to provide hints • Better inferencing algorithms • video (4:19 min)

  50. Gamut Contributions • Primary Contributions • UI techniques for positive and negative examples that reduce modes • Hints by asking to highlight • Most powerful inferencing algorithm • Most ambitious coverage by inference

More Related