1 / 72

Programming by Example

Programming by Example. Henry Lieberman Media Laboratory Massachusetts Institute of Technology Cambridge, MA, USA lieber@media.mit.edu http://www.media.mit.edu/~lieber PBE Web site: http://www.media.mit.edu/~lieber /PBE/. Your Wish is My Command: Programming by Example.

annot
Download Presentation

Programming by Example

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Programming by Example • Henry Lieberman • Media Laboratory • Massachusetts Institute of Technology • Cambridge, MA, USA • lieber@media.mit.edu • http://www.media.mit.edu/~lieber • PBE Web site: http://www.media.mit.edu/~lieber /PBE/

  2. Your Wish is My Command:Programming by Example • Morgan Kaufmann, 2001

  3. Programming should be like teaching • User plays the role of a teacher • Computer plays the role of a student

  4. Programming by Example • (Also called "Programming by Demonstration") • Agent "watches what you do" in an interactive interface • Records sequence of operations, data • Generalizes program so that you can use an analogous procedure in new examples • "Macros on Steroids"

  5. Programming in the User Interface • The same interface used by end-users can be used for programming • Empowers end-users to do programming • No switching between user and programmer interfaces • Empowers "visual thinkers"

  6. PBE and the "Intelligent Agent" movement • There's been a recent movement toward "intelligent agents" -- proactive, assistive, (possibly anthropomorphic) interfaces • PBE can be used to make agents instructible • Agent personalities: • Butler/Servant/Secretary • Teacher/Student, Master/Apprentice

  7. Issues in Programming by Example • The Generalization Problem (= Data Description) • How do I describe actions and data? • How similar do the actions/data have to be next time? • Feedback • How does the system show the user what it's learned? • Advice • How can the user influence or modify what the system learns?

  8. Issues in Programming by Example • How should we record the user's actions? • At what granularity should things be recorded? • Can the user interact with the agent during recording? If so, how? • Do we represent the agent directly in the interface? • Do we anthropomorphize the agent? • Can users give advice during recording? • Turning recording on and off

  9. Issues in Programming by Example • How should the recorded program be represented? • Should the user see the recorded program? • What should it look like? • Should the user interact directly with the recorded program? If so, how? • How to represent examples? • How to represent generalizations?

  10. Issues in Programming by Example • How should the user edit an already-recorded program? • How do we keep the representations consistent during editing? • Can the user edit previous generalizations? • How can users debug programs? • Can users step/trace/break programs?

  11. Machine learning in Programming by Example • Lots of work in machine learning • Careful: Watch for hidden assumptions • Few examples, learns fast, explainable • Explanation-based Generalization • Case Based reasoning • Lots of examples, slow but more robust • Neural networks, PAC learning, data-mining

  12. Users of the world, unite! • You have nothing to lose but your error messages! • Almost "political" message of empowerment of users • Users should not be at the mercy of software providers • User procedures and data should not be held hostage by software companies

  13. "Poor man's"Programming by Example • Macro recorders in applications, AppleScript • Emacs keyboard macros • Interface builders / Scripting languages • Quickeys / OneClick …

  14. Hooking PBE systems to conventional applications • PBE systems would like to use conventional interfaces as tools rather than re-implement apps • Conventional interfaces aren't set up to be operated by an external program • Issues: Access to application data, sharing the interface, parallelism • Article: Lieberman, IUI 98 "Attaching Interface Agents to Conventional Applications"

  15. Watch What I Do • Allen Cypher, ed. MIT Press 1993

  16. A Small Matter of Programming • Bonnie Nardi, MIT Press, 1994. • End-User Programming • Scripting languages • Application-specific languages • "Gardeners"

  17. Eager - Allen Cypher • Programming by example for Hypercard • Notices patterns in user actions • Anticipation highlighting of predicted actions • Anthropomorphization of learning agent • Example: Collect subject lines from messages

  18. Eager

  19. Mondrian: An Instructible Graphical Editor • Mondrian = • An object-oriented graphical editor + • An agent that records user interface actions + • Learning and generalization heuristics

  20. Mondrian: An Instructible Graphical Editor

  21. Wizard of Oz Experiment • Human simulates agent • What kinds of communication between human and agent are plausible? • What kind of knowledge does the agent need to know to perform the task?

  22. Wizard of Oz experiment task • Reformat a Bibliography • Lieberman, H., The One True Path to Intelligent Agents, International Conference on Buzzwords and Hype, Tahiti, 1996. • [Lieberman 96] The One True Path to Intelligent Agents, Int. Conf. on Buzzwords and Hype, Tahiti, (1996). • Oz, Wizard Of, Pay No Attention to the Man Behind the Curtain, Journal of Smoke and Mirrors, August 1942, Emerald City, Oz.

  23. Wizard of Oz results • Users can make mistakes • Information can be incomplete • Agents can ask the user if they encounter an error situation • Agent could ask the user to supply missing information • Agent could ask the user to edit the example • Skip steps that don't work

  24. Wizard of Oz experiment method • User performs task once • “Thinking aloud”: goals and methods • Agent (secretly) constructs description of actions • In terms of simple “implementable” operations • Agent offers help • Agent performs help according to action description • Agent gives user feedback • Agent accepts advice from user

  25. Survey of "Your Wish" systems • Astonishingly wide variety of application domains: • Kids, Games, CAD, GIS, Text editing, Graphical editing, Web browsing, Animation, Databases, etc. • Illustrates variety of design decisions in PBE systems • Different reps for programs, agents, etc. • How much inference, generalization, initiative?

  26. Stagecast Creator - Smith, Cypher • Programming for kids, grid game world • If you see this pattern of squares, replace it with that pattern of squares • Viz rep of rules, editor/stepper • Generalization on object types, introduce variables • Commercially available, success with thousands of users

  27. A Stagecast Creator Rule

  28. A Stagecast Creator Game

  29. Toontalk - Ken Kahn • Programming language is a video game • Most radical programming model • Concurrent logic programming • Abstraction by removing detail • Commercially available, success with kids in schools

  30. Toontalk's Visual Language

  31. Generalizing by Removing Detail

  32. Toontalk Sort program

  33. How much intelligence?- Brad Myers, Rich McDaniel • CMU group one of most active and consistent in PBE • Multitude of systems, "gems" of PBE • Peridot, Lapidary, Jade, Gilt, Tourmaline, C32, Pursuit, Gold, Marquise, Katie, Turquoise, Topaz, Gamut • Experience with range of conservative to radical PBE systems • "Spectrum of Instructibility" • Tradeoff between conveience and control

  34. Pursuit's Visual Language

  35. Selecting Generalizations in Marquise

  36. Internet Scrapbook - Sugiura • Everybody's favorite application area - the Web! • Automates Browse & Cut & Paste applications • How do you describe a Web page selection? • Partial matching, Web design heuristics • Commercially available - preloaded on NEC PCs

  37. Internet Scrapbook

  38. TriAS - Bauer, Meyer, Dengler, Paul • Automating Browse & Copy & Paste, like Scrapbook • Collaborative, mixed-initiative dialog • At any time, agent or user can propose selections, generalizations • What Next? Can I Stop? • Integrates wrapper description language HyQL, InfoBeans dynamic Web sources

  39. Collaborative Wrapper Definition

  40. PBE for GIS: CSprl - Traynor, Williams • Study of how end users use Geographical Information Systems • "Gardeners", "Surrogate Users" • Comic-strip PBE language

  41. A Geographical Info Systems PBE Program

  42. Visual Language for Geographic Information Systems

  43. PBE for Computer Aided Design -Girard, Pierra • True "Industrial Strength" PBE, probably the champ • CAD users are sophisticated in application domain, willing to specify a lot • Provides complete editing and debugging • Key issues: Naming, Ambiguity in spec/solutions, general control structures: conditionals, loops

  44. Example-Based Programming in Computer-Aided Design

  45. Demonstrating "Hidden Features" - McDaniel, Myers • PBE for animated computer games • WYSIWYG - "What you get is all you have" • Demonstrate "hidden features" by reifying them • Guides, e.g. Arrows for directions • Sophisticated heuristics, lots of inference • Stop This! • Do Something!

  46. Demonstrating Hidden Features

  47. Report Generation - Masuishi • Mundane, but hey, could save a lot of time • Format generalization • Application-specific iteration • Distributed to wide user community

  48. Report Generation by Example

  49. Dynamic Macro - Masui • Notices loops in text editing commands • Simple, automatic: No start/stop recording • Wide community of users • Also: POBox predictive input for PDAs, phones, etc.

  50. Dynamic Macro

More Related