Symmetric Multi-modal Intelligent Interactive Development Environment (SMIIDE) Seedling

1. Symmetric Multi-modal Intelligent Interactive Development Environment (SMIIDE) Seedling Update as of September 21, 2009 BAE AIT: Greg Sullivan, Basil Krikeles, Mike Cook MIT CSAIL: Randy Davis, Howie Shrobe, James Oleinik, Kyle Mill

2. Symmetric Multimodal Interactive Intelligent Development Environments • Dramatic reduction in time and cost of constructing software designs and related engineering artifacts • Production of higher quality artifacts through interactive design. • Metrics to be refined during seedling. Let’s create a data interface component Does this process need access to the database? STATUS QUO QUANTITATIVE IMPACT • Current Integrated Development Environments (IDEs) are developer-centric, detail-intensive, and error-prone • Modality is essentially paper-based and human developer-centric • IDEs support limited consistency checking and debugging • Multi-modal Mixed-initiative User Interface • Supports sketching, gestures, natural language. • System can query human for clarification, suggestion, etc. • System can take initiative in design and implementation, freeing the human from detailed specification of lower-level details • Adaptive Domain Specific Model-Based Software Engineering tools • Support evolution of domain “vocabulary” • Support diagramming using symbology specific to a domain. • ASSUMPTIONS AND LIMITATIONS: • Anticipated productivity gains can be demonstrated on well-defined domains. • Current natural language technology is sufficient. • • Demonstrate program feasibility and develop program structure • Prototype combination of symmetric multimodal HCI with model-based code generation technology • Determine feasibility of program leading to mixed-initiative high-productivity environments • Identify technology gaps and refine DARPA-hard program challenges. • Demonstrate on a DoD-relevant domain • Survey and position with respect to previous research Multi-modal User Interfaces can now integrate sketch, gesture,& natural language to capture design, rationale more naturally. Mixed-Initiative User Interfaces enable symmetric collaboration between human and computer-based designers. Adaptive, Domain-Specific Model-Based Software Engineering tools enable flexible collaboration at a human level of abstraction. END-OF-SEEDLING GOAL NEW INSIGHTS Symmetric multimodal interaction brings intelligence to model-based development SMIIDEling Overview 2 Distribution authorized to U.S. Government Agencies only

3. Some seedling goals(from Jim’s slides at August meeting) • Arrive at a focused set of technical challenges that form the basis of a new program • The purpose of the seedling is to create a new program initiative brief • A “Scientific American” statement of the technical goals of the program • A clear connection to compelling operational need • A convincing case for how this can be achieved • The nature of the technical challenges, and which focused challenges will form the basis for a program? • How far can we reach in terms of operational capability in the scope of a program? • What is the potential operational payoff? • What options for demonstrating the concept and measuring success? SMIIDEling Overview

4. Outline • The Heilmeyer Questions • A “Scientific American” statement • Compelling operational need • Research Topics for proposed program • Quick review of topics from August meeting. • “Lots of Little Languages”. • Rationale Capture via Natural Interaction • Related research (“Why now?”) • Sample Scenarios • Geospatial reasoning in Deep Green • Building a wiki for after-action reports SMIIDEling Overview

5. 1. The Heilmeyers… • What is the problem, why is it hard? • Software design, development, and maintenance. • Low level, error-prone, complex, hard to verify. • Slow to adapt to mission needs. • How is it solved today? • Code-level “forward engineering”. That is, human-intensive high-to-low level specification with computer “automaton” support for detail tracking (e.g. parsing) and high-to-low level translation (e.g. compilation). • What is the new technical idea; why can we succeed now? • Natural interaction and DSML interpretation combine to enable mixed initiative interaction, with useful initiative from computer, augmented by captured rationale. • Drill down and machine learning enable both computer and human input at all levels of abstraction. • What is the impact if successful? • Revolutionary increase in productivity for creating software-intensive systems. Enables fast development and deployment of latest technology, rather than traditional slow DoD software acquisition process. • Dramatic increase in development / adaptation allows rapid response to mission needs. • How will the program be organized? • How will intermediate results be generated? • How will you measure progress? • What will it cost? SMIIDEling Overview

6. A “Scientific American” statement of the technical goals of the program • Every day, countless conversations take place between software engineers in front of white boards. The engineers take turns grabbing markers, and talk while drawing. In response to questions from each other, the engineers elaborate or erase parts of their diagrams. The interactions are highly verbal, and the drawings include text, traditional software diagrams, and ad-hoc notations made up on the spot. • Now, what if one of those engineers was a computer? • The goal of the SMIIDE program is investigate whether the combination of several cutting edge research areas can enable computer-based agents to become active participants in the software design, development, and maintenance process. • The computer should be able to ask relevant questions, suggest useful approaches and designs, and thereby substantially contribute to the implementation of the eventual system. Research Areas: Sketch recognition, Natural Language, Rationale Capture, Domain Specific Modeling, Dialogue management. SMIIDEling Overview

7. Compelling Operational Need • The problem: in a fast-changing world, it’s easy to miss the mission window • Response: Rapidly tailor systems to mission needs • Can leverage rationale captured in natural modalities. • Why were certain decisions made? • What were assumptions? • Increased speed of development while maintaining correctness • Decreased costs What current operational needs are addressed by a dramatic increase in speed and flexibility? SMIIDEling Overview

8. Research Topics identified during August meeting • Grounding complex abstract diagrams in sufficient meaning to support the design task; • Determining how complex specifications can be segmented in ways that are functionally meaningful for reasoning and dialog; • Determining what kind of human-computer interactions are needed at any given moment and mediating that interaction effectively; • Determining the HCI principles that lead to productivity enhancements not attainable with model-based software engineering tools alone (including the advanced versions produced in the current DMT-SWP program). • Is there a body of kn about MBSE applicable across domains? • Dev of DSMLs suggest any common body of language, lexicon,… • How wide a variety of specification forms to consider? tabular, text, graphical • Graphical models vs. behavioral specifications? How do we get the benefits of graphical structural models for behavioral specs? SMIIDEling Overview

9. Core Research Topic: Related Families of Overlapping DSMLs path location unit • Benefits of DSLs: • Small language decreases errors. • More abstraction increases productivity, improves analysis • DSMLs capture expertise of domain experts • How to get coverage / generality? • Lots of DSLs • Overlaps in domains gain coverage by transitivity. • Evolution of DSLs • Generators / Interpreters / Transforms • Provide overlap by translation • Often high-to-low level (where “level” is loosely defined). • Generator logic can be hand-generated or partially learned. Geospatial Course of Action Geometry Correspondence, Transformations Tenneygrams UML classdiagrams Correspondence, Transformations Make Java Scala SMIIDEling Overview

10. Core Research Topic: Natural Rationale Capture • Can natural interaction make rationale capture (nearly) painless? • Why were certain decisions made? • What were assumptions? • Can captured rationale provide the basis for “parametric design” of software? SMIIDEling Overview

11. Rationale Capture - Rationale • Every software project involves thousands of design decisions • Most are routine (i.e., obvious to an experienced programmer); many crucial ones are not • Writing/modifying/debugging/ software requires constantly re-absorbing those decisions. • This would be orders of magnitude easier if all the program authors were always there to explain how they made those decisions and why. • Rationale capture can achieve this effect. SMIIDEling Overview

12. Rationale Capture – Technical Challenges • Why isn’t rationale captured now? • Cost is too high (writing documentation is a lot of work) • Task is aversive (writing documentation is unpleasant work) • Task is disruptive (writing documentation interrupts design) • The value equation is out of whack: • The person incurring the cost is (almost) never the one who gains the benefit. • Cost is now, benefit is in the future. • Consequences: documentation of small but crucial decisions is deferred. Can we change the equation by drastically lowering the cost and making it more like talking to another designer? SMIIDEling Overview

13. Core Research Topics: Sketch Understanding, Symmetric Multimodal Interaction • Sketch understanding • grounding complex abstract diagrams in sufficient meaning to support the design task; • structure (e.g., data structure) is easy to draw, but a lot of software is about process, behavior • can we find ways to draw behavior that feel as natural? • Symmetric multimodal interaction requires an intelligent observer, e.g., knowing what questions to ask and when • How task- and domain-specific is that knowledge? • Is there a body of knowledge about MBSE applicable across domains? Or is the problem AI-complete? SMIIDEling Overview

14. Central Hypotheses • Natural interaction + modestly intelligent observer = drastic increase in agility and lower cost • Domain specificity enables intelligence of computer-based observer. • Evolvable and overlapping languages enables generality SMIIDEling Overview

15. How far can we reach in terms of operational capability in the scope of a program? • What to rule out? • “Programmer’s common sense” • Domain expertise encoded in DSML “smarts”. • Mixed initiative: human in the loop. • System must be “smart about something” but cannot be smart about everything. SMIIDEling Overview

16. What options for demonstrating the concept and measuring success? • Prototype and storyboards for demonstrating concept. SMIIDEling Overview

17. Metrics • Ultimate metric: order of magnitude faster with system – allows missions / capabilities that simply weren’t available before. • # of lines of code automated versus hand written. • Some measure of complexity of code generated. In other words, abstraction + automation results in less quantity and lower complexity over course of task. SMIIDEling Overview

18. Program Structure • Individual pieces, with an integrator • Or entire systems from each team? SMIIDEling Overview

19. Related Research • Sketch understanding • MIT (Davis), UWash (Landay), UC Riverside (Stahovich), Illinois (Forbus) • Dialogue management • Grosz (Harvard), Sidner (BAE) • DSMLs • Karsai et al. (Vanderbilt), Metacase (Kelly, Tolvanen), Gray (Alabama), Sprinkle (Arizona). • See dsmforum.org • Natural Language • Mixed initiative SMIIDEling Overview

20. Random Thoughts • Community aspect? • related to Framework Adoption Seedling • learning over many interactions with same and different users. • Related to Bootstrapped Learning? • Teaching computer to program. • Might be relevant to helping SMIIDE system learn. SMIIDEling Overview