Interactive Cognitive Intelligence

1. Interactive Cognitive Intelligence • Problem: Easy Customization & Extension of Software • Sometimes irritating bugs persist with no recourse • Simple extensions can take a long time to officially code • Solution: Have a conversation about the problem to learn a new procedure • Interaction with feedback for apprenticeship (vs. book) learning • Demonstration of some behaviors multi-modally (vs. only words) • Sharing of models learned at each site for overall system scaling • Challenges: • Natural language platform for reliable interaction • Representing the state of the “world” for referencing • Composing and refining objects and procedures in the world robot learning video

2. Building applications • Frontware • The current system is implemented on a mobile robotic platform for performing fetch-and-carry tasks in an eldercare scenario. This is incidental to main objective of the project where our focus is primarily a software layer which can be used with any application. • Core system could be adapted to work with generic software interfaces by implementing a mouse driver and click generator and a screen reader that can find and read menu options. • Special recognizers might have to be built for domain-specific display entities (e.g. circles representing sets of people). • Linguistic interaction • The central Natural Language and dialog components would remain largely unchanged – there is still a need to specify objects of interest verbally, and a need for inducing procedural control structures from language. • To speed up the learning process it is important that the computer be allowed to ask the user questions in order to direct attention to whatever parts it finds ambiguous. Some of this might be via mouse gesture rather than purely through text or speech. • Knowledge sharing • The knowledge fragments need to be indexed for retrieval in relevant situations. • There also should be a mechanism for resolving conflicts when more than one downloaded piece of remotely acquired expertise might apply.

3. Functional Architecture Current Robot-specific Demo Setup Eli Robot at Watson Vision Objects Visual models ASR Parser Vocabulary Supervisor Reasoning Semantic memory Action models Talk Kinematics Sequencer

4. Some representative scenarios • Automating and removing rough edges from software • Siri: “Find pizza near Croton”  Stony Point • Restaurant is only 1.5 miles away, but it is all water (Hudson River) • Say: “Stony point is not really near Croton so omit these answers” • Surveillance robot commonly tasked to guard the rear exit of a building • Manually drive to other side, aim camera at door, enable person detection • Say: “Cover the back means doing this sequence <demo>” • Adaptive washing machine • Special instructions: Observe mix of colors  normal perma-press cycle • However husband likely to get itchy when he sweats in polyester • Say: “This is my husband’s bowling shirt. Make sure it gets an extra rinse” • Alter default settings: Observe T-Shirt inserted  set water to hot for cotton • But hot water can cause lettering to peel off • Say: “Always ask if the item is silk-screened before using hot water.”

5. mouse File Edit View Data Help Data Scatter plot Time series Bar chart Pie chart Bar chart mouse Data analysis scenario • Learning group names • “These columns <circles with mouse> are called the standard dump” • Learning procedures • “I am going to show you generate the divisional scorecard”. • Select from menu: multi-source bar chart • Select from y axis options: major ticks at $10M • Select from x axis options: bars labeled by quarter • “That’s how you do it” • Emergence of a “higher level” language • “Get me the divisional scorecard of the standard dump.” • Can share “macro” with colleagues via bulletin board

6. mouse mouse Trainable wingman • Learning formation names • “These units near the base <circles with mouse> are called the reserve forces” • Learning procedures • “Let me show you how to carpet bomb something” • “First, load up your Overlords with Banelings” • “Send them over the target” • “And then … <click> <click> <click> <click>” • “See?” • Emergence of a “higher level” language • “Okay, now carpet bomb his reserve forces” • Can share “macro” with other players via app store