Center for Integrated Cognitive Systems

1. Center for Integrated Cognitive Systems A Vision and Collaborative Proposal

2. Engineering Method for RAIR Lab’sNext-Generation Logic-based Artificial Intelligence • Isolate and dissect human ingenuity. • Mathematize a weak correlate to this ingenuity courtesy of advanced logical systems. • Implement this correlate in working programs. • Augment the correlate with machine-specific power.

3. Selected Research Projects in RAIR Lab • Integrated Cognition via PERI and “Psychometric AI” • PERI is a robot with human-level intelligence. • http://www.cogsci.rpi.edu/peri • The Slate System: Intelligent Assistant in support of Intelligence Analysis and Beyond • Slate is developed to help analysts @ NSA, CIA, etc. • Formalizing and Modeling Adversarial Beliefs and Ethical Codes in Support of Wargaming • predict what the enemy is going to do in modern-day asymetrical conflict • New World Records in the -Cracking Project: A New-Millennium Attack • Learning by Reading! -- and the MARMML Machine Reasoning System • This project is deveoted to building a machine system capable of learning by reading.

4. Mind is What Brain Does • How do we think? Solve problems? Learn new things? Remember old things? • What cognitive processes are involved in reading a book? Recognizing your Grandmother? Writing a letter? Designing a scientific experiment? • How many things can we pay attention to at once? How does the design of computer software affect the way we think? • What is cognition in the world? How does it differ from cognition in the head? • Can we design our environment to facilitate thought? So the tools we use become “invisible” and we can get on with the task at hand?

5. What is Cognitive Science? (A 2-slide introduction) • Cognitive science is the scientific study of mind; more specifically, the systematic attempt to investigate, understand, simulate, and replicate cognitive systems in computational, representational, and dynamical frameworks • It is a rapidly growing interdisciplinary field that brings together researchers from parts of six fields: artificial intelligence (AI), psychology, philosophy, education, linguistics, and neuroscience

6. What is Cognitive Science? (A 2-slide introduction) • Cognitive scientists use empirical, formal, and computational methods • Cognitive Science views the mind as a cognitive system that receives, stores, retrieves, transforms, and transmits symbols or representations • The operations performed on these symbols and representations are termed information processes. Hence, most of cognitive science can be understood as the application of information technology (and information science) to problems in perception, attention, motor control, memory, language, problem solving, and reasoning

7. What is Computational Cognitive Science? • Computational Cognitive Science is what Rensselaer Polytechnic Institute excels at! • CCS • is the application of cognitive science theory to human factors practice • seeks to build systems that monitor, support, aid, and extend human cognitive processes • includes the engineering of artificially intelligent systems that capture, in computation, human-level “smarts”

8. What is Cognitive Engineering? • Cognitive Engineering is what Rensselaer Polytechnic Institute excels at! • Cognitive engineering is the application of cognitive science theory to human factors practice • Cognitive engineering seeks to build systems that monitor, support, aid, and extend human cognitive processes

9. Who Supports Cognitive Engineering? • Academic work in cognitive engineering is primarily supported by: • Air Force Office of Scientific Research (AFOSR) • Advanced Research & Development Agency (ARDA -- consortium of Intelligence Agencies) • Army Research Institute • Defense Advanced Research Projects Agency (DARPA) • Office of Naval Research (ONR)

10. Who Supports Cognitive Engineering? • Government work in cognitive engineering is primarily supported by: • Air Force Research Laboratories • Human Effectiveness Directorate (Mesa, AZ & Wright-Patterson, AFB, Ohio) & • Information Directorate (Rome, NY) • Advanced Research & Development Agency (ARDA -- consortium of Intelligence Agencies) • Army Research Laboratory (Aberdeen, MD) • Naval Research Laboratories (DC) • Naval Underwater Weapons Center (Newport, RI)

11. Who Supports Cognitive Engineering? • Leading Industrial Labs in cognitive engineering include: • PARC (formerly XEROX PARC) • Microsoft Research Laboratories • Sandia Laboratories • BBN • IBM Watson • MITRE

12. Center for Integrated Cognitive Systems & the Air Force Research Laboratories • Building on the foundation of excellence at RPI Air Force funds will enable CICS to become internationally renown • This will be achieved by • Support for leading edge basic research • Hiring additional post-doctoral researchers and faculty • Attracting and supporting a cohort of next-generation researchers (i.e., grad students) • Enmeshing CICS researchers in the challenges and opportunities provided by AFRL projects

13. Center for Integrated Cognitive Systems & the Air Force Research Laboratories • CICS can provide AFRL with • Access to leading-edge research and leading-edge researchers • Increase the awareness within the cognitive community of the types of problems encountered by Air Force researchers • Annual Spring symposium that brings the best and brightest of the cognitive community together to contemplate Air Force problems • Fall colloquium alternating between Troy and Syracuse each year • Advanced degree program • MS in Cognitive Science tailored to AFRL schedules and needs • PhD program that takes into account AFRL constraints • Student intern program -- Rensselaer students who are supported by Air Force funds will be required to spend a summer or semester working at the Rome lab

14. Center for Integrated Cognitive Systems & the Air Force Research Laboratories • Working together to • Advance the foundations of Cognitive Science • Bridge the gap between theory and application • Target Air Force needs by incorporating leading edge cognitive theory into AF projects

15. Cognitive Science applied to real-world problems Project Ernestine (how cognitive modeling saved the phone company big $$$) Project Nemo: Cognitive Models of Situation Assessment (currently funded by Office of Naval Research) Nature, detection, and correction of errors in rule-based programming tasks (currently funded by National Science Foundation) Interactive routines and cognitive workload (currently funded by Air Force Office of Scientific Research)

16. The Précis of Project Ernestine An Overview of a Validation of Cognitive Modeling • Interest • Modeling: parallel performance; event-driven; complexity in the world; transfer • Application: can modeling be used to evaluate design; operating expenses = $3 million/sec • Theory: can cognitive theory be applied via cognitive modeling to predict time-critical human performance in the real-world • Validation of both modeling & the cognitive approach • See Gray, John, & Atwood (1993) for more details

17. Nature, detection, & correction of errors Errors and other situated pitfalls in a “walk-up and use” device • Topic: Can we predict “where” and “when” errors are made based upon a cognitive analysis of the procedural knowledge? • Interest • Theory: display-based problem-solving; situated cognition; errors; role of “verifies” in procedural skill • Application: development of guidelines for providing feedback in “walk-up and use” devices • Modeling: data-driven vs in-the-head cognition • Technology: models written in ACT-R “read” from and “press” buttons on a VCR device simulation written in HyperCard

18. Cognitive Models of Situation Assessment • Subjects: Submarine Action Officers • Question: Before deciding what to do, they must first decide what is going on. How do they do this? • (Can’t open the window and look out.) • All data is passive (noises in the ocean) • Low signal-to-noise ratios are the norm • Interest • Modeling: situation assessment aspect of decision making; display-based problem-solving • Application: models will provide input into to the design of displays and work groups for the New Attack Submarine • Theory: schema-theory; display-based cognition; situated action • ONR funded.

19. Interactive Routines and Cognitive Workload • Mental workload is a colloquial term that various researchers have attempted to translate into information processing terms • Interactive routines • Elements of interactive routines are individual productions that take from 30 to 300 msec to occur • The interactive routines themselves require 300 msec to 3 sec to execute (ballpark estimate of time required for system to do about 10 productions) • Requirements of one interactive routine may influence the execution of another interactive routine (that is, individual interactive routines may be considered as elements in larger strategy)

20. Cognitive Science applied to more real-world problems Resolving the Paradox of the Active User Profile before Optimizing: A Cognitive Metrics Approach to Workload Analysis Soft constraints in Judgment & Decision-Making Recognizing User Affective States for Active Assistance

21. Paradox of the Active User • Inefficiencies in the form of stable suboptimal performance may persist even after years of specialized experience and daily use • These inefficiencies result in: • Increased cognitive workload • Subtle effects on performance and errors • Shifts between interaction-intensive strategies that rely on knowledge in-the-world versus memory-intensive strategies that rely on knowledge in-the-head • Can we predict it? • Can we eliminate it?

22. Profile before Optimizing: A Cognitive Metrics Approach to Workload Analysis • Intelligence Analysts are the designated user of a score or more projects that will put new and innovative information technologies on their desktops • Our role is to assist the Intelligence Analysts as user of these technologies • Vital that a new type of usability assessment be made; one that provides metrics for cognitive workload that address various cognitive factors • Our proof-of-concept is a model that performs a task that the IAs find challenging • From the trace of the model, we derive a cognitive metrics profile that pinpoints dynamic changes in workload demands on human cognitive, perceptual, or action systems

23. Soft constraints in Judgment & Decision-Making • Is decision-making like buying a train ticket or like sailing a boat? • Is a decision about how to solve a problem made up front and never altered until the solution is reached? • Or does each step bring new problems and each solution new perspectives? • How does the cost-benefit structure of the task environment influence the strategies that the decision-maker adopts?

24. Recognizing User Affective States for Active Assistance • Innovative research between RPI’s Engineering Department and Cognitive Science Department • What can happen when you combine a system that can parse emotions in real-time with . . . • A computationally based, architecture of cognition? • And put both in service of determining when and how to intervene to assist the human operator in a real-time, safety critical task?