Cognitive Model Comparisons: The Road to Artificial General Intelligence?

1. Cognitive Model Comparisons:The Road to Artificial General Intelligence? Christian Lebiere (cl@cmu.edu) Cleotilde Gonzalez (coty@cmu.edu) Carnegie Mellon University Walter Warwick (wwarwick@alionscience.com) Alion Science & Technology

2. Challenges in AI & Cognitive Science • Both fields have similar history of challenge problems despite compatible ends but different means • Artificial Intelligence: maximize task performance • Started with ambitious but poorly defined test (Turing Test) • Evolved narrow, precise, overspecialized challenges (Chess) • Recently attempted broader tests (Robocup, Grand Challenge) • Cognitive Science: fit human capabilities (design guide) • Started with ambitious, ill-defined capacities list (Newell Test) • Organized a series of complex task comparisons (AMBR, HEM) • Is taking on broader but integrated challenges (DSF?) Artificial General Intelligence Conference

3. Cognitive Challenge Pitfalls • Challenge is fundamentally about the task, not cognition • Too much task analysis and KE, too little cognitive theory • Task is too narrow; too much data available • Reduces to data fitting – favors parameterization over principle • Task is too specialized (typical cognitive psychology) • Single cognitive aspect – misses generality, integration • Lack of common simulation environment • Each framework/theory only tackles what they do well • Lack of comparable human data • Emphasizes functionality – loses cognitive constraints Artificial General Intelligence Conference

4. Desirable Challenge Attributes • Lightweight • Limit integration overhead and task analysis/knowledge eng. • Fast • Rapid model development and collection of monte carlo runs • Open-ended and dynamic • Less parameterization, generalization to emergent behavior • Simple and tractable • Direct relation from cognitive mechanisms to behavioral data • Integrated • Toward integrated agent capturing architectural interactions Artificial General Intelligence Conference

5. DSF Challenge Comparison • Dynamic Stocks and Flows – Instance of Dynamic Decision Making • Control a dynamic system given unexpected environmental fluctuations • Simple version of real-world situations (financial, ecological, technical, game) • Integrated tasks • Anticipate events • Control system • Cognitive functions • Sequence learning - PC • Action selection - BG • Implementation • VB on Windows • Text socket protocol Artificial General Intelligence Conference

6. Generalization Scenarios • Humans learn to control system over time for simple functions • Highly variable but quantifiable performance over learning process • Complexity of task scalable along a number of cognitive dimensions • Environmental i/o • Complex sequences • Stochastic noise • Multiple variables • System dynamics • Feedback delay • Non-linear effects • Real-time control • Multi-agent system • Other controllers • Payoff manipulations Artificial General Intelligence Conference

7. DSF Comparison Schedule • Official announcement expected March 15 • Task environment with socket connection for model, data and documentation available on web site • Symposium April 1st at BRIMS conference (Sundance) • Model submission by May 15 • Best entries invited to symposium at European cognitive modeling conference (travel supported) • Email DSFChallenge@gmail.com to be added to distribution list for official announcements/updates Artificial General Intelligence Conference