Brain-Related Computing beyond Moore’s Law

1. Brain-Related Computing beyond Moore’s Law Thomas Sterling Professor of Informatics and Computing Chief Scientist and Associate Director Center for Research in Extreme Scale Technologies (CREST) School of Informatics and Computing Indiana University February 24, 2014

2. The Human Brain • 100 billion neurons (Felix says “89”) • 10,000 synaptic junctions per neuron • < 1 KHz pulse repetition rate • 1400 cc volume • 2-4 Exa-Ops • 20 Watts power consumption • Hierarchical architecture • Neuro-function • Analog summation in time and space • Time varying comparison threshold • Pulsed signal propagation • Modulated synaptic connections

3. Brain-inspired • Consciousness • Identity • Self-aware • perception • Thinking • Recognition of patterns, objects, people, concepts • Planning, decision making, learning, action, inference • Intelligence • Extreme complexity • 70 neurons/cubic mm • 20 Atto Joules per operation • Localized asynchronous functions • Emergent operational behavior

4. Neuron Function

5. Neuro-Inspired • Neurons embody state and function • State both internal and at interfaces • Function both internal and at interface • Neurons perform localized operations • Ops to first order are internal and isolated • complex • independent in time • Asynchronous and slow, event driven • Enormous connectivity • On the order of 10,000 input junctions • On the order of 10,000 output contacts • Limited connectivity • Only 0.00001% of system • Nonlocal • Operates in a context for emergent behavior • Execution model, graph structured subsystems • Ultra low power

6. Analog of Neuro-Inspired Computational Element • Vannevar Bush Machine • aka Analog Computer • Solves sets of first-order differential equations • Core device satisfies Neuro-Inspired definition • Internal state and complex functionality • Emergent behavior in context of total system of like devices • Many technologies • Mechanical, vacuum tubes, transistors, op amps • One execution model • Governing abstraction

7. Artificial Neural Networks (ANN) • An interconnected network of functional elements that reflect some properties of neurons and their interconnections • Derived functions through learning from observations • Training set • Adaptive weights • Applications • Identification and pattern recognition • Image, signature, character recognition, ATR • Data mining, clustering, classification • Control of real time mechanical systems • Diagnosis • Robotics

9. Expert Systems • Also “production systems” or “rule-based systems” • Makes decisions in the presence of a set of external and internal state • Comprises a set of rules or “productions” of two parts: • Condition: “if” – determines satisfiability • Action: “then” – updates system state • Productions are executed • When conditions are satisfied • Conflict resolution and prioritization • RETE algorithm optimizes performance • Languages: OPS-5, ART, Keys, …

10. Two Brain-Inspired Projects • CCA • Continuum Computer Architecture • Low-level abstraction • Brain Inspired: • Fine grain functionality • Massive co-existing elements • Distributed control • Quasi independent operation • Emergent behavior • Localized function & state • Event-driven • Logically highly interconnected • CRIS • Cognitive Real-time Interactive System • High-level abstraction • Brain Inspired: • Intelligence • Self-aware • Emergent behavior • Concurrent agents • Objective function-driven • Engages neighborhood of interrelated objects • Responsive in real-time

11. Relationship within System Structure Users CRIS Graph API Runtime System Operating System CCA Logic/Store/Comm Devices

12. Peak Resources X100 arithmetic/logic units massive bisection and memory bandwidth Merge functionality into a single simple block (fonton) communications memory logic Global cellular structure, 2.5 or 3-D mesh Data/instruction structures distributed across fontons Local rules determine fonton operations – cellular automata Synergism among fontons yields emergent global behavior of general parallel computing model Natural graph structure store CCA Fundamental Concepts

13. Small block of fully associative tagged memory Basic logical and arithmetic unit Instruction register directs control to set data paths Nearest neighbor communications with switching Assoc. Memory Inst. Reg. ALU Control CCA Structure: Fonton

15. Emulates Neuron Structures with Hardware in Software Localized functionality achieved by fontons or groups of fontons Packet switching through fonton pathways achieves synaptic broadcast In aggregate builds up dynamic irregular time-varying graph structures Data migration objects are copied to adjacent fontons copying exploits fine grain data parallelism, even for irregular data structures objects may transit by means of wormhole routing Data objects are virtual named Address translation an intrinsic function Emulates Neuron Structures in 3-D

16. CRIS Motivation • Inspired to mechanize key properties of mental condition: thinking • Machine Intelligence • Question: How big a machine? • Derive a lower bound of required resources • OPS, • storage, • communication, • energy • Premise: “Intelligence” • Independent of the mammalian mental condition • An algorithm • Does not embody many properties of mental condition • Determine balance of resource utilization to exhibit intelligence

17. Abstract Architecture: CRIS

18. Abstract Architecture: Knowledge State

19. Abstract Architecture: Objective Function

20. Execution Model Phase Change Von Neumann Model 1949 • Guiding principles for system design and operation • Semantics, Mechanisms, Policies, Parameters, Metrics • Driven by technology opportunities and challenges • Historically, catalyzed by paradigm shift • Decision chain across system layers • For reasoning towards optimization of design and operation • Essential for co-design of all system layers • Architecture, runtime and OS, programming models • Reduces design complexity from O(N2) to O(N) • Enables holistic reasoning about concepts and tradeoffs • Empowers discrimination, commonality, portability • Establishes a phylum of HPC class systems SIF-MOE Model 1968 Vector Model 1975 SIMD-array Model 1983 CSP Model 1991 ? Model 2020

21. ParalleX Execution Model- A Virtual Brain? • Lightweight multi-threading • Divides work into smaller tasks • Increases concurrency • Message-driven computation • Move work to data • Keeps work local, stops blocking • Constraint-based synchronization • Declarative criteria for work • Event driven • Eliminates global barriers • Data-directed execution • Merger of flow control and data structure • Shared name space • Global address space • Simplifies random gathers

22. Isomorphism between ParalleX and Neuron Precepts • Compute Complexes (instantiated threads) • Neurons • Performs local operations • Parcels • Connectivity • Event driven • Local Control Objects • Defines satisfiability constraints for firing • Manages asynchrony • Global Address Space & hierarchical name space (processes) • Brain • Provides single global context • Can be used directly for parallel emulation in near-term

23. Summary Conclusions • Dennard scaling and Moore’s Law are at an end • Neuro-inspired computing and computational elements • Power perspective in expanding beyond conventional practices • A long history of prior art • New technologies advancing new opportunities • Execution models are fundamental abstraction required for success of any general or special purpose computing strategy • Neo-digital era or non-von Neumann architectures • Revolutionary • Isomorphic computing • Neuro-Inspired computing