Erol Gelenbe Professor in the Dennis Gabor Chair Imperial College ee.ic.ac.uk/gelenbe

1. Self-Aware Networks: QoS & Performance http://san.ee.ic.ac.uk Summary in E. Gelenbe, Communications of the ACM, July 2009Introduction in S. Dobson et al. “Autonomic Communications”, ACM Trans. Adapt. Aut. Systems, 2006 Erol Gelenbe Professor in the Dennis Gabor Chair Imperial College www.ee.ic.ac.uk/gelenbe e.gelenbe@imperial.ac.uk Member of Academy of Sciences of Turkey www.tuba.gov.tr Academy of Sciences of Hungary www.mta.hu Academie des Technologies (France) www.academie-technologies.fr

2. Self Awareness and Adaptation in Networked Natural Systems is our Speculative Topic of Investigation: Spiking Neurons, Gene Regulatory Networks, Chemistry, Digital Economy Extension: Gene Regulatory Networks

5. A Communication Network such as the Internet is a Control System whose function is to Adaptively and Robustly vehicule Information between Users and offer Services to Users • Old Goals include Quality of Service (Loss, Delay, Jitter, Low Overhead) • New Goals include Security, Privacy, Low Energy Consumption, Resilience to Attacks Computer-Communication Networks are Robust Control Systems

6. Network Control cannot be based on “set point” approaches because such simple points do not exist • Models are routinely used for Preliminary Design of Topologies and Capacity Optimisation • The use of Models for Network Control is limited because of dimensionality, model accuracy, parameter identification, control computation and control delays • Self-Awareness in this Area can be based on On-line Measurement and Continuous, Distributed Adaptation Computer-Communication Networks  Self-Aware Systems

7. Self-Awareness must Address Network Security via Detection of Attacks and Adaptive Countermeasures • Energy Consumption by Networks and ITC represents 2% of worldwide consumption  Self-Aware Control needs to address this issue to limit any further increase while ITC pervasively extends smart homes, smart vehicles, smart grid, e-learning • Network Self-Awareness must incorporate the application: digital economy, smart grid  Co-Networks [Comms+Application] •  On-line Measurement and Continuous, Distributed Adaptation Co-Networks  Self-Aware Systems

8. IP Core Technology also supports MobilityHot Stuff happens at the periphery: parasitic and symbiotic relationships

9. Fixed and Mobile Convergence IP is obviously – for the time being – the universal solution provider for Communications

10. Applications as Networks: The Family’s Private Network The FPN is a Network User, and its is a Service for the Family Private to the family and specific services: Home Security, Housekeeper, Parents Fully known to itself (self-aware, self-monitoring) cheap, secure, private and dependable Must dynamically and economically use all available communication modalities: Sensor Networks, IP etc., public fixed and mobile telephony, WiFi, .. You guess it .. Children and Parents have PDAs and Lap-Tops Children use home “learning center” based on multiple external services Parents’ cars are connected Infants, grand parents, teen-agers are being monitored House security system, Home temperature, sound sensors & actuators are on-line Home appliances, security, entertainment, heating, lighting are monitored and controlled Autonomic Self-Aware Management of Connections, U[grades, Costs, QoS, Policies, Virtual Services, Network Resources, Monitoring …

11. The Internet Architecture: A Historical Perspective- It was Invented “Against” the Telecoms Industry because of DoD’s Dissatisfaction with Telephony for Dependable Communications- The Initial System was Built on top of Analog Telephony- The Telecoms Industry Fought it for Many Years and Created Competing Services (e.g. Minitel, Teletext) .. and so did some of the Computer Industry (IBM’s EARN, Frame Relay, Transpac)- The First Major Application was File Transfer .. Then E-mail ran under FTP- The Internet grew in a Time of Deregulation based on Loose Standardization (e.g. IETF)

12. Historical Internet Protocols: OSI Layers • TCP/IP is a layered protocol stack • Application handles particular applications, Presentation handles compression and encryption of data, Session controls establishment, management, termination of sessions • Transport provides flow of data • Network handles the transmission of packets in the network • Data-link is responsible for the interaction of the device driver in the operating system and the network card in the machine • Physical defines electrical and mechanical specifications

13. IP Architecture in Reality:An Assembly of Inter-dependent Protocols- The Web is a “Standard User Interface”- TCP the Transmission Control Protocol: Controls Packet Flow for a Connection as a Function of Correctly Received or Lost Packets (TCP Reno, Vegas, etc.) & Retransmits Lost Packets - BGP: Determines Paths between Clouds of Routers belonging to Autonomous Systems (AS)- MPLS: Carries out fast Packet Switching based on Pre-determined Paths within ASs using Labels, Implements Traffic Engineering within ASs - IP Implements Shortest Path Routing within ASs. Variants of IP Address QoS (e.g. DiuffServ, IPV6), Weighted Fair Queueing, Congestion Control through Packet Drop …

14. IP Architecture: A Distributed System which has Evolved through Usage and Practice- It grew in a Time of Deregulation- The Web was developed to store technical papers- TCP is a Rudimentary Congestion Control Mechanism- BGP Allows Different ISPs to Exchange Traffic- MPLS Exploits ATM-like Fixed Path Routing and Reduces the Overhead of Packet Switching- The IP (Internet Protocol) provides Router Based Staged Decisions & Shortest Paths to Minimize Overhead

15. Critique from the Founding Funders (DARPA) in August 2004 - “Flaws in the basic building blocks of networking and computer science are hampering reliability, limiting flexibility and creating security vulnerabilities” (Note that DARPA paid for most of these developments !!)- DARPA wants to see the IP and the OSI protocol stack revamped- “The packet network paradigm … needs to change … we must … have some mechanism for assigning capabilities to different users … today’s networks are stationary and have a static infrastructure … (mobile) nodes should be able to automatically sign on to networks in their vicinity …

16. The “Historical” Need for QoS- Exploit different technologies dynamically, for the users’ benefit- Identify and eliminate undesirable traffic flows, e.g. SPAM- Protect from Malicious Attacks: Viruses and Worms, Denial of Service Attacks- Identify services and charge for them- Provide security to connections- Demonstrable service levels and monitored agreements- Wireless makes it all the more urgent

17. The Means for Quality of Service- Effective user interfaces & Access Control- Pro-active monitoring of users- On-line monitoring of flows- Pro-active measurement - Monitoring of QoS/QoE and Service Level Agreements- Technical approach that Crosses Protocol LayersCoordinated Measurement Based Dynamic Adaptation

18. In Theory, QoS Optimisation Problems can be defined and Solved

19. The network is very large – for specific users, optimization is relevant for a subset of routes at a time • The system is large .. information delay, control delay and combinatorial explosion: global algorithms can be very slow and come too late • The system is highly dynamic – traffic varies significantly over short periods of time • There are large quantities of traffic in the pipes – congestion can occur suddenly, reaction and detours must be very rapid • Measurements are local to subset of users, and adaptivity is needed which is relevant to the users most concerned by the measurements In Practice Optimisation is Impossible

20. - Many applications have implicit or explicit QoS requirements • Voice, Video conferencing, Network games and simulation, Commerce , Banking • QoS techniques based on “parameter setting” such as IntServ, DiffServ and IPv6 have been unsuccessful • Our proposal: • Users and Services formulate their Needs and QoS Goals • The Network Service adaptively offers Services to Users and the Resources to Users and Services • The Network Service Monitors the outcome and Adaptively re-allocates Services and Resources as needed • The Approach is that of Adaptive Control in a Random Environment On-Line Measurement and Adaptation “Self-Aware Networks”

21. Sensible Routing Theorem [Gelenbe, Computational Management Science ‘03] Consider any decision point x at with n choices Let q(x,i) be a r.v. representing the QoS that results from choice i, and lrt W(x,i) = E[u(q(x,i))] be the expected outcome of choice i Theorem If p(k)(x,i) = [W(x,i)]-k/ Snm=1 [W(x,m)]-k and W(k)(x) = Snm=1 W(x,m) p(k)(x,i), then W(k+1)(x) < W(k)(x) How about theory?

22. Simulation of Routing with Perfect IgnoranceAverage Travel Time E[T] vs Average Time-Out 1/r for a regular 8 neighbour grid with d=1 and D=10 (b=0, c=1.5)

23. Packets go from some source S to a Destination (that may move) that is initially at distance d • The wireless range is d << d, there are no collisions • Packets can be lost in [t,t+Dt] with probability lDt anywhere on the path • There is a time-out R (in time or number of hops), or timed-out in [t,t+Dt] with probability rDt with a subsequent retransmission delay M • Packets may or not know the direction they need to go – we do not nail down the routing scheme with any specific assumptions • We avoid assumptions about the geography of nodes (in some m-dimensions) but assume temporal and spatial homogeneity and temporal and spatial independence along the distance to the destination – each physical setting will have a different distance .. General Context

24. Destination Source Simulation examples in an infinite grid: d=1 (D=6) or d=sqrt(2) (D=4)

25. Simulations of Average Travel Time vs Constant Time-Out d=1, D=10, M=20, No LossPerfect Ignorance: b=0, c=1

26. Brownian Motion ModelAssumes homogeneity with respect to distance to the destinationAfter each Loss or Time-Out, the Source waits M time units and then retransmits the packet under identical statistical conditions

28. Theoretical Result: Time to Travel to a Destination at Distance D • Average Time E[T] • Drift b < 0 or b>0, 2nd Moment Param. c>0 • Average Time-Out 1/r , M=1/m, then

29. Theory: Average Travel Time for D=10, b=0, 4-Neighbours and M=50 with Losses

30. Theory: what happens when loss rate increases

31. What happens if we send multiple copies of each packet and accept the first one that arrives ?

32. What happens if we send multiple copies of each packet and accept the first one that arrives ?

33. Send N copies of each packet and accept the first that arrives ..

34. Mathematical model assumes exponentially distributed loss time, time-out and re-start time after time-out • It assumes temporal and spatial homogeneity • It allows for generally distributed times between hops • Closed form expressions are obtained for E[T] from the diffusion model with jumps for al values of D, b, c, r, l, m • If b<0 (good) the average total travel time E[T] is finite • If b>0 (bad), E[T] is finite provided that time-outs (hara-kiri) exist and c>0 • There is a single minimum of E[T] as a function of r • Additional results: Multiple Copies of Each Packet – You can actually save on energy and improve timeliness Sufficient conditions under which the packet gets to the destination when the destination moves Summary

35. Let the Measurements and On-Line Adaptation be under “user” or meta-user control • Let the user make his/her own QoS and economic decisions • Remain compatible with the core IP protocol .. i.e. be able to run CPN and IP simultaneously and tunnel IP packets through CPN sub networks QoS Routing Pragmatics :The Cognitive Packet Network Approach

36. OSI Layers & CPN • TCP/IP is a layered protocol stack • Application handles particular applications, Presentation handles compression and encryption of data, Session controls establishment, management, termination of sessions • Transport provides flow of data • Network handles the transmission of packets in the network • Data-link is responsible for the interaction of the device driver in the operating system and the network card in the machine • Physical defines electrical and mechanical specifications CPN

37. CPN operates seamlessly with IP and creates a self-aware network environment • Users select QoS goals • The User’s Packets collectively learn to achieve the goals • Learning is performed by sharing information between packets • User Packets sharing the same goals can be grouped into classes • Nodes (Cognitive Routers) are storage centers, mailboxes and processing units CPN Principles

38. Smart Packets route themselves based on QoS Goals, e.g., Minimise Delay or Loss or Combination Minimise Jitter (for Voice) Maximise Dispersion (for security) Minimise Cost Optimise Cost/Benefit Smart Packets make observations & take decisions ACK Packets bring back observed data and trace activity Dumb Packets execute instructions, carry payload and also may make observations CPN and Smart Packets

39. QoS Goals are obtained from Path Measurements: Traffic, Delay, Loss, Cost, Power, Security Information – this is the “Sufficient Level of Information” for Self-Aware Networking • Smart packets collect path information and dates • ACK packets return Path, Delay & Loss Information and deposit W(K,c,n,D), L(K,c,n,D) at Node c on the return path, entering from Node n in Class K • Smart packets use W(K,c,n,D) and L(K,c,n,D) for decision making using Reinforcement Learning Cognitive Adaptive Routing

40. Decision System: a “neural” network of networks (neural networks embedded in each router) Internal State of Neuron i, is an Integer xi> 0 Network State at time t is a Vector x(t) = (x1(t), … , xi(t), … , xk(t), … , xn(t)) Is the Internal Potential of Neuron I If xi(t)> 0, we say that Neuron i is excited and it may fire at t+ in which case it will send out a spike If xi(t)=0, the Neuron cannot fire at t+ When Neuron i fires: : It sends a spike to some Neuron k, w.p. pik Its internal state changes xi(t+) = xi(t) - 1

41. Rates and Weights If xi(t)> 0, then Neuron i will fire with probability riDt in the interval [t,t+Dt] , and as a result: From Neuron i to Neuron m - Excitatory Weight or Rate is wim+ = ri pim+ - Inhibitory Weight or Rate is wim- = ri pim - - Total Firing Rate is ri = Snm=1 wim+ + wim– To Neuron i, from Outside the Network - External Excitatory Spikes arrive at rate Li - External Inhibitory Spikes arrive at rate li

42. State Equations

43. External Arrival Rate of Excitatory Spikes Probability that Neuron I is excited Firing Rate of Neuron i External Arrival Rate of Inhibitory Spikes

44. The Goal Function to be minimized is selected by the user, for instance G = [1-L]W + L[T+W] • On-line measurements and probing are used to measure L and W, and this information is brought back to the decision points • The value of G is estimated at each decision node and used to compute the estimated reward R = 1/G • The RNN weights are updated using R stores G(u,v) indirectly in the RNN which makes a myopic (one step) decision Goal Based Reinforcement Learning in CPN

45. Routing with Reinforcement Learning using the RNN • Each “neuron” corresponds to the choice of an output link in the node • Fully Recurrent Random Neural Network with Excitatory and Inhibitory Weights • Weights are updated with RL • Existence and Uniqueness of solution is guaranteed • Decision is made by selecting the outgoing link which corresponds to the neuron whose excitation probability is largest

46. The decision threshold is the Most Recent Historical Value of the Reward • Recent Reward Rl If then else Reinforcement Learning in CPN

47. Re-normalize all weights • Compute q = (q1, … , qn) from the fixed-point • Select Decision k such that qk > qi , i=1, …, n

48. Test-Bed Measurements On-Line Route Discovery by Smart Packets

49. Route Adaptation without Obstructing Traffic

50. Packet Round-Trip Delay with Saturating Obstructing Traffic at Count 30