Aggregation and propagation

1. Aggregation and propagation David Clark September 2003

2. The problem • Find an approach to building a distributed cognitive system. • Global scale. • Avoid “event implosion”. • Minimize global standards. • No assumption that all parts work correctly. • Focus on examples that stress the distributed nature, like fault resolution.

3. The approach • Cognitive-mediated “publish-subscribe” system. • Sort of… • A spectrum from multicast to content-based routing. This extends the “rich” end of the scale. • Rules for aggregation to control “implosion”. • A fan-in pattern, not fan-out. • Economic model to control usage

4. The phases of operation • Specification of global agreement. • Establishment of a graph for propagation. • Propagation of subscription. • Advertisement, offer, attractor, etc. • Propagation of notification. • Request, query, event, etc. • My example: FIXIT • Return of reply. • Evaluation of reply • Note: Crafty asymmetry • Subscription less common, more state.

5. Global agreement • Representation and more. • Ties subscriptions to notifications. • Constrains allowed operations. • My approach: array of tuples. • Lets talk about this… • Allows match based on exact and wild-card match on values. • Allows rewrite rules based on syntax. • Lets discuss other sorts of manipulation.

6. A Failure Story • (story VALUE “Read a Web page”)(Hardware WHAT-KIND Dell)(System WHAT-KIND WindowsXP)(Browser WHAT-KIND IE 5.1)(URL VALUE http://www.google.com/search?q=content++routing&ie=UTF-8&oe=UTF-8)(DomainName VALUE www.google.com)(LocalDNSState (HOW-MANY 2) (Address1 VALUE 18.26.0.18) (Address2 VALUE 18.26.0.32))(DNSQuery (Address1 DomainName) RETURNED xxx.yyy.zzz.aaa)(TCPSession FinalState SYN-SENT)(Failure HOW-URGENT Acute)(Trouble-Ticket VALUE LocalIDnonce)

7. Subscription • (Advertisement, offer, etc.) • Explicit and implicit/built-in. • Three parts (at least): • A set of tuple patterns to match. • A set of aggregation/rewrite/forwarding rules.. • Who is subscribing. (Return path.) • Install state in aggregation and forwarding agent. • Propagation cost-limited.

8. Notification • (Request, query, event, etc.) • Not just requests, but facts, etc. • Propagate from source in stages. • In each stage of propagation: • Process in a notification resolution (NR) agent. • My old request satisfaction (AF) agent. • Process in an AF agent. • Notifications propagate until: • They run out of money before they reach a matching subscription. • Resolution may not not terminate propagation.

9. The propagation picture A propagation stage: a Think Point? Cognitiveprocessing Cognitiveprocessing? Notificationresolution agent Aggregation& forwarding agent Notification A/F stateNotification state Subscription

10. Notification state • Notifications trigger transient state. • Not like subscription state. • Undo aggregation on a reply. • Hold notification as part of aggregation rule.

11. Aggregation • Aggregation of notifications is critical. • Avoid “event implosion”. • Allow observation at a distance without overwhelming cost. • Where do aggregation/forwarding rules come from? • Subscriptions. • Defaults. • Success stories. • Higher level mechanisms • Support contracts. • Latter two lead to “directed” notifications.

12. The propagation picture A propagation stage: a Think Point? Cognitiveprocessing Cognitiveprocessing? Notificationresolution agent Aggregation& forwarding agent Notification A/F stateNotification state Subscription

13. Why this isn’t publish/subscribe • Fan-in, not fan-out. • Notifications don’t always go all the way. • Aggregation, resolution • Notifications get rewritten. • Some notifications trigger replies. • Some notifications carry their own subscription (directed notifications.) • Evaluation stage.

14. Untrustworthy agents • Have some fanout at each stage so single disfunctioning agent cannot “eat” notification or subscription. • Evaluate reply and generate consensus on good and bad agents.

15. The region model • Any element with enough power can think for/about itself. • But need some higher-level unit of forwarding and processing. • The Region. • Associated with an administrative unit. • Maps to logical topology. • Approximates units of trust and tussle. • But another abstract topology might be better for some class of notification.

16. The propagation picture redux A propagation stage: a Think Point? Cognitiveprocessing Notificationresolution agent Aggregation& forwarding agent Notification A/F stateNotification state Subscription Region map

17. Where is the cognition? A propagation stage: a Think Point? Cognitiveprocessing Cognitiveprocessing? Notificationresolution agent Aggregation& forwarding agent Notification A/F stateNotification state Subscription

18. Is A/F “cognitive? • Must assume a baseline of common activity for A/F agent. • Must respond to subscriptions and notifications in a predictable way. • Can/should it do more? • Observe patterns and reason. • Learn new “default” rules. • Priority of forwarding rules. • Who do you trust. • Don’t mess up the baseline. • Lots of options for learning.

19. What is a reply? • Like a notification. • Same formats, etc. Perhaps more. • But has a destination address • In this respect, like a subscription or directed notification. • Propagated hop by hop, not tunneled to destination. • Aggregation/privacy rules may rewrite destination. • NR agents may peak and remember replies.

20. What is missing in this talk? • How cognitive “stuff” propagates. • What is stored in NR state? • Past notifications? Replies? Rules? Experimental results? Models? • Should notifications be “active”? • Where do success stories come from? • Privacy concerns.

21. Map of Think Points TP TP YourISP Yourhost TP Networkregion TP TP Networkregion Myhost TP MyISP TP Networkregion TP TP Networkregion Networkregion