Designing Internet-Based Software

1. Designing Internet-Based Software Steven Reiss Brown University

2. Idea: Show Where News Occurs • Display a map of the world • Color countries by amount of news • Highlight recent/breaking news • Have a student build it

3. Design for NewsVisualizer History Crawler Display Control Page Analyzer

4. This Worked Fine … BUT • Put a significant load on user’s machine • Crawling and parsing 400 pages every 15 minutes • Startup was slow • Country determination algorithm • Simplistic and slow • What if this became popular • Would google/cnn/… want 100000 users each looking at 100 pages every 15 minutes? • Need to share data to achieve Internet scale

5. NewsView: Redesign Manager Crawler Page Analyzer Control Display Crawler Page Analyzer History Crawler Page Analyzer

6. NewsView: Redesign • Develop other page analyzers • Course assignment; some maintained • Find and parse by-lines • Look at standard links on the page • Look at URL and how it is organized • Shared crawlers and reader • Run crawlers on grid nodes

7. This Worked Fine … BUT • Grid nodes crash • Byline formats change • Link names change • Directory structure for links changes • Page analyzer code changes at random • Can’t run it from a browser • (Did I fail to mention that this was needed?) • Can’t run as a dashboard widget

8. NewsView Redesign History Proxy Control Mapper Fault Tolerant Crawler Manager Crawler Page Analyzer Crawler

9. Result Live

10. What Does This Show • Software is a moving target • Design is constraint satisfaction • Maximize quality, minimize cost • Subject to requirements and other constraints • But the constraints change continually • In significant ways • In unanticipated ways • BUT the software is expected to adapt • Designing the intangible • Designing the undesignable • Not a science

11. What Else Does This Show • Designer doesn’t control the system • Grid nodes go down • Page formats on news sites change • Web structure of news sites change • Others change their code and break it • Everything is subject to change • In unanticipated and unexpected ways • At unpredictable times • BUT the software is expected to keep working • Designing without control • Designing the undesignable • Not a science

12. The Challenge of Software Design • Develop techniques to deal with • Unanticipatable change • Lack of control • Without these, design doesn’t matter • Even the best design can be wrong by the time the software is ready to be used • Using Newtonian physics to study light-speed travel

13. Potential Solution • Design in terms of abstract components • Separate interface and implementation (apple pie) • But do it in a way so that • Anticipate that interfaces are going to change • Implementations evolve independently of interfaces • Implementation can change to adapt • Implementation can run anywhere (grid; remote; local) • Implementation can be open source Interface Implementation

14. BUT This Isn’t Enough • Need to understand the interface • In order to code to it • JCP works for Java; won’t work in general • Need a efficient way of defining what the interface does • Need to ensure the implementation actually implements that particular interface • This has to remain true even if implementation changes • This has to remain true even if interface changes • BUT current techniques don’t provide this • They only do syntactic matching

15. Intelligent Interfaces  Outerfaces • The interface includes the semantics • Implementations must obey the semantics Implementation Outerface Semantics Check Syntax of Calls Bind

16. What Does This Buy You • Implementations can change • But if it does it has to be rechecked • Applications should still work • Multiple implementations can be available • Can change implementations as needed • Any implementation should work • Implementations can be developed externally • Market for software components • Semantics provide a basis for trust

17. This is Nice … BUT • It all depends on semantics being • Consistent • Complete • Depends on what is meant by semantics • Semantics here need to be • Practical • Checkable • Cover all dimensions of programming

18. Functional Semantics • This is what is generally meant by semantics • Alternatives • Formal specs : nice but impractical • Contracts : nice, but still need to be checked at runtime • Hard to fully cover everything • Test cases : incomplete, but practical • Use contracts and test cases • User can add new test cases to an outerface • All test cases must succeed before an implementation can be used

19. Recovery Semantics • What to do if a component fails • Should be part of the outerface • Not in the code (ala Remote Exception) • Allow implementation to restart • Without terminating the application • Or allow a new implementation to restart • Outerface should include a recovery model • Client maintains this model • Implementation can be restarted from this model • This should happen automatically

20. Security and Privacy Semantics • Security: what the implementation requires of the client • Certain rights or permissions • Authentication • Privacy: what the clients require of the implementation • Limits on what it can do with passed in data • Limits on what it can access (files …) • Should be integrated with underlying system security • This should be part of the outerface • Required for trust • Essential part of a system • Checked against the implementation

21. Economic Semantics • How to choose the implementation • What is important • Resource utilization • How it will be run • Cost of using the implementation • Cost of running the implementation • Level of security or privacy • Data warranties • This is economics • Economic model should be part of outerface

22. This Seems Reasonable … BUT • Can this work • Is it possible to specify these different semantics • Is it possible to check these implementations • Can you code against outerfaces and then choose implementations • Is it practical • Is it something programmers would want to use • To find out • We are building a system called TAIGA

23. IMPLEMENTATIONS Outerfaces + Code OUTERFACES Outerfaces + Code Package + Semantics Package + Semantics P2P BACKBONE TAIGA SHARED FACILITIES CHECK & BIND APPLICATIONS

24. Taiga Applications

25. Taiga Status • These applications demonstrate • The approach is feasible • The approach can work in some cases • But there is much more to do • Our work so far has created as many questions as it has answered • What are the challenges

26. Taiga Challenges: Testing • Challenges • Testing real applications without side effects • Testing the untestable (e.g. map from zip code) • Providing large test cases • Testing in a distributed environment • Approaches • Bundling test cases and data • Distributed sandbox for running tests

27. Taiga Challenges: Recovery • Challenges • Dealing with saved external data • Handling recovery of shared components • Dealing with exceptions and partial calls • Approaches • Incorporate a transaction model • Develop implementation-side recovery model

28. Taiga Challenges: Security • Challenges • Checking privacy statically or dynamically • Handling black-box implementations • Distributed security and privacy • Approaches • Virtual machine-based security sandboxes • Static analysis

29. Taiga Challenges: Economics • Challenges • Determining what can & should be specified • Providing a weighting function • Determining accurate cost values • Approaches • Develop a “currency”

30. Conclusion • Interfaces must include semantics • Semantics in a broad sense • Semantics in a checkable sense • TAIGA is a start • Demonstrates feasibility • But there is a long way to go • This should be the focus of software design • Static design just won’t cut it any more

31. More Information • Downloads • Newsapplet: taiga.cs.brown.edu/newsapplet.html • Newswidget: taiga.cs.brown.edu/NewsViewWdgt.zip • Newsview:ftp.cs.brown.edu/u/spr/newsview.distrib.zip • Source: ftp.cs.brown.edu/people/spr/taiga.tar.gz • Taiga: http://taiga.cs.brown.edu/taiga • Contacts • http://www.cs.brown.edu/people/spr • spr@cs.brown.edu

32. Questions and Comments