IACT 424/924 The Design Process: Evaluating Plans

1. IACT 424/924The Design Process: Evaluating Plans William Tibben SITACS University of Wollongong 17 September 2002 Written by William Tibben

2. Overview • Definitions • Brief example to illustrate why we evaluate • A real life example of an evaluation process – Case Study of Commonwealth Gov Tender • Move to the theory – the need to determine priorities • Final focus on technical matters. Written by William Tibben

3. Definition • Evaluation – to determine the value or worth of, to appraise (Cassell Concise English Dictionary) • Some common anxieties that evaluation should relieve • Will it work? • Will it meet the design criteria? • How much is it going to cost? Written by William Tibben

4. What is evaluation in this context? • Evaluation in this context refers to a recognised point in time where the detail of the design is evaluated in relation to the primary goals that have been established • Is distinguished from the “normal” and ongoing evaluation that one undergoes while doing work (eg am I doing it right?) Written by William Tibben

5. What is evaluation in this context? • Represents a point where goals can be re-assessed to determine if the detailed design will meet the goals. • A point in which the detailed design can be altered (fine tuned) to better meet goals • A point in which goals can altered to take account of contradictions between goals Written by William Tibben

6. Tender 1. $120,000 Main Serial Line – Telstra ISDN line Back up Line – Dial- up modem using PSTN Tender 2 $130,000 Main Serial Line – Telstra ISDN Line Back up line – ‘Line-of-sight’ microwave link owned by business. Scenario 1: The Design Criteria states that a backup line needs to be provided for the ISDN serial link between office 1 and office 2 Written by William Tibben

7. Which tender? • Which tender would you choose? • Tender 1 - the cheapest Or • Tender 2 – a separate and higher bandwidth back-up link Or • Go for a change in design criteria – make the main link the microwave link (avoid costs for the ISDN line from Telstra) and use a dial up modem backup Written by William Tibben

8. In summary, we often need to balance the desire for technical excellence and cost • It can be a point where management and engineering collide. Written by William Tibben

9. Evaluating and resolving conflicting objectives One example of an evaluation task is a Commonwealth Government Tender Assessment Procedure • Technical Proposal • if OK then move onto • Budget Written by William Tibben

10. Comm Government Tender • Technical Proposal – • knowledge and experience in relevant fields – 50% • Managerial and financial capabilities – 20% • General capabilities – 30% Written by William Tibben

11. Knowledge and Experience in Relevant fields • The design document should detail how the proposed design will meet the the design requirements • The document should detail a timeline as to how the project will be executed Written by William Tibben

12. Managerial and Financial Capabilities • Does the tenderer have appropriate personnel (CV)? • Is the tenderer able to substitute a similarly qualified person should the primary person become unavailable? • Is the tenderer able to complete the work within the budget allocated? Written by William Tibben

13. General Capabilities • Is the tenderer or good reputation? • Does the tenderer have a good understanding of your business and the environment in which it works? • Does the tenderer have good contacts within the industry? Written by William Tibben

14. Lastly,… • Once the tenders have satisfactorily satisfied the technical requirements then tenders should be compared on the basis of cost Written by William Tibben

15. Four foci of evaluation (Cotterall and Hughes1995) • Strategic Assessment • Technical Assessment • Cost-benefit assessment • Risk Analysis • Where do each of these forms of evaluation relate to when considering the Comm. Gov. Tender? Written by William Tibben

16. Strategic Assessment Technical Assessment Cost-benefit assessment Risk Analysis Technical Proposal – knowledge and experience in relevant fields – 50% Managerial and financial capabilities – 20% General capabilities – 30% Budget Written by William Tibben

17. Strategic Assessment Technical Assessment Cost-benefit assessment Risk Analysis Technical Proposal – knowledge and experience in relevant fields – 50% Managerial and financial capabilities – 20% General capabilities – 30% Budget Written by William Tibben

18. Four foci of evaluation (Cotterall and Hughes1995) • Strategic Assessment • Objectives: support for corporate vision • Information System (IS) Plan: legacy systems • Organisation Structure: enhance of destroy? • Management Information System (MIS) • Personnel: manning levels and skill base • Corporate Image: will it affect customer perceptions Written by William Tibben

19. Four foci of evaluation (Cotterall and Hughes1995) • Technical Assessment • Functionality: will the end product work? End-end connectivity issues Applications Security Metrics – refer lecture 8: design requirements • Scalability: is the network able to grow without major problems • Adaptability: will the project be able to incorporate new technologies in the future? • Manageability: can we monitor network operations and make necessary changes easily? Written by William Tibben

20. Four foci of evaluation (Cotterall and Hughes1995) • Cost-benefit Analysis • Costs • Development cost • Setup cost • Operational costs • Benefits • Direct benefits (reduction in salary bills) • Indirect benefits (increased accuracy, increased timeliness, more user friendly) • Intangible benefits (Better customer and supplier relationships, better information flows/problem solving) Written by William Tibben

21. Four foci of evaluation (Cotterall and Hughes1995) • Cost-benefit Analysis • If benefits > cost, that is good • If benefits < cost – that is bad • How do you measure intangible benefits? Written by William Tibben

22. Four foci of evaluation (Cotterall and Hughes1995) • Risk Analysis • What is the likelihood that an event will result in the project not meeting its objectives? • A cost-benefit style of analysis can also be used to quantify possible losses. “What if” analysis • The difficulty is determining the likelihood of an event occurringor accounting for an unpredictable set of circumstances Written by William Tibben

23. Resolving Conflicting Goals • It is in the evaluation phase when goals can collide. How does one resolve these? • McCabe (1998) suggests that you prioritise design gaols. • What kind of prioritisation occurred with the Comm Gov Tender? Written by William Tibben

24. Resolving Conflicting Goals • McCabe (1998) also suggests that prioritisation of goals allows one to better clarify future directions. This was demonstrated in Scenario 1described at the beginning of the lecture. Written by William Tibben

25. Technical Assessment • How does one determine the technical characteristics of a proposal? • Functionality: will the end product work? End-end connectivity issues Isolate specific services and determine whether flows indicated on the diagram are appropriate Written by William Tibben

26. Written by William Tibben Teare, 1999, p. 384

27. Written by William Tibben Teare, 1999, p. 385

28. A distributed database for record keeping Interactive training modules demonstrating fine art of coffee and tea making Streaming Audio and Video for meetings Ask the question whether specific technologies are suitable for the kind of service you wish to deliver. Best effort Deterministic service Guaranteed Service Milestone 3 Scenario Written by William Tibben

29. Criteria used in your last mile stone exercise • Response Time • Accuracy • Availability • Maximum Network Utilisation • Throughput • Efficiency • Latency Written by William Tibben

30. Fill in the blanks Written by William Tibben

31. Technical Assessment • Scalability: is the network able to grow without major problem • Are group sizes appropriate? • McCabe (1998, p. 185) • In a broadcast environment (eg Ethernet) background broadcast traffic should be < 2% • Need to remember that routing protocols also account for background traffic and need to be considered • e.g Connection oriented protocols need to exchange state information between end points in order to maintain track of the packet sequence, errors and buffer overflow Written by William Tibben

32. Technical Assessment • Scalability: is the network able to grow without major problem • Is there sufficient room for growth in the protocols you have chosen? • All protocols have a common upgrade path to higher bandwidths. You should be choosing protocols to ensure that they can be speeded-up as increased traffic demands Written by William Tibben

33. Written by William Tibben McCabe, 1998, p.193

34. Technical Assessment • Adaptability: will the project be able to incorporate new technologies in the future? • Best effort may be OK today but your applications may demand Deterministic Service or Guaranteed Service in the future • If you can foresee that there will be a future requirements for more demanding specifications you will need to consider whether you should move to those protocols now. Written by William Tibben

35. McCabe, 1998, p.195 NBMA=Non-Broadcast Multiple Access Written by William Tibben

36. McCabe, 1998, p.196 Written by William Tibben

37. Technical Assessment • Manageability: can we monitor network operations and make necessary changes? • In-band monitoring – relies on existing network to transmit management data • Out-of-band monitoring uses a separate network to transfer information (e.g. a dialup line using the PSTN) Written by William Tibben

38. Prototyping • Prototyping represents another important evaluation mechanism • Begins to stray into the area of next lecture validating plans Written by William Tibben

39. References • MacCabe, D. D. 1998, Practical Computer Network Analysis and Design, Morgan Kaufman Publishers, San Francisco, California. • Teare, D. 1999, Designing Cisco Networks, Cisco Press Indianapolis. Written by William Tibben