The Insufficiency of Formal Design Methods - the necessity of an experimental approach for the understanding and contro

1. The Insufficiency of Formal Design Methods- the necessity of an experimental approach for the understanding and control of complex MAS Bruce EdmondsCentre for Policy ModellingManchester Metropolitan University Joanna BrysonDepartment of Computer Science,University of Bath Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-1

2. Main Themes • The need for a shift of emphasis: • From verification towards validation • From the “10%” towards the “90%” (in the adage) • From engineering(of new systems) to adaptation(of existing systems) • This goes against implicit assumptions (in SE): • Design/engineering is “better” than testing/adaptation • Computational systems are deterministic (in reality) • Since (aspects of) software systems can be represented as formal objects that formalist approaches will be able to deliver (increased) reliability • Whilst this shift may be resistible with single, closed systems it will be unavoidable with MAS Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-2

3. Outline of the talk • Some formal limitations to formal approaches (why a “Hilbert Programme” for MAS won’t work) • Software production strategies(the primary SE goal and strategies to get there) • A more “Scientific” approach for SE(utilising the classic experimental method) Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-3

4. Part 1: Some Formal Limitations of Formal Methods - how simple MAS can beand it still be impossible to bridge thespecification program gap Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-4

5. Formal Approach Idea is to write specifications in a formal language Often of a logical or set-theoretic nature Two undisputed advantages: • Specification is unambiguous and concise • Specifications can be syntactically manipulated (as in proofs, checkers etc.) Is thus a sort of lingua franca for software engineers As with any language, there are difficulties that arise when attempting to translate to and from it To it (from the informal) - the “requirements problem” From it (to programs) - the “problems” that follow Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-5

6. T A translating program? Py Sx A given formal specification Any program that satisfies Sx The Programming Problem Posed Is there a general, effective or systematic method of finding a program that satisfies a given specification? Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-6

7. T A translating program? Py Sx A given formal specification Any program that satisfies Sx The Programming ProblemAnswered No, if the language of specification is expressive enough Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-7

8. T A checking program? Py Sx A given formal specification A given program The Checking Problem Posed Is there a general, effective or systematic method of checking whether a given program satisfies a givenspecification? Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-8

9. T A checking program? Py Sx A given formal specification A given program The Checking ProblemAnswered No, if the language of specification is expressive enough Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-9

10. What “expressive enough” means E.g. Those that are able to express basic arithmetic statements about the behaviour of programs That is, able to express statements such as: • Output of program Px is always < 100 • Program Px will not finish before time 1000 • Output of Px is always greater than Py • Output of Px is a approximates that of Py (±10%) (There are other ways such expressiveness can be established: 2-place predicates etc.) Thus in SE terms “expressive enough” is, in fact, the minimum necessary for almost all real systems Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-10

11. Proof Sketch (programming problem answer) Define the “nth limited halting problem”, LHn(x,y), as:DoesPxever halt with input y where bothx,yn? Each LHn(x,y) is computable as a finite lookup table LHn(x,y) is effectively expressible in a language with arithmetic via the construction in (Gödel 1933) as SHm - that is, m here is computable from n Now if there were a translating program,T, then: given x and y; let z=max(x,y); compute SHz; use T to find a program to compute LHz(x,y) from SHm; and use this to find whether Px(y) halts; but this is impossible(Turing 1936). Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-11

12. An example: how “simple” a MAS can be and still be beyond formal methods • Giving Agent System with Plans (GASP) • Fixed number of agents: A1, A2, …An • Each agent, Ax, has • a single store, Sx • a fixed number of plans: Px1, Px2, … • Each Plan, Pxy, is composed of instructions: • A fixed number of “give one to …” • And one final test instruction: If Si is zero then do plan j next, otherwise plan k next • Each time click, all do: get 1 unit; use current plan to: [do giving (while they have); do test instruction; note next plan]. Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-12

13. An illustration of a GASP system Agent 1 Plan 1: G3 G2 JZ2,1,3 Plan 3: G2 G2 G2 JZ2,3,3 Plan 2: JZ1,2,3 Agent 2 Agent 3 1 2 3 4 1 2 3 Etc. Store: 27 Check if zero Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-13

14. Facts about GASP systems • They are Turing Complete (see paper for proof outline), hence: …many questions about their behaviour are (in general)undecidableeven when given their program • In particular, a specification adequate to specifying the behaviour of GASP systems will need basic arithmetic and so… …the previous results about the (difficulties of the) relationship between an adequate specification language and systems hold Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-14

15. Part 2: Software Production Strategies - Specification and Design Strategies and their problems Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-15

16. Goals and Strategies for engineering computational (IT) systems To produce IT systems that work well in practice when working in their operational context One particular strategy to achieve this is the “formal design strategy” (FDS), summarised as: • Agree the goals for the system; • Write a specification that would meet these goals; • Implement a system that meets this specification. • Works well for relatively simple, closed, static and analysable cases • FDS is obviously a “straw man” but consistent with much rhetoric in MAS and true to the extent that there is an over-emphasis on these stages Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-16

17. Some (well known) causes of problems with the FDS • Context of operation is (at least partially) unknown to designers • Good in practice operation requires meaningful, complex and abstract goals, • Thus, either one has a: High-level specification, in which case you can’t guarantee that the system works according to its specification A Low-level specification in which case you can’t guarantee that the specification achieves the goals Or many staged levels of specificationwith many chances for errors due to repeated translation Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-17

18. Some kinds of complexity • Syntactic Complexity • When the computational ‘distance’ between initial conditions and outcomes is too great to be analytically bridgeable  There are different views of a system • Semantic Complexity • When any formal representation is necessarily incomplete • Models are context-dependent • You probably need many of them Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-18

19. Inseparable system embedding When the particular system is embedded into a wider system such that… the wider system can not be separated from the particular system to aid analysis… without changing the behaviour of both particular system and the wider system,… so that off-line analysis and design is difficult and ineffective. When “particular system” = agent(i.e. MAS), such embedding can emerge in a wide variety of kinds of systems and environment making off-line and formal design strategies inappropriate Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-19

20. Some (well known) software production strategies Engineered Agent Approach • Beliefs, intentions, etc. • Automatic verification • Ontologies, protocols • Agents, groups, teams • Logics • Roles • Deferred decision making • Testing, model checking • Adaptive Approach • Actor • Simulation • Agents, groups, societies • Social&biological analogies • Duplication, competition • Social&individual learning • Post hoc exploration/experimentation • Abstraction • Automation • Standardisation • Modularity • Formalisation • Transparency • Redundancy • Adaptivity • Testing Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-20

21. Why such over-emphasis on (the rhetoric of) formal design strategies? Some guesses: • Illusion that computational systems are deterministic at macro level in practice because they are in theory at micro level • Blame is contained to specification  system production stages, and hence substantially defrayed from designer to user • Fixed goals suit management, external consultants and academics • Used to dealing with simple, closed systems • Engineering is seen as “better” than adaptation Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-21

22. Part 3: An Experimental Approach - an alternative for messy systems and situations Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-22

23. What if the properties of most MAS are messy - more like Biology than Logic? • Lots of kindsof agents, teams, trust, communication forms, etc. • Lots of observation and exploration before any abstraction into theory possible • A priori foundationalist studies based on plausibility probably worse than useless • Success coming more from: what works in context, reliability, and adaptation to the unexpected • And less from: abstraction, modularity and formal analysis Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-23

24. The Classic Experimental Method • Theory developed empirically (not a priori) • Applied using well-validated processes and tools • How the theory can be applied using what approximations is empirically established • The conditions under which a theory can be safely applied (and how) developed over time • Useful properties can only be deducedafter theory has been validated • Has worked in messy systems(in science) where there is little that can be completely generalised Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-24

25. What this might give us… • Explicit, relevant and testable models/hypotheses concerning the properties of certain MAS • With sets of conditions under which it has survived trials/testing (and the extent of success) • Also a set of situations where the hypotheses failed, to indicate the limits of its applicability • Giving confidence and guidance to those who wish to use these systems • From which inferences can be made • A scientific basis on which to build sound engineering practices (e.g. systematisation) Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-25

26. E.g. Engineering a bridge • Use of well-validated general designs and strategies (e.g. arches, columns, suspension) • Multiple approximate calculations (maximum stress, weight, compression) • Use of well-validated components or components made using well-validated techniques (e.g. standard girders or cable) • Simulations of the set-up (e.g. oscillations) …still the unexpected may occur - no illusion that design proof can be used on whole systems to achieve reliability Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-26

27. A Comparison of Approaches (at present time) Formal/Inferential • Precise criteria for success • Well-developed methodology/tools • Taken from formal sciences • Limited applicability • For Small components • As a check with simplified models • Inductive/Experimental • Weak criteria of success • Methodology/tools need improving • From natural sciences • Wide applicability • On real systems • No certainty • An eternal task Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-27

28. Inference or results Setting or set-up Observableoutcome (OO) Adjustableaspect (AA) Strengthening the whole chain of inference about MAS Formal or Computational Model Strengthening the inference stage is counterproductive if this weakens the whole chain It is the strength of the whole chain from AA to OO that matters Target MAS Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-28

29. Some practical steps… • Change the rhetoric and expectations of AOSE • Import well-tried testing techniques from other sciences, e.g. [Bryson, Lowe & Stein 2000] • Independently replicating simulation experiments [Axtel et. al. 1996, Edmonds & Hales 2003] • Extend the open source model to include hypotheses, test results etc. [Edmonds 2004] • Develop experimental experience and methodology [MABS, MAMABS, ESOA, JASSS] • Tools for analysis of MAS [Barber & Lam 2004] • Show how this approach can help achieve robust engineering [Hales & Edmonds 2004] Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-29

30. Conclusion –for messy MAS and MAS environments we am suggesting… • a move away from a foundationalist approach towards a more empiricistapproach… • with less emphasis on prior verificationand more on post hoc validation… • where reliability is sought from experimentally tested theories of system behaviour… • (formalisation entering once there is such theory) • which may entail a loss of theoretical certainty… • but offers greater potential for adaptation and hence in practice performance, including somerobustness in uncertain and complex situations Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-30

31. The End Bruce Edmonds bruce.edmonds.name Joanna Bryson www.cs.bath.ac.uk/~jjb Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-31