Conceptual Model Evaluation. Towards more Epistemological Rigor

1. Conceptual Model Evaluation. Towards more Epistemological Rigor Presentation at EMMSAD ’05 Jan Recker Queensland University of Technology June 14, 2005

2. Why Philosophy? [a] good part of the answer to the question “why philosophy?” is that the alternative to philosophy is not no philosophy but bad philosophy. The ‘unphilosophical’ person has an unconscious philosophy, which they apply in their practice – whether of science or politics or daily life. Collier (1994), p. 17

3. Agenda • Motivation • Philosophy in IS Research • Conceptual Model Evaluation • A Paradigmatic Discussion Framework • Sample Application • Conclusions

4. The Need for Evaluation Motivation Philosophy in IS Model Evaluation Discussion Framework Application Conclusions “No problem-solving process can be considered complete until evaluation has been carried out. It is the evaluation which helps us to measure the effectiveness of the problem-solving process and the problem solver in the 'problem situation' – unless this element is considered there is no way of establishing that the 'problems' have been successfully resolved” Jayaratna (1994), p. 108

5. Scope, Idea, and Objective Motivation Philosophy in IS Model Evaluation Discussion Framework Application Conclusions • Scope: Evaluation of conceptual models • Idea: Transfer Insights from Philosophy to the field of Evaluation: Paradigmatic Analysis • Objective: Explicate the implications of paradigmatic presuppositions onto the evaluation of conceptual models • Research method: Critique: Explicating and Discussing paradigmatic presuppositions onto the evaluation of conceptual models to determine possibilities, scope, and limits.

6. Philosophy and IS Research Motivation Philosophy in IS Model Evaluation Discussion Framework Application Conclusions • IS discipline at the intersection of multiple research fields • “State of diversity” • Diverse approaches towards cognition, reality, truth etc. are being used

7. Paradigms in IS Research Motivation Philosophy in IS Model Evaluation Discussion Framework Application Conclusions • Paradigm: a specific way of thinking about problems based on a set of achievements that are acknowledged as a foundation of further research practice Avison & Fitzgerald (1995) • Predominant paradigms in IS: • Positivism • Interpretivism Chen & Hirschheim (2004)

8. Paradigmatic Framework Motivation Philosophy in IS Model Evaluation Discussion Framework Application Conclusions

9. Models in IS Research Motivation Philosophy in IS Model Evaluation Discussion Framework Application Conclusions • Conceptual Models • facilitate, systemize, and aid the process of information system engineering. • describe object systems (e.g. an information system) of some domain in semantic terms, using an abstract yet formalized language. • Core (if not the) artefact(s) of the discipline

10. Selected Evaluation Approaches(empirical) Motivation Philosophy in IS Model Evaluation Discussion Framework Application Conclusions Cf. Siau & Rossi (1998)

11. Selected Evaluation Approaches(non-empirical) Motivation Philosophy in IS Model Evaluation Discussion Framework Application Conclusions Cf. Siau & Rossi (1998)

12. Preliminary Conclusions Motivation Philosophy in IS Model Evaluation Discussion Framework Application Conclusions • Plethora of model evaluation methods available • Evaluation design is coined by philosophical presuppositions of designing and applying researchers • Discussion of presuppositions aids explicating evaluation approaches

13. Construction (1/2) Motivation Philosophy in IS Model Evaluation Discussion Framework Application Conclusions • Investigation of existing model evaluation techniques based on • Philosophical aspects • Ontological aspect • Epistemological aspect • Paradigmatic approach upon aspects • Positivist approach • Interpretivist approach • Paradigmatically coined perception of • Models • Evaluation • Quality

14. Construction (2/2) Motivation Philosophy in IS Model Evaluation Discussion Framework Application Conclusions

15. The BWW Ontology Motivation Philosophy in IS Model Evaluation Discussion Framework Application Conclusions • Wand and Weber applied an ontology based on Bunge’s work to the field of conceptual modeling • The BWW Ontology specifies “reality” constructs that a conceptual modeling language should be able to depict • The BWW ontology serves as a reference point in evaluation • Evaluation is conducted by mapping language constructs against ontology constructs and thereby assessing ontological completeness and ontological deficiency within the modeling language Cf. Bunge (1977) Cf. Wand & Weber (1993)

16. Applying the framework Motivation Philosophy in IS Model Evaluation Discussion Framework Application Conclusions Representation Correspondence Clarity and Completeness

17. Findings: Some Implications Motivation Philosophy in IS Model Evaluation Discussion Framework Application Conclusions • Application of BWW approach restricted to contexts adhering to same paradigm • Positivist stance of BWW approach problematic “Positivism” should no longer even be mentioned in discussion of theory or epistemology in information systems as a defensible position. Gregor (2004), p. 4 • Reconsideration and modificaiton of the BWW ontology?

18. Conclusions Motivation Philosophy in IS Model Evaluation Discussion Framework Application Conclusions • Evaluation is problematic due to its dependancy on paradigmatic viewpoints onto • Object of evaluation (models) • Evaluation methodology (obtainment of truth) • Evaluation target objective (quality) • Multi-paradigm research during artefact design and evaluation is proscribed • Need for more paradigmatic rigor in IS evaluation research

19. Thank you for listening Any Questions?

20. Selected References (1/2) Motivation Philosophy in IS Model Evaluation Discussion Framework Application Conclusions • Avison, D.E. and Fitzgerald, G. (1995) Information Systems Development: Methodologies, Techniques and Tools, McGraw-Hill Companies, London. • Berger, P.L. and Luckmann, T. (1966) The Social Construction of Reality : A Treatise in the Sociology of Knowledge, Doubleday, Garden City. • Bunge, M.A. (1977) Treatise on Basic Philosophy Volume 3: Ontology I - The Furniture of the World, Kluwer Academic Publishers, Dordrecht. • Collier, A. (1994) Critical Realism: An Introduction to Roy Bhaskar's Philosophy, Verso, London. • Gregor, S. (2004) "The Struggle Towards an Understanding of Theory in Information Systems" in D. Hart and S. Gregor (eds.) Information Systems Workshop: Constructing and Criticising, School of Business and Information Management, Canberra, pp. 1-11. • Gruber, T.R. (1993) What is an Ontology?, Retrieved from http://www.ksl.stanford.edu/kst/what-is-an-ontology.html.

21. Selected References (2/2) Motivation Philosophy in IS Model Evaluation Discussion Framework Application Conclusions • Jayaratna, N. (1994) Understanding and Evaluating Methodologies: Nimsad, a Systematic Framework, McGraw Hill, New York et al. • Losee, J. (2001) A Historical Introduction to the Philosophy of Science, Oxford University Press, Oxford. • Siau, K. and Rossi, M. (1998) "Evaluation of Information Modeling Methods -- A Review" in D. Dolk (eds.) 31st Hawaii International Conference on System Sciences, Computer Society Press, Big Island, pp. 314-322. • Stufflebeam, D.L. (2001) Evaluation Models. New Directions for Evaluation, Jossey-Bass, San Francisco. • von Glasersfeld, E. (1987) The Construction of Knowledge. Contributions to Conceptual Semantics, Intersystems Publications, Seaside. • Wand, Y. and Weber, R. (1993) On the Ontological Expressiveness of Information Systems Analysis and Design Grammars, Journal of Information Systems, 3, 217-237.