Quality Determination

1. Quality Determination for Web-based Applications Hazura Zulzalil

2. Overview of MCDA General definition MCDM process MCDA methods Evaluation of WBA Quality Attribute Relationships Aggregation by Choquet Integral Implementation Case study and results Outlines

3. Aims to give the decision-maker some tools in order to enable him to advance in solving a decision problem where several – often contradictory-points of view must be taken into account. What is MCDA?

4. Highly structured, disciplined and formal approach to decision making evaluating the alternatives in the given set A against the set C of criteria Aggregating the individual evaluations to produce global evaluation Could be used for selection the best possible alternatives or for ranking the alternatives What is MCDM?

5. Set of Alternatives Set of Criteria C1, C2,………Cn A1 x11……..………x1n A2 x21……..………x2n . . Am xn1……..………xmn Weights wi / Importance of Criteria Aggregation Measure Overall worth of an alternative Ai MCDM Process

6. Criteria – interdependence, completeness, non-linear preferences Weights – transparency of process, type of weights, meaning Solution finding procedure – ranking, option Project constraints – cost, time Evaluation of MCDA methods

7. Structure of problem solving process – stakeholder participant, tool for learning transparency, actors communication Data Situation Type of data - qualitative or quantitative Risk/uncertainties – probabilities, thresholds, fuzzy numbers, sensitive analysis Data processing amount Non-substitutability Evaluation of MCDA methods

8. Quality of web site is hard to evaluate Consists of multiple criteria to be measured Simple weighted average cannot be used to summaries the various quality measurements into a single score. Inability to account for dependency among the quality criterion. Tend to construct independent criteria, or criteria that are supposed to be so Causing some bias effect in evaluation Evaluation of WBA

9. Single criteria usability aspects(Collins, 1996; Stefani & Xenos, 2001; Hassan & Li, 2005), content and structure (Bauer & Scharl, 2000). accessibility (Vigo et al., 2007) WBA Evaluation Approaches

10. Multi-criteria WEBQEM (Olsina et al., 1999) EWAM (Schubert & Selz, 1998) WebQual (Barnes and Vidgen, 2002) WAI (Miranda et al., 2006) FQT4Web (Davoli et al., 2005) WBA Evaluation Approaches

11. Stated or implied needs ISO 9126 & other technical info Requirement definition Managerial requirement Quality requirement specification Quality Requirement Definition Metric Selection Rating level definition Assessment criteria definition Preparation Software Development Products Measured value Measurement Evaluation Rated value Rating Result (acceptable or unacceptable) Assessment ISO/IEC 9126 Evaluation Process

12. e-commerce e-learning e-education e-government etc. APPLICATION DOMAIN S C I T S Y I T R I Functionality Reliability Usability Efficiency Maintainability Portability L E A T U C A Q R A H C understandability S Adaptability C time behaviour learnability Analysability maturity suitability I installability T resource operability changeability fault tolerance accuracy S coexistence I utilisation attractiveness stability recoverability R interoperability replaceability efficiency expliciteness testability availability E T security portability customisability compliance manageability degradability C compliance traceability clarity reusability reliability A R helpfulness maintainability compliance functionality A user-friendliness compliance H compliance C usability compliance B U S Indicators, scales and preferred values Quality Model

13. Define software product qualities as a hierarchy of factors, criteria and metrics. Quality factor represents behavioral characteristics of the system Quality criterion is an attribute of a quality factor that is related to software production and design Quality metrics is a measure that captures some aspect of a quality criterion. Quality Attributes for WBA

14. Factor A is split up into three criteria a1, a2, and a3. Criteria a1 with the weight 4 is considered four times as important as criteria a2 and twice as important as criteria a3. Similarly, we can set different weight for each factor to indicate its importance. Overall Quality Score Factor A Factor B Factor C Criteria a1, weight 4 Criteria a2, weight 1 Criteria a3, weight 2

15. Name Description Functionality The capability of the Web site to provide functions and properties which meet stated and implied needs when the site is used under specified conditions Usability The capability of the Web site to be understood, learned and liked by the user, when used under specified conditions Reliability The capability of the Web site to maintain a specified level of performance when used under specified conditions. Efficiency The capability of the site to provide appropriate performance, relative to the amount of resource used, under stated conditions Maintainability The capability of the site to be modified. Modifications may include corrections, improvements or adaptation of the site to changes in environments, and in requirements and functional specifications Portability The capability of the site to be transferred from one environment to another Definition of Quality Attributes

16. Three types of relationships Positive, i.e. a good value of one attribute result in a good value of the other (synergistic goals). Relationships definitions: If characteristics A is enhanced, then characteristics B is likely to be enhanced (+) Negative, i.e. a good value of one attribute result in a bad value of the other (conflicting goals). Relationships definitions: If characteristics A is enhanced, then characteristics B is likely to be degraded (-) Independent, i.e. the attributes do not affect each other. Relationships definitions: If characteristics A is enhanced, then characteristics B is unlikely to be affected (0) Quality Attributes Relationships

17. Interrelationships between quality factors (Perry, 1987)

18. Relationship Chart (Gillies, 1997)

19. Ref Attributes Purpose Techniques used [8, 9] Correctness, Reliability Integrity, Usability Efficiency, Maintainability Testability, Flexibility Portability. Reusability Interoperability To study the relations of different quality goals attribute in developing software Survey -questionnaire [10] Performance Adaptability Maintainability To address the importance of design decision made during software development Case Study - Interview [11] Usability Time to market Reliability, Usability Correctness, Portability To increase the understanding of software quality attributes and their relations Research Literature and Survey –structured interview [12] Quality attributes in 3 different perspectives: management, developer and user perspective To merge different view and discuss the relationships between the quality attributes Discussion (meeting and offline discussion) Techniques to explore the relationships

20. Quality Attributes Relationships for WBA

21. method of combining several numerical values into a single one, so that the result of aggregation takes into account in a given manner all the individual values What is Aggregation?

22. use simple weighted average approach methods are not transparent assume independency the choice of summarization method somehow should depend on the certain parameters E.g. the kind of importance parameters (weights) and the type of dependency and interaction the definition of the quality factors and their relationships must be clearly specified Aggregation issues

23. Common aggregation operators Quasi-arithmetic means (arithmetic, geometric, harmonic, etc.) Not stable under linear transformation and consider criteria as non interacting Median Typical ordinal operator – defined the middle value of the ordered list Weighted minimum and maximum Possible to increase one of the weights without having any change in the result Ordered weighted averaging operators Can express vague quantifiers 23

24. mathematical properties Properties of extreme values Idempotence Continuity Monotonicity Commutativity Decomposability Stability under the same positive linear transformation Properties of an aggregation operator

25. behavioural properties express the decisional behavior, interaction between criteria, interpretability of the parameters and weights on the arguments Properties of an aggregation operator

26. Aggregation by fuzzy integral Different methods have been developed according to type of information to be aggregated and the properties have to be satisfied. 26

27. Fuzzy measures and integral Definition 1: A fuzzy measure on the set X of criteria is a set function  : Ƥ (X) [0,1], satisfying the following axioms  ()=0,  (X)=1. A  B  X implies (A)  (B) (A) represent the weight of importance of the set of criteria A. Additive : if (AB) = (A) + (B); A  B= Superadditive: if (AB)  (A) + (B); A  B= Subadditive if (AB)  (A) + (B); A  B= If a fuzzy measure is additive, then it suffices to define n coefficients (weights) ({ I}), … ({ n}) 27

28. Choquet integral Definition 2: Let  be a fuzzy measure on X. The choquet integral of a function ƒ: (X) [0,1] with respect to  is defined by n C (f(x1),…. f(xn)):=  (f(x(i)) - f(x(i-1))) (A(i)) ƒ ((0)) = 0 i = 1 • Fuzzy integral model does not need to assume independency • Fuzzy integral of ƒwith respect to  gives the overall evaluation of an alternative 28

29. Importance and interaction of criteria Problem of evaluation of student with respect to three subjects: mathematics (M), Physics (P) and literature (L). By weighted sum (3 , 3, 2) result: 29

30. Solved by fuzzy measure  and the choquet integral Scientific subjects are more important than literature;  ({M}) =  ({P}) =0.45;  ({L}) = 0.3 M and P are redundant,  ({M, P}) = 0.5 < 0.45 + 0.45 Students equally good at scientific subjects and literature,  ({L, M}) = 0.9 > 0.45 + 0.3  ({L, P}) = 0.9 > 0.45 + 0.3  ()=0,  ({M, P, L})=1 30

31. Result by applying fuzzy measure: * The initial ratio of weight (3, 3, 2) is kept unchanged 31

32. Complexity of the model Number of coefficients grows exponentially with the number of criteria to be aggregated. 3 approaches (to reduce the number of coefficients) Identification based on semantics Importance of criteria Interaction between criteria Symmetric criteria Veto effects Identification based learning data Minimization of squared error Constraint satisfaction Combining semantics and learning 32

33. Apply 2-additive Choquet integral provide the information about the relationships among criteria (redundancy or support among criteria) and the preference among alternatives Derive fuzzy measures by constraint satisfaction Proposed solution

34. Techniques to explore how the different attributes are related to each other: Experience Based Approach Mathematical Modeling Statistical Technique (Correlation Analysis) measures the strength of a linear relationship among different quality factors The main result of a correlation is called the correlation coefficient (r) Explore relationships

35. Correlation Result

38. Definition of the initial preferences. Convert into Choquet integral form Identify threshold values. If solution exists, calculate the Choquet integral, Shapley value and Interaction indices Implementation of Choquet Integral

39. Define preference thresholds

40. Convert into Choquet integral form

41. Three preference thresholds C, Sh &I have to be determined before the aggregation take part. Range of : 0 to 1 no rule to fix the , we need to compare the solutions obtain with different value of . Once the solution exist, Choquet integral will be calculated Define preference thresholds

42. Calculate the Choquet integral

43. Calculate the Shapley value with Shapley index can be interpreted as a kind of average value of the contribution of element i, individual criteria, alone in all coalitions. Summation of these Shapley values for a given set of elements would represent the importance of the complete set

44. Calculate the Interaction Index With The interaction index Iij can be interpreted as a kind of average value of the added value given by putting i and j together, all coalitions being considered. When Iij is positive (resp. negative), then the interaction is said to be positive (resp. negative).

45. Perform on 3 types of WBA Academic E-commerce Museum Four quality factor were evaluated Usability,Functionality, Reliability, Efficiency Each has different preference, importance and interaction Case Study

46. Result for academic website

48. Threshold C= 1, Sh = 0.1, I =0.1,