Evaluating Software Clustering Algorithms

1. EvaluatingSoftware Clustering Algorithms

2. The problem • How do we know that a particular decomposition of a software system is good? • What does ‘good’ mean anyway? • We can compare against a • Mental model • Benchmark standard • Can be done either manually or automatically COSC6431

3. Manual evaluation • Have experts evaluate automatic decompositions • Very time-consuming, impractical • Also quite subjective • Need an automatic, objective way of doing it COSC6431

4. Automatic evaluation • Usually measures the similarity of an automatic decomposition A to an authoritative decomposition B (prepared manually) • Major drawback: Assumes there exists one ‘correct’ decomposition • Other evaluation approaches are possible, such as measuring the stability of an SCA COSC6431

5. Precision/Recall • Standard Information Retrieval measures • Were applied in a software clustering context by Anquetil • Definitions: • Intra pair: A pair of software entities in the same cluster • Inter pair: A pair of entities in different clusters COSC6431

6. Precision/Recall • Precision: Percentage of intra pairs in A which are also intra in B • Recall: Percentage of intra pairs in Bfound also in A • A good algorithm should exhibit high values in both measures COSC6431

7. 9 4 8 2 1 3 2 9 4 1 5 3 7 5 8 6 7 6 10 10 Precision / Recall example Decomposition A Decomposition B Pair 1-5 : Intra-pair in A but not in B Precision: 16/20 = 80% Recall: 16/21 = 76.2% COSC6431

8. Problems with P/R • Sensitive to the size and number of clusters • Differences are exaggerated if you have small/many clusters • Two values makes comparison harder • The two values are interchangeable if there is no “reference” decomposition COSC6431

9. Koschke – Eisenbarth measure • Loosely based on Precision/Recall • Attempts to be less strict • Definitions: • GOOD match: Two clusters (one in A, one in B) with both precision and recall larger than a threshold p (typical value 70%) • OK match: Two clusters with only one of the measures larger than p COSC6431

10. B2 B4 A5 A1 A1 A2 A4 B1 A3 B5 B3 A5 A4 B2 B3 B5 B1B4 A2A3 Koschke – Eisenbarth metric Good OK COSC6431

11. Koschke – Eisenbarth metric • Does not take edges into account • In extreme situations (each cluster contains only one element) may provide strange results (similarity of 100%) • No penalty for joining clusters COSC6431

12. MoJo distance • The distance between two different partitions of the same software system is defined as the minimum number of Move and Join operations to transform one to the other • Move: Remove an object from a cluster and put it in a different cluster • Join: Merge two clusters into one • Split: Has to be simulated by Move operations COSC6431

13. 1 1 2 2 3 3 4 4 5 5 7 7 6 6 8 8 MoJo example Decomposition A Decomposition B COSC6431

14. Why only Move and Join? • Two clusters can be joined in only one way. One cluster can be split in two in an exponential number of ways • Joining two clusters only means that we performed more detailed clustering than required • Splitting is effectively assigned a weight equal to the cardinality of the smaller of the two resulting clusters COSC6431

15. Computing MoJo distance • Give each object in partition A a tag j if it belongs in cluster Bj in partition B • For each cluster in A, assign it to group Gk if it contains more objects with tag k than any other tag. • To resolve ties, use the maximum bipartite matching method to maximize the number of groups. • Join clusters within same group. Then move all objects with tag k to the clusters in group Gk. COSC6431

16. MoJo Distance Calculation • MoJo Distance is M+J, where M is the number of Moves, and J is the number of Joins • The moves for each cluster is , and the total moves is • Assume the number of non-empty groups is g, from G1 to Gg. The joins of each group is|Gk|-1. Then the number of Joins is COSC6431

17. Outline of Proof • Move & Join operations are commutative • The algorithm uses the minimum number of Moves • Any algorithm that also uses the minimum number of Moves can not beat us • For any algorithm that does not use the minimum number of Moves, there exists another one which is better or equal and uses the minimum number of Moves COSC6431

18. MoJoFM Effectiveness Metric • Assumes the existence of a ‘correct’ authoritative partition • MoJo Distance can be used to measure the distance between two arbitrary partitions COSC6431

19. MeCl and EdgeSim • Other measures do not consider edges • Edges might convey important information • EdgeSim: • Rewards clustering algorithms for preserving the edge types • Penalizes clustering algorithms for changing the edge types • MeCl: • Rewards the clustering algorithm for creating cohesive “subclusters” COSC6431

20. EdgeSim Inter-edge: Edge between clusters Intra-edge: Edge within a cluster Y: set of edges that are of the same type in both A and B COSC6431

21. EdgeSim example In this example, EdgeSim(A,B) = 52.6% COSC6431

22. EdgeSim counterexample COSC6431

23. EdgeMoJo philosophy • A similarity measure cannot make assumptions as to what cluster a particular object should belong to • Dissimilarity between decompositions should increase if the misplacement of an object results in the misplacement of a large number of edges COSC6431

24. EdgeMoJo calculation • Apply MoJo and obtain a series of Move and Join operations • Perform all Join operations • The cost of each Move operations increases from 1 to COSC6431

25. 1 1 2 2 3 4 3 4 7 5 7 5 6 6 8 8 EdgeMoJo example Decomposition A Decomposition B m(5) = 1 + |4-1|/(4+1) = 1.6 COSC6431

26. Real data experiments COSC6431

27. Synthetic data experiments COSC6431