Measuring & Managing Performance in Software Projects: Dysfunction and Models

1. Measuring & Managing Performance EECS811 : SW Project Management Presenter : Amit Dandekar Instructor : Dr. Hossein Saiedian

2. Contents • Introduction • Measurement dysfunction defined • Measurement dysfunction model • Designing measurement system • Designing incentive system • Management by delegation • Popular measurement models & dysfunction • Conclusion

3. Introduction • Measurement: The Good • always a constructive practice • no downside • requires little or no justification • popular belief • Measurement: The Bad • most things extremely difficult to measure • enormous cost of measurement mechanisms • can inhibit productivity • unconstructive uses possible

4. Introduction • Measurement: The ugly • dysfunction !!! • measurements unknowingly put to unconstructive use • measurements do more harm than good • distorted incentive system

5. Measurement dysfunction defined • Measurement way it is suppose to be • measure <parameter> in hopes of improving <goal> • parameter gets better as goal comes near • Measurement dysfunction • parameter gets better even if goal stays as distant • this becomes a rule and not just an anomaly • measured standards actually interfere with achievement of intended goals

6. Measurement dysfunction defined Measurement Indicators Level of performance True performance Time

7. Characteristics of measurement dysfunction • Key measure do not provide clear account of performance • Substitute characteristics used as indicators • measurement of timeliness of delivery as project success • Assuming bi-directional nature of correlation between indicator and performance • measurement standard based on physical appearance • Measured standards interfere with achievement of intended effect • Employee rewards not associated with performance of organization

8. Measurement dysfunction model • 3 roles • principal • agent • customer • Principal • typically a manager • motivated by profits

9. Measurement dysfunction model • Agent • typically an employee • motivated by fondness for money • dislike for work • possibly by internal motivation • Customer • purchaser of services generated • final judge of the value of efforts

10. Measurement dysfunction model • Principal cannot determine value of agents production directly • must measure proxies of value • Agent & principal have opposite interests • principle wants max value for min money • agent wants max money for min works

11. Measurement dysfunction model • Apply this model to a typical scenario • DOD contracted a firm to develop software • DOD expects thorough unit testing to ensure quality • roles defined per model • principal: manager from the firm • agent: development engineer at the engineer • customer: DOD • no of lines of code written is measurable • code coverage achieved via unit tests is not easily measurable

12. Measurement dysfunction model • What DOD wants ? • more effort on coding is better • more effort on testing is better • coding and testing , both activities are critical

13. Measurement dysfunction model • Ways engineer can allocate his effort capacity • DOD’s most preferred allocation

14. Measurement dysfunction model • Effect of increasing engineers effort capacity • DOD’s preference: best mix path

15. Measurement dysfunction model • Possible ways in which engineer can achieve increased target level set by manager

16. Measurement dysfunction model • Can engineer meet higher productivity targets by not achieving complete code coverage in unit tests? • how closely engineers activities are being monitored ? • can manager measure code coverage achieved by unit tests ? • In most cases measuring engineer’s activities fully is not possible • too costly or impossible.

17. Measurement dysfunction model • Factors that affect measurement costs • repetitiveness of the task • offer more opportunities for observing all phases of agent’s work • permit use of statistical sampling techniques • inexpensive measurement enabling full supervision • complexity of task • overtaxes principal’s cognitive capabilities • simple tasks easier to become fully supervised

18. Measurement dysfunction model • Factors that affect measurement costs • newness of task • principal more likely to be familiar with older tasks • older tasks provide more measurement opportunity over time • more likely to become fully supervised • specialized knowledge requirement • difficult/expensive to find someone qualified to do measurement & interpret results • typically include tasks that are pushing technology beyond boundary • full supervision unlikely to be achieved

19. Measurement dysfunction model • Factors that affect measurement costs • interdependence and in-separable effort • tasks of numerous workers tightly coupled • all have ability to hinder others tasks • impossible to distinguish how much effort each member is contributing • full supervision unlikely • environmental covariates & noise • outside factors affect agent’s performance • external random factors for which no measure available affect performance • full supervision unlikely

20. Measurement dysfunction model • Measurement costs dictate supervision model • Three ways of supervision based on measurement costs • no supervision • full supervision • partial supervision

21. Measurement dysfunction model • No supervision • nothing observed on effort allocation • impossible to measure / measurement cost too high • agent does usefulworkdue to internal motivationonly • payment not contingent upon performance • effort distribution along best path • typically short of customer expectation • because agent dislikes work

22. Measurement dysfunction model • No supervision

23. Measurement dysfunction model • Full supervision • principal can measure every critical dimension of performance • principal sets target at ideal point on best mix path • agent will meet target exactly • incentive distortion unlikely • effort increase can be forced to be exactly on best mix path • ideal but very uncommon in real life • finding and measuring all critical dimensions not easy • measure infeasible / too expensive

24. Measurement dysfunction model • Partial supervision • most likely real world scenario • some but not all critical dimensions measured • Dysfunction likely • incentive system likely to induce distortion in effort mix • value to customer reaches a point where its same /lower than unsupervised value • thus agent is given incentive to deliver less values to customer • dysfunction !!!

25. Measurement dysfunction model • Partial supervision

26. Measurement dysfunction model • Applying model to our case study • measuring code coverage achieved by unit tests is too expensive • only partial supervision possible • incentive provided to engineer to write more code • engineer changes effort allocation changed to write more code and less tests • engineer can easily slip under radar without writing adequate unit tests • value to customer reaches a point where its same /lower than value without additional incentive • engineer given incentive to deliver less values to customer • dysfunction !!!

27. Designing measurement system • Comprehensiveness is desirable • must measure all critical dimensions • essential to prevent dysfunction • difficult to verify performance of knowledge workers • quality is notoriously difficult to measure • Motivational uses of measurement • agent can not be prevented from reacting to measure • results in competitive dynamics that distorts measure • Often causes warning signs to be concealed • extremely difficult to achieve informational only use

28. Designing measurement system • Use measurements for information only • convince agents that measure is informational only • promote people longer time • no sudden rising stars • reduces competition within employees • aggregation of measurement • individual see measures at their or higher level • need procedural methods to enforce aggregation • gives manager a self assessment • managerial job shifts from evaluation to inspiration and assistance to subordinates

29. Designing incentive system • Identify mode of supervision possible • dictated by structure of measurement costs

30. Designing incentive system • Common mistake • assuming full supervision and setting targets too far out • Most experts agree that comprehensive measurement impossible • Most of the times only partial supervision possible • How to achieve better than moderate performance ? • internal motivation & management by delegation

31. Introducing internal motivation

32. Management by delegation • Management by delegation • management style that relies on internal motivation • agents preferences changed as compared to management by measurement • unsupervised effort allocation • always moves along best mix path • dysfunction can not occur • in conflict with measurement based management • principal must choose between the two

33. Management by delegation • Management by delegation involves costs • cost benefit analysis needs to be done • Delegation costs • delegation involves changes in weighing of agents utility components • needs taking time out of production activities to promote group identity • extending lifetime employment

34. Management by delegation • Factors that affect delegation costs • organizational size • large organizations are less personal than small due to sheer number of employees and size • difficult to inspire organizational identification and loyalty • public or private organizations • employees of public bureaucracies less motivated • higher delegation costs in public organizations • cultural homogeneity of organization • organizations with large cultural diversity have higher delegation costs • difficult to align objectives of firm with employees

35. Management by delegation • Factors that affect delegation costs • duration of relationship between principal and agent • longer relationship will develop long term relationships • principal and agents objectives can get closely linked • frequency of interaction between employees • frequent interactions within employees help organizational identification • delegation costs lower in geographically less dispersed organizations • organizational prestige • organizational identification easy to cultivate with firms that are widely known

36. Management by delegation • Factors that affect delegation costs • extent to which employee needs are met • non-material needs for companionship, quality of life • employee satisfaction with work environment lower delegation costs • Perceived level of mutual commitment • easier to cultivate loyalty if organizations takes responsibility for employee well being • Japanese practice of assuring lifetime employment greatly increases loyalty

37. Management by delegation • Recommended management models

38. Management by delegation • What to do when nothing works ? • no supervision possible • delegation costs too high • Convert the situation to manageable one • make delegation appropriate • make measurement appropriate

39. Management by delegation • How to make jobs more measurable ? • standardization • almost all processes repetitive at some level of abstraction • identify phases and establish standard methods of execution • decide appropriate product properties and agent behaviour at a certain stage • e.g. 4 phases of RUP

40. Management by delegation • How to make jobs more measurable ? • specification • construct detailed model of the process • standardise every step in a process • variances from specification can be noted at every step • subdivision • decompose jobs into subtasks and group similar tasks • grouping makes repetition & self similarity visible making measurement easier • people doing similar activities can be assigned overseers that have specialized knowledge as workers

41. Management by delegation • How to make situation suitable to delegation? • promoting organizational intimacy • make hierarchy flatter & work groups smaller • reconfigure work spaces to promote casual contatcs between managers and workers in same team • promoting trust • facilitate mutual trust between employer and employees • introducing mutual vulnerabilities into relationship • investments that suggest permanence of employee base such as investing heavily in training

42. Management by delegation • How to make situation suitable to delegation? • better leadership • skilful expression of vision • management styles that emphasize inspiring and trusting employees rather than analyzing and coercing

43. Popular measurement models & dysfunction • Popular measurement systems • Malcom Baldrige quality award • ISO 9000 certification • SEI SCE • Goal Question Metrics (GQM) • Malcom Baldrige quality award • NIST designed award scoring system • 7 category, 1000 point, 3 level judging • 75% points allocated to process criteria • Only 25% to actual product quality • Site visits by experts and interviews by experts

44. Popular measurement models & dysfunction • ISO 9000 • five quality standards set by ISO • ISO establishes requirements for production systems • not product standard • documents all processes comprehensively • document what you do and do what you document • ISO certification required to maintain market access

45. Popular measurement models & dysfunction • SEI SCE • Determine organization’s ability to deliver particular product • on time • on budget • With high quality • uses standard questionnaires & interviews • standard scoring algorithm used • Limited empirical data showing relation between maturity levels and quality of product

46. Popular measurement models & dysfunction • GQM (Goal Question Metric) • fashionable in software metrics circles • summary of six-step GQM process • develop set of business and associated measurement goals • generate questions to define those goals as completely as possible • associate set of metrics with questions to answer questions in a measurable way • advocates measuring high profile areas associated with organizational goals • not concerned with making measures comprehensive

47. Popular measurement models & dysfunction • Similarities among these popular models • audit from external examiner • measure process characteristics than product • standard and publicly known scoring algorithm • each firm has tremendous incentive to achieve favourable result • each employ a measurement based control system

48. Popular measurement models & dysfunction • Problems with the popular models • cover only certain critical dimensions • likely to lead to dysfunction • rely on point in time measurement • firms learn how to exploit program measures • firms ramp up to corrupt one time measure • examiners often turn consultants • conflict of interest • costs do not necessarily outweigh benefits • each of these popular systems can lead to measurement dysfunction !!!

49. Conclusion • Measurement dysfunction always possible • comprehensive measurement extremely difficult • Never use measurement to monitor individual performance • individuals always react to it • Internal motivation is a viable mean to achieve satisfactory effort levels • external motivation works only in limited environments

50. References • R. D. Austin, “Measuring and Managing Performance in Organizations”, Dorset House Publishing Company, 1996.