Application Domain Knowledge and Programmers’ Mental Representations

1. Application Domain Knowledge and Programmers’ Mental Representations Teresa M. Shaft University of Oklahoma Iris Vessey Indiana University

2. Application Domain Knowledge • Application Domain refers to the context of the problem to be addressed by the computer software – in contrast to the solution or computing domain (Blum 1989)

3. Application Domain Conceptual Models Implementation Domain Formal Models adapted from Blum, 1989 The Software Development Process

4. Importance of Application Domain Knowledge in Software Development • Those who can map deep application domain knowledge into a computation solution often serve as project gurus • “the thin spread of application domain knowledge” was one of the most salient problems in large scale development • Curtis, Krasner & Iscoe 1988 • Significant effort during software development is associated with acquiring needed application domain knowledge • Walz, Elam & Curtis 1993

5. Application Domain Knowledge • Acknowledged as critical to the ability to develop, comprehend, and maintain software (Brooks, 1990, Pennington & Grabowski 1990, Vans, von Mayrhauser & Somlo 1999) • Infrequently studied (Glass & Vessey 1998). • Most studies of software comprehension or maintenance either do not consider the application domain or use materials the researchers argue do not require application domain knowledge.

6. Application Domain Knowledge in Software Comprehension • Programmers use a more top-down (hypothesis driven) comprehension process when they have relevant application domain knowledge (Shaft & Vessey 1995) • During software comprehension, programmers use application domain knowledge in addition to (rather than to replace) programming knowledge (Shaft & Vessey 1998)

7. Current Study • Examines the influence of application domain knowledge on programmers’ mental representations during the software comprehension and enhancement process • Examines the influence of different types of enhancement tasks on the development of programmers’ mental representations

8. Comprehension Process Enhancement Process Conceptual Model of the Software Comprehension and Enhancement Program Code and Documentation Enhancement Specification Mental Representation of Computer Program Mental Representation of Enhanced Program Program/Programmer Specific Knowledge Base Program/Programmer Specific Knowledge Base Enhanced Computer Program Application Domain Knowledge Programming Domain Knowledge Programmer’s Knowledge Base

9. Programmers’ Mental Representations • Internal knowledge structure of the contents of the computer program • Numerous types of information are embedded in a computer program (Brooks 1987, Pennington 1987, Green 1977, Corritore & Wiedenbeck 1999) • Comprehension is the process of extracting that information from the computer program • Programmers’ mental representations reflect their comprehension of the different types of information embedded in the programs

10. Knowledge Categories • Function • Data flow • Control flow • State • Pennington 1987a, 1987b • These categories are relevant to procedural programs

11. Initial Comprehension of a Computer Program • Driven by the knowledge programmers’ bring to the comprehension task

12. Hypothesis 1: Initial Comprehension • H1a: Programmers have higher levels of initial comprehension when they are familiar with the application domain of the program than when they are unfamiliar with the application domain. • H1b: When programmers are familiar with the application domain, they have better comprehension of function and data flow information than of control flow and state information.

13. Comprehension After Conducting an Enhancement • Influenced by the nature of the enhancement task as well as the knowledge programmers’ possess • Different types of enhancement tasks rarely considered • Consider two types of enhancement tasks: • Function – high need for application domain knowledge • Control flow – low need for application domain knowledge

14. Hypotheses 2: Comprehension After Conducting An Enhancement • H2a: Comprehension after conducting an enhancement will be greater when programmers are familiar with the application domain (than when they are unfamiliar with the application domain). • H2b: Programmers have greater comprehension of the type of knowledge emphasized in the enhancement task than the other types of knowledge. • Control flow task -> control flow knowledge • Function task -> function knowledge

15. Changes in Programmers’ Comprehension • Influenced by programmers’ knowledge • Influenced by the nature of the enhancement task

16. Hypotheses 3: Changes in Comprehension • H3a: Programmers experience greater changes in comprehension when they are familiar with the application domain (than when they are unfamiliar) • H3b: Programmers who conduct a control flow task have greater increases in comprehension than those who conduct a function task

17. Methodology • 24 professionals studied and enhanced two computer programs • Minimum of 2 years professional experience • Computer programs from two application domains • Familiar application domain: accounting • Unfamiliar application domain: hydrology • COBOL programs • Equivalent size, data density, decision density

18. Methodology (continued) • Assessed programmers’ comprehension at two points: • After an initial study period • After conducting an enhancement task • Comprehension assessed via questions • Two questionnaires per program • 20 questions per questionnaire • 5 questions for each knowledge category

19. Methodology (continued) • Half of the programmers conducted control flow enhancements on both programs; half conducted function enhancements • Control flow tasks: insert a new level of control break in an existing control break report • Function tasks: create a new capability • Enhanced programs of equivalent size, data density & decision density

20. Analysis of Hypothesis 1 Source DF SS Mean Square F Value Pr > F Application Domain (AD) 1 14700.00 14700.00 16.74 .001 Error 23 20200.00 878.26 Knowledge Category (KC) 3 8625.00 2875.00 7.61 .01 Error 69 26075.00 377.90 AD * KC 3 466.67 155.56 .44 .73 Error 69 24633.33 357.00

21. Hypothesis 1 – Initial Comprehension • H1a: supported • programmers’ have higher levels of comprehension in the familiar application domain • H1b: not supported • No interaction between knowledge category and application domain familiarity • Programmers’ comprehension of state knowledge was significantly greater than data flow, regardless of knowledge of the application domain

22. 100 90 80 73.33 70.84 69.17 70 Familiar (Accounting) Percent Correct 59.17 Unfamiliar (Hydrology) 60 50 53.33 50.00 40 38.33 30 Function Data Flow Control Flow State Knowledge Category Initial Comprehension by Knowledge Category

23. Analysis of Hypothesis 2 Source DF SS Mean Square F Value Pr > F Type of Enhancement Task (TET) 1 1752.08 1752.08 1.44 .24 Error 22 26845.83 1220.27 Application Domain (AD) 1 10502.08 10502.08 31.21 <.001 AD * TET 1 4680.75 468.75 1.40 .25 Error 22 7389.58 335.42 Knowledge Category (KC) 3 7289.58 2429.86 10.81 <.001 KC * TET 3 522.92 174.31 .78 .51 Error 66 14837.50 224.81 AD * KC 3 5972.92 1990.97 6.65 .001 AD * KC * TET 3 2106.25 702.08 2.34 .08 Error 66 19770.83 299.56

24. Hypothesis 2 – Comprehension After Conducting the Enhancement • H2a: supported • Comprehension in the familiar application domain is greater than in the unfamiliar application domain • H2b: not supported • Type of enhancement task did not influence which knowledge categories are better comprehended • Application domain knowledge influenced comprehension of the different knowledge categories: • Data flow, control flow and state knowledge more accurately understood in the familiar application domain • Within familiar application domain: state knowledge more accurately understood than other categories • Within unfamiliar application domain: state and function more accurately understood than data flow knowledge

26. Analysis of Hypothesis 3 Source DF SS Mean Square F Value Pr > F Type of Enhancement Task (TET) 1 176.04 176.04 0.06 .81 Phase 1 5551.04 5551.04 16.14 <.001 TET * Phase 1 2109.38 2109.38 6.13 .02 Error 22 7564.58 343.84 Application Domain (AD) 1 25026.04 25026.04 38.85 <.001 AD * TET 1 26.04 26.04 0.04 .84 Error 22 14172.92 644.22 Knowledge Category (KC) 3 15586.46 5195.49 15.51 <.001 KC * Task 3 786.46 262.15 0.78 .51 Error 66 22102.08 334.88 Phase * AD 1 176.04 176.04 0.29 .59 Phase*AD*TET 1 651.04 651.04 1.09 .31 Error 22 13197.92 599.91 Phase*KC 3 328.13 109.38 .41 .75 Phase*KC*TET 3 969.79 323.26 1.21 .31 Error 66 17577.08 266.32 ADK * KC 3 3253.13 1084.38 4.05 .01 ADK* KC* TET 3 1553.13 517.71 1.93 .13 Error 66 17668.75 267.31 Phase*ADK*KC 3 3186.46 1062.15 2.65 .06 Phase*ADK*KC*TET 3 844.79 281.60 .71 .55 Error 66 26443.75 400.66

27. Hypothesis 3 – Changes in Comprehension • H3a: not supported, application domain knowledge did not allow programmers to gain more knowledge • H3b: supported, those who conducted a control flow enhancement showed greater gains in comprehension • Application domain influenced the types of knowledge programmers understood: • Higher levels of comprehension of all knowledge categories in the familiar application domain • Within familiar application domain: state knowledge more accurately understood than function and data flow • Within unfamiliar application domain: function, control flow and state knowledge more accurately understood than data flow

30. Summary of Results

31. Discussion • When programmers possess application domain knowledge they have higher levels of comprehension initially and after enhancing a program • Consistent for two types of enhancements • Even in the context of program maintenance, a software development task that is seen as relatively far from the original domain, relevant application domain knowledge benefits software comprehension

32. Discussion • Conducting a control flow enhancement task led to gains in comprehension • Not all types of tasks will give programmers’ the same opportunity to develop their mental representation of a computer program • A more complete taxonomy of tasks would be helpful to future researchers

33. Implications – Types of Enhancement Tasks • Long-term vs. short-term employees • Programmers’ mental representations develop if they engage in tasks that require changes to multiple locations of the program • Programmers who conduct enhancements which do not require interacting with much of the original program, did not experience significant increases in comprehension

34. Implications – Application Domain Knowledge • Lengthy start-up time for new IT employees • Training and selection • Outsourcing and off-shoring • Activities that require less application domain knowledge would seem to be better choices • Develop mechanisms to provide application domain knowledge

35. Conclusions • Not all enhancement tasks encourage development of programmers’ mental representation of a computer program. • Traditionally, software comprehension and enhancement were seen as residing in the programming domain and requiring little, if any, application domain knowledge. • Application domain knowledge is beneficial to software comprehension, and to the understanding of all knowledge categories.