Sundar Gopalakrishnan, Guttorm Sindre, and John Krogstie:

1. Sundar Gopalakrishnan, Guttorm Sindre, and John Krogstie: Adapting UML activity diagrams for mobile work process modelling: Experimental comparison of two notation alternatives PoEM 2010 Delft, 9.-10. Sept.

2. Agenda • Motivation • Background • Research method • Results • Discussion of threats to validity • Summary and further work

3. Traditionalperspectives to modeling • Structural • Behavioral • Functional • Goal and rule-oriented • Object-oriented • Social communication • Actor/role-oriented Whataboutlocation/place/space... ?

4. Zachman Framework

5. Motivation • ’Where’ is increasingly relevant • Outsourcing • SupplyChainManagement/Logistics • Virtualorganization • Mobile applications and information systems • And it is possible to utilize ’where’ to a largerdegree (also real time) to know whereusers, equipment and goodsshould be, are or where at a certain time • Tracking (RFID, UWB, GPS, GSM, WiFi, Ultrasound…) • Internet ofThings (IoT) • This paper: • Presents somedifferentnotation alternatives basedon UML activity diagrams vs. Modeling mobile information systems • Summarizes an analyticalevaluation from a previouspaper (I-ESA’10) • Makes an experimentalcomparisonofthetwo most promisingones

6. Possible notations looked at(I-ESA’10) • Standard UML, using annotation boxes to indicate context or location • Redefining swimlanes to indicate context / location • Using colour for context / location • These three were compared analytically, using a home care case supported by a mobile IS as an example

7. Alt 1: Using annotation boxes • Advantages • Smallest deviation from standard UML AD • Enhanced understandability for those who already know AD • Disadvantages • Greatly increases # nodes in diagram (poor expressive economy) • May be confusing if you also need to use notes for something else in addition to context / location

8. Alt 2: Swimlanes for where • Advantages • Shifting ”who” to stick figures: AD more uniform with UCD • Swimlanes intuitively indicate location • Disadvantages • Many lines from stick figures to activities, poor readability (would be even worse with bigger example) • New usage of swimlanes may confuse those already familiar with AD

9. Alt 3.: using colour • Advantages • No increase in # nodes or lines, better readability • With two process design alternatives beside each other, it is easy to spot differences in location • Disadvantages • Larger deviation from standard UML • Must add legend • Possible challenge for colour blind users

10. Analytical comparison Analytical comparison (I-ESA’10): • Alt 1. (trad. UML with annotations) and Alt 3. (Colours) came out as the two most promising • Proposed further work: Make experimental comparison

11. Experimental design • Compare two diagram alternatives, annotation and colour • Controlled experiment looking at the participants’ • Performance using the notation • Opinion about the notation • Within-subjects design (Latin squares) • Controls better for selection bias • ”Doubles” the N • Measured variables: • Performance: • Understanding: score on 12 True/False questions about a case after reading textual description and seeing diagram • Error_detection: score on identifying 5 deliberately seeded errors in a diagram relative to textual description • Opinion: • Perceived Ease of Use, Perceived Usefulness, and Intention to Use, measured by answers to a TAM-inspired questionnaire w 14 questions

12. Hypotheses • Since the colour notation was best in the analytical comparison, this was hypothesized to have advantages, i.e.: • H1: understanding scores will be better for the colour notation than for the annotated notation • H2: error detection scores will be better for the colour notation than for the annotated notation • H3: participants’ opinion about the colour notation will be more positive than for the annotated notation

13. Experimental tasks • Answering a pre-experiment questionnaire investigating relevant competence • Reading tutorial about first diagram notation (annotatated or colour depending on group) • Reading text and diagram for case (home care or traffic control), answering 12 T/F questions • Answering post-task questionnaire giving opinion about the notation • Repeating steps 2-4 with the opposite case and notation • Repeating 2-6 with the error detection task

14. Latin squares design

15. Results • 46 students participated, randomly assigned into the four Latin squares groups • Clear advantage for colour notation in performance • 3 students performed very poorly in error detection (not doing a serious job); might be considered outliers? • Slight, but not significant advantage for colour when it comes to opinion

16. Conclusion on hypotheses • H1: coloured notation would be better than annotated for understanding (answering T/F questions): CONFIRMED • H2: coloured notation would be better than annotated for detecting errors: CONFIRMED • H3: participants’ opinion about the coloured notation would be more positive: REJECTED

17. Threats to validity • Conclusion validity: significant results but small to moderate effects, should have had larger N to make a strong claim about results • Construct validity: many other ways of understanding a model than answering T/F questions, and many other work tasks than identifying errors. But at least, this is a relevant task, and the ability to answer questions correctly should indicate to some extent whether a model has been understood

18. Threats to validity, cont. • Internal validity: Latin squares design and pre-exp. questionnaire should control very well for any selection bias. Notations were presented in equal detail and style in tutorials, and no preferred or hypothesized outcome was signalled to the students • External validity: The biggest challenge: • Students are not practitioners and motivation may be limited in an experiment which has no impact on their job or a delivered product. But the comparative nature of the experiment should mean that performance with both notations are equally hurt by low competence or motivation. • Small experimental tasks are not representative of the more complex tasks in ”real” mobile IS development.

19. Further work • Also experiment with different pattern fills instead of colour (e.g. better for colour blind users) • More experiments, possibly including practitioners and/or including collaboration among several persons instead of just individuals answering questions • Larger industrial case studies, to try out alternative notations with larger and more realistic work tasks