The Value of USAP in Software Architecture Design

1. The Value of USAP in Software Architecture Design Presentation by: David Grizzanti

2. Introduction • Usability is an important quality attribute of interactive systems. • Since the 1980’s, usability has been treated as a subset of user interface functionality. • Since usability is not considered early in the design process, problems found in user testing are very costly.

3. Introduction • The authors research between usability and software architecture has lead to the development of Usability-Supporting Architectural Patterns (USAPs). • Each of these address a usability concern not addressed by separation alone.

4. Introduction • Each USAP consists of: • An architecturally sensitive usability scenario. • A list of general responsibilities derived from forces inherent in the task and environment, human capabilities and desires, and software state. • A sample solution implemented in a larger separation-based design pattern.

5. Introduction • In their paper, the authors described a controlled experiment design to assess the value of using USAP components in modifying a software architecture design. • Participants were asked to redesign an existing architecture to allow for users to cancel a long-running command. • This experiment measured whether the architectural solutions produced using all the USAP components was more beneficial than using only certain subsets.

6. Experiment - Participants • 18 computer science graduate students at Carnegie Mellon participated in the experiment. • The 15 male and 3 female participants ranged in age from 23 to 30, and all had completed work for a master’s degree. • Participants also reported spending an average of 22.9 hours per work programming.

7. Experiment Design & Materials • Participants were randomly assigned to one of 3 teams. • Participants in each group received a different version of a “Training Document.” • All received the same architecture redesign task.

8. Groups • Group 1 received: • A usability scenario describing user circumstances. • Group 2 received: • The usability scenario • A list of responsibilities to consider with a cancel command. • Group 3 received: • The usability scenario • List of responsibilities • Sample solution for adding cancellation function.

9. Experiment • The task given to participants was to redesign an existing architecture that did not support cancellation, to support cancellation. • Chose the Plug-in Architecture for Mobile Devices (PAMD), developed by students at Carnegie Mellon. • Developed for the Palm OS4.

10. Experiment • Task Instructions included 7 elements: • General description of PAMD architecture • Example PAMD scenario • List of responsibilities for PAMD operations • List of Component Interaction Steps detailing PAMD run-time operation • A Component Interaction Diagram (Fig. 1) • A Sequence Diagram of PAMD of run-time component interaction (Fig. 2) • Final page instructed participant to add the ability to cancel a plug-in to the PAMD architecture design.

11. Fig. 1

12. Fig. 2

13. Experiment • Answer Paper was designed so participants could easily and efficiently record the information relevant to their design. • Component diagrams were identical to those provided to the Task Instructions, except that run-time steps and assignments were removed. • Participants were instructed to use the diagrams as a bases for their designs.

14. Procedure • Participants were randomly assigned to 3 groups and given unlimited time to complete their task. • They were told that they were participating in a study about fixing one kind of usability problem in a specific software architecture design. • Participants were then given the Training Document and after reading it, received the task instructions.

15. Results • The time to complete the redesign task ranged from 39 to 138 minutes. • Participants given only the USAP scenario considered 2-4 cancellation responsibilities. • Those who received both the USAP and list of 19 responsibilities considered 4-15. • Participants who were given all 3, considered 5-15 cancellation responsibilities.

16. Results - Overview • Average for 3 groups: • 1st group considered an average of 3.17 • 2nd group considered an average of 7.7 • 3rd group considered an average of 9.5 • An analysis of variance between groups showed that the time on task had no significant effect on the number of cancellation responsibilities considered.

17. Discussion • Experiment showed a strong main effect of providing the full USAP on the number of responsibilities considered. • Participants who received only the cancellation scenario considered, on average, a third of the responsibilities of the participants who received the full USAP. • This indicates that the full USAP helped the participants in remembering which responsibilities to consider.

18. Discussion • The absence of a correlation between the time on task and performance showed no speed/accuracy tradeoff. • However, administering the full USAP allowed participants to create a more complete solution in nearly the same amount of time. • Though the group was small, factors such as self-reported experience or familiarity with technology did not interact with experiment conditions.

19. Discussion • For 15 of the 19 Cancellation Responsibilities (CRs), Chi-squared tests revealed no significant difference in training conditions, however the following was observed: • CR1, CR5, and CR10 were considered by the majority of participants. • CR16 was not considered by a single participant. • CR4, CR13, CR17, and CR18 were considered very differently between those who received the cancellation scenario only and those who received the full USAP.

20. Conclusions • Using a full USAP increased the number of responsibilities considered. • Participants who used all 3 parts of the USAP were able to identify 3 times as many cancellation responsibilities, in the same amount of time and without having more work experience or formal training prior to the task.

21. Conclusions • Assumption in SE world that programmers know how to implement whatever requirements are given to them. • Mostly true, however, not the case for usability. • Usability issues have seen to comprise more than 60% of requirements-related defects in some professional software projects. • Usability heuristics for software design are not obvious and should be made more explicit to software designers as well as SE students.