Ana Paiva & João Pascoal Faria {apaiva,jpf}@fe.up.pt Grupo de Engenharia de Software

1. Temas para doutoramento: Projecto Amber-iTest (An Automated Model-Based User Interface Testing Environment) Ana Paiva & João Pascoal Faria {apaiva,jpf}@fe.up.pt Grupo de Engenharia de Software www.fe.up.pt/~softeng Departamento de Engenharia Informática www.fe.up.pt/si/unidades_geral.visualizar?p_unidade=151 Faculdade de Engenharia da Universidade do Porto www.fe.up.pt

2. Research interests • Software Quality – our society depends ever more on software, so its crucial to assure its quality through various means • Product quality – testing and certification of software products (the customer perspective) • particularly model-based user interface testing • partnerships: MSR/FSE, etc. • Process quality– improving the software development process to assure the production of high-quality software in a predictable and cost-effective manner (the supplier perspective) • particularly the combination of CMMI/PSP/TSP/Agile methods for fast process improvement • partnerships: SEI, etc. • Formal methods – mathematically based software specification and verification methods • particularly lightweight and automated methods integrated with commonly used development environments • partnerships: UM, etc.

3. Model-based User Interface Testing • UI testing is a fundamental but currently very time consuming and poorly systematized V&V activity • Microsoft: number of testers ≈ number of developers • Model-based testing can help to systematize and automate the UI testing process (through automatic test case generation from formal models), but poses specific challenges: • Test case explosion problem: how to generate a test suite of manageable size that still assures adequate UI testing? • How to bridge the gap between the model and the implementation? • How to model GUIs? • How to reduce the effort in the construction of the model?

4. An example of a MBUIT environment • Model-based testing tool: Spec Explorer (FSE/MSR) • Automates test case generation and execution for API conformance testing • Test sequences are derived from a FSM which is generated by bounded exploration of the model • Formal modelling language: Spec# (FSE/MSR) • Extends C# with contracts and high-level constructs • Can be used as modelling or programming language • GUI testing extensions (from previous work at FEUP) • GUI modelling techniques, promoting abstraction and reuse • FSM reduction tool for hierarchical GUIs, to handle the state/test case explosion problem • GUI mapping tool, to bridge the gap between the model and its implementation

5. GUI modelling with Spec#: example • class FindDialog { • string findWhat = ""; • bool matchCase = false; • UpDown direction = Down; • [Action] void SetFindWhat(string value) • requires IsEnabled("FindDialog"); • { • findWhat = value; • } • [Action] void FindNext() • requires IsEnabled("FindDialog") && findWhat != ""; • { • Notepad.Find(findWhat, direction, matchCase); • } • [Action] void Cancel() • requires IsEnabled(“FindDialog”) • { • WindowManager.RemoveWindow("FindDialog"); • } • … • } (imaginar em inglês) Janela -> classe Botão ou opção menu -> acção Controlo com estado -> var. estado + acções

6. Current issues • Time and effort required to build the GUI models • Reluctance of GUI testers and modellers in writing textual formal specifications (they strongly prefer graphical notations with which they are familiar) • UML models better accepted by testers, but the detail and rigour of UML models is insufficient for being used as test oracles • Effort required to configure the test generation process to guarantee the quality of the test cases generated. • There is the need for coverage criteria best adapted for GUI testing, such as coverage of navigation maps • With Spec Explorer, the bounded exploration process is based on parameter domain values provided by the tester, but there is no feedback mechanism to evaluate the quality of the test cases obtained based on appropriate test adequacy criteria for GUIs

7. AMBER-iTest project Goals • Develop a set of tools and techniques to automate specification based GUI testing, solving the shortcomings found in our previous work, so that it can be used in industrial environments. • Industrial partner: Critical Software • Duration: January 2008 – December 2010

8. Mapear acções do modelo Informação de mapeamento Amber-iTest overview 2. Ferramenta para mapear acções do modelo com objectos de interacção da GUI 3. Engenharia Reversa Modelação Modelo do SW SW a testar Requisitos ≈30% linhas de código 1. Modelação visual e teste Geração automática de casos de teste 4. Definição de critérios de cobertura para teste de GUIs Casos de teste (abstractos) Execução automática dos testes Lista de erros • Microsoft Research: • Spec Explorer • Spec#/AsmL NModel 5. Validação industrial da abordagem

9. Theme/Task 1. Development of a visual GUI modelling front-end • study different graphical notations used to model GUIs, and adapt (extend or combine) one notation or a combination of notations according to the requirements already described; • since it is not a goal of the project to develop a new visual editor to support the edition of the graphical notation, a freely available visual editor will be selected and adapted to support the graphical view of the GUI model; • definition of a set of rules for the translation between the graphical notation and the formal textual notation; • construction of a tool for the automatic translation and round-trip engineering between the two notations. • Preliminary results: Paper MBT’07

10. m()pre post {body} m()pre post Test coverage analysis Test case generation Model to implementation mapping oracle new Coverage information GUI mapping code & data (adapter) Spec Explorer Test Execution GUI Application Under Test Spec Explorer Test results Test suite (abstract) Visual GUI modelling front-end overview Visual to formal model translation Visual modelling Refinement new UML visual model Spec# formal model (explicit spec) Spec# formal model (implicit spec)

11. Theme/Task 2. Development (generalization) of a model to implementation mapping tool • support the majority of the Windows controls used in Windows applications; • support Java SWT applications; • support Web applications (HTML, DHTML, etc.); • support a one-to-many mapping between logical model actions and physical GUI actions, so that a model action can correspond to a sequence of physical actions; the sequence of physical actions will be captured in way similar to the recording mechanism of capture-replay testing tools. • Previous work: Paper ICFEM’05

12. Theme/Task 3. Development of a GUI reverse engineering tool • In a first phase, discover the navigation map of the application and represent it by a state machine. • In a second phase, the goal will be to discover state variables within each window or page and effects of user actions on state variables. • In a third phase, it will be introduced the ability to mix automatic exploration with manual exploration, in order to be able to access parts of the application functionality that are protected by a key or that depend on specific input data (this feature represents an improvement to the state-of-art). • The reverse engineering tool will generate, at the same time, the mapping information between the model and the implementation that is needed for test execution (see task 2). • Preliminary results: Paper FMICS’07

13. Reverse engineering example // Main window [Action] void AddressBook_NewContact() requires IsEnabled(“AddressBook”); ensures IsEnabled(“Contact”); { AddWindow(“Contact”); } //Contact dialog [Action]void SetLastName(string str) requires IsEnabled(“Contact”); { LastName = str; } //… // Find dialog [Action]void SetFindWhat(string str) requires IsEnabled(“Find”); ensures FindButton.IsEnabled(); { FindButton.Enabled = true; } // RECORD MODE [Action(Kind=ActionAttributeKind.Scenario)]) void PasswordScenario() requires IsEnabled(“Password"); { TextBox = “mypassword"; Ok(); }

14. Theme/Task 4. Development of GUI test coverage analysis and enforcement tools • identification of test coverage criteria that are best suited for model-based GUI testing, and that can be used to evaluate the quality and guide the generation of test cases; • techniques and tools to determine the degree of coverage reached by a given test suite, according to the criteria previously identified; • techniques and tools to control and guide the generation of test cases in order to meet specified test coverage goals: • stop the generation of test cases (or the intermediate FSM) as soon as the coverage goals are met; • remove redundant test cases (or redundant parts of the FSM) with respect to the coverage goals specified, extending our previous work presented at ASM´05; • automatically generate domain values that allow achieving the coverage goals specified (e.g., using constraint satisfaction algorithms); • Previous work: paper ASM’05

15. Amber-iTest - Vantagens • Projecto financiado pela FCT • 2 ofertas de bolsa ainda em aberto, duração 2 anos • Já existe trabalho feito em muitas das tarefas • Envolvimento com a indústria • Projecto na área da qualidade de sw (área em expansão no mundo empresarial) • Interacção/integração com/no Grupo de Engenharia de Software (investigação e ensino) • Obrigada pela vossa atenção! • Mais informação: http://paginas.fe.up.pt/~softeng/wiki/doku.php?id=projects:amber_itest:start

16. References and further information • “Reverse Engineered Formal Models for GUI Testing”. Ana C. R. Paiva, João C. P. Faria, Pedro M. C. Mendes. 12th International Workshop on Formal Methods for Industrial Critical Systems (FMICS 2007). July 1-2, 2007, LNCS vol. 4916, Berlin, Germany. • "Towards the Integration of Visual and Formal Methods for GUI Testing" Ana C. R. Paiva, João C. P. Faria, Raul F. A. M. Vidal. MBT'06 - Third Workshop on Model Based Testing, 31 March - 1 April, 2007, Braga, Portugal • "A Model-to-implementation Mapping Tool for Automated Model-based GUI Testing". Ana C. R. Paiva, João C. P. Faria, Nikolai Tillmann, Raul F. A. M. Vidal.ICFEM 2005 - Seventh International Conference on Formal Engineering Methods, 1-4 November 2005, Manchester, UK. • "Modeling and Testing Hierarchical GUIs". Ana C. R. Paiva, João C. P. Faria, Raul F. A. Moreira, Nikolai Tillmann. ASM 2005 - 12th International Workshop on Abstract State Machines, March 8-11, 2005, Paris, France. • Automated Specification-Based Testing of Graphical User Interfaces, PhD Thesis, Ana Cristina Paiva Pimenta, FEUP, November 2006 • AMBER iTest Project: http://paginas.fe.up.pt/~softeng/wiki/doku.php?id=projects:amber_itest:start