Starting from Scratch with RC, Python, and Pysaunter

2. Starting from Scratch with RC, Python, and Pysaunter Mary Ann May-Pumphrey Adobe EchoSign QA 02/28/12

3. “From Scratch” • > 1000 manual test cases in TestLink • No Selenium automation • Very little Selenium knowledge amongst the rest of the QA team members • > 20% of my time getting assigned to help execute manual tests

4. Why RC? • Selenium-RC had several years’ worth of bug fixes when EchoSign decision was made in 06/11. Selenium-WebDriver was just about to be released for the first time. • I had experience with RC, but not WebDriver, and producing tests asap was a priority.

5. Why Python? • Selenium-1 supported C#, Java, Perl, PHP, Python, & Ruby. • Selenium-2 quietly dropped Perl & PHP. • Java & Python seem best-suited for long-term survival.

6. Why Python? • Interpreted languages are simpler to write scripts in. • Python is easier for manual testers to learn than Java, and we want our manual testers to do at least some automation. • An IDE is less critical for Python; scripts can be easily written & debugged with a terminal/cmd window.

7. Why Pysaunter? • Priority was on getting tests developed asap, rather than designing our own customized framework. • Pysaunter implements Page Objects.

8. Why Pysaunter? • Pysaunter’s creator--Adam Goucher--is a widely-known and respected Selenium contributor & consultant. • Framework supports both RC and WebDriver tests, and provides examples of both.

9. Page Object Model • A test case comprises business logic, understandable by anyone who reads it. • A page object file comprises locators, assertion methods, action methods, strings, etc., that are specific to a single page (or even part of a single page).

10. Page Object Model • Selenium API calls should only appear in the page object files, not in the test scripts themselves. • If everything is done correctly, in theory, the entire test suite could be switched to something other than Selenium without any modifications to the test scripts. (!?!)

11. pytest Flags Every EchoSign test method has a decorator containing pytest flags, ordered from most general to most specific, e.g., @pytest.marks('regression','search','new_search', 'new_search_for_company_name', 'EA-627')) deftest_new_search_for_company_name(self):

12. Executing the Tests To run just the smoke tests: pysaunter.py –m smoke –v To run all the tests: pysaunter.py –m ‘smoke or regression’ –v

13. Structure of Test Suite conf conftest.py files logs modules pytest.ini scripts support

14. Structure of Test Suite:scripts & modules/pages dirs Each file of test scripts, e.g., scripts/AddToDocument.py, scripts/Upgrade.py, etc. Has a corresponding PO file… modules/pages/AddToDocumentPage.py, modules/pages/UpgradePage.py, etc.

15. Example Test in scripts/Registration.py @pytest.marks('regression','negative','registration','register_without_any_input_values','EA-594') def test_register_without_any_input_values(self): r = RegistrationPage() r.open_default_url() r.do_register_without_input_values() for msg in r.missing_messages.keys(): self.assert_(r.is_missing_message_present(msg),"No or wrong error message present for " + msg)

16. Relevant PO Code: modules/pages/RegistrationPage.py missing_messages = { "missing_email": "Please enter your email address", ... "missing_country": "Please specify your country", } def do_register_without_input_values(self): self.se.click(locators['submit_button']) def is_missing_message_present(self,key): return self.se.is_text_present(self.missing_messages[key])

17. Structure of Test Suite:modules/*py • Non-PO files • imap.py, retrieve.py, extract.py all deal with fetching email • helpers.py contains very general functions

18. Structure of Test Suite:files dir • EchoSign-specific, not part of Py.saunter • Contains various files to be uploaded, archived, added to library, etc. by the test cases.

19. Structure of Test Suite:logs dir • latest.xml + dated .xml files contain execution results • One dir/test with snapshots taken during run • Demo

20. Integration with TestLink • TestLink Keyword “manual” vs. “automated” allows manual testers to run queries to retrieve just “manual” tests for regression. • Final pytest “mark” is always the TestLink id of corresponding manual test, i.e., @pytest.marks('smoke','registration', 'register_free_account','EA-583')

21. Integration with Jenkins • CI test runs on our internal build system are in our near-term plans. • In the meantime, two artifacts: • echosign_selenium.tar.gz makes it very easy to install the latest checked-in version of test suite—no P4 client needed. • list.html shows entire list of auomated tests to date.

22. Integration with FogBugz Any bug found by Selenium automation has two features: • Name of automated test somewhere in the bug report • automation tag for metrics purposes

23. Best Practices:Coding Standards • Adhering to very basic items from the Style Guide for Python Code: • Function & variable names: lowercase, underscores_for_readability • Class names should be CapitalizeEachWord • Indentation should be 4 spaces per level—no tabs

24. Best Practices: skipif() • skipif() decorators, for tests that only work in certain environments @pytest.mark.skipif('helpers.get_env() == "preview"') @pytest.mark.skipif('helpers.get_env() != "preview"')

25. Best Practices: skip() • skip() decorators, for tests that aren’t quite finished, or have been found to have a problem @pytest.mark.skip()

26. Best Practices: xfail() • xfail() decorators, for tests that are expected to fail @pytest.mark.xfail(reason="FogBugz 13864")

27. Best Practices: Error Handling • Provide an error message for every assertion in case it fails. • Put all these error messages in a dictionary at top of class. • Use self-documenting dictionary keys.

28. Best Practices: Error Handling error_strings = { "is_archived_count_not": "The Archived count is not ", . . . "manage_page_not_displaying_only_archived_folder" : "The Manage page is not displaying the Archived folder OR is showing more folders also.”,

29. Best Practices: Error-Handling self.assert_ ( m.is_only_folder_displayed("widget"), self.error_strings['manage_page_not_displaying_only_widget_folder'] )

30. Best Practices: Code Reviews • Code reviews of test code required for check-in

31. Best Practices: Testing the Test • Temporarily put a pdb.set_trace() just before the assertion in a new/passing test. • Run the test. • When the pdb prompt appears, delete a file or an email or do <whatever> will best simulate a failure situation. • Ensure that the test fails.

32. Best Practices: Important Tests vs. More Tests • Try to balance development of smoke tests (important but hard) with development of regression tests (less important but easier).

33. What’s Worked Well • Pysaunter creator Adam Goucher has been very supportive, both in terms of bug fixes & feature requests. • The Python RC API has presented virtually no issues.

34. What’s Worked Well • Pysaunter’s image-capturing feature is very useful. Much faster than video and usually just as effective. • Pysaunter is “Just Enough Framework” for our current needs. • Non-Selenium colleagues have been able to add new tests soon after getting started.

35. Concerns • Lack of a Pysaunter community => no peer support or best-practice discussion • Pysaunter’s future unclear • RC’s longevity unclear, as is LOE for migration to WebDriver • Selenium/Python job-seekers seem fewer in number than Selenium/Java job-seekers

36. Resources • Instructions on how to install RC/Python/Pysaunter from my blog • PushToTest-Sponsored Webinar: “Selenium Python Page Objects” • The Quick Python Book, 2nd ed.

37. Resources • PushToTest-Sponsored Webinar: “Selenium--You Are Doing It Wrong” • Python Module of the Week • The Element-34 Blog

38. Q&A? Now! or… maryann@adobe.com