OCAP-RI Vendor Forum

1. OCAP-RI Vendor Forum August 10, 2011 Presenters: NDS and CableLabs Dev Teams Location: CableLabs Summer Conference – Keystone, CO

2. Agenda Introductions Ground Rules Release Roadmap NDS Topics CableLabs Topics Miscellaneous Topics

3. Production / Debug Builds CableLabs occasionally tests releases in both debug and “release” build configurations In “release”, compiler optimizations are enabled and much logging is disabled Changes in timing can exponse thread synchronization bugs in the stack Harder to debug

4. Extension-based Builds Stack can be configured with OCAP base and any combination of OCAP extensions DVR, HomeNetworking, FrontPanel, etc Application developers must be careful to structure their apps so as not to trigger accidental class loading Class file verification can trip you up JVM spec (http://java.sun.com/docs/books/jvms/second_edition/html/ClassFile.doc.html)

5. Extension-based app example public class App { public void doSomething() { MyInterface myIf; if (System.getProperty(“ocap.api.option.dvr”) != null) { myIf = new MyIntefaceDVRImpl(); } else { myIf = new MyIntefaceBaseImpl(); } } }

6. Testing Overview QA - Bug tracking Trunk and 1.1.4 Branch Unit Testing CTP attended/unattended, CTP smoke, atelite, DLNA & UPnP CTT, JUnit Integration Testing Integration Xlets, AutoXlet, CATS TDK MSO Guides – ODN, Trio, Barcelona QA: TestLink We want your tests - Appeal for your tests - Please

7. Bug Tracking Private Database - OCORI A mix of publicly visible issues, and some are not To protect confidential information, exposed case by case Mirrors the issues in the private data Public database - IT For these issues, all information in the private databases is also in the public database

8. Trunk and Branch Testing Schedule Trunk and 1.1.4 vendor branch Daily Guide smoke tests (trunk) Weekly Guide smoke tests (1.1.4 branch) Weekly CTP test runs Full regression run prior to releases Internal Branch Testing CTP tests are run prior to merging back into the trunk

9. CTP Unattended Tests • Test Setup • 5600+ unattended CTP tests. • 1 ATE dedicated to testing the RI. • Tests run against Windows and Linux platforms. • Run against all test suites requires 3 days. • Test Schedule • Test all suites against the trunk (frequency: once per week). • Test all suites against the 1.1.4 branch (frequency: once per week). • Test all suites or a subset of tests against developers branches before merge to trunk (frequency: as needed). • Test subset of suites against separable builds: Base, DVR only, HN+DVR, HN only (frequency: when ATE available).

10. Attended CTP Tests 400+ CTP tests require operator interaction Attended CTP tests can be roughly divided into two groups: Tests which demonstrate interactivity. Tests which require visual inspections. Automated attended CTP testing addresses the tests which require only visual inspection. Test run is automated. Screenshot and verification question are captured and saved for each test. After test run completes, operator analyzes saved screenshots and verification questions. Goal of automated attended CTP testing Increase scope of tests run for each release cycle while using QA resources more efficiently.

11. Automated Attended CTP Testing utilizes programs developed by the RI team to capture test results, incorporate results into a spreadsheet and pull aggregated results in to a GUI for analysis by QA.

12. DLNA CTT v 1.5.57.03 Unit tests which check the conformance of the Ri against the DLNA specification Run the DMS (server) tests. Player tests are restricted by PC features supported by the RI. 267 unit level tests are run.

13. UPnP CTT 2.0.18.5079 Unit tests which check the conformance of the Ri against the UPnP specifications 110 unit level tests are run.

14. JUnit Tests Modified version of the JUnit test runner Tests are available as part of the distribution Executed by development engineers as part of their workflow.

15. Integration Tests $OCAPROOT/apps/qa/org/cablelabs/xlets Each test application should have a readme.txt in its directory. Currently undergoing clean-up and rework Latest status is found in $OCAPROOT/apps/qa/docs/IntegrationTestInfo.doc New test applications will be added going forward for integration testing of new features.

16. List of tests run with every RI release:

17. Integration Test Automation Autoxlet – a framework that automates xlet launching and dispatch of remote control events. XletDriver – AUTOSTART application that loads xlets and sends remote control events based on the XML script file. To automate a xlet, the xlet should implement the Driveable interface. DispatchEvent(…) provides a method for receiving and interpreting key events. Uses IXC communication for logging and result reporting. Deterministic xlet results are provided using JUnit-like asserts. $OCAPROOT/apps/qa/Autoxlet/

18. MSO Guide Tests Guides ODN Barcelona Trio Run a subset of the tests used by the MSOs for their own testing

20. Home Networking InterOp Testing List scenarios run at the May Interop Discover and playback RI recordings on your STB RI discover and playback recordings from your STB Trick modes Delete recordings

21. TDK and CATS Will run the same tests as NDS talked about earlier At this moment, still working through installation issues Our goal is to run all the TDK tests prior to each RI release

22. TestLink Web based test case management tool Archive of test results Repository for test procedures

23. TestLink Results Entry

24. Testing Overview - Summary QA - Bug tracking Trunk and 1.1.4 Branch Unit Testing CTP attended/unattended, CTP smoke, atelite, DLNA & UPnP CTT, JUnit Integration Testing Integration Xlets, AutoXlet, CATS TDK MSO Guides – ODN, Trio, Barcelona QA: TestLink We want your tests - Appeal for your tests - Please

25. Questions and Suggestions RI QA is a small team with limited resources Do you have tests to contribute? Do you have suggestions for new tests?