WG5 P02 Proposal 2014 Qualification of Standard Scripts

1. WG5 P02 Proposal 2014Qualification of Standard Scripts

2. Proposal through CSS 2014http://www.phusewiki.org/wiki/index.php?title=File:FDA_Scrips.ppt • Anyone should be able to submit a script, according to a check list • Categorize scripts according to complexity • Complexity: low, medium, high, software • Output: tabulated data, analysis data, table, figure, listing • Metadata for script should indicate • Type of output: tabulated data, analysis data, table, figure, listing • Study design: parallel, crossover, etc • State of qualification

3. Proposal through CSS 2104 • Test data • Overall project should have minimum test data (SDTM & ADaM) • Scripts can propose new test data, must pass (Data fit? Open CDISC?) • Share program to produce test data, never binary test data • 2 levels of qualification to match script complexity/output • Light vs. Heavy qualification • Common elements include • header • good programming practices • clearly declared scope of script (e.g., study design(s)) • test data matches scope & passes "FDA Data Fit" assessment (?) • documentation inputs/outputs/dependencies/usage

4. Proposal through CSS 2104 • Heavy qualification • Beta package includes minimal elements for contribution • Specification & Documentation (could be in pgm header) • Test data (Data Fit? or Open CDISC or other, as appropriate) • Tests & Expected results defined • Peer Review: GPP, Specs & Docn reviewed, Tests reproduced • Draft • Write qualification plan, Review tests for completeness/suitability (e.g., Branch testing – are all conditional blocks/combos tested?) • Test • Peer Review: Write qualification report, incl. log/output from tests • Final

5. Proposal through CSS 2104 • Light qualification • Beta package includes skip if >1 yr production usewithout ERROR • Draftminimal elements for contribution • Specification & Documentation (could be in pgm header) • Test data (Data Fit? or Open CDISC or other, as appropriate) • Tests & Expected results defined • Peer Review: GPP, Specs & Docn reviewed, Tests reproduced • Write qualification plan, Review tests for completeness/suitability (e.g., Branch testing – are all conditional blocks/combos tested?) • Test • Peer Review: Write qualification report, incl. log/output from tests • Final

6. Proposal through CSS 2104

7. Proposalmeaningful terms in blueQualificationhttp://www.phusewiki.org/wiki/index.php?title=File:WG5_P02_Proposal_-_2014.pptx • Certification applies to new scripts and tests • Confirmation applies to updates of existing scripts • States: Contributed, Development, Testing, Qualified • Roles • Contributor: Anyone with appropriate skills & interests • Developer: CSS Working Group 5 volunteer** familiar with objectives • Tester: CSS WG 05 volunteer** • Environment Tester: Anyone in industry community able to set up automatic test replication in their work environment • Reviewer: Author of white papers, designers of script targets ** suggests an quick-start onboarding page in CSS Phusewiki

8. ProposalQualification • End-user Objectives • Clear overview of purpose and resources • Inspire confidence from first sight • Ease of use, clear messaging from first run • Consistency of scripts, learning first one makes remaining familiar • Ease of converting users to contributors • Contributor Objectives • Standardize routine steps • Modularize routine components • Automate testing, issue identification • Centralize & consolidate information & results

9. ProposalQualification • TransitionsContributed is the original State of all scripts • to Development checklist includes by Developer & Reviewer • D reviews components • D works with Contributor to complete minimum components[ including Test Data and Coverage of defined tests ] • D adds standard parameter, dependency checking • R confirms contributed output matches/approximates target[ may require analysis details, specs, from contributor ] • D writes Qualification instructions .docx (see template) • to Testing by Tester • Review Qualification instructions, consider coverage of tests • Execute Qualification instructions • Work with Developer to complete execution successfully

10. ProposalQualification • Transitionscontinued • to Qualified by Tester & Environment Tester • T updates posted test outputs from certification/confirmation • E updates local tests and executes (posting PASS/FAIL results) • Achieve "Qualified" state when all tests in all test environments PASS (i.e., match outputs that T has certified and/or confirmed

11. ProposalQualification • Efforts Required • Finalize Qualification states, roles, workflow and templates • Design test structure in google code • Develop scripts that will allow Environment Testing • Develop general components (e.g. parameter, dependency checking) • Identify Environment Testers based on • Host environment • SAS or R version • Identify opportunities to automate qualification. E.g., • Docx format for Qualification instructions is not easily machine readable • Environment Testers to post results back as machine readable • Script green-light/red-light qualification matrix on Phusewiki

12. Doing now what patients need next