IHTSDO Quality Assurance Framework Jane Millar Chief Quality Officer

1. IHTSDO Quality Assurance Framework Jane MillarChief Quality Officer www.ihtsdo.org

2. Quality Assurance Framework • Background • Purpose and scope • Description • Applying the framework • Example applications of the Quality Assurance Framework • Tool for capturing quality metrics • Next steps and future plans

3. Quality Assurance + Framework “Quality assurance (QA) refers to a program for the systematic monitoring and evaluation of the various aspects of a project, service, or facility to ensure that standards of quality are met” (Wikipedia, 2009). The Quality Assurance Framework will be used to properly monitor the quality of the product (SNOMED CT) and to be consistent with the broader understanding of quality assurance, i.e. systematic monitoring and evaluation of the various aspects of an IHTSDO project, service, or facility to ensure that standards of quality are met

4. Background • Developed by the IHTSDO Quality Assurance Committee • Is a merger of models including: • world of software quality engineering • healthcare quality assurance • existing terminology quality assurance processes • Piloted across a number of varied IHTSDO activities • Approved for use and roll out across the IHTSDO by the General Assembly and Management Board in October 2009 • To meet the requirement to set targets and demonstrate quality standards across all IHTSDO projects and services • Now reinforced by the IHTSDO Strategic Direction to 2015

5. Origins of the quality framework • IEEE software definitions • ISO terminology definitions ISO/IEC 9126 software quality characteristics

6. www.ihtsdo.org Purpose of the Quality Assurance Framework • The Quality Assurance Framework, and supporting Toolkit, has been developed to support a consistent, systematic and ubiquitous approach to the incorporation of quality assurance practices into any IHTSDO activities and products, whether as: • surveys or studies to identify priority areas for quality improvement, • the specification and introduction of novel quality assurance practices to an established or newly introduced activity • the documentation and presentation of established quality assurance activities, • The use of the Quality Assurance Framework will allow this to be done in a consistent, comparable and systematic way.

7. Scope of the Quality Assurance Framework • All (any) identifiable aspects of IHTSDO activity and products • Standing committees (Quality Assurance, Content, Technical, Implementation and Innovation), Special Interest Groups (SIG’s), Project Groups, Central functions, (? National Release Centres) • Organisational processes and support • Data products (terminology reference data, mappings, translations, subsets) • Documentation • IHTSDO-responsible services and tooling provision

8. Definitions • IHTSDO Quality • The degree to which the IHTSDO meets its specified objectives, in terms of its organisational and product development processes, as well as the services and products it provides. • IHTSDO Quality Assurance • A planned and systematic pattern of actions necessary to provide adequate confidence that the IHTSDO meets its specified objectives, in terms of its organisational and product development processes, as well as the services and products it provides.

9. Definitions (cont) • Quality Metrics • Agreed methods and means for measuring the agreed levels of achievement, performance or conformance of a component or its quality characteristic(s) • Quality Targets • Agreed levels of achievement, performance or conformance of a Component for any given Quality Characteristic • Quality Component • Sub-parts of a product, process or service. Categorized as structural, process or outcome that can be quantified or measured in order to improve the quality of IHTSDO activities and products

10. Definitions (cont) • Quality Characteristics • A typology of attributes of an IHTSDO component by which its quality is assessed or measured. • Terminology Quality • The degree to which a terminology, component or process, meets agreed customer requirements and satisfies stated IHTSDO purposes

11. Quality Characteristics Process activities e.g. project management, balloting Efficiency Clarity Adaptability Usability Transparency Equitability (fairness) Agile Standardized Non-redundant Service-based activities e.g. release dissemination, call centre activity Functional Reliable Responsiveness Transportable Timeliness Improvable Comprehensiveness Affordability/sustainability Consistent

12. General quality assurance framework for integration into IHTSDO activities Activity Structure Component 1 Process Characteristics Outcome • Quality • Metrics • Targets Component n

13. Request submission process - example REQUEST SUBMISSION PROCESS Define and set targets Activity • Quality • Metrics • Targets Effective process in place Component 1 Structure Turn around times for requests Responsiveness Component 2 Characteristics Process

14. www.ihtsdo.org Quality Metrics example Request submission process example

15. Quality framework at a glance

16. Implementing the framework • Where do we apply the framework ? • Anywhere where you are undertaking a project or specific work item, particularly where you will be required to show the outcome of that piece of work • How do we apply the framework • Quality Assurance Framework, Framework Toolkit, Framework template, • Who do we tell ? • Quality Assurance Committee

17. Applying the quality assurance framework • General Quality Assurance requirement • Set targets and demonstrate quality standards for all IHTSDO projects and services • The IHTSDO Quality Assurance Framework • Does not dictate what the quality standards should be • Does • Provide a consistent mechanism and framework for identifying project or service components • Specifies the quality characteristics/attributes of each component • Sets standards or targets for each characteristic • Identifies a realistic mechanism for measuring (and demonstrating) whether such standards are achieved

18. Metrics • Agreed methods and means for measuring the agreed levels of achievement, performance or conformance of a component-characteristic • Description: What is to be measured and how this is believed to demonstrate the quality of the associated component-characteristic • Target: Agreed levels of achievement, performance or conformance of a component-characteristic that would be felt to demonstrate adequate quality • Plan: Description of how measurement is to be carried out • Level achieved: Agreed reporting format for the metric once measured (units, timescale) • Response: Agreed response steps to follow when this metric is reported (in particular if targets are not achieved) or when a target is revised

19. Example of metrics • Data production and publication • Schema and relational integrity conformance tests • Conformance metrics • Change request management • Infrastructure and procedure for responding to change request • Change request response time metrics • Editorial rule adherence • Editorial rules and their implementation • Degree of compliance with the editorial rules

20. www.ihtsdo.org Quality Metrics example Committee management

21. www.ihtsdo.org Quality Metrics example Travel expenses process

22. Quality Metrics example Tooling and Technology Request Submission Technical Solution

23. Engagement in developing quality approach • Quality and assurance is central to all IHTSDO activities • Quality is a section included in IHTSDO Project Proposals and projects will not be accepted without • Similarities across projects in terms of Quality Assurance • Sharing and reuse of metrics across IHTSDO projects (Do not re-invent the wheel . . . ) via use of a repository • Consistent set of metrics across activities • Collect and share instantiated templates • Support by Chief Quality officer and committee

24. Quality Assurance Framework documentation • IHTSDO Quality Assurance Framework • Introduction and description of IHTSDO quality assurance framework • IHTSDO Quality Framework Toolkit • Framework description and example/prototype application • Templates A and B • www.ihtsdo.org/publications/quality-assurance/

25. Quality Assurance Framework documentation • IHTSDO Quality Assurance Framework • Introduction and description of IHTSDO quality assurance framework

26. “SNOMED CT is a quality product, produced by the IHTSDO, which is an organisation with quality at its centre”