Integrating Performance Measures and Quality Systems

1. Jeffrey C. WorthingtonChair, Energy and Environmental DivisionAmerican Society for QualityOEI Director of QualityOffice of Environmental InformationU.S Environmental Protection Agency Integrating Performance Measures and Quality Systems straight from the mouse's mouth - Mickey Mouse Lorena Romero CedenoProgram AnalystOffice of Environmental InformationU.S Environmental Protection Agency ASQ 32nd Annual National Energy & Environmental Conference San Antonio, Texas September 19-20, 2005 iaidq

2. Jeffrey Worthington- BIO • Director of Quality for the USEPA Office of Environmental Information. Jeff has served as the Director of Quality Assurance for the USEPA Office of Research and Development (ORD) National Risk Management Research Laboratory (NRMRL) and as the Director of Quality Assurance for TechLaw, Inc. He is an American Society for Quality (ASQ) Certified Quality Manager and ASQ Certified Quality Auditor. Jeff, Senior ASQ member, founding member of the Education Division, Chairs the ASQ Energy and Environment Division and participates on the ASQ Division Affairs Council. He is a founding member and serves on the Board of Directors for the recently established International Association for Information and Data Quality (IAIDQ). Jeff is a member of the Editorial Board of Quality Assurance, Science, and the Law and previously served as Editorial Board member for: the Journal of Environmental Forensics, Environmental Laboratory magazine, and Environmental Testing and Analysis magazine. • He has been with the Federal Government since 1994. Jeff supported environmental engineering quality at NRMRL, joining a team authoring the combined quality and management system for EPA’s Environmental Technology Verification program. He co-lead the EPA team developing EPA’s Information Quality Guidelines. Jeff co-authored a peer review journal paper and received the USEPA Science and Technological Achievement Award, Level III for equating EPA policies and procedures to U.S. Supreme Court Sound Science Criteria (2002). Jeff has spoken at numerous national and regional conferences on the subjects of quality management, audit management, information quality planning and assessment, data authenticity, data quality, and data integrity.

3. Lorena Romero Cedeno - BIO Lorena R. Romero-Cedeño is the Manager of Quality for the Office of Planning, Resources and Outreach for the Office of Environmental Information. She graduated in 1999 from Colorado State University and currently is finalizing two master degrees in Community and Regional Planning and Latin American Studies from the University of New Mexico. Lorena has a passion for the integrity of environmental data being translated into Spanish targeted to the Spanish speaking public. She is currently conducting research on environmental data for her master thesis and holds two years of experience teaching Spanish Language at the University of New Mexico.

4. DISCLAIMER The opinions expressed in this technical presentation are those of the author and do not necessarily reflect the views of the US EPA.

5. SPONSORED BY MICKEY……….. WHY MICKEY? • Statue theme in 2005 • He’s cool!!! • He is a performer… must know something about performance measurements • He is a genius • Wears the best clothes • Never ages • Is very popular • Is everywhere • Has a lot of friends • Certified quality inspector (usually wears white gloves) Mickey Mouse

6. MICKEY FACTS • Cherry Blossoms • Age • Oscar • First words • First cartoon character to ___ • Quotes Mickey Mouse

7. Lobsta' Mickey

8. Information Mickey The Mouse is in The House

9. Quality Mickey Q Arithmetic is being able to count up to twenty.... without taking your shoes off

10. Selected Performance Measurement Systems Used inEPA and Federal Government • GPRA – Government Performance and Results Act • FMFIA – Federal Managers Financial Integrity Act (management controls) • EVM – Earned Value Management • CPIC – Capital Planning and Investment Coordination (especially information technology) • Balanced Scorecard

11. GPRA – Government Performance and Results Act • Mandated by OMB • Effort to identify and track project/program/process outcomes (vs. outputs) • Managed at a high level in the organization • In EPA, may include: • QA/QC procedures • Data quality reviews • Data limitations • Error estimate • Each office negotiates their own list of measures with OCFO and OMB • Annual performance plan submittals – www.omb.gov

12. Outcomes vs. Outputs • Outputs = productivity or efficiency metrics such as the number of reports written per month • Outcomes = the impact that the output has on success of the process organization

13. FMFIA – Federal Managers Financial Integrity Act (management controls) • FMFIA Act of 1982 • Agency weaknesses vs. • Annual Management Accomplishments and Challenges report • Office’s self-identify adequacy of “management controls” • OIG, GAO, and OMB overview comments may be part of overview • Action plan to make corrections and identify schedule and measures to correct

14. EVM -Earned Value Management • Standard - ANSI/EIA-748-A Standard for Earned Value Management Systems • EV = value of completed work expressed in terms of the budget assigned to that work • Objective measure of work accomplished • Based on budgeted value of the work • ‘What you got for what it costs you.” • Compliance standard criteria grouped in 5 areas • Planning, scheduling, and budgeting • Organization • Analysis and management reports • Revisions and data maintenance • Accounting considerations • Performance measurement • Establish a baseline • Monitor in time units • Review both cost variance and schedule variance from baseline

15. Office of Environmental Information Report- Earned Value Management: Fiscal Year 2004Lessons Learned December 2004 • Applied to assist IT project managers consistent with the ANSI/EIA-748 standard and in line with requirements of the OMB Exhibit 300. • 3 categories of LESSONS LEARNED • Refinement of EVM methods • Increasing consistency of project reporting • Facilitating management analysis of EVM data

16. LESSON LEARNED 1Refinement of EVM methods • Use EVM across all phases of mixed life-cycle projects • Separate milestones and associated resources for system life-cycle phase • Separate milestones and associated costs by contractor wherever possible • Keep milestones from getting too large in duration, cost, or scope • Attempt to limit milestones to a single fiscal year (or less) • Establish objective measures for determining earned value

17. LESSON LEARNED 2Increasing consistency of project reporting • Use standard template for reporting • Institute standard reporting cycles

18. LESSON LEARNED 3Facilitating management analysis of EVM data • Provide both numerical and graphical representations of EVM data • Use color-coded standardized scoring system

19. Balanced Scorecard A Management System (not only a measurement system) that enables organizations to clarify their vision and strategy and translate them into action. • Provides feed back for both internal processes and external outcomes in order to continuously improve strategic performance and results. Balanced Scorecard Institute www.balancedscorecard.org

20. Balanced Scorecard – view organization from 4 perspectives • Learning and growth perspective • Business process perspective • Customer perspective • Financial perspective

21. Balanced ScorecardBusiness Process Perspective • Products and services conforming to customer requirements • Mission-oriented processes AND support (repetitive) processes • (easier to benchmark repetitive processes)

22. Balanced ScorecardFinancial Perspective • Timely and accurate funding data • Cost-benefits

23. Balanced ScorecardCustomer Perspective • Customer focus • Customer satisfactions • Matching products and services to customer groups

24. Balanced ScorecardLearning and Growth Perspective • Employee training • Organizational culture toward personal and professional growth • Keeping knowledge workers in a continuous learning mode

27. BRM – OMB’s Business Reference Model • From OMB’s Federal Enterprise Architecture Program Management Office (FEAPMO) • Identifying and categorizing an organization’s lines of business and sub-functions to relate them to the enterprise architecture

28. PRM – OMB’s Performance Reference Model • From OMB’s Federal Enterprise Architecture Program Management Office (FEAPMO) • A standardized approach to IT performance • Define measurement indicators • Establish relationship inputs and outcomes • Sets baselines and targets for improvement • Closely related to the BRM

29. GENERALIZATION – for various performance measurement systems • Relationship to organization’s vision and mission may need to be verified • Need to verify measures: • Are real • Are meaningful • Are not redundant • Include quality (limitations on use) • Are we measuring this item because it is easy to measure? • Is the goal convenient or challenging? • Applicability - What does measuring this particular thing tell you about the organization’s performance? • Completeness – Are enough measures being tracked to characterize a broader sense of organizational performance?

30. How to Speak the Language of Senior ManagementStephen George Quality Progress May 2003 OBSERVATIONS • The language of senior management is very different than the language of quality managers • To have their suggestions heard and accepted, quality professionals need to learn management’s financial vocabulary • There are seven steps you can take to improve communication with management

31. Speaking the Language of Senior ManagementThe Seven Steps • PROVE– quality professionals have to prove the need and then prove the value of proposed improvement processes • COACH – quality managers must help senior management move goals, objectives, and strategies into action, …. To move from strategies to measures to projects • INFLUENCE – involve senior management in quality processes, participate as trainers, incporated quality in reviews • PARTNER – with the CEO if possible, or with any other senior manager, focus on the area of greatest need, ask for guidance from senior management • PILOT – initiate a project in an area of value to the organization • BENCHMARK – with senior managers in other organizations • ALIGN – align the quality system language with the senior management language, everyone should be speaking the same language

32. Balanced scorecard for a quality system – HOW?? • Look for ways for the quality system to demonstrate an improvement in each of the four perspectives • Try to use a balanced scorecard for the quality system itself – can you really do this? • Performance measures of the organization vs. performance measures for individual processes (e.g., quality systems) – does this mean there is a new customer to consider in the balanced scorecard, the internal customer? • How much you are spending internally on your own process, for example • When do the management system and the quality system come into full alignment?

33. Balanced scorecard for a quality system • OUTPUTS • OUTCOMES

34. Balanced scorecard for a quality system • Learning and growth perspective • Business process perspective • Customer perspective • Financial perspective

35. Mickey says: - arithmetic is counting to twenty without taking off our shoes Quality Mickey says: Performance measurement of quality is measuring both outputs and outcomes

36. HOW CAN AN INTERESTED QUALITY MANAGER TRAIN THEMSELVES? • www.balancedscorecard.com

37. WHAT ELSE IS GOING ON AT EPA FOR QUALITY NOW? • OEI is re-developing its 5 year QMP • OEI has delegated authority for directives for Agency information policy, including quality policy

38. THE END See Ya Real Soon Mickey Mouse

39. CONTACT INFORMATION • Jeffrey Worthington 202-566-0997 worthington.jeffrey@epa.gov • Lorena Romero-Cedeno 202-566-0978 romeo-cedeno.lorena@epa.gov

40. A Body of Knowledge for Information and Data Quality WHY? FREEDOM from the tyranny of quality chaos

49. FREEDOM from the tyranny of quality chaos

50. Statue of Freedom - 1855, 1862, 1863