European Conference on Quality in Official Statistics 2008 July 8 – 11, 2008

1. Using Quality Audits to Improve the 2007 Economic Census European Conference on Quality in Official Statistics 2008 July 8 – 11, 2008 presented by: Deborah Stempowski U.S. Census Bureau Economic Planning & Coordination Division 1

2. Agenda • Economic Programs Directorate • 2007 Economic Census • Quality Management and the 2007 Economic Census • Role of Quality Audits in Quality Management • Quality Audits and the Macro Processing and Dissemination Team Process • Macro Processing and Dissemination Processes • Macro Team Process • Software Quality Assurance (SQA) Evaluation Procedure • Conclusions • Current Status • Successes and Areas for Improvement for SQA • Quality Management Lessons Learned • Next Steps 2

3. Economic Programs Directorate • Measure and profile businesses and government organizations of the United States • Nine organizations in the directorate • Five subject matter divisions • Four directorate wide support divisions • Economic Planning and Coordination Division (EPCD) • Economic Statistical Methods and Programming Division (ESMPD) • Office of Statistical Methods and Research for Economic Programs (OSMREP) • Center for Economic Studies (CES)

4. 2007 Economic Census • Economic Census is conducted every 5 years • Data collection focuses on years ending in 2 and 7 • Questionnaire mailout occurs in December of the collection year • Data processing and dissemination activities take place for 3 years following mailout • Covers over 23 million business establishments • Core Census Programs: • Manufactures • Mineral Industries • Construction Industries • Retail and Wholesale Trade • Service Industries • Finance, Insurance, and Real Estate • Transportation, Communication, and Utilities • Use of enterprise systems to collect, process and disseminate data • There are approximately 1700 separate data releases over a 2.5 year period 4

5. Quality Management and the 2007 Economic Census • Short-Term Goals • Ensure adherence to Statistical Standards • Ensure adherence to Project Management process • Ensure adherence to Software Development process • Adherence measured through quality audits • Long-Term Goals • Improve product quality • Improve process quality 5

6. Role of Quality Audits in Quality Management • Evaluation of work products and processes based on established standards • Documentation of audit results • Communication with project manager, project team and SQA staff • Reporting of noncompliance issues to senior management as needed • Tracking of noncompliance issues to closure • Documentation of status of audit process • Evaluation of audit effectiveness against objectives of SQA program 6

7. Organization to Conduct Quality Audits • Separate Staff Dedicated to Quality Audits • Independently Supervised • Objective Review of Management Products Produced • Objective Review of Activities Performed • Feedback from QA Staff to Manager • Private Audit Process 7

8. Macro Processing and Dissemination Processes • Macro Processing and Dissemination -- Phase 3 of the Economic Census -- involves all processes related to the macro analysis and dissemination of the Economic Census data and its related programs. • There are 11 major IT systems • Each system is supported by a cross-divisional team comprised of staff from: • EPCD – Provides the Business Analysts and the Project Managers. Often times, both of these roles are played by the same individual. • Economic Statistical Methods and Programming Division (ESMPD) – Provides the programming and system development staff. • Subject matter areas (business units) – Provides the subject matter (data) experts and system users. • These teams strive to maintain, and if possible, improve the standardization of these systems. 8

9. Quality Audits and the Macro Processing and Dissemination Team Process • In the conjunction with ESMPD, EPCD developed the ‘Macro Processing and Dissemination Team Process’. • Re-used best practices from the 2002 Economic Census and used them to develop our own software development methodology. • Implemented a minimum process for software development methods and tools. • Developed standardized tools and procedures that would be used for all macro teams. • Ensured clear understanding of roles and responsibilities for all team members. • Throughout the project lifecycle, work products and processes are reviewed to ensure compliance to agreed upon standards (SQA Audits). 9

10. Macro Processing and Dissemination Team Process 10

11. Quality Audits and the Macro Processing and Dissemination Team Process 11

12. Conclusions • Current Status • Successes and Areas for Improvement • Lessons Learned • Next Steps 12

13. Current Status: • Status of the 2007 Economic Census: • All phase 3 projects are currently in development and testing. • Several of the project teams have made preliminary deliveries of production applications. • All systems are expected to be ready on or before October 31, 2008. • Status of SQA activities: • We have completed the first two cycles of SQA evaluations for all 11 IT systems. Neither of these two cycles identified noncompliance issues. • Third cycle of evaluations is scheduled to be complete July 22, 2008. • Project teams have conducted lessons learned sessions to gain feedback.

14. Successes and Areas for Improvement for SQA • Successful initiatives from our progress to date: • Clearly defined project processes and products as well as individual roles help to reduce ambiguity of tasks. • Utilization of standard templates and documents reduces the learning curve for team members who serve on several teams. • Independent assessment provides valuable feedback all during the project. • Initiatives and processes that could be more effective in subsequent projects: • Auditing process and products for presence only does not offer much value. • Having the SQA staff trained to review the contents of each document and not just the template that is used, would offer more value. • After several rounds of auditing with no noncompliance issues, it might be acceptable to only audit a subset of projects. • When established business processes have been working well to date, focus resources on other aspects of the process. • All team members should be included in the methodology development process to ensure consideration of work from all viewpoints is considered. • Improved communication plans would make everything flow much easier. 14

15. Quality Management Lessons Learned • Quality Management is a continuous process. It cannot be turned on for the Economic Census and turned off for all projects between Censuses. • Focus should be on the content of documents reviewed in the audit and not just the presence of that document. • Clear, specific, detailed standards must be developed early to get everyone educated on the process and its expectations. • Independent review and reporting of adherence to standards is of great value. • Goal of quality audits is to tell senior management whether a project is adhering the standards. Failure to adhere to standards should be viewed as a teaching opportunity to improve skills. • Planning and training for project management should include quality assurance processes and staff at all stages. • Decisions on tools used for project management should be made early . Some quality problems occur because we do not make the best use of tools available. 15

16. Next Steps: • Receive feedback from all participants during each phase of the project and make modifications where feasible. • Continue to carry out each project and perform an evaluation of the ‘Macro Processing and Dissemination Team Process’ upon the closeout of all projects. • Upon final evaluation, make modifications to the process and begin planning phases for the next cycle of projects to be audited. 16

17. Contact Information deborah.m.stempowski@census.gov 17