Considerations for Developing a Performance Measurement System

1. Considerations for Developing a Performance Measurement System MTAC Workgroup #114 April 13, 2007

2. Agenda • Introduction • Performance Measurement Methodology • General Study Considerations • Sample Design • Mail Fabrication • Induction • Receipt • Data Analysis • Reporting • Summary Performance Measurement Methodology

3. Introduction • The USPS requested that we present today based on the requirement from the Postal Accountability and Enhancement Act to implement performance measurement for all Market Dominant Products • IBM’s Background in External Performance Measurement • Have conducted USPS transit-time measurement studies since 1990 • Mail measurement experience in more than 35 countries on various domestic and international initiatives • Methodology developed specifically for this work • Simple in concept • Complexity is in all the details Performance Measurement Methodology

4. Performance Measurement Methodology Quality Assurance StatisticalDesign Test MailFabrication DataAnalysis Test MailInduction Test MailReceipt Reporting Panel Management Performance Measurement Methodology

5. General Study Considerations • What objectives need to be met? • Who are the various stakeholders? • What are their interests and priorities? • Standards and best practices • What are other countries doing? Performance Measurement Methodology

6. Sample Design • Representative of real mail flows • Base the design on real data (e.g. ODIS) • Need a defined measurement period and precision level • Monthly, quarterly, annual estimates to +/- X% • What are you interested in measuring? • Usually on-time rate • Could also consider measuring: • Percent of mail delivered within a delivery window • Percent early, on time and late • Percent delivered within J+1 days, J+2 days… Performance Measurement Methodology

7. Sample Design (cont) • Provide for continuous measurement • Measure across the period you are interested in • Include all days, weeks, months in the period • Follow live mailing patterns to the extent possible • Across days of the week • Across weeks of the quarter/year • Define the field of study that you want to measure • Performance of a mailing • Performance of letters, cards, flats • Performance of all Standard Mail Performance Measurement Methodology

8. Sample Design (cont) • Meet desired stratification and geographic coverage by spreading inductions and receipts accordingly • Could use available ODIS data: • Origin/destination ZIP Code pairs • Address type (handwritten, machine-addressed) • Barcoded • Shape or weight • Postage type • Day of week mailed/received • Could also use other characteristics: • Physical appearance (color, size, window) • Presort level • Sender type • Method of acceptance/collection • Receiver type Performance Measurement Methodology

9. Sample Design (cont) • Provide for a geographic stratification to ensure coverage across all areas covered by the product • Some systems place o/d links into sample groups and sample links from the groups to represent the population • Major metro to major metro • Major metro to rural areas of size X • Rural areas to rural areas • Some systems seek to estimate performance for: • All mail originating from an area with certain precision • All mail destinating to an area • Origin/destination combination is more complex Performance Measurement Methodology

10. Sample Design (cont) • Sample size required is largely determined by: • Number of sample groups to be measured • Desired level of precision for more detailed cuts of the data • Time period for which precision is needed • Variability of the population • Larger sample sizes can improve the precision • A simple random sample size of 1,067 achieves precision of +/-3% • A sample size of 4 times that size achieves +/-1.5% • Sample must represent the field of study well to be accurate Performance Measurement Methodology

11. Sample Design (cont) • What else drives sample size? • Variability of the results • Closer to 50% on time - more sample needed • Closer to 100% on time - less sample needed • Number of and correlation between test pieces mailed at the same time • The more pieces are correlated, the less information you have • Number of usable pieces out of your sample (after business rules are applied) Performance Measurement Methodology

12. Mail Fabrication • How are test pieces created / chosen • Sampling • Created by mailers in different locations • Centrally by a third party • Will test pieces need to be shipped for induction • Matching test pieces to participants • Seeding receiver names into mailings • Timing and coordination • What addressing standards are needed • Quality of address: • Avoiding mail forwarding and other delays • What checks are done up front versus laterin the process Performance Measurement Methodology

13. Induction • How will you manage senders / mailers • Where are they needed based on design • Recruiting, training, and performance monitoring • What types of mail acceptance are needed • Collection boxes • Post Office • DDU, SCF, ADC, AADC, BMC or BMEU • How will the Start-the-Clock data be validated • What will the induction rules be • When will induction data be removed Performance Measurement Methodology

14. Receipt • Are receivers needed and how many • What types of delivery addresses are needed • Recruiting, training, and monitoring • Handling performance issues • How many pieces can be delivered to the same address: • Without compromising the confidentiality of the study • Without overburdening receivers • How will the Stop-the-Clock data be validated • What will the rules be • When will receipt data be removed Performance Measurement Methodology

15. Data Analysis • Processes will be needed to validate the statistical assumptions and results • Design will rely on having data about the characteristics of the population to ensure it is representative • Weighting methodology for metrics will need to be validated • Must develop a methodology for estimation that is agreed to and understood • Business rules about what data are suspicious and how to identify, investigate and decide whether or not to keep data are needed • Pieces missing valid start-the-clock or stop-the-clock information • Pieces that were forwarded, returned to sender, etc • A receiver’s data when there is a pattern of “bad” reporting • Extreme circumstances (e.g. hurricanes, blizzards, blackouts) Performance Measurement Methodology

16. Reporting • What deliverables will be needed • Reports • Data files • Who are the recipients • USPS • Mailers • PRC / Congress • What level of reporting is needed • National • Mailer-level • By sample group • How frequently should deliverables be released • Preliminary versus final results Performance Measurement Methodology

17. Quality • Credibility of the measurement system depends on quality processes • Must be able to withstand scrutiny: • Mailer and USPS stakeholders • GAO • Congress • Ability to identify and respondto issues Performance Measurement Methodology

18. Summary Quality Assurance StatisticalDesign Test MailFabrication DataAnalysis Test MailInduction Test MailReceipt Reporting • There are numerous considerations in external measurement that vary depending on: • Objectives and priorities of your stakeholders • Which product you are measuring • What level of detail you require Performance Measurement Methodology