Statistical Testing for AMR Programs

1. Statistical Testing for AMR Programs CB Associates AMR Seminar March 30, 2004

2. Why Use a Statistical Testing Plan? • Focuses testing on the proper meters. • Minimizes number of meters to be tested; • usually requires less than 30% of what a • periodic testing plan requires. • Can provide data and analysis tools for use in • understanding what is happening with meters • installed in the field or for use in the purchasing • of new meters.

3. Acceptable Statistical Testing Plans ANSI C12.1-2001 Code for Electricity Metering Guidance Paragraph 5.1.4.3.3 Statistical sampling plan “The statistical sampling plan used shall conform to accepted principles of statistical sampling based on either variables or attributes methods. Meters shall be divided into homogeneous groups, such as manufacturer and manufacturer’s type. The groups may be further divided into subdivision within the manufacturer’s type by major design modifications.” NOTE - Examples of statistical sampling plans can be found in ANSI/ASQC Z1.9, the ANSI version of MIL-STD-414 and ANSI/ASQC Z1.4, the ANSI version of MIL-STD-105.

4. Homogeneous Population(s) • The groups or populations being sampled and tested • are made up of the same or similar items, items • which operate in the same way and were made in the • same manner. • For electric meters, this has traditionally been • interpreted as being meters of a specific meter • type from a manufacturer (i.e. AB1, J5S, MX, etc.). • AMR programs have helped to make the overall • populations more homogenous. This makes a utility • with an AMR system better prepared to take • advantage of a statistical sampling plan.

5. Suitably Sized Samples • The sample size for each group must be large • enough to provide a statistically valid sample • for the group’s population. • The larger the group’s population, the greater the savings for statistical testing over periodic testing and the more statistically reliable the testing • AMR implementation generally results in larger • group populations. The larger the population, the • more suitable for statistical testing.

6. Random Sample Selection • Every item within the group or population has • an equal chance of being selected as part of the • sample for testing. • Random sample selection is critical to providing • for a statistically valid sample. • AMR programs help to update and overhaul • meter record systems. Having the records for the • entire meter population updated allows for a better • chance that any meter may be selected as part of • the sample for testing.

7. Population Fits the Statistical Model • The statistical model being used for the sampling/testing plan needs to match the actual distribution of the population. • In most circumstances, one is looking at a normal or Gaussian distribution (i.e. a Bell curve). • This can be checked using a histogram plot or a chi-square analysis. For mechanical and electromechanical meters, a normal distribution fits the actual data very well. • For electronic or solid-state meters, there is some question due to the failure modes of these meters. These meter types are fairly recent designs, and not enough data has been seen yet to verify a normal distribution.

8. Population Fits the Statistical Model • AMR programs put either retrofitted electromechanical meters in the field or solid state meters. Electric Utilities must be in a position in the near future to determine if the solid state meters have a normal distribution. The only way to determine this is to aggressively begin testing and evaluating the in-service solid state meters.

9. Statistical Testing Plan w/ AMR • By definition an AMR system no longer has a pair of human eyes checking the installation each month. Statistical testing allows the Utility to quickly identify which areas may have a problem. • Potential problems that could be caught by aggressive testing. • A faulty batch of meters • Design or premature equipment failures • Poor installation due to a poorly trained crew • Location related failures • Energy Diversion

10. Population Fits the Statistical Model • Test an installation and not just a meter. Test programs for AMR systems need to involve testing and checking the meter performance as well as checking and testing the installation. This more extensive test check list should be done for the higher revenue C&I customers.

11. Statistical Sampling and Revenue Protection One of the significant benefits to the statistical sampling of AMR meters is the potential to spot energy diversion more readily. Statistical testing of meters will indicate the overall health of the meter population. Coupled with historical revenue information and meter tamper flags statistical testing can become a powerful tool for combating energy diversion. Utilities will be in a better position than ever to spot trends toward energy diversion more readily and on a closer to real time basis.

12. Statistical Testing with AMR • Statistical testing to monitor AMR programs will also point up; • design or manufacturing deficiencies • installation or post-installation problems (some of which may or may not be energy diversion). • All should be pursued and the root cause understood.

13. Statistical Testing with AMR Statistical Testing in preparation for AMR

14. Statistical Testing Plan w/ AMR • For the business case a good statistical testing program can determine how well the existing system is working and what revenue gains might be expected from replacing all of the meters over a 36 to 48 month time period. • A good statistical testing program can also be used to make sound business decision as to which meters should be retrofit and which should be replaced.

15. Statistical Testing with AMR Setting up Testing Programs to monitor AMR installations and post AMR performance

16. Statistical Testing with AMR • As you implement your AMR program problems and exceptions will seem to pour out of the wood work. The key is to stay focused on the primary issues and not on the isolated occurrence. Statistical testing can help you to differentiate between the two. • Testing during installation will help utilities to spot trends early on. Root cause analysis will help to determine if there is a design issue, a manufacturing issue, an installation issue, a communication issue, or a training issue. All of these will occur to one degree or another. Some can be corrected easily. All will require expensive revisit work.

17. Statistical Testing with AMR • As you are completing your AMR installation the following is a brief check list and discussion of topics to cover; • What waivers may be available from your Utility Commission? • How long will these waivers be effective? • How will revenue protection work with your AMR program? • Identify people responsible and whether or not they are interested in working together to develop a more comprehensive and informative program.

18. Statistical Testing with AMR • Areas to cover continued; • Who is responsible for the meter performance - the utility or the vendor? • Who determines when there is or is not a problem inside and outside the Utility? • Frequency of sampling and objectives • Design concerns • Support concerns • Installation concerns

19. Statistical Testing with AMR • Example of an AMR test program; • Best time to start to develop the program is while the meters are being installed. • Use installation reports to determine if there is any initial concerns about the meters being installed. • Typical reports that should be available; • Failed Meter Report, Project to Date • Electric Meters on Network Report

20. Statistical Testing with AMR • For this utility the failed Meter Report listed nearly 30 failure or return categories into which the manufacturer classified returned meters which failed either before, during, or after installation. Additionally, returned meters which were pulled because of abnormal remote polling but passed a multi-function test (MFT) are listed in a separate category. The data for each category was broken down into three groups: • Recalls - Meters that were recalled prior to installation. These normally came from automatic recalls generated by meter module status flags. Some recalls were manually generated. • Not Installed - Meters with problems found by the installer and not installed. • Maintenance - Meter problems reported outside of the automated process and changes using a manual paper process.

21. Statistical Testing with AMR • Of the various failure or return categories, the seven largest failure categories associated with start-up failures were analyzed in detail for possible trends. These seven categories represent 85% of all failures and 95% of failures not related to programming errors or electric surge damage. The seven categories included: • New meters – Unable to Read Meter Module • Retrofits – Unable to Read Meter Module • Abnormal Cumulative Count • Packet Error in Meter Module • Broken Leg/Base/Glass • Burnt Meter/Base/Leg • Defective 1S Meters

22. Statistical Testing with AMR • Data for the categories was tabulated into a spreadsheet • Data tables and graphs for each category were created. • Summarized data and graphs for these seven categories, both collectively and individually were evaluated and presented to the management team. • The graphs provided a visual picture of the growth of each failure category and were based on the associated summary data. Where appropriate and useful, graphs showing the percentage of the meter population for a failure category were included.

23. Statistical Testing with AMR • Percentage graphs were done for the following categories: • Summary of Top 7 Failure Categories and Failed Meters Passing MFT Testing • New Meters – Unable to Read Meter Module • Retrofits – Unable to Read Meter Module • Meters Reported as Failed but Passing MFT Testing • Failure percentages were calculated for all categories, but due to the small percentages involved for some categories, graphs were only produced for the above four categories. Percentage data is tabulated on the data table for each category. • Data on the installed new meters and overall AMR meter population was obtained from the Electric Meters on Network reports.

24. Statistical Testing with AMR • Data Evaluation and Conclusions • After monitoring the situation for nearly 2-1/2 years and evaluating 32 months worth of data the following conclusions were made regarding statistical testing and monitoring of the newly implemented AMR system meters: • The overall meter failure rate, including returned meters passing MFT testing, was X% of the installed population. Of this half are actual failures and half are returned meters passing MFT testing. • This final number is considered to be fairly accurate since two months were allowed to pass to let the backlog of failed meters returned to the manufacturer be tested and added to the Failed Meter Report.

25. Statistical Testing with AMR • For most failure categories, it was not possible to breakdown the failures between new meters and other AMR meters. The Unable to Read Meter Modules categories was the exception. The failure rate in this category for new meters was eleven times that of retrofit meters.New meters – Unable to Read Meter Module is the largest failure category representing about half of the actual electric meter failures for the AMR project. • Of the lesser failure categories, the failure rates were well under 0.10%.

26. Statistical Testing with AMR • Recommendations • Since the AMR deployment has been completed, the following recommendations are made for follow-on monitoring of the AMR meter population: • In-service testing will be critical for determining the actual state of the installed meter population. For the random sample in-service testing program, all efforts should be made to ensure that a sufficient sample of new meters (at least 200 per group) is pulled from the field for in-service testing. • Minor design changes in the new meter over the course of the AMR deployment could mean that the in-service performance of the meters may differ depending on the exact age of a meter and its design variations. Therefore, the in-service test results for the new meters should be analyzed in detail to see if there are obvious performance differences between different sub-groups of meters.

27. Why Do All of this Testing? • Installation of AMR programs move at seemingly breakneck speeds with all focus on schedule. At the same time, problems and exceptions seem to be pouring out of the woodwork. Upper management wants to hear about project milestones and budgets and not about the problems. Especially not any publicly embarrassing problems associated with an AMR installation.

28. Why Do All of this Testing? • The meter engineer will have only limited resources to address this multitude of problems and exceptions. Statistical testing will allow you to more readily identify where the problems are and where there were simply anomalies. The testing will help differentiate between training and equipment problems. The testing will also help to identify potential weak areas in the system that may bear closer scrutiny as the system goes into service. Putting a good testing system into place during the implementation will help to keep you on schedule, on budget, and out of trouble during the installation and will ensure that there will be a good system in place with the self discipline and understanding to administer the system.

29. Summary • AMR provides the Utility with the opportunity to get even more and better business information from their installed meter base. Statistical Sampling of these in-service meters can help to point up deficiencies in the installed system during installation as well as shortly after system implementation. The sampling can help to identify potential energy diversion and can help catch design inadequacies in the meters. Once a problem is identified additional statistical testing can help to zero in on a problem and help to identify potential solutions. • Statistically testing the installed meter population will allow the utility to more fairly meter the entire population without unfairly charging any one customer and without unfairly subsidizing any group of customers. • Statistical sampling plans are also lower cost plans to use than the traditional periodic plans.