Statistical Methods Compliance checking for MAC-EQSs

1. Statistical Methods Compliance checking for MAC-EQSs John Batty DEFRA UK Bratislava November 2010

2. Legal Background For any given surface water body, applying the MAC-EQS means that the measured concentration at any representative monitoring point within the water body does not exceed the standard. (Note 2 of Annex 1Part B of 2008/105/EC) However in accordance with Section 1.3.4 of Annex V of 2000/60/EC: • Member States may introduce statistical methods, such as a percentile calculation, to ensure an acceptable level of confidence and precision for determining compliance with the MAC-EQS. • If they do so, such statistical methods shall comply with detailed rules laid down in accordance with the regulatory procedure referred to in Article 9(2) of this Directive.

3. The Problem The International Standard ISO 5667-20 looks at Absolute limits in Section 5.3 • The purpose of a threshold is compromised if it is defined in a way that ignores the fact that compliance will be checked by sampling. • A threshold that runs this risk is the maximum value (or absolute limit). This type of threshold is popular because it is easy to understand and use, especially in legal actions, but : • increasing sampling will lead to more failed samples • taking fewer samples gives the illusion of improved performance

4. “Ideal” Environmental Quality Standards • ISO 5667-20 suggests that to achieve consistency an EQS should include: • A limit - e.g. a concentration of 10ug/litre • A “Summary statistic” - how often the limit may be exceeded- e.g. 5% of the time • The period over which compliance assessed e.g. one year • The statistical confidence required to demonstrate failure e.g. 95%confidence for reporting failure or taking legal action • The proportion of reports for which failure to meet the standard is acceptable over which assessment takes place for which failure is acceptable e.g. 1 in 10 calendar years

5. Sample Numbers and Compliance Low levels of failure do not usually present problems because the EQS standards adopted are precautionary The case below shows a site that truly exceeds its MAC limit for 1% of the time It is clear that sampling frequency can distort the levels of perceived failure. An illusion of improved performance can be created by actually taking fewer samples! Surely this is not the best way forward! Member states sampling more frequently will appear less compliant

6. Statistical Methods for MAC StandardsWhy are they necessary? • We usually assess compliance by taking samples of water and measuring the concentration of the substance • A MAC standard implies that this concentration must never be exceeded. Does this mean that no sample can fail, no matter how many we take? • If we only take 4 to 12 samples each year we have absolutely no idea of the levels present for the vast majority of the time– We cannot therefore confirm that no exceedences of the MAC standard occur in the year. It effectively means that we may actually be allowing the MAC to fail. • Continuous monitoring is required to prove absolute compliance with the MAC – both expensive and impractical

7. Types of exceedence • Levels of toxicants in water bodies are normally less than EQS values • Experience in the UK suggests that any failures observed are usually of short duration • Adopting an absolute compliance approach means one short term duration exceedence for one substance results in that water body failing compliance for that period: • even though the failure may be marginal • not representative of broader compliance • not ecologically significant • Is this what is intended? We paint the maps red??

8. Types of exceedence – a real time example This real time failure lasted 15 minutes • This type of incident is more common than you may think • The question is whether it is environmentally significant • Remember EQS standards are set using a precautionary approach –safety factors are applied to a NOEC • and even for acute values the exposure period will be rather longer than 15 minutes

9. Case Studies • To try and illustrate the problem we constructed a year-long dataset with a sample taken each day. • We included a known number of failures against the 95%ile and 99%ile standards • We then prepared 16 randomly chosen dates for i)quarterly, ii)monthly and iii)weekly samples. • The full dataset is depicted below

10. Case Studies • The graph shows a set of data derived from monitoring for a given pollutant • There are 20 occasions where the 95%ile limit of 100 is exceeded • There are 5 occasions where the 99%ile limit of 140 is exceeded

11. What do the sample sets show? Quarterly sampling is unlikely to identify any failures. Only one failure identified in the sixteen sets we prepared

12. Case Study Summary • Using our randomly selected set of sixteen sampling results for quarterly, monthly and weekly monitoring: • Quarterly monitoring did not pick up any of the 140 + values and only one of the 100+ value exceedences • Monthly monitoring revealed 12 of the identified failures • While weekly monitoring provided a much more comprehensive picture • BUT REMEMBER ANOTHER RANDOM SELECTION COULD REVEAL ENTIRELY DIFFERENT RESULTS! IT IS PURELY A QUESTION OF CHANCE

13. Compliance - The Proposed Solution ISO 5667-20 suggests defining the MAC as a percentile. A look-up table can be set up for any combination of percentile value and confidence level. Our suggestion is that the MAC should be a 95%ile standard and failure of the 95%ile is declared at 95% confidence. The benefit of such an approach is consistency, clarity, and the control of bias in reporting. A level playing field !

14. Pollution Control-Dealing with individual failures • The real value of a MAC standard is for pollution control rather than determining water body compliance • It is vital to separate these two issues • A positive approach should be adopted to pollution control • Any failed sample should trigger an immediate investigation on the ground especially if there are repeated or high level failures • Such an approach is vital to ensure confidence

15. Recommendation to WG-E CMEP is invited to consider the proposals made in this paper and prepare a recommendation to WG-E on the preferred way forward on this topic Thanks for your attention Any Questions?