Risk-limiting Audits

1. Risk-limiting Audits • A tool for building trust

2. Introductions • Miriam Campbell, OSOS VIS Manager • Garth Fell, Snohomish County Elections Manager • Jennifer Morrell, Consultant for Democracy Fund

3. Breakout Session Plan • Risk Limiting Audit Exercise • What is a Risk Limiting Audit (RLA) • Snohomish County Pilot • Next Steps • Resources

4. Risk Limiting Audit Exercise

5. Risk-limiting Audit A risk-limiting audit is • a post-election procedure • that uses statistical methods and tools • to audit the elections process and • provides a known chance of identifying and correcting wrong electoral outcomes.

6. Snohomish County RLA Pilot

7. Pilot Goals • To follow the procedural steps required for a risk-limiting audit as outlined in state statute, administrative code and best practice • To determine the time requirements and constraints of a risk-limiting audit as currently outlined in state statute and administrative code • To understand the staff, equipment, facility, and supply resources required to conduct a risk-limiting audit

8. Pilot Goals (cont.) • To understand the impacts of conducting a risk-limiting audit on the continuing needs of the election • To better understand the benefits of a risk-limiting audits

9. Pre-pilot day activity • Outlined the steps required in statute / WAC • Held a series of conference calls with the OSOS and Jennifer Morrell • Identified how we would maintain and create a ballot manifest • Developed tools and documents to aid in pulling ballots

10. Ballot manifest • Comprehensive of all valid ballots • Independent of the tabulation system • Contains information that will allow you to efficiently find specific ballots

11. Documents to aid in pulling ballots • Ensures the correct ballot to audit is pulled • Serves as a placeholder in the original storage box

12. Pilot day

13. Random seed

16. Generate a list of ballots to audit

18. Pull the ballots

23. Enter ballot markings

25. Generate reports

27. Lessons Learned and Outstanding Questions • Consistent, documented ballot handling and storage practices are critical • Maintenance of a ballot manifest • Storage of boxes/batches matches your ballot manifest • Post-scan imprint of the image number very helpful • Contemplate impacts on other types of audits or recounts

28. Lessons Learned and Outstanding Questions • An uncertain time and resource commitment • Number of ballots to pull and hand count is based on the risk-limit, number of ballots returned, and closeness of target race. • Opportunistic audits add time. • The audit process is designed to expand your audit if there are discrepancies, potentially conducting a full hand count.

29. Lessons Learned and Outstanding Questions • RLA tool improvements • Requires that ballots and markings be input in a particular order • Unnecessary key strokes • Online nature of the tool needs to be considered from a security/privacy standpoint

30. Lessons Learned and Outstanding Questions • How do talk to the public about risk-limiting audits? • The “large chance” vs. perfect

31. Next Steps • RLA Tool • Rulemaking • Notification • Training • Cybersecurity Team (SOC) • Promoting voter confidence

32. Questions

33. Resources • A Gentle Introduction to Risk-limiting Audits – Mark Lindeman and Philip B. Stark (https://www.stat.berkeley.edu/~stark/Preprints/gentle12.pdf) • Tools for Comparison Risk-Limiting Election Audits (https://www.stat.berkeley.edu/~stark/Vote/auditTools.htm) • Knowing It’s Right, Parts One and Two– Jennifer Morrell, Democracy Fund, May 2019 (https://www.democracyfund.org/publications/knowing-its-right)