Know it’s Broke Before Your Customers Do.

1. Know it’s Broke Before Your Customers Do. David Thomas dthomas@dthomasdigital.com

2. Acknowledgements Nagios World Conference Ethan Galstad And of Course all of you!

3. David Thomas Director of Information Technology and Emergency Communications Sandoval County, State of New Mexico A little about me, and what I do. Over 20 years in the IT field, certified ITIL, Six Sigma, HDI and a bunch of other stuff.

4. Sandoval County Sandoval County a pretty unique place. The county has a total area of 3,714 square miles. 5 square miles of water. Population of 131, 561 12 Indian reservations (pueblos), The second most of any county in the United States. Bandelier National Monument Cibola National Forest Tent Rocks National Monument Santa Fe National Forest Valles Caldera National Preserve 5 National Protected Areas The county is also flanked by two national laboratories Sandia National laboratory to the south Los Alamos National laboratory to the north Numerous nuclear waste and weapons transportation routes run through the Sandoval County as well.

5. Sandoval County A Unique place with Unique Issues: Day-to-day operations • Over 70 Buildings throughout the 3,714 square miles of Sandoval County • County Sherriff facilities including 60 sheriff vehicles (PC and radio functions) • County Fire including 24 Fire Stations 80 vehicles (PC and radio functions) • County Detention Center 500+ inmate facility • County dispatch center • And all the other functions of county government; Treasure, Assessor, public works, ext. • County Heath Clinic including dental, dietary, indigent populations operation • Tribal governments of the 12 pueblos

6. Sandoval County Customers • So this presentation is called • So Who we talking about? • EVERYONE! • Population of 131, 561 our constituents • All County Government • All Municipality Governments • All 12 Indian Pueblos and their governments Know it’s Broke Before Your Customers Do.

7. Sandoval County Network Pretty typical midsize network

8. Here is where it get’s interesting! Talk about terrain we start on a hill shoot across the river valley back over the river valley up to a mountain peak to get to Cuba, NM 75 miles from Bernalillo, NM where we started. Not much of a chose if we are to get high speed networks to the far reaches of the county.

9. So Much for the background. • Now for something completely different. • It’s still about customers! • And in this case when it broke it broke bad!

10. Real emergencies in need of real solutions In 2011 Sandoval County had to activate it’s Emergency Operations Center Twice. • February 2011 – Regional wide natural gas outage during a record breaking -40 degree temperature outbreak • June & July 2011 – Los Conches Fire - Largest wildfire fire in New Mexico History Nagios played a pivotal role in the successful emergency operations of Sandoval County during these events!

11. Regional wide natural gas outage During a record breaking cold spell demand was such that gas companies made the decision to cut gas to less populated areas so urban areas would not be affected. The result was a sudden lost of gas for many areas across the desert southwest. What did Sandoval County Information Technology and Emergency Communications do? • What happened: Activation of County Emergency Operations Center and information center Code Red emergency notification to citizens activated 24 hour information line set up manned by IT Ham radio coordination with county shelters and red cross Updated information on County web sites 24/7 FEMA and NIMS forms compliance Operations lasted 120 hours

12. Los Conches Fire • What happened: Down power line turns into largest wildfire in New Mexico history. What did Sandoval County Information Technology and Emergency Communications do? • Activation of County Emergency Operations Center and information center • Code Red emergency notification to citizens activated • Operations lasted 336 hours • Mobile emergency radio communications and command unit activated IT manned • COM/L activation on site communication coordination • Updated information on County web sites 24/7 • FEMA and NIMS forms compliance • Massive evacuations including the city of Los Alamos and LANL • Ham radio coordination with county shelters and red cross • Monsoon rains now major flooding.

13. Smoke plumes from my backyard. This image showing the Las Conchas wildfire raging in the Jemez Mountains of the Santa Fe National Forest in north-central New Mexico the was taken by a crew member aboard the ISS on June 27 Residences: 63 destroyed Outbuildings:49 destroyed; 2 damaged Injuries to Date:15 Size:156,593 acres

14. The Role Nagios Played and it’s a big role • In both situations when the County EOC had to be activated Nagios turned into a very useful tool. Nagios was constantly being monitored by EOC staff We have remote devices in remote locations that where in the fire All fire vehicles and sheriffs Vehicles could be monitored by VPN’ed laptops Thanks to IP based radios and repeaters those were monitored too Generator power was monitored on remote microwave shelters Mobile command truck could monitor and we could monitor it from the EOC Don’t forget standard uses for Nagios even more important in an emergency All our front line people have access to view including dispatch center All of this has to be documented for FEMA, and best of all it can be used as your change management database for ITIL functions. Nagois has all that built in. We use Cacti too!

15. Nagios played a vital roll: Nagios played a vital roll and will be a tool we use time and again in emergency situations and the day-to-day operations of Sandoval County, NM Know it’s Broke Before Your Customers Do. Or If it’s frozen or burning down Use advanced monitoring tools to help Save lives and property. David Thomas dthomas@dthomasdigital.com