University of New Mexico “The Agony & the Ecstasy… the Trials, Tribulations & Thrills of an Enormous Project”

1. University of New Mexico “The Agony & the Ecstasy…the Trials, Tribulations & Thrills of an Enormous Project” Mark Reynolds Associate Director ITS Communications Network Services

2. The University of New Mexico: A City Within a City • Founded in 1889 with 600 acres along old Route 66 • 27,000 students on primary campus; 7,000 at branch campuses • 20,000 faculty and staff statewide including UNMH • Branch campuses in: • Gallup • Los Alamos • Taos • Valencia County

3. Health Science Center (HSC) University of New Mexico Hospital (UNMH) • HSC, established in 1994, is largest academic health complex in the state • UNM Hospital (HSC's primary clinical component): • Consistently one of the 100 top-performing hospitals in the US • Primary teaching hospital for the UNM School of Medicine • Home of the highly regarded UNM Children's Hospital • New Mexico's only Level 1 Trauma Center that treats more than 1,300 critically injured patients a year

4. UNM Children’s HospitalCommunications Challenges ($260 Million Project) • 480,000 square feet - six stories high • 2000+ voice & data drops using Category Augmented 6 cable plant • Telecommunications building in planned footprint

5. What UNM Has… • 24,000 ports • 19 PBX’s, multiple Key systems, DRUs • 9,000 voicemail subscribers, 90 auto attendants • 64 Automatic Call Distribution groups, 300 agents • 2,200 cell phones, 4,000 pagers

6. PBX SystemsSimplified Version

7. First StepWhat to do with the University’s Main Central Office PBX? • Option 1: Move 11,000 port PBX to another location (estimate $10 million) • This would require: • New building and power plant • New PBX • Move all copper, fiber and carrier connections Decision: Expense and time unacceptable to Hospital • Second option: Leave the PBX where it is & protect the PBX building • Hospital estimated $400,000 to fortify the PBX building • Actual final cost was $2 million plus new office building Decision: Work with design team to protect the University’s communication systems

8. Next Steps - Survival Plans • Identify risks and probability of system failures • Develop a mitigation & contingency plan • Develop a disaster recovery / incident action plan • Make improvements to construction plans • Develop detailed work plan • Perform audits of switch-room, electrical and cable plant • Rehabilitate cable vault • Replace voice processing system

9. Next Steps – Survival Plans • Reroute carrier facilities, negotiate with LEC for two additional POPs & make Hospital PBX independent • Reroute cell phone services • Reroute alarm / emergency services • Develop emergency services agreement with outside firm • Create position to oversee protection of switch room and manage cable plant

10. Next Steps - Telephone Planning Study • Five year plan with input from stakeholders • Ensure high quality customer service and “state of industry” systems • Enable cost controls • Ensure self-financial support • Plan for technical and operational convergence • Move thoughtfully toward convergence, evolve with current manufacturer (NEC) • Use multi-vendor solutions where practical • Plan / carry out VOIP trials • Perform cost allocation study to determine true costs • Develop rates and revenue strategies

11. Existing PBX room

12. Construction Begins • Construct new office building for telecom staff • Install and cutover new voicemail system • Demolish building around the PBX room without structural damage • Keep PBX room sealed, provide positive pressure to keep dust out and build vestibules at both entrances • Carrier changes (two additional POP’s) • Power design and changes • HVAC system - “clean” the duct work (original plan)

13. Construction Continues • Remove copper cable feeding existing telecom building • Move the carrier fiber paths that feed the PBX room • Move two cell sites • Overcome public utility power pole complications • Remove ceiling tiles to avoid equipment damage when removing copper plant • Design new lighting fixture layout • Decommission Halon system and install FM200 system • Halon dump accident • New piping system

14. Construction Continues • Remove existing HVAC system which consisted of: • Indoor Liebert Unit • Ceiling Unit for backup • Install two 20 ton units using “chilled water” and DX backup • Hole in roof for exhaust of portable systems • Portable units in room to keep temperature at 68 degrees with 25% humidity • Install new duct system • Install new generator to replace existing one • New grounding system • Power within the PBX room rerouted and re-cabled

15. Construction Time Lapse Video

16. Vibration Beam Installation Dust Biggest Construction Concerns

17. Vibration and Core Drilling

18. Beam Installation

19. The Best Laid Plans…. • Dust • No vestibules built, see daylight through walls and roof • PBX equipment and room fill with dust • Heat and aging PBX concerns—could fail • Engaged external and internal professional services to document • Contractor • Installed devices to assist in damage control, but the damage was already done

20. Solution • Bonding authority received for switch replacement and upgrades--$5 million • RFP developed • Replace the 20+ year old, 11,000 port main PBX • Upgrade the Hospital 10,000 port PBX • Various ancillary system and PBX upgrades • Outsource the maintenance and installation functions • Telephone Planning Study recommendations followed

21. Next Steps • RFP completed • Negotiation & award of contract • Upgrade all systems, replace sets not ADA compliant • Transition maintenance group from existing contractor to new contractor • Installation of fiber paths to the University, Hospital, and data center with full fail over

22. Next Steps • Establish PBX upgrade/replacement project and communications plan • Maintain support for work orders and trouble tickets during PBX implementations • Build new Hybrid PBX and distribution frame within same room as existing PBX (space limitations) • Tie hybrid PBX’s to existing PBX for migration strategy • Design and install temporary emergency PBX for the hospital PBX upgrade

23. Next Steps • Customer service responsible for “getting the word out” • Communication methods included: • Flyers posted on entry doors and throughout affected buildings • Telephone calls to key department representatives • Email to building coordinators and mass emails • UNM News Minute update • Schedules, FAQ’s and other documentation posted on the web • Posting on the outage notification whiteboard • Communication process changed as the criticality of the situation increased.

24. The Trouble Begins! • Roof • Roof leaked during snow and rain fall • Installed sump pumps with alarm system • New roof and drains • HVAC system • 20 Ton units come on line, backup system does not work

25. The Trouble Intensifies! • Our WORST FEAR! • PBX starts to fail • No dial tone, call forwarding, auth code failures • Cannot shut down the PBX • Weekend – PBX fails but comes back • New PBX is working but not all customers have been moved over • Fast track the project • Cut over 4000 pair a day • 2-3,000 end users affected daily • Work day and night to cut-over • Move E911, voicemail & special circuits • Affected Hospital auto attendant & clinics

26. New University Switch Room

27. UNMHeart – Courage - Wisdom

28. Celebrate! A Job Well Done!

29. Thank You!