1 / 29

University of New Mexico “The Agony & the Ecstasy… the Trials, Tribulations & Thrills of an Enormous Project”

University of New Mexico “The Agony & the Ecstasy… the Trials, Tribulations & Thrills of an Enormous Project”. Mark Reynolds Associate Director ITS Communications Network Services. The University of New Mexico: A City Within a City. Founded in 1889 with 600 acres along old Route 66

leann
Download Presentation

University of New Mexico “The Agony & the Ecstasy… the Trials, Tribulations & Thrills of an Enormous Project”

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. University of New Mexico “The Agony & the Ecstasy…the Trials, Tribulations & Thrills of an Enormous Project” Mark Reynolds Associate Director ITS Communications Network Services

  2. The University of New Mexico: A City Within a City • Founded in 1889 with 600 acres along old Route 66 • 27,000 students on primary campus; 7,000 at branch campuses • 20,000 faculty and staff statewide including UNMH • Branch campuses in: • Gallup • Los Alamos • Taos • Valencia County

  3. Health Science Center (HSC) University of New Mexico Hospital (UNMH) • HSC, established in 1994, is largest academic health complex in the state • UNM Hospital (HSC's primary clinical component): • Consistently one of the 100 top-performing hospitals in the US • Primary teaching hospital for the UNM School of Medicine • Home of the highly regarded UNM Children's Hospital • New Mexico's only Level 1 Trauma Center that treats more than 1,300 critically injured patients a year

  4. UNM Children’s HospitalCommunications Challenges ($260 Million Project) • 480,000 square feet - six stories high • 2000+ voice & data drops using Category Augmented 6 cable plant • Telecommunications building in planned footprint

  5. What UNM Has… • 24,000 ports • 19 PBX’s, multiple Key systems, DRUs • 9,000 voicemail subscribers, 90 auto attendants • 64 Automatic Call Distribution groups, 300 agents • 2,200 cell phones, 4,000 pagers

  6. PBX SystemsSimplified Version

  7. First StepWhat to do with the University’s Main Central Office PBX? • Option 1: Move 11,000 port PBX to another location (estimate $10 million) • This would require: • New building and power plant • New PBX • Move all copper, fiber and carrier connections Decision: Expense and time unacceptable to Hospital • Second option: Leave the PBX where it is & protect the PBX building • Hospital estimated $400,000 to fortify the PBX building • Actual final cost was $2 million plus new office building Decision: Work with design team to protect the University’s communication systems

  8. Next Steps - Survival Plans • Identify risks and probability of system failures • Develop a mitigation & contingency plan • Develop a disaster recovery / incident action plan • Make improvements to construction plans • Develop detailed work plan • Perform audits of switch-room, electrical and cable plant • Rehabilitate cable vault • Replace voice processing system

  9. Next Steps – Survival Plans • Reroute carrier facilities, negotiate with LEC for two additional POPs & make Hospital PBX independent • Reroute cell phone services • Reroute alarm / emergency services • Develop emergency services agreement with outside firm • Create position to oversee protection of switch room and manage cable plant

  10. Next Steps - Telephone Planning Study • Five year plan with input from stakeholders • Ensure high quality customer service and “state of industry” systems • Enable cost controls • Ensure self-financial support • Plan for technical and operational convergence • Move thoughtfully toward convergence, evolve with current manufacturer (NEC) • Use multi-vendor solutions where practical • Plan / carry out VOIP trials • Perform cost allocation study to determine true costs • Develop rates and revenue strategies

  11. Existing PBX room

  12. Construction Begins • Construct new office building for telecom staff • Install and cutover new voicemail system • Demolish building around the PBX room without structural damage • Keep PBX room sealed, provide positive pressure to keep dust out and build vestibules at both entrances • Carrier changes (two additional POP’s) • Power design and changes • HVAC system - “clean” the duct work (original plan)

  13. Construction Continues • Remove copper cable feeding existing telecom building • Move the carrier fiber paths that feed the PBX room • Move two cell sites • Overcome public utility power pole complications • Remove ceiling tiles to avoid equipment damage when removing copper plant • Design new lighting fixture layout • Decommission Halon system and install FM200 system • Halon dump accident • New piping system

  14. Construction Continues • Remove existing HVAC system which consisted of: • Indoor Liebert Unit • Ceiling Unit for backup • Install two 20 ton units using “chilled water” and DX backup • Hole in roof for exhaust of portable systems • Portable units in room to keep temperature at 68 degrees with 25% humidity • Install new duct system • Install new generator to replace existing one • New grounding system • Power within the PBX room rerouted and re-cabled

  15. Construction Time Lapse Video

  16. Vibration Beam Installation Dust Biggest Construction Concerns

  17. Vibration and Core Drilling

  18. Beam Installation

  19. The Best Laid Plans…. • Dust • No vestibules built, see daylight through walls and roof • PBX equipment and room fill with dust • Heat and aging PBX concerns—could fail • Engaged external and internal professional services to document • Contractor • Installed devices to assist in damage control, but the damage was already done

  20. Solution • Bonding authority received for switch replacement and upgrades--$5 million • RFP developed • Replace the 20+ year old, 11,000 port main PBX • Upgrade the Hospital 10,000 port PBX • Various ancillary system and PBX upgrades • Outsource the maintenance and installation functions • Telephone Planning Study recommendations followed

  21. Next Steps • RFP completed • Negotiation & award of contract • Upgrade all systems, replace sets not ADA compliant • Transition maintenance group from existing contractor to new contractor • Installation of fiber paths to the University, Hospital, and data center with full fail over

  22. Next Steps • Establish PBX upgrade/replacement project and communications plan • Maintain support for work orders and trouble tickets during PBX implementations • Build new Hybrid PBX and distribution frame within same room as existing PBX (space limitations) • Tie hybrid PBX’s to existing PBX for migration strategy • Design and install temporary emergency PBX for the hospital PBX upgrade

  23. Next Steps • Customer service responsible for “getting the word out” • Communication methods included: • Flyers posted on entry doors and throughout affected buildings • Telephone calls to key department representatives • Email to building coordinators and mass emails • UNM News Minute update • Schedules, FAQ’s and other documentation posted on the web • Posting on the outage notification whiteboard • Communication process changed as the criticality of the situation increased.

  24. The Trouble Begins! • Roof • Roof leaked during snow and rain fall • Installed sump pumps with alarm system • New roof and drains • HVAC system • 20 Ton units come on line, backup system does not work

  25. The Trouble Intensifies! • Our WORST FEAR! • PBX starts to fail • No dial tone, call forwarding, auth code failures • Cannot shut down the PBX • Weekend – PBX fails but comes back • New PBX is working but not all customers have been moved over • Fast track the project • Cut over 4000 pair a day • 2-3,000 end users affected daily • Work day and night to cut-over • Move E911, voicemail & special circuits • Affected Hospital auto attendant & clinics

  26. New University Switch Room

  27. UNMHeart – Courage - Wisdom

  28. Celebrate! A Job Well Done!

  29. Thank You!

More Related