600 likes | 614 Views
Learn about UC Merced's process of selecting, deploying, and migrating to Office 365, including infrastructure deployment, user support, and next steps.
E N D
Deploying Office 365 At UC Merced Nick Dugan and Todd Van Zandt UC Merced Information Technology UCCSC 2013
Introductions • Nick Dugan, Senior Systems Administrator • ndugan@ucmerced.edu • Todd Van Zandt, Director of Academic Technology and User Services • tvanzandt@ucmerced.edu
Agenda • Background • Selection and Pilot Process • Infrastructure Deployment • Migration • User Support • Next Steps and Conclusions
Background UC Merced • Opened 2005 • Population in Fall 2012 • 5,800 students • 140 faculty • 1,000 staff • 10,000 students by 2020
Background • Technology • ~8,000 active accounts • ~10,000 alumni / former student accounts • Lots of infrastructure from ~2005 • Sun Messaging Server • Oracle Calendar • SPARC Hardware
Background • Sun -> Oracle == $$$ • SPARC hardware refreshes • Maintenance and license costs • Oracle Calendar EOL • Heavy use of Outlook with Oracle Calendar connector • Incompatible with Outlook 2010 • Client software not supported in Mac OS X Mountain Lion • Email Growing Pains • Quotas, Compromised accounts, SMTP blacklisting • Aging webmail client
Background • Minimally Utilized Windows Infrastructure • Identity Management: Sun Waveset • LDAP: Sun Directory Server • Active Directory • Fed from IDM • Lab support • Printing • NAS Authentication • Minimal account attributes populated
Selection and Pilot Process • Selection Committee • Convened November 2011, charged with developing functional requirements and recommending a solution • Representatives from faculty, staff, and student populations • Locally hosted options ruled out • Microsoft vs. Google • Work completed March 2012
Selection and Pilot Process • Microsoft vs. Google • No clear consensus among committee members • Subjective and objective scoring gave a slight advantage to Office 365 • Both better than current solution • High level of satisfaction with both • High satisfaction with Lync among staff • Final decision left to IT after consideration of committee report and analysis of costs
Selection and Pilot Process • IT Review • March – May 2012 • Microsoft reduced price of A2 plan to $0 • Evaluated technical requirements for implementation • Reviewed committee input • Spoke with other UC CIOs and IT organizations • Final recommendation for Office 365 delivered to Chancellor and Provost
Infrastructure Deployment • Consultant Engagement • Decision made to employ professional services for tenant deployment and migration support • Interviewed and vetted approx. 10 companies in July 2012 • Selected Cloudbearing, initial engagement Sept. 2012
Infrastructure Deployment • Active Directory Enhancements • Domain Controllers in four physical locations • (Merced x2, Berkeley, Fresno) • ADFS Servers • 2x ADFS, 2x ADFS proxy, load balanced across two physical locations • DirSync Server • HA deployment not possible
Infrastructure Deployment • Why ADFS and DirSync? • Apprehension about pushing passwords to the cloud • IDM Already fed Active Directory • Infrastructure vs. custom development trade-offs made sense
Infrastructure Deployment • Lesson – Time is everything • ADFS is sensitive to clock skew • ADFS Proxies are not domain-joined by design • No automatic clock sync with DCs • Small time discrepancies lead to authentication outages • Proxies need reliable NTP
Infrastructure Deployment • Identity Management Integration • Sun Waveset (Java) on Solaris • EOL, Replacement in 2014 • Office 365 == PowerShell • IDM provision/deprovision events now trigger licensing actions on a remote PowerShell server • Lots of waiting built into process due to (un)timeliness of events in o365 cloud • Account provisioning went from real-time to “an hour or two”
Infrastructure Deployment • Shibboleth / SSO integration • Goal: Never see the ADFS sign-on page, OWA integration into campus portal
Infrastructure Deployment • Shibboleth configured as a relying party in ADFS • ADFS Home Realm Discovery page modified to bypass IdP selection box • ADFS Logout handler modified to ignore SAML errors and redirect to global logout page • We will help • office365@ucdavis.edu
Infrastructure Deployment • Dynamic Mailing Lists • Sun Messaging does this well • Office 365 does this… differently. • Limited to default attributes • We have dynamic lists based on lots of attributes we don’t push to AD • Interim Solution: maintain Sun Messaging Server at lists.ucmerced.edu • Distribution lists in o365 forward to @lists for expansion and ultimate delivery back to o365 • Long-term Solution: Next-Gen IDM with proper group management
Migration - Email • Approach and timing • Early adopters, opt-ins (early-mid December) • Faculty and Staff by affiliation/geographic location (Jan 2-20) • Students alphabetically (Jan 14-25) • Mailing lists (Feb 2013) • Alumni (March-April 2013)
Migration – Email • Technical considerations • Respecting user settings • Forwarding • Vacation messages (didn’t do) • Messages that won’t migrate (>25 MB) • Identify and notify
Migration - Email • Technical Process • Set DirSync flag in AD so accounts are pushed to Cloud • Assign licenses using PowerShell (not too early) • wait • Copy forwarding settings using PowerShell • Change LDAP routing address • user@merced.onmicrosoft.com • Set migration password • Migrate mail • Remove migration password
Migration – Email • Tool #1: Office 365 IMAP Migration Tool • Pros: • Free • Fast (not subject to the same bandwidth throttling as non-MS tools) • Easy • Cons: • ZERO reporting/logging. • Only provides a count of “skipped” messages for each account – not which messages or why they were skipped.
Migration - Email • Lesson: Never trust a tool that has no logging. • Half-way through faculty/staff migration, number of skipped messages went through the roof • Hundreds or thousands on some accounts, zero on others • All attempts at resolution failed • Session/resource issue with local IMAP server? – No • Too many concurrent migrations? – No • Microsoft able to provide an explanation or resolution? – No • Panic? – Yes
Migration – Email • Migration Tool #2 – MigrationWiz • Contract with Cloudbearing provided up to 300 mailbox licenses • Pros: • Excellent Reporting Capabilities • Excellent Support • Easy to incorporate into migration workflow • Cons: • $11/mailbox == $$$$ • Subject to Microsoft inbound traffic throttling
Migration - Email • Lesson: Ask what other people are doing first. • office365@ucdavis.edu
Migration - Email • Migration Tool #3: imapsynchttp://imapsync.lamiral.info/ • Pros: • $50 (might as well be free) • Excellent support • Great reference from UCSB • Cons: • Required a bit of work to incorporate into our migration workflow • Subject to Microsoft inbound traffic throttling
Migration - Email • Schedule, bandwidth, timings • “Just in time” mailbox provisioning was difficult • Start data migration by 10pm, run until done • Largest migration batch was ~1,100 accounts • 40 simultaneous connections • Generally finished by 8:00am
Migration - Email Lesson: Your carefully constructed process will experience its worst breakdown the day you migrate the Chancellor. Story
Migration – Email • Jan 3, 2013 – Migration of Chancellor’s office • 8:00am – Departmental account identified that did not provision successfully (sAMAccountName >20 chars) • 10:00am – Sysadmin attempts to rectify problem by deleting and then re-provisioning account to Active Directory • Sysadmin accidentally deletes ou=People instead of individual account
Migration – Email • Recovery • First thought – pull the (virtual) network plug on one of the DCs before changes propagate • Too late • Second thought – shut down DirSync server NOW • Preserved accounts and mailboxes in cloud tenant • ADFS non-functional, so as auth tokens expired, users lost access • Third thought – Reprovision from IDM? • No IDM group management, share permissions would be lost
Migration - Email • Recovery • Decision made to restore from tape and rebuild AD • Backups were good, but VM restore process was failing • 3:00pm – Support case with Symantec • Overnight – work case with Symantec, upgrade server and client software • Jan 4 morning – single DC restored • Windows sysadmins rebuild AD forest • Jan 5 9am – Full functionality restored
Migration – Email • Lessons: • AD Recycle Bin would have made this a non-issue • Requires 2008 R2 domain functional level • UCM was at 2003 • Verify backups (AND restores) • Prepare for the worst when migrating campus executives
Migration - Calendar • Cloudbearing contract included migration of calendar data through third party (CalMover) • All calendar data migrated to Office 365 at the same time • Thursday, Jan 17 5:00pm – Oracle Calendar database extracted and uploaded to CalMover • Friday, Jan 18 – receive PSTs from CalMover • Fri-Sun – import PSTs to Office 365 • Monday Jan 22 – Office 365 official UCM Calednar
Migration - Calendar • Issues, side effects • Modification or cancelation of migrated meetings would not notify attendees • Contacts, tasks, and other non-calendar data was excluded from import • Resource delegates, sharing and booking permissions did not migrate
Migration – Calendar • Timing, speed, historical events • Cloudbearing recommended limiting historical data to 12 months • PST import could be very slow, might not finish over weekend window • Individuals needing complete historical data would be accommodated through other means of import/export • Application/service updates made the import much quicker than anticipated • Allocated 3 days, completed in 6 hours
User Support • Campus Communication • Pilot Support • Migration Support • Documentation • Training • Client Issues
User Support - Communication • Campuswide announcements • News articles • MSOs • Info sessions • Record and make available • Inform about • Benefits • Cons / Changes (more important) • Process , expectations • Direct emails to users at the time of migration
User Support - Communication • Website • Large Banner to draw attention • O365 project section included • Migration schedule • Configuration Guides • List by device, not just service or client • Training & Getting Started Guides • Explanation of what is coming (large mailboxes, mobile support) , but also limitations and differences • Update all old information pages, aside from Config Guides
User Support - Pilot • Dedicated a technician to assist with installs • Did not provide a lot of help • Asked pilot group to review draft documentation • IT Student Employees • Engaged Resident Assistants (additional support) • Lessons Learned • Needed to have created formal testing plans • Needed to have been more engaged in support
User Support - Migration • Get Extra Help! • We used temps, technically savvy dept staff, and IT students and staff • Dedicated 2 (of 5) Desktop Support technicians to this effort once migrations started • 2 DSS departed as we were starting • Hired 4 temps to help with migration appointments • First used them to create and finalize documentation for various client set ups • Followed up on problems, after migrations completed for the day • Identify technical leads in departments and target for early adoption • Student Technology Consultants • Interested IT staff
User Support - Migration • Temps: • The Bad: Not as engaged, less reliable, had to replace two • The Good: Easy to hire, able to release when not needed, trained for the future, targeted at O365 tasks • Suggestions: Interview more so you have extras “on-hand”; Have projects lined up for them in downtime; Keep an eye on them
User Support - Migration • Coordinated with Depts (opt-in) • Took a LOT of effort • Difficulties with scheduling • 60-120 per group; • Only so many depts on a day • Plus additional one off early adopters • Coordinating postponements for out of office users • Allowed us to test our processes and improve documentation
User Support - Migration • Informed Depts for mass moves • Chose by location; focused support to an area • Tables in the lobbies (rarely used) • Self-help documentation was good • Morning apptsfor high profile groups • Most done within 4-6hrs • Everyone wanted help immediately • OWA was an option • Many users (faculty and lecturers) not present
User Support - Migration • Staff • Very interested; needed help and assistance • Many Windows Outlook users • Remove Outlook Oracle Connector • Reattach Personal Folders and Archives • Import Address Books and Autocomplete lists • Upgrade to Office 2010 (became too lengthy) • Faculty • Many were out of office • Many self-configured their clients
User Support - Migration • Students • Mostly web access users so just adapted to using OWA • Or still forwarded their email to Gmail, etc. • Switched webmail in Campus Portal if migrated • Did not read email to understand process; if went to direct to MyMail link • Migration of lots of students took longer for data to copy, new mailbox looked empty except for new • Offered trainings, did not attend • More critical for Grad students (instruction and research) • Finally moved them en masse; should’ve planned that from start
User Support - Documentation • Documentation • Configuration of Clients • Outlook, Thunderbird, MacMail, Mac Outlook • What about calendar for Oracle Calendar Client users • What about Lightning, Mac Calendar? • Configuration of Mobile Devices • Use of new systems (OWA)