Lucas TOI – CUC Upgrade and Rollback

1. Lucas TOI –CUC Upgrade and Rollback Monica Price Cohen

2. Agenda • L2 Upgrades • Upgrade • Switchover • Rollback • Switchover • Partial Data Migration, gotchas • Troubleshooting

3. L2 Upgrade • L2 Upgrade consists of 2 Phases • Upgrade and switchover, same a CCM • GUI Upgrade - On the OS Administrator page, pick “Software Upgrades | Install/Upgrade” • CLI Upgrade - utils system upgrade* • Manual Switchover • Needed if you choose not to do automatic reboot • GUI – On the OS Administrator page, pick “Setting | Version”. Click button to switch version. • CLI Switchover – utils system switch-version • Beware – A result is returned almost immediately from both the CLI and GUI as successful. In reality, switchover is still in progress.

4. Upgrade • Fresh install on passive partition • Most of the system is function during this phase • For Cores: • The majority of the CCM user data is migrated at this time • No User CRUD should be performed at this time. • New binaries, new database installed • Unlike CCM, there is no CUC data migration during this phase • Locales will be carried forward for ES and maintenance releases only • Major and minor upgrades will require all locales be reapplied (similar to Unity)

5. L2 Switchover • Connection services are stopped • A full CUC database migration is performed between the active and passive partitions • Different than CCM, which migrates only dynamic data at this time • On a co-res system connection will stop it’s services first, do the migration and then CCM will stop it’s services for it’s data migration. • Expected down time is about 20 mintues. Results may vary depending on user count, etc. • A grub swap is performed at the very end and the box is rebooted to the new partition

6. Rollback • Reverts the system back to the most recent version prior to upgrade. • Only mailbox (message) data is migrated. User data is not migrated during Rollback • Mostly the same code used for switchover during upgrade. • Only possible to revert one version in the past

7. Rollback Example: • System is upgraded from version A -> B • After the reboot A will on the passive partition and B will be on the active partition. • During upgrade from B->C, B will be still be active and C will be installed on the partition where A used to reside. • After reboot, a rollback to B is your only option.

8. Rollback gotchas • CUC’s priorities during rollback • Get the system back to a working state on previous version. • Make a best effort to retain messages and message states (New, Saved, Deleted) • Two types of successful rollback: • Gold Star Rollback - Successful CUC mail store database migration (achieves 1 and 2) • Silver Star Rollback - Failed CUC mail store migration, but rollback completed (achieves 1)

9. Gold Star Rollback • Users inboxes will contain messages in the same state as before the rollback • No mailbox data (messages, message state) should be lost • One caveat – If a user was created after upgrade, then their messages will exist on the rolled back system, but their mailbox, subscriber objects will not. So these messages are forward to the “Undeliverable Messages” mailbox.

10. Silver Star Rollback • In the event that an error is encountered while migrating message data, the mail store migration will be aborted and the message database will be restored to it’s original state at the time before upgrade. • The users mailbox size, and counts will be restored to the pre-upgrade state. • Message received after upgrade will be lost • Message deleted after upgrade will seem to reappear • Many of these “reappearing” messages will be missing their wav attachment. The message header has been returned, but the wav attachment was discarded after the user deleted the message on the upgraded system. The TUI handles this case, but it still can be unpleasant for the end user. • Not a desirable situation, but better than a busted system.

11. Troubleshooting • With root access: Install log are located at /common/log/install for both CCM and CUC install logs • Access these files through RTMT. • Trace and Log Central | Remote Browse | Trace Files. • Click next until you see “Select System Service/Applications” on the tab • Scroll down towards the bottom of list • Check Upgrade and Install logs and click finish • cuc-*.* files show the details for upgrade and switchover

12. Troubleshooting - Upgrade • install_log_<date/time>.log shows the overall progress of the upgrade • upgrade-results.xml shows the results of the upgrade (not switchover) and a good place to start Failed upgrade-result.xml

13. Troubleshooting – Upgrade (cont) • Same as troubleshooting a fresh install • If errors are pointing at cuc, cuc-install.log and cuc-dbinstall.log show the details for the upgrade (installing rpm packages and the database install respectively)

14. Troubleshooting - Switchover • GUI returns with “Update successful”, even though the switchover is still in progress • Watch the contents in the install log directory for updates • In case of error during switchover, review the install_log<date/time>.log file for details • If errors are pointing at cuc, cuc-version-switch*.* files should be reviewed for errors

15. Troubleshooting – Switchover (cont) • Ex: Successful Rollback (Gold or Silver Start) cuc-version-switch_<date/time>.log

16. Troubleshooting – Switchover (cont) • Ex: Silver Star Rollback (rollback succeeds, but mailbox store data is not migrated) • This should only the case when a defect is present. cuc-version-switch_<date/time>.log