Lecture no 10: Server upgrades

1. Lecture no 10: Server upgrades TDT4285 Planlegging og drift av IT-systemer Spring 2008 Anders Christensen, IDI TDT4285 Planl&drift IT-syst

2. Server upgrades • Make a service checklist • Verify compatibility • Make verification tools • Make a back-out plan • Choose a maintenance window • Inform the users • Perform the tests • Perform the upgrade • Perform the tests • On errors, perform the back-out plan • Inform the users TDT4285 Planl&drift IT-syst

3. Step 1:Make a service checklist It should specify dependencies and configuration related to: • Which services are implemented by the server? • Who are the users of these services • Which software is installed? TDT4285 Planl&drift IT-syst

4. Step 2:Verify compatibility Verify that every software package installed at the server is compatible with the new OS. If this is not the case, then: • Upgrade the software while still under the old OS • Upgrade the software in parallel with the OS upgrade of the server • Discontinue support for this software • Let the server continue to run the old OS as a legacy system. TDT4285 Planl&drift IT-syst

5. Step 3:Make verification tools • Make tests that verifies a broad set of the most used/most important functionality in the software running on the system. • Preferably automate these tests (or retrieve them from your test-library) • Make a regression test from the automatic execution of these tests. TDT4285 Planl&drift IT-syst

6. Step 4:Make a back-out plan • A back-out plan is a plan for how you get back to the state from before that start of the change: • Reversal of all steps taken • Restore of a backup/snapshot taken before first change • Reinstall of old system from scratch TDT4285 Planl&drift IT-syst

7. Step 5: Choose a maintenance window • Which maintenance windows are available • How do this change fit in with other changes • When is the change supposed to be effective • How much time is needed to do the change • How much time is needed to do the back-out plan if the change is unsuccessful. TDT4285 Planl&drift IT-syst

8. Step 6:Inform the users • Who and what is affected? • What will happen? • Why is this necessary? • When will it happen? • How to give feedback if there are strong reasons for postponing this change. • How to contact the sysadmin staff TDT4285 Planl&drift IT-syst

9. Step 7:Preform the tests • This should be done right before the maintenance window • May indicate recent but still uncovered problems, so you know these are unrelated to the upgrade. • Any problems should be fixed before starting on the upgrade. • Any problems should ideally have been uncovered and fixed during the planning of the upgrade. TDT4285 Planl&drift IT-syst

10. Step 8:Perform the upgrade • Follow normal standards for the upgrade • Consider doing the upgrade in pairs, which allows you constant peer review of all actions. • If the upgrade takes to much time, or there are any uncertainties, consider a back-out. TDT4285 Planl&drift IT-syst

11. Step 9:Perform testing after upgrade • All the tests from before the upgrade must be run again after the upgrade. • If there are any errors, find and correct them and rerun the tests • If there are any strange results, investigate • If you are running out of time, activate the back out plan TDT4285 Planl&drift IT-syst

12. Step 10:If problems, activate back-out plan • The back-out plan should not be postponed in anticipation that the problem will be solved • Somebody should have the suprime authority to activate the back-out plan • Enough time must be reserved, so that the back-out plan can be completed without overrunning the maintainance window. TDT4285 Planl&drift IT-syst

13. Step 11:Inform the users • With all down-time, tell the users what is/was down, and what is up again. • After upgrades, tell the users what has changed, and why that is better for them. • After an attempted upgrade and following back-out, tell the users which functionality that was anticipated but not implemented. TDT4285 Planl&drift IT-syst

14. Availability and monitoring • Monitor the system in order to verify that the service works, and that normal use has resumed. Let this extra monitoring be limited in time. • Keep the helpdesk up-to-date on all changes • For a period, make sure there is enough personel to jump right on any reported problem related to the upgrade. TDT4285 Planl&drift IT-syst

15. Upgrade: Preserves parts of the old setup Usually takes less time Fokuses on the current, actual configuration Maintainance Reinstall Excludes problems from earlier installations Focuses on the configuration as it is supposed to be Easier to ensure that it works Upgrade of reinstall? TDT4285 Planl&drift IT-syst

16. Testing the upgrade procedure • Complete test, where where a upgrade on a similar system is performed. • ”Mime” test, where all cabling are stretched, and all phyical opertions are ”mimed” • Timing tests, where parts of the tasks are performed in order to get timing data to plan the upgrade. TDT4285 Planl&drift IT-syst

17. Upgrade-models • Complete a new server off-line, and swap with the old server in one operations, the old server is a cold back-out. • Take the old server down, upgrade/reinstall it, and take it up afterwards. • Gradually phase the users from the old server to the new server, and take the old machine down when all users are phased over. TDT4285 Planl&drift IT-syst

18. Reuse of the tests • Put the tests in a ”library” of tests, which can be a source for later test suites • Use the tests for monitoring • Use the tests as detectors for degration and bit-rot • Use the tests as a reference point that defines how certain components behaved at a certain time. TDT4285 Planl&drift IT-syst