Where’s My Data ? Application Migration & Integration

1. Where’s My Data?Application Migration & Integration CPTE 228 John Beckett

2. The “Green Field” Myth • Myth: There is no difference between starting a new IT process from scratch and migrating to a new process. • Truth: Migrating is often more difficult, because you must carry forward your data. • Truth: Managers actually know this about buildings, but fail to apply it to data.

3. For example: • Install new email client. • Oops: All your saved mail is lost. • Can’t convert, find, or move folders for some reason. • Oops: Your address book is gone. • Same problem • And this is an easy task! • Imagine a real-life scenario with multiple databases.

4. Solving the Email Client Issue • Find the former assets • Look in c:/Users/username/AppData • Probably the “roaming” profile • Look for the app’s name • Is the mailbox format compatible? • Perhaps you can import • Or…Copy to the new location • A messy process – but experience helps

5. The Bad News • You probably aren’t going to do anything “green field” unless you are starting a new company. • Even then, you’ll probably start small and do a larger-scale version later. • So you’ll almost always be migrating data at implementation time.

6. Data Migration Approaches • Static • Hand Entry • Semi-Dynamic • Phased Roll • Process-Based • Live • Screen Scraping • EDI/EAI Connection

7. Migration AuditingHow Do You Know It’s Right? • Totals • Number of transactions, amount of money • Codas • Identifiable batches that you can compare • May have to be time periods • Statistical Methods • Are we seeing reasonable ratios (inc/expense, etc) • Select sample transactions at random, follow the path

8. Method: Hand Entry • No electronic connection to old system needed. • Cleanup of old data done at data entry. • Appears to be simplest for computer people. • What about data entry error cleanup? • May be lots of work for clients. • Maybe not as much as you think • Default if other methods fail. • Probably used for some components anyhow. • How are you going to “prove” it is correct?

9. Hand Entry Choice Factors When to use • Small amount of data • Data is stable during conversion period • No other way to do it • Conceptually difficult to map former data onto new system • Perhaps the way the new system works is why you are changing! When not to use • Lots of data • Data changes during conversion period • Quality typing unavailable

10. Method: Phased Roll • “Roll” data from old system to new system on a per-component basis • For example: This week we’ll “roll” Payroll. Next month we’ll “roll” Accounts Payable. • Requires building a roll process for each set of data • Inter-Relationships can be complex

11. Campus Shop Roll (circa 1977) • Administrative system was being copied to new system every night • Campus Shop system was ready to go on new system (and old was unusable) • “Your ID card will work at the Campus Shop tomorrow” • Individual could be entered manually • Campus shop transactions copied back to old system at end-of-month statement time This is actually a very simple example. Real life is usually far more complex.

12. Phased Roll Choice Factors When to use When not to use Data relationships change during conversion period Data is tightly coupled between systems E.g. If the Campus Shop data needed to be available on the old system up-to-the-minute You aren’t converting everything • Data involved in relationships is stable during conversion period • People could tolerate the ID card delay • Data is loosely coupled between systems • Only one file needed to come back from Campus Shop, monthly

13. Method: Iterative • Like phased roll, except that: • A complete conversion is done repeatedly (daily?) • New systems are available only for read-access until all systems are complete • This is a Beckett-designed approach. Life is rarely so simple that you can do it this way – but I’ve done it twice

14. Iterative Method Choice Factors When to use • Large amount of data • Especially if large part is foundational to others (e.g. Academic Records) • Codas available • Data is not stable during conversion period • Need to show early success When not to use • Conversion matrix is too complex • No specific foundational data • Time required for conversion run is long • Extensive hand-manipulation is necessary

15. Method: Screen-Scraping • “Live” process emulates old workstations, captures data. • Usually requires some sort of “middle-ware” product • Requires former method to be character-mode (less likely in the future) • You could, however, use an HTML Mash-up

16. Screen-ScrapingChoice Factors When to use • No alternative access is available • Former system is frozen • Occasional errors are: • Detectable • Correctable • Acceptable if detected & corrected When not to use • Direct access to data is available • Former system is under development • Zero tolerance for data errors • Problem: No feedback Any time you have a better alternative

17. Method: Hand-Coded Interface • Write a program for each connection between source and destination • Or add module into existing program • Have the program invoked either continuously or periodically to transport across

18. Hand-Coded Model

19. Hand-CodedChoice Factors When to use • Very few interfaces • You can achieve the “liveliness” needed When not to use • Many interfaces are required

20. Method: EAI Electronic Applications Integration • “Middleware” product used to keep current data on both sides. • Use EAI language (or modules you program) to define interfaces

21. EAIChoice Factors When to use • Integrating with system under development • You need live data • Bandwidth is available • Interfaces work When not to use • Bandwidth and/or server overhead is intolerable • “Snapshot” view is preferred • Extreme security concerns about updating

22. EAI model

23. Myths of EAI • Myth: • (N(N-1))/2 interfaces required for hand coding. • Only N interfaces required for EAI • Truth: • You don’t have all the modules sending data to each other. • EAI doesn’t excuse you from designing all your interfaces. • But…EAI systematizes the process, and that is a very good thing – probably as important as the magic that was promised!

24. HTML Mash-Up • Write server-side code that reads a Web site • Parse the site to get the data you want • Present it in a useful manner to users

25. Choice Factors for Mash-Up Yes No The source data site changes often You have no further resources to spend on the project Downtime is intolerable • The source data site is stable • The mash-up is not more complex than you care to maintain • Downtime is tolerable if the source changes • Until you fix the problem, because you have no warning

26. What’s the Best Way? • It’s a designer’s choice: • Meet the needs • Accept the risks • Accept the costs

27. The Good News • This is in the middle a CSA/CIS person’s expertise. • Modest programming skills will help. • Success at data migration is a key to moving up in the organization. • It’s a bullet point you can add to your resume’.

28. Case Study: Hazel & Co.“Because you have other things to do” • Maid service • Began with three maids working from one location • Now has 500 locations with 3,000 maids • 200 locations company-owned • 300 locations franchised

29. Business Challenges • Quality service from minimum-wage people • Connect with external sources of info • Arrest records • Payroll data • Out-source (or in-source by our co. for franchisees)

30. History • Ran out of a checkbook register • Excel • Quickbooks • Got a payroll system • Got a General Ledger/Accts Pay system • Wrote conversion system to grab data from PR, GL, AP -> put into Quickbooks

31. Wrote a Web-based system that analyzes current info from PR, GL, AP – makes it look like Quickbooks • Requests for more / better info: • Write add’l modules for new system that gives that information • Write modules that create Excel sheets with desired analyses