200 likes | 385 Views
Ford Motor Company. QAD Infrastructure Upgrade Solaris to Linux Mike Van Buren Program Manager Ford Motor Co Subbu Balakrishnan Thirdware Solution Inc. Agenda. Business Case Development Program Funding Development Platform Choices Developing The Plan Standardized Migration Plan
E N D
Ford Motor Company QAD Infrastructure Upgrade Solaris to Linux Mike Van Buren Program Manager Ford Motor Co Subbu Balakrishnan Thirdware Solution Inc
Agenda • Business Case Development • Program Funding Development • Platform Choices • Developing The Plan • Standardized Migration Plan • Migration Results • Lessons Learned: • Linux Technical Features • Migration Tools – Pro Dump & Load • Intel Chipset • Autosys • Program Management: Key Points • End Results
Business Case Development • Business Drivers: • Dated Solaris Infrastructure 7 Years Old / 2007 • Operating System Solaris 8 End OF Life Support • Internal Hardware Compliance Issues (Preferred Vendor) • Capacity Constraints Month End Loads 100% Utilization • Disaster Recovery Capacity Oversubscribed • Growing Interfacing / Integration Requirements • Chinese Growth Demand • Robust Batch Job Scheduling Tool • Challenges: • Program Capital • Cost Sharing Multiple Business Units • Maintaining Ongoing Operations • Chinese Growth Demand (Cost Sharing)
Business Case Development: Funding • Key Points: • Multiple Business Units & Business Models Supported • Significant Variability in Service Demand • Multiple Financial Approvals From • Joint Venture Partners: China & Japan.. • Internal Approval Processes - Complicated • Delivery Of a Consumption Based Assessment Model • Intense Customer Based Cost Scrutiny • Results: • Delivered a Consumption Based Assessment Model Based upon License Usage • Isolated Large Assembly Operations from Smaller Operations • Separate Joint Venture Funding Approval & Participation • Transparent License Based Assessment Model • Customers Understand How It Works
Platform Choices • Compared IBM P550, IBM X3950, Sun E6900 Sun T2000
Platform Selection: • Initially Selected IBM P3950 • IBM 3950’s Scaled Upwards On Capacity Bench Marking Exercise • SUN T2000 Single Processor Multi Threading Architecture – To New • IBM Was Identified As Preferred Vendor • CPT (Cost Per Transaction) Formula Supported P3950 Arch • Initially Acquired 6 Quad Core Servers in Three Phases To Minimize Capital Outflows Until Ready For Migration • At the time of acquiring the last servers in the program the preferred vendor had switched to HP DL580 16 core machines. • 12 Months after the initial acquisition, capacity doubled at similar cost levels for the HP Servers • Suse Linux SLES – 9 SP4 • Mixed Environment of IBM & HP Servers Suse Linux SLES – 9 • Linux Provided Lower Ongoing Operating Cost
Developing The Plan: • Funding Was Approved: What’s Next? • Identify Dedicated and Competent Team Members: • Understood the meaning of One Team • Willing To Work Weekends For 15 Months • Technical & Communication Skills (Multi Cultural) • Understanding The Importance of Sticking To The Plan • Build a Standardized & Repeatable Migration Plan • The Plan Had To Be: • Business Readable • Milestone Driven • Realistic & Deliverable • Planning Challenges: • Determining Migration Windows: (Closing Periods – Out) • Global Holiday Calendar (14 Countries) • Language – English Is A Second Language
The Plan Highlights: • Results: The Standard Migration Plan
The Plan Execution: • Migration Schedule
Pre-Migration Production Environment • Server Configuration: • 900MHZ Sparc 64 Bit Chipset • Solaris 8 Operating System • Four Production Servers: • SUN V1280 (12@900 MHz, 34GB RAM) • SUN V1280 (8@900 MHz, 34GB RAM) • SUN E4800 (6@900 MHz, 32GB RAM) • SUN Fire 880 (2@900Mhz, 32GB RAM) • Configured By Time Zone Support Window • New Zeeland through South Africa • Reliable and Stable • Progress 9.1D 64 Bit Database • QAD EB2/SP1 Application Software (Early Adopter of EB2)
Post-migration environment • Total No. of Production Servers –5 • Processors - 44 • RAM - 32 GB*5 • OS – Linux SUSE 9 SLES 9 • IBM x3950-[8878PCC](4@3.16GHz Intel Xeon Dual Core, 32GB RAM) - 3 Nos. • HP ProLiant DL580 G5 (16 Intel Xeon CPU’s @ 2.9GHz, 32GB Memory) – 2 Nos.
Migration Statistics • Migrated 40 Business Entities in 14 Countries • Closed & Archived 16 Business Entities • 350 Databases (QA & Prod) • 550 Printers • 900+ Batch Jobs Migrated to Autosys • Last Site Migrated on March 22, 2009
Lessons Learned: Technical Updates • Progress Database • Upgraded to 9.1E from 9.1D • 9.1D Was Not Supported on Suse Linux SLES 9 • 64 Bit Progress 9.1E Was Not Supported by QAD • Progress 10.1A /64 Bit was looked at on EB2 but not supported by QAD (Legacy Support Concerns Issue) • Attempted run EB2 on 10.1A/64 Bit Unsuccessfully • Suse Linux Operating System • Kernel Parameter Settings Changes from Solaris • Introduced Self Service Tool for Linux Password Resets • Reduced Administrative Overhead • Discovered CPU Utilization Issues with the newer Intel Chipsets. Required OS Patches From Novell
Lessons Learned: Technical Updates • Suse Linux Operating System (continued) • Home Directory Permission Issues • Script Changes Required For Faxing • Echo Emulation Setting – Character Mode • Runaway Tivoli Performance Management Process • Stabilized After 9 Months Of Service • Common Unix Print Server: (CUPS) • Cups Is The Native File & Print Server on Linux • CUPS Is Not Ready for International Language Support • CUPS Print Conversion Filter English Only • Print Drivers for Older and Unusual Devices Are Not Available i.e.: International Check Printers • CUPS Was Removed and Replaced with the LPRNG • Major Rework Issue
Lessons Learned: Pro Dump & Load • Pro Dump & Load Was Acquired For Large Migrations • Tool Worked As Expected Reducing Migration Outage Windows • No Missing Records After Migration • Training Occurred on Live Program Migration & Worked Well • Requires The Progress Editor – Security Issue through 10.1C • No Audit Trail On Progress Editor Changes • Compensating Control Processes Are required. • Bravepoint Made Core Product Changes For Replication Trigger Logic To Support Our Program • Pro Dump & Load Reduced Business Outage Window by 95% • Don’t Skip The Product Training
Lessons Learned: Intel Chipset • Intel Chipset • Initially Skeptical Moving From SPARC Architecture • Concerned About Month End Loads • Results: • Overall Performance Once OS Tuning Was Completed Has Been Outstanding • Month End Processing Load Hasn’t Exceeded 50 of CPU Capacity • Swap Memory Usage Running At 99% • Batch Jobs Completing 50-60 % Faster • Reduction in Server Foot Print In Future • Sizing Rule Of Thumb 40% Of Modern Intel Chipset to 1 Legacy Unix Chipset.
Lessons Learned: Batch Scheduling Autosys • Batch Scheduling Issue: • Native Cron Tab Access Had Been Identified As a Security & Control Risk For The Lack Of Audit Trails on Changes • Alternative: Implement a Batch Scheduling Tool • Autosys From Computer Associates Was Selected • Action: • Moved All Batch Jobs To Autosys • Result • Modified All Batch Job Return Codes To Indicate Success or Failure with Daily Batch Job Status Reporting • Eliminated Audit Comments • Reduced Ongoing Batch Management Support • Increased Customer Satisfaction
Key Points: Program Management • Due Diligence: • Avoid Shortening Upfront Planning Activities • Encourage Cross Functional Participation In Developing & Executing the Plan • Leave Some Flexibility in Plan Dates For Unforeseen Events • Testing & Sizing: • It’s Not Perfect • Migration Methodology • Be Aware of Local Customs When Presenting and Executing The Plan • Never Sacrifice Implementation Quality • Count The Number Of Programs Between Old and New • Make Sure End Users Test Logging In Before Migration
Key Points: Program Management • Security & Controls: • Don’t Forget Encryption Technology and Processes When Moving From The Legacy Environment • Customers May Initially Be Reluctant To Update/ Review Control Documents • End User Security / Role Management Activities Take Time • Communication: • Know Your Customer Before and After Migration • Effective Status Reporting & Escalation Procedures Are Communication Tools to Keep Programs on Track • Cultural Diversity Is An Advantage • Program Execution & Delivery: • Never Get to Comfortable • Traditional Work Days Are Changing - Time Zones Don’t Matter Any Longer
End Results • Program Completed 2 Months Early Under Budget • 3 post Launch Support Issues Encountered • Significant Capacity Improvement • Clear Visibility On Scheduled Batch Jobs (Autosys) • Enhanced Disaster Recovery Capability • Removal Of Legacy QAD Software (7.4, 8.5, 9.0, EB) • Satisfied Customers