"Ford’s FUSE initiative - Implementing VERITAS products”

1. "Ford’s FUSE initiative - Implementing VERITAS products” Chris Rampson FORD Motor Company crampson@ford.com Mick Evans - VERITAS mevans@veritas.com

2. FORD Motor Company • FORD’s vision: To become the leading CONSUMER company for automotive products and services. • World Headquarters and the main “campus” are located in and around Dearborn MI. • FORD is committed to IT standards, and has its own standards process called ATAD (Architecture for Technology and Development). • FORD’s IT department is THE lowest cost provider (by 2x) over major vendors.

3. FORD Computer Environment • Main Dearborn (MI) Campus area. • Gigabit WAN over fiber • Switched 100baseT to desktop over cat5 • Worldwide - 260 plants and 140 countries. • Over 100,000 PCs. • Over 10,000 UNIX workstations (Sun,HP,SGI). • Over 1500 UNIX servers (Sun,HP). • Over 100Tb of Total FORD data.

4. FORD C3P Process • CAD/CAM/CAE/PIM = C3P • C3P is the main process for creating parts, organizing them and distribution. • Engineer uses SDRC Ideas MasterSeries to design a part. • Part is checked into server storage (TDM). • Oracle database keeps track of part data and metadata. • Metaphase product allows parts to be copied from one server (TDM) to another (TDM). • The C3P upgrade process is well planned and takes place twice per year.

5. VERITAS products at FORD • The C3P Foundation - VxVM, VxFS. • C3P Server Environment - TDM, Metaphase. • C3P Backup - NetBackup. • C3P futures - VCS, SRVM.

6. C3P Foundation - VxVM, VxFS • VxFS - Journalled file system insures quick recovery of very large file systems. • VxFS - Extent based file systems with on-line maintainability ideal for C3P environment. • VxVM - Large partitions requiring high data integrity are very common. • VxVM - Majority of C3P uses RAID 0+1 for increased data availability. • VxVM - Ford’s Enterprise Disk Management software for several years.

7. C3P Server Environment • TDM - SDRC I-DEAS, file access through NFS. • Benchmark results - 34% performance increase using QuickLog (NFS Acc.) • Metaphase - back end database is Oracle. • Quick I/O implementation on core Metaphase/Oracle servers.

8. C3P Backup - NetBackup • All client and server machines backed up using NBU. • C3P data is lifeblood of Ford. Ford chose NBU (site license). • Ford implements standard backup strategy (full and incremental). • Researching Oracle “Hot” backup (7x24).

9. C3P Futures • Disaster Recovery - SRVM, Oracle Database replication. • High Availability - VCS. • Storage Resource Management.

10. FUSE origins • In order to deploy a large client-server project, you need standards for consistency across the enterprise. • The UGC (UNIX Global Client) OS load was developed for UNIX clients. • A snapshot of a working C3P server was used as the server standard (MDSS).

11. MDSS shortcommings • MDSS (Master Data Set Server) was created with ONLY the C3P project in mind. • Limited Scope - Other non-C3P areas also had a need for a standard server load. • Not extensible - It was not designed as a base to build upon - more like a monolithic image. • Solaris Only - We DO have other vendors. • Ad-hoc process - No software engineering on this guy.

12. Ford UNIX Server Environment (FUSE) • We wanted a clean sheet, ground up approach for creating server OS loads. • Our target was the “barely experienced” system admin - we wanted a consistent load process regardless of who performed it. • We wanted a process that could be replicatable/reusable on many different platforms (HP, SUN, Linux, maybe NT . . .) • We wanted to be able to run multiple server “types” (NIS,DNS,TDM,Metaphase,Oracle) on one box - with the admin deciding which ones he needs. • Has to be fast! Minimal downtime

13. FUSE Standards • PERL was chosen as the standard development environment. • CURSES was chosen for the UI. • System V package format was chosen for packaging auxiliary apps. • Application directory structure was formalized. • Root image “snapshot” into dumpfile. • Server “profiles” were developed for each server “type”. • Extra software packages needed. • Special disk voluming instructions. • Kernel parameters needed.

14. FUSE Structure • “Front end” process is PERL code driving CURSES - which asks all relevant questions concerning configuring a server. • Configuration Data is captured into a file. • Disks are “carved” and root drive image file is dumped. • “Back end” loads additional software as stated in the Configuration Data File.

15. FUSE advantages • Reduces complexity - less platform-specific scripts • Reduces cost - 90% of software cost is in maintenance, so less complex = less cost • Easy training - a single interface for all platforms means that an admin trained on a Sun can easily load a HP (or Linux!). • Fast turnaround - anytime a new vendor OS is introduced, there is very little “tweaking” needed to create a new FUSE load. • Standard - FUSE was created with standards (such as Perl), which are closely aligned with the kind of talent you can find in the workforce.

16. Issues/Concerns • FUSE created a need for a common structure across platforms. • Sun Online Disk Suite and Enhanced UFS. • HP LVM and Online JFS. • C3P uses VERITAS Foundation Suite (on Sun). The logical conclusion is to use VERITAS - if only it was available on HP . . .

17. The Solution • We discovered that VERITAS was working on a HP version of Foundation Suite. • FORD agreed to become an official beta test site for the HP VERITAS Foundation Suite. • VERITAS agreed to work closely with FORD to supply scripts in support of the FUSE methodology.

18. VERITAS Foundation Suite beta test for HP • Equipment used: HP A-class, SCSI JBOD, 640Mb RAM • OS level: HP-UX 11.0 12/98 32bit - including • Veritas Patches: • PHCO_17009, PHCO_17453, PHCO_17454, PHCO_17455, PHKL_VXVMTMP, PHSS_17419, PHSS_17535

19. Beta Test Highlights • Basic Functionality. • Enhanced Functionality (VxVM 3.x). • DMP (Dynamic Multi Pathing)

20. Basic Functionality • RAID - 0, 1, 0+1, 5. • Volume/File System manipulation. • Snap shot, hot relocation, etc. • Import/deport disk groups.

21. Enhanced Functionality • RAID 1+0. • On-line relayout. • vmsa - Volume Manager Storage Administrator (new gui). • Icons less intuitive than vxva. • Menu structures better. • Lacks drag-n-drop. • Transition simple - core components unchanged. • LVM Conversion Utility

22. DMP • Utilized DMP configuration test tool from VERITAS Engineering. • Addition of new paths and disks. • Path removal. • Path swapping. • I/O Failover - active/active, active/passive

23. 20/20 Hindsight • What would you have done differently? • I would have volunteered for the HP Foundation Suite beta program much sooner. • What tips can you share with others? • Get the vendor involved early. • Top 3 things you learned. • vmsa isn’t so bad . . . • VxVM has the same look and functionality across platforms. • Install and configuration process identical. • Unexpected: • The beta version of HP Foundation Suite does not support root disk encapsulation.

24. Plans for the Future • What will you do next with VERITAS or your implementation? • Look at product suites, their application at FORD and how they will benefit the environment + its performance. • Current products targeted to be implemented on all servers.

25. Summary • Summarize the business benefits • Standardizing on products + installation configurations saves money in product cost inside of FMC and maintenance costs. • Partnerships between FORD and their vendors and FORD requiring their vendors to partner together, allows high level design and quicker time to market. • Standardization allows for stable products + improved quality of deliverables. • Final words of wisdom • Partnering with best-in-class vendors + standardization of end user tools allows for maintaining costs and controlling quality within the organization.