1 / 56

OCBIG PRESENTATION BI Tool Evaluations A Methodology and Case Studies

“Delivering Strategic Insight”. OCBIG PRESENTATION BI Tool Evaluations A Methodology and Case Studies. Robert Snoyer Managing Partner Steve Jones Managing Partner. April 28, 2005. Agenda. Red Sky Overview An Approach to BI Tool Evaluations Components Methodology Case Study #1 – Telco

teresa
Download Presentation

OCBIG PRESENTATION BI Tool Evaluations A Methodology and Case Studies

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. “Delivering Strategic Insight” OCBIG PRESENTATIONBI Tool EvaluationsA Methodology and Case Studies Robert SnoyerManaging PartnerSteve JonesManaging Partner April 28, 2005

  2. Agenda • Red Sky Overview • An Approach to BI Tool Evaluations • Components • Methodology • Case Study #1 – Telco • Case Study #2 – Health Care government agency • Questions BI Tool Evaluations: A Methodology & Cases Studies

  3. Agenda • Red Sky Overview • An Approach to BI Tool Evaluations • Components • Methodology • Case Study #1 – Telco • Case Study #2 – Health Care government agency • Questions BI Tool Evaluations: A Methodology & Cases Studies

  4. Red Sky Data Provides Solutions Red Sky Datais a Canadian consulting company that specializes in providing: • Business Performance Management (BPM) • Business Intelligence (BI) • and Data Warehouse (DW) solutions The partners of Red Sky Data have provided cutting edge solutions for several decades BI Tool Evaluations: A Methodology & Cases Studies

  5. Partners leave to start DVI-Dec ’98 RSI1989 – Mar ’00 Data VisionsDec ’98 – Jun ’01 CGI Red Sky DataJan ’05 – present CognicaseJun ’01 – Jan ’03 Microforum Acquired by Cognicase-Jun ’01 Acquired by CGI-Jan ’03 Acquired byMicroforum-Mar ’00 Partners form RSD -Jan ’05 consulting group acquired by Cognicase-Sept ’02 Red Sky’s Lineage BI Tool Evaluations: A Methodology & Cases Studies

  6. Key Differentiator -The Team - • 22 data warehouse and business intelligence specialized consultants • Industry-recognized technical experts • Regular presenters at industry seminars • TOUG, IOUG, Informatica World, Oracle World, Master Classes, IRMAC, OCBIG • Certified professionals • Informatica, Oracle, MicroStrategy, Cognos, Microsoft • Dedicated to the client, our organization, and each other BI Tool Evaluations: A Methodology & Cases Studies

  7. Key Differentiator - AdvantageNOW v4.0 - • AdvantageNOW is a framework for delivering BPM / Data Warehouse and Business Intelligence projects on time and on budget • Based on Ralph Kimball and “best practices” • Continuous improvement through real-life team experiences – living framework • Comprised of: • Methods • Loading • Cleansing • Data Retention and Archiving • Reporting • Templates • Project Management Processes (PMI) • Industry specific designs and code examples BI Tool Evaluations: A Methodology & Cases Studies

  8. Solutions Enterprise BI Technology • Data Warehouse (DW) • Data Marts (DM) • Extract, Transform, Load (ETL) • BI Front-End • Meta Data Bus. Focused OLAP Ad-Hoc Query Reporting BPM Dashboard (KPI’s) Visualization • Data Marts (DM) • Extract, Transform, Load (ETL) • BI Front-End • Meta Data ETL Informatica DataStage DTS Services BI Strategic Services Cognos MicroStrategy Business Objects Readiness Assessments Product Evaluations Audits Proof of Concepts Strategies Scope Exercises Training & Education Services Database Mentoring & Training Best Practices & Approaches Workshops Oracle DB2 SQLServer Red Sky Offers Full BI/DW Solutions AdvantageNOW Framework BI Tool Evaluations: A Methodology & Cases Studies

  9. Our BI/DW Expertise By Vertical Government • Cancer Care Ontario • Greater Toronto Airport Authority (GTAA) Fin. Services • Dynamic Mutual Funds • ICICI Bank of Canada Communications Allstream Rogers Cable Systems Rogers Wireless Telus Mobility Carswell Thompson MRD • McDonalds • Toyota • Direct Energy • Cardinal Meat Specialists BI Tool Evaluations: A Methodology & Cases Studies

  10. Agenda • Red Sky Overview • An Approach to BI Tool Evaluations • Components • Methodology • Case Study #1 – Telco • Case Study #2 – Health Care government agency • Questions BI Tool Evaluations: A Methodology & Cases Studies

  11. Components of an Evaluation • Short-Listing • Paper Evaluation • RFP • Vendor Presentations • Proof of Concept • Bake-Off BI Tool Evaluations: A Methodology & Cases Studies

  12. Short-Listing- Steps & Duration - • Develop 5-10 broad criteria for selection (ie in business >10 years) • High level research of the tools on the market for the purposes of answering your evaluation criteria • Prepare your findings Duration: 1 week BI Tool Evaluations: A Methodology & Cases Studies

  13. STRENGTHS Greatly simplifies all future steps WEAKNESSES Possibility of prematurely eliminating a valid contender Short-Listing- Strengths & Weaknesses - BI Tool Evaluations: A Methodology & Cases Studies

  14. Paper Evaluation- Steps & Duration - • Gain a good understanding of this market segment and what makes for a successful BI tool (see Sources later) • Gather the business requirements from your users • Develop the Evaluation Criteria • Conduct in-depth research into the short-listed tools and their capabilities (see Sources later) • Evaluate the tools against your criteria • Prepare your findings Duration: 2-3 weeks BI Tool Evaluations: A Methodology & Cases Studies

  15. Paper Evaluation- Sources - • Organizations / Web-Sites • The Data Warehouse Institute (TDWI) tdwi.org • olapreport.com • datawarehouse.com • Datawarehousing.com • datawarehousingonline.com • Industry Publications • Intelligent Enterprise (intelligenctenterprise.com) • DM Review (dmreview.com) • Vendor Web-Sites BI Tool Evaluations: A Methodology & Cases Studies

  16. Paper Evaluation- Sources - • IT Advisory Groups • Gartner Group • Giga Information Group • Yankee Group • Meta Group • Body of Knowledge • Participation in previous evaluations • Experience with the tools BI Tool Evaluations: A Methodology & Cases Studies

  17. STRENGTHS Quickest turn-around time Least demand on resources ($ and people) WEAKNESSES Doesn’t provide opportunity to see the tools in action (look & feel) Doesn’t provide opportunity to clarify information with the vendors Doesn’t validate the features and functionality Paper Evaluation- Strengths & Weaknesses - BI Tool Evaluations: A Methodology & Cases Studies

  18. RFP- Steps & Duration - • Formalized version of the Paper Evaluation • Same steps as the Paper Evaluation, with the addition of: • Prepare and distribute the RFP document • Provide for a formal Q & A process Duration: 4-8 weeks BI Tool Evaluations: A Methodology & Cases Studies

  19. STRENGTHS Places the onus on the vendor to provide the responses Seen as more impartial and legally binding Opportunity to clarify information with the vendor WEAKNESSES Doesn’t provide opportunity to see the tools in action Doesn’t validate the features and functionality (but are they legally bound anyway?) Additional effort to prepare the RFP (beyond preparing the criteria) RFP- Strengths & Weaknesses - BI Tool Evaluations: A Methodology & Cases Studies

  20. Vendor Presentations • Must be focused - ‘Dog & Pony’ shows not permitted • How do you guarantee this? • Detailed evaluation criteria • Involve both business and technical teams • Clearly communicate the format, criteria and purpose of the presentations to the vendors BI Tool Evaluations: A Methodology & Cases Studies

  21. Vendor Presentation- Steps & Duration - • Prepare the detailed evaluation criteria • 1st presentation: focused on business users and business functionality (‘day in the life of a user’) • Post-presentation review and scoring • 2nd presentation: focused on IT users and technical functionality • Post-presentation review and scoring • Summarize and prepare findings Duration: 4 weeks + (depending on # of vendors) BI Tool Evaluations: A Methodology & Cases Studies

  22. STRENGTHS Opportunity to clarify information with the vendor Opportunity to see the tools in action WEAKNESSES Doesn’t validate the features and functionality As you increase the # of vendors it can become confusing (‘they meld into one’) Increasing demand on resources ($ and people) Vendor Presentations- Strengths & Weaknesses - BI Tool Evaluations: A Methodology & Cases Studies

  23. Proof of Concept • Single vendor brought in to prove their tool meets your functional requirements • Typically done in conjunction with a Paper Evaluation or an RFP BI Tool Evaluations: A Methodology & Cases Studies

  24. Proof Of Concept- Steps & Duration - • Select the vendor for the PoC (Paper Eval, RFP, other) • Develop the criteria and use cases for the PoC • Setup the environment (servers, s/w) • Determine the subject area for PoC • Find/build data model, data sources and business rules • Develop the BI environment (cubes, reports) • Put the tool through its paces • Evaluate the outcome against expected results • Prepare your findings Duration: 2-4 weeks (not including effort for item 1) BI Tool Evaluations: A Methodology & Cases Studies

  25. STRENGTHS Opportunity to see the tool in action Opportunity to clarify information with the vendor Validates the features and functionality WEAKNESSES Doesn’t provide a ‘safety valve’ should the tool not meet your expectations Increasing demand on resources (if it is one of the components of your BI Evaluation process) Proof of Concept- Strengths & Weaknesses - BI Tool Evaluations: A Methodology & Cases Studies

  26. Bake-Off • Just an extension of the PoC • Involves 2-3 vendors putting their tool through the paces • Provides a ‘safety valve’ should one (or two) tools not meet your expectations BI Tool Evaluations: A Methodology & Cases Studies

  27. Bake-Off- Steps & Duration - • Same process as the PoC, with the addition of these steps for the 2nd/3rd tool: • Develop the BI environment (cubes, reports) • Put the tool through its paces • Evaluate the outcome against expected results Duration: 5-8 weeks (not including effort to select the short-listed tools) BI Tool Evaluations: A Methodology & Cases Studies

  28. STRENGHTS Opportunity to see the tool in action Opportunity to clarify information with the vendor validates the features and functionality Provides a ‘safety valve’ WEAKNESSES Greatest demand on resources ($ and people) Bake-Off- Strengths & Weaknesses - BI Tool Evaluations: A Methodology & Cases Studies

  29. Red Sky’s Evaluation Criteria Methodology • Process to Create Evaluation Criteria • Determine the evaluation criteria (70-90) • Determine the ‘key differentiators’ – Type II • Other criteria are the ‘base criteria’ – Type I • Group the criteria into categories • Evaluation Criteria Scoring • Type II Criteria – 5 points (0-5) • Type I Criteria – 1 point (0-1) • Prioritize the top 4 categories and weight them 5-2. All other categories are weighted 1 • Final category scores based on: (Avg. Type I score x 1) + (Avg. Type II score x 5) BI Tool Evaluations: A Methodology & Cases Studies

  30. Red Sky’s PoC/Bake-Off Methodology • Preparation • Identify ‘key differentiators’, ‘base cases’ & categories • Create 3-5 use cases (i.e. build a specific report) • Create an expected results ‘answer sheet’ • Involve the users throughout the process • Execution • Assign an IT resource to work closely with the vendor throughout the build process (learning & assessment) • Eliminate generic BI learning curve by having users evaluate the tools in small, parallel steps • Lots of hands-on time for users BI Tool Evaluations: A Methodology & Cases Studies

  31. Agenda • Red Sky Overview • An Approach to BI Tool Evaluations • Components • Methodology • Case Study #1 – Telco • Case Study #2 – Health Care government agency • Questions BI Tool Evaluations: A Methodology & Cases Studies

  32. Approach • Short-Listing • Vendor Presentations • Bake-Off BI Tool Evaluations: A Methodology & Cases Studies

  33. Short-Listing • Client was very familiar with BI technology and knew what was out there • Joint decision on the vendors was based on the clients knowledge and our experience • Decision was made to evaluate 2 vendors • Duration: < 1 week (1 week was the bench-mark) BI Tool Evaluations: A Methodology & Cases Studies

  34. Vendor Presentations- Steps & Duration - • Prepare the detailed evaluation criteria • 1st presentation: focused on business users and business functionality (‘day in the life of a user’) • Post-presentation review and scoring • 2nd presentation: focused on IT users and technical functionality • Post-presentation review and scoring • Summarize and prepare findings Duration: 6 weeks (4 weeks was the bench-mark) BI Tool Evaluations: A Methodology & Cases Studies

  35. Vendor Presentations - Vendor Evaluation Criteria - • Evaluation Criteria Metrics • Evaluation criteria (75) • ‘key differentiators’ – Type II (30) • ‘base criteria’ – Type I (45) • Criteria categories (12) • 12 Categories • Functionality • Ease of Use/Manageability • Training and Support • Meta Data • Security • Debugging/Error Handling • Deployment Management • Scheduler Capabilities • Change Management • Administration/Operational • Backup and Recovery • Technical Environment BI Tool Evaluations: A Methodology & Cases Studies

  36. Vendor Presentations - Vendor Evaluation Criteria - • Evaluation Criteria Scoring • Type II Criteria – 5 points (0-5) • Type I Criteria – 1 point (0-1) • Prioritize the top 5 categories and weight them 5-2. All other categories are weighted 1 • Final category scores based on: (Avg. Type I score x 1) + (Avg. Type II score x 5) • Top 5 Categories • 5 - Functionality • 4 - Ease of Use/Manageability • 3 - Training and Support • 2 - Meta Data • 2 - Security BI Tool Evaluations: A Methodology & Cases Studies

  37. Vendor Presentations- Sample Detailed Evaluation - • Sample scoring for a category with 4 base criteria and 2 key differentiators • Note how A and B have the same actual score but C has a higher average due to a better key differentiator score BI Tool Evaluations: A Methodology & Cases Studies

  38. Vendor Presentations- Evaluation Summary - BI Tool Evaluations: A Methodology & Cases Studies

  39. Bake-Off- Steps and Duration - • Select the vendors (paper eval, RFP, vendor presentation) • Develop the evaluation criteria and use cases • Setup the environment (servers, s/w) • Determine the subject area for the Bake-Off • Find/build data model, data sources and business rules • Develop the BI environment (1 cube, 3 reports) • Put the tool through its paces • Evaluate the outcome against expected results • Prepare your findings • Duration: 6 weeks (5-8 weeks was the bench-mark) BI Tool Evaluations: A Methodology & Cases Studies

  40. Bake-Off- Evaluation Criteria - • All Type I criteria were eliminated • Type II’s were re-visited to identify the most critical (9 removed, leaving 21) • This reduced the Categories from 12 to 8 • Remaining Categories • Functionality • Ease of Use/Manageability • Meta Data • Security • Deployment Management • Scheduler Capabilities • Administration/Operational • Technical Environment • Eliminated Categories • Change Management • Backup and Recovery • Debugging/Error Handling • Training and Support BI Tool Evaluations: A Methodology & Cases Studies

  41. Bake-Off- Evaluation Criteria - Chose 5 components and weighted them according to importance • 45% - Use Cases • Built to test the 21 Evaluation Criteria • Business – 3/4 • IT – 1/4 • 25% - Product Pricing • 10% - Customer Support & Service • 10% - Development Evaluation • 10% - User’s Personal Experiences 100% BI Tool Evaluations: A Methodology & Cases Studies

  42. Bake-Off - Sample Detailed Bake-Off Evaluation - • The Business Use Cases with the detailed use cases for the Canned Reports Features BI Tool Evaluations: A Methodology & Cases Studies

  43. Bake-Off- Evaluation Summary - 116/180 x 45 = 29 BI Tool Evaluations: A Methodology & Cases Studies

  44. Agenda • Red Sky Overview • An Approach to BI Tool Evaluations • Components • Methodology • Case Study #1 – Telco • Case Study #2 – Health Care government agency • Questions BI Tool Evaluations: A Methodology & Cases Studies

  45. Approach • RFP • Short-Listing • Proof of Proposals (POP’s) – Part 1 • Proof of Proposals (POP’s) – Part 2 BI Tool Evaluations: A Methodology & Cases Studies

  46. RFP • Client was new to BI technology • Very aggressive timelines • Assisted client in preparing RFP and facilitating RFP evaluation process • Defined Mandatory and Rated Requirements • Critical Rated Requirements = “Key Differentiators” Duration: 6 weeks (4-8 weeks was the bench-mark) • 12 Categories • Functionality • Ease of Use/Manageability • Training and Support • Meta Data • Security • Debugging/Error Handling • Deployment Management • Scheduler Capabilities • Change Management • Administration/Operational • Backup and Recovery • Technical Environment BI Tool Evaluations: A Methodology & Cases Studies

  47. Short-Listing • RFP responses evaluated against: • Mandatory (18) • Rated Requirements (140) • Rated Requirements: • Assigned a score (1 - lowest to 5 - highest) • Assigned a weight (1 - lowest to 5 - highest) • Highest possible score for any one requirement: 5x5=25 BI Tool Evaluations: A Methodology & Cases Studies

  48. Rated Requirements Scoring- Sample - BI Tool Evaluations: A Methodology & Cases Studies

  49. Short-Listing - Con’t • Financial responses evaluated=TCO over 5 years: • Initial price per component • Annual support and maintenance • Development, Test, Production environments • Best Value Determination=Overall Weighted Score • 80% - Rated Requirements scores • 20% - Price • Top 3 vendors selected to proceed to PoP process Duration: < 1 week BI Tool Evaluations: A Methodology & Cases Studies

  50. Best Value Determination- Example - BI Tool Evaluations: A Methodology & Cases Studies

More Related