Why QlikView/QlikTech is the Fastest Growing BI Company in the World

1. Why QlikView/QlikTech is the Fastest Growing BI Company in the World By Neil Hepburn (Dir. of Education, IRMAC) This presentation by IRMAC is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 2.5 Canada License.Based on a work at wikipedia.org.

2. Speaker Bio • Information Management practice consultant for PwC Canada (PriceWaterhouseCoopers LLP), specializing in Agile Analytics and Data Quality • 20 years experience in IS/IT, both in full time and external consulting capacities with a focus on Data Management over past 7 years • Have worked for very large Fortune 500 organizations as well as small start-ups and mid sized companies across numerous industries • Education: • Honours Bachelor of Mathematics in Computer Science from the University of Waterloo • Certified Data Management Professional (Mastery Level) • PMI Certified (CAPM) • Director of Education for IRMAC (Toronto chapter of DAMA-I) • Advised Wall Street equity analyst on technology company, prior to IPO in 2010 • Routinely involved with small companies and not-for-profits in spare time: • GM of marketing for: Innovative iPhone App for Internet Radio Discovery • Data architect for Institute of Bioforest Technology

3. Presentation Roadmap • A brief history of QlikTech • How Traditional BI works • How QlikView works • What is Agile BI/Agile Analytics • Demo of QlikView • How QlikView works internally? • Criticism of QlikView • Novel Features of QlikView • QlikView Competition • QlikViewblindspots • What are the implications for OLAP and the data warehouse

4. A Brief History of QlikTech • Founded in Lund, Sweden in 1993 by Björn Berg and StaffanGestrelius originally as a consultancy • Originally called “QuikView” as in “Quality, Understanding, Interaction, Knowledge” • Product was designed to mimic the way the brain works • A key aspect was the colour-coding scheme whereby selected values are highlighted in green, linked values in white, and excluded values in highlighted grey • First two versions were basically written in Excel using VLOOKUPs • HåkanWolgé was later hired as lead software engineer to re-architect/re-write QlikView from the ground up as an in-memory application • Renamed as “QlikView” in 1996 • IPOed on Nasdaq in 2010 under symbol “QLIK” and had 7th best IPO of 2010 • Now has over 24,000 customers in 100 countries and employs over 1,000 people worldwide • Market cap: $2.5 billion

5. How Traditional BI tools works • Traditional OLAP/cube technologies primarily provide the ability to drill up and down through “dimension” hierarchies, allowing the end-user to see pre-aggregated “measures” • Dimensions and measures must be know a priori • A small team is usually required to complete a BI project • A data warehouse or data mart is usually required as a pre-requisite before OLAP cubes can be built • This can often lie on the critical path of other data warehouse projects. Since data warehouse usage cannot be anticipated, a “single version of the truth” can often bog down development • ETL is very slow to test, which in turn slows down development time • If a detail drill down report (e.g. to see all point-of-sale records), a “drill through” query link is made to the operational data store to retrieve these data • Introduces another point-of-failure • Associations between dimensions are not computed – only resulting measures (e.g. counts)

6. How QlikView works (cont’d) • The “secret sauce” is: An experienced QlikView can build and test a dashboard solution (including user acceptance testing) faster than any other BI tool I have evaluated • This makes “Agile BI” possible • Users and developers can remain focused on insights and outcomes • The resulting dashboards are effectively by-products of the analysis process • More flexible data model allows normalized data to be imported with fewer transformations • ETL development is in-memory. ETL jobs can be tested orders of magnitude faster than traditional ETL tools • All data is automatically profiled on import • QlikView uses the word “associative” to distinguish itself from other BI vendors • Associative is a tricky concept to explain, but most people will “get it” when they see it • “Associative” puts emphasis on understanding how sets of data relate to one another • All those tricky SQL queries involving “NOT EXISTS” or “LEFT/RIGHT OUTER JOIN” are but a mouse click away

7. How QlikView Works • QlikView uses the word “associative” to distinguish itself from other BI vendors • Associative is a tricky concept to explain, but most people will “get it” when they see it • “Associative” puts emphasis on understanding how sets of data relate to one another • All those tricky SQL queries involving “NOT EXISTS” or “LEFT/RIGHT OUTER JOIN” are but a mouse click away

8. Traditional BI workflow

9. QlikView workflow

10. What is Agile BI/Agile Analytics • Fulfills the promise that Agile Software Development failed to • Very little software development is truly “green field”/”blue sky” • Functional requirements are inflexible • Once a shared data architecture is in place, impact analysis must be performed, breaking the Agile methodology. Basically all Agile brings to the table is improved UX (User Experience), since presentation layers are the only thing that can quickly be modified without extensive impact analysis • Agile BI is not about software. It’s about business insights and business outcomes • Dashboards are a by-product of Agile BI • Agile BI iterations are typically bi-daily, with a single Agile BI developer/analyst performing all technical functions • A separate warehouse is not required, a “single version of the truth”, and other data quality issues should not always be a roadblock • Occasionally specialized tools are required (e.g. fuzzy matching, cluster discovery, etc.)

11. QlikView Demo Time

12. How Does QlikView Work Internally? (Cont’d) • QlikView does not reveal the specifics of its inner workings. However, the following gives us clues: • From Curt Monash’s DBMS2 blog: “The main ingredient of the performance secret sauce in QlikView is that selections are compiled straight into machine code. (QlikTech gave me the impression that this post is the first time that will be publicly revealed.)” • We can also look at their main patent, with HåkanWolgélisted as the inventor. This is the first part of their first, and most important claim. Note, the “final multi-dimensional cube”

13. How Does QlikView Work Internally? • At the centre of QlikView is a large “Multi-Dimensional Cube Table”, with one column for each table, and each row containing pointers back to the original table’s row index • Also uses a: Global Symbol Table; Value Tables; and Data Tables • The “machine code” most likely refers to bitmap indexes. QlikView heavily relies on bitmap indexes to perform its JOINs • QlikView may have the best known solution to Kimball’s “Big JOIN” problem (JOINing a billion dimensions with a trillion facts), since a single row is effectively being represented by a single bit • Consider that a 64 rows can be JOINed in less than a clock cycle • Intel and AMD now support “Active Vector Extensions” (AVX), which will allow 256 rows to be JOINed in less than a clock cycle • Unclear if this architecture lends itself to map/reduce • The embedded example shows in detail how the indexes work

14. Criticism of QlikView • QlikView is the biggest threat to established BI vendors. Not surprisingly, there is more criticism directed to QlikView than any other product. • Some criticism is valid, but most of it either misunderstands the product or distorts the truth • Criticism #1: You can’t fit very much data in memory • Used to be true. When when 32-bit OSes were the norm, upper limit of 20 GB uncompressed data • Now I can buy an HP Integrity Superdome 2 /w 4 TB RAM, and load 40 TB of uncompressed data – about a years worth of call-detail-record data for a major Canadian telecom • Criticism #2: QlikView forces you to rename foreign and/or primary key columns to be the same • This is true, QlikView relies on Natural Joins. This is what you what you want as it leads to a more intuitive [or Natural] user experience. The effort to rename columns is negligible • Criticism #3: QlikView stores data in proprietary files • QlikView now supports an open QVX format with a published spec and SDK • Third party tools (e.g. Expressor – an ETL tool) integrate with the QVX format

15. Novel Features of QlikView (3rd party visualization platform) • 3rd party Visualization platform allows 3rd party developers to develop custome extensions

16. Novel Features of QlikView (3rd party visualization platform) • Associative Experience (already discussed) • Advanced Excel transformations. Allows quick prep of Excel data. E.g. filling in merged cells • AND/NOT selections (as opposed to OR) • Semantic model support • Define relationship and inverse relationship. E.g. locale all predecessors and successors • Collaborative BI. Similar to two-way desktop sharing or Google Docs sharing. • Can be done through web browser with no client plug-in required. • Client-side data access • Can access and analyze Google Analytics data via client browser, simplifying security • Metadata can be imported and linked to attributes

17. QlikView Competition • Only true competitor is Microsoft Power Pivot • Available as free plug-in for Excel 2010 and can be deployed in SharePoint 2010 • Started as Project Gemini, which was announced 21 months in advanced – the farthest out for any MS project • MS has done their best to mimic QlikView’s associative experience • Will now be rolling out “Power View” as part of SQLServer 2012 SSRS • Other vendors have greatly simplified the cube/OLAP approach, and can be considered somewhat Agile, although they lack the “Associative” experience. Primarily: • Tableau • TIBCO SpotFire • Many vendors have jumped on the “in memory” bandwagon, but ultimately have just moved their existing cubes “in memory” – effectively just speeding up user interaction, but offering nothing new in terms of user experience or development timelines • Some “big data” analytical DB vendors (e.g. SAP HANA) are feigning competition with QlikView – but none of these get to the “last mile” of user experience

18. QlikView blindspots • Raw calculation engine has never been completely exposed • It is possible to integrate QlikView server objects into other web applications (e.g. SharePoint) • It is also possible use client-side JavaScript to extract data from charts • However there is no pure data interface into QlikView, apart from the QVX interface, which is not the same

19. What are the implications for OLAP and the data warehouse? • No longer need to maintain star schemas • The data warehouse is going through a transition, and will likely be much simpler to maintain • Bitemporal data types, which can already be found in TeraData and DB2, and have been ratified in ISO SQL:2011 will handle all issues related to Slowly Changing Dimensions, and other time related issues (e.g. when data was loaded vs. when original transaction occurred) • Change Data Capture tables should be used to load data warehouse • Data quality, de-duplication, and fuzzy matching should be treated as operational issues, e.g. fuzzy matching tables should be maintained operationally • Dashboard schemas will be built in tools like QlikView, as needed. • Star and snowflake schemas are still useful, but should be built as needed on-the-fly • The data warehouse should more-or-less be a time invariant mirror of the ODS, and more-or-less maintain itself