Dr. Vishal Sikka Chief Software Architect SAP AG

1. Data Management in Enterprise Apps: Some Perspectives Dr. Vishal Sikka Chief Software Architect SAP AG

2. A Brief Introduction to SAP and Data Management in Our Applications The Current Situation: Some Existing and Emerging “Divides” Our Approach to Two of These Divides The Lessons Learned and Some Open Problems

3. Widgets Portal • Mobile • Duet • Forms Voice RSS RSS Embedded RFID • SAP GUI • Project Muse • Dashboards CRM SRM PLM SCM mySAP ERP SAP Legacy Home Grown / ISV IndustryStandards Bizpartner Bizpartner SAP at a Glance • Who we are • Founded in 1972 • 2005 revenues: €8.5 Billion • 34,600+ customers • 37,500+ employees • 12+ Million users in 120 countries • 1,600+ partners • What we do • Largest enterprise applications company in the world • Serve most back-end and front-end business processes • Leader in ERP, CRM, SCM, … • Leading platform to build and run apps on • 25+ industry solutions Other Composites SAP Composites SAP NetWeaver Data Infrastructure Infrastructure

4. Our data management requirements are massive mySAP ERP HCM Customer with payroll calculations for 500,000 employees in 3 hours SAP for Engineering & Construction Customer with 5,000 concurrent active users SAP for Consumer Products Customer with 1.4 million sales order line items per day SAP NetWeaver Portal Customer with 300,000 users (20,000 concurrent) mySAP SCM Customer with 4.5 million characteristic combinations & 512 GB memory in live cache mySAP Business Suite SAP for Utilities 25 million business partners – 85 million service and sales orders per year SAP NetWeaver BICustomer with 40 TB database live Average DB size of top 10 live BI customers: 5.5TB mySAP ERP A customer with 5 users on a laptop

5. There is >10 PB of transactional and analytical data processed by SAP apps worldwide We are the largest applications consumer and reseller of data worldwide Our data is of many different types, shape and sizes: Transactional, Analytical, Text/Unstructured, Master, Events, … Data has different requirements & different optimizations Significant need for deriving value from this data Data Management from SAP’s Perspective SAP Applications Unstructured Event Transactional Master Analytical

6. Data through the SAP Lens – “Not All Data Is Alike” Progression Over Time Textual and Unstructured Data Analytical Data Transactional Data Event Data Master Data • Order ~ 100G • Write > read • Many changes • Accurate • Consistent • Performance • All back-end apps • Order > Tb • Read only • Slow changes • Many queries • Flexibility • Performance • Order ~ 1G • Mostly read • Mid change • Many queries • Distributed • Order < Tb • Many writes • Few queries • Distributed • Filtering • Correlation • Order > Tb • Mostly read • Slow change • Many queries • Unstructured • Contextual

7. 3-tier C/S Architecture of Basis: Our Application Server

8. Memory Management in Basis outside the DBMS • Buffers in the application server help significantly improve performance. In a classical 3-tier system, network round trips mitigated benefits of the DBMS cache, while TCO optimization required one DB for >10+ app servers. • Application level locking (Enqueue and Application LUW) mitigates the absence of fine granularity of locking in DBMS and transaction support needed by Application Servers (multiple users accessing the same DB, complex screen processing with workflow on front-end). • Numerous other optimizations and DB abstractions.

9. Bringing Data Closer to Applications: SAP LiveCache • LiveCache is a main-memory DB component used in SAP SCM’s APO • Rapid Planning Matrix in the Automotive Industry • Common ERP system: Plan the mfg of 20,000 Cars / Day • Needed volumes are much higher • liveCache enables planning 500,000 Cars / Hour • Demand Planning (DP): • Interactive planning: 10x performance gain compared to DB based solution • Consistent storage of data (no need for aggregation/disaggregation batch jobs) • Production Planning (PP/DS): • Performance gain of 15x in rescheduling production runs and DS heuristics • Data volume 5x higher in planning board compared to common ERP system • Consolidation of data structures via generic liveCache data types: • E.g. 1 order data type 1 order type with multiple attributes instead of a few dozen different specific order types in ERP • Bringing development teams closer together • LiveCache applications team bridges technology knowledge with business process knowledge by working together with the application team on the usage of the liveCache, as well as in optimization of business logic. • Common team working together for several years  3000+ happy deployments.

10. A Brief Introduction to SAP and Data Management in Our Applications The Current Situation: Some Existing and Emerging “Divides” Our Approach to Some of These Divides The Lessons Learned and Some Open Problems

11. New needs: Innovate, Be flexible, Stay high-performant “ Once my system is up and running, you, SAP, can touch my core processes once every 5 years ... and it needs to be a Saturday … and my CEO wants me to innovate every quarter” CIO, Fortune 1000 Manufacturing Company

12. More decoupled business processes More visible Physical-Digital divide Infrastructure subjected to much higher volumes (events, sensors, …) Greater need for in-context usage Multiple UIs More visible work-personal divide Users are a lot more used to search, lack of structure is academic to them Different requirements on front-end than on back-end e.g. easier front-end application composition Many more deployment options Greater flexibility  easy integration, better components semantics • Widgets Portal • Mobile • Duet • Forms Voice Other Composites SAP Composites RSS RSS Embedded RFID • SAP GUI • Project Muse • Dashboards CRM PLM SCM SRM mySAP ERP SAP NetWeaver SAP Legacy Home Grown / ISV IndustryStandards Bizpartner Bizpartner Data Infrastructure Infrastructure New requirements, New “divides” New application architectures are necessary: SOA is the biggest component, but there are others

13. Technology Shifts • Architectural Shift • Technology Drivers • 1990 • 2006 • 1990 • 2006 • Improvement • 0.05 • MIPS/$ • 7.15 • MIPS/$ • 143x • Disk based data storage • Simple consumption of applications (Fat client UI, EDI) • General-purpose, application-agnostic database • In-memory data stores • Multi-channel UI, high event volume, cross industry value chains • Application-aware and intelligent data management • CPU • 0.02 • MB/$ • 5 • MB/$ • 250x • Memory • 64 • Bits • 2x • 16 • Bits • 48 • Addressable Memory • 10 • Gbps • 100x • 100 • Mbps • Network Speed • 5 • Kilo RPM • 3x • 15 • Kilo RPM • Disk • Speed

14. A Brief Introduction to SAP and Data Management in Our Applications The Current Situation: Some Existing and Emerging “Divides” Our Approach to Two of These Divides The Lessons Learned and Some Open Problems

15. Any source, any tool legacy Addressing DB Architecture Gap: SAP BI Accelerator Performance 1 Billion records analyzed in 3 seconds Delivery Off the shelf hardware, appliance setup Predictability Consistent response, no tuning, fast load Integration Built for & closely integrated with SAP NW BI

16. Addressing DB Architecture Gap: SAP BI Accelerator • Performance 1 Billion records analyzed in 3 seconds • Affordability Off the shelf hardware, appliance setup • Agility Consistent response, no tuning, fast load • Integration Closely integrated with SAP BI

17. BI Accelerator Key Technology BI Application Server SAP BI AppServer SAP BI Accelerator • Main memory technology • Inspired by text search • On the fly aggregation • L2 cache miss optimization • Column based data structures • Highly compressed, dictionary based, golomb, sparse, ... • Fast updates with write-optimized delta mechanism • Compressed data structures for read access • Parallel and distributed execution engine • Distributed joins, horizontal table split • Intelligent partitioning (along join paths) • Data distribution optimizer • Model based data layer • Exploit data model for performance optimization and data distribution Storage subsystem Database Server Scalability by adding blades

18. Key Benefits • Predictable (near constant) query response time • Query execution shifted from DB to BI Accelerator • Fast in memory full table scans guarantee stable response times • Column based data structures support fast joins • Intelligent partitioning and data distribution allows massive parallelization • Reduced maintenance costs • Simplified cube modeling (normalization for semantic reasons only) • No more aggregates (or aggregate administration) • Less need for DB optimization • Reduced hardware costs • Commodity hardware (blades) with standard equipment • Linear scalability with number of processors / cores • Use of blade infrastructure instead of big SMP box • Packaged as an appliance

19. Search in the enterprise Business objects Business context awareness Role Authorizations, Compliance Current work context Graceful degradation with decreasing structure Multiple clients Stand alone and embedded into applications Integration into non-SAP sources SAP Enterprise Search is a stand alone business search xApp and a framework for search as a service SAP Enterprise Search Portal Devices Desktop Office SAP Enterprise Search SAP NetWeaver Business Process Platform DesktopSearchService InternetSearchService R/3 via BAPI’s Search Indexing my SAPBus.Suite 3rdparty Docu-ments

20. SAP Enterprise Search • Access more information from any place • Get the right answer to enterprise questions anywhere, anytime • Access data from your workplace or mobile device. • Simple to use: Open to everyone • Pre-build common queries • Smart context • Better Answers:Leverage context information and meta data • Support targeted search for object types • Enhance search and displays by contextual meta data: related queries, object scoping • Go Deep:Find the right information – Across all your sources • Penetrate entire corporate data sources including Search for documents and business objects simultaneously • Ensure service-oriented, multi-device scalable operation • Reach Out:Embed search into everyday tools • Design simple search front ends that are compliant to the respective devices, including Portal, Desktop, SMS, e-mail, mobile

21. The Argo Widget

22. Enterprise Search Example

23. Enterprise Search Example (Cont’d)

24. A Brief Introduction to SAP and Data Management in Our Applications The Current Situation: Some Existing and Emerging “Divides” Our Approach to Some of These Divides The Lessons Learned and Some Open Problems

25. Master Data Management • Characterized By • Business Entities with • Multiple data models • Multiple application sources • Reference Models • Single logical model • Multiple physical models • Source of Truth • No single source of truth • Access Characteristics • Serves as reference data • Few systems write • Many systems read • 360 ° view of data • Full analytics view • Full operational view Master Data Management Architecture MDM Application Services Quality Visibility Governance Validation Analytics Meta-dataMaster Unified Data Management Layer Distributed Query Data Federation Multiple Data Source Management Data Mappings Legacy Data Unstructured Data Structured Data Connectivity Fabric Events Services

26. Event Processing • Characterized By • Continuous Streams of near real-time data • High data flow rate and large volume needs parallel processing • Significant main memory processing • Continuous evaluation of rules • Edge Devices as data producers • (RFID, sensor data) generate significant number of events • orders of magnitude scale data e.g., shop floor sensor devices • Large volume of event data dictates pre-processing for consumption • Events externalized non-invasively for several forms of consumption • Automatic correlation and context determination of business events Business Events Actions Query Results BI/Reports Alerts Event Streams Data (IN) Output Streams Input Streams Event Management Response Filters Correlation Engine Correlation Rules Event Memory/Storage

27. Lessons Learned • It’s not the technology, stupid. Application perspectives provide grounding for data management.  So learn what the apps needs are • One size does not fit all. Applications’ data mgmt needs are changing and this requires a rethink in data mgmt architecture.  So let’s go rethink data mgmt for the enterprise