Selecting and Implementing An Embedded Database System

1. Selecting and Implementing An Embedded Database System Presented by Jeff Webb March 2005 Article written by Michael Olson IEEE Software, 2000

2. Overview • Key Strategy – Focus on application requirements • Embedded DB products vary • Some do less than what you need • Some will do more • Choose the tool that most closely matches your needs

3. Overview • After choosing OS, HW platform and DB you must design for reliability • Design for performance up front • Evaluate performance once the application is built

4. Evaluate Database Services Available • High end RDM systems provide: • Concurrency • Transaction processing • Disaster recovery • Although these features may be needed, enterprise systems are seldom a good choice due to: • platform differences • packaging differences

5. Evaluate Database Services Available • Several embedded DB systems use the same techniques as enterprise systems but in smaller packages. • Often, full blown disaster recovery is not needed • Many embedded databases are configurable allowing inclusion or exclusion of services

6. Consider Services Required • Now… consider which services you need. For example: • Locking? • Will run faster without locking • Recovery from failures matter? • Lack of, or disabling will increase performance

7. Operating Systems For Embedded systems • Hundreds of OS for variety of processor hardware • DB developers have an enormous job porting their software to the variety of platforms • Few OS dominate the market • MS Win CE • Neutrino • VxWorks • Wind Rivers

8. Databases For Embedded systems • Classification • Client Server relational systems • Client Server OO systems • Embedded libraries

9. Classification • Client Server relational systems • + Programmers already know SQL • - Extra run-time cost of client server communications

10. Classification • Client Server OO systems • Appear to be a good choice however • Most are designed for Unix systems and their deeply engrained assumptions about memory management and interprocess communications are difficult to port to embedded OS.

11. Classification • Embedded libraries • Created specifically for embedded systems • Provide simple language API that does not require SQL • + Faster execution, increased reliability • - Require developers to master nonstandard programming interfaces

12. Platform Support • Consider combination of: • OS • Processor • Storage system • Exotic processor boards • Rare combinations can be difficult to fit • Often embedded OS vendors maintain lists of partner companies whose products run on their systems

13. Performance Considerations • High concurrency? • Size of database • Multiple control threads? • Can not rely on standard benchmark measuring systems • Evaluation of actual performance is a must

14. Designing the Application • Design for performance • Consider: • Speed • Predictability & Reliability

15. Designing the Application Speed Predictability & Reliability • Data representation • Access patterns • Configuration

16. Designing for SpeedData representation/Access patterns/Configuration • Most embedded DB tools operate on a fixed set of data types • Every fetch and store may require translation • A few systems, mostly library types allow storage in program-native format

17. Designing for Speed Data representation/Access patterns/Configuration • Consider the queries that the application will need to make • Data should be laid out accordingly • Can keys be used that will allow related records to be physically stored together? • B+tree storage typically performs faster than hash table algorithms

18. Designing for Speed Data representation/Access patterns/Configuration • Must understand the chosen systems configuration options • Memory use for secondary cache • Write data to disk or store in memory? • Locking system granularity • Entire locking system on/off • Vendors often choose the wrong defaults

19. Designing the Application • Design for performance • Consider: • Speed • Predictability & Reliability

20. Predictability & Reliability • May need to run with no humans present • Not easy • Fanatically check return values and error indicators • Resource exhaustion • More common in embedded systems • Track and release resources yourself

21. Predictability & Reliability • Are transactions required so that changes won’t be lost after a crash? • The recovery system must be callable from the application program on start up?

22. Performance tuning • Common causes for poor performance • Contention – 2 or more threads contending for same data • Disk-to-memory transfers • Deadlocks

23. Contention/Disk-to-memory transfers/ Deadlocks • When several threads are waiting for the same resources • Use record level locking if possible • Use smaller pages if possible to make page level locks more like record level locks • Touch the Hot data last and hold it for short periods of time

24. Contention/Disk-to-memory transfers/ Deadlocks • Disk latency – mechanical • Flash RAM • More memory for buffer cache • Indexes

25. T2 Obj O2 locked Contention/Disk-to-memory transfers/ Deadlocks T1 waits for a lock on O2 T1 Obj O1 locked T2 waits for a lock on O1

26. Deadlocks • Turn off locking if it is not needed and if the application permits • Break large transactions into several smaller transactions • Write transactions so that they all acquire the same resources in the same order • Consider Optimistic Concurrency Control

27. Price • Helps make final decision • Some are available at no cost • Licensing methods vary • Per developer • Per application using it • Per deployment platform • Per user is less common in embedded systems • During development / deployment

28. Questions