NoSQL Databases

1. NoSQL Databases

2. Introduction • The NoSQL movement • Key–value stores • Tuple and document stores • Column-oriented databases • Graph-based databases • Other NoSQL categories

3. The NoSQL Movement • RDBMSs put a lot of emphasis on keeping data consistent. • Entire database is consistent at all times (ACID) • Focus on consistency may hamper flexibility and scalability • As the data volumes or number of parallel transactions increase, capacity can be increased by • Vertical scaling: extending storage capacity and/or CPU power of the database server • Horizontal scaling: multiple DBMS servers being arranged in a cluster

4. The NoSQL Movement • RDBMSs are not good at extensive horizontal scaling • Coordination overhead because of focus on consistency • Rigid database schemas • Other types of DBMSs needed for situations with massive volumes, flexible data structures, and where scalability and availability are more important  NoSQL databases

5. The NoSQL Movement • NoSQL databases • Describes databases that store and manipulate data in formats other than tabular relations, i.e., non-relational databases (NoREL) • NoSQL databases aim at near-linear horizontal scalability by distributing data over a cluster of database nodes for the sake of performance as well as availability • Eventual consistency: the data (and its replicas) will become consistent at some point in time after each transaction

6. The NoSQL Movement

7. Key–Value Stores • Key–value-based database stores data as (key, value) pairs • Keys are unique • Hash map, or hash table or dictionary

8. Key–Value Stores importjava.util.HashMap; importjava.util.Map; publicclassKeyValueStoreExample { publicstaticvoid main(String... args) { // Keep track of age based on name Map<String, Integer> age_by_name = newHashMap<>(); // Store some entries age_by_name.put("wilfried", 34); age_by_name.put("seppe", 30); age_by_name.put("bart", 46); age_by_name.put("jeanne", 19); // Get an entry intage_of_wilfried = age_by_name.get("wilfried"); System.out.println("Wilfried's age: " + age_of_wilfried); // Keys are unique age_by_name.put("seppe", 50); // Overrides previous entry } }

9. Key–Value Stores • Keys (e.g., “bart”, “seppe”) are hashed by means of a so-called hash function • A hash function takes an arbitrary value of arbitrary size and maps it to a key with a fixed size, which is called the hash value • Each hash can be mapped to a space in computer memory

10. Key–Value Stores • NoSQL databases are built with horizontal scalability support in mind • Distribute hash table over different locations • Assume we need to spread our hashes over three servers • Hash every key (“wilfried”, “seppe”) to a server identifier • index(hash) = mod(hash, nrServers) + 1

11. Key–Value Stores Sharding!

12. Key–Value Stores • Example: Memcached • Implements a distributed memory-driven hash table (i.e., a key–value store), which is put in front of a traditional database to speed up queries by caching recently accessed objects in RAM • Caching solution

13. Key–Value Stores importjava.util.ArrayList; importjava.util.List; importnet.spy.memcached.AddrUtil; importnet.spy.memcached.MemcachedClient; publicclassMemCachedExample { publicstaticvoid main(String[] args) throws Exception { List<String> serverList = newArrayList<String>() { { this.add("memcachedserver1.servers:11211"); this.add("memcachedserver2.servers:11211"); this.add("memcachedserver3.servers:11211"); } };

14. Key–Value Stores MemcachedClientmemcachedClient = newMemcachedClient(AddrUtil.getAddresses(serverList)); // ADD adds an entry and does nothing if the key already exists // Think of it as an INSERT // The second parameter (0) indicates the expiration - 0 means no expiry memcachedClient.add("marc", 0, 34); memcachedClient.add("seppe", 0, 32); memcachedClient.add("bart", 0, 66); memcachedClient.add("jeanne", 0, 19); // SET sets an entry regardless of whether it exists // Think of it as an UPDATE-OR-INSERT memcachedClient.add("marc", 0, 1111); // <- ADD will have no effect memcachedClient.set("jeanne", 0, 12); // <- But SET will

15. Key–Value Stores // REPLACE replaces an entry and does nothing if the key does not exist // Think of it as an UPDATE memcachedClient.replace("not_existing_name", 0, 12); // <- Will have no effect memcachedClient.replace("jeanne", 0, 10); // DELETE deletes an entry, similar to an SQL DELETE statement memcachedClient.delete("seppe"); // GET retrieves an entry Integer age_of_marc = (Integer) memcachedClient.get("marc"); Integer age_of_short_lived = (Integer) memcachedClient.get("short_lived_name"); Integer age_of_not_existing = (Integer) memcachedClient.get("not_existing_name"); Integer age_of_seppe = (Integer) memcachedClient.get("seppe"); System.out.println("Age of Marc: " + age_of_marc); System.out.println("Age of Seppe (deleted): " + age_of_seppe); System.out.println("Age of not existing name: " + age_of_not_existing); System.out.println("Age of short lived name (expired): " + age_of_short_lived); memcachedClient.shutdown(); } }

16. Key–Value Stores • Request coordination • Consistent hashing • Replication and redundancy • Eventual consistency • Stabilization • Integrity constraints and querying

17. Request Coordination • In many NoSQL implementations (e.g., Cassandra, Google’s BigTable, Amazon’s DynamoDB), all nodes implement the same functionality and are all able to perform the role of request coordinator • Need for membership protocol • Dissemination • Based on periodic, pairwise communication • Failure detection

18. Consistent Hashing • Consistent hashing schemes are often used, which avoid having to remap each key to a new node when nodes are added or removed • Suppose we have a situation in which ten keys are distributed over three servers (n = 3) with the following hash function: • h(key) = key modulo n

19. Consistent Hashing

20. Consistent Hashing • At the core of a consistent hashing setup is a so-called “ring”-topology, which is basically a representation of the number range [0,1]:

21. Consistent Hashing

22. Consistent Hashing • Hash each key to a position on the ring, and store the actual key–value pair on the first server that appears clockwise of the hashed point on the ring

23. Consistent Hashing • Because of the uniformity property of a “good” hash function, roughly 1/n of key–value pairs will end up being stored on each server • Most of the key–value pairs will remain unaffected in the event that a machine is added or removed

24. Consistent Hashing

25. Replication and Redundancy • Problems with consistent hashing: • If two servers end up being mapped close to one another, one of these nodes will end up with few keys to store • In the case that a server is added, all of the keys moved to this new node originate from just one other server • Instead of mapping a server s to a single point on our ring, we map it to multiple positions, called replicas • For each physical server s, we hence end up with r (the number of replicas) points on the ring • Note: each of the replicas still represents the same physical instance (↔ redundancy) • Virtual nodes

26. Replication and Redundancy • To handle data replication or redundancy, many vendors extend the consistent hashing mechanism so that key–value pairs are duplicated across multiple nodes • e.g., by storing the key–value pair on two or more nodes clockwise from the key’s position on the ring

27. Replication and Redundancy • It is also possible to set up a full redundancy scheme in which each node itself corresponds to multiple physical machines, each storing a fully redundant copy of the data

28. Eventual Consistency • Membership protocol does not guarantee that every node is aware of every other node at all times • it will reach a consistent state over time • State of the network might not be perfectly consistent at any moment in time, though will become eventually consistent at a future point in time • Many NoSQL databases guarantee so-called eventual consistency

29. Eventual Consistency • Most NoSQL databases follow the BASE principle • Basically Available, Soft state, Eventual consistency • CAP theorem states that a distributed computer system cannot guarantee the following three properties at the same time: • Consistency (all nodes see the same data at the same time) • Availability (guarantees that every request receives a response indicating a success or failure result) • Partition tolerance (the system continues to work even if nodes go down or are added)

30. Eventual Consistency • Most NoSQL databases sacrifice the consistency part of CAP in their setup, instead striving for eventual consistency • The full BASE acronym stands for: • Basically Available: NoSQL databases adhere to the availability guarantee of the CAP theorem • Soft state: the system can change over time, even without receiving input • Eventual consistency: the system will become consistent over time

31. Stabilization • The operation which repartitions hashes over nodes in case nodes are added or removed is called stabilization • If a consistent hashing scheme is being applied, the number of fluctuations in the hash–node mappings will be minimized.

32. Integrity Constraints and Querying • Key–value stores represent a very diverse gamut of systems • Full-blown DBMSs versus caches • Only limited query facilities are offered • e.g. put and set • Limited to no means to enforce structural constraints • DBMS remains agnostic to the internal structure • No relationships, referential integrity constraints, or database schema can be defined

33. Tuple and Document Stores • A tuple store is similar to a key–value store, with the difference that it does not store pairwise combinations of a key and a value, but instead stores a unique key together with a vector of data • Example: • marc -> ("Marc", "McLast Name", 25, "Germany") • No requirement to have the same length or semantic ordering (schema-less!)

34. Tuple and Document Stores • Various NoSQL implementations do, however, permit organizing entries in semantical groups (aka collections or tables) • Examples: • Person:marc -> ("Marc", "McLast Name", 25, "Germany") • Person:harry -> ("Harry", "Smith", 29, "Belgium")

35. Tuple and Document Stores • Document stores store a collection of attributes that are labeled and unordered, representing items that are semi-structured • Example: { Title = "Harry Potter" ISBN = "111-1111111111" Authors = [ "J.K. Rowling" ] Price = 32 Dimensions = "8.5 x 11.0 x 0.5" PageCount = 234 Genre = "Fantasy" }

36. Tuple and Document Stores • Most modern NoSQL databases choose to represent documents using JSON{ "title": "Harry Potter", "authors": ["J.K. Rowling", "R.J. Kowling"], "price": 32.00, "genres": ["fantasy"], "dimensions": { "width": 8.5, "height": 11.0, "depth": 0.5 }, "pages": 234, "in_publication": true, "subtitle": null }

37. Tuple and Document Stores • Items with keys • Filters and queries • Complex queries and aggregation with MapReduce • SQL after all …

38. Items with Keys • Most NoSQL document stores will allow you to store items in tables (collections) in a schema-less manner, but will enforce that a primary key be specified • e.g. Amazon’s DynamoDB, MongoDB ( _id ) • A primary key will be used as a partitioning key to create a hash and determine where the data will be stored

39. Filters and Queries importorg.bson.Document; importcom.mongodb.MongoClient; importcom.mongodb.client.FindIterable; importcom.mongodb.client.MongoDatabase; importjava.util.ArrayList; importstaticcom.mongodb.client.model.Filters.*; importstaticjava.util.Arrays.asList; publicclassMongoDBExample { publicstaticvoid main(String... args) { MongoClientmongoClient = newMongoClient(); MongoDatabasedb = mongoClient.getDatabase("test"); // Delete all books first db.getCollection("books").deleteMany(new Document()); // Add some books db.getCollection("books").insertMany(newArrayList<Document>() {{ add(getBookDocument("My First Book", "Wilfried", "Lemahieu", 12, new String[]{"drama"})); add(getBookDocument("My Second Book", "Seppe", “vandenBroucke", 437, new String[]{"fantasy", "thriller"})); add(getBookDocument("My Third Book", "Seppe", “vandenBroucke", 200, new String[]{"educational"})); add(getBookDocument("Java Programming", "Bart", "Baesens", 100, new String[]{"educational"})); }});

40. Filters and Queries // Perform query FindIterable<Document> result = db.getCollection("books").find( and( eq("author.last_name", “vandenBroucke"), eq("genres", "thriller"), gt("nrPages", 100))); for(Document r : result) { System.out.println(r.toString()); // Increase the number of pages: db.getCollection("books").updateOne( newDocument("_id", r.get("_id")), newDocument("$set", newDocument("nrPages", r.getInteger("nrPages") + 100))); } mongoClient.close();} publicstatic Document getBookDocument(String title, String authorFirst, String authorLast, intnrPages, String[] genres) { returnnew Document("author", new Document() .append("first_name", authorFirst) .append("last_name", authorLast)) .append("title", title) .append("nrPages", nrPages) .append("genres", asList(genres));}}

41. Filters and Queries Document{{_id=567ef62bc0c3081f4c04b16c, author=Document{{first_name=Seppe, last_name=vanden Broucke}}, title=My Second Book, nrPages=437, genres=[fantasy, thriller]}}

42. Filters and Queries // Perform aggregation query AggregateIterable<Document> result = db.getCollection("books") .aggregate(asList( new Document("$group", new Document("_id", "$author.last_name") .append("page_sum", new Document("$sum", "$nrPages"))))); for (Document r : result) { System.out.println(r.toString()); } Document{{_id=Lemahieu, page_sum=12}} Document{{_id=Vanden Broucke, page_sum=637}} Document{{_id=Baesens, page_sum=100}}

43. Filters and Queries • Queries can still be slow because every filter (such as “author.last_name = Baesens”) entails a complete collection or table scan • Most document stores can define a variety of indexes • unique and non-unique indexes • compound indexes • geospatial indexes • text-based indexes

44. Complex Queries and Aggregation with MapReduce • Document stores do not support relations • First approach: embedded documents { "title": "Databases for Beginners", "authors": ["J.K. Sequel", "John Smith"], "pages": 234 } { "title": "Databases for Beginners", "authors": [ {"first_name": "Jay Kay", "last_name": "Sequel", "age": 54}, {"first_name": "John", "last_name": "Smith", "age": 32}], "pages": 234 } BUT: Data duplication!

45. Complex Queries and Aggregation with MapReduce • Second approach: create two collections book collection: { "title": "Databases for Beginners", "authors": ["Jay Kay Rowling", "John Smith"], "pages": 234 } authors collection: { "_id": "Jay Kay Rowling", "age": 54 } BUT: Need to resolve complex relational queries in application code!

46. Complex Queries and Aggregation with MapReduce • Third Approach: MapReduce • A map-reduce pipeline starts from a series of key–value pairs (k1,v1) and maps each pair to one or more output pairs • The output entries are shuffled and distributed so that all output entries belonging to the same key are assigned to the same worker (e.g., physical machines) • Workers then apply a reduce function to each group of key–value pairs having the same key, producing a new list of values per output key • The resulting, final outputs are then (optionally) sorted per key k2 to produce the final outcome

47. Complex Queries and Aggregation with MapReduce • Example: get a summed count of pages for books per genre • Create a list of input key–value pairs • Map function is a simple conversion to a genre–nrPages key–value pair function map(k1, v1) emitoutput record (v1.genre, v1.nrPages) end function

48. Complex Queries and Aggregation with MapReduce • Workers have produced the following three output lists, with the keys corresponding to genres • Aworking operation will be started per unique key k2, for which its associated list of values will be reduced • e.g., (education,[120,200,20]) will be reduced to its sum, 340 function reduce(k2, v2_list) emit output record (k2, sum(v2_list)) end function

49. Complex Queries and Aggregation with MapReduce • Final output looks like this: • Can be sorted based on k2 or v3

50. Complex Queries and Aggregation with MapReduce • Suppose we would now like to retrieve an average page count per book for each genre • Reduce function becomesfunctionreduce(k2, v2_list)emitoutput record (k2, sum(v2_list) / length(v2_list)) end function • After mapping the input list, workers produce the following three output lists