F# and NoSQL Introducing FoSQL

1. John C. Zablocki Development Manager, HealthcareSource Organizer, Beantown ALT.NET New England F# User Group 2011-09-12 F# and NoSQLIntroducing FoSQL

2. Agenda • NoSQL Overview • MongoDBBasic Concepts • MongoDB Shell • MongoDBC# Driver • CouchDB Basic Concepts • CouchDB and cURL • LoveSeat • NoSQLDesign Considerations • Questions?

3. NoSQL Not Only SQL

4. What is NoSQL? • Coined in 1998 by Carlos Strozzi to describe a database that did not expose a SQL interface • In 2008, Eric Evans reintroduced the term to describe the growing non-RDBMS movement • Broadly refers to a set of data stores that do not use SQL or a relational data model • Popularized by large web sites such as Google, Facebook and Digg

5. NoSQL Databases

6. NoSQL Databases • NoSQL databases come in a variety of flavors • XML (myXMLDB, Tamino, Sedna) • Wide Column (Cassandra, Hbase, Big Table) • Key/Value (Redis, Memcached with BerkleyDB) • Object (db4o, JADE) • Graph (neo4j, InfoGrid) • Document store (CouchDB, MongoDB)

7. Introducing MongoDB

8. MongoDB - Concepts • Schema-less documents stored in collections • Documents are stored as BSON (Binary JSON) • JavaScript used to query and manipulate documents and collections • Each document in a collection has a unique BSON ObjectId field named _id • Collections belong to a database

9. Installing MongoDB on Windows • Download the binaries from mongodb.org • Extract to Program Files directory (or wherever) • Create a directory c:\data\db • Run mongod.exe from the command line with the --install switch • See http://bit.ly/aed1RW for some gotchas • To run the daemon without installing, simply run mongod.exe without arguments • Run mongo.exe to verify the daemon is running

10. MongoDB - Shell • The MongoDB interactive JavaScript shell (mongo.exe) is a command line utility for working with MongoDB servers • Allows for CRUD operations on collections • May be used for basic administration • Creating indexes • Cloning databases • Also useful as a test-bed while building apps

11. MongoDB - Shell /* This script file demonstrates the basics of MongoDB from the interactive shell.    It's not intended to be a best-practive example for anything! */ /*Connect to a server:port/database (defaults are localhost:27017/test ):*/mongo.exelocalhost:27017/AltNetGroup//Switch database:use CodeCamp//View collections in a database:showcollections//create an index on Name fielddb.Posts.ensureIndex({ Name : 1 });//copy one database to anotherdb.copyDatabase("CodeCamp", "AltNetGroup")

12. MongoDB - Shell //create a documentvarpost = { Title: "On Installing MongoDB as a Service on Windows" }//insert a document, if the collection doesn't exist it's createddb.Posts.insert(post);//verify that the document was createddb.Posts.find();//write a query to find a post with a valid titlevarquery = { Title: { $ne: null} }//use that query to find the postvarpost = db.Posts.findOne(query);//this line will actually set the content after pressing enterpost.Content = "When installing MongoDB as a service on Windows..."

13. MongoDB - Shell //update the content to include an author using collection update method db.Posts.update( { Title : "On Installing MongoDB as a Service on Windows" }, { Author : "John Zablocki" } ) //check that the post was updated db.Posts.findOne() //where'd my document go? updates are in place, replacing entire docs! //need to use the $set operator to update partial documents - start over //first remove the new document. Notice remove takes a function argument. //find and findOne also accept functions as arguments db.Posts.remove(function (e) { returnthis.Author == "John Zablocki" }) //rerun the first statements up to but not including the db.Posts.update(... db.Posts.update({ Title: "On Installing MongoDB as a Service on Windows" }, { $set: { Author: "John Zablocki" } }) //verify that the update worked db.Posts.findOne()

14. MongoDB - Shell //add two more tags db.Posts.update( { Title: "On Installing MongoDB as a Service on Windows" }, { $pushAll: { Tags: ["windows", "nosql"] } }) //add another post db.Posts.insert( { Author : "John Zablocki", Title : "On MapReduce in MongoDB", Tags: ["mongodb", "nosql"] }) //verify that last insert worked db.Posts.findOne(function (e) { returnthis.Title.indexOf("MapReduce") != -1; })

15. MongoDB - Shell //add a "like" counter to the post. The booleanarguments tell //updatenot to insert if the document doesn't exist and to //update all documents, not just one respectively db.Posts.update({ Author: "John Zablocki" }, { $set: { Likes: 0} }, false, true) //increment the likes counter for the mapreduce article db.Posts.update({ Title: /mapreduce/i }, { $inc: { Likes: 1} }) //check that the counter was incremented db.Posts.findOne({ Title: /mapreduce/i }).Likes

16. MongoDB - Shell //use MapReduce to get counts of the tagscreate the map and reduce functionsvarmap = function() {if (!this.Tags) { return; } for (varindexinthis.Tags) { emit(this.Tags[index], 1);                 }            };//conceptually, reduce gets called like://reduce("mvc", [1, 1]);//reduce("norm", [1]varreduce = function(key, vals) {varcount = 0;for (varindexinvals) { count += vals[index];                 }returncount;             };/*

17. MongoDB - Shell run the mapreduce command on the Posts collectionusing the map and reduce functions defined abovestore the results in a collection named Tags*/varresult = db.runCommand(    {mapreduce : "Posts",map : map,reduce : reduce,out : "Tags"   });db.Tags.find()

18. MongoDB - Shell //first, insert some datadb["UserActions"].insert({ Username : "jzablocki", Action : "Login"})db["UserActions"].insert({ Username : "jzablocki", Action : "Login"})db["UserActions"].insert({ Username : "jzablocki", Action : "Login"})db["UserActions"].insert({ Username : "jzablocki", Action : "PasswordChange"})db["UserActions"].insert({ Username : "mfreedman", Action : "PasswordChange"})db["UserActions"].insert({ Username : "mfreedman", Action : "PasswordChange"})db["UserActions"].insert({ Username : "mfreedman", Action : "Login"})//now run the group bydb.UserActions.group(     { key :  { Username : true, Action : true },cond : null,reduce : function(doc, out) { out.count++; },initial: { count: 0 }});

19. Something Completely Different

20. MongoDB C# Driver • 10gen developed and supported • Consists of two primary components, a BSON serializer and the MongoDB driver • Support for typed and untyped collections, MapReduce, and all CRUD operations • Currently lacking a LINQ provider • Current version (as of 5/5/11) is 1.0.4098.x

21. MongoDB C# Driver – The Basics //MongoServer manages access to MongoDatabaseletmongoServer=MongoServer.Create("mongodb://localhost:27017")//MongoDatabase used to access MongoCollection instancesletmongoDatabase=mongoServer.GetDatabase("FSUG");//reference the collectionletmongoCollection=mongoDatabase.GetCollection<Artist>(COLLECTION)letdoSetup=//drop collection before running samples    ifmongoCollection.Exists()thenmongoCollection.Drop()

22. MongoDB C# Driver - CRUD //Insert a document into a typed collection let artist ={ Name ="The Decembrists"; Genre ="Folk"; Id =ObjectId.GenerateNewId(); Albums =[]; Tags =[]}mongoCollection.Insert(artist)|> ignore//Updating (replacing) a document in a typed collectionletupdatedArtist={ artist with Name ="The Decemberists"}mongoCollection.Save(updatedArtist)|> ignore//Updating a nested collectionmongoDatabase.GetCollection<Artist>(COLLECTION).Update(Query.EQ("Name",BsonValue.Create("The Decemberists")),Update.PushAll("Albums",BsonArray.Create(["Picaresque";"Hazards of Love";"The Crane Wife"])))|> ignore

23. MongoDB C# Driver - CRUD //Find all documents in a typed collectionlet artists =mongoDatabase.GetCollection<Artist>(COLLECTION).FindAll()Console.WriteLine("Artist name: "+artists.FirstOrDefault().Name)//Query with a document speclet artist =mongoDatabase.GetCollection<Artist>(COLLECTION) .FindOne(Query.EQ("Name",BsonValue.Create("The Decemberists")))Console.WriteLine("Album count: {0}",artist.Albums.Count())//Count the documents in a collectionlet count =mongoDatabase.GetCollection<Artist>(COLLECTION).Count()Console.WriteLine("Document count: {0}", count)

24. MongoDB C# Driver - Queries let artists =mongoDatabase.GetCollection<Artist>(COLLECTION)//Find items in typed collectionletartistsStartingWithFoo=artists.Find(Query.Matches("Name",BsonRegularExpression.Create(newRegex("foo",RegexOptions.IgnoreCase))))Console.WriteLine("First artist starting with Foo: {0}",artistsStartingWithFoo.First().Name);//Find artists without pulling back nested collectionsletartistsWithDecInTheName=artists.Find(Query.Matches("Name",BsonRegularExpression.Create("Dec"))).SetFields("Name");Console.WriteLine("First artist with dec in name: {0}",artistsWithDecInTheName.First().Name);//Find artists with a given tagletartistsWithIndieTag=artists.Find(Query.In("Tags",BsonArray.Create(["Indie"])))Console.WriteLine("First artist with indie tag: "+artistsWithIndieTag.First().Name);

25. MongoDB C# Driver - MapReduce //Add some tagsmongoDatabase.GetCollection<Artist>(COLLECTION).Update(Query.EQ("Name",BsonValue.Create("The Decemberists")),Update.PushAll("Tags",BsonArray.Create(["Folk rock";"Indie"])))|> ignorelet artist ={ Name ="Foo Fighters";                   Genre ="Hard rock";                   Albums =mutableListAddRange(["The Colour and the Shape";"Wasted Light"]);                   Tags =mutableListAddRange(["Hard Rock";"Grunge"]); Id =ObjectId.Empty}mongoDatabase.GetCollection<Artist>(COLLECTION).Save(artist)|> ignore

26. MongoDB C# Driver - MapReduce //Create map and reduce functonslet map =@"function() {                    if (!this.Tags ) { return; }                    for (index in this.Tags) { emit(this.Tags[index], 1); }                    }";let reduce =@"function(previous, current) {var count = 0;                        for (index in current) { count += current[index]; }                        return count;                    }";let result =mongoDatabase.GetCollection<Artist>(COLLECTION).MapReduce(BsonJavaScript.Create(map),BsonJavaScript.Create(reduce),MapReduceOptions.SetKeepTemp(true).SetOutput(MapReduceOutput.op_Implicit("Tags")))let collection =mongoDatabase.GetCollection<Tag>(result.CollectionName);Console.WriteLine("Tag count: {0}",collection.Count())

27. MongoDB C# Driver - GroupBy //add one more artist for good measurelet artists =mongoDatabase.GetCollection<Artist>(COLLECTION)let artist ={ Name ="The Fratellis"; Genre ="Rock"; Id =ObjectId.GenerateNewId(); Albums =mutableListAddRange(["Costello Music"]); Tags =new List<string>()}artists.Insert(artist)|> ignorelet reduce =BsonJavaScript.Create("function(obj, out) { out.count += obj.Albums.length; }")letgroupBy=mongoDatabase.GetCollection<Artist>(COLLECTION).Group(Query.Null,GroupBy.Keys("Name"),newBsonDocument("count", BsonInt32.Create(0)), reduce,null)for item ingroupBydoConsole.WriteLine("{0}: {1} Album(s)",item.GetValue(0),item.GetValue(1));

28. Introducing CouchDB

29. About CouchDB • Open source, Apache supported project • Document-oriented database • Written in Erlang • RESTfulAPI (POST/PUT/GET/DELETE) for managing CouchDB: • Servers • Databases • Documents • Replication

30. CouchDB - Concepts • Schema-less documents stored as JSON • RESTful API for database and document operations (POST/PUT/GET/DELETE) • Each document has a unique field named “_id” • Each document has a revision field named “_rev” (used for change tracking) • Related documents may have common “type” field by convention – vaguely analogous to collections or tables

31. Design Documents • Design Documents represent application boundaries (users, blogs, posts, etc.) • Used to define views, shows, lists and validation functions, attachments, etc. • Views allow for efficient querying of documents • Show and List functions allow for efficient document and view transformations • Validation functions place constraints on document creation

32. Sample Design Document { "_id": "_design/artist", "validate_doc_update" : "function(newDoc, oldDoc) { if (!newDoc.name) { throw({ forbidden : 'Name is required'}); } }", "shows" : { "csv" : "function(doc, req) { return doc._id + ',' + doc.name }" }, "views": { "all" : { "map" : "function(doc) { emit(null, doc) }" }, "by_name" : { "map" : "function(doc) { emit(doc.name, doc) }" }, "by_name_starts_with" : { "map" : "function(doc) { var match = doc.name.match(/^.{0,3}/i)[0]; if (match) { emit(match, doc) } }" }, "by_tag" : { "map" : "function(doc) { for(i in doc.tags) { emit(doc.tags[i], doc) } }" }, }, "lists" : { "all_csv" : "function(head, row) { while(row = getRow()) { send(row.value._id + ',' + row.value.name + '\\r\\n'); } }" } }

33. Installing CouchDB on Windows • Download an installer from https://github.com/dch/couchdb/downloads • Download curl at http://curl.haxx.se/download/curl-7.19.5-win32-ssl-sspi.zip, unzip and set path • Run the following from the command linecurl.exe http://127.0.0.1:5984 • If all is running, response should be {“couchdb” : “Welcome”, “version”, “1.1.0”} • Check out http://wiki.apache.org/couchdb/Quirks_on_Windows for some gotchas

34. cURL Basics • cURL is an open source, command line utility for transferring data to and from a server • cURL supports all common Internet protocols, including SMTP, POP3, FTP, IMAP, GOPHER, HTTP and HTTPS • Examples: • curl http://www.bing.com • curl –F email=test@live.comhttp://www.live.com • curl –X GET http://www.bing.com?q=couchdb

35. cURL and CouchDB • Check server version • curl http://localhost:5984 • Create database • curl –X PUT http://localhost:5984/albums • Delete database • curl –X Delete http://localhost:5984/cds • Get a UUID • curl http://localhost:5984/_uuids • Create document • curl –X POST http://localhost:5984/albums -d “{ \”artist\” : \”The Decembrists\” }” –H “Content-Type: application-json” • Get document by ID • curl http://localhost:5984/artists/a10a5006d96c9e174d28944994042946

36. CouchDB - Futon

37. CouchDB - Futon • Futon is a simple web admin for managing CouchDB instances and is accessible at http://127.0.0.1:5984/_utils/ • Used for setting server configuration • Allows for database administration (create/delete, compact/cleanup, security) • Allows for CRUD operations on documents • Creating and testing views • Creating design documents

38. CouchDB .NET Client Libraries • SharpCouch – simple CouchDB wrapper and GUI client. Last commit 2008 • Divan – Nearly API complete. Some LINQ support. Last commit 2010 • Relax – Built with CQSR consideration. Complex library. Recent commit (May 2011)

39. LoveSeat • Document, View, List and Show API complete. • Fluent HTTP API for non-implemented API features, such as creating design documents • Support for strongly typed documents, using generics and Type convention • Last commit August 2011 by jzablocki.

40. CouchDBLoveSeat – The Basics let DESIGN_DOC ="artist"let DATABASE ="vtcodecamp"//database names cannot have uppercase charactersletcouchClient=newCouchClient("127.0.0.1",5984,null,null)letcouchDatabase=couchClient.GetDatabase(DATABASE)couchDatabase.SetDefaultDesignDoc(DESIGN_DOC)letdoSetup=ifcouchClient.HasDatabase(DATABASE)thencouchClient.DeleteDatabase(DATABASE)|> ignorecouchClient.CreateDatabase(DATABASE)|> ignore

41. CouchDBLoveSeat – Design Doc //Create map and reduce functons for tag countsvar design = string.Format(@"{{ ""_id"": ""_design/artist"",                            ""all"" : {{                                    ""map"" : ""function(doc) {{ emit(null, doc) }}""                                }},                                ""by_name"" : {{                                    ""map"" : ""function(doc) {{ emit(doc.name, doc) }}"" }});varrequest=newCouchRequest("http://127.0.0.1:5984/music/_design/artist”);var response = request.Put().Form() .ContentType("multipart/formdata") .Data(JObject.Parse(design)).GetResponse();

42. CouchDBLoveSeat - CRUD //Create POCO instancelet artist =newArtist(Guid.NewGuid(),"The Decembrists",null,[],[],["Boston";"Boston";"Hartford";"Burlington"])//Inserting a document into a typed collection - GUID Id will be created prior insert in property, not by driverlet result =couchDatabase.CreateDocument(new Document<Artist>(artist))//Updating (replacing) a document in a typed collection//after creating document, document revision id is in result, but POCO not updatedletupdatedArtist=newArtist(artist.Id,"The Decemberists",result.Last.Last.ToString(),artist.Albums,artist.Tags,artist.TourStops)let foo =couchDatabase.SaveDocument(new Document<Artist>(updatedArtist))    DATABASE

43. Design Considerations • Your object graph is your data model • Don't be afraid to store data redundantly • Your graph might be redundant! • Not everything has to fit in 1 document • Don't be afraid to store aggregate statistics with a document.

44. Object Graph as Data Model • Generally speaking, most MongoDB drivers will serialize an object graph as a single document • The relationships of your classes creates an implied schema! • Migrating this schema is not trivial if you are trying to deserialize properties that did not or no longer exist • Consider use cases carefully to avoid inefficiently structured documents • Projection queries will be your friend

45. Redundant Data is OK • Optimize documents for quick reads and writes • Your application layer will have to maintain referential integrity! • If every time you access a Post document, you need some of an Author document's data, store that data with Post • Design simple classes for this redundant data for reusability (see AuthorInfo in Meringue)

46. Don’t Stuff it All In One Document • Nothaving formal relationships does not mean throwing away relationships • Consider a user and his or her logged actions • The user would likely have a User class/doc with properties for name, email, etc. • User actions are generally write heavy and read out of band. • Don't clutter user documents - create a separate collection for user actions

47. Don't Be Afraid of Extra Data • The schema-less nature of documents makes it easy to store meta data about that document – particularly aggregate data • Consider a blog post with a rating feature • Each rating would be stored as a nested document of the post • Rather than compute vote totals and averages real time, simply add these properties to the document and update on writes

48. Final Thoughts Eat food. Not too much. Mostly Plants. - Michael Pollan

49. Final Thoughts Write code. Not too much. Mostly C#. - John Zablocki

50. Links • http://dllHell.net - my blog • http://www.CodeVoyeur.com- my code • http://www.linkedin.com/in/johnzablocki • http://twitter.com/codevoyeur • http://mongodb.org - Official MongoDBsite • http://couchdb.org- Official CouchDB site • http://bitbucket.org/johnzablocki/meringue • http://bitbucket.org/johnzablocki/codevoyeur-samples • http://about.me/johnzablocki