.NET and MongoDB

1. Building Applications with NoRM and MongoDB Fairfield/Westchester Code Camp 2010 2010-11-06 John C. Zablocki Development Lead, MagazineRadar Adjunct, Fairfield University .NET and MongoDB

2. Agenda • What is NoSQL? • NoSQL Databases • Introducing MongoDB • .NET Drivers for MongoDB • Introducing NoRM • Case Study: RateMySnippet.com • Questions?

3. NoSQL

4. What is NoSQL? • Coined in 1998 by Carlos Strozzi to describe a database that did not expose a SQL interface • In 2008, Eric Evans reintroduced the term to describe the growing non-RDBMS movement • Broadly refers to a set of data stores that do not use SQL or a relational model to store data • Popularized by large web sites such as Google, Facebook and Digg

5. Wide Column, Key/Value and Document Store... Oh My! NoSQL Databases

6. NoSQL Databases • NoSQL databases come in a variety of flavors • XML (myXMLDB, Tamino, Sedna) • Wide Column (Cassandra, Hbase, Big Table) • Key/Value (Redis, Memcached with BerkleyDB) • Object (db4o, JADE) • Graph (neo4j, InfoGrid) • Document store (CouchDB, MongoDB)

7. Why NoSQL?

8. RDBMSs and the Environment • RDBMS Administrators are highly paid • Highly paid individuals often buy larger than average homes or cars • Larger than average homes and cars require more energy than smaller home and cars • Therefore RDMBSs contribute to global warming more than NoSQL databases which typically do not require the addition of a DBA

9. RDBMSs and the Environment • RDBMSs often require high end servers that are taxing on disks • High end servers consume more electricity than mid-range servers • Taxed disks fail more often than untaxed disks • Therefore RDBMSs require more energy and produce more waste (lots of hard drives in landfills) than NoSQL DBs, which run on mid-range servers.

10. Why NoSQL?

11. NoSQL and Healthcare • The current healthcare crisis requires talented software engineers to fix the outdated or non-existent IT systems of the hospital system • Talented software engineers spend a great deal of time mapping objects to tables in RDBMSs • Talented software engineers are unable to fix healthcare because they are mapping objects to tables • Therefore RDBMSs are causing illnessnes

12. Dropping ACID

13. Dropping ACID • CAP Theorem and Eventual Consistency • A distributed computer system cannot provide Consistency, Availability and Partition Tolerance at the same time • Immediate consistency is expensive • Some systems can tolerate dirty reads • Removing relational storage structure helps to reduce transactional requirements • Inserts/Updates written to fewer places

14. Dropping ACID • ACID support is not required for blogs, status updates, product listings, etc. • Would a 2 second delay in finding out who the mayor of Turnpike Spirits is matter? • Or that your friend is embarrassed to admit that she loves Jersey Shore? • Modern relational databases simply do not scale to Internet proportions • Sites like Digg, Facebook and Ebay have data sets 10s or 100s of TB or even PB large

15. NoSchema Databases • NoSQL databases generally do not impose a formal schema • Optimized storage for unstructured data

16. NoSchema Documents • Related data is stored in a single document • The ubiquitous Blog example • Each post is a document • Post might have reference to Blog or User document or redundantly store blog/user data • East post document has a nested collection of tags and comments • Application layer must be smarter about referential integrity

17. NoSchema Documents varpost = { Author : { Name : "John Zablocki", _id : "497ce96f..." },Title : "On Running NerdDinner on MongoDB with NoRM",Tags : ["mvc", "mongodb", "norm"], CreatedDate : newDate('6-14-2010'),Text : "At the Hartford Code Camp..." }

18. NoSchema Documents <post> <author> <name>John Zablocki</name> <id>497ce96f395f2f052a494fd4</id> </author> <title>On Running NerdDinner on MongoDB - Part 1</title> <tags> <tag>mvc</tag> <tag>mongodb</tag> <tag>norm</tag> </tags> <created_date>6-14-2010</created_date> <text>At the Hartford Code Camp...</text> </post>

19. Something Completely Different

20. Introducing MongoDB • Open source, document-oriented database • 10gen corporate entity behind development • 10gen supports official drivers for many platforms, but not .NET!

21. MongoDB – The Basics • Schema-less documents stored in collections • Documents are stored as BSON (Binary JSON) • JavaScript used to query and manipulate documents and collections • Each document in a collection has a unique BSON ObjectId field named _id • Collections belong to a database

22. MongoDB – Server Features • Indexes • Secondary, compound and unique • Replication • Master/slave and replica sets • Auto-sharding (1.6) • MapReduce • GridFS • Large file storage

23. MongoDB – Hidden Features • MongoDB will help you: • Make more friends • Be more confident in social situations • Be more attractive to members of the opposite (or same) sex

24. MongoDB on Windows

25. Installing MongoDB on Windows • Download the binaries from mongodb.org • Extract to Program Files directory (or wherever) • Create a directory c:\data\db • Run mongod.exe from the command line with the --install switch • See http://bit.ly/aed1RW for some gotchas • To run the daemon without installing, simply run mongod.exe without arguments • Run mongo.exe to verify the daemon is running

26. MongoDB - Shell • The MongoDB interactive JavaScript shell (mongo.exe) is a command line utility for working with MongoDB servers • Allows for CRUD operations on collections • May be used for basic administration • Creating indexes • Cloning databases • Also useful as a test-bed while building apps

27. MongoDB - Shell /*Connect to a server:port/database (defaults are localhost:27017/test ):*/ mongo.exelocalhost:27017/AltNetGroup //Switch database: use CodeCamp //View collections in a database: showcollections //create an index on Name field db.Artists.ensureIndex({ Name : 1 }); //copy one database to another db.copyDatabase("CodeCamp", "AltNetGroup")

28. MongoDB – Shell CRUD //Insert an item into a collection db.Artists.insert({ Name : “TheShins” }); //Find an item in a collection: db.Artists.findOne({ Name: “Radiohead”}); //Find items in a collection: db.Artists.find({ Name : /The/i}); //Count items in a collection db.Artists.count();

29. MongoDB – Shell CRUD //Update an item in a collection db.Artists.update({ Name : “JackJohnson” }, $set : { Name : “JackJohnson” } }); /*Update items in a collection Without $set in this example, whole document would be replaced!*/ db.Artists.update({ Name : { $ne : null } }, { $set : { Category : “Rock” } }, false, true); /*$ denotes special operators and operations $push, $pop, $pull, etc.*/

30. MongoDB – MapReduce • MapReduce is used for batch manipulation and aggregation of data • A Map function transforms a collection into a series of key/value pairs • A Reduce function operates on that key/value pair series to produce output based on those input values • An optional Finalize function is called after the reduce function is invoked

31. MongoDB – Shell MapReduce //Create some data db.Posts.insert( { Name : "On Installing MongoDB as a Service On Windows", Tags : ["mongodb"] }); db.Posts.insert( { Name : "On Running NerdDinner on MongoDB with NoRM", Tags : ["mongodb", "norm", "mvc"] }); db.Posts.insert( { Name : "On A Simple IronPython Route Mapper for ASP.NET MVC", Tags : ["mvc", "ironpython"] });

32. MongoDB – Shell MapReduce /* Create the map function This function creates a view that looks like{ “mvc”, [1, 1] }, { “norm”, [1] }*/ varmap = function() { if (!this.Tags) { return; } for (varindexinthis.Tags) { emit(this.Tags[index], 1); } };

33. MongoDB – Shell MapReduce /* Create the reduce function conceptually, reduce gets called like: reduce("mvc", [1, 1]); reduce("norm", [1] /* varreduce = function(key, vals) { varcount = 0; for (varindexinvals) { count += vals[index]; } returncount; };

34. MongoDB – Shell MapReduce /* Run the mapreduce command on the Posts collection using the map and reduce functions defined above store the results in a collection named Tags */ varresult = db.runCommand( { mapreduce : "Posts", map : map, reduce : reduce, out : "Tags" }); db.Tags.find()

35. .NET and MongoDB

36. .NET and MongoDB • No officially supported 10gen driver • mongo-csharp • Actively being developed and supported • Supports typed collections • Simple-mongodb • Json-centric • Actively developed, but less so than other drivers like mongo-csharp and NoRM

37. NoRM

38. NoRM • OK, so it is pronounced No R M! • Actively developed by Andrew Theken and James Avery, among others • Active community with reliable support • I received help even as I prepared this slide! • Support for typed and untyped collections, MapReduce, Property Mapping, LINQ, Expando, nearly all CRUD operations

39. NoRM – The Basics //Connections managed with IDisposable pattern //Connect to test database on localhost using (IMongomongo=Mongo.Create("mongodb://localhost/test")) { Console.WriteLine(mongo.Database.DatabaseName); } //Mongo instance has MongoDatabase property /*MongoDatabase has GetCollection<T> methods for accessing MongoCollection instances*/ //CRUD operations performed on collections

40. NoRM - CRUD //Class properties map to document structure publicclassArtist { //ObjectId property mapped to _id field in document publicObjectIdId { get; set; } publicstringName { get; set; } //IList property will be mapped to JavaScript Array privateIList<string>_albums=newList<string>(0); publicIList<string>Albums { get { return_albums ; } set { _albums =value; } } }

41. NoRM - CRUD varartist=newArtist() { Name="The Decembrists" }; //Inserting a document into a typed collectionmongo.Database.GetCollection<Artist>("Artists").Insert(artist); //Updating (replacing) a document in a typed collection artist.Name="The Decemberists"; mongo.Database.GetCollection<Artist>("Artists").Save(artist); //Updating a nested collection mongo.Database.GetCollection<Artist>("Artists").UpdateOne( new { Name="The Decemberists"}, new { Albums=M.Push("Picaresque") } );

42. NoRM - CRUD //Find all documents in a typed collection varartists=mongo.GetCollection<Artist>("Artists").Find(); Console.WriteLine(artists.FirstOrDefault().Name); //Query with a document spec varartist=mongo.GetCollection<Artist>("Artists") .FindOne( new { Name="The Decemberists" }); Console.WriteLine(artist.Albums.Count); //Count the documents in a collection longcount=mongo.GetCollection<Artist>("Artists").Count(); Console.WriteLine(count);

43. NoRM - MapReduce //Add a Tags collection added to Artist class privateIList<string>_tags; publicIList<string>Tags { get { return_tags; } set { _tags=value; } } //Add some tags – use Set not PushAll, since Tags was nullmongo.Database.GetCollection<Artist>("Artists").UpdateOne( new { Name="The Decemberists" }, new { Tags=M.Set(newList<string>() { "Alternative", "Folk Rock }) });

44. NoRM - MapReduce //Create map and reduce functons stringmap=@"function() { if (! this.Tags ) { return; } for (index in this.Tags) { emit(this.Tags[index], 1); } }"; stringreduce=@"function(previous, current) { var count = 0; for (index in current) { count += current[index]; } return count; }";

45. NoRM - MapReduce //MapReduce class is responsible for calling mapreduce command MapReducemr=mongo.Database.CreateMapReduce(); //Represents the document passed to the //db.runCommand in the shell example MapReduceOptionsoptions=newMapReduceOptions("Artists") { Map=map, Reduce=reduce, OutputCollectionName="Tags" }; MapReduceResponseresponse=mr.Execute(options); varcollection=mongo.Database.GetCollection<Tag>("Tags"); Console.WriteLine(collection.Count());

46. NoRM - LINQ //LINQ provider exposed via AsQueryable method of MongoCollection varartists=mongo.Database.GetCollection<Artist>("Artists") .AsQueryable(); //Find items in typed collection varartistsWithDecInName=fromainmongo.Database.GetCollection<Artist>().AsQueryable() wherea.Name.Contains("Dec") selecta; Console.WriteLine("First containing Dec in name: "+artistsWithDecInName.First().Name);

47. NoRM - LINQ //Find artists without pulling back nested collections //Note use of Regex name search varartistsWithoutAlbums= fromainmongo.Database.GetCollection<Artist>().AsQueryable() whereRegex.IsMatch(a.Name, "ber", RegexOptions.IgnoreCase) selectnew { Name=a.Name }; Console.WriteLine(artistsWithoutAlbums.First().Name); //Find artists with a given tag varartistsWithFolkRockTag=mongo.Database.GetCollection<Artist>("Artists") .AsQueryable().Where(a=>a.Tags.Any(s=>s=="Folk Rock")); Console.WriteLine(artistsWithFolkRockTag.First().Name);

48. Case Study: RateMySnippet.com

49. Final Thoughts Eat food. Not too much. Mostly plants. - Michael Pollan

50. Final Thoughts Write Code. Not too much. Mostly C#. - John Zablocki