Introduction to the GAE Datastore

1. CS569 Selected Topics in Software EngineeringSpring 2012 Introduction to theGAE Datastore

2. Suppose you want to harness lots of machines to manage data • Brewer’s CAP theorem says you can only have 2 of the following 3 properties: • Consistency (all machines see the same values) • Availability (all requests must be handled) • Partition tolerance (messages might be lost) • Scaling up RDBMS is based on a “master” with several replicated “slaves” • With very poor partition tolerance • And therefore limited ability to scale out

3. Other annoying things about RDBMS • Generally tuned for small, rare writes • Bulk loads can take forever • All of the rows in a table have to have the same schema • Making it difficult to add and remove columns

4. Imagine this instead… • An enormous table • With multiple rows per entity • One row per attribute of the entity • Only two columns • One for a key that includes the type of the entity and the name of a member variable • One for the value of that variable • And we store different parts of the table on different machines on a fast network

5. That’s the general idea of the data storage behind the GAE Datastore • Can be implemented with Google’s BigTable • Which also includes compression, intelligent grouping of rows, timestamps, type metadata, … • Wrapped with Java Data Object (JDO) layer • 1. Instantiate object • 2. Tell DataStore, “Save this for me till later” • 3. Use SQL-like queries to retrieve objects • This “Not Only SQL” (noSQL) approach is easily 100 times more scalable than RDBMS

6. How to use GAE datastore • Configure the datastore with XML files • Instantiate a PersistenceManagerFactory • Hold that factory reference in memory • Ask the factory for a connection to datastore • Instantiate or query for some objects • Set values on those objects • Save the objects • Close the connection

7. Connecting to the datastore package edu.oregonstate.mobilecloud; import javax.jdo.JDOHelper; import javax.jdo.PersistenceManagerFactory; public final class PMF { private static final PersistenceManagerFactorypmfInstance = JDOHelper.getPersistenceManagerFactory("transactions-optional"); public static PersistenceManagerFactorygetPMF() {return pmfInstance;} private PMF() {} } Instantiate factory Hold reference in static variable In JSP or servlet, request a manager (connection) Remember to close when finished (use try/finally) <%@ page import="javax.jdo.PersistenceManager" %> <%@ page import="edu.oregonstate.mobilecloud.PMF" %> <% PersistenceManagerpm = PMF.getPMF().getPersistenceManager(); try { out.write(""+pm); } finally { pm.close(); } %> test1.jsp

8. Defining an object that can be stored package edu.oregonstate.mobilecloud.lectures.clouddatastore; import javax.jdo.annotations.PersistenceCapable; import javax.jdo.annotations.Persistent; import javax.jdo.annotations.PrimaryKey; @PersistenceCapable public class Test2Simple { @PrimaryKey @Persistent private String title; public String getTitle() { return title != null ? title : ""; } public void setTitle(String title) { this.title = title; } } Must be PersistenceCapable Declare a PrimaryKey that is Persistent (details later)

9. Saving an object <%@ page import="javax.jdo.PersistenceManager" %> <%@ page import="edu.oregonstate.mobilecloud.PMF" %> <%@ page import="edu.oregonstate.mobilecloud.lectures.clouddatastore.*" %> <% PersistenceManager pm = PMF.getPMF().getPersistenceManager(); try { Test2Simple obj = new Test2Simple(); obj.setTitle("an object: " + Math.random()); pm.makePersistent(obj); } finally { pm.close(); } %> Get a connection Instantiate the object makePersistent Close connection test2.jsp

10. Inspecting the datastore • Go to http://localhost:8888/_ah/admin/ • Browse through the data • The Full Key will be shown (more details later) • Blobs / clobs will not be shown

11. Saving more fields package edu.oregonstate.mobilecloud.lectures.clouddatastore; … @PersistenceCapable public class Test3Members { @Persistent private intcoursenum; @Persistent private float difficulty; @Persistent private java.util.Date creation; @Persistent private com.google.appengine.api.datastore.Text description; // … getters and setters follow Members can be ints, longs, floats, Strings (< 500 chars), Text objects (clobs), serializable objects.

12. Compound objects… serializable children • @PersistenceCapable • public class Test4Compound { • @Persistent(serialized = "true") • private Test4Child serializableChild; • …. Parent class declares that one of its members is serialized. Child class declares that it is serializable. import java.io.Serializable; @SuppressWarnings("serial") public class Test4Child implements Serializable { …

13. Compound objects… arrays of children • package edu.oregonstate.mobilecloud.lectures.clouddatastore; • import java.util.ArrayList; • import java.util.Date; • import javax.jdo.annotations.IdGeneratorStrategy; • import javax.jdo.annotations.PersistenceCapable; • import javax.jdo.annotations.Persistent; • import javax.jdo.annotations.PrimaryKey; • @PersistenceCapable • public class Test4Compound { • @Persistent • private ArrayList<Date> importantDates = new ArrayList<Date>(); • public ArrayList<Date> getImportantDates() { • return importantDates; • } • public void setImportantDates(ArrayList<Date> importantDates) { • this.importantDates = importantDates; • } Parent should use ArrayList, HashSet, or TreeSet (or interface/abstract class equivalent)

14. Changing the schema(aka “soft schemas”) • Not all entities of a certain kind must have the same member variables • If you add members to a class, instances loaded from the Datastore will lack those members. • So in this case, use nullable Object types (such as Long, Integer, Float) when you add members • If you delete members to a class, it’s ok. • Do not change the type of members. It is bad.

15. Object keys • Full Key = app engine application ID + separator + entity kind + separator + entity group + separator + entity ID key • Actually, each part is hashed before concatenation • Typical ways to specify the entity ID key • You set it yourself as a String (or Long) • You let GAE automatically assign a Long • You let GAE automatically set an encoded String • You set it yourself as a custom Key object

16. Setting your own String as a key • package edu.oregonstate.mobilecloud.lectures.clouddatastore; • import javax.jdo.annotations.PersistenceCapable; • import javax.jdo.annotations.Persistent; • import javax.jdo.annotations.PrimaryKey; • @PersistenceCapable • public class Test2Simple { • @PrimaryKey • @Persistent • private String title; • public String getTitle() { • return title != null ? title : ""; • } • public void setTitle(String title) { • this.title = title; • } • } Declare @PrimaryKey Before trying to makePersistent, be sure to set the value Any existing entity of that type with that key is overwritten

17. Letting GAE automatically assign a Long key package edu.oregonstate.mobilecloud.lectures.clouddatastore; import javax.jdo.annotations.IdGeneratorStrategy; import javax.jdo.annotations.PersistenceCapable; import javax.jdo.annotations.Persistent; import javax.jdo.annotations.PrimaryKey; @PersistenceCapable public class Test3Members { @PrimaryKey @Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY) private Long id; public Long getID() { return id; } Declare @PrimaryKey Specify IDENTITY generator type Do not try to assign the ID when instantiating object

18. Listing all entities of a certain kind <%@ page import="javax.jdo.Query" %> <%@ page import="javax.jdo.PersistenceManager" %> <%@ page import="edu.oregonstate.mobilecloud.PMF" %> <%@ page import="edu.oregonstate.mobilecloud.lectures.clouddatastore.*" %> <%@ page import="java.util.*" %> <%@ page import="edu.oregonstate.mobilecloud.PMF" %> <% PersistenceManager pm = PMF.getPMF().getPersistenceManager(); try { Query query = pm.newQuery(Test3Members.class); List<Test3Members> allTest3 = (List<Test3Members>)query.execute(); for (Test3Members obj : allTest3) { out.write("<B>"+obj.getDepartment()+obj.getCoursenum() +": difficulty "+obj.getDifficulty()+"</B>"); out.write("<div style='height:100; “ +”overflow-y:auto'>"+obj.getDescription()+"</div>"); out.write("<BR><BR><BR>"); } } finally { pm.close(); } %> Create Query Execute Cast to List of objects test5.jsp

19. Retrieving a subset of entities based on a value <%@ page import="javax.jdo.Query" %> <%@ page import="javax.jdo.PersistenceManager" %> <%@ page import="edu.oregonstate.mobilecloud.PMF" %> <%@ page import="edu.oregonstate.mobilecloud.lectures.clouddatastore.*" %> <%@ page import="java.util.*" %> <%@ page import="edu.oregonstate.mobilecloud.PMF" %> <% PersistenceManager pm = PMF.getPMF().getPersistenceManager(); try { Query query = pm.newQuery(Test3Members.class, "department == :dd && coursenum >= :cn"); List<Test3Members> allTest3 = (List<Test3Members>)query.execute("CS", 400); for (Test3Members obj : allTest3) { out.write("<B>"+obj.getDepartment()+obj.getCoursenum()+": difficulty "+obj.getDifficulty()+"</B>"); out.write("<div style='height:100; overflow-y:auto'>"+obj.getDescription()+"</div>"); out.write("<BR><BR><BR>"); } query.closeAll(); } finally { pm.close(); } %> Create Query with parameters Bind parameter values on execute Cast to List of objects Close query when done test5a.jsp

20. Retrieving an object by key <%@ page import="javax.jdo.Query" %> <%@ page import="javax.jdo.PersistenceManager" %> <%@ page import="edu.oregonstate.mobilecloud.PMF" %> <%@ page import="edu.oregonstate.mobilecloud.lectures.clouddatastore.*" %> <%@ page import="java.util.*" %> <%@ page import="edu.oregonstate.mobilecloud.PMF" %> <% PersistenceManagerpm = PMF.getPMF().getPersistenceManager(); try { Query query = pm.newQuery(Test3Members.class); query.setOrdering("id desc"); List<Test3Members> allTest3 = (List<Test3Members>)query.execute(); long maxId = allTest3.get(0).getID(); Test3Members highest = pm.getObjectById(Test3Members.class, maxId); out.write(highest+"..."+highest.getID()); query.closeAll(); } finally { pm.close(); } %> Use getObjectById to retrieve If object might not exist, catch JDOObjectNotFoundException or JDOException test5b.jsp

21. Deleting entities <%@ page import="javax.jdo.Query" %> <%@ page import="javax.jdo.PersistenceManager" %> <%@ page import="edu.oregonstate.mobilecloud.PMF" %> <%@ page import="edu.oregonstate.mobilecloud.lectures.clouddatastore.*" %> <%@ page import="java.util.*" %> <%@ page import="edu.oregonstate.mobilecloud.PMF" %> <% PersistenceManager pm = PMF.getPMF().getPersistenceManager(); try { Query query = pm.newQuery(Test3Members.class, "coursenum >= 600"); List<Test3Members> toDelete = (List<Test3Members>)(query.execute()); pm.deletePersistentAll(toDelete); query.closeAll(); } finally { pm.close(); } %> Retrieve list of objects to delete (which seems inefficient?!?!) Invoke pm.deletePersistentAll test5c.jsp

22. A few final notes • If you don’t want a member persisted, use @NotPersistent • Make all member variables private • JDO might actually modify the variable’s bytecode • Query has a closeAll() method. • close the query when you’re done with results • Query has a deletePersistentAll() instance method. It is painfully slow. • Do not use until Google fixes this problem (which is ironic, considering its intended purpose)