190 likes | 357 Views
Supporting Persistent Objects In Python Jeremy Hylton jeremy@zope.com. What is persistence?. Data lives longer than programs Files, pipes, relational databases, etc. But, discontinuity of representation Orthogonal persistence Automatic management of program state
E N D
Supporting Persistent ObjectsIn Python Jeremy Hyltonjeremy@zope.com
What is persistence? • Data lives longer than programs • Files, pipes, relational databases, etc. • But, discontinuity of representation • Orthogonal persistence • Automatic management of program state • Independent of data type or longevity • Allow programmer to focus on application data model
Python approach • Goals • Minimize changes to existing programs • Work with standard interpreter • Zope Object Database (ZODB) • Support Zope application server • Originally targeted for web development • Separate database and storage layers
Not on today’s agenda • Benefits of persistence • Discussion of related work • Early work • PS-algol, Napier88 • Much recent work on Java • PJava, ObjectStore PSE
ZODB key ideas • Persistence by reachability • Any reachable object is saved • Transactions to control updates • Safe sharing among clients • Implement with Persistent mixin • Provides hook for persistence • Compromise on transparency
Persistent base class • C extension type that defines • Four _p_ attributes • Custom attribute accessors • Efficient C API • Marks serialization boundaries • Interacts with database • Load objects on demand • Register modifications with TM
Persistence by reachability • Any object reachable from ZODB root is persistent
A simple example • from Persistence import Persistentfrom Transaction import get_transactionfrom ZODB.FileStorage import DBclass Counter(Persistent): _value = 0 def inc(self): self._value += 1def main(): fs = DB(“data.fs”) conn = db.open(); root = conn.root() obj = root[“myobj”] = Counter() get_transaction().commit() obj.inc() get_transaction().commit()
Object serialization • Standard pickle library • Serializes arbitrary object graph • Raises TypeError for sockets, files, &c. • Instance vars serialized via dictionary • Hooks to define custom state • __getstate__() / __setstate__() • Persistent mixin ignores _v_ attributes
Pickling persistent objects • Stores objects in separate records • Persistent objs pickled as oid + class • Works with cache to maintain identity • Handling non-persistent objects • Copied into record of containing object • Sharing by persistent objs is problematic
Object identity / caching • Cache maintains oid obj mapping • Guarantees only one copy of object • Unpickler loads all referenced objects • Ghost objects • Only Persistent header initialized • No instance state loaded • State loaded on first object access • LRU cache of recent objects
Attribute access handlers • Persistent implements C wrappers • Override tp_getattro, tp_setattro slots • Mediate access to instance variables • Crucial Python feature
Example __getattribute__() hook class Example(object): _p_state = False def _p_activate(self): print "activate" self._p_state = True def __getattribute__(self, attr): print "intercept", attr if not attr.startswith("_p_") and not self._p_state: self._p_activate() return super(Example, self).__getattribute__(attr) >>> obj = Example(); obj.value = "test" >>> print obj.value intercept value intercept _p_state intercept _p_activate activate test
Transactions • Supports multiple threads, processes • Independent database connections • Updates visible at transaction boundaries • Optimistic concurrency control • When conflict occurs, abort and retry • On error, abort to restore consistency • Reverts to last saved state
Concurrency and conflicts • Invalidations sent at commit time • Clients process at transaction boundaries • Conflicting transactions aborted • Write conflict at commit time • Read conflict on object access • Application must retry on conflict • Can use generic wrapper • Can define conflict resolution method
Retrying conflicts • Example wrapper for retries def transact(f, retries=3): def wrapper(*args, **kwargs): n = retries while n: try: try: return f(*args, **kwargs) finally: get_transaction().commit() except ConflictError: n -= 1 if n == 0: raise except: get_transaction().abort() raise return wrapper
Other features • Undo support • Storage stores multiple revisions • Transactional undo reverts to earlier state • BTrees: efficient persistent containers • Storing code in database
Limitations • Schema evolution • Must code manually in __setstate__() • Database management • Manual pack() to remove revisions, do GC • Sharing of non-persistent objects • Integration with legacy code • Multiple inheritance helps • Factory classes
Getting the software • http://www.zope.org/Wikis/ZODB • info central for ZODB • ZODB 3.1 released Oct. 2002 • Uses ExtensionClass • ZODB4 alpha planned for Dec. 2002 • Based on Python 2.2 type support • Fewer onions in the varnish