Open Data Protocol (OData) Deep Dive

1. Open Data Protocol (OData)Deep Dive A common, open, RESTful Web protocol for querying and updating data that provides a uniform way to unlock data and free it from silos that exist in applications today.

2. OData = REST Resource-oriented Entities modeled as URI-addressable Resources Relationships modeled as links (URIs) CRUD = POST, GET, PUT/PATCH, DELETE Hypermedia-driven Navigate from Service Document to Sets to Members to related items to… Links in Payload for editing, navigation, operations, etc. + Data Model URLs, operations, namespaces, derived from declarative model + Common Conventions Common query string options Representation within Formats + Common Formats ATOM, JSON

3. USE of HTTP • HTTP Methods • GET, POST, PUT/PATCH, DELETE • HTTP Headers • Content Negotiation • Accept, Accept-Charset, Accept-Encoding, Accept-Language • Content-Type, Content-Length, Content-Encoding, Content-Language • Concurrency • Etags, If-Match, If-None-Match • Etc… • HTTP Result Codes • 2xx Success Codes • 3xx Redirection • 4xx Client Error Messages

4. OData Feature Areas • Versioning • Service Metadata • Data Requests • Data Modification • Relationships • Content Types • Extensibility

5. Versioning

6. Protocol Versioning • DataServiceVersion header • Describes the protocol version according to which the request/response should be interpreted • MaxDataServiceVersion Request Header • The maximum protocol version that the client can accept • MinDataServiceVersion Request Header • The minimum protocol version the client can accept -Clients SHOULD specify a MaxDataService -Services return a response formatted according to the latest protocol version the service supports that is less than MaxDataServiceVersion but at least MinDataServiceVersion

7. Schema Versioning • When a data model or other breaking change is made to the service, the service URL is generally versioned. For example: • odata.netflix.com/v1/Catalog • odata.netflix.com/v2/Catalog

8. Service Metadata

9. Service Document • Root of service exposes a “Service Document” • Describes collections of Entities GET /odata.netflix.com/v2/Catalog HTTP/1.1 <servicexmlns="http://www.w3.org/2007/app"xmlns:app="http://www.w3.org/2007/app" xmlns:atom="http://www.w3.org/2005/Atom"xml:base="http://odata.netflix.com/v2/Catalog/"> <workspace> <atom:title>Default</atom:title>-<collectionhref="Genres"> <atom:title>Genres</atom:title> </collection>-<collectionhref="Titles"> <atom:title>Titles</atom:title> </collection>-<collectionhref="TitleAudioFormats"> <atom:title>TitleAudioFormats</atom:title> </collection>-<collectionhref="TitleAwards"> <atom:title>TitleAwards</atom:title> </collection>-<collectionhref="People"> <atom:title>People</atom:title> </collection>-<collectionhref="TitleScreenFormats"> <atom:title>TitleScreenFormats</atom:title> </collection>-<collectionhref="Languages"> <atom:title>Languages</atom:title> </collection> </workspace> </service>

10. Entity Data Model • Exposed on /$metadata • Describes Entity Model of Data Service • Entity Types • Complex Types • Association Types • EntityContainers GET /odata.netflix.com/v2/Catalog/$metadata HTTP/1.1 <edmx:Edmxxmlns:edmx="http://schemas.microsoft.com/ado/2007/06/edmx"Version="1.0"> <edmx:DataServicesm:DataServiceVersion="1.0" xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata"> <Schemaxmlns="http://schemas.microsoft.com/ado/2008/09/edm" xmlns:d="http://schemas.microsoft.com/ado/2007/08/dataservices" Namespace="Netflix.Catalog.v2"> +<EntityTypeName="Genre"> +<EntityTypeName="Title"m:HasStream="true"> +<ComplexTypeName="BoxArt"> +<AssociationName="Genre_Titles"> +<EntityContainerName="NetflixCatalog"m:IsDefaultEntityContainer="true"> </Schema> </edmx:DataServices> </edmx:Edmx>

11. Entity Type • Named structures with keys • Support Inheritance • Properties • String, Integer, Boolean, DateTime, Spatial datatypes • Collections, Complex Types • Relationship Properties • May be marked as "Open" • May return "dynamic" properties not described in $metadata <EntityTypeName="Genre"> <Key> <PropertyRefName="Name"/> </Key> <PropertyName="Name"Type="Edm.String"/> <NavigationPropertyName="Titles"Relationship="Self.Genre_Titles" ToRole="Genre_Titles_Target" FromRole="Genre_Titles_Source"/> </EntityType>

12. Complex Type • Named Structures w/o keys • Support Inheritance • Don't support relationships • Because they don't have an independent ID <ComplexTypeName="BoxArt"> <PropertyName="SmallUrl"Type="Edm.String"/> <PropertyName="MediumUrl"Type="Edm.String"/> <PropertyName="LargeUrl"Type="Edm.String"/> </ComplexType>

13. Associations • Define relationships (navigation paths) between Entities • Bi-directional • Specify Cardinality of each end • Can also impose integrity constraints <AssociationName="Genre_Titles"> <EndType="Self.Genre"Multiplicity="*" Role="Genre_Titles_Source"/> <EndType="Self.Title"Multiplicity="*"Role="Genre_Titles_Target"/> </Association>

14. Entity Container • Contains • EntitySets – collections of Entity Instances • AssociationSets – relationships between entities • FunctionImports – Actions/Functions exposed by service <EntityContainerName="NetflixCatalog" m:IsDefaultEntityContainer="true"> <EntitySetName="Genres"EntityType="Self.Genre"/> <EntitySetName="Titles"EntityType="Self.Title"/> <AssociationSetName="Genre_Titles"Association="Self.Genre_Titles"> <EndRole="Genre_Titles_Source"EntitySet="Genres"/> <EndRole="Genre_Titles_Target"EntitySet="Titles"/> </AssociationSet> </EntityContainer>

15. Data Requests

16. Querying Data • Querying a Set • GET /v2/Catalog/Genres HTTP/1.1 • Requesting an Individual Entity by ID • GET /v2/Catalog/Genres('Adventures') HTTP/1.1 • Requesting an Individual Property • GET /v2/Catalog/Genres('Adventures')/Name HTTP/1.1 • Requesting an Individual Property Raw Value • GET /v2/Catalog/Genres('Adventures')/Name/$value HTTP/1.1

17. $Filter • Basic predicates, built-in functions • GET /v2/Catalog/Titles?$filter=Name eq 'The Sting' HTTP/1.1 • Filter on Properties of a derived type • Only returns entities of that type that match the predicate • GET /v2/Catalog/Awards?$filter=Netflix.AcademyAward/Category HTTP/1.1 • Query based on collection Membership using Any/All • GET /v2/Catalog/Titles?$filter=Genres/Any(g:g/Name eq 'Adventure') HTTP/1.1

18. Operators • Logical Operators • eq, ne, gt, ge, lt, le, and, or, not • Mathematic Operators • add, sub, mult, div, mod • Grouping Operator • () • NULL literal • null

19. Built-in Functions • String Functions • substringof, endswith, startswith, length, indexof, substring, tolower, toupper, trim, concat • DateTime Functions • year(), month(), day(), hour(), minute(), second() • Math Functions • round(), floor(), ceiling() • Type Functions • isof() • Casting • Insert type in path • Geo Functions • geo.length, geo.intersects, geo.distance

20. $Orderby • Comma separated list of properties, expressions • GET /v2/Catalog/Titles?$orderby=AverageRating,ReleaseYearHTTP/1.1 • Can specify asc (default) or desc • GET /v2/Catalog/Titles?$orderby=AverageRatingdesc HTTP/1.1

21. $top/$skip • Enables Client-side paging through large result sets • GET /v2/Catalog/Titles?$top=10&$orderby=AverageRatingdesc HTTP/1.1 • GET /v2/Catalog/Titles?$top=10&$skip=10 &$orderby=AverageRatingdescHTTP/1.1 • If no $orderby, server guarantees stable ordering

22. $expand • Specify comma separated list of navigation property paths to include in response • GET /v2/Catalog/Titles?$expand=Cast,Awards HTTP/1.1 • Result are returned inline according to the appropriate format

23. $select • Narrow the set of fields returned • GET /v2/Catalog/Titles?$select=Name,Synopsis,Rating HTTP/1.1 • Related properties must be specified in $expand • GET /v2/Catalog/Titles?$select=Name,Cast/Name&$expand=Cast HTTP/1.1 • Can include all properties using * • Does not include navigation properties • GET /v2/Catalog/Titles?$select=* HTTP/1.1

24. $count/$inlinecount • Get just the count • Can include additional query operators • GET /v2/Catalog/Titles/$count HTTP/1.1 • Include the count with the results • $inlinecount ignores $top/$skip but includes $filter • GET /v2/Catalog/Titles?$inlinecount=allpages&$top=10 HTTP/1.1

25. $FORMAT • Specify a particular format • "atom" • application/atom+xml, • application/atomsvc+xml for service document • "json" • application/json • "xml" • application/xml • Overrides content type specified in headers

26. Server Driven Paging • Server controls the maximum number of records to return at a time • Each "page" of results includes a "next link" for retrieving the next page of results • Pages can vary in size; some may be blank • i.e., return records as computed or across a federation

27. Data Modification

28. Insert • POST entity to the EntitySet collection • Returns inserted entity • Use PREFER header to request no content returned • Media Resources have special rules according to AtomPub: • POST media resource to the collection • Returns Media Link Entry • Update the Media Link Entry

29. Update • PUT to the edit-link to replace • Unspecified values are set to null or default • PATCH to the edit-link to affect only specified values • Unspecified values are left unchanged • Use PREFER header to request content returned • Use entity etag in if-match for concurrency control • Obtained from header or payload

30. Delete • DELETE to the edit-link to delete • Use entity etag in if-match for concurrency control • Obtained from header or payload

31. Batch Requests • "Batch" multiple statements in a single multi-part mime request • Results for each statement returned in a multi-part mime response • Multiple Data Modification statements may be grouped in a single "ChangeSet" • ChangeSets are atomic • Reference the results of other statements in the changeset • i.e., add a child to an added parent • May have multiple ChangeSets per Batch

32. Relationships

33. Requesting Relationship Links • Use $links to request the relationship links for a particular navigation property • Results are returned as an array of URIs • GET /v2/Catalog/Titles/Genres('Adventures')/$links/Titles HTTP/1.1

34. Creating Links to New Entities • Create a new Entity with a new related entity by POSTing the entity with the related entity as inline content ("Deep Inserts") • Create a new entity related to an existing entity by POSTing to the relationship collection of the existing entity • POST /v2/Catalog/Titles/Genres('Adventures')/$links/Titles HTTP/1.1 <entry>…</entry>

35. Creating Links to Existing Entities • Create a relationship between two existing entities by POSTing the URL of the one entity to the appropriate $links collection of the other entity • POST /v2/Catalog/Titles/Genres('Adventures')/$links/Titles HTTP/1.1 <uri>http://odata.netflix.com/v2/Catalog/Titles('13kbf')</uri>

36. DELETING Links • Remove a relationship by sending a DELETE request to the $links collection specifying the url to remove • DELETE /v2/Catalog/Titles/Genres('Adventures')/$links/Titles HTTP/1.1 <uri>http://odata.netflix.com/v2/Catalog/Titles('13kbf')</uri>

37. Content Types (Formats)

38. Atom Format • Multiple entities represented as a <feed> • Count optionally returned in <metadata:count> element • Next link returned as <link> element • Each entity represented as an <entry> • <id> is unique id • <category> specifies type • Self, edit <link>s • <link> element for Relationships • Inline content as child <metadata:inline> element containing feed or element • <link> element for editing media • Properties nested in <metadata:properties> element • Individual properties namespace qualified with data namespace • For Media Resources, <content> has a src property to the media stream • <metadata:properties> is a sibling of <content> • For non-Media Resources, <metadata:properties> is a child of <content> • Functions/Actions returned as <metadata:function> and <metadata:action> elements

39. Atom Payload - Example <?xmlversion="1.0"encoding="utf-8" ?> <feedxml:base="http://odata.netflix.com/v2/Catalog/" xmlns:d="http://schemas.microsoft.com/ado/2007/08/dataservices" xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata" xmlns="http://www.w3.org/2005/Atom"> <titletype="text">Genres</title> <id>http://odata.netflix.com/v2/Catalog/Genres/</id> <updated>2012-07-26T18:37:18Z</updated> <linkrel="self"title="Genres"href="Genres" /> <m:count>326</m:count> <entry> <id>http://odata.netflix.com/v2/Catalog/Genres('Adventures')</id> <titletype="text">Adventures</title> <updated>2012-07-26T19:08:35Z</updated> <author> <name/> </author> <linkrel="edit"title="Genre"href="Genres('Adventures')" /> <linkrel="http://schemas.microsoft.com/ado/2007/08/dataservices/related/Titles" type="application/atom+xml;type=feed"title="Titles" href="Genres('Adventures')/Titles" /> <categoryterm="Netflix.Catalog.v2.Genre" scheme="http://schemas.microsoft.com/ado/2007/08/dataservices/scheme" /> <contenttype="application/xml"> <m:properties> <d:Name>Adventures</d:Name> </m:properties> </content> </entry> <linkrel="next"href="Genres/?$skiptoken='Adventures'" /> </feed>

40. JSON Verbose Format • Results wrapped in a "d" wrapper • Multiple entities returned as JSON Array property named "Result" • Optionally has __Count property for number of elements in query result (across all pages) • Optionally has "__next" property for next link • Each Entity has • A "__metadata" property containing: • URI • Type • Media Link, edit-media link, content type, etag for media resources • A "__deferred" property for each un-expanded relationship containing the URL for the related entity/collection • Properties a Name/Value pairs • Each Complex Type has a "__metadata" property containing the Type

41. JSON PAYLOAD - Example { "d" : { "results": [ { "__metadata": { "uri":"http://odata.netflix.com/Genres('Adventures')", "type": "Netflix.Catalog.v2.Genre" }, "Name": "Adventures", "Titles": { "__deferred": { "uri":"http://odata.netflix.com/Genres('Adventures')/Titles"} } } ], "__count": "326", "__next": " Genres/?$skiptoken='Adventures'" } }

42. New JSON OData Format • Effort to simplify JSON • More readable JSON format • Reduce size • Preserve OData-ness • Still Hypermedia-driven • Uses Namespacing Mechanism • Removes almost all metadata from payloads • Calculates values from conventions or templates in metadata • 80-90% more efficient than Atom • Future contribution to TC

43. NEW JSON PAYLOAD - Example { "odata.metadata":"http://odata.netflix.com/$metadata#Netflix/Genres", "count":"830", "value": [ { "Name": "Adventures", } ], "nextLink":" http://odata.netflix.com/Genres('Adventures')/Titles " }

44. Extensibility

45. Custom Actions • Side-effecting operations that may or may not return a value • May be exposed on instances (Hypermedia-driven) • Use entity etag in if-match for concurrency control • Obtained from header or payload

46. Custom Functions • Extend query language • May be exposed on instances (Hypermedia-driven) • Use entity etag in if-match for concurrency control • Obtained from header or payload

47. Vocabularies An OData Vocabulary defines a shareable set of Annotations that may be applied to metadata or instance data • Extended Property Information • Units of measurement • Read-only, read-write • Ontologies • This entity represents the common concept of a person • Validation • This integer property is between 10 and 20 • Display hints • Caption field, grouping, navigation • Service Capabilities • Query limitations, etc. • Extension • Analytics

48. Annotations Consist of: • A Target: The construct against which the annotation is applied • May be any Model Construct (type, property, etc.) • A Term: The global name of the annotation • For example, org.odata.display.Title • A Value: The value for the annotation • May be a static value, path, or expression A Vocabulary is a set of terms in a common namespace

49. Annotations may… • Be defined in a common format • Machine readable (i.e., for validation) • Be embedded in a model • To define extended metadata • Be applied through an external annotation application file • Externally annotate existing services • Be applied to Instance Data • Information that may vary per instance

50. Types of Annotations • ValueTerm • A single annotation term applied to a type, property, association, etc. • Complex TypeTerm • Generally used to describe an is-a relationship • Relationships may be modeled as properties of individual or collections of other complex type terms • Entity TypeTerm • Most prescriptive; define keys, relationships