Clouds and Web2.0 II: Case Study and Tutorial

Clouds and Web2.0II: Case Study and Tutorial CTS08 Tutorial Hyatt Regency Irvine California May 19 2008 Geoffrey Fox, Marlon Pierce Community Grids Laboratory, School of informatics Indiana University http://www.infomall.org/multicore gcf@indiana.edu, http://www.infomall.org 1

Polar Grid: A Web 2.0 Case Study Using Microformats, GeoRSS, REST, and KML Slides from Yu Ma

More Detailed Information • See Marie’s blog: http://tethealla.blogspot.com/ • We use blogs in the lab extensively as lab books.

PolarGrid Datasets • A typical processed dataset consists of: • One plain text header file describing the overall system and metadata of the particular chunk of measurements. • Multiple JPG images processed from the corresponding raw data. • Collection timestamp and fundamental properties such as waveform and transmitting (TX) /receiving (RX) antenna IDs are embedded in file names. • Images are processed by Kansas University’s Matlab-based code, GOAP

Approaches • Convert unstructured metadata description into microformats. • Construct a relational database to enable fast and dynamic queries. • Provide geographic annotation embedded into web feeds that can be plotted by Google Maps and Google Earth • Technologies: • GeoRSS: RSS syndication with geo-location • Microformats: extensions to XHTML

Microformat Conversion • Microformats capture “semantic” metadata information in XHTML format. • Principals for converted microformats: • Simple: capture the most basic information unit. • Reusable: capture shared features among data. • Composable: when combined, can reflect more complex relationships among data. • Most of our microformat classes are prefixed with “pg:” denoting the PolarGrid project namespace, except when existing generic ones such as the geo microformat are used.

How To Extend XHTML • Use either <div> or tags. • Display of these tags can be controlled with style sheets • You can also dynamically control these with JavaScript. • Next generation of browsers will have built-in support for well-known microformats such as hCard.

Simple Example: Geo Format   -2.193  52.686

Polar Grid Microformat Examples • Simple descriptions: MCRDS • Measurements with unit: 2.5 us

Microformat Examples • Identification for cross-references: 1 • Commonly shared features: (e.g. both TX and RX antennas record same kind of information)  0 TX/RX 0

Microformat Examples • Structured collections/Arrays:  2  0 TX 0   1  TX  0

Microformat Examples • Complex objects: (e.g. spatial and temporal information that uniquely defines a processed data chunk)  Start/Stop 1202755351.892651   -2.193  52.686

GeoRSS-Simple Embedded Atom Feed <?xml version="1.0" encoding="utf-8"?> <feed xmlns="http://www.w3.org/2005/Atom" xmlns:georss="http://www.georss.org/georss"> <title>PolarGrid Expedition</title> <id>tag:pg3.ucs.indiana.edu,2008:polargrid-expedition-2008-05</id> <updated>2008-05-11T02:02:54Z</updated> <author> <name>Polar Grid</name> </author> <link type="application/atom+xml" rel="self" href="http://pg3.ucs.indiana.edu/cgi-bin/getPolarGridFeed"/>

GeoRSS-Simple Embedded Atom Feed <entry> <title>Data Chunk: 2008-03-11 11:55:41</title> <id>tag:pg3.ucs.indiana.edu,2008:polargrid expedition-2008-05-262</id> <updated>2008-03-11T11:55:41Z</updated> <content type="html"> … (data chunk information described in microformats) … </content> <georss:point>77.585957 -52.448461</georss:point> <georss:point>77.580976 -52.435148</georss:point> </entry> </feed>

Google RSS Feed Reader

Google Maps

Corresponding KML <?xml version="1.0" encoding="UTF-8"?> <kml xmlns="http://earth.google.com/kml/2.2"> <Document> <name>PolarGrid Expedition</name> <Placemark id="tag:pg3.ucs.indiana.edu,2008:polargrid-expedition-2008-05-262”> <name>Data Chunk: 2008-03-11 11:55:41</name> <description><![CDATA[ … (data chunk information described in microformats) … ]]></description> <Point><coordinates>-52.448461,77.585957</coordinates></Point> <Point><coordinates>-52.435148,77.580976</coordinates></Point> </Placemark> </Document></kml>

Google Earth and KML

Web 2.0 Lessons • Web 2.0 approaches can be applied to management of scientific information. • With the right choices, you get a lot for free. • Microformats: simple ways to encode name-value pairs, build up semantic descriptions. • GeoRSS: orders data by both time and space • Works in all standard RSS/Atom readers. • Google Maps supports natively

Web 2.0 Tutorial Map

User Layer Browser + JavaScript Libraries Browser + JavaScript Libraries Browser + JavaScript Libraries AJAX, JSON, REST, RSS Server-Side Gdata Apps Facebook Apps Gadgets, Gadget Aggregators SOAP, REST, RSS Blogs, Calendars, Docs, etc Facebook Social Gadget Containers System Cloud Layer

Map Key • Red blocks represent browsers and things that run in them (JavaScript). • This is the “user” level. • Client side mashups • Green blocks represent Web servers and their applications. • This is the “developer” level. • Server-side mashups. • These can run on any hosting environment: your web server, Amazon EC2, Google GAE, etc. • Blue blocks represent third party services. • This is the “system cloud” layer. • Arrows represent network communications. • Everything goes over HTTP • REST, AJAX: communication patterns. • RSS, ATOM, JSON, SOAP: message format.

Using Google’s GData API How to Write Server-Side Mash-Ups

GData: the API to Google Services • Google provides an extensive set of online services with well-known user interfaces. • Docs • Spreadsheet • YouTube • Blogger/Blogspot • Calendar • Picasso Photo Sharing • GMail • These have programming interfaces as well as user interfaces. • Server side: Java, .NET, PHP, Python • Client side: JavaScript • Client interfaces include visual components suitable for mash-ups • Server-side APIs don’t include visual components but do allow you to make server-side mash-ups.

Server-Side Mash-Up Use Case • Imagine you have a Web service that runs a large parallel finite element code. • Output can include images, movies, output files, metadata, etc. • Your web service manages the interaction with the queuing system (another topic…) • With a server-side mash-up, your Web Service can • Post status information about your jobs (“Step 521 Completed!”) on your blog. • Post URLs pointing to your output files on your blog. • Upload your visualizations to Picasso and YouTube (and post links). • Generate RSS/Atom feeds of the above. • Post results to Google Calendar • We’ll look at some Java examples.

Getting Started with GData • Create a Blog, make sure you can login to YouTube, etc. • You will need a clientID and associated developer key. • Get these from http://code.google.com/apis/base/signup.html. • Get the Java code • http://code.google.com/p/gdata-java-client/downloads/list • See any number of Google examples • http://code.google.com/apis/base/javadevguide.html • One trick: You can get your blogger ID by examining the your Blog’s Atom feed. • It will be in the header and look something like this: <id>tag:blogger.com,1999:blog-19457310</id> • The blog ID to use in the code below would be 19457310 in this example. • The full working example is here • http://communitygrids.blogspot.com/2008/03/googles-gdata-java-api.html • We will simplify for pedagogical reasons.

Posting Results to Your Blogspot Blog String userName="me@gmail.com"; String password= "qwerty"; String content=“…”; GoogleService myService= new GoogleService("blogger",""); myService.setUserCredentials(userName,password); Entry myEntry = new Entry(); myEntry.setTitle(new PlainTextConstruct(title)); myEntry.setContent(new PlainTextConstruct(content)); URL postUrl = new URL("http://www.blogger.com/feeds/" + blogID + "/posts/default"); myService.insert(postUrl, myEntry); Replace these with your values Login in Create the entry to insert Insert the new entry using REST’s POST

Getting a YouTube Feed String feedUrl="http://gdata.youtube.com/feeds/api/videos/"; String scarFace5sec="http://gdata.youtube.com/feeds/api/videos/egwB7hVIIEc"; YouTubeService service = new YouTubeService(clientID, developer_key); service.setUserCredentials(userName,password); VideoFeed videoFeed = youTubeService.getFeed(new URL(feedUrl),VideoFeed.class); VideoEntry videoEntry = youTubeService.getEntry(new URL(scarFace5sec), VideoEntry.class); //Loop over feed entries and display. String feedContent= "Title: " + videoEntry.getTitle().getPlainText()+”"; feedContent+=videoEntry.getMediaGroup(). getDescription().getPlainTextContent()+”";

Posting To Google Calendar CalendarService calService=new CalendarService(clientID); calService.setUserCredentials(userName,password); URL postUrl = new URL("http://www.google.com/calendar/feeds/"+userName+"/private/full"); CalendarEventEntry myEntry = new CalendarEventEntry(); myEntry.setTitle(new PlainTextConstruct(titleOfEvent)); myEntry.setContent(new PlainTextConstruct(contentToPost)); DateTime startTime = new DateTime(new Date()); When eventTimes = new When(); eventTimes.setStartTime(startTime); myEntry.addTime(eventTimes); CalendarEventEntry insertedEntry = calendarService.insert(postUrl, myEntry); CalendarEventEntry myEntry2 = new CalendarEventEntry(); String now=(new Date()).toString(); myEntry2.setContent(new PlainTextConstruct("Test post at "+now)); myEntry2.setQuickAdd(true); // Send the request and receive the response: CalendarEventEntry insertedEntry2 =calendarService.insert(postUrl, myEntry2); Log in to the calendar service Create a new Calendar entry Create the event time and insert An alternative way to post the entry

Some Other Things You Can Do with GData Server Code • YouTube • Search movies by keyword tag, “related”, categories • Upload and download movies. • Put movies into collections • Calendar • Retrieve events as RSS feed • Spreadsheet • Remotely retrieve and insert or change row and cell data. • Use structured queries to retrieve data ranges. • Remotely invoke batch operations

Making FaceBook Applications Using Facebook as a backend service.

Full Examples and More Detailed Notes • PHP and JavaScript notes • http://communitygrids.blogspot.com/2008/02/quick-facebook-javascript-api-notes.html • JavaScript example: • http://communitygrids.blogspot.com/2008/02/little-more-jsfacebook-hacking.html

Why Use FaceBook as a Portal? • Because it has 10’s of millions of users. • The largest Web 2.0-style Web portal • Easy to create Social Networks: groups of friends and share applications, communicate, share information, etc. • And more importantly, tools for building and managing these networks • Can leverage many third party applications • For example, photo albums • Of course many of them are trivial, silly

FaceBook APIs • These come in two flavors • Embedded: make your application available through FaceBook. • PHP APIs • Embedding: use Facebook authentication and social network data in your application. • JavaScript API • In both cases, you run your application on your Web server. • You application do anything you can implement. • And you can query FaceBook for social network information, user metadata.

Using The JavaScript API • Getting Started: • You just need a Web server to host your application. • Register as a developer • Place Facebook’s xd_receiver.htm on your web server • Make sure the relative path used is correct. • What will happen? • Your page will run outside of Facebook. • Anyone loading your application will be directed to Facebook to login. • User is then redirected back to your page. • Your application can make calls back to Facebook. • And you can do anything else you want. • So you could embed this application in a portlet, interact with a local database, • You can build your application with .NET or Java Server Faces, etc.

HTML Page Example <textarea style="width:500px;height:300px;" id="_traceTextBox"> </textarea> <script src="http://static.ak.facebook.com/js/api_lib/FacebookApi.debug.js" type="text/javascript"> var api = new FB.ApiClient('<your_key_here>', '/xd_receiver.htm', null); … </script>

Interact with FaceBook api.requireLogin(function(exception) { api.friends_get(null,getResults); var myinfo= 'last_name, first_name, hometown_location, work_history, pic_small'; api.users_getInfo(api.get_session().uid, myinfo, getInfo); });

Notes on Previous Slides • The requireLogin, friends_get, and users_getInfo are FaceBook API calls. • The getResults and getInfo methods are callback methods that I defined. • Shown in next slide • The myinfo data structure is taken from the Facebook XML user profile. • http://wiki.developers.facebook.com/index.php/Users.getInfo

Developer Defined Call-Back Functions function getResults(result,exception) { Debug.dump(api.get_session().uid,’You'); Debug.dump(result,’Your Friends’); } function getInfo(result,exception) { Debug.dump(result,’Your info'); } These two methods receive the output returned from Facebook by the API’s friends_get() and users_getInfo() methods. The results are dumped to the text window but you can do more interesting things.

Output Info Dumped to Text Area you: 627774031 Your friends: {Array} [Deleted] Your Info: {Array} [0]: {Object} first_name: Marlon hometown_location: {Object} city: state: country: zip: last_name: Pierce pic_small: http://profile.ak.facebook.com/profile6/1797/98/t627774031_2463.jpg uid: 627774031 work_history: {Object} User’s Facebook ID and the ID numbers of his friends User’s profile metadata. Note some fields are strings, some are Arrays, and some are structured objects of strings.

Facebooks’ Query Language • All of the FaceBook API is basically a set of wrappers around their SQL like FQL query language. So you can, if you prefer, make custom methods out of FQL query strings. • var myQuery='SELECT name FROM user WHERE uid='+api.get_session().uid; • api.fql_query(myQuery, getFQLResponse); getFQLResponse() is a developer-written callback function

Working with Facebook Groups • To list information about groups you are a member of, use the following. • You can replace the value of uid with any UID you know. • 18629081888 is the Group ID for the Open Grid Forum’s uninteresting Facebook group • api.groups_get(api.get_session().uid,null,getGroups); • api.groups_getMembers(18629081888,getGroupMembers);

More Information • The JavaScript documentation is available from here • http://wiki.developers.facebook.com/index.php/JavaScript_Client_Library • It is pretty minimal, but it is essentially the same as the better documented PHP API. • Replace PHP’s "." with a "_". • So PHP’s users.getInfo becomes users_getInfo(...) in JavaScript.

Open Social API and Gadgets

What Is a Gadget? Simple gadgets for getting a Grid proxy credential and running remote commands. Both run on my own Web server.

Gadget Definition <?xml version="1.0" encoding="UTF-8" ?> <Module> <ModulePrefs title="MyProxy Gadget Example" scrolling="true" height="500"/> <Content type="url" href="http://156.56.104.143:8080/GTLAB/examples/MyProxyExample.jsf"> </Content> </Module> • This XML page is web accessible via http://156.56.104.143:8080/GTLAB/MyProxyGadget.xml • This URL points to the actual application (written in JSF) • Google uses an HTML IFrame to load the page. • You can also use content type=“html” for more integrated applications. • For example, you can send messages between gadgets in the client.

Gadgets and Open Social • Open Social is a consortium of Facebook’s major competitors. • Orkut, MySpace, Hi5, LinkedIn, Friendster, etc. • The Open Social API has two parts • JavaScript libraries for building social gadgets. • Must run in a gadget container (server) • REST APIs for exchanging data between container servers. • The ideas: • JS APIs let you make portable gadgets that will work in different containers (Hi5LinkedIn) but data will be different. • REST will let you export/import data.

Getting Started • You need a Gadget container to host your gadgets. • LinkedIn is notably selective about the gadgets it will host. • Orkut provides a sandbox, but you have to upload your gadgets. • Apache Shindig is an extremely simple Java-based container. • De facto reference implementation of Open Social • Not suitable for production by a long ways. • But it will work on an air plane. • Shindig is something of a moving target. • My February notes were obsolete…

Getting Started with Shindig • Visit the Web site for information • http://incubator.apache.org/shindig • Check out the code with SVN • mkdir $HOME/shindig; cd shindig • svn co http://svn.apache.org/repos/asf/incubator/shindig/trunk/ • Build and run it with Apache Maven • Run “mvn” from $HOME/shindig/ • Run “mvn -Prun” from $HOME/shindig/java/server • Point browser to http://localhost:8080/gadgets/files/container/sample1.html and look through other samples. • You can also load these HTML files using file:// if you don't want to run the Jetty server. • Periodically check for updates and rebuild • svn update • The READMEs have better/more up-to-date information than the website.

Clouds and Web2.0 II: Case Study and Tutorial