1 / 43

Lecture IV: REST Web Service with Google App Engine

Develop a web service on Google App Engine to analyze daily keyword trends based on Google Search API. Record query results over time and provide statistics on keyword frequency and weekly trends. The service should deploy on GAE and output in JSON or XML. Choose a meaningful topic and website for search to enhance results. Use additional web services as needed. Evaluation criteria include fetching API results, keyword extraction, and presenting results. Deliver demo, code, comments, user guide by due dates.

lottiew
Download Presentation

Lecture IV: REST Web Service with Google App Engine

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Lecture IV: REST Web Service with Google App Engine CS 4593Cloud-Oriented Big Data and Software Engineering

  2. Outline • Course Project • Google App Engine • Google Development Console • Google Custom Search API 2

  3. Course Project • Topic • Keyword trends analysis services for a certain topic • Platform • Google App Engine • Any programming language supported by GAE • Collaboration • Work individually • Hopefully you are not working on the same idea, but it is fine if you happen work on one by accident 3

  4. Course Project • Requirements • Develop a web service based on Google Search API to follow the weekly trends of keywords on a certain topic • Search one or multiple websites each day with Google Search API free quota and record the query results during a period of time (hopefully more than a week) • Perform statistics on query results to provide daily keywords frequency and the trends of the week 4

  5. Course Project • Requirements • Provide at least two web services: one for the keyword frequency of each day, and the other for trend of the week • Provide output as either JSON or XML • The developed web service should be deployed on Google App Engine 5

  6. Course Project • Advanced Requirements • You are going to choose the topic and the website to search, but the results should be meaningful (searching a static website for a week will not give you any different results) • Provide HTML presentation for the web services 6

  7. Course Project • Additional Information • Feel free to use additional free web services to support your task • Searching a more volatile website often provides you more interesting results, e.g., searching "Obama" in CNN or NY Times. 7

  8. Course Project • Evaluation • fetching and recording Google Search API results (2.5 points) • extracting keywords (2.5 points) • presentation of daily results (2.5 points) • presentation of weekly results (2.5 points) • The advanced features will be evaluated by quality. 8

  9. Course Project • Deliverables • Demo • Code • Commends and User Guide • Blackboard • Due date • Demo: Oct. 5th • Code: Oct. 4th 9

  10. Outline • Course Project • Google App Engine • Google Development Console • Google Custom Search API 10

  11. Google App Engine: Overview • Google App Engine (GAE) is a PaaS cloud computing platform for developing and hosting web applications in Google-managed data centers. • Google App Engine lets you run web applications on Google's infrastructure. • Easy to build. • Easy to maintain. • Easy to scale as the traffic and storage needs grow 11

  12. infrastructure vs. platform - What is “The Platform”? Application-specific code Libraries: shared by multiple applications Platform: same for all applications infrastructure: hidden by platform

  13. Scaling • Low-usage apps: many apps per physical host • High-usage apps: multiple physical hosts per app • Stateless APIs are trivial to replicate • Datastore built on top of Bigtable; designed to scale well • Abstraction on top of Bigtable • API influenced by scalability • No joins • Recommendations: denormalize schema; precompute joins 13

  14. Google App Engine: Overview • Free • 1GB storage • 1 virtual machine instance • 5 million page views / month • 10 applications / Google account 14

  15. Preserving Fairness Through Quotas • Everything an app does is limited by quotas, for example: • request count, bandwidth used, CPU usage, datastore call count, disk space used, emails sent • If you run out of quota that particular operation is blocked (raising an exception) until replenished 15

  16. Preserving Fairness Through Quotas • Free quotas are tuned so that a well-written app (light CPU/datastore use) can survive • The point of quotas is to be able to support a very large number of small apps (analogy: baggage limit in air travel) • Large apps can request raised quotas • Three types of quotas • Free Quota • Billed Limits • Safety Limits 16

  17. Google App Engine: Programming Languages • Java • App Engine runs JAVA apps on a JAVA 7 virtual machine (currently supports JAVA 6 as well). • Uses JAVA Servlet standard for web applications: • WAR (Web Applications ARchive) directory structure. • Servlet classes • Java Server Pages (JSP) • Static and data files • Deployment descriptor (web.xml) • Other configuration files 17

  18. Google App Engine: Programming Languages • Python • Uses WSGI (Web Server Gateway Interface) standard. • Python applications can be written using: • Webapp2 framework • Django framework • Any python code that uses the CGI (Common Gateway Interface) standard 18

  19. Google App Engine: Programming Languages • PHP • Local development servers are available to anyone for developing and testing local applications. • Only whitelisted applications can be deployed on Google App Engine. (https://gaeforphp.appspot.com/). 19

  20. Google App Engine: Programming Languages • Go • Go is an Google’s open source programming environment. • Tightly coupled with Google App Engine. • Applications can be written using App Engine’s Go SDK. 20

  21. Google App Engine: Data Storage • App Engine Data Store • NoSQL schema-less object based data storage, with a query engine and atomic transactions. • Data object is called a “Entity” that has a kind (~ table name) and a set of • properties (~ column names). • JAVA JDO/ JPA interfaces and Python datastore interfaces. 21

  22. Google App Engine: Data Storage • Google Cloud Store • RESTful service for storing and querying data. • Fast, scalable and highly available solution. • Provides Multiple layers of redundancy. All data is replicated to multiple data centers. • Provides different levels of access control. • HTTP based APIs. 22

  23. Google App Engine: Services • Google Services • URL Fetch • Mail • Memcache 23

  24. Google App Engine: Security • Sandboxes • All hosted applications run in a secure environment that provides limited access to the underlying operating system. • Sandbox isolates the application in its own secure, reliable environment that is independent of hardware, operating system and physical location of a web server. 24

  25. Google App Engine: Security • Sandboxes: Limitations • Application can only interact with other computers over internet using URL fetch, email, and HTTP/ HTTPS requests on the standard ports • Applications cannot write to local file system in any of the runtime environments. • Application code runs only in response to a web request, a queued task or a scheduled task and must return the response data within 60 seconds. 25

  26. Google App Engine: Demo • Create a project on Google App Engine • At https://console.developers.google.com/project • Initialize the environment of Restlet + eclipse + GAE • Tutorial at • http://restlet.com/technical-resources/restlet-framework/guide/2.3/introduction/first-steps/first-application • Download Restlet-GAE, GAE SDK, and Eclipse-GAE plugin 26

  27. Google App Engine: Demo • Deployment of Applications • Follow https://console.developers.google.com/start/appengine?_ga=1.67402004.429901446.1441394923 • Google Search API • Generate your search engine • Manage API usages • https://cse.google.com/manage/all 27

  28. Google Map API • Provide Map Services • Http: http://maps.googleapis.com/maps/api • Android • IOS • JavaScript • Parameters • Center • Zoom • Size • Language 28

  29. Google Map API • Pricing • Free Quota: 2500 / day • Bill Quota: 1000 / $0.5 • Safety Quota: 100, 000 / day 29

  30. Google Place API • Searching for places around certain map location • Usage Example • Finding all food places within 500 miles from (long,lat) • https://maps.googleapis.com/maps/api/place/nearbysearch/json?location=long,lat&radius=500&types=food&name=cruise&key=API_KEY • Place ID • A ID of a place that can be used later (e.g., Costco at I 10 @ UTSA Boulevard) 30

  31. Google Place API • Parameters • Type: food, airport, hospital, … • https://developers.google.com/places/supported_types • Pricing • Free quota: 1000/day • Free up to 150,000/day with credit card info 31

  32. Google Place API: response {         "geometry" : {            "location" : {               "lat" : -33.870775,               "lng" : 151.199025            }         }, …         "name" : "Rhythmboat Cruises",         "opening_hours" : {            "open_now" : true         },         "photos" : […],         "place_id" : "ChIJyWEHuEmuEmsRm9hTkapTCrk",         "scope" : "GOOGLE",         "types" : [ "travel_agency", "restaurant", "food", "establishment" ],         "vicinity" : "Pyrmont Bay Wharf Darling Dr, Sydney" },

  33. Google+ API Intro Usages Parameters Pricing 33

  34. Google+ API • A series of APIs working with Google+ • People: Get public profiles with query or user ID • Activities: List activities of a user or searching activities with query • Comments: List comments of an activity or searching comments with query • Moments: Retrieve, insert or delete moments • Usage • https://www.googleapis.com/plus/v1/ • 10,000/day free quota 34

  35. Google Translation API • Translate strings to other languages • Parameters • source • target • q(query) • Pricing • $20 for 1M characters • 50M / day limit 35

  36. Youtube Data API • Provide various supports to retrieve and manage data from Youtube • Some important services • Video: List, search, Insert, Delete, Rate, … • Playlist: Insert, delete, List, … • Captions: List, insert, update, download, delete, … 36

  37. Youtube Data API • Parameters • Part • videoCategoryId • myRating • Pricing • Unit based quota calculation • Services with different parameters may cost different quota • https://developers.google.com/youtube/v3/determine_quota_cost • 50M unit / day, and 30K unit / second, always free 37

  38. Facebook Graph API • A web API library for Facebook social network • Provide support for various platforms • RESTful • IOS • Android • JavaScript • PHP 38

  39. Facebook Graph API • Usages • Provide similar functions to google+ APIs • List friends, List events, Fetch comments, … • Uploading videos, photos, … • Pricing • Free • Need to contact Facebook for more than 100M API calls per day 39

  40. MediaWiki API • API to get information from Wikipedia programmatically • Entry:  https://en.wikipedia.org/w/api.php • Parameters • Action: query, modules • Titles: specify the query • Limit: max results • Section: retrieve from which section • Pricing • Free: with unofficial limits 40

  41. Bing API • Search API from Microsoft • Root: • https://api.datamarket.azure.com/Bing/Search/v1/ • Parameters • Query • NewsCategory • Sources: web, image, video, news, … 41

  42. Bing API • Pricing • 5000 / month free • After it: 10K/month for $20 42

  43. Summary Google App Engine Application Deployment on Google App Engine Popular free Web Services 43

More Related