1 / 61

Programmatic Interaction with Open Access Repositories

Learn how to interact programmatically with open access repositories using APIs and the OAI-PMH standard protocol. Explore manual resource uploads and techniques for searching, downloading, and uploading data. Understand the importance of reproducibility in scientific research and discover the role of e-Infrastructures in supporting the scientific method. Discover the concept of open science and its tools, and explore opportunities to promote open science in Africa.

drumm
Download Presentation

Programmatic Interaction with Open Access Repositories

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Programmatic Interaction with Open Access Repositories Roberto Barbera – University of Catania – Italy (roberto.barbera@ct.infn.it) WACREN e-Research Hackfest – Lagos (Nigeria)

  2. Outline Part 1 • Introduction: definitions and context Part 2 • Manually resource upload by submit interface • Programmatic interaction with an Open Access Repository using APIs for data • Searching • Downloading • Uploading • MARCXML tags overview • Programmatic interaction with an Open Access Repository using the OAI-PMH-standard protocol Part 3 • Get authorship of research products

  3. Part 1

  4. The Scientific Method • Examples of IR: • Classical Mechanics • Newton’s Gravitation Theory • Examples of DR: • General Relativity • Standard Model of Particle Physics G. Galilei

  5. The “output” of the Scientific Method Marked a realScientificRevolutionbut… itis the samesincealmost 4 centuries!

  6. The Pillars of the Scientific Method • Repeatability • The closeness of agreement between independent results obtained with the same method on identical test material, under the same conditions (same operator, same apparatus, same laboratory and after short intervals of time) • Affected by random errors • Reproducibility • The closeness of agreement between independent results obtained with the same method on identical test material but under different conditions (different operators, different apparatus, different laboratories and/or after different intervals of time) • Affected by systematic errors Is science reallyreproducible?

  7. Challenges in irreproducible research(http://www.nature.com/nature/focus/reproducibility/index.html)

  8. The “reproducibility crisis” Out of 18 microarray papers, results from 10 could not be reproduced • Ioannidis et al., 2009. Repeatability of published microarray gene expression analyses. Nature Genetics 41: 14 • Science publishing: The trouble with retractionshttp://www.nature.com/news/2011/111005/full/478026a.html • Bjorn Brembs: Open Access and the looming crisis in science https://theconversation.com/open-access-and-the-looming-crisis-in-science-14950

  9. Repeatability and Reproducibility are not all…

  10. Evolution of Distributed Computing Cost of hw Cost of networks Time 90’s-00’s Grid Computing 00’s-10’s Cloud Computing 80’s-90’s Cluster Computing Mainframe Computing Power of COTS WAN bandwidth 10

  11. e-Science • Virtual Research Communities • e-Infrastructure • Applications • Data • Instruments/sensors

  12. How do e-Infrastructures support the Scientific Method? Data Infrastructures Open Access Doc. Repos. Data Repos. Challenge: «walk» across the knowledgepathboth ways HTC/HPC Clusters Grids, Clouds Data preservation Semantic-web enrichment of linked data

  13. Open Science (definitions)(http://book.openingscience.org and http://dx.doi.org/10.1787/5jrs2f963zs1-en) “Open Science refers to a scientific culture that is characterized by its openness. Scientists share results almost immediately and with a very wide audience” “Open science is a means and not an end in itself and it is much more than just open access to publications or data; it includes many aspects and stages of research processes thus enabling full reproducibility and re-usability of scientific results.”

  14. Open Science (tools)

  15. The European Open Science Cloud and the 3 Os of the EC(http://ec.europa.eu/research/openscience/index.cfm?pg=open-science-cloud, https://goo.gl/6Nm39H)

  16. Some global “connections” Challenge: makeAfrican science/scientists more «visible» Opportunity: exploit e-Infrastructures to do it Vision: promote Open Science in Africa

  17. The Dakar Declaration on Open Science in Africa www.sci-gaia.eu/dakar-declaration/

  18. The Sci-GaIAFederated Platform for an Open Science Commons in Africa www.sci-gaia.eu/osp/

  19. The Knowledge Workflow

  20. Concepts and definitions(Source: Wikipedia) Open Access repositories are powered by Digital Asset Management Systems (DAMSes), which are “intertwined structures incorporating both software and hardware that take care of management tasks and decisions surrounding the ingestion, annotation, cataloguing, storage, retrieval and distribution of digital assets” A digital asset in essence is “anything that exists in a binary format and comes with the right to use” “Types of digital assets include, but are not exclusive to, photography, logos, illustrations, animations, audio-visual media, presentations, spreadsheets, Word and/or PDF documents, data and a multitude of other digital formats and their respective metadata”

  21. Some of the most common DAMSes Others, more business or social oriented, are listed at www.capterra.com/digital-asset-management-software/

  22. Sci-GaIA Task 3.1: Support the creation of federated and interoperable Open Access Document and Data Repositories in Africa, compliant with EU and other international guidelines Planned activities: • Identification of already existing Open Access Document and Data Repositories in the region and inclusion in web based directories such as OpenDOAR and the CHAIN-REDS Knowledge Base • Promotion of the Open Access Initiative (OAI) standards and of the OpenAIREguidelines to make contents (both papers and data) stored on the African repositories more discoverable, searchable and hence visible worldwide • Federation, through the use of Linked Data standards and Semantic Web technologies, of African Open Access Document and Data Repositories and to make them accessible and searchable from a unique entry point included in the project website • Feasibility study for the creation of a pilot service to issue Persistent Identifiers (PIDs) compliant with the Handle System to be associated to documents and data • Provision of a ready-to-install-and-configure appliance to quickly build and populate Open Access Repositories compliant with OAI, OpenDOAR and OpeAIREstandards/guidelines

  23. The Sci-GaIA Open Access Repository Requirements: • Open source • Distributed under a free license • Deployable on a local infrastructure (i.e., not a hosted service) • Standard compliant • Well supported • Scalable, up to O(106) – O(107) resources (to begin with) Choice: Invenio(latest stable version: v1.2.1 + Sci-GaIA add-ons) Motivations: • Fully compliant with all most important library standards, e.g. DCMI, Marc21 and OAI-PMH; • Co-developed by an international collaboration comprising institutes such as CERN, DESY, EPFL, FNAL, SLAC and used as institutional repository by about 30 scientific institutions worldwide; • INSPIRE, SCOAP3and ZENODO (the OpenAIRE flagship archive) repositories are based on Invenio; • The CERN Document Server operates since 2002 and manages about 1.3 million records; • UNESCO and UEMOA are leading an initiative to create a virtual library based on Invenio in 8 African countries (Benin, Burkina Faso, Côte d’Ivoire, Guinea Bissau, Mali, Niger, Senegal and Togo).

  24. The Sci-GaIA Open Access Repository(http://oar.sci-gaia.eu/) Resources can be: Manually uploaded Automatically harvested and ingested from external sources Sci-GaIA add-ons to Invenio: The possibility to mint DataCite Digital Object Identifiers (DOIs) and assign them to the records stored in the OAR If existing, direct links to the altmetrics of each of the records contained in the OAR The correct metadata structure and the right OAI-PMH endpoint configuration to make the OAR compliant with version 3.0 of the OpenAIREGuidelines federated authentication

  25. Compliance with standards(Full conforming with Open Archive Initiative’s standards & registered as an OpenDOAR data provider)

  26. The Knowledge Workflow First demonstrated @ ICT2015

  27. Research packages

  28. The Sci-GaIA OAR itself as a research package 6 clones of the Sci-GaIA OAR are being deployed, both in Africa and Europe

  29. Part 2

  30. Submit a resource

  31. Image submit Item 1 Item 2

  32. Programmatic Interaction with an (Invenio-based) Open Access Repository Search Engine API There are three kind of APIs you can use: XML API JSON API Python API

  33. Programmatic Interaction XML API Syntax: • GET /search?param1=value1&param2=value2&param3=value3… Example: • Get the first 10 records in XML format • http://oar.sci-gaia.eu/search?jrec=1&rg=10&of=xm • where • jrec= jump to record ID (e.g. 1 for first hit) • rg=records-in-group-of (e.g. 10 hits per page) • of= output format (e.g. Xm for XML format)

  34. Programmatic Interaction XML API Set ‘jrec’ and ‘rg’ appropriately to paginate the output Example: • http://oar.sci-gaia.eu/search?of=xm&jrec=1&rg=10 • http://oar.sci-gaia.eu/search?of=xm&jrec=11&rg=10 • http://oar.sci-gaia.eu/search?of=xm&jrec=22&rg=10 Do not set “rg” to high – there is a server-wide safety limit for it

  35. Programmatic Interaction XML API Example: • Get the first 10 records that contains the string “Sci-GaIA Winter School” in the title: http://oar.sci-gaia.eu/search?p=Sci-GaIA%20Winter%20School&f=title&jrec=0&rg=10&of=xm • where: • p=pattern (e.g. your query) • f= field to search within (e.g. “title”, “athors”..) • Get a record from a given DOI • http://oar.sci-gaia.eu/search?p=doi:10.15169/sci-gaia:1466352420.24&of=xm • Get all records uploaded from a given date (e.g. 2016-03-21) to another given date (e.g. today) http://oar.sci-gaia.eu/search?of=xm&d1=2016-03-21&d2=2016-07-05 • where • d1=first date YYYY-mm-dd format • d2=second date YYYY-mm-ddformat

  36. Output of : http://oar.sci-gaia.eu/search?of=xm&d1=2016-03-21&d2=2016-07-05

  37. Programmatic Interaction JSON API You can ask for JSON output format “of=recjson” to obtain it Use the same parameters as XML API Example: • Get a record from a DOI: • http://oar.sci-gaia.eu/search?p=doi:10.15169/sci-gaia:1466352420.24&of=recjson • Get all records uploaded from a given date (e.g. 2016-03-21) to another given date (e.g. today):http://oar.sci-gaia.eu/search?d1=2016-03-21&d2=2016-07-05&of=recjson • where • d1=first date YYYY-mm-ddformat • d2=second date YYYY-mm-ddformat

  38. Output of : http://oar.sci-gaia.eu/search?of=recjson&d1=2016-03-21&d2=2016-07-05

  39. Programmatic Interaction JSON API Example: • Get only the abstract, title and authors of resources: • http://oar.sci-gaia.eu/search?of=recjson&ot=abstract,title,authors • where • ot=output tags (e.g. ‘’ to get all fields, ‘title’ to get titles only)

  40. Programmatic Interaction Python API Invenio Search Engine can be called from within your Python programs via both a high-level and low-level API interface. Use the same parameters as XML and JSON API To know more about Python, XML and JSON API visit this guide: http://oar.sci-gaia.eu/help/hacking/search-engine-api

  41. Programmatic Interaction Download records We need: PUBLIC KEY PRIVATE KEY SIGNATURE Calculate signature: myquery= http://oar.sci-gaia.eu/search?apikey=PUBLIC-KEY&jrec=0&rg=10&of=xm Signature=HMAC-SHA1(myquery,Private-Key) http://oar.sci-gaia.eu/search?apikey=PUBLIC-KEY&jrec=0&of=xm&rg10&signature=SIGNATURE Provided by the system We have to calculate

  42. Programmatic Interaction Upload records We have to: Send an authorisathion request for your IP address to admin@sci-gaia.eu Create a MARCXML file as input (e.g. your_file.xml) Example: • curl –T your_file.xml http://oar.sci-gaia.eu/batchuploader/robotupload/insert -A invenio_webupload -H “Content-Type: application/marcxml+xml” • To know more about Upload: • http://oar.sci-gaia.eu/help/admin/bibupload-admin-guide#2

  43. YOUR_FILE.XML MARC format is the standard in the library world <?xml version="1.0" encoding="UTF-8"?> <collection xmlns="http://www.loc.gov/MARC21/slim"> <record xmlns="http://www.loc.gov/MARC21/slim"> </record> <record xmlns="http://www.loc.gov/MARC21/slim"> </record> <record xmlns="http://www.loc.gov/MARC21/slim"> </record> … </collection>

  44. your_file.xml • <?xml version="1.0" encoding="UTF-8"?> • <collection xmlns="http://www.loc.gov/MARC21/slim"> • <record xmlns="http://www.loc.gov/MARC21/slim"> • <datafieldtag=“ "ind1=" "ind2=" "> • <subfield code=“”></subfield> • <subfield code=“”></subfield> • ….. • </datafield> • <datafieldtag=“ "ind1=" "ind2=" "> • <subfield code=“”></subfield> • <subfield code=“”></subfield> • ….. • </datafield> • …… • </record> • </collection>

  45. your_file.xml <?xml version="1.0" encoding="UTF-8"?> <collection xmlns="http://www.loc.gov/MARC21/slim"> <record xmlns="http://www.loc.gov/MARC21/slim"> <datafieldtag="024" ind1="7"ind2=" "> <subfield code="a">DOI identifier</subfield> <subfield code="2">Type of identifier</subfield> </datafield> <datafieldtag="100"ind1=" "ind2=" "> <subfield code="a">First author</subfield> <subfield code="v">Affiliation</subfield> <subfield code="w">Country</subfield> <subfield code="j">orcid</subfield> </datafield> …… </record> </collection>

More Related