1 / 23

Manipulating SharePoint Lists with curl

Learn how to manipulate SharePoint lists using curl command-line tool. Explore use cases like authenticating, uploading/downloading files, and CRUD operations. Join us for a networking event at Reston Town Center.

ehines
Download Presentation

Manipulating SharePoint Lists with curl

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Housekeeping…

  2. Thanks to all our Sponsors!!!

  3. Join us at #SharePint! Why? To network with fellow Microsoft professionals What? SharePint!!! When? 5:45 PM Where? Lets take over the Crafthouse at Reston Town Center!

  4. Manipulating Share Point Lists using curl Presented by don@hautsch.com

  5. Agenda Use cases Authenticating Upload/Download files SP List Metadata CRUD - Create, read, update and delete Scripts

  6. Use cases

  7. Authenticating On premises you can use the --ntlm with curl to easily upload or download a file. Create an entry in your $HOME/.netrc file machine sharepoint login ACME\scott password tiger To upload or download SP_SHRD_DOCS=http://sharepoint/sites/eng/Shared%20Documents curl -n --ntlm --upload-file $HOME/my_file.txt $SP_SHRD_DOCS/myfile.txt curl -n --ntlm -o $HOME/my_file.txt $SP_SHRD_DOCS/myfile.txt

  8. Authenticating to sharepoint.com Authenticating at sharepoint.com is a bit harder than on premises Authentication via https://en.wikipedia.org/wiki/Federated_identity This script https://github.com/dhautsch/etc/blob/master/scripts/xfer_sp_file will handle file upload or download for local or cloud SharePoint SP_SHRD_DOCS=https://acme.sharepoint.com/sites/etl/Shared%20Documents xfer_sp_file /etc/hosts $SP_SHRD_DOCS/hosts.txt xfer_sp_file $SP_SHRD_DOCS/hosts.txt /tmp/hosts.txt

  9. Authenticating to sharepoint.com continued Most likely you will have to deal with a HTTPS proxy Set the following environment variable for curl export HTTPS_PROXY=https://proxy.acme.com:8080 Your $HOME/.netrc will have to have the site name from sharepoint.com machine acme.sharepoint.com login ACME\scott password tiger

  10. Authenticating to sharepoint.com continued Create a saml request file saml.xml to get a security token <s:Envelope xmlns:s='http://www.w3.org/2003/05/soap-envelope' xmlns:a='http://www.w3.org/2005/08/addressing' xmlns:u='http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd'> <s:Header> <a:Action s:mustUnderstand='1'>http://schemas.xmlsoap.org/ws/2005/02/trust/RST/Issue</a:Action> <a:ReplyTo><a:Address>http://www.w3.org/2005/08/addressing/anonymous</a:Address></a:ReplyTo> <a:To s:mustUnderstand='1'>https://login.microsoftonline.com/extSTS.srf</a:To> <o:Security s:mustUnderstand='1' xmlns:o='http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd'> <o:UsernameToken><o:Username>scott@acme.com</o:Username><o:Password>tiger</o:Password></o:UsernameToken> </o:Security> </s:Header><s:Body> <t:RequestSecurityToken xmlns:t='http://schemas.xmlsoap.org/ws/2005/02/trust'> <wsp:AppliesTo xmlns:wsp='http://schemas.xmlsoap.org/ws/2004/09/policy'> <a:EndpointReference><a:Address>acme.sharepoint.com</a:Address></a:EndpointReference> </wsp:AppliesTo> <t:KeyType>http://schemas.xmlsoap.org/ws/2005/05/identity/NoProofKey</t:KeyType> <t:RequestType>http://schemas.xmlsoap.org/ws/2005/02/trust/Issue</t:RequestType> <t:TokenType>urn:oasis:names:tc:SAML:1.0:assertion</t:TokenType> </t:RequestSecurityToken> </s:Body></s:Envelope>

  11. Authenticating to sharepoint.com continued Make request to get security token curl --connect-timeout 10 --max-time 120 -n --proxy-anyauth -o curl_reponse.xml \ --data @saml.xml --ciphers AES256-SHA \ -H 'accept:application/atom+xml;charset=utf-8' \ https://login.microsoftonline.com/extSTS.srf Extract token from file and deswizzle the string, convert “&amp;” to “&” SAML_TOKEN=$( perl -lane 'print $1 if m!<wsse:BinarySecurityToken[^>]*>(.*)</wsse:BinarySecurityToken!' curl_reponse.xml | \ perl -p -e 's/&amp;/&/sg;s/&lt;/</sg;s/&gt;/>/sg;s/&quot;/"/sg;' )

  12. Authenticating to sharepoint.com continued Make a form sign in request to get cookies, these cookies last a while echo $SAML_TOKEN > form.dat curl --connect-timeout 10 --max-time 120 -n --proxy-anyauth -o /dev/null \ -c cookies.txt \ --data @form.dat \ --ciphers AES256-SHA \ -H "HOST:acme.sharepoint.com" \ -L "https://acme.sharepoint.com/_forms/default.aspx?wa=wsignin1.0" COOKIE1=$(perl -lane 'print "$1=$2" if m!(rtFa)\s+(\S+)!' cookies.txt) COOKIE2=$(perl -lane 'print "$1=$2" if m!(FedAuth)\s+(\S+)!' cookies.txt) Now it gets easy again

  13. Updating a SharePoint list needs a digest Request an authorization digest from SharePoint which will last a while but expire before the cookies, for example, 1 hour versus 8 hours. CLOUD_SP=http://acme.sharepoint.com/sites/eng curl --connect-timeout 10 --max-time 120 -n --proxy-anyauth \ --data "''" \ --ciphers AES256-SHA \ -H "Cookie:$COOKIE1;$COOKIE2" \ -H 'accept:application/atom+xml;charset=utf-8' \ -o digest.xml \ "$CLOUD_SP/sites/eng/_api/contextinfo" DIGEST=$(perl -lane 'print "Authorization:Bearer $1" if m!<d:FormDigestValue[^>]*>(.*)</d:FormDigestValue!' digest.xml) DIGEST_TIMEOUT=$(perl -lane 'print $1 if m!<d:FormDigestTimeoutSeconds[^>]*>(.*)</d:FormDigestTimeoutSeconds!' digest.xml)

  14. Upload/Download file sharepoint.com Upload curl --connect-timeout 10 --max-time 120 -n --proxy-anyauth \ -H "Cookie:$COOKIE1;$COOKIE2" \ -H "$DIGEST" \ --ciphers AES256-SHA \ --data-binary @my_file.txt \ -o curl_response.xml \ "$CLOUD_SP/_api/web/getfolderbyserverrelativeurl('Shared%20Documents')/Files/Add(url='my_file.txt',overwrite=true)" Download curl --connect-timeout 10 --max-time 120 -n --proxy-anyauth \ -H "Cookie:$COOKIE1;$COOKIE2" \ -H "$DIGEST" \ --ciphers AES256-SHA \ -o my_file.txt \ "$CLOUD_SP/Shared%20Documents/my_file.txt"

  15. SP List Metadata - Get metadata, write ListItemEntityTypeFullName to dot file curl -s --max-time 30 -n --proxy-anyauth \ --dump-header curl_header.txt \ --ciphers AES256-SHA \ -H 'accept:application/atom+xml;charset=utf-8' \ -H "Cookie:$COOKIE1;$COOKIE2" \ -H "$DIGEST" \ -o curl_response.xml \ "$CLOUD_SP/_api/lists/getByTitle('Bogus')" for s in Created EntityTypeName Id ItemCount \ LastItemDeletedDate LastItemModifiedDate \ LastItemUserModifiedDate ListItemEntityTypeFullName Title do perl -lane 'print "$1=$2" if m!<d:('$s')[^>]*>([^<]+)!' curl_response.xml done | tee dot_file.txt

  16. CRUD - Create list item, write Id to dot file source dot_file.txt curl -s --max-time 30 -n --proxy-anyauth \ --dump-header curl_header.txt \ --ciphers AES256-SHA \ -H 'accept:application/atom+xml;charset=utf-8' \ -H "Cookie:$COOKIE1;$COOKIE2" \ -H "$DIGEST" \ -o curl_response.xml \ -X POST \ --data "{ '__metadata': { 'type': '$ListItemEntityTypeFullName' }, 'Title': 'New-18999' }" \ -H 'content-type:application/json;odata=verbose' \ -H 'IF-MATCH:*' \ "$CLOUD_SP/_api/lists/getByTitle('Bogus')/items" for s in Created Id Modified Title do perl -lane 'print "$1=$2" if m!<d:('$s')[^>]*>([^<]+)!' curl_response.xml done | tee dot_file.txt

  17. CRUD - Update list item, look for 204 in header source dot_file.txt curl -s --max-time 30 -n --proxy-anyauth \ --dump-header curl_header.txt \ --ciphers AES256-SHA \ -H 'accept:application/atom+xml;charset=utf-8' \ -H "Cookie:$COOKIE1;$COOKIE2" \ -H "$DIGEST" \ -o curl_response.xml \ -X MERGE \ --data "{ '__metadata': { 'type': '$ListItemEntityTypeFullName' }, 'Title': 'Updated-18999' }" \ -H 'content-type:application/json;odata=verbose' \ -H 'IF-MATCH:*' \ "$CLOUD_SP/_api/lists/getByTitle('Bogus')/items($Id)" egrep '^HTTP' curl_header.txt

  18. CRUD - Read list item, note we are using $filter rather than /items($Id) curl -s --max-time 30 -n --proxy-anyauth \ --dump-header curl_header.txt \ --ciphers AES256-SHA \ -H 'accept:application/atom+xml;charset=utf-8' \ -H "Cookie:$COOKIE1;$COOKIE2" \ -H "$DIGEST" \ -o curl_response.xml \ "$CLOUD_SP/_api/lists/getByTitle('Bogus')/items?%24filter=ID%20eq%20$Id" for s in Created Id Modified Title do perl -lane 'print "$1=$2" if m!<d:('$s')[^>]*>([^<]+)!' curl_response.xml done | tee dot_file.txt

  19. CRUD - Delete list item, look for 200 in header curl -s --max-time 30 -n --proxy-anyauth \ --dump-header curl_header.txt \ --ciphers AES256-SHA \ -H 'accept:application/atom+xml;charset=utf-8' \ -H "Cookie:$COOKIE1;$COOKIE2" \ -H "$DIGEST" \ -o curl_response.xml \ -X DELETE \ -H 'IF-MATCH:*' \ "$CLOUD_SP/_api/lists/getByTitle('Bogus')/items($Id)" egrep '^HTTP' curl_header.txt

  20. Scripts The CRUD examples are from https://github.com/dhautsch/etc/blob/master/scripts/sp_curl_examples For a general perl script that works on Linux and Sun https://github.com/dhautsch/etc/blob/master/perl/get_sp_list_items.pl For a perl script that updates a file from list when list is newer https://github.com/dhautsch/etc/blob/master/perl/update_file_from_sp.pl

  21. Scripts - get_sp_list_items.pl examples get_sp_list_items.pl -create -data "{ '__metadata': { 'type': 'SP.Data.BogusListItem' }, 'Title': 'New_bogus-$$' }" \ http://sharepoint/sites/etl Bogus get_sp_list_items.pl -update -id $ID -data "{ '__metadata': { 'type': 'SP.Data.BogusListItem' }, 'Title': 'Updated-$$' }" \ http://sharepoint/sites/etl Bogus echo "{ '__metadata': { 'type': 'SP.Data.BogusListItem' }, 'Title': 'New_bogus-$$' }" | get_sp_list_items.pl -create -data @- \ http://sharepoint/sites/etl Bogus echo "{ '__metadata': { 'type': 'SP.Data.BogusListItem' }, 'Title': 'Updated-$$' }" | get_sp_list_items.pl -update -id $ID -data @- \ http://sharepoint/sites/etl Bogus get_sp_list_items.pl -create -data "@create_data.txt" http://sharepoint/sites/etl Bogus get_sp_list_items.pl -update -id $ID -data "@update_data.txt" http://sharepoint/sites/etl Bogus get_sp_list_items.pl -delete -id $ID http://sharepoint/sites/etl Bogus get_sp_list_items.pl -meta http://sharepoint/sites/etl Bogus get_sp_list_items.pl -digest http://sharepoint/sites/etl get_sp_list_items.pl http://sharepoint/sites/etl Bogus get_sp_list_items.pl -query '$top=5' http://sharepoint/sites/etl Bogus

  22. Scripts - get_sp_list_items.pl called from python

  23. Scripts - get_sp_list_items.pl usage

More Related