Deploying Hybris on Windows Azure for Cloud Data Management

1. Hybris – cloud - bigdata V1.0 19/11/2014 Yassine MEJRI

2. Hybris-cloud-bigdata Cloud Windows Azure Deploying Hybris on Windows Azure Elasticsearch Kibana Use cases : Analytics, Machine learning. Agenda

3. CLOUD Cloud Computing A standardised IT capability (services, software or infrastructure) delivered via internet technologies in a pay-per-use, self-service way Cloudservices are shared services, under virtualised management, accessible over the internet A style of computing where massively scalable IT-related capabilities are provided “as a service” using internet technologies to multiple external customers

4. CLOUD History 1960 : John McCarthy’s Concept “Computation may someday be organized as a public utility." “Pioneered the concept of delivering enterprise applications via a simple website” 1999 : Salesforce.com 2000 : Microsoft 2001 : IBM “Expanded Sass Concept through web service” 2005 : Amazon “Launch of Amazon web services” 2007 : Google and IBM “Start researching Cloud Computing” 2008 : Gartner Research “Start using Cloud Computing in many organization”

5. Cloud http://www.cloudscreener.com/ Cloud computing providers

6. CLOUD WINDOWS AZURE

7. WINDOWS AZURE WINDOWS AZURE LAYERS

8. WINDOWS AZURE Cloud service model

9. Windows azure Geo-location Datacenter US Europe Asia South Central US North Central US Western Europe South East Asia West US East US Northern Europe East Asia

10. WINDOWS AZURE Building and running apps

11. WINDOWS AZURE Building and running apps

12. Windows Azure Blob Storage

13. Windows azure blob storage Azure Blob storage is a service for storing large amounts of unstructured data, such as text or binary data, that can be accessed from anywhere in the world via HTTP or HTTPS. Common uses of Blob storage include: Serving images or documents directly to a browser Storing files for distributed access Streaming video and audio Performing secure backup and disaster recovery Architecture Services: PutBlob, GetBlob, DeleteBlob, CopyBlob, SnapshotBlob, LeaseBlob…

14. Windows azure blob storage Connexion String : publicstaticfinalString storageConnectionString ="DefaultEndpointsProtocol=http;"+"AccountName=your_storage_account;"+"AccountKey=your_storage_account_key"; Create container : CloudStorageAccount storageAccount =CloudStorageAccount.parse(storageConnectionString); CloudBlobClient blobClient = storageAccount.createCloudBlobClient(); CloudBlobContainer container = blobClient.getContainerReference("images"); container.createIfNotExists(); Java API

15. Windows azure blob storage Change permissions : BlobContainerPermissions containerPermissions =newBlobContainerPermissions(); containerPermissions.setPublicAccess(BlobContainerPublicAccessType.CONTAINER); container.uploadPermissions(containerPermissions); Upload blob : finalString filePath ="C:\\myimages\\myimage.jpg"; CloudBlockBlob blob = container.getBlockBlobReference("myimage.jpg"); File source =newFile(filePath); blob.upload(newFileInputStream(source), source.length()); Download blob : for(ListBlobItem blobItem : container.listBlobs()){ if(blobItem instanceofCloudBlob){ CloudBlob blob =(CloudBlob) blobItem; blob.download(newFileOutputStream("C:\\mydownloads\\"+ blob.getName())); } } Java API

16. Windows azure blob storage Tables NoSQL http://<account>.table.core.windows.net/<table> Services: Insert, Update, Delete, Query, Entity Group Transaction…

17. Windows azure blob storage Queue http://<account>.queue.core.windows.net/<queue>/messages Services: Put, Get, Peek, Delete, Update…

18. CLOUD Windows Azure Management Console

19. cloud Windows azure SDK : Import-AzurePublishSettingsFile -PublishSettingsFile "full path to downloaded file“ New-AzureAffinityGroup -Name pslab-group -Location "East US“ New-AzureQuickVM -ImageName $VMImage -Windows -Name $myVMName -ServiceName $myVMName -AdminUsername $myAdminName -Password $myAdminPwd -AffinityGroup pslab-grou Stop-AzureVM -Name $myVMName -ServiceName $myVMName Start-AzureVM -Name $myVMName -ServiceName $myVMName Restart-AzureVM -Name $myVMName -ServiceName $myVMName   Azure SDK : Powershell, Node.js, Java …

20. HYBRIS Use Case : Deploying Hybris on Windows Azure Deploy Hybris

21. Deployhybris Architecture : auto-scalable horizontal and vertical VIP : windows Azure Load Balancer (Failover, Round Robin, Performance) CDN HTTP/HTTPS Ni N2 N1 N1 N2 Ni Cloud Service F.O Cloud Service B.O AZURE SQL SERVER Azure Blob Storage : Medias, Files, Attachements, orders.pdf…

22. Hybris-CLOUD Windows Azure Blob provides a simple web services interface that can be used to store and retrieve any amount of data. You can configure a specific MediaFolder to store binary data of a Media item directly in Windows Azure Blob. To configure your folder to use Windows Azure Blob you need to have: Windows Azure account Properly created Access Keys For more details read http://www.windowsazure.com/en-us/develop/net/how-to-guides/blob-storage/. Azure cloud Extension

23. Hybris-cloud Azure cloud Extension

24. Hybris-CLOUD https://wiki.hybris.com/display/release5/Using+Windows+Azure+Blob+Media+Storage+Strategy Import extension : azurecloud Configure blob storage in local.properties: Global settings : media.globalSettings.accountKey= media.globalSettings.accountName= media.globalSettings.connection=UseDevelopmentStorage\=True media.globalSettings.endPointProtocol=http media.globalSettings.local.cache=true media.globalSettings.public.base.url=http://127.0.0.1:10000/devstoreaccount1 media.globalSettings.secured=true media.globalSettings.storage.strategy=windowsAzureBlobStorageStrategy media.globalSettings.url.strategy=windowsAzureBlobURLStrategy Azure cloud Extension

25. Hybris-CLOUD 3. How to create new blob storagefolder : …….. media.folder.invoices.accountKey= media.folder.invoices.accountName= media.folder.invoices.connection=UseDevelopmentStorage\=True media.folder.invoices.endPointProtocol=http media.folder.invoices.local.cache=true media.folder.invoices.public.base.url=http://127.0.0.1:10000/devstoreaccount1 media.folder.invoices.secured=true media.folder.invoices.storage.strategy=windowsAzureBlobStorageStrategy media.folder.invoices.url.strategy=windowsAzureBlobURLStrategy …….. Azure cloud Extension

26. Hybris-CLOUD 4. StoringMedia Files : finalMediaModel media =modelService.create(MediaModel.class);media.setCatalogVersion(catalogVersionService.getCatalogVersion("productCatalog","Staged")); finalMediaFolderModelfolder=mediaService.getFolder("invoices"); media.setFolder(folder); mediaService.save(media); Azure cloud Extension

27. Hybris-cloud Secure media access

28. Hybris-cloud You can enable secure media access for specific Media folder by putting in your local.properties file the following property set to true: media.folder.<folderName>.secured=true It means that only secure URL will be rendered for each Media item stored in these folders. It also means that access to these medias will be filtered only by the SecureMediaFilter. ManagingPermissions: Use the MediaPermissionService Using hMC You can grant or deny access to a Media item for a give principal by opening specific Media item and going to Security tab. Using ImpEx Below you can find the example of an ImpEx import script for granting access to a Media item with code 1017895.jpg for the editor principal: INSERT_UPDATE media; code[unique=true]; catalogVersion(catalog(id),version)[unique=true]; permittedPrincipals(uid);;1017895.jpg; clothescatalog:Staged;editor; Secure media access

29. Hybris-CLOUD http://hybrisazure.blob.core.windows.net/hybris/sys_master/root/h3e/hd7/8796157378590.jpg Initialze or Update Hybris : Keep in mind that even if name of custom container is myContainer, then prefix with tenantId is added automatically, so finally container name is sys-master-myContainer. The pattern is sys-<tenantID>-<containerName>. To control cleaning Windows Azure storage on fresh initialization use following global property: media.globalSettings.windowsAzureBlobStorageStrategy.cleanOnInit={true or false} Azure cloud Extension

30. Deployhybris Azure Cloud Service ? VIP : windows Azure Load Balancer (Failover, Round Robin, Performance) CDN HTTP/HTTPS Ni N2 N1 N1 N2 Ni Cloud Service F.O Cloud Service B.O AZURE SQL SERVER Azure Blob Storage : Medias, Files, Attachements, orders.pdf…

31. Deployhybris AzureRunMe

32. Deployhybris Windows Azure Services are described by two important artifacts: Service Definition (*.csdef) Service Configuration (*.cscfg) Your code is zipped and packaged with definition (*.cspkg) Encrypted(Zipped(Code + *.csdef)) == *.cspkg Windows Azure consumes just (*.cspkg + *.cscfg) Packaging and Deploy Hybris

33. Deployhybris # import Azure dll $env:PSModulePath=$env:PSModulePath+";"+"C:\Program Files (x86)\Microsoft SDKs\Windows Azure\PowerShell  Import-Module Azure # Connexion Import-AzurePublishSettingsFile$pubsettings Select-AzureSubscription-SubscriptionName $selectedsubscription Set-AzureSubscription-CurrentStorageAccount $storageAccountName-SubscriptionName $selectedsubscription # Create New deployement $opstat=New-AzureDeployment-Slot $slot-Package $packageLocation-Configuration $cloudConfigLocation-label $deploymentLabel-ServiceName $serviceName # Upgrade deployement $setdeployment=Set-AzureDeployment-Upgrade -Slot $slot-Package $packageLocation-Configuration $cloudConfigLocation-label $deploymentLabel-ServiceName $serviceName-Force # swap deployment, staging  production Move-AzureDeployment -ServiceName $serviceName Devops :  Azure PowerShell cmdlets

34. Hybris-CLOUD Demo : AzureRunMe and Windows Azure Emulator AzureRunMe

35. Elasticsearch Elasticsearch

36. ElasticSearch https://github.com/elasticsearch Java Apache Lucene Plug and play Document Oriented Scalable Clustering Lucene Sharding and replication REST/ JSON Client Apache2 license Elasticsearch

37. Elasicsearch SQL VS ES

38. Elasticsearch Architecture

39. ElasticSearch Core types : String, Integer, Long , Double, Boolean, Date, Binary …. IP type : "address" : { "type" : "ip", "store" : "yes" } { "name" : "Tom PC", "address" : "192.168.2.123" } Geo point type : "location" : {  "type" : "geo_point"} Attachement type :   "my_attachment" : { "type" : "attachment" } Token count type : The token_count field type allows us to store index information about how many words the given field has instead of storing and indexing the text provided to the field. "address_count" : { "type" : "token_count", "store" : "yes" } Mapping fields types

40. ElasticSearch Object types : JSON documents are hierarchical in nature, allowing them to define inner "objects" within the actual JSON. "tweet" : {        "properties" : {            "person" : {  "type" : "object",                "properties" : {                    "name" : {"type" : "object",                        "properties" : {                            "first_name" : {"type" : "string"},                            "last_name" : {"type" : "string"}                        }                    },                    "sid" : {"type" : "string", "index" : "not_analyzed"}                }            },            "message" : {"type" : "string"}} } Mapping fields types

41. ElasticSearch Nested Types : The nested type works like the object type except that an array of objects is flattened, while an array of nested objects allows each object to be queried independently. To explain, consider this document: Mapping : {    "type1" : {        "properties" : {            "users" : {  "type" : "nested",                 "properties": {                    "first" : {"type": "string" },                    "last"  : {"type": "string" }                }            }        }    }} Mapping fields types

42. ElasticSearch Array types : JSON documents allow to define an array (list) of fields or objects. "Product" : [ { "id" : 12 "title" : "iphone", "categories" : [1,3,5,7], "tag" : ["iphone4", "iphone5","iphone6"], "author" : [ { "firstname" : "Francois", "lastname": "francoisg", "id" : 18 }, { "firstname" : "Gregory", "lastname" : "gregquat" "id" : "2" } ]}} Mapping fields types

43. ELASTICsearch Relationnel vs denormalize

44. ELASTICsearch "translation": { "_routing" : { "required" : true, "path" : "project_id" }, "_id" : { "path" : "id" }, "_all" : { "enabled" : "false" }, "dynamic" : "strict", "properties" : { "id" : { "type" : "string", "index" : "not_analyzed" }, "public_id" : { "type" : "integer", "index" : "not_analyzed" }, "project_id" : { "type" : "string", "index" : "not_analyzed" }, "title_na" : { "type" : "string", "index" : "not_analyzed" }, "title" : { "type" : "string", "index" : "analyzed", "analyzer" : "trans_standard" }, "title_cs" : { "type" : "string", "index" : "analyzed", "analyzer" : "trans_cs" }, "description" : { "type" : "string", "index" : "analyzed", "analyzer" : "trans_standard" }, "description_cs" : { "type" : "string", "index" : "analyzed", "analyzer" : "trans_cs" }, "resource_file_id" : { "type" : "integer", "index" : "not_analyzed" }, "created_at" : { "type" : "long", "index" : "not_analyzed" }, "updated_at" : { "type" : "long", "index" : "not_analyzed" }, "any_empty" : { "type" : "boolean", "index" : "not_analyzed" }, "all_empty" : { "type" : "boolean", "index" : "not_analyzed" }, "status" : { "type" : "string", "index" : "not_analyzed" }, "phrases" : { "_id" : { "path" : "id" }, "type" : "nested", "properties" : { "id" : { "type" : "string", "index" : "not_analyzed" }, "iso2_lang" : { "type" : "string", "index" : "not_analyzed" }, "content" : { "type" : "string", "index" : "analyzed", "analyzer" : "trans_standard" }, "content_cs" : { "type" : "string", "index" : "analyzed", "analyzer" : "trans_cs" }, "created_at" : { "type" : "long", "index" : "not_analyzed" }, "updated_at" : { "type" : "long", "index" : "not_analyzed" }, "status" : { "type" : "string", "index" : "not_analyzed" } } } } } Relationnel vs denormalize

45. ELASTICSEARCH Insert Data: $ cat data.json { "index" : { "_index" : "requests" , "_type" : "request" , "_id" : 33 } } { "client" : "client1" , "country" : "FR" , "id" : 1, "ip" : "100.1.1.3", "password" : "test" , "sensor" : "test" , "session" : "EFRFR34344" , "success" : "OK" ,"timestamp" : "1414183085848", "username" : "test" } $ curl -XPOST http://localhost:9200/requests -d @data.json Update : $curl -XPOST 'localhost:9200/test/type1/1/_update'-d '{"doc":{"name":"new_name"}}'}}‘ Delete : $ curl -XDELETE 'http://localhost:9200/twitter/tweet/1‘ Elasticsearch : CRUD

46. Elasticsearch $ curl -XPOST http://localhost:9200/_search?<YOUR_QUERY> Query DSL

47. Elasticsearch 'http://localhost:9200/requests/_search?pretty' -d '{ "query": { "filtered": { "query": { "bool": { "should": [ { "query_string": { "query": "marketing.cars >100" } }, { "query_string": { "query": "marketing.music > 100" } }, { "query_string": { "query": "marketing.electronics > 00" } }, { "query_string": { "query": "marketing.fashion > 100" } } ] } }, "filter": { "bool": { "must": [ { "match_all": {} }, { "exists": { "field": "location" } } ] } } } }, "fields": [ "location", "remoteAddr" ], "size": 1000 }' Query DSL

48. ElasticSearch SearchRequestBuilder requestOne = node.client().prepareSearch().setQuery(QueryBuilders.matchQuery("name","test1")).setSize(1); SearchRequestBuilder requestTwo = node.client().prepareSearch().setQuery(QueryBuilders.matchQuery("name","test2")).setSize(1);MultiSearchResponse response = node.client().prepareMultiSearch().add(requestOne ).add(requestTwo ).execute().actionGet(); // You will get all individual responses from MultiSearchResponse#getResponses()long nbHits =0;for(MultiSearchResponse.Item item : sr.getResponses()){SearchResponse response = item.getResponse();    nbHits += response.getHits().getTotalHits();} MULTI SEARCH API

49. ElasticSearch The bulk API makes it possible to perform many index/delete operations in a single API call. This can greatly increase the indexing speed. Example $ cat requests{"index":{"_index":"test","_type":"type1","_id":"1"}}{"field1":"value1"}$ curl-s -XPOST localhost:9200/_bulk--data-binary@requests;echo{"took":7,"items":[{"create":{"_index":"test","_type":"type1","_id":"1","_version":1}}]} Bulk API

50. ElasticSearch The following snippet captures the basic structure of aggregations: "aggregations" : {    "<aggregation_name>" : {        "<aggregation_type>" : {            <aggregation_body>        }        [,"aggregations" : { [<sub_aggregation>]+ } ]?    }    [,"<aggregation_name_2>" : { ... } ]*} Aggregations