Microsoft IT Integration Agenda: Composable Architecture & Exception Handling

1. Ashish Jain, Mike Bizub, & Vignesh Sukumar • Microsoft CSE&O How Microsoft IT does Integration

2. Agenda • Recap of who we are and what we do • Composable architecture and how we’ve implemented the Routing Slip Pattern • Hot-Path Telemetry • ALM, Dev-ops, and Governance • Business Continuity and Enterprise Disaster Recovery • Migration Tools update • Appendix

3. Microsoft Integration Landscape 150M+ Messages Per Month 1,000 + Partners – Internal and External 3,400+ Integration Streams Centrally Managed Enabling Business Processes Supply Chain Contract Management Licensing Sales Benefits Legal Payroll Finance Travel Suppliers Distributors … etc 200+ Microsoft Systems $1.6 B Daily bank treasury transaction 100+ BizTalk Servers in use today Message Standards X12 – 30% EDIFACT-14% XML-12% SWIFT-12% Other – 32% Multi-Platform BTS 2016, MABS, Logic Apps

4. Recap – The People (giving credit where it’s due)

5. Recap – B2B Approach

6. Composable Architecture Adapter Router Msg Pub Adapter Business Prop Promotion Msg Publisher Source BTS On-P Transform Get Context Source File Trigger File Processor File Deleter Adapter Router Msg Pub Adapter Business Prop Promotion Get Context FFDecode APIM Source SFTP Trigger SFTP Processor APIM Policy, FFDecode, Business Prop Promotion, Adapter Router Pub / Sub Source HTTP Request Trigger APIM Policy Destination BTS On-P Msg Subscriber File Processor Destination FFEncode Transform Get Context Adapter Router Msg Subscriber Processor APIM Destination SFTP Processor Determines physical endpoint adapter Destination HTTP POST

7. Telemetry • Hot-path • Run Scope for exception handling within logic apps (similar to a try/catch) • All exceptions are not created equal; may need to enumerate through exceptions before posting • Single logic app concept to move exceptions into an Event Hub (not the same used by Logic Apps) • Azure Functions consume exceptions from Event Hub for both App Insights and internal Ticket management system • Warm-path • OOB Logic App infrastructure plus Integration Account • Event Hub and OMS • OMS drives non-flow business exceptions

8. ALM and Dev-ops IST PST Enterprise Commerce / Volume Licensing Supply Chain / Core Integration Service CFE Common Backlog (shared and triaged across regions) Feature Branches Feature Branches Feature Branches sprint Sprint teams live here: iterating through tasks and user stories until features are ready or sprint ends. What’s passing tests makes it to release Pull request into develop ‘develop’ CI / CD DRI’s work here: Executing PR’s into ‘release’ and managing the deployment across primary/secondary regions. Issues found here can be dealt with by DRI following Git best practices for this scenario Pull request into release ‘release’ Pre-production (PPE) DRI’s work here: Executing PR’s into ‘master’ and managing the deployment across primary/secondary regions. Issues found here can be dealt with by DRI following Git best practices for this scenario Pull request into master ‘master’ Production

9. ALM and Dev-ops • Git for source control. • Each code check-in goes thru pull request (code review), at least 2 approvals required and enforced thru branch policies • Code is compiled – gated check in build • Policies for security or governance enforcement (e.g. scan for potential security risks or ensure the person checking in code is supposed to be) • Azure DevOps (formerly known as Visual Studio Online (VSO)) for build and release management. • Two builds • Gated check in – code check in and performs compilation of the code executes unit tests • Continuous Integration build – used to deploy/release, compiles the code, runs the unit tests and creates a build structure to be consumed by release/deployment. • Release • One release pipeline for each environment • Each release pipeline consists of steps to perform various tasks to complete tasks in a given environment and run functional tests • Branching Strategy • Following a basic Git practice of using a ‘develop’, ‘release’, and ‘master’ branch • Deployments into CI, PPE (Pre-Production), and Production • Deploy core services components on a sprint cadence • Unit test / functional test • Unit test what can be unit tested • Functional test for things like Logic Apps, APIM Policies, and any other non-unit testable capability that makes up the integration service • In general • Agile != ‘No Process’ • Dev-ops != ‘No Governance’

10. Security and Governance • IP Whitelist to further secure logic apps (Migration to ISE) • Managed Identities for AAD auth between logic apps (HTTP Actions/Triggers). Remove SAS token from URL • Things to consider in a hybrid environment • Access Management – who has access to my system when considering PaaS and on-premises resources (e.g. BizTalk) • Release Management – Separation of duties between developers and release management in production • Privacy – How to handle highly confidential data or personal data (e.g. cosmosdb, table/blob storage, Azure SQL • Cost Optimization – Once you begin to figure this PaaS thing out, how do you corral the cost

11. AIS Migration Story Milestones Achieved Metadata driven Architecture Migration Accelerators • Metadata Creation • Schemas Json • Maps Json • End Point • Flow Context • Partner Flow Context • Schema Provisioning • Partner – Agreement json creation • Maps to XSLT conversion • Metadata creation • AB Testing

12. Enterprise Integration Disaster Recovery • Scheduled Sync Solution’s • Runtime Sync • LogicApps scheduled to run every 3 mins to update secondary region Integration account to make them sync and vice versa post DR. • AS2 Synchronization (MIC Algorithm) • X12 Synchronization (Control Numbers Sync) • EDIFACT Synchronization (Control Numbers Sync) • Artifacts Sync • Integration Account Artifacts sync scheduled to run every 1 hour to secondary region and vice versa if needed. • Schemas • Maps • Certificates • Partners • Agreements • Assembly • Batch Configurations • RosettaNet PIP • Pre/Post DR Scripts Execution • Bump Interchange Control Numbers in secondary region and vice versa post DR. • Flip Azure Traffic Manager(ATM) routing from Primary to secondary APIM and vice versa post DR. • Disable/Enable LogicApps. • Lookup Azure Storage Table sync.

13. Appendix • Open-source Accelerator Tools, Demos and scripts available at Github https://aka.ms/Integrate2019 • Write to us for any integration queries: btsmigrationtool@microsoft.com

14. Recap – B2B Approach • Use one Integration Account and set of logic apps across multiple Businesses and Trading Partners (Partner info, Agreements, Document Types, Schemas, Maps, Batch Configurations) • Flow concept static but type can be overridden based on message (e.g. different logic apps that process X12 or EDIFACT messages) • Metadata used to populate runtime properties specific to a TP or a LOB (currently stored in a CosmosDB and accessed via Azure Function with caching) • A ‘Routing Slip’ can override the URL of next logic app to load (can be overridden using APIM Policy)

15. Recap – B2B Approach Trading Partner Gateway Exchange Protocol Document Protocol Business Process / Orchestration Line of Business Integration On-premises Connectivity LOB Application • MTPB/HTTPS • SFTP • AS2 • RNIF • BTF • EDI X12 • EDI EDIFACT • OAGIS • RosettaNet • BTF • Custom message processing • Transforms • Business Property Promotion • Flat file encoding • LOB specific enhancement (custom processing) • EHC • EIS Adapter • BizTalk Trading Partner Gateway Exchange Protocol Document Protocol Business Process / Orchestration Line of Business Integration On-premises Connectivity LOB Application • Flat file decoding • LOB specific enhancement (custom processing) • EIS Adapter • BizTalk • APIM/HTTPS • SFTP • AS2 • RNIF • BTF • EDI X12 • EDI EDIFACT • OAGIS • RosettaNet • BTF • Custom message processing • Transforms • Business Property Promotion

16. Recap – B2B Approach Secondary Logic Apps Service Lines (Business Units) • Gateway provides an Active/Active store and forward mechanism for B2B type traffic. Trading Partners post messages to endpoint which round robins between the two regions. • There are two deployed installations of B2B Service. However, only one is active and any time. • At deployment time, artifacts are deployed to both regions. This includes all static data including Logic Apps, Functions, Metadata • During runtime, there are properties that are updated in the Enterprise connectors that must be replicated across both regions for business continuity. In the passive region, there is a set of logic apps responsible for listening to events in primary region and updating the appropriate Integration Account Artifacts • EHC provides secure connectivity to on-premises resources. B2B Service uses corpnet BTS 2016 servers • BTS 2016 provides pass-thru connectivity to existing LOB Applications over a wide range of technologies • BTS 2016 provides the necessary capabilities to send and receive messages from LOB Applications Corpnet BTS2016 APIM EHC Gateway TP Replication for Busn. Continuity Logic Apps Service Lines (Business Units) Corpnet BTS2016 Gateway APIM EHC Primary

17. Recap – B2B Approach PAExchange ProtocolAS2V2 PADocument ProtocolX12V2 PADebatch GoodMessagesV2 PABusiness Process Orchestration V2 PALob IntegrationV2 PALobAdapter BTSV2 EHC BTS On-premises LOB Application PATPC MessageInfoV2 Adapter APIM APIM PADocument ProtocolEdifactV2 PADebatch BadMessages V2 Integration Account Integration Account Integration Account Azure Function PADebatch GeneratedAcks V2 CosmosDB Integration Account PADebatch ReceivedAcks V2 APDocument EdifactV2 Integration Account Integration Account Integration Account APExchange ProtocolAS2V2 APDocument ProtocolX12V2 APBusinessProcess OrchestrationV2 APLob IntegrationV2 BTS On-premises LOB Application Integration Account APIM APIM APBatch MessagesX12V2 APMessage SenderX12V2 APAsyncMDN ProcessorV2 Azure Function Integration Account CosmosDB

18. Composable Architecture • Patterns and Composable Services • Composable Flows for integrations • Leverage existing patterns or compose a new one • Microservices should be autonomous • Refactor to net-new, don’t change an existing pattern or service

19. Composable Architecture • Routing Slip Pattern • APIM, API, and Policies to expose and describe pattern • Context to describe the flow specific parameters that allows for multiple scenarios to flow through one pattern • Generic ARM Templates to describe concepts that may need implementation specific properties at deployment time (e.g. FileAdapter, SFTP, Message publishing and subscribing patterns) • AdapterRouter concept for endpoint implementation details