Migrating BizTalk to Azure iPaaS – Part 1

By Bill Chesnut

This is the 1st of a multipart series of Blog Posts to help companies understand Migrating BizTalk to Azure Integration Platform as a Service (IPaaS).

Should I be migrating from BizTalk to Azure iPaaS? This is a question I am being ask more and more lately for a couple of different reasons.

BizTalk development has always been a very specialized skill set with a limited number of resources available in the market, so companies are now starting to look at Azure iPaaS as a viable alternative to BizTalk in terms of finding developers with the correct skill sets.

There are 3 other reasons many companies are looking at moving from BizTalk to Azure iPaaS:

  • cost, consumption pricing instead of product licensing;
  • location of data, many companies are dealing with their consumers over the internet, so having their integration resources in Azure makes more sense;
  • Infrastructure, companies are seeing the advantages of not having to maintain their own infrastructure and using Platform as a Service (PaaS) and Software as a Service (SaaS) offerings as a cost-effective alternative.

Unlike the BizTalk middleware products that covered most features required for all integration scenarios, Azure iPaaS is a collection of Azure services that may or may not be utilised for each scenario.

The base components that make up Azure iPaaS are Service Bus, Logic Apps, API Management and Event Grid. For some integration scenarios you may also use services like but not limited to: SQL Database, Storage, Function App, Web Site (hosing your WebAPI or WebJobs), Cosmos Db and Virtual Machines.  In this series of blog posts, I will look at the different scenarios and the Azure services required.

Instead of spending time here to explain each of the Azure iPaaS services, I have included links to Azure documentation on each product/service mentioned:

All Azure Product/Services can be found here: https://docs.microsoft.com/en-us/azure/#pivot=products

Before you jump full steam into migrating from your on-premises (or cloud hosted) BizTalk to Azure iPaaS, there are a few things that you need to look at to see if the migration makes sense:

  • Location of your data
  • Sensitivity of your data
  • Security Policies for accessing cloud-based resources
  • Location of consumers/partners

Let’s look at each of these items in detail:

Location of your data, if most of your data currently used by BizTalk is located inside of your data center it would be hard to justify moving all of that data to and from Azure to use Azure iPaaS.

If your data is from your consumers/partners and it is coming in and going out via the internet, iPaaS can be a great solution in terms of cost, scalability and availability.

Sensitivity of your data, again this will relate back to the first item, if the data is coming in and going out over the internet, your company has already accepted the risk, so iPaaS will be a viable solution, otherwise, a risk assessment will need to be undertaken.

Security Policies for accessing cloud based resources can sometimes be one of the hardest nuts to crack, it requires educating your management to the benefits and security safe guards of Azure. It also means designing your iPaaS solution to use some of the additional security features available in Azure.  The final point to look at is location of consumers/partners, again if everyone using the integration service of BizTalk are located inside of your data center, having them traverse to Azure and back for data does not make sense, but if many of your consumers/partners are outside of your offices or mobile users, Azure make perfect sense for availability and scale.

Once a company has decided that moving to Azure iPaaS makes sense, the next item to consider is the structure of their existing BizTalk integration solution, does the existing BizTalk solutions follow best practice:

  • having incoming data mapped to an internal/canonical format
  • processed in BizTalk using the internal/canonical format
  • then mapped back to the outbound format before leaving BizTalk

if so, this will make the migration to Azure iPaaS a less disruptive and staged process.

If you would like to speak more about this topic or other ways Azure can help to transform your business processes, I would be glad to come out and present or chat directly with you, please use the Contact Us page to reach out to me.

This series will continue with the following blog posts:

  • Migrating inbound messaging to Azure using Logic Apps, Event Grid, Service Bus and the BizTalk Claim Check Pipeline component that I have built.
  • Migrating outbound messaging to Azure, using the same tooling as inbound
  • Choices for message archiving in Azure
  • End to End Monitoring of Azure iPaaS
  • EDI (EDIFACT, X12 & AS2) Capabilities of Azure iPaaS
  • Azure iPaaS CI/CD

Cross Posted to https://www.biztalkbill.com

Why Azure API Management

By Bill Chesnut

Being a big fan of Azure API Management (APIM), I get ask often “why should you use Azure API Management to publish my APIs? They are in Azure in an App Service as WebAPI”. To address this question we are going to go through the components and features of APIM. One thing I will not be coving is cost, too many times people first look at the cost of a product/service before they look at the features, I will leave the cost analysis to later and concentrate on the features.

Components

There are 3 main components of APIM: API gateway, Azure Portal and Developer Portal, let’s talk quickly about each of these. Complete details can be found here: https://docs.microsoft.com/en-us/azure/api-management/api-management-key-concepts

API gateway

This is the gatekeeper, it accepts calls, routes them to the location of your API, verifies access, enforces quotas and rate limits, caches backend responses, manipulates the requests and responses and provides logging and analytics. There is nothing in the gateway that you could not do in your own code, but I strongly believe that you should not try to reinvent the wheel, concentrate on your code functionality not the plumbing.

Azure Portal / Management APIs

Originally when Microsoft purchased APIM, there was a Publisher Portal where all the management of your APIs was done. In the last year or so Microsoft have migrated all of this functionality to the Azure Portal, offering the added benefit of RBAC (Resource Based Access Control). As I am talking about the Azure Portal, almost all of the portal functionality is available via the management APIs. The Azure portal allows the definition or import of the API schemas, the packaging of the APIs into products, configuration of policies, and the management of users and analytics. They have also recently added the ability to test APIs directly from the Azure portal.

image

Developer Portal

This is the main entry point for developers that want to use your API. The developers register themselves on the developer portal, you can decide if you want to approve their registration and what method you want them to use to register: Username & Password, Azure Active Directory, Azure Active Directory B2C, Facebook, Google, Microsoft Account or Twitter. The developer portal allows the user to discover your Product (which is made up of your APIs), manage their subscriptions to your Products, test the APIs, get sample code to call the API in different languages (Curl, #, Java, JavaScript, ObjC, PHP, Python and Ruby) and view the analytics of their own usage.

image

Features

To really start talking about the answer to the question “why should you use APIM”, we need to start looking at the features of APIM. Not every company has the same requirements for publishing APIs, so there are features of APIM that your company may or may not use.

When you start looking at feature, one of the things you need to look at is are your APIs going to be Public or Private. If you need them available to external parties including partners & customers, they will need to be public. If they are indented for internal developer use only, they can be private. I would actually argue that they should all be public It is amazing the number of times I have seen companies build APIs for internal use only to then have to figure out how to make them public after the fact. Public does not  mean available to everyone, public typically means available to partners or customers.

The feature that I will look at in some depth are: Security, Policies, Importing APIs, Analytics/Logging and Versioning/Revisions

Security

Now that we have decided to publish our APIs, how are we going to protect them? There are several security layers and ways to protect your APIs. The first is Azure API Management subscriptions, by default in APIM each set of APIs are part of a Product and users of a Product get a subscription to that Product, The subscription has a primary and secondary key and one of these needs to be passed in the header of the request to the APIM. This protects your API from being called by anyone without a subscription key. Request without a key are stopped at the APIM gateway, never reaching your API backend. The 2nd layer that you can choose is OAuth 2.0 or OpenID Connect, of which there are many providers. When you configure your APIs to use either OAuth 2.o or OpenID Connect, the caller would need to supply a valid authorization token in the request header. The final layer of security is between APIM and your APIs and there are several options for this, you can pass the OAuth 2.0 or OpenID Connect token through, you can using Client Certificates, you can user Basic Authentication and finally you can use the static IP address of your APIM instance.

image

Policies

One of the biggest reasons I point out to clients thinking about APIM is the power of the polices. There is so much you can do with APIM policies, they can be implement at the Product level, so they apply to all APIs that are part of the Product (Note: APIs can be in multi Product). They can be implemented at API level so they apply to all operation in the API, or they can be implemented for a single operation. There are 6 categories of polices:

image

I would like to call out a few of my favourites:

  • Validate JWT – Enforces existence of validity of a JWT
  • Control flow – Conditionally applies policy statements based on the evaluation of Boolean expressions
  • Log to Event Hub – Send message in the specified format to a message target defined by the Logger entity
  • The whole set of Caching polices – being able to add caching to your APIs without touching the code
  • Convert JSON to XML and Convert XML to JSON
  • Set body (specifically using a Liquid template) – used primarily for the REST to SOAP, but not limited to that

That is a small number of the available polices, but do give you a good idea of the power of polices in APIM

Importing APIs

This feature makes it easy to get started with APIM, it is how you get your APIs into APIM. Microsoft have just implemented a couple of standards here but they have made it quick and easy by giving you lots of options. These are the available options:

image

Analytics/Logging

Analytics and Logging are 2 distinct types of data that needs to be collected from your APIs. I have typically seen very good logging in API code, but rarely good Analytics. I know I have spent my fair share of time in IIS logs trying to figure out some usage analytics. Analytics gives you access to usage, health and activity data around your APIs. This can be viewed in the APIM Publisher Portal (still to be migrated to the Azure Portal) or in Azure Portal under Azure Monitor. If you need to log data in APIM before or after your API call, you can use the Log to Event Hub policy to capture that information, very helpful in the SOAP to REST scenarios. The final piece is a bit of a mixture, Microsoft have recently enabled the ability to connect Application Insights to APIM.  Application Insights provides insights into the request, exceptions and dependencies.

imageimage

Versioning/Revisions

These are 2 very helpful features of APIM, Many of us start our our API journey without thinking about how we are going to manage change. If we were to go back to the early days of COM programing the rule was never change the interface, you can change the code but never the interface. In the real world interfaces need to change, so how we manage that change is the important. With APIM, if we originally published our APIs without versioning this feature will allow you to maintain the original without a version but then add a new versioned copy of the same API. With versioning there may also be a need to maintain several version of the same API because of different usage scenarios, APIM will allow each version to point to a different backend set of APIs, or you can use policies to update earlier version to conform to the new backend. APIM supports the version in either the path, header or query string. Revisions are useful when you change the code behind your API but not the interface. Revisions allow you deploy and test the new revision without making it the active revision of your API.

image

Other Consideration for Using APIM

There are a few other things that may help to make your decision to publish your APIs in APIM:

  • APIM instances have a static IP address
  • All APIM instances above developer are highly available
  • APIM Developer and Premium can be connected to a Virtual Network for VPN connectivity
  • APIM Premium can have instances in multiple Azure Regions and includes an internal Traffic Manager and shared configuration
  • APIM standard instance can be placed behind Traffic Manager, but configuration is not shared

I hope the information provide in this blog post can help you decide if APIM is the right solution for publishing your APIs.

Cross Posted to https://www.biztalkbill.com