Hi there! This blog post intends to explain a possible way to implement a claim-check pattern with different Azure Integration Services components and highlights a few pitfalls.
The claim-check pattern
The claim-check pattern is designed to facilitate the handling of large messages by reducing processing time and the load on the components involved in message handling. It’s especially valuable in scenarios where there’s a need to bypass certain limitations, such as the message size limit in the Service Bus REST API.
Project Background
During a recent project, our team developed an API to facilitate the reception of messages from partners, queuing these messages on Service Bus, processing them through Logic Apps, and ultimately forwarding them to the backend system. The setup involved several key components:
- API on APIM: Hosts a secure HTTP endpoint with a proper Open API specification.
- Service Bus: Ensures messages are queued during outages or maintenance windows for the Logic App.
- Logic Apps: Handles message translation and utilizes connectors for sending data to the backend system.
This system worked seamlessly, incorporating a Service Bus Topic Subscription system based on to API Subscription Keys, allowing for messages from each partner to be queued in distinct Topic Subscriptions. However, challenges were encountered with messages larger than 1MB.
The Challenge with Large Messages
Despite utilizing a Premium tier Service Bus, a notable limitation was encountered: the REST API’s message size limit is strictly 1MB, contrasting with AMQP’s size limitation. This limitation necessitated an innovative solution to handle large messages effectively.
Implementing the Claim-Check Pattern
Enter the claim-check pattern!
Implementing this pattern consists of having APIM store the message body on storage account, constructing metadata about the message (including the Blob URL) and sending that metadata to Service Bus.
A Logic App can then use that metadata to load the message from Blob and process it further. Watch out, in this scenario we do not have an Application Gateway with Web Application Firewall (WAF) and body inspection enabled in front of the API. There are limits on the WAF as well (V1:128Kb, V2: 2Mb): Web application firewall request size limits in Azure Application Gateway – Azure portal | Microsoft Learn
Focus on APIM Implementation
The implementation process begins with APIM, where a variable for the message body is defined.
<set-variable name=“messageBody” value=“@(context.Request.Body.As<string>(preserveContent: true))” />
It’s important to use the preserveContent flag, if not, the Body is no longer accessible as APIM will perform a destructive read.
Now we also want to generate a Blob Url with a unique Blob name. Replace “BlobEndpointAndContainer” with your storage account URL and container name. For example: “https://demo.blob.core.windows.net/containername”
<set-variable name=“FullBlobUrl” value=“@{var requestId = context.RequestId; return string.Format(“{0}/{1}.xml“, “BlobEndpointAndContainer“, requestId);}” />
The extension is hardcoded to xml. You could build some logic to parse the Content-Type Header to support JSON and XML.
Time to construct the upload HTTP PUT request to the BLOB endpoint while using Managed Identity security.
<send-request mode=“new” response-variable-name=“blobResponse” timeout=“20” ignore-error=“false”>
<!– PUT Blob –>
<set-url>@((string)context.Variables.GetValueOrDefault(“FullBlobUrl”))</set-url>
<set-method>PUT</set-method>
<set-header name=“x-ms-blob-type” exists-action=“override”>
<value>BlockBlob</value>
</set-header>
<set-header name=“x-ms-version” exists-action=“override”>
<value>2019-07-07</value>
</set-header>
<set-body>@{
return (string)context.Variables[“messageBody”];
}</set-body>
<authentication-managed-identity client-id=“123456489-1234-1234-9cce-8e7c905457c7” resource=“https://storage.azure.com” />
</send-request>
Ensuring APIM first uploads the message to BLOB storage, followed by forwarding the metadata to Service Bus through the backend URL, is crucial.
Previously, we attempted the reverse process and observed a hiccup: about one in every twelve messages stumbled at the Logic App stage. The reason? The message was processed earlier than APIM has written the message to Storage Account. This mismatch led to a ‘Blob not found’ error on the processing Logic App.
Anyway, if you are here looking for a way to upload messages to Service Bus via a APIM POST request, find below the example code. Remember, the size limit as of writing this article is 1MB.
<send-request mode=“new” response-variable-name=“serviceBusResponse” timeout=“20” ignore-error=“false”>
<set-url>@(“https://yourServiceBusNamespaceHere.servicebus.windows.net/yourTopicNameHere/messages”)</set-url>
<set-method>POST</set-method>
<set-header name=“Content-Type” exists-action=“override”>
<value>application/json</value>
</set-header>
</set-header>
<set-body>
@{
var message = new {
MessageType = “YourMessageType”,
FileName = (string)context.Variables.GetValueOrDefault(“FullBlobUrl”)
};
var serializedMessage = JsonConvert.SerializeObject(message);
return serializedMessage;
}
</set-body>
<authentication-managed-identity resource=“https://servicebus.azure.net/” client-id=“ManagedIdentityGuidHere” ignore-error=“false” />
</send-request>
With the message already on Storage, we continue sending the metadata to Service Bus. Let’s add a rewrite URI policy snippet just to make sure the API operation name does not get appended to the backend URL:
<rewrite-uri template=“/” copy-unmatched-params=“false” />
You can set the Backend URL in many ways, either via a Backends resource, via the API Specification or via policy:
<set-backend-service base-url=“https://yourServiceBusNamespaceHere.servicebus.windows.net/yourTopicHere/messages” />
Now, we need to make sure we are using the correct HTTP Verb, Content-Type header and managed identity. Also, find again some code to generate a metadata JSON.
<set-method>POST</set-method>
<set-header name=“Content-Type” exists-action=“override”>
<value>application/json</value>
</set-header>
<set-body>@{
var message = new {
MessageType = “LargeMessageType”,
FileName = (string)context.Variables.GetValueOrDefault(“BlobFileName”)
};
var serializedMessage = JsonConvert.SerializeObject(message);
return serializedMessage;
}</set-body>
<authentication-managed-identity resource=“https://servicebus.azure.net/” client-id=“12345678-1234-1234-1234-1234567890” ignore-error=“false” />
One more pitfall for having APIM send to Service Bus via REST API, is that we noticed that seemingly at random there might be “transient” errors while doing so. It is important to build in some ‘retry mechanism’ to cope with this. In the Backend part of the policy, add below policy snippet:
<backend>
<retry condition=“@(context.Response.StatusCode >= 500)” count=“2” interval=“5” max-interval=“10” delta=“1” first-fast-retry=“true”>
<forward-request buffer-request-body=“true” timeout=“15” />
</retry>
</backend>
Find info on the retry policy settings here: Azure API Management policy reference – retry | Microsoft Learn
The Logic App implementation is straight forward. Read from Service Bus using Peek-Lock-Complete, get the BLOB based on the Blob URL in the message and you are of to processing the message however you see fit!
What we also like to do is split the Logic App into a Service Bus receive Logic App and a processing Logic App. This makes resubmitting a lot easier.
Here is the Read from Blob part in the “processing” Logic App where we split the BlobUrl on ‘/’ to get to the container and filename.
If this approach sparks any questions or thoughts, we’d love to hear from you. Please, let us, or our colleagues from Ntegration.team know your thoughts!
– By Nathan Moerman
