1

I have an Azure function (based on the new C# functions instead of the old .csx functions) that is triggered whenever a message comes into an Azure Service Bus Queue. Once the function is triggered, it starts processing the service bus message. It decodes the message, reads a bunch of databases, updates a bunch of others, etc... This can take upwards of 30 minutes at times.

Since, this is not a time sensitive process, 30 minutes or even 60 minutes is not an issue. The problem is that in the meanwhile, Azure Function seems to kick in again and picks up the same message again and again and reprocesses it. This is an issue and cause problems in our business logic.

So, the question is, can we force the Azure function to run in a singleton mode? Or if that's not possible, how do we change the polling interval?

3
  • Was your Function App created under the App Service Plan or Consumption Plan? Under App Service Plan, make sure that AlwaysOn is turned on. Under the Consumption Plan, the Function App is terminated after 5 minutes (configurable to extend to 10 minutes, see "functionTimeout" value at github.com/Azure/azure-webjobs-sdk-script/wiki/host.json), which may explain the behavior you are seeing.
    – Ling Toh
    Jul 11, 2017 at 23:22
  • I have the App Service Plan. My function is not terminating. In fact, the opposite is happening. Multiple functions are processing the message.
    – Yasir
    Jul 14, 2017 at 18:15
  • @Yasir Even I faced the same issue, as it was processing multiple message in parallel it was creating duplicate records in upsert. I managed to handle it by making the maxConcurrentCalls to 1. Refer stackoverflow.com/a/75898239/3057246 for more details. Mar 31 at 11:57

5 Answers 5

6

AutoRenewTimeout is not as great as suggested. It has a downside that you need to be aware of. It's not a guaranteed operation. Being a client side initiated operation, it can and sometimes will fail, leaving you in the same state as you are today.

What you could do to address it is to review your design. If you have a long running process, then you process the message, and hand off processing to something that can run longer than MaxLockDuration. The fact that your function is taking so long, indicates you have a long running process. Messaging is not designed for that.

One of the potential solutions would be to get a message, register processing intent in a storage table. Have another storage table triggered function to kick-off processing that could take X minutes. Mark it as a singleton. By doing so, you'll be processing your messages in parallel, writing "request for long running processing" into storage table, completing Service Bus messaging and therefore not triggering their re-processing. With the long running processing you can decide how to handle failure cases.

Hope that helps.

4
  • When you say "Mark it as a singleton", what is the it you are referring to? And how do I mark it as a singleton?
    – Yasir
    Jul 12, 2017 at 17:37
  • The other storage table triggered function. I've implemented somewhat similar process with NServiceBus and Service Bus. You'll find the code and the diagram here: docs.particular.net/samples/azure/… With functions approach, your long running Function (Processor) doesn't have to poll, it would be triggered. Jul 12, 2017 at 17:39
  • 1
    I'd suggest using the github.com/Azure/azure-functions-durable-extension, which provides the long-running / orchestration capabilities you need
    – Matt Mason
    Jul 14, 2017 at 20:32
  • 1
    Matt, durable Functions are not there yet in terms of maturity. Jul 14, 2017 at 21:15
5

The issue is related to Service Bus setting...

What is happening is that the message is added to the queue, the message is then given to the function and a lock is placed on that message so that no other consumer can see/process that message while you have a lock on it.

If within that lock period you do not tell service bus that you've processed the file, or to extend the lock, the lock is removed from the message and it will become visible to other services that will then process that message, which is what you are seeing.

Fortunately, Azure Functions can automatically renew the lock for you. In the host.json file there is an autoRenewTimeout setting that specifies for how long you want Azure Functions to keep on renewing the lock for.

https://github.com/Azure/azure-webjobs-sdk-script/wiki/host.json

"serviceBus": {
  // the maximum duration within which the message lock will be renewed automatically.
  "autoRenewTimeout": "00:05:00"
},
2

So your question is on how to make the Azure Function Message trigger to process one message at a time, and a message has to be processed once only.

I have been able to achieve the same functionality using the following host.json configuration.

{
  "serviceBus": {
    "maxConcurrentCalls": 1,
    "autoRenewTimeout": "23:59:59",
    "autoComplete": true
  }
}

Note that I set the autoRenewTimeout to 24 hour, which is long enough for a really long running process. Otherwise you can change it to a duration that fits your need.

Many will argue the suitability of using Azure Function for a long running operation. But that is not the question that needs an answer here.

0

I also experience the same issue, what I did is remove the default rule and add a custom rule on subscriber.

(OriginalTopic='name-of-topic' AND OriginalSubscription='name-of-subcription-sub') OR (NOT Exists([OriginalTopic]))
0

Message Lock Duration and Max Delivery Count

There is something called as Message Lock Duration and Max Delivery Count, so as per the config below it will be locked by the client for 5 minutes to be processed once the timeout happens it will cause an inner exception and will try to redeliver it incrementing the delivery count and up to the Max Delivery Count as below 10.

enter image description here

Scan the log and If this is happening due to the Lock Duration Timeout you can either try to manually renew it using BrokeredMessage.RenewLock Method in your code or automatically with OnMessageOptions.AutoRenewTimeout Property


Function Auto Scaling

You can also disable Auto Scaling by making it to 1 which is by default 200, but this will have no/little effect to ServiceBus Triggerd Function since even with a single instance it will make concurrent calls to service bus to read message.

enter image description here


Singleton

Now for your answer you can try Singalton but I think even with singleton if the function scales out it will be multiple singleton as the function is not in a single box with a single instance. It might work in other triggers but not with service bus trigger.


maxConcurrentCalls in host.json

If your issue is not because of Message Lock Duration it might be because of concurrent calls to service bus which is by default 32. So you can add the following to your host.json and minimize the values to 1 to test with outcome.

{
    "version": "2.0",
    "extensions": {
        "serviceBus": {
            "prefetchCount": 100,
            "messageHandlerOptions": {
                "autoComplete": true,
                "maxConcurrentCalls": 32,
                "maxAutoRenewDuration": "00:05:00"
            },
            "sessionHandlerOptions": {
                "autoComplete": false,
                "messageWaitTimeout": "00:00:30",
                "maxAutoRenewDuration": "00:55:00",
                "maxConcurrentSessions": 16
            },
            "batchOptions": {
                "maxMessageCount": 1000,
                "operationTimeout": "00:01:00",
                "autoComplete": true
            }
        }
    }
}

But if you just what to read the message is sequence from a ServiceBus Trigger Function you just need to add the following in the host.json to tell him stop making concurrent calls. And if you log the Sequence Number of the message in the console you can verify it as it will now follow a sequence.

  "extensions": {
    "serviceBus": {
      "messageHandlerOptions": {
        "maxConcurrentCalls": 1
      }
    }
  }

This will even avoid duplicates if you are doing upsert as the records are created one by one in sequence but the downside it will be slow and compromise the performance.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.