Azure Function App processing multiple queue messages at a time despite batchSize set to 1

Sadeep Mihiranga 0 Reputation points
2024-08-01T09:26:00.26+00:00

I have an Azure Function written in Python that's meant to process messages from an Azure Storage Queue. In my host.json file, I have set "batchSize": 1 and "newBatchThreshold": 0 under the queues settings. This should mean that the function dequeues and processes one message at a time. The function execution time will take minimum 1 minuets as per the business logic. This arrangement works just fine locally.

The function code

import azure.functions as func
import logging
import json
import os
import time

app = func.FunctionApp()

@app.queue_trigger(arg_name="msg", queue_name=os.getenv("REQUEST_QUEUE"), connection="AzureWebJobsStorage")
def ustorescraper(msg: func.QueueMessage):
    
    received_message = msg.get_body().decode('utf-8')
    json_data = json.loads(received_message)
    request_type_str = json_data["requestType"]

    logging.info('Process started : %s', request_type_str)
    time.sleep(65)
    logging.info('Processing end : %s', request_type_str)

host.json

{
  "version": "2.0",
  "logging": {
    "applicationInsights": {
      "samplingSettings": {
        "isEnabled": true,
        "excludedTypes": "Request"
      }
    },
    "logLevel": {
      "default": "Information"
    }
  },
  "extensionBundle": {
    "id": "Microsoft.Azure.Functions.ExtensionBundle",
    "version": "[4.*, 5.0.0)"
  },
  "extensions": {
    "queues": {
      "batchSize": 1,
      "newBatchThreshold": 0,
      "maxPollingInterval": "00:05:00",
      "visibilityTimeout": "00:00:30",
      "maxDequeueCount": 3,
      "messageEncoding": "base64"
    }
  }
}

However, when I deploy to Azure, the function seems to be processing multiple messages at a time, which is not what I want.

In an attempt to override the settings on Azure, I added "AzureFunctionsJobHost__extensions__queues__batchSize": "1" and "AzureFunctionsJobHost__extensions__queues__newBatchThreshold": "1" as environment variables to the Function App settings in the Azure portal, as per Microsoft's documentation.

Some of Environment variables opened as Advanced Edit

  {
    "name": "AzureFunctionsJobHost__extensions__queues__batchSize",
    "value": "1",
    "slotSetting": true
  },
  {
    "name": "AzureFunctionsJobHost__extensions__queues__newBatchThreshold",
    "value": "0",
    "slotSetting": true
  },
  {
    "name": "REQUEST_QUEUE",
    "value": "request-queue",
    "slotSetting": false
  },
  {
    "name": "AzureWebJobsStorage",
    "value": "xxx",
    "slotSetting": false
  }

Additionally, I've also disabled the scale count of my function app, to prevent Azure from scaling out instances.

Despite all these configurations, my function app continues to dequeue and process multiple messages concurrently.

Looks like the application not reading from the host.json file on azure environment.

Has anyone encountered this ?

Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
5,132 questions
Azure Queue Storage
Azure Queue Storage
An Azure service that provides messaging queues in the cloud.
109 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. Sadeep Mihiranga 0 Reputation points
    2024-08-04T06:08:29.2533333+00:00

    The host.json file is located under the "App Files" section of the Azure Function App in the Azure portal.

    Despite having the correct queue configurations, the application initially fails to dequeue messages sequentially. However, if I place the same set of messages in the queue over 4 or 5 times, the queue trigger works correctly. In this case, the function processes one message at a time, only dequeuing the next message after completing the current one. This behavior is consistent each time.

    0 comments No comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.