Python Flex Consumption: memory growth, worker replacements, and missing timer completion records at 512 MB

Question

Python Flex Consumption: memory growth, worker replacements, and missing timer completion records at 512 MB

Veeral 0

I run a Python 3.11 Function App on Linux Flex Consumption at 512 MB. A timer refreshes dashboard data every 5 minutes, with additional price chart processing occurring approximately hourly.

Memory grows across successful executions, for example, Python process RSS increased from approximately 270 MB to 410 MB over five refreshes. Workers are repeatedly replaced. Some earlier executions have start/progress logs but no completion checkpoint, request result, or exception.

Production testing of PYTHONMALLOC=malloc for over 12 hours did not stop the growth. We have also performed local profiling and allocator investigations without identifying a confirmed fix.

My questions are:

How can I determine why a Flex Consumption worker was replaced? Which logs distinguish memory-related termination from normal recycling, maintenance, scaling, or runtime faults?

How can I distinguish interrupted executions from missing telemetry? Application Insights sampling excludes Request, yet some invocation results are missing.

What should I investigate for NoScriptHost and “Exception encountered while listening to EventStream. The HTTP/2 connection faulted”? How can I associate these errors with a particular timer instance?

Where can I obtain total instance memory and termination evidence? My checkpoints measure Python RSS, but cannot retrieve cgroup memory usage.

What profiling approach would identify the retained allocations in the actual production workload? My goal is to fix the cause while preserving reliable five-minute refreshes and hourly processing at 512 MB.

The app uses Recreate deployments, which explains replacements during updates. Other observed replacements do not coincide with the listed deployment/configuration updates. I have not confirmed an OOM kill or established that memory growth causes the interruptions.

0 comments

2 answers

Your answer

Answer 1

The current evidence does not establish an OOM kill; enable scale-controller logs, correlate each timer InvocationId with instance-level memory and lifecycle events, and use Azure support when no customer-visible signal supplies a termination reason.

1. Capture worker lifecycle evidence

Enable verbose scale-controller logging temporarily:

az functionapp config appsettings set \
  --name <FUNCTION_APP_NAME> \
  --resource-group <RESOURCE_GROUP_NAME> \
  --settings SCALE_CONTROLLER_LOGGING_ENABLED=AppInsights:Verbose

Verbose logs record reasons for scale-controller worker-count changes and are written under ScaleControllerLogs; this feature is preview and increases telemetry cost, so disable it after capturing the problem ^1^.

Query count changes:

traces
| extend d = todynamic(tostring(customDimensions))
| where d.Category == "ScaleControllerLogs"
| where message == "Instance count changed"
| project timestamp,
          Reason=tostring(d.Reason),
          Previous=tostring(d.PreviousInstanceCount),
          Current=tostring(d.CurrentInstanceCount)
| order by timestamp asc

This identifies scaling decisions, but it is not documented as a complete termination ledger for OOM, host faults, maintenance, or infrastructure replacement.

For each unexplained replacement:

Function App → Diagnose and solve problems.
Run Function App Down or Reporting Errors (Preview) for the affected interval. It checks startup/offline history, deployments, scaling, high memory/CPU, host health, and stuck executions ^2^.
Query AzureActivity for start, stop, write/configuration, and trigger-sync operations. These are control-plane records, not evidence of every platform recycle ^3^.
If neither diagnostics nor telemetry supplies a cause, open an Azure support case with UTC timestamps, region, app name, cloud_RoleInstance, deployment IDs, and invocation IDs. A definitive host/container termination reason may require platform-side evidence unavailable to the app.

2. Distinguish interruption from missing telemetry

Excluding Request from sampling is necessary but not sufficient. Host.Results produces invocation records in requests, and it must remain at Information; setting it to Error records only failed invocations. Function start/completion traces also require the Function category at Information ^4^.

Use:

{
  "logging": {
    "logLevel": {
      "default": "Warning",
      "Host.Results": "Information",
      "Function": "Information",
      "Worker": "Information"
    },
    "applicationInsights": {
      "samplingSettings": {
        "isEnabled": true,
        "excludedTypes": "Request;Exception"
      }
    }
  }
}

Verify actual sampling rather than assuming all trace/checkpoint records survived:

union requests, dependencies, exceptions, traces
| where timestamp between (datetime(<START_UTC>) .. datetime(<END_UTC>))
| summarize RetainedPercentage=100/avg(itemCount)
            by itemType, bin(timestamp, 5m)

A value below 100 means that telemetry type was sampled ^5^.

Correlate one timer execution by InvocationId:

let invocation = "<INVOCATION_ID>";
union traces, requests, exceptions, dependencies
| where tostring(customDimensions["InvocationId"]) == invocation
| project timestamp, itemType, operation_Id, cloud_RoleInstance,
          message, name, success, resultCode, customDimensions
| order by timestamp asc

Every invocation has an InvocationId; operation_Id correlates the wider logical operation ^6^.

Interpretation:

Start and checkpoint records, followed by a normal requests result: completed invocation.
Start/checkpoints but no completion or Host.Results, coincident with instance disappearance or host shutdown: consistent with interruption, but not proof of OOM.
Missing traces while the request result exists and trace retention is below 100%: telemetry loss is more likely.
No records anywhere: cannot distinguish “never triggered” from loss without an external durable audit record.

3. Interpret `NoScriptHost` and HTTP/2 faults

NoScriptHost means the health check found no available script host; it does not identify why. Investigate adjacent startup, storage, deployment, and worker logs.

The EventStream HTTP/2 fault can occur during normal shutdown when the language worker exits before its gRPC channel closes gracefully; maintainers describe this as noisy shutdown telemetry rather than proof of an application networking failure ^7^. Associate it with a timer only when the same timestamp, cloud_RoleInstance, operation_Id, or InvocationId overlaps that execution. Otherwise treat it as host/worker lifecycle evidence, not the timer’s exception.

4. Measure total instance memory

Use Function App → Monitoring → Metrics:

Metric: Memory working set and Average memory working set
Split/filter by: Instance
Add Automatic Scaling Instance Count
Use the smallest available time grain around each replacement

These Flex Consumption metrics represent app-level memory and can be filtered per instance ^3^. Python RSS omits the Functions host and other processes. A 512-MB instance also has a separate 272-MB platform buffer, while function code and the host share instance resources ^8^.

5. Profile without risking the five-minute production timer

Do not deploy memory-profiler to the production app; Microsoft explicitly limits it to development environments ^9^.

Instead:

Clone the app into an isolated staging/canary Function App.
Replay production-shaped dashboard and hourly chart inputs repeatedly.
Decorate internal synchronous processing functions—not the trigger entry point—with @memory_profiler.profile.
Compare line-level increments across successive invocations.
Separately capture Python tracemalloc snapshot deltas around logical phases to identify allocation tracebacks.
Test suspected fixes under the same sequence and concurrency.

For immediate reliability while investigating, make each refresh idempotent and write start/phase/commit records to external storage. Timer retries only act on uncaught exceptions, and a timer retry does not resume on a new instance after instance failure ^10^. If 512 MB remains too tight, temporarily test 2,048 MB—Microsoft recommends that size for most Flex workloads—without treating the improvement as proof of an OOM root cause ^8^.

References

Answer 2

Rukshan edirisinghe 2,445

Hi @Veeral

Your numbers already point to a likely cause. At 512 MB instance memory, your function code and the Functions host share that 512 MB (Microsoft adds about 272 MB extra only for system processes). A Python process growing from about 270 MB to 410 MB leaves very little room for the host, so a worker being replaced partway through a run is consistent with hitting the memory limit.

Here's how I'd confirm it and stabilise things:

Raise instance memory first. Change it to 2,048 MB, which Microsoft recommends as the default for most Flex Consumption apps. You can change it at any time. If replacements stop, memory was the cause.
Stop sampling the records you need. In host.json, under logging.applicationInsights.samplingSettings, set "excludedTypes": "Request;Exception". Then every invocation result and exception is kept, and you can tell a real interruption from missing telemetry.
Track replacements by instance. In Application Insights, group your start and completion log lines by operation_Id and cloud_RoleInstance. An invocation that has a start line, no completion line, and a new instance name right after it was cut off by a replacement, not lost telemetry.
Check the portal detectors. Open Diagnose and solve problems on the app and search for Flex Consumption and availability detectors around the times workers were replaced.
Look at the hourly chart job. Memory that grows across runs in Python is usually module-level state that survives between invocations. If you use matplotlib, call plt.close(fig) (or plt.close('all')) after saving each chart, and avoid caching DataFrames in globals.

Do the replacements line up with the hourly chart processing runs?

If the larger instance size stops the replacements, please consider accepting it as the answer. If they continue at 2,048 MB, let me know and I'll be happy to keep helping.

Reference:

Veeral 0 Reputation points

2026-11-01T06:59:19.9966667+00:00
AV_INSTANCE_MEMORY_COMMUNITY_20261030.txtThanks for reviewing this. My goal is to identify and fix the cause while retaining 512 MB.

I collected additional per-instance MemoryWorkingSet metrics and matched them to the exact timer instances:

Two executions missing completion records had captured readings up to 688.3 MiB and 659.9 MiB.

Another instance completed its refresh successfully, reported 709.0 MiB, and later stopped its timer listener orderly.

There is an unusual pattern: that orderly instance’s final readings jump from approximately 647 MiB to 1,300 MiB, then become zero. Five HTTP instances with successful requests also show a near-doubling in their final positive sample. These are one-minute Average values converted from bytes.

Also, our hourly “price chart” job retrieves and exports AudienceView pricing data; it does not use matplotlib. Memory growth occurs across ordinary refreshes too.

Request records were already excluded from sampling in the captured configuration. We have correlated invocation/instance IDs and reviewed the available Flex diagnostics, but have no confirmed OOM event or termination reason.

Do you know:

Whether this final-sample doubling is expected, and exactly what MemoryWorkingSet includes on Flex?

Which customer-accessible signal can distinguish memory-related termination from normal instance replacement?

I’m happy to share the relevant metric samples and timelines.Thanks for reviewing this. My goal is to identify and fix the cause while retaining 512 MB.

I collected additional per-instance MemoryWorkingSet metrics and matched them to the exact timer instances:

Two executions missing completion records had captured readings up to 688.3 MiB and 659.9 MiB.

Another instance completed its refresh successfully, reported 709.0 MiB, and later stopped its timer listener orderly.

There is an unusual pattern: that orderly instance’s final readings jump from approximately 647 MiB to 1,300 MiB, then become zero. Five HTTP instances with successful requests also show a near-doubling in their final positive sample. These are one-minute Average values converted from bytes.

Also, our hourly “price chart” job retrieves and exports AudienceView pricing data; it does not use matplotlib. Memory growth occurs across ordinary refreshes too.

Request records were already excluded from sampling in the captured configuration. We have correlated invocation/instance IDs and reviewed the available Flex diagnostics, but have no confirmed OOM event or termination reason.

Do you know:

Whether this final-sample doubling is expected, and exactly what MemoryWorkingSet includes on Flex?

Which customer-accessible signal can distinguish memory-related termination from normal instance replacement?

I’m happy to share the relevant metric samples and timelines.

Python Flex Consumption: memory growth, worker replacements, and missing timer completion records at 512 MB

2 answers

1. Capture worker lifecycle evidence

2. Distinguish interruption from missing telemetry

3. Interpret NoScriptHost and HTTP/2 faults

4. Measure total instance memory

5. Profile without risking the five-minute production timer

Your answer

3. Interpret `NoScriptHost` and HTTP/2 faults