Azure Compute: Architecture recommendation for hosting a domain-wide keyword web scraping script

Welcome to Microsoft Q&A,

Hello @Faheem Haider I hope you are doing well,

For this workload, I would lean toward Azure Container Apps rather than Azure Functions or a dedicated VM.

Because the crawler can run for an extended period and the workload varies depending on the target domain, Container Apps gives you a good balance between scalability and operational overhead. You can package the existing scraper into a container and run it as a background workload without having to manage an operating system.

A possible architecture would be:

Container Apps → scraper container → queue/storage → results

The crawler can process a domain in smaller jobs, allowing large sites to be divided into multiple tasks instead of keeping one very large execution running. Azure Storage or a queue can be used to persist URLs and results.

I would consider:

Azure Functions if the scraping jobs can be broken into short, independent executions.

Container Apps if the scraper is long-running and you want managed container hosting with the ability to scale.

• Azure VM if the application requires full OS-level control or has dependencies that are difficult to containerize.

I would avoid designing the solution as one extremely long-running process. For large domains, checkpointing the crawl state and processing URLs in batches will make the solution more resilient and easier to scale.

Also ensure the crawler respects the target site's robots.txt, terms of service, rate limits, and any applicable legal requirements.

If cost is the primary concern, start with a single Container Apps workload and scale only when the queue of domains/URLs increases. This avoids paying for a dedicated VM when the scraper is idle.

If my answer helped you, please consider marking it as accepted. This helps others in the community find similar solutions.

Hey @Faheem Haider — sounds like you’re looking to migrate a long-running domain crawl/scraping job to Azure, where the runtime can vary a lot based on the target site size/load, and you’re specifically worried about serverless execution/timeouts.

Based on the provided docs, here’s the most practical guidance we can give for architecture direction and “how to avoid pain” (even though the docs you provided don’t spell out exact timeout values).

Architecture recommendation (high level)

1) Prefer breaking the crawl into decoupled “work items”

The provided Azure hosting guidance for web apps notes that individual logical processes that can be decoupled from the rest of the application may be deployed independently to Azure Functions. In your case, that typically means splitting “crawl the whole domain” into smaller units (for example: discover links, then process pages in batches).

Why this helps: it keeps each function invocation more bounded and lets the platform scale out based on the queue of work rather than keeping one single long-running execution alive.

2) Use a job/work queue pattern to orchestrate crawling

From the provided Azure Functions documentation guidance on common scenarios and bindings, a typical pattern for long-running background work is:

• trigger work on a schedule or from an event
• enqueue units of work
• have Functions process those units independently

The Functions doc explicitly calls out run scheduled tasks via a TimerTrigger as a common scenario, and the hosting guidance mentions Azure offers queue/messaging options for transient orchestration/event-based data (it references Service Bus or Storage Queues, with Service Bus recommended for non-trivial messaging).

So, a cost- and scaling-friendly direction is:

• TimerTrigger (or some starter trigger) to kick off a “crawl run”
• Queue/Service Bus to store and distribute page-processing tasks
• Azure Functions workers to process tasks in parallel

3) Choose a Functions hosting plan based on “you need to manage execution behavior”

The Functions architecture best practices doc clarifies that Functions are serverless and run on App Service plans, with multiple hosting plans:

• Consumption plan: fully managed; doesn’t let you access the underlying App Service instance
• Dedicated / Premium plans: provide ability to configure/manage the App Service instance

If your concern is that a single attempt could run “too long” (depending on your target domain size), then the key is still to partition the work (step 1) so each execution stays within reasonable bounds. If, however, you truly need longer runtime per worker, then the doc suggests moving from Consumption to Dedicated/Premium gives more operational control.

4) Data/storage: store results efficiently for queryability and scale

Your scraper will likely generate lots of structured-ish results (e.g., URL + keyword matches + metadata). The provided documentation suggests:

• Transactional relational data → Azure SQL Database
• High-performance read-mostly → Redis backed by Azure SQL
• Unstructured JSON with strong querying at scale → Azure Cosmos DB (recommended for large numbers of JSON documents that must be queried)

A common approach is:

• write crawl findings to Cosmos DB (if you need flexible querying across JSON-y documents)
• or Azure SQL if your schema is relational and stable

How this addresses your timeout concern (practically)

Even though the docs you provided don’t give explicit “timeout limits,” the general mitigation supported by the docs is: don’t rely on one continuous long execution. Instead:

• use queue-based decomposition (Functions for decoupled logical processes)
• use scheduled/event-driven triggers (Functions scenarios)
• scale by processing multiple smaller tasks rather than one huge crawl

This keeps the system resilient under varying domain sizes and load, and it aligns with how Functions are designed to scale “based on demand” while charging for consumed operation time.

Hello @Faheem Haider If the resolution was helpful, kindly take a moment to click on and click on Yes for was this answer helpful. And, if you have any further query do let us know.