Share via

Scaling on Azure AI Search - indexes content

Tomás Novo 0 Reputation points
2024-11-19T14:13:52.89+00:00

Hello,

I've been using RAG chat app with Azure OpenAI and Azure AI Search for a year now. I've ingested many documents, which resulted in some necessary costs in order to have better results with the help of Azure Document Intelligence.

The next step to improve my Azure Web App, would be to also ingest metadata about the documents. In the middle of the metadata ingestion process, I got the message:

"Storage quota has been exceeded for this service. You must either delete documents first, or use a higher SKU for additional quota. Please note that vector index size quota is not a separate allowance and consumes from the total storage quota for the service."

Thinking about the future, it makes sense to upgrade my Azure AI Search service, which currently has the Basic tier. However, it's not possible to directly scale the tier on this service. As far as I understood, I need to create a new Azure AI Search service with the desired new tier (Standard) and somehow migrate the content.

My question is: is there a simple way to export the content from the Basic service and import it on the Standard service? I would like to avoid more costs for re-ingesting documents I've already processed.

Thanks in advance and best regards.

Azure AI Search
Azure AI Search
An Azure search service with built-in artificial intelligence capabilities that enrich information to help identify and explore relevant content at scale.
1,083 questions
Azure AI services
Azure AI services
A group of Azure services, SDKs, and APIs designed to make apps more intelligent, engaging, and discoverable.
2,931 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Sina Salam 12,816 Reputation points
    2024-11-19T20:57:01.88+00:00

    Hello Tomás Novo,

    Welcome to the Microsoft Q&A and thank you for posting your questions here.

    I understand that you would like to know if there is a simple way to export the content from the Basic service and import it on the Standard service.

    Yes, there is a way to export your content from the Basic Azure AI Search service and import it into a new Standard service without re-ingesting all your documents. The below method will ensure a smooth transition to a higher tier service while maintaining your existing data integrity, this is a step-by-step guide to help you through the process:

    1. You can use the Azure AI Search .NET utilities to export data from your current index. This involves creating JSON files that contain your index schema and documents. The export-data command in the utility can help you partition and export your data efficiently. - https://learn.microsoft.com/en-us/samples/azure-samples/azure-search-dotnet-utilities/export-data
    2. In the Azure portal, create a new Azure AI Search service with the Standard tier. Make sure to configure it according to your needs.
    3. Use the import-data command or the Azure portal’s import wizard to load the exported JSON files into your new Standard service. This can be done using the REST API or Azure SDKs for various programming languages. - https://learn.microsoft.com/en-us/azure/search/search-what-is-data-import and https://learn.microsoft.com/en-us/azure/search/search-import-data-portal
    4. After importing, use the Search explorer in the Azure portal to verify that your data has been correctly imported and is queryable, - https://learn.microsoft.com/en-us/azure/search/search-what-is-data-import

    If you can try the above - you can migrate your data without incurring additional costs for re-ingesting documents.

    I hope this is helpful! Do not hesitate to let me know if you have any other questions.

    Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful.

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.