Quickstart: Search for multimodal content in the Azure portal

In this quickstart, you use the Import and vectorize data wizard in the Azure portal to get started with multimodal search. The wizard simplifies the process of extracting, chunking, vectorizing, and loading both text and images into a searchable index.

Unlike Quickstart: Vector search in the Azure portal, which processes simple text-containing images, this quickstart supports advanced image processing for multimodal RAG scenarios.

This quickstart uses a multimodal PDF from the azure-search-sample-data repo. However, you can use different files and still complete this quickstart.

Prerequisites

• An Azure account with an active subscription. Create an account for free.

• An Azure AI Search service. We recommend the Basic tier or higher.

• An Azure Storage account. Use Azure Blob Storage or Azure Data Lake Storage Gen2 (storage account with a hierarchical namespace) on a standard performance (general-purpose v2) account. Access tiers can be hot, cool, or cold.

• A supported extraction method.

• A supported embedding method.

• Familiarity with the wizard. See Import data wizards in the Azure portal.

Supported extraction methods

For content extraction, you can choose either default extraction via Azure AI Search or enhanced extraction via Azure AI Document Intelligence. The following table describes both extraction methods.

¹ For billing purposes, you must attach your Azure AI multi-service resource to the skillset in your Azure AI Search service. Unless you use a keyless connection to create the skillset, both resources must be in the same region.

Supported embedding methods

For content embedding, you can choose either image verbalization (followed by text vectorization) or multimodal embeddings. Deployment instructions for the models are provided in a later section. The following table describes both embedding methods.

¹ The endpoint of your Azure OpenAI resource must have a custom subdomain, such as https://my-unique-name.openai.azure.com. If you created your resource in the Azure portal, this subdomain was automatically generated during resource setup.

² Azure OpenAI resources (with access to embedding models) that were created in the Azure AI Foundry portal aren't supported. You must create an Azure OpenAI resource in the Azure portal.

³ For billing purposes, you must attach your Azure AI multi-service resource to the skillset in your Azure AI Search service. Unless you use a keyless connection (preview) to create the skillset, both resources must be in the same region.

⁴ phi-4 is only available to Azure AI Foundry projects.

Public endpoint requirements

All of the preceding resources must have public access enabled so that the Azure portal nodes can access them. Otherwise, the wizard fails. After the wizard runs, you can enable firewalls and private endpoints on the integration components for security. For more information, see Secure connections in the import wizards.

If private endpoints are already present and you can't disable them, the alternative is to run the respective end-to-end flow from a script or program on a virtual machine. The virtual machine must be on the same virtual network as the private endpoint. Here's a Python code sample for integrated vectorization. The same GitHub repo has samples in other programming languages.

Check for space

If you're starting with the free service, you're limited to three indexes, three data sources, three skillsets, and three indexers. Make sure you have room for extra items before you begin. This quickstart creates one of each object.

Configure access

Before you begin, make sure you have permissions to access content and operations. We recommend Microsoft Entra ID authentication and role-based access for authorization. You must be an Owner or User Access Administrator to assign roles. If roles aren't feasible, you can use key-based authentication instead.

Configure the required roles and conditional roles identified in this section.

Required roles

Azure AI Search and Azure Storage are required for all multimodal search scenarios.

Conditional roles

The following tabs cover all wizard-compatible resources for multimodal search. Select only the tabs that apply to your chosen extraction method and embedding method.

Prepare sample data

This quickstart uses a sample multimodal PDF, but you can also use your own files. If you're on a free search service, use fewer than 20 files to stay within the free quota for enrichment processing.

To prepare the sample data for this quickstart:

1. Sign in to the Azure portal and select your Azure Storage account.

2. From the left pane, select Data storage > Containers.

3. Create a container, and then upload the sample PDF to the container.

4. Create another container to store images extracted from the PDF.

Deploy models

The wizard offers several options for content embedding. Image verbalization requires an LLM to describe images and an embedding model to vectorize text and image content, while direct multimodal embeddings only require an embedding model. These models are available through Azure OpenAI and Azure AI Foundry.

To deploy the models for this quickstart:

1. Sign in to the Azure AI Foundry portal.

2. Select your Azure OpenAI resource or Azure AI Foundry project.

3. From the left pane, select Model catalog.

4. Deploy the models required for your chosen embedding method.

Start the wizard

To start the wizard for multimodal search:

1. Sign in to the Azure portal and select your Azure AI Search service.

2. On the Overview page, select Import and vectorize data.

   

3. Select your data source: Azure Blob Storage or Azure Data Lake Storage Gen2.

   

4. Select Multimodal RAG.

   

Connect to your data

Azure AI Search requires a connection to a data source for content ingestion and indexing. In this case, the data source is your Azure Storage account.

To connect to your data:

1. On the Connect to your data page, specify your Azure subscription.

2. Select the storage account and container to which you uploaded the sample data.

3. Select the Authenticate using managed identity checkbox. Leave the identity type as System-assigned.

   

4. Select Next.

Extract your content

Depending on your chosen extraction method, the wizard provides configuration options for document cracking and chunking.

Embed your content

During this step, the wizard uses your chosen embedding method to generate vector representations of both text and images.

Store the extracted images

The next step is to send images extracted from your documents to Azure Storage. In Azure AI Search, this secondary storage is known as a knowledge store.

To store the extracted images:

1. On the Image output page, specify your Azure subscription.

2. Select the storage account and blob container you created to store the images.

3. Select the Authenticate using managed identity checkbox. Leave the identity type as System-assigned.

   

4. Select Next.

Map new fields

On the Advanced settings page, you can optionally add fields to the index schema. By default, the wizard generates the fields described in the following table.

You can't modify the generated fields or their attributes, but you can add fields if your data source provides them. For example, Azure Blob Storage provides a collection of metadata fields.

To add fields to the index schema:

1. Under Index fields, select Preview and edit.

2. Select Add field.

3. Select a source field from the available fields, enter a field name for the index, and accept (or override) the default data type.

4. If you want to restore the schema to its original version, select Reset.

Schedule indexing

For data sources where the underlying data is volatile, you can schedule indexing to capture changes at specific intervals or specific dates and times.

To schedule indexing:

1. On the Advanced settings page, under Schedule indexing, specify a run schedule for the indexer. We recommend Once for this quickstart.

   

2. Select Next.

Finish the wizard

The final step is to review your configuration and create the necessary objects for multimodal search. If necessary, return to the previous pages in the wizard to adjust your configuration.

To finish the wizard:

1. On the Review and create page, specify a prefix for the objects the wizard will create. A common prefix helps you stay organized.

   

2. Select Create.

When the wizard completes the configuration, it creates the following objects:

• An indexer that drives the indexing pipeline.

• A data source connection to Azure Blob Storage.

• An index with text fields, vector fields, vectorizers, vector profiles, and vector algorithms. During the wizard workflow, you can't modify the default index. Indexes conform to the 2024-05-01-preview REST API so that you can use preview features.

• A skillset with the following skills:

  • The Document Extraction skill or Document Layout skill extracts text and images from source documents. The Text Split skill accompanies the Document Extraction skill for data chunking, while the Document Layout skill has built-in chunking.

  • The GenAI Prompt skill verbalizes images in natural language. If you're using direct multimodal embeddings, this skill is absent.

  • The Azure OpenAI Embedding skill, AML skill, or Azure AI Vision multimodal embeddings skill is called once for text vectorization and once for image vectorization.

  • The Shaper skill enriches the output with metadata and creates new images with contextual information.

Check results

This quickstart creates a multimodal index that supports hybrid search over both text and images. Unless you use direct multimodal embeddings, the index doesn't accept images as query inputs, which requires the AML skill or Azure AI Vision multimodal embeddings skill with an equivalent vectorizer. For more information, see Configure a vectorizer in a search index.

Hybrid search combines full-text queries and vector queries. When you issue a hybrid query, the search engine computes the semantic similarity between your query and the indexed vectors and ranks the results accordingly. For the index created in this quickstart, the results surface content from the content_text field that closely aligns with your query.

To query your multimodal index:

1. Sign in to the Azure portal and select your Azure AI Search service.

2. From the left pane, select Search management > Indexes.

3. Select your index.

4. Select Query options, and then select Hide vector values in search results. This step makes the results more readable.

   

5. Enter text for which you want to search. Our example uses energy.

6. To run the query, select Search.

   

   The JSON results should include text and image content related to energy in your index. If you enabled semantic ranker, the @search.answers array provides concise, high-confidence semantic answers to help you quickly identify relevant matches.

   "@search.answers": [
      {
         "key": "a71518188062_aHR0cHM6Ly9oYWlsZXlzdG9yYWdlLmJsb2IuY29yZS53aW5kb3dzLm5ldC9tdWx0aW1vZGFsLXNlYXJjaC9BY2NlbGVyYXRpbmctU3VzdGFpbmFiaWxpdHktd2l0aC1BSS0yMDI1LnBkZg2_normalized_images_7",
         "text": "A vertical infographic consisting of three sections describing the roles of AI in sustainability:  1. **Measure, predict, and optimize complex systems**: AI facilitates analysis, modeling, and optimization in areas like energy distribution, resource allocation, and environmental monitoring. **Accelerate the development of sustainability solution...",
         "highlights": "A vertical infographic consisting of three sections describing the roles of AI in sustainability:  1. **Measure, predict, and optimize complex systems**: AI facilitates analysis, modeling, and optimization in areas like<em> energy distribution, </em>resource<em> allocation, </em>and environmental monitoring. **Accelerate the development of sustainability solution...",
         "score": 0.9950000047683716
      },
      {
         "key": "1cb0754930b6_aHR0cHM6Ly9oYWlsZXlzdG9yYWdlLmJsb2IuY29yZS53aW5kb3dzLm5ldC9tdWx0aW1vZGFsLXNlYXJjaC9BY2NlbGVyYXRpbmctU3VzdGFpbmFiaWxpdHktd2l0aC1BSS0yMDI1LnBkZg2_text_sections_5",
         "text": "...cross-laminated timber.8 Through an agreement with Brookfield, we aim  10.5 gigawatts (GW) of renewable energy to the grid.910.5 GWof new renewable energy capacity to be developed across the United States and Europe.Play 4 Advance AI policy principles and governance for sustainabilityWe advocated for policies that accelerate grid decarbonization",
         "highlights": "...cross-laminated timber.8 Through an agreement with Brookfield, we aim <em> 10.5 gigawatts (GW) of renewable energy </em>to the<em> grid.910.5 </em>GWof new<em> renewable energy </em>capacity to be developed across the United States and Europe.Play 4 Advance AI policy principles and governance for sustainabilityWe advocated for policies that accelerate grid decarbonization",
         "score": 0.9890000224113464
      },
      {
         "key": "1cb0754930b6_aHR0cHM6Ly9oYWlsZXlzdG9yYWdlLmJsb2IuY29yZS53aW5kb3dzLm5ldC9tdWx0aW1vZGFsLXNlYXJjaC9BY2NlbGVyYXRpbmctU3VzdGFpbmFiaWxpdHktd2l0aC1BSS0yMDI1LnBkZg2_text_sections_50",
         "text": "ForewordAct... Similarly, we have restored degraded stream ecosystems near our datacenters from Racine, Wisconsin120 to Jakarta, Indonesia.117INNOVATION SPOTLIGHTAI-powered Community Solar MicrogridsDeveloping energy transition programsWe are co-innovating with communities to develop energy transition programs that align their goals with broader s.",
         "highlights": "ForewordAct... Similarly, we have restored degraded stream ecosystems near our datacenters from Racine, Wisconsin120 to Jakarta, Indonesia.117INNOVATION SPOTLIGHTAI-powered Community<em> Solar MicrogridsDeveloping energy transition programsWe </em>are co-innovating with communities to develop<em> energy transition programs </em>that align their goals with broader s.",
          "score": 0.9869999885559082
      }
   ]
   

Clean up resources

This quickstart uses billable Azure resources. If you no longer need the resources, delete them from your subscription to avoid charges.

Next steps

This quickstart introduced you to the Import and vectorize data wizard, which creates all of the necessary objects for multimodal search. To explore each step in detail, see the following tutorials:

• Tutorial: Image verbalization and Document Extraction skill
• Tutorial: Image verbalization and Document Layout skill
• Tutorial: Multimodal embeddings and Document Extraction skill
• Tutorial: Multimodal embeddings and Document Layout skill