Share via

Question on fairly complicated rag code base from Azure-Samples that doesnt't use Azure document skillset api

Tung Nguyen Xuan 40 Reputation points
2024-11-06T04:38:45.55+00:00

I'm interested in creating a pdf document ingestion pipeline. I followed the tutorial here

https://github.com/Azure/azure-search-vector-samples/blob/main/demo-python/code/integrated-vectorization/azure-search-integrated-vectorization-sample.ipynb

and managed to create a simple splitting --> embedding skillset, and used it to create an index.

Now I want to be able to make the pipeline more adaptive, since some pdf doesn't contain any text (probably scanned), and must be processed with vision techniques (gpt4v). Otherwise, the pdf is assumed to contain text, then it should be chunked after image description is generated (GPT) and stored away for retrieval later (in another image index, for instance). This gets complicated quickly, and I can't find an example of implementing these ideas using Azure doc skillset. Some Azure-Samples repo do implement these ideas, but not using the skillset api at all. Instead they create their own sophisticated and dedicated application.

https://github.com/Azure-Samples/rag-as-a-service-with-vision/tree/main

https://github.com/Azure-Samples/multimodal-rag-code-execution

My question is: is it realistic to expect to implement these processing ideas using just the skillset? Or is skillset just for creating quick demos with happy case scenario? How flexible can it be to accommodate for complexities in business requirements?

Thanks

Azure AI Search
Azure AI Search
An Azure search service with built-in artificial intelligence capabilities that enrich information to help identify and explore relevant content at scale.
1,083 questions
Accepted answer
  1. ajkuma 27,101 Reputation points Microsoft Employee
    2024-11-06T19:07:03.32+00:00

    Tung Nguyen Xuan. Following-up on this,

    1.We already have a new sample of how to call GPT-4o mini (and can be used with any LLM) from a custom skill, so you may do your own queries within a skillset calling your own custom skill.

    AzureOpenAICustomInferenceSkill

    Based on your requirement/ask- Just to highlight, the skillset functionality has been available for several years and is currently utilized by thousands of users in production environments.
    You may perform stress tests to determine the capabilities you require. The built-in indexers are designed to easily transfer data to an index. When scheduled, they run incrementally and continue processing items until completion.

    It is recommended to test with your own data to determine your specific performance requirements for indexing and to assess whether the built-in indexers are suitable for your scenario.

    2.AI Search has built-in skills already, you may check the ones that are available:

    Skills for extra processing during indexing (Azure AI Search) - however, as shown in #1, with a custom skill (Custom Web API skill in an Azure AI Search enrichment pipeline), typically you can call any Azure service you need that is not directly/natively supported (and perform any custom transformation from it with custom code).

    (I'll convert my comment to answer -If my answer helped (pointed, you in the right direction) > please click Accept Answer - it will benefit the community/users to find the answer quickly.

    1 person found this answer helpful.

0 additional answers

Sort by: Most helpful
Most helpful Newest Oldest

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.