Any Azure AI service can help find possible duplicated customer data?

Hi @Henry Zhang yes, this is possible with Azure AI Search

Create an Azure AI Search Index:

• Set up a new Azure AI Search service within your Azure subscription.
  • Create a new index and define the fields that will store your customer data (e.g., firstName, lastName, phoneNumber, address, dateOfBirth).
    • Ensure that the fields are indexed appropriately for efficient search.
    Index Your Customer Data:
    • Use the Azure AI Search SDK or REST API to index your customer data into the newly created index. Each customer record should be represented as a document with the corresponding field values.

Create a Similarity Search Query:

• Construct a similarity search query using the Azure AI Search query language. This query will specify the fields to compare and the similarity metric to use. For example, you could use the cosine similarity metric to compare text fields like names and addresses.

{
    "search": {
        "query": "Henry Zhang",
        "searchType": "similarity",
        "similarityFields": [
            "firstName",
            "lastName",
            "phoneNumber",
            "address"
        ],
        "similarityMetrics": [
            {
                "fields": ["firstName", "lastName"],
                "metric": "cosine"
            },
            {
                "fields": ["phoneNumber"],
                "metric": "exact"
            }
        ]
    }
}

Evaluate Results:

• Execute the similarity search query and analyze the results. Azure AI Search will return documents that are similar to the query based on the specified similarity metrics.
  • You can set a threshold for similarity scores to determine which results are potential duplicates. For instance, if the cosine similarity score is above 0.8, it might indicate a high likelihood of duplication.

To get the best result make sure that your data is clean and consistent before indexing. Inconsistencies in spelling or formatting can affect similarity scores.

Best,

Grace