Share via

Implement learning to an AI Model

Nidhi Priya 571 Reputation points
2025-12-11T12:35:43.4233333+00:00

Hello experts!

We’ve built an AI-powered time tracking application that automatically logs time intervals for Teams meetings. Our app uses the Microsoft Graph API to fetch meetings from the Teams calendar, and we pull task data from Intervals.

We’ve deployed GPT‑5 in Azure, which receives the raw meeting data and corresponding task lists. The LLM performs semantic and contextual matching to align meeting names with task names. The whole process is fully automated.

However, sometimes the LLM fails to find a confident or accurate match — in such cases, the meeting is sent into a review stage, where the user can manually select the correct task. In other scenarios, it might map a meeting to the wrong task altogether, causing time entries to post incorrectly.

We want to enhance this system by adding a memory layer — essentially, enabling learning from user feedback. When a meeting goes into review and the user manually corrects the task mapping, the LLM should “remember” that correction. The next time a similar meeting appears, the model should leverage that past feedback to produce a better mapping automatically.

Has anyone implemented a similar memory or adaptive learning mechanism for LLM-based apps on Azure or with OpenAI APIs? Would it make more sense to use a vector database, fine-tuning, or some form of dynamic retrieval memory?

Any insights or examples would be greatly appreciated!

Azure Language in Foundry Tools
Azure Language in Foundry Tools
An Azure service that provides natural language capabilities including sentiment analysis, entity extraction, and automated question answering.

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. SRILAKSHMI C 14,655 Reputation points Microsoft External Staff Moderator
    2025-12-18T17:09:10.34+00:00

    Hello Nidhi Priya,

    Welcome to Microsoft Q&A and thank you for the more information.

    What you are trying to achieve is learning from user feedback at runtime, not traditional model training. This is a common pattern in LLM-based automation systems, and it’s best handled outside the model itself.

    • GPT-5 models do not currently support fine-tuning in Azure OpenAI / AI Foundry.
    • Even when fine-tuning is available (for older models), it is not suitable for per-user or per-interaction learning.
    • LLMs are stateless any “memory” must be implemented externally.

    Recommended approach (production-proven pattern)

    The correct solution is a retrieval-augmented memory layer, not fine-tuning.

    1. Capture user corrections explicitly

    Whenever a meeting goes to review and the user selects the correct task:

    Store meeting metadata (title, participants, keywords, context)

    Store the correct task ID / name

    Scope this data per tenant or per user

    This becomes trusted ground-truth feedback.

    1. Use embeddings + vector storage

    Generate embeddings for:

    Meeting descriptions

    Task names

    Confirmed meeting → task mappings

    Store these in a vector-capable store such as:

    Azure AI Search (vector search)

    Cosmos DB with vector indexing

    Any managed vector database

    This enables semantic similarity matching.

    1. Dynamic retrieval at inference time

    For each new meeting:

    Retrieve similar past meetings and their corrected mappings

    Pass the retrieved examples to the LLM as context

    Ask the model to choose the best task using both current input and historical feedback

    This gives you adaptive behavior without retraining.

    1. Confidence-based automation

    High confidence → auto-map

    Low confidence → send to review

    Each review automatically improves future results because it feeds the memory layer.

    Why fine-tuning is not recommended here

    Not supported for GPT-5

    Requires offline datasets and retraining cycles

    Cannot learn immediately from individual corrections

    Difficult to scope per user or tenant

    Slower iteration compared to retrieval-based memory

    Fine-tuning works best for static domain behavior, not dynamic, user-driven feedback loops.

    This is a memory and retrieval problem, not a model training problem

    Use vector search + stored user feedback

    Keep the model stateless

    Let learning happen through retrieval, not weight updates

    This architecture aligns with how copilots and adaptive AI systems are implemented on Azure today.

    Please refer this

    I Hope this helps. Do let me know if you have any further queries.

    Thank you!

    1 person found this answer helpful.
    0 comments

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.