Transparency Note for Azure OpenAI Service

As part of the fully managed Azure OpenAI Service, the GPT-3 models analyze and generate natural language, Codex models analyze and generate code and plain text code commentary, and the GPT-4 models can understand and generate natural language and code. These models use an autoregressive architecture, meaning they use data from prior observations to predict the most probable next word. This process is then repeated by appending the newly generated content to the original text to produce the complete generated response. Because the response is conditioned on the input text, these models can be applied to various tasks simply by changing the input text.

The GPT-3 series of models are pretrained on a wide body of publicly available free text data. This data is sourced from a combination of web crawling (specifically, a filtered version of Common Crawl, which includes a broad range of text from the internet and comprises 60 percent of the weighted pretraining dataset) and higher-quality datasets, including an expanded version of the WebText dataset, two internet-based books corpora and English-language Wikipedia. The GPT-4 base model was trained using publicly available data (such as internet data) and data that was licensed by OpenAI. The model was fine-tuned using reinforcement learning with human feedback (RLHF).

Learn more about the training and modeling techniques in OpenAI's GPT-3, GPT-4, and Codex research papers. The guidance below is also drawn from OpenAI's safety best practices.

Fine tuning refers to using Supervised Fine Tuning to adjust a base model’s weights to provide better responses based on a provided training set. All use cases and considerations for large language models apply to fine-tuned models, but there are additional considerations as well.

Key terms

The fully managed service provides API access to Azure OpenAI DALL·E 2, DALL·E 3, and GPT-4 Turbo with Vision APIs.

Azure OpenAI DALL·E APIs enable the generation of rich imagery from text prompts and image inputs in an application. This powerful, multimodal AI model was developed by OpenAI and can generate images that capture both the semantics and style of the text input.

You can learn more about the training and modeling techniques for DALL·E in the OpenAI DALL·E research paper and about DALL·E 3 safety from the OpenAI DALL·E 3 system card.

Azure OpenAI GPT-4 Turbo with Vision can accept multimodal (image and text) inputs and generate natural language and code responses. This API enables a richer and more comprehensive understanding of image and video inputs.

You can learn more about the training and modeling techniques for GPT-4 Turbo with Vision in the OpenAI GPT-4 research paper OpenAI GPT-4 research paper and GPT-4 Turbo with Vision safety from the system card.

Azure OpenAI GPT-4o can accept multimodal (image and text) inputs and generate natural language and code responses, similar to Azure OpenAI GPT-4 Turbo with Vision.

The following guidance is drawn from OpenAI's safety best practices and Microsoft research insights and best practices for responsible AI.

Key terms

The gpt-4o-realtime-preview model in Azure OpenAI service enables robust speech-to-speech and text-to-speech interactions. This model integrates advanced speech recognition and synthesis capabilities, allowing it to understand spoken input, generate accurate transcriptions, and produce fluent spoken output in a target language. With its ability to translate speech in real time, gpt-4o-realtime-preview facilitates dynamic conversations across languages.

The Whisper model in Azure OpenAI service enables access to a model that performs robust speech recognition and translation tasks. The Whisper model is an Automatic Speech Recognition (ASR) model that was developed by OpenAI and is capable of transcribing speech audio files into the language that was spoken as well as translated into English.

You can learn more about training and modeling techniques for the Whisper model in the OpenAI Whisper research paper. The Whisper model is also used in the Azure AI Speech service. Learn more about the Azure AI Speech integration and evaluate it for your use here.

Key terms

The o1, GPT-4, GPT-3, and Codex models, and Azure OpenAI evaluation use natural language instructions and examples in the prompt to identify the task. The model then completes the task by predicting the most probable next text. This technique is known as "in-context" learning. These models are not retrained during this step but instead give predictions based on the context you include in the prompt.

There are three main approaches for in-context learning. These approaches vary based on the amount of task-specific data that is given to the model:

Few-shot : In this case, a user includes several examples in the prompt that demonstrate the expected answer format and content. The following example shows a few-shot prompt providing multiple examples:

Convert the questions to a command: 
Q: Ask Constance if we need some bread 
A: send-msg `find constance` Do we need some bread? 
Q: Send a message to Greg to figure out if things areready for Wednesday. 
A: send-msg `find greg` Is everything ready forWednesday? 
Q: Ask Ilya if we're still having our meeting thisevening 
A: send-msg `find ilya` Are we still having a meetingthis evening? 
Q: Contact the ski store and figure out if I can getmy skis fixed before I leave on Thursday 
A: send-msg `find ski store` Would it be possible toget my skis fixed before I leave on Thursday? 
Q: Thank Nicolas for lunch 
A: send-msg `find nicolas` Thank you for lunch! 
Q: Tell Constance that I won't be home before 19:30tonight — unmovable meeting. 
A: send-msg `find constance` I won't be home before19:30 tonight. I have a meeting I can't move. 
Q: Tell John that I need to book an appointment at10:30 
A:  

The number of examples typically ranges from 0 to 100 depending on how many can fit in the maximum input length for a single prompt. Few-shot learning enables a major reduction in the amount of task-specific data required for accurate predictions.

One-shot : This case is the same as the few-shot approach except only one example is provided. The following example shows a one-shot prompt:

Convert the questions to a command:
Q: Ask Constance if we need some bread
A: send-msg `find constance` Do we need some bread?
Q: Send a message to Greg to figure out if things are ready for Wednesday.
A:

Zero-shot: In this case, no examples are provided to the model and only the task request is provided. The following example shows a zero-shot prompt:

Convert the question to a command:
Q: Ask Constance if we need some bread
A:

Chain-of-thought : Azure OpenAI's o1-preview and o1-mini models have new advanced reasoning capabilities using chain-of-thought (CoT) techniques. CoT techniques generate intermediate reasoning steps before providing a response, enabling o1 to address more complex challenges through step-by-step problem solving. o1 demonstrates improvements in benchmarks for reasoning-heavy domains such as research, strategy, science, coding and math, among others. o1 has safety improvements from advanced reasoning capabilities, with the ability to reason through and apply safety rules more effectively. This results in better performance alongside safety benchmarks such as generating illicit advice, choosing stereotyped responses, and succumbing to known jailbreaks.

For greater detail on the o1 family of models’ capabilities, see the OpenAI o1 System Card.

Azure OpenAI Evaluation

The evaluation of large language models is a critical step in measuring their performance across various tasks and dimensions. This task is especially important for fine-tuned models, where assessing the performance gains (or losses) from training is crucial. Without thorough evaluations, it can become challenging to understand how different versions of the model may impact your specific application .

Azure OpenAI Evaluation is a UI-based experience to evaluate data, including generated datasets from an Azure OpenAI deployment, or other manually curated files.

Azure OpenAI Evaluation has an optional step of generating responses. If the user opts into this step, we provide a prompt (System/User Message) to instruct the model how to generate responses.

Azure OpenAI Evaluation includes 9 categories of tests to score results. Some require ground truth data (like factuality), while others do not (schema validation). Graders are a mixture of CPU-based and model-based. Here is the list of testing criteria: Factuality, Sentiment, Valid JSON or XML, Criteria Match, Custom Prompt, Semantic Similarity, Contains string, Matches Schema and Text quality.

DALL·E APIs in Azure OpenAI Service

The DALL·E 2 and DALL·E 3 APIs use natural language prompts to generate new content. These models were trained on pairs of images and corresponding captions that were drawn from publicly available sources and other sources that OpenAI has licensed.

These generative AI models present myriad opportunities for developers, artists, designers, educators, and others. The models can bridge the gap between what you can imagine and what you can create. They allow for cross-domain, general understanding, and zero-shot translation between text prompt and images, often with a high degree of realism.

The main capabilities of the Azure OpenAI DALL·E APIs are:

• Text-to-image: The model takes in a text prompt to generate images.

  Example text prompt	Example generated image
  "Watercolor painting of the Seattle skyline"

• DALL·E 3 text-to-image with prompt transformation: Before a prompt is sent to the Azure OpenAI DALL·E 3 model to generate images,a safety and quality mitigation is applied to the prompt. Prompt transformation enhances the prompt with the goal of generating more diverse and higher-quality images.

  Prompt transformation is applied to every Azure OpenAI DALL·E 3 generation.

  After prompt transformation is applied to the original prompt, content filtering is applied as a secondary step prior to image generation; see Content Filtering for more information.

  Learn more about image generation prompting in OpenAI’s DALL·E documentation.

  Example text prompt	Example generated image without prompt transformation	Example generated image with prompt transformation
  "Watercolor painting of the Seattle skyline"

GPT-4 Turbo with Vision and GPT-4o in Azure OpenAI Service

GPT-4 Turbo with Vision and GPT-4o use natural language and image prompts to generate natural language or code responses. The models accept text only, image only, and interleaved text and image inputs. These models are part of the GPT family and are extensions of GPT-4.

The Azure AI Vision Service provides Video enhancement for GPT-4 Turbo with Vision (Preview), which can also accept video inputs to generate natural language or code responses. This capability is not available with the latest GPT-4 Turbo with Vision model release.

Example prompt	Example completion
"What is this used for?"	These are tomatoes. They are commonly used as a food ingredient in various dishes, both raw and cooked. Tomatoes can be used in salads, sandwiches, sauces, soups, juices, and many other culinary applications. They are also known for their rich nutritional content, including vitamins, minerals, and antioxidants.
How can I use to make this?	You can use tomatoes to make tomato soup by following these general steps: Ingredients: Tomatoes Onion Garlic Olive oil or butter Chicken or vegetable broth Salt Pepper Optional: herbs like basil, oregano, or thyme Instructions: Preparation: Wash the tomatoes thoroughly. Cut the tomatoes into halves or quarters. Peel and chop onions and garlic…

Face blurring

For inputs to GPT-4 Turbo with Vision, GPT-4o, and GPT-4o mini that contain images or videos of people, the system will first blur faces prior to processing to return the requested results.

For any inputs that contain images or videos of people, the system will blur faces prior to analyzing the images or videos. Blurring helps protect the privacy of the individuals and groups involved, and protect against restricted use cases. Blurring shouldn't affect the quality of your completions, but you might see the system refer to the blurring of faces in some instances.

The gpt-4o-realtime-preview model in Azure OpenAI service enables advanced speech-to-speech capabilities, providing real-time speech recognition, translation, and synthesis. GPT-4o allows for fluid communication by converting spoken input into coherent output in a target language.

Azure OpenAI service offers these functionalities through the realtime API:

• Processing spoken input to generate spoken output
• Translating spoken input into another language
• Transcribing input and output speech

The service is designed for rapid speech-to-speech processing of spoken interactions, facilitating real-time conversations with minimal latency.

The Whisper model in Azure OpenAI service enables speech-to-text transcription and translation of audio files. The model was trained on multilingual and multitask supervised data that was collected from the internet. These large and diverse datasets enabled improved robustness to attributes such as accents, background noise and technical language, while enabling transcription into multiple languages, and translation into English.

Azure OpenAI service provides these functionalities through two REST APIs:

• Transcribing a provided audio file
• Translating a provided audio file into English

The service provides the ability to synchronously process single audio files as fast as possible. This is limited to less than 15 to 30 minutes of audio depending on audio compression.

The service provides highly readable transcripts that often remove disfluencies and provide more accurate sentence boundaries, punctuation, and capitalization. You can also leverage prompts to improve the quality of the model outputs as fit for your scenario. See Best practices for improving system information for more information.

For greater detail on 4o model’s capabilities, see the OpenAI 4o System Card.

Intended uses

The o1, GPT-4, GPT-3, and Codex models, and Azure OpenAI evaluation can be used in multiple scenarios. The following list isn't comprehensive, but it illustrates the diversity of tasks that can be supported for models with appropriate mitigations:

• Chat and conversation interaction : Users can interact with a conversational agent that responds with responses drawn from trusted documents such as internal company documentation or tech support documentation. Conversations must be limited to answering scoped questions.
• Chat and conversation creation : Users can create a conversational agent that responds with responses drawn from trusted documents such as internal company documentation or tech support documentation. Conversations must be limited to answering scoped questions.
• Code generation or transformation scenarios : For example, converting one programming language to another, generating docstrings for functions, converting natural language to SQL.
• Journalistic content : For use to create new journalistic content or to rewrite journalistic content submitted by the user as a writing aid for predefined topics. Users cannot use the application as a general content creation tool for all topics.
• Question-answering : Users can ask questions and receive answers from trusted source documents such as internal company documentation. The application does not generate answers ungrounded in trusted source documentation.
• Reason over structured and unstructured data : Users can analyze inputs using classification, sentiment analysis of text, or entity extraction. Examples include analyzing product feedback sentiment, analyzing support calls and transcripts, and refining text-based search with embeddings.
• Search : Users can search trusted source documents such as internal company documentation. The application does not generate results ungrounded in trusted source documentation.
• Summarization : Users can submit content to be summarized for predefined topics built into the application and cannot use the application as an open-ended summarizer. Examples include summarization of internal company documentation, call center transcripts, technical reports, and product reviews.
• Writing assistance on specific topics : Users can create new content or rewrite content submitted by the user as a writing aid for business content or pre-defined topics. Users can only rewrite or create content for specific business purposes or predefined topics and cannot use the application as a general content creation tool for all topics. Examples of business content include proposals and reports. For journalistic use, see above Journalistic content use case.
• Data generation for fine-tuning: Users can use a model in Azure OpenAI to generate data which is used solely to fine-tune (i) another Azure OpenAI model, using the fine-tuning capabilities of Azure OpenAI, and/or (ii) another Azure AI custom model, using the fine-tuning capabilities of the Azure AI service. Generating data and fine-tuning models is limited to internal users only; the fine-tuned model may only be used for inferencing in the applicable Azure AI service and, for Azure OpenAI service, only for customer’s permitted use case(s) under this form.

Fine-tuned use cases

The following are additional use cases we recommend for fine-tuned text and code models. Fine tuning is most appropriate for:

• Steering the style, format, tone or qualitative aspects of responses via examples of the desired responses.
• Ensuring the model reliably produces a desired output such as providing responses in a specific format or ensuring responses are grounded by information in the prompt.
• Use cases with many edge cases that cannot be covered within examples in the prompt, such as complex natural language to code examples.
• Improving performance at specific skills or tasks such as classification, summarization, or formatting – that can be hard to describe within a prompt.
• Reducing costs or latency by utilizing shorter prompts, or swapping a fine-tuned version of a smaller/faster model for a more general-purpose model (e.g. fine tuned GPT-3.5-Turbo for GPT-4).

As with base models, the use case prohibitions outlined in the Azure OpenAI Code of conduct apply to fine-tuned models as well.

Fine tuning alone is not recommended for scenarios where you want to extend your model to include out-of-domain information, where explainability or grounding are important, or where the underlying data are updated frequently.

o1 use cases

The advanced reasoning capabilities of the o1-preview and o1-mini models may be best suited for reasoning-heavy uses in science, coding, math, and similar fields. Specific use cases could include:

• Complex code generation, analysis and optimization: Algorithm generation and advanced coding tasks to help developers execute multi-step workflows, better understanding the steps taken in code development.
• Advanced problem solving: Comprehensive brainstorming sessions, strategy development and breaking down multifaceted issues.
• Complex document comparison: Analyzing contracts, case files, or legal documents to discern subtle differences in document contents.
• Instruction following and workflow management: Handling workflows that require shorter context.

For greater detail on o1’s intended uses, visit the OpenAI o1 System Card.

Azure OpenAI evaluation use cases

Azure OpenAI evaluation is a text-only feature and can’t be used with models that support non-text inputs. Evals can be used in multiple scenarios including but not limited to:

• Text matching/comparison evaluation: This is helpful for scenarios where the user wants to check if the output matches an expected string. Users can also compare two sets of values and score the relationships. Examples include, but are not limited to, multiple-choice questions where answers are compared to an answer key, and string validation.
• Text quality: Text quality assesses response quality with methods such as Bleu, Rouge or cosine algorithms and is widely used in various natural language processing tasks such as machine translation, text summarization, and text generation, among others.
• Classification-based evaluation: Classification-based evaluation assesses the performance of a model by assigning responses to predefined categories or labels or by comparing the model’s output to a reference set of correct answers. Automated grading, sentiment analysis, and product categorization are among some of the common use cases.
• Conversational quality evaluation: Conversational quality evaluation involves comparing responses against predefined criteria using a detailed chain-of-thought (CoT) prompt. Common use cases include customer support, chatbot development, and educational assessments, among others.
• Criteria-based evaluation: One common scenario for criteria-based evaluation is factuality. Assessing factual accuracy involves comparing a submitted answer to an expert answer, focusing solely on factual content. This can be useful in educational tools to improve the accuracy of answers provided by LLMs or in research assistance tools to assess the factual accuracy of responses generated by LLMs in academic settings.
• String validity evaluation: one common scenario would be to check if model’s response follows a specific schema or is valid JSON or XML content.

Considerations when choosing a use case

We encourage customers to use the Azure OpenAI GPT-4, GPT-3, and Codex models in their innovative solutions or applications as approved in their Limited Access registration form. However, here are some considerations when choosing a use case:

• Not suitable for open-ended, unconstrained content generation. Scenarios where users can generate content on any topic are more likely to produce offensive or harmful text. The same is true of longer generations.
• Not suitable for scenarios where up-to-date, factually accurate information is crucial unless you have human reviewers or are using the models to search your own documents and have verified suitability for your scenario. The service does not have information about events that occur after its training date, likely has missing knowledge about some topics, and may not always produce factually accurate information.
• Avoid scenarios where use or misuse of the system could result in significant physical or psychological injury to an individual. For example, scenarios that diagnose patients or prescribe medications have the potential to cause significant harm. Incorporating meaningful human review and oversight into the scenario can help reduce the risk of harmful outcomes.
• Avoid scenarios where use or misuse of the system could have a consequential impact on life opportunities or legal status. Examples include scenarios where the AI system could affect an individual's legal status, legal rights, or their access to credit, education, employment, healthcare, housing, insurance, social welfare benefits, services, opportunities, or the terms on which they're provided. Incorporating meaningful human review and oversight into the scenario can help reduce the risk of harmful outcomes.
• Avoid high stakes scenarios that could lead to harm. The models hosted by Azure OpenAI service reflect certain societal views, biases, and other undesirable content present in the training data or the examples provided in the prompt. As a result, we caution against using the models in high-stakes scenarios where unfair, unreliable, or offensive behavior might be extremely costly or lead to harm. Incorporating meaningful human review and oversight into the scenario can help reduce the risk of harmful outcomes.
• Carefully consider use cases in high stakes domains or industry: Examples include but are not limited to healthcare, medicine, finance, or legal.
• Carefully consider well-scoped chatbot scenarios. Limiting the use of the service in chatbots to a narrow domain reduces the risk of generating unintended or undesirable responses.
• Carefully consider all generative use cases. Content generation scenarios may be more likely to produce unintended outputs and these scenarios require careful consideration and mitigations.

Intended use cases

DALL·E in Azure OpenAI

The DALL·E 2 and DALL·E 3 APIs in Azure OpenAI service can be used for various image- generation scenarios. The following list isn't comprehensive, but it illustrates the diversity of tasks that can be supported with appropriate mitigations.

• Accessibility features: Use to generate image-based visual descriptions.
• Art and design: Use to generate imagery, for artistic purposes only, for designs, artistic inspiration, mood boards, or design layouts.
• Communication: Use to create imagery for business-related communication, documentation, essays, newsletters, blog posts, social media, or memos.
• Education: Use to create imagery for enhanced or interactive learning materials, either for use in educational institutions or for professional training.
• Entertainment: Use to create imagery to enhance entertainment content such as video games, movies, TV, videos, recorded music, podcasts, audiobooks, or augmented or virtual reality.
• Journalistic content: Use to create imagery to enhance journalistic content.
• Marketing: Use to create marketing materials for product or service media, product instructions, business promotion, or advertisements. Should not be used to create advertisements that are personalized or targeted to individuals.
• Prototyping and conceptual development: Use to generate imagery for ideation or visualization of products or services. This use is applicable only in the context of the scenarios above.

GPT-4 Turbo with Vision and GPT-4o in Azure OpenAI

• Chat and conversation interaction: Users can interact with a conversational agent that responds with information drawn from trusted documentation such as internal company documentation or tech support documentation. Conversations must be limited to answering scoped questions.
• Chatbot and conversational agent creation: Users can create conversational agents that respond with information drawn from trusted documents such as internal company documentation or tech support documents. For instance, diagrams, charts, and other relevant images from technical documentation can enhance comprehension and provide more accurate responses. Conversations must be limited to answering scoped questions.
• Code generation or transformation scenarios: Converting one programming language to another or enabling users to generate code by using natural language or visual input. For example, users can take a photo of handwritten pseudocode or diagrams illustrating a coding concept and use the application to generate code based on that material.
• Reason over structured and unstructured data: Users can analyze inputs using classification, sentiment analysis of text, or entity extraction. Users can provide an image alongside a text query for analysis.
• Summarization: Users can submit content to be summarized for predefined topics built into the application and cannot use the application as an open-ended summarizer. Examples include summarization of internal company documentation, call center transcripts, technical reports, and product reviews.
• Writing assistance on specific topics: Users can create new content or rewrite content submitted by the user as a writing aid for business content or predefined topics. Users can only rewrite or create content for specific business purposes or predefined topics and cannot use the application as a general content creation tool for all topics. Examples of business content include proposals and reports.
• Image tagging: Users can detect and tag visual elements, including objects, living beings, scenery, and actions within an image. The service is not intended, and may not be used for identifying individuals or verifying individual identities.
• Image captioning: Users can generate descriptive natural language captions for visuals. Beyond simple descriptions, the application can identify and provide textual insights about specific subjects or landmarks within photos. If shown an image of the Eiffel Tower, the system might offer a concise description or highlight intriguing facts about the monument. The service may not be used to identify or verify individual identities.
• Object detection: For use to identify the positions of individual or multiple objects in an image by providing their specific coordinates. For instance, in an image that has scattered apples, the application can identify and indicate the location of each apple. Through this application, users can obtain spatial insights regarding objects captured in images. The service may not be used to identify or verify individual identities.
• Visual question answering: Users can ask questions about an image and receive contextually relevant responses. For instance, when shown a picture of a bird, users might ask, "What type of bird is this?" and receive a response like, "It's a European robin." The application can identify and interpret context within images to answer queries. For example, if presented with an image of a crowded marketplace, users can ask, "How many people are wearing hats?" or "What fruit is the vendor selling?" and the application can provide the answers. The system should not be used to answer identifying questions about people.
• Brand and landmark recognition: The application can be used to identify commercial brands and popular landmarks in images or videos from a preset database of thousands of global logos and landmarks.

Considerations when choosing a use case

We encourage customers to use the Azure OpenAI DALL·E 2, DALL·E 3, GPT-4 Turbo with Vision, and GPT-4o APIs in their innovative solutions or applications as approved in their Limited access registration form. However, here are some considerations when choosing a use case:

• Do not use for tracking or facial recognition, identification, or verification purposes. Examples include using models for surveillance of individuals and using models to verify two individuals pictured in two separate locations are the same person.
• Not suitable for scenarios where up-to-date, factually accurate information is crucial unless you have human reviewers or are using the models to search your own documents and have verified suitability for your scenario. The service does not have information about events that occur after its training date, likely has missing knowledge about some topics, and might not always produce factually accurate information.
• Avoid scenarios in which use or misuse of the system could have a consequential impact on life opportunities or legal status. Examples include scenarios in which the AI system could affect an individual's legal status, legal rights, or their access to credit, education, employment, healthcare, housing, insurance, social welfare benefits, services, opportunities, or the terms on which these rights and services are available.
• Avoid high-stakes scenarios that could lead to harm. The models hosted by Azure OpenAI Service might reflect certain societal views, biases, and other undesirable content present in the training data or in the examples provided in the prompt. As a result, we caution against using the models in high-stakes scenarios in which unfair, unreliable, or offensive behavior might be extremely costly or lead to harm.
• Avoid scenarios in which use or misuse of the system could spread false narratives about sensitive topics or people. Examples include the creation and distribution of deepfake synthetic media, misinformation about highly sensitive events, and generation of photorealistic imagery of real people in circumstances that reflect a false narrative.
• Carefully consider scenarios in which open-ended, unconstrained content generation is allowed. Scenarios in which users can generate content on any topic are more likely to produce offensive, harmful, or misleading content.
• Carefully consider scenarios that involve generating media in the style of known artists with published works. For example, there could be scenarios in which users distribute generated content that is in a style of an artist who maintains intellectual property (IP) rights in original art. Cases like these could have legal consequences or consequential impacts on opportunities for artists. Consider creating a process for artists to limit the creation of images associated with their names in your product or service.
• Carefully consider scenarios that involve generating images that include real people. Content that includes images of real people could be misused in a way that negatively affects life opportunities or legal status, as well as public perception and trust. Exercise caution when generating images of real people, living or dead, or of similar likeness.
• Carefully consider all use cases in high-stakes domains or industry: Examples include but are not limited to healthcare, education, finance, and legal.

Intended uses

The gpt-4o-realtime-preview model can be used for a variety of natural language processing tasks in a similar fashion to existing GPT models. However, this model specifically targets speech and audio use cases. Examples of tasks that this model supports include, but are not limited to, the following:

• Voice chat creation and interaction: Users can create and voice chat with a conversational agent that responds with voice capabilities.
• Language Translation: Users can translate between languages during conversation in real time. Subtitling can be supported with real-time input and output audio transcriptions. Language supportability is in line with existing gpt-4o model versions.
• Question-answering: Users can ask written and/or spoken questions and receive spoken answers from the model.
• Content generation: Users can create new audio content from user-defined input. The model will check for and prevent the creation of copyright protected material or impersonation of voices.

For greater detail on 4o model’s capabilities, see the OpenAI 4o System Card.

The Whisper model in Azure OpenAI service can be used for various speech recognition and speech translation tasks. The following list isn’t comprehensive, but it illustrates the diversity of tasks that can be supported with appropriate mitigations.

• Dictation: Users can transcribe spoken words in the form of audio files into text. For example, a user might use the service to transcribe call center recordings into reports.
• Captioning or transcription: Users can transcribe audio files into text for natural language understanding or analytic tasks like summarization and sentiment analysis. Examples include speech as input into dialog systems, making audio content more accessible, or creating a text translation to be analyzed.
• Translation: Users can translate multilingual audio files into English. This may be particularly attractive in multilingual scenarios.

Considerations when choosing other use cases

We encourage customers to use the gpt-4o-realtime-preview and Whisper models in Azure OpenAI service in their innovative solutions or applications. Here are some considerations when choosing a use case:

• Avoid scenarios in which the use or misuse of the system could have a consequential impact on life opportunities or legal status. Examples include scenarios in which the AI system could affect an individual's legal status, legal rights, or their access to credit, education, employment, healthcare, housing, insurance, social welfare benefits, services, opportunities, or the terms on which these items are available.
• Carefully consider all use cases in high-stakes domains or industries: Examples include but are not limited to healthcare, education, finance, and legal.
• Whisper in Azure Speech service: The OpenAI Whisper model is also available within Azure AI Speech services, enhancing the experience with advanced features like multi-lingual recognition and readability. Depending on your scenario, you might explore Azure Speech services and the service's additional capabilities like diaritization, customization, or processing multiple audio files per request.

The steps in conducting an Azure OpenAI evaluation are:

1. Provide data for evaluation: Either an uploaded flat file in JSONL format, or generated data based on a series of prompts.
2. Specify test cases to evaluate the data: Select one or more test cases to score the provided data with passing / failing grades.
3. Review and filter results: Each test includes a definition of passing and failing scores. After an evaluation runs, users can review their row-by-row results to see individual test results, or filter on passed / failed.

For additional information on how to evaluate and integrate these models responsibly, please see the RAI Overview document.

For additional information on how to evaluate and integrate these models responsibly, please see the RAI Overview document.

The OpenAI Whisper model is also available within Azure AI Speech services, enhancing the experience with advanced features like multi-lingual recognition and readability. Depending on your scenario, you might explore Azure Speech services and the service’s additional capabilities like diaritization, customization, real-time streaming, or processing multiple audio files per request. For additional information on how to evaluate and integrate the Whisper model responsibly, please see the Azure Speech services transparency note.