Live transcribe audio from a microphone with Foundry Local

Use Foundry Local's live audio transcription API to stream microphone audio and receive transcription results in real time. In this article, you create a console application that captures audio from your microphone, streams it to a local speech model, and prints transcription output as you speak.

Prerequisites

.NET 9.0 SDK or later installed.
A working microphone connected to your computer.

Samples repository

The complete sample code for this article is available in the foundry-samples GitHub repository. To clone the repository and navigate to the sample use:

git clone https://github.com/microsoft-foundry/foundry-samples.git
cd foundry-samples/samples/csharp/foundry-local/live-audio-transcription-example

Install packages

If you're developing or shipping on Windows, select the Windows tab. The Windows package integrates with the Windows ML runtime — it provides the same API surface area with a wider breadth of hardware acceleration.

Windows
Cross-Platform

dotnet add package Microsoft.AI.Foundry.Local.WinML
dotnet add package OpenAI

dotnet add package Microsoft.AI.Foundry.Local
dotnet add package OpenAI

The C# samples in the GitHub repository are preconfigured projects. If you're building from scratch, you should read the Foundry Local SDK reference for more details on how to set up your C# project with Foundry Local.

Install the NAudio package for microphone capture:

dotnet add package NAudio

Live transcribe from microphone

The following code initializes the Foundry Local SDK, loads a streaming speech model, captures audio from your microphone using NAudio, and streams it to the live transcription API. Partial results appear as you speak, and final results are printed on a new line.

Copy and paste the following code into Program.cs:

// Live Audio Transcription — Foundry Local SDK Example
//
// NAudio's WaveInEvent is Windows-only. On non-Windows platforms, the sample
// falls back to synthetic PCM audio.

using Microsoft.AI.Foundry.Local;
using NAudio.Wave;

Console.WriteLine("===========================================================");
Console.WriteLine("   Foundry Local -- Live Audio Transcription Demo");
Console.WriteLine("===========================================================");
Console.WriteLine();

var config = new Configuration
{
    AppName = "foundry_local_samples",
    LogLevel = Microsoft.AI.Foundry.Local.LogLevel.Information
};

await FoundryLocalManager.CreateAsync(config, Utils.GetAppLogger());
var mgr = FoundryLocalManager.Instance;

await Utils.RunWithSpinner("Registering execution providers", mgr.DownloadAndRegisterEpsAsync());

var catalog = await mgr.GetCatalogAsync();

// English-only:
var modelAlias = "nemotron-speech-streaming-en-0.6b";
// Multi-lingual (supports 30+ languages including auto-detect):
// var modelAlias = "nvidia-nemotron-3.5-asr-streaming-multilingual-0.6b";

var model = await catalog.GetModelAsync(modelAlias) ?? throw new Exception($"Model \"{modelAlias}\" not found in catalog");

await model.DownloadAsync(progress =>
{
    Console.Write($"\rDownloading model: {progress:F2}%");
    if (progress >= 100f)
    {
        Console.WriteLine();
    }
});

Console.Write($"Loading model {model.Id}...");
await model.LoadAsync();
Console.WriteLine("done.");

var audioClient = await model.GetAudioClientAsync();
var session = audioClient.CreateLiveTranscriptionSession();
session.Settings.SampleRate = 16000;  // Default is 16000; shown here to match the NAudio WaveFormat below
session.Settings.Channels = 1;
session.Settings.Language = "en";                  // English (default)
// Multi-lingual examples:
// session.Settings.Language = "de";     // German
// session.Settings.Language = "zh-CN";  // Chinese (Simplified)
// session.Settings.Language = "auto";   // Auto-detect language

await session.StartAsync();
Console.WriteLine("       Session started");

var readTask = Task.Run(async () =>
{
    try
    {
        await foreach (var result in session.GetStream())
        {
            var text = result.Content?[0]?.Text;
            if (result.IsFinal)
            {
                Console.WriteLine();
                Console.WriteLine($"  [FINAL] {text}");
                Console.Out.Flush();
            }
            else if (!string.IsNullOrEmpty(text))
            {
                Console.ForegroundColor = ConsoleColor.Cyan;
                Console.Write(text);
                Console.ResetColor();
                Console.Out.Flush();
            }
        }
    }
    catch (OperationCanceledException) { }
});

bool useSynth = args.Contains("--synth");

// NAudio WaveInEvent is Windows-only. On other platforms, fall back to synthetic audio.
if (!useSynth && OperatingSystem.IsWindows())
{
    using var waveIn = new WaveInEvent
    {
        WaveFormat = new WaveFormat(rate: 16000, bits: 16, channels: 1),
        BufferMilliseconds = 100
    };

    // Use a bounded channel to avoid unbounded fire-and-forget AppendAsync calls.
    // NAudio's DataAvailable callback is synchronous, so we enqueue PCM chunks and
    // await AppendAsync on a dedicated task to respect SDK backpressure.
    var audioChannel = System.Threading.Channels.Channel.CreateBounded<byte[]>(
        new System.Threading.Channels.BoundedChannelOptions(50)
        {
            FullMode = System.Threading.Channels.BoundedChannelFullMode.DropOldest
        });

    var appendTask = Task.Run(async () =>
    {
        await foreach (var chunk in audioChannel.Reader.ReadAllAsync())
        {
            await session.AppendAsync(chunk);
        }
    });

    waveIn.DataAvailable += (sender, e) =>
    {
        if (e.BytesRecorded > 0)
        {
            var buffer = new byte[e.BytesRecorded];
            Buffer.BlockCopy(e.Buffer, 0, buffer, 0, e.BytesRecorded);
            audioChannel.Writer.TryWrite(buffer);
        }
    };

    Console.WriteLine();
    Console.WriteLine("===========================================================");
    Console.WriteLine("  LIVE TRANSCRIPTION ACTIVE");
    Console.WriteLine("  Speak into your microphone.");
    Console.WriteLine("  Transcription appears in real-time (cyan text).");
    Console.WriteLine("  Press ENTER to stop recording.");
    Console.WriteLine("===========================================================");
    Console.WriteLine();

    waveIn.StartRecording();
    Console.ReadLine();
    waveIn.StopRecording();

    audioChannel.Writer.Complete();
    await appendTask;
}
else
{
    if (!OperatingSystem.IsWindows() && !useSynth)
    {
        Console.WriteLine("NAudio mic capture is Windows-only. Falling back to synthetic audio...");
    }

    // Synthetic PCM fallback: 440Hz sine wave, 2 seconds
    Console.WriteLine("Pushing synthetic audio (440Hz sine, 2s)...");
    const int sampleRate = 16000;
    const int duration = 2;
    var totalSamples = sampleRate * duration;
    var pcmBytes = new byte[totalSamples * 2];
    for (int i = 0; i < totalSamples; i++)
    {
        double t = (double)i / sampleRate;
        short sample = (short)(short.MaxValue * 0.5 * Math.Sin(2 * Math.PI * 440 * t));
        pcmBytes[i * 2] = (byte)(sample & 0xFF);
        pcmBytes[i * 2 + 1] = (byte)((sample >> 8) & 0xFF);
    }

    int chunkSize = (sampleRate / 10) * 2; // 100ms
    for (int offset = 0; offset < pcmBytes.Length; offset += chunkSize)
    {
        int len = Math.Min(chunkSize, pcmBytes.Length - offset);
        await session.AppendAsync(pcmBytes.AsMemory(offset, len));
        await Task.Delay(100);
    }

    Console.WriteLine("✓ Synthetic audio pushed");
    await Task.Delay(3000); // Wait for remaining transcription results
}

await session.StopAsync();
await readTask;

await model.UnloadAsync();

The CreateLiveTranscriptionSession method returns a session that accepts raw Pulse-code modulation (PCM) audio and yields transcription results as an async stream. NAudio's WaveInEvent captures microphone audio at 16-kHz mono—the format the session expects.

Run the application:

dotnet run

Speak into your microphone. You see real-time transcription output:

Listening... (press Ctrl+C to stop)
Hello, this is a test of the live transcription feature.
It transcribes audio from the microphone in real time.

Press Ctrl+C to stop recording. The model finishes processing any remaining audio and the application exits.

Prerequisites

Node.js version 20 or later installed.
A working microphone connected to your computer.

Samples repository

The complete sample code for this article is available in the foundry-samples GitHub repository. To clone the repository and navigate to the sample use:

git clone https://github.com/microsoft-foundry/foundry-samples.git
cd foundry-samples/samples/javascript/foundry-local/live-audio-transcription-example

npm install foundry-local-sdk-winml openai

npm install foundry-local-sdk openai

Install the naudiodon2 package for microphone capture:

npm install naudiodon2

Live transcribe from microphone

The following code initializes the Foundry Local SDK, loads a streaming speech model, captures audio from your microphone using naudiodon2, and streams it to the live transcription API. Partial results appear as you speak, and final results are printed on a new line.

Copy and paste the following code into app.js:

// Live Audio Transcription Example — Foundry Local JS SDK
//
// Demonstrates real-time microphone-to-text using the JS SDK.
// Requires: npm install foundry-local-sdk naudiodon2
//
// Usage: node app.js

import { FoundryLocalManager } from 'foundry-local-sdk';

console.log('╔══════════════════════════════════════════════════════════╗');
console.log('║   Foundry Local — Live Audio Transcription (JS SDK)      ║');
console.log('╚══════════════════════════════════════════════════════════╝');
console.log();

// Initialize the Foundry Local SDK
console.log('Initializing Foundry Local SDK...');
const manager = FoundryLocalManager.create({
    appName: 'foundry_local_samples',
    logLevel: 'info'
});
console.log('✓ SDK initialized');

// Get and load the nemotron model
// English-only:
const modelAlias = 'nemotron-speech-streaming-en-0.6b';
// Multi-lingual (supports 30+ languages including auto-detect):
// const modelAlias = 'nvidia-nemotron-3.5-asr-streaming-multilingual-0.6b';
let model = await manager.catalog.getModel(modelAlias);
if (!model) {
    console.error(`ERROR: Model "${modelAlias}" not found in catalog.`);
    process.exit(1);
}

console.log(`Found model: ${model.id}`);
console.log('Downloading model (if needed)...');
await model.download((progress) => {
    process.stdout.write(`\rDownloading... ${progress.toFixed(2)}%`);
});
console.log('\n✓ Model downloaded');

console.log('Loading model...');
await model.load();
console.log('✓ Model loaded');

// Create live transcription session (same pattern as C# sample).
const audioClient = model.createAudioClient();
const session = audioClient.createLiveTranscriptionSession();

session.settings.sampleRate = 16000;  // Default is 16000; shown here for clarity
session.settings.channels = 1;
session.settings.bitsPerSample = 16;
session.settings.language = 'en';                  // English (default)
// Multi-lingual examples:
// session.settings.language = 'de';     // German
// session.settings.language = 'zh-CN';  // Chinese (Simplified)
// session.settings.language = 'auto';   // Auto-detect language

console.log('Starting streaming session...');
await session.start();
console.log('✓ Session started');

// Read transcription results in background
const readPromise = (async () => {
    try {
        for await (const result of session.getStream()) {
            const text = result.content?.[0]?.text;
            if (!text) continue;

            // `is_final` is a transcript-state marker only. It should not stop the app.
            if (result.is_final) {
                process.stdout.write(`\n  [FINAL] ${text}\n`);
            } else {
                process.stdout.write(text);
            }
        }
    } catch (err) {
        if (err.name !== 'AbortError') {
            console.error('Stream error:', err.message);
        }
    }
})();

// --- Microphone capture ---
// This example uses naudiodon2 for cross-platform audio capture.
// Install with: npm install naudiodon2
//
// If you prefer a different audio library, just push PCM bytes
// (16-bit signed LE, mono, 16kHz) via session.append().

let audioInput;
try {
    const { default: portAudio } = await import('naudiodon2');

    audioInput = portAudio.AudioIO({
        inOptions: {
            channelCount: session.settings.channels,
            sampleFormat: session.settings.bitsPerSample === 16
                ? portAudio.SampleFormat16Bit
                : portAudio.SampleFormat32Bit,
            sampleRate: session.settings.sampleRate,
            // Larger chunk size lowers callback frequency and reduces overflow risk.
            framesPerBuffer: 3200,
            // Allow deeper native queue during occasional event-loop stalls.
            maxQueue: 64
        }
    });

    const appendQueue = [];
    let pumping = false;
    let warnedQueueDrop = false;

    const pumpAudio = async () => {
        if (pumping) return;
        pumping = true;
        try {
            while (appendQueue.length > 0) {
                const pcm = appendQueue.shift();
                await session.append(pcm);
            }
        } catch (err) {
            console.error('append error:', err.message);
        } finally {
            pumping = false;
            // Handle race where new data arrived after loop exit.
            if (appendQueue.length > 0) {
                void pumpAudio();
            }
        }
    };

    audioInput.on('data', (buffer) => {
        // Single copy: slice the underlying ArrayBuffer to get an independent Uint8Array.
        const copy = new Uint8Array(buffer.buffer, buffer.byteOffset, buffer.byteLength).slice();

        // Keep a bounded queue to avoid unbounded memory growth.
        if (appendQueue.length >= 100) {
            appendQueue.shift();
            if (!warnedQueueDrop) {
                warnedQueueDrop = true;
                console.warn('Audio append queue overflow; dropping oldest chunk to keep stream alive.');
            }
        }

        appendQueue.push(copy);
        void pumpAudio();
    });

    console.log();
    console.log('════════════════════════════════════════════════════════════');
    console.log('  LIVE TRANSCRIPTION ACTIVE');
    console.log('  Speak into your microphone.');
    console.log('  Press Ctrl+C to stop.');
    console.log('════════════════════════════════════════════════════════════');
    console.log();

    audioInput.start();
} catch (err) {
    console.warn('⚠ Could not initialize microphone (naudiodon2 may not be installed).');
    console.warn('  Install with: npm install naudiodon2');
    console.warn('  Falling back to synthetic audio test...');
    console.warn();

    // Fallback: push 2 seconds of synthetic PCM (440Hz sine wave)
    const sampleRate = session.settings.sampleRate;
    const duration = 2;
    const totalSamples = sampleRate * duration;
    const pcmBytes = new Uint8Array(totalSamples * 2);
    for (let i = 0; i < totalSamples; i++) {
        const t = i / sampleRate;
        const sample = Math.round(32767 * 0.5 * Math.sin(2 * Math.PI * 440 * t));
        pcmBytes[i * 2] = sample & 0xFF;
        pcmBytes[i * 2 + 1] = (sample >> 8) & 0xFF;
    }

    // Push in 100ms chunks
    const chunkSize = (sampleRate / 10) * 2;
    for (let offset = 0; offset < pcmBytes.length; offset += chunkSize) {
        const len = Math.min(chunkSize, pcmBytes.length - offset);
        await session.append(pcmBytes.slice(offset, offset + len));
    }

    console.log('✓ Synthetic audio pushed');
    console.log('Waiting briefly for final transcription results...');
    await new Promise((resolve) => setTimeout(resolve, 3000));
    await session.stop();
    await readPromise;
    await model.unload();
    console.log('✓ Done');
    process.exit(0);
}

// Handle graceful shutdown
process.on('SIGINT', async () => {
    console.log('\n\nStopping...');
    if (audioInput) {
        audioInput.quit();
    }
    await session.stop();
    await readPromise;
    await model.unload();
    console.log('✓ Done');
    process.exit(0);
});

The createLiveTranscriptionSession method returns a session that accepts raw PCM audio and yields transcription results as an async generator. The naudiodon2 AudioIO captures microphone audio at 16 kHz mono 16-bit — the format the session expects.

Run the application:

node app.js

Speak into your microphone. You see real-time transcription output:

Listening... (press Ctrl+C to stop)
Hello, this is a test of the live transcription feature.
It transcribes audio from the microphone in real time.

Press Ctrl+C to stop recording. The model finishes processing any remaining audio and the application exits.

Prerequisites

Python 3.11 or later installed.
A working microphone connected to your computer.

Samples repository

The complete sample code for this article is available in the foundry-samples GitHub repository. To clone the repository and navigate to the sample use:

git clone https://github.com/microsoft-foundry/foundry-samples.git
cd foundry-samples/samples/python/foundry-local/live-audio-transcription

Install packages

Windows
Cross-Platform

pip install foundry-local-sdk-winml openai

pip install foundry-local-sdk openai

Install PyAudio for microphone capture:

pip install pyaudio

Live transcribe from microphone

The following code initializes the Foundry Local SDK, loads a streaming speech model, captures audio from your microphone using PyAudio, and streams it to the live transcription API. Partial results appear as you speak, and final results are printed on a new line.

Copy and paste the following code into app.py:

# Live Audio Transcription — Foundry Local SDK Example (Python)
#
# Tries PyAudio mic capture first; falls back to synthetic PCM if unavailable.
#
# Usage:
#   pip install -r requirements.txt
#   python src/app.py              # Live microphone
#   python src/app.py --synth      # Synthetic 440Hz sine wave

import math
import signal
import struct
import sys
import threading
import time

from foundry_local_sdk import Configuration, FoundryLocalManager

use_synth = "--synth" in sys.argv

print("===========================================================")
print("   Foundry Local -- Live Audio Transcription Demo (Python)")
print("===========================================================")
print()

config = Configuration(app_name="foundry_local_samples")
FoundryLocalManager.initialize(config)
manager = FoundryLocalManager.instance

manager.download_and_register_eps()

# English-only:
model_alias = "nemotron-speech-streaming-en-0.6b"
# Multi-lingual (supports 30+ languages including auto-detect):
# model_alias = "nvidia-nemotron-3.5-asr-streaming-multilingual-0.6b"
model = manager.catalog.get_model(model_alias)
if model is None:
    raise RuntimeError(f'Model "{model_alias}" not found in catalog')

model.download(
    lambda progress: print(f"\rDownloading model: {progress:.2f}%", end="", flush=True)
)
print()
print(f"Loading model {model.id}...", end="")
model.load()
print("done.")

audio_client = model.get_audio_client()
session = audio_client.create_live_transcription_session()
session.settings.sample_rate = 16000
session.settings.channels = 1
session.settings.language = "en"  # English (default)
# Multi-lingual examples:
# session.settings.language = "de"     # German
# session.settings.language = "zh-CN"  # Chinese (Simplified)
# session.settings.language = "auto"   # Auto-detect language

session.start()
print("✓ Session started")

# --- Background thread reads transcription results (mirrors JS readPromise) ---


def read_results():
    for result in session.get_stream():
        text = result.content[0].text if result.content else ""
        if result.is_final:
            print()
            print(f"  [FINAL] {text}")
        elif text:
            print(text, end="", flush=True)


read_thread = threading.Thread(target=read_results, daemon=True)
read_thread.start()

# --- Microphone capture (mirrors JS naudiodon2 / C++ PortAudio) ---
# Try PyAudio for mic input; fall back to synthetic PCM on failure.

RATE = 16000
CHANNELS = 1
CHUNK = RATE // 10  # 100ms of audio = 1600 frames

stop_event = threading.Event()
mic_active = False
pa = None
stream = None

if not use_synth:
    try:
        import pyaudio

        pa = pyaudio.PyAudio()
        stream = pa.open(
            format=pyaudio.paInt16,
            channels=CHANNELS,
            rate=RATE,
            input=True,
            frames_per_buffer=CHUNK,
        )
        mic_active = True

        print()
        print("===========================================================")
        print("  LIVE TRANSCRIPTION ACTIVE")
        print("  Speak into your microphone.")
        print("  Press Ctrl+C to stop.")
        print("===========================================================")
        print()

        def capture_mic():
            while not stop_event.is_set():
                try:
                    pcm_data = stream.read(CHUNK, exception_on_overflow=False)
                    if pcm_data:
                        session.append(pcm_data)
                except Exception as e:
                    print(f"\n[ERROR] Microphone capture failed: {e}")
                    stop_event.set()
                    break

        capture_thread = threading.Thread(target=capture_mic, daemon=True)
        capture_thread.start()

    except Exception as e:
        print(f"Could not initialize microphone: {e}")
        print("Falling back to synthetic audio test...")
        print()
        mic_active = False
        if stream:
            stream.close()
        if pa:
            pa.terminate()
        pa = None
        stream = None

# Fallback: push synthetic PCM (440Hz sine wave) — mirrors JS catch block
if not mic_active:
    print("Pushing synthetic audio (440Hz sine, 2s)...")
    duration = 2
    total_samples = RATE * duration
    pcm_bytes = bytearray(total_samples * 2)
    for i in range(total_samples):
        t = i / RATE
        sample = int(32767 * 0.5 * math.sin(2 * math.pi * 440 * t))
        struct.pack_into("<h", pcm_bytes, i * 2, sample)

    chunk_size = (RATE // 10) * 2  # 100ms
    for offset in range(0, len(pcm_bytes), chunk_size):
        end = min(offset + chunk_size, len(pcm_bytes))
        session.append(bytes(pcm_bytes[offset:end]))
        time.sleep(0.1)

    print("✓ Synthetic audio pushed")
    time.sleep(3)  # Wait for remaining transcription results


# --- Graceful shutdown (mirrors JS SIGINT handler / C++ SignalHandler) ---


def shutdown(*_args):
    print("\n\nStopping...")
    stop_event.set()

    if stream:
        stream.stop_stream()
        stream.close()
    if pa:
        pa.terminate()

    session.stop()
    read_thread.join(timeout=5)
    model.unload()
    print("✓ Done")
    sys.exit(0)


signal.signal(signal.SIGINT, lambda *a: shutdown())

if mic_active:
    # Block until Ctrl+C
    stop_event.wait()
else:
    shutdown()

The create_live_transcription_session method returns a session that accepts raw PCM audio and yields transcription results as you stream chunks. PyAudio captures microphone audio at 16 kHz mono 16-bit — the format the session expects.

Run the application:

python src/app.py

Speak into your microphone. You see real-time transcription output:

Listening... (press Ctrl+C to stop)
Hello, this is a test of the live transcription feature.
It transcribes audio from the microphone in real time.

Press Ctrl+C to stop recording. The model finishes processing any remaining audio and the application exits.

Prerequisites

Rust and Cargo installed (Rust 1.70.0 or later).
A working microphone connected to your computer.

Samples repository

The complete sample code for this article is available in the foundry-samples GitHub repository. To clone the repository and navigate to the sample use:

git clone https://github.com/microsoft-foundry/foundry-samples.git
cd foundry-samples/samples/rust/foundry-local/live-audio-transcription-example

Install packages

Windows
Cross-Platform

cargo add foundry-local-sdk --features winml
cargo add tokio --features full
cargo add tokio-stream anyhow

cargo add foundry-local-sdk
cargo add tokio --features full
cargo add tokio-stream anyhow

The sample uses the cpal crate for cross-platform microphone capture. The dependency is already listed in Cargo.toml.

Live transcribe from microphone

The following code initializes the Foundry Local SDK, loads a streaming speech model, captures audio from your microphone using cpal, and streams it to the live transcription API. Partial results appear as you speak, and final results are printed on a new line.

Replace the contents of src/main.rs with the following code:

// Live Audio Transcription — Foundry Local Rust SDK Example
//
// Tries CPAL mic capture first; falls back to synthetic PCM if unavailable.
//
// Usage:
//   cargo run                  # Live microphone (press Ctrl+C to stop)
//   cargo run -- --synth       # Synthetic 440Hz sine wave

use std::env;
use std::io::{self, Write};
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;

use cpal::traits::{DeviceTrait, HostTrait, StreamTrait};
use foundry_local_sdk::{FoundryLocalConfig, FoundryLocalManager, LiveAudioTranscriptionSession};
use tokio_stream::StreamExt;

// English-only:
const ALIAS: &str = "nemotron-speech-streaming-en-0.6b";
// Multi-lingual (supports 30+ languages including auto-detect):
// const ALIAS: &str = "nvidia-nemotron-3.5-asr-streaming-multilingual-0.6b";

// Global flag for Ctrl+C graceful shutdown (mirrors JS process.on('SIGINT'))
static RUNNING: AtomicBool = AtomicBool::new(true);

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let use_synth = env::args().any(|a| a == "--synth");

    // Install Ctrl+C handler (mirrors JS SIGINT / C++ SignalHandler)
    let running = Arc::new(AtomicBool::new(true));
    let running_for_signal = running.clone();
    ctrlc::set_handler(move || {
        RUNNING.store(false, Ordering::SeqCst);
        running_for_signal.store(false, Ordering::SeqCst);
    })?;

    println!("===========================================================");
    println!("   Foundry Local -- Live Audio Transcription Demo (Rust)");
    println!("===========================================================");
    println!();

    let manager = FoundryLocalManager::create(FoundryLocalConfig::new("foundry_local_samples"))?;
    let model = manager.catalog().get_model(ALIAS).await?;
    println!("Model: {} (id: {})", model.alias(), model.id());

    if !model.is_cached().await? {
        println!("Downloading model...");
        model
            .download(Some(|progress: f64| {
                print!("\r  {progress:.1}%");
                io::stdout().flush().ok();
            }))
            .await?;
        println!();
    }

    println!("Loading model...");
    model.load().await?;
    println!("✓ Model loaded\n");

    let audio_client = model.create_audio_client();
    let mut session = audio_client.create_live_transcription_session();
    session.settings.language = Some("en".into());    // English (default)
    // session.settings.language = Some("de".into());    // German
    // session.settings.language = Some("zh-CN".into()); // Chinese (Simplified)
    // session.settings.language = Some("auto".into());  // Auto-detect language
    let session = Arc::new(session);
    session.start(None).await?;
    println!("✓ Session started\n");

    // --- Background task reads transcription results (mirrors JS readPromise) ---
    let mut stream = session.get_stream().await?;
    let read_task = tokio::spawn(async move {
        while let Some(result) = stream.next().await {
            match result {
                Ok(r) => {
                    if let Some(content) = r.content.first() {
                        let text = &content.text;
                        if r.is_final {
                            println!();
                            println!("  [FINAL] {text}");
                        } else if !text.is_empty() {
                            print!("{text}");
                            io::stdout().flush().ok();
                        }
                    }
                }
                Err(e) => {
                    eprintln!("\n[ERROR] Stream error: {e}");
                    break;
                }
            }
        }
    });

    // --- Microphone capture (mirrors JS naudiodon2 / C++ PortAudio / Python PyAudio) ---
    // Try CPAL for mic input; fall back to synthetic PCM on failure.

    let mut mic_active = false;

    if !use_synth {
        match try_start_mic(&session, &running).await {
            Ok(()) => {
                mic_active = true;
            }
            Err(e) => {
                eprintln!("Could not initialize microphone: {e}");
                eprintln!("Falling back to synthetic audio test...\n");
            }
        }
    }

    // Fallback: push synthetic PCM (440Hz sine wave) — mirrors JS catch block
    if !mic_active {
        println!("Pushing synthetic audio (440Hz sine, 2s)...");
        let pcm_data = generate_sine_wave_pcm(16000, 2, 440.0);
        let chunk_size = 16000 / 10 * 2; // 100ms
        let chunk_interval = std::time::Duration::from_millis(100);
        for offset in (0..pcm_data.len()).step_by(chunk_size) {
            if !running.load(Ordering::SeqCst) {
                break;
            }
            let end = std::cmp::min(offset + chunk_size, pcm_data.len());
            session.append(&pcm_data[offset..end], None).await?;
            tokio::time::sleep(chunk_interval).await;
        }
        println!("✓ Synthetic audio pushed");

        // Wait for remaining transcription results
        tokio::time::sleep(std::time::Duration::from_secs(3)).await;
    }

    // Graceful shutdown (mirrors JS SIGINT handler)
    println!("\n\nStopping...");
    session.stop(None).await?;
    read_task.await?;
    model.unload().await?;
    println!("✓ Done");
    Ok(())
}

/// Try to open the default microphone with CPAL and forward PCM to the session.
/// Blocks until Ctrl+C is pressed.
async fn try_start_mic(
    session: &Arc<LiveAudioTranscriptionSession>,
    running: &Arc<AtomicBool>,
) -> Result<(), Box<dyn std::error::Error>> {
    let host = cpal::default_host();
    let device = host
        .default_input_device()
        .ok_or("No input audio device available")?;
    let default_config = device.default_input_config()?;
    let device_rate = default_config.sample_rate().0;
    let device_channels = default_config.channels();
    let sample_format = default_config.sample_format();

    let mic_config = cpal::StreamConfig {
        channels: device_channels,
        sample_rate: cpal::SampleRate(device_rate),
        buffer_size: cpal::BufferSize::Default,
    };

    // Bounded channel (cap=100) mirrors JS appendQueue / C++ AudioQueue
    let (audio_tx, mut audio_rx) = tokio::sync::mpsc::channel::<Vec<u8>>(100);
    let err_fn = |err| eprintln!("Microphone stream error: {err}");

    // CPAL may deliver f32, i16, or u16 depending on the device/host. Convert
    // each supported sample format to f32 in [-1.0, 1.0] before resampling.
    let input_stream = match sample_format {
        cpal::SampleFormat::F32 => {
            let tx = audio_tx.clone();
            device.build_input_stream(
                &mic_config,
                move |data: &[f32], _: &cpal::InputCallbackInfo| {
                    let bytes = convert_audio(data, device_channels, device_rate);
                    if !bytes.is_empty() {
                        let _ = tx.try_send(bytes);
                    }
                },
                err_fn,
                None,
            )?
        }
        cpal::SampleFormat::I16 => {
            let tx = audio_tx.clone();
            device.build_input_stream(
                &mic_config,
                move |data: &[i16], _: &cpal::InputCallbackInfo| {
                    let samples: Vec<f32> = data
                        .iter()
                        .map(|&s| s as f32 / i16::MAX as f32)
                        .collect();
                    let bytes = convert_audio(&samples, device_channels, device_rate);
                    if !bytes.is_empty() {
                        let _ = tx.try_send(bytes);
                    }
                },
                err_fn,
                None,
            )?
        }
        cpal::SampleFormat::U16 => {
            let tx = audio_tx.clone();
            device.build_input_stream(
                &mic_config,
                move |data: &[u16], _: &cpal::InputCallbackInfo| {
                    let samples: Vec<f32> = data
                        .iter()
                        .map(|&s| (s as f32 / u16::MAX as f32) * 2.0 - 1.0)
                        .collect();
                    let bytes = convert_audio(&samples, device_channels, device_rate);
                    if !bytes.is_empty() {
                        let _ = tx.try_send(bytes);
                    }
                },
                err_fn,
                None,
            )?
        }
        other => {
            return Err(format!("Unsupported input sample format: {other:?}").into());
        }
    };
    drop(audio_tx);

    input_stream.play()?;

    println!("===========================================================");
    println!("  LIVE TRANSCRIPTION ACTIVE");
    println!("  Speak into your microphone.");
    println!("  Press Ctrl+C to stop.");
    println!("===========================================================");
    println!();

    // Pump audio from channel to session (mirrors JS pumpAudio / C++ pump loop)
    let session_clone = Arc::clone(session);
    let forward_task = tokio::spawn(async move {
        while let Some(bytes) = audio_rx.recv().await {
            if let Err(e) = session_clone.append(&bytes, None).await {
                eprintln!("Append error: {e}");
                break;
            }
        }
    });

    // Block until Ctrl+C
    while running.load(Ordering::SeqCst) {
        tokio::time::sleep(std::time::Duration::from_millis(100)).await;
    }

    drop(input_stream);
    forward_task.await?;
    Ok(())
}

fn convert_audio(data: &[f32], channels: u16, sample_rate: u32) -> Vec<u8> {
    let mono: Vec<f32> = if channels > 1 {
        data.chunks(channels as usize)
            .map(|frame| frame.iter().sum::<f32>() / channels as f32)
            .collect()
    } else {
        data.to_vec()
    };

    let resampled = if sample_rate != 16000 {
        resample(&mono, sample_rate, 16000)
    } else {
        mono
    };

    let mut bytes = Vec::with_capacity(resampled.len() * 2);
    for &s in &resampled {
        let clamped = s.clamp(-1.0, 1.0);
        let sample = (clamped * i16::MAX as f32) as i16;
        bytes.extend_from_slice(&sample.to_le_bytes());
    }
    bytes
}

fn generate_sine_wave_pcm(sample_rate: i32, duration_seconds: i32, frequency: f64) -> Vec<u8> {
    let total_samples = (sample_rate * duration_seconds) as usize;
    let mut pcm_bytes = vec![0u8; total_samples * 2];

    for i in 0..total_samples {
        let t = i as f64 / sample_rate as f64;
        let sample =
            (i16::MAX as f64 * 0.5 * (2.0 * std::f64::consts::PI * frequency * t).sin()) as i16;
        let bytes = sample.to_le_bytes();
        pcm_bytes[i * 2] = bytes[0];
        pcm_bytes[i * 2 + 1] = bytes[1];
    }

    pcm_bytes
}

fn resample(input: &[f32], from_rate: u32, to_rate: u32) -> Vec<f32> {
    if from_rate == to_rate || input.is_empty() {
        return input.to_vec();
    }

    let ratio = from_rate as f64 / to_rate as f64;
    let out_len = (input.len() as f64 / ratio).ceil() as usize;
    let mut output = Vec::with_capacity(out_len);

    for i in 0..out_len {
        let src_idx = i as f64 * ratio;
        let idx = src_idx as usize;
        let frac = src_idx - idx as f64;
        let s0 = input[idx.min(input.len() - 1)];
        let s1 = input[(idx + 1).min(input.len() - 1)];
        output.push(s0 + (s1 - s0) * frac as f32);
    }

    output
}

The create_live_transcription_session method returns a session that accepts raw Pulse-code modulation (PCM) audio and yields transcription results as an async stream. The cpal input stream captures microphone audio at 16-kHz mono, which is the format the session expects.

Run the application:

cargo run

Speak into your microphone. You see real-time transcription output:

Listening... (press Ctrl+C to stop)
Hello, this is a test of the live transcription feature.
It transcribes audio from the microphone in real time.

Press Ctrl+C to stop recording. The model finishes processing any remaining audio and the application exits.

Feedback

Was this page helpful?

Last updated on 2026-06-15

Live transcribe audio from a microphone with Foundry Local

Prerequisites

Samples repository

Install packages

Live transcribe from microphone

Prerequisites

Samples repository

Install packages

Live transcribe from microphone

Prerequisites

Samples repository

Install packages

Live transcribe from microphone

Prerequisites

Samples repository

Install packages

Live transcribe from microphone

Related content

Feedback

Additional resources