> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/argmaxinc/WhisperKit/llms.txt
> Use this file to discover all available pages before exploring further.

# Streaming Transcription

> Real-time audio transcription from microphone input

# Streaming Transcription

WhisperKit supports real-time streaming transcription using the `AudioStreamTranscriber` class. This enables live transcription from the microphone with automatic voice activity detection and segment confirmation.

## AudioStreamTranscriber

The `AudioStreamTranscriber` actor manages the complete streaming pipeline:

* Captures live audio from the microphone
* Detects voice activity
* Transcribes audio in real-time
* Manages segment confirmation and state updates

See [AudioStreamTranscriber](~/workspace/source/Sources/WhisperKit/Core/Audio/AudioStreamTranscriber.swift:23)

## Basic Setup

### Initialize Streaming Transcriber

```swift theme={null}
import WhisperKit

let whisperKit = try await WhisperKit()

let streamTranscriber = AudioStreamTranscriber(
    audioEncoder: whisperKit.audioEncoder,
    featureExtractor: whisperKit.featureExtractor,
    segmentSeeker: whisperKit.segmentSeeker,
    textDecoder: whisperKit.textDecoder,
    tokenizer: whisperKit.tokenizer!,
    audioProcessor: whisperKit.audioProcessor,
    decodingOptions: DecodingOptions(),
    stateChangeCallback: { oldState, newState in
        print("Text: \(newState.currentText)")
        print("Confirmed segments: \(newState.confirmedSegments.count)")
    }
)
```

See [AudioStreamTranscriber.init](~/workspace/source/Sources/WhisperKit/Core/Audio/AudioStreamTranscriber.swift:40-71)

### Start and Stop Streaming

```swift theme={null}
// Start transcription
try await streamTranscriber.startStreamTranscription()

// Transcription runs continuously...

// Stop transcription
await streamTranscriber.stopStreamTranscription()
```

See [AudioStreamTranscriber.swift:73-93](~/workspace/source/Sources/WhisperKit/Core/Audio/AudioStreamTranscriber.swift:73-93)

## State Management

The `AudioStreamTranscriber.State` tracks the current transcription state:

```swift theme={null}
public struct State {
    var isRecording: Bool
    var currentFallbacks: Int
    var lastBufferSize: Int
    var lastConfirmedSegmentEndSeconds: Float
    var bufferEnergy: [Float]
    var currentText: String
    var confirmedSegments: [TranscriptionSegment]
    var unconfirmedSegments: [TranscriptionSegment]
    var unconfirmedText: [String]
}
```

See [AudioStreamTranscriber.State](~/workspace/source/Sources/WhisperKit/Core/Audio/AudioStreamTranscriber.swift:7-18)

### State Properties

<ParamField path="isRecording" type="Bool">
  Whether audio is currently being recorded and transcribed.
</ParamField>

<ParamField path="currentText" type="String">
  The most recent transcription text (may be unconfirmed).
</ParamField>

<ParamField path="confirmedSegments" type="[TranscriptionSegment]">
  Segments that have been confirmed and are unlikely to change.
</ParamField>

<ParamField path="unconfirmedSegments" type="[TranscriptionSegment]">
  Segments that may still be refined as more audio is processed.
</ParamField>

<ParamField path="bufferEnergy" type="[Float]">
  Audio energy levels for voice activity detection.
</ParamField>

## Configuration Options

Customize the streaming behavior with initialization parameters:

```swift theme={null}
let streamTranscriber = AudioStreamTranscriber(
    audioEncoder: whisperKit.audioEncoder,
    featureExtractor: whisperKit.featureExtractor,
    segmentSeeker: whisperKit.segmentSeeker,
    textDecoder: whisperKit.textDecoder,
    tokenizer: whisperKit.tokenizer!,
    audioProcessor: whisperKit.audioProcessor,
    decodingOptions: DecodingOptions(
        language: "en",
        wordTimestamps: true
    ),
    requiredSegmentsForConfirmation: 2,  // Segments needed before confirming
    silenceThreshold: 0.3,               // VAD silence threshold
    compressionCheckWindow: 60,          // Token window for hallucination check
    useVAD: true,                        // Enable voice activity detection
    stateChangeCallback: { oldState, newState in
        handleStateChange(newState)
    }
)
```

See [AudioStreamTranscriber.init](~/workspace/source/Sources/WhisperKit/Core/Audio/AudioStreamTranscriber.swift:40-71)

### Configuration Parameters

<ParamField path="requiredSegmentsForConfirmation" type="Int" default="2">
  Number of segments that must be decoded before earlier segments are confirmed. Higher values provide more stability but increase latency.
</ParamField>

<ParamField path="silenceThreshold" type="Float" default="0.3">
  Energy threshold for voice activity detection. Lower values are more sensitive to quiet speech.
</ParamField>

<ParamField path="compressionCheckWindow" type="Int" default="60">
  Number of tokens to check for repetition/hallucination. Helps detect when the model is producing gibberish.
</ParamField>

<ParamField path="useVAD" type="Bool" default="true">
  Enable voice activity detection to skip silent segments. Improves performance and accuracy.
</ParamField>

## State Change Callback

The callback receives both old and new states for comparison:

```swift theme={null}
let transcriber = AudioStreamTranscriber(
    // ... other parameters
    stateChangeCallback: { oldState, newState in
        // Check if new segments were confirmed
        if newState.confirmedSegments.count > oldState.confirmedSegments.count {
            let newSegments = newState.confirmedSegments.suffix(
                newState.confirmedSegments.count - oldState.confirmedSegments.count
            )
            
            for segment in newSegments {
                print("Confirmed: \(segment.text)")
                saveToDatabase(segment)
            }
        }
        
        // Update UI with current text (confirmed + unconfirmed)
        let fullText = newState.confirmedSegments.map { $0.text }.joined() +
                      newState.unconfirmedSegments.map { $0.text }.joined()
        updateUI(fullText)
    }
)
```

See [AudioStreamTranscriberCallback](~/workspace/source/Sources/WhisperKit/Core/Audio/AudioStreamTranscriber.swift:20)

## Segment Confirmation

The transcriber uses a sliding window approach:

1. Audio is continuously buffered and transcribed
2. New segments are added to `unconfirmedSegments`
3. When segment count exceeds `requiredSegmentsForConfirmation`, earlier segments move to `confirmedSegments`
4. Confirmed segments are unlikely to change as more audio is processed

```swift theme={null}
// Example with requiredSegmentsForConfirmation = 2
// 
// Initial state: []
// After 1st transcription: unconfirmed = [seg1]
// After 2nd transcription: unconfirmed = [seg1, seg2]
// After 3rd transcription: confirmed = [seg1], unconfirmed = [seg2, seg3]
// After 4th transcription: confirmed = [seg1, seg2], unconfirmed = [seg3, seg4]
```

See [AudioStreamTranscriber.transcribeCurrentBuffer](~/workspace/source/Sources/WhisperKit/Core/Audio/AudioStreamTranscriber.swift:123-190)

## Voice Activity Detection

When `useVAD` is enabled, the transcriber skips silent segments:

```swift theme={null}
// VAD checks relative energy levels
let voiceDetected = AudioProcessor.isVoiceDetected(
    in: audioProcessor.relativeEnergy,
    nextBufferInSeconds: nextBufferSeconds,
    silenceThreshold: silenceThreshold  // 0.3 by default
)

if !voiceDetected {
    // Skip transcription for this buffer
    return
}
```

See [AudioStreamTranscriber.transcribeCurrentBuffer](~/workspace/source/Sources/WhisperKit/Core/Audio/AudioStreamTranscriber.swift:139-154)

### Benefits of VAD

* Reduces unnecessary computation during silence
* Improves transcription accuracy by avoiding false positives
* Lowers battery consumption
* Reduces hallucinations from background noise

## Early Stopping

The transcriber implements early stopping to prevent hallucinations:

```swift theme={null}
private static func shouldStopEarly(
    progress: TranscriptionProgress,
    options: DecodingOptions,
    compressionCheckWindow: Int
) -> Bool? {
    // Check for high compression ratio (repetition)
    if currentTokens.count > compressionCheckWindow {
        let checkTokens = currentTokens.suffix(compressionCheckWindow)
        let compressionRatio = TextUtilities.compressionRatio(of: checkTokens)
        if compressionRatio > options.compressionRatioThreshold ?? 0.0 {
            return false  // Stop early
        }
    }
    
    // Check for low log probability (low confidence)
    if let avgLogprob = progress.avgLogprob,
       let threshold = options.logProbThreshold {
        if avgLogprob < threshold {
            return false  // Stop early
        }
    }
    
    return nil  // Continue
}
```

See [AudioStreamTranscriber.shouldStopEarly](~/workspace/source/Sources/WhisperKit/Core/Audio/AudioStreamTranscriber.swift:204-223)

## Complete Example

```swift theme={null}
import SwiftUI
import WhisperKit

class StreamingTranscriptionViewModel: ObservableObject {
    @Published var currentText = ""
    @Published var confirmedText = ""
    @Published var isRecording = false
    
    private var whisperKit: WhisperKit?
    private var streamTranscriber: AudioStreamTranscriber?
    
    func setup() async {
        do {
            whisperKit = try await WhisperKit()
            
            streamTranscriber = AudioStreamTranscriber(
                audioEncoder: whisperKit!.audioEncoder,
                featureExtractor: whisperKit!.featureExtractor,
                segmentSeeker: whisperKit!.segmentSeeker,
                textDecoder: whisperKit!.textDecoder,
                tokenizer: whisperKit!.tokenizer!,
                audioProcessor: whisperKit!.audioProcessor,
                decodingOptions: DecodingOptions(
                    language: "en",
                    task: .transcribe,
                    wordTimestamps: true
                ),
                requiredSegmentsForConfirmation: 3,
                useVAD: true,
                stateChangeCallback: { [weak self] _, newState in
                    Task { @MainActor in
                        self?.isRecording = newState.isRecording
                        self?.currentText = newState.currentText
                        self?.confirmedText = newState.confirmedSegments
                            .map { $0.text }
                            .joined()
                    }
                }
            )
        } catch {
            print("Failed to initialize: \(error)")
        }
    }
    
    func startRecording() async {
        do {
            try await streamTranscriber?.startStreamTranscription()
        } catch {
            print("Failed to start: \(error)")
        }
    }
    
    func stopRecording() async {
        await streamTranscriber?.stopStreamTranscription()
    }
}

struct StreamingView: View {
    @StateObject private var viewModel = StreamingTranscriptionViewModel()
    
    var body: some View {
        VStack {
            Text("Confirmed:")
                .font(.caption)
            Text(viewModel.confirmedText)
                .padding()
                .background(Color.green.opacity(0.1))
            
            Text("Current:")
                .font(.caption)
            Text(viewModel.currentText)
                .padding()
                .background(Color.yellow.opacity(0.1))
            
            Button(viewModel.isRecording ? "Stop" : "Start") {
                Task {
                    if viewModel.isRecording {
                        await viewModel.stopRecording()
                    } else {
                        await viewModel.startRecording()
                    }
                }
            }
            .buttonStyle(.borderedProminent)
        }
        .task {
            await viewModel.setup()
        }
    }
}
```

## Permissions

Streaming transcription requires microphone access:

```xml theme={null}
<!-- Info.plist -->
<key>NSMicrophoneUsageDescription</key>
<string>We need microphone access to transcribe your speech</string>
```

The transcriber automatically requests permission:

```swift theme={null}
guard await AudioProcessor.requestRecordPermission() else {
    print("Microphone access denied")
    return
}
```

See [AudioStreamTranscriber.startStreamTranscription](~/workspace/source/Sources/WhisperKit/Core/Audio/AudioStreamTranscriber.swift:75-78)

## Performance Considerations

<CardGroup cols={2}>
  <Card title="Model Size" icon="layer-group">
    Use smaller models (tiny, base) for real-time streaming. Larger models may not keep up with live audio.
  </Card>

  <Card title="Buffer Management" icon="memory">
    The transcriber maintains an audio buffer. Long recordings consume more memory.
  </Card>

  <Card title="VAD Optimization" icon="gauge-high">
    Enable VAD to skip silent portions and reduce computation.
  </Card>

  <Card title="Confirmation Latency" icon="clock">
    Higher `requiredSegmentsForConfirmation` improves stability but increases latency before segments are confirmed.
  </Card>
</CardGroup>

## Next Steps

<CardGroup cols={2}>
  <Card title="Voice Activity Detection" icon="microphone" href="/whisperkit/voice-activity-detection">
    Deep dive into VAD configuration
  </Card>

  <Card title="Configuration" icon="gear" href="/whisperkit/configuration">
    Advanced configuration options
  </Card>
</CardGroup>