> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/argmaxinc/WhisperKit/llms.txt
> Use this file to discover all available pages before exploring further.

# VoiceActivityDetector

> Base class for voice activity detection in audio streams

## Overview

The `VoiceActivityDetector` class is an abstract base class for implementing Voice Activity Detection (VAD). VAD is used to identify segments of audio that contain human speech versus silence or background noise. WhisperKit uses VAD to intelligently chunk long audio files for more efficient transcription.

## Class Definition

```swift theme={null}
open class VoiceActivityDetector
```

## Initializer

```swift theme={null}
public init(
    sampleRate: Int = 16000,
    frameLengthSamples: Int,
    frameOverlapSamples: Int = 0
)
```

<ParamField path="sampleRate" type="Int" default="16000">
  Sample rate of the audio signal in Hz
</ParamField>

<ParamField path="frameLengthSamples" type="Int">
  Length of each analysis frame in samples
</ParamField>

<ParamField path="frameOverlapSamples" type="Int" default="0">
  Number of samples overlapping between consecutive frames
</ParamField>

## Properties

<ResponseField name="sampleRate" type="Int">
  The sample rate of the audio signal in samples per second (typically 16000 Hz for WhisperKit)
</ResponseField>

<ResponseField name="frameLengthSamples" type="Int">
  The length of each analysis frame in samples
</ResponseField>

<ResponseField name="frameOverlapSamples" type="Int">
  The number of samples overlapping between consecutive frames. Useful for catching speech at frame boundaries.
</ResponseField>

## Methods

### voiceActivity(in:)

Analyzes audio waveform to detect voice activity. **Must be implemented by subclasses.**

```swift theme={null}
open func voiceActivity(in waveform: [Float]) -> [Bool]
```

<ParamField path="waveform" type="[Float]">
  Array of audio samples to analyze
</ParamField>

<ResponseField name="return" type="[Bool]">
  Array where `true` indicates voice activity and `false` indicates silence. Each element corresponds to one frame.
</ResponseField>

### voiceActivityAsync(in:)

Async version of voice activity detection.

```swift theme={null}
open func voiceActivityAsync(in waveform: [Float]) async throws -> [Bool]
```

<ParamField path="waveform" type="[Float]">
  Array of audio samples to analyze
</ParamField>

<ResponseField name="return" type="[Bool]">
  Array indicating voice activity per frame
</ResponseField>

### calculateActiveChunks(in:)

Identifies continuous segments of audio containing voice activity.

```swift theme={null}
public func calculateActiveChunks(
    in waveform: [Float]
) -> [(startIndex: Int, endIndex: Int)]
```

<ParamField path="waveform" type="[Float]">
  Array of audio samples
</ParamField>

<ResponseField name="return" type="[(startIndex: Int, endIndex: Int)]">
  Array of tuples containing start and end sample indices for each active segment
</ResponseField>

### voiceActivityIndexToAudioSampleIndex(\_:)

Converts a voice activity frame index to an audio sample index.

```swift theme={null}
public func voiceActivityIndexToAudioSampleIndex(_ index: Int) -> Int
```

<ParamField path="index" type="Int">
  Voice activity frame index
</ParamField>

<ResponseField name="return" type="Int">
  Corresponding audio sample index
</ResponseField>

### voiceActivityIndexToSeconds(\_:)

Converts a voice activity frame index to time in seconds.

```swift theme={null}
public func voiceActivityIndexToSeconds(_ index: Int) -> Float
```

<ParamField path="index" type="Int">
  Voice activity frame index
</ParamField>

<ResponseField name="return" type="Float">
  Corresponding time in seconds
</ResponseField>

### findLongestSilence(in:)

Finds the longest continuous period of silence.

```swift theme={null}
public func findLongestSilence(
    in vadResult: [Bool]
) -> (startIndex: Int, endIndex: Int)?
```

<ParamField path="vadResult" type="[Bool]">
  Voice activity detection results
</ParamField>

<ResponseField name="return" type="(startIndex: Int, endIndex: Int)?">
  Start and end indices of the longest silence, or nil if no silence found
</ResponseField>

### voiceActivityClipTimestamps(in:)

Generates timestamp pairs for active audio segments.

```swift theme={null}
func voiceActivityClipTimestamps(in waveform: [Float]) -> [Float]
```

<ResponseField name="return" type="[Float]">
  Flat array of timestamps: \[start1, end1, start2, end2, ...]
</ResponseField>

### calculateNonSilentSeekClips(in:)

Calculates seek positions for non-silent segments.

```swift theme={null}
func calculateNonSilentSeekClips(
    in waveform: [Float]
) -> [(start: Int, end: Int)]
```

<ResponseField name="return" type="[(start: Int, end: Int)]">
  Array of seek clip start/end positions in samples
</ResponseField>

### calculateSeekTimestamps(in:)

Calculates timestamp pairs for seeking through active segments.

```swift theme={null}
func calculateSeekTimestamps(
    in waveform: [Float]
) -> [(startTime: Float, endTime: Float)]
```

<ResponseField name="return" type="[(startTime: Float, endTime: Float)]">
  Array of start/end timestamp pairs in seconds
</ResponseField>

## Built-in Implementation: EnergyVAD

WhisperKit includes a built-in energy-based VAD implementation:

```swift theme={null}
let vad = EnergyVAD(
    sampleRate: 16000,
    frameLengthSamples: 1600,  // 100ms at 16kHz
    frameOverlapSamples: 160,   // 10ms overlap
    energyThreshold: 0.022
)
```

### How EnergyVAD Works

EnergyVAD detects voice activity by:

1. Dividing audio into frames
2. Calculating RMS energy for each frame
3. Comparing energy against a threshold
4. Frames above threshold are marked as containing voice

## Custom Implementation

To create a custom VAD, subclass `VoiceActivityDetector` and implement `voiceActivity(in:)`:

```swift theme={null}
class CustomVAD: VoiceActivityDetector {
    override func voiceActivity(in waveform: [Float]) -> [Bool] {
        let frameCount = waveform.count / frameLengthSamples
        var result = [Bool]()
        
        for i in 0..<frameCount {
            let start = i * frameLengthSamples
            let end = min(start + frameLengthSamples, waveform.count)
            let frame = Array(waveform[start..<end])
            
            // Implement your VAD logic here
            let hasVoice = detectVoiceInFrame(frame)
            result.append(hasVoice)
        }
        
        return result
    }
    
    private func detectVoiceInFrame(_ frame: [Float]) -> Bool {
        // Your custom voice detection logic
        return true
    }
}
```

## Example Usage

### Basic Voice Activity Detection

```swift theme={null}
import WhisperKit

// Load audio
let audioArray = try AudioProcessor.loadAudioAsFloatArray(
    fromPath: "/path/to/audio.wav"
)

// Create VAD instance
let vad = EnergyVAD(
    sampleRate: 16000,
    frameLengthSamples: 1600  // 100ms frames
)

// Detect voice activity
let vadResult = vad.voiceActivity(in: audioArray)
print("Voice activity: \(vadResult)")
```

### Find Active Segments

```swift theme={null}
let vad = EnergyVAD()
let audioArray = try AudioProcessor.loadAudioAsFloatArray(
    fromPath: "/path/to/audio.wav"
)

let activeChunks = vad.calculateActiveChunks(in: audioArray)

for (index, chunk) in activeChunks.enumerated() {
    let startTime = Float(chunk.startIndex) / 16000.0
    let endTime = Float(chunk.endIndex) / 16000.0
    let duration = endTime - startTime
    
    print("Segment \(index): \(startTime)s - \(endTime)s (\(duration)s)")
}
```

### Use VAD with WhisperKit

```swift theme={null}
// Configure WhisperKit with VAD-based chunking
let config = WhisperKitConfig(
    model: "base",
    voiceActivityDetector: EnergyVAD()
)

let whisperKit = try await WhisperKit(config)

// Transcribe with VAD chunking
let options = DecodingOptions(
    chunkingStrategy: .vad
)

let results = try await whisperKit.transcribe(
    audioPath: "/path/to/long_audio.wav",
    decodeOptions: options
)
```

### Visualize Voice Activity

```swift theme={null}
func visualizeVAD(audioArray: [Float], vad: VoiceActivityDetector) {
    let vadResult = vad.voiceActivity(in: audioArray)
    
    print("Voice Activity Visualization:")
    print("Time (s) | Activity")
    print("---------|----------")
    
    for (index, hasVoice) in vadResult.enumerated() {
        let time = vad.voiceActivityIndexToSeconds(index)
        let marker = hasVoice ? "█████████" : "         "
        print(String(format: "%6.2f   | %@", time, marker))
    }
}

let vad = EnergyVAD()
let audio = try AudioProcessor.loadAudioAsFloatArray(
    fromPath: "/path/to/audio.wav"
)

visualizeVAD(audioArray: audio, vad: vad)
```

### Find Silence for Split Points

```swift theme={null}
let vad = EnergyVAD()
let audio = try AudioProcessor.loadAudioAsFloatArray(
    fromPath: "/path/to/audio.wav"
)

let vadResult = vad.voiceActivity(in: audio)

if let silence = vad.findLongestSilence(in: vadResult) {
    let startTime = vad.voiceActivityIndexToSeconds(silence.startIndex)
    let endTime = vad.voiceActivityIndexToSeconds(silence.endIndex)
    
    print("Longest silence: \(startTime)s - \(endTime)s")
    print("Duration: \(endTime - startTime)s")
    
    // Use this to split audio at a natural break
}
```

### Custom Energy Threshold

```swift theme={null}
// More sensitive (detects quieter speech)
let sensitiveVAD = EnergyVAD(
    energyThreshold: 0.01
)

// Less sensitive (only loud speech)
let conservativeVAD = EnergyVAD(
    energyThreshold: 0.05
)

// Compare results
let audio = try AudioProcessor.loadAudioAsFloatArray(
    fromPath: "/path/to/audio.wav"
)

let sensitiveResult = sensitiveVAD.voiceActivity(in: audio)
let conservativeResult = conservativeVAD.voiceActivity(in: audio)

let sensitiveCount = sensitiveResult.filter { $0 }.count
let conservativeCount = conservativeResult.filter { $0 }.count

print("Sensitive VAD: \(sensitiveCount) active frames")
print("Conservative VAD: \(conservativeCount) active frames")
```

### Process Live Audio with VAD

```swift theme={null}
let processor = AudioProcessor()
let vad = EnergyVAD()

try processor.startRecordingLive { audioBuffer in
    // Accumulate enough samples for VAD analysis
    let minSamples = vad.frameLengthSamples * 10  // 10 frames
    
    if processor.audioSamples.count >= minSamples {
        let samples = Array(processor.audioSamples.suffix(minSamples))
        let vadResult = vad.voiceActivity(in: samples)
        
        let voiceFrames = vadResult.filter { $0 }.count
        let totalFrames = vadResult.count
        let voiceRatio = Float(voiceFrames) / Float(totalFrames)
        
        if voiceRatio > 0.5 {
            print("Speech detected (\(Int(voiceRatio * 100))% active)")
        } else {
            print("Mostly silence (\(Int(voiceRatio * 100))% active)")
        }
    }
}
```

## Best Practices

1. **Frame Length Selection**: Use 20-100ms frames (320-1600 samples at 16kHz). Shorter frames = more responsive but noisier.

2. **Threshold Tuning**: Adjust energy threshold based on your audio:
   * Clean recordings: 0.01-0.02
   * Noisy environments: 0.03-0.05
   * Test with representative samples

3. **Overlap for Continuity**: Use frame overlap (10-20ms) to avoid missing speech at frame boundaries.

4. **Post-Processing**: Apply smoothing to reduce false positives:
   ```swift theme={null}
   // Remove isolated voice frames
   func smoothVAD(_ vad: [Bool], minLength: Int = 3) -> [Bool] {
       var smoothed = vad
       for i in 1..<(vad.count - 1) {
           if vad[i] && !vad[i-1] && !vad[i+1] {
               smoothed[i] = false  // Remove single frame
           }
       }
       return smoothed
   }
   ```

5. **Chunking Strategy**: When using `.vad` chunking strategy with WhisperKit, ensure your VAD is tuned to avoid creating too many tiny segments.