> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/argmaxinc/WhisperKit/llms.txt
> Use this file to discover all available pages before exploring further.

# AudioProcessor

> Audio processing and recording functionality for WhisperKit

## Overview

The `AudioProcessor` class handles audio loading, conversion, preprocessing, and live recording for WhisperKit. It provides utilities for converting audio to the required format (16kHz mono) and streaming audio from input devices.

## Class Definition

```swift theme={null}
open class AudioProcessor: NSObject, AudioProcessing
```

## Initializer

The `AudioProcessor` uses the default initializer:

```swift theme={null}
let processor = AudioProcessor()
```

## Properties

### Audio State

<ResponseField name="audioSamples" type="ContiguousArray<Float>">
  Current buffer of recorded audio samples (16kHz mono)
</ResponseField>

<ResponseField name="audioEngine" type="AVAudioEngine?">
  The audio engine used for live recording
</ResponseField>

<ResponseField name="audioEnergy" type="[(rel: Float, avg: Float, max: Float, min: Float)]">
  Array of energy measurements for each audio buffer
</ResponseField>

<ResponseField name="relativeEnergy" type="[Float]">
  Array of relative energy values (0-1) normalized to the quietest buffer
</ResponseField>

### Configuration

<ResponseField name="relativeEnergyWindow" type="Int" default="20">
  Number of past buffers to use for calculating relative energy baseline (\~2 seconds at 100ms buffers)
</ResponseField>

<ResponseField name="minBufferLength" type="Int">
  Minimum buffer length in samples (default: 0.1 seconds = 1600 samples at 16kHz)
</ResponseField>

<ResponseField name="audioBufferCallback" type="(([Float]) -> Void)?">
  Callback invoked when a new audio buffer is available
</ResponseField>

<ResponseField name="isInputSuppressed" type="Bool">
  When true, replaces input buffers with silence while maintaining timing
</ResponseField>

## Static Methods

### Loading Audio

#### loadAudio(fromPath:channelMode:startTime:endTime:maxReadFrameSize:)

Loads audio from a file path and converts it to the required format.

```swift theme={null}
public static func loadAudio(
    fromPath audioFilePath: String,
    channelMode: ChannelMode = .sumChannels(nil),
    startTime: Double? = 0,
    endTime: Double? = nil,
    maxReadFrameSize: AVAudioFrameCount? = nil
) throws -> AVAudioPCMBuffer
```

<ParamField path="audioFilePath" type="String">
  Path to the audio file
</ParamField>

<ParamField path="channelMode" type="ChannelMode" default=".sumChannels(nil)">
  How to handle multi-channel audio:

  * `.sumChannels(nil)` - Mix all channels with peak normalization
  * `.sumChannels([0, 1])` - Mix specific channels
  * `.specificChannel(0)` - Use only one channel
</ParamField>

<ParamField path="startTime" type="Double?" default="0">
  Start time in seconds to read from
</ParamField>

<ParamField path="endTime" type="Double?">
  End time in seconds to read until (nil = end of file)
</ParamField>

<ParamField path="maxReadFrameSize" type="AVAudioFrameCount?">
  Maximum frames to read at once (for memory management)
</ParamField>

<ResponseField name="return" type="AVAudioPCMBuffer">
  Audio buffer in 16kHz mono float32 format
</ResponseField>

#### loadAudioAsFloatArray(fromPath:channelMode:startTime:endTime:)

Loads audio and returns it as a float array.

```swift theme={null}
public static func loadAudioAsFloatArray(
    fromPath audioFilePath: String,
    channelMode: ChannelMode = .sumChannels(nil),
    startTime: Double? = 0,
    endTime: Double? = nil
) throws -> [Float]
```

<ResponseField name="return" type="[Float]">
  Array of 16kHz mono audio samples
</ResponseField>

#### loadAudio(at:channelMode:)

Loads multiple audio files concurrently.

```swift theme={null}
public static func loadAudio(
    at audioPaths: [String],
    channelMode: ChannelMode = .sumChannels(nil)
) async -> [Result<[Float], Swift.Error>]
```

<ParamField path="audioPaths" type="[String]">
  Array of audio file paths
</ParamField>

<ResponseField name="return" type="[Result<[Float], Error>]">
  Array of results, one per input file
</ResponseField>

### Resampling

#### resampleAudio(fromFile:toSampleRate:channelCount:channelMode:frameCount:maxReadFrameSize:)

Resamples audio from a file.

```swift theme={null}
public static func resampleAudio(
    fromFile audioFile: AVAudioFile,
    toSampleRate sampleRate: Double,
    channelCount: AVAudioChannelCount,
    channelMode: ChannelMode = .sumChannels(nil),
    frameCount: AVAudioFrameCount? = nil,
    maxReadFrameSize: AVAudioFrameCount = Constants.defaultAudioReadFrameSize
) -> AVAudioPCMBuffer?
```

<ParamField path="audioFile" type="AVAudioFile">
  Input audio file
</ParamField>

<ParamField path="sampleRate" type="Double">
  Target sample rate (typically 16000)
</ParamField>

<ParamField path="channelCount" type="AVAudioChannelCount">
  Target channel count (typically 1 for mono)
</ParamField>

<ResponseField name="return" type="AVAudioPCMBuffer?">
  Resampled audio buffer
</ResponseField>

#### resampleAudio(fromBuffer:toSampleRate:channelCount:)

Resamples an audio buffer.

```swift theme={null}
public static func resampleAudio(
    fromBuffer inputBuffer: AVAudioPCMBuffer,
    toSampleRate sampleRate: Double,
    channelCount: AVAudioChannelCount
) -> AVAudioPCMBuffer?
```

### Channel Conversion

#### convertToMono(\_:mode:)

Converts multi-channel audio to mono.

```swift theme={null}
public static func convertToMono(
    _ buffer: AVAudioPCMBuffer,
    mode: ChannelMode
) -> AVAudioPCMBuffer?
```

<ParamField path="buffer" type="AVAudioPCMBuffer">
  Input audio buffer (possibly multi-channel)
</ParamField>

<ParamField path="mode" type="ChannelMode">
  How to convert to mono
</ParamField>

<ResponseField name="return" type="AVAudioPCMBuffer?">
  Mono audio buffer
</ResponseField>

### Energy and Voice Activity

#### calculateAverageEnergy(of:)

Calculates RMS energy of an audio signal.

```swift theme={null}
public static func calculateAverageEnergy(of signal: [Float]) -> Float
```

<ResponseField name="return" type="Float">
  RMS energy value
</ResponseField>

#### calculateEnergy(of:)

Calculates detailed energy metrics.

```swift theme={null}
public static func calculateEnergy(
    of signal: [Float]
) -> (avg: Float, max: Float, min: Float)
```

<ResponseField name="return" type="(avg: Float, max: Float, min: Float)">
  Tuple containing average (RMS), maximum, and minimum energy values
</ResponseField>

#### calculateRelativeEnergy(of:relativeTo:)

Calculates energy relative to a reference baseline.

```swift theme={null}
public static func calculateRelativeEnergy(
    of signal: [Float],
    relativeTo reference: Float?
) -> Float
```

<ParamField path="signal" type="[Float]">
  Audio signal to analyze
</ParamField>

<ParamField path="reference" type="Float?">
  Reference energy level (typically the minimum energy in recent buffers)
</ParamField>

<ResponseField name="return" type="Float">
  Normalized energy value from 0 to 1
</ResponseField>

#### isVoiceDetected(in:nextBufferInSeconds:silenceThreshold:)

Detects if voice is present in audio.

```swift theme={null}
public static func isVoiceDetected(
    in relativeEnergy: [Float],
    nextBufferInSeconds: Float,
    silenceThreshold: Float
) -> Bool
```

<ResponseField name="return" type="Bool">
  True if voice is detected above the threshold
</ResponseField>

#### calculateNonSilentChunks(in:)

Identifies non-silent segments of audio.

```swift theme={null}
public static func calculateNonSilentChunks(
    in signal: [Float]
) -> [(startIndex: Int, endIndex: Int)]
```

<ResponseField name="return" type="[(startIndex: Int, endIndex: Int)]">
  Array of start/end index pairs for non-silent segments
</ResponseField>

#### calculateVoiceActivityInChunks(of:chunkCount:frameLengthSamples:frameOverlapSamples:energyThreshold:)

Calculates voice activity for audio chunks.

```swift theme={null}
public static func calculateVoiceActivityInChunks(
    of signal: [Float],
    chunkCount: Int,
    frameLengthSamples: Int,
    frameOverlapSamples: Int = 0,
    energyThreshold: Float = 0.022
) -> [Bool]
```

<ParamField path="energyThreshold" type="Float" default="0.022">
  Energy threshold for detecting speech
</ParamField>

<ResponseField name="return" type="[Bool]">
  Array indicating voice activity for each chunk
</ResponseField>

### Utility Methods

#### padOrTrimAudio(fromArray:startAt:toLength:saveSegment:)

Pads or trims audio to a specific length.

```swift theme={null}
public static func padOrTrimAudio(
    fromArray audioArray: [Float],
    startAt startIndex: Int = 0,
    toLength frameLength: Int = 480_000,
    saveSegment: Bool = false
) -> MLMultiArray?
```

<ParamField path="audioArray" type="[Float]">
  Input audio samples
</ParamField>

<ParamField path="startIndex" type="Int" default="0">
  Starting index in the array
</ParamField>

<ParamField path="frameLength" type="Int" default="480000">
  Target length in samples (default is 30 seconds at 16kHz)
</ParamField>

<ResponseField name="return" type="MLMultiArray?">
  Padded/trimmed audio as MLMultiArray for Core ML
</ResponseField>

#### convertBufferToArray(buffer:chunkSize:)

Converts AVAudioPCMBuffer to float array.

```swift theme={null}
public static func convertBufferToArray(
    buffer: AVAudioPCMBuffer,
    chunkSize: Int = 1024
) -> [Float]
```

<ResponseField name="return" type="[Float]">
  Array of audio samples
</ResponseField>

#### requestRecordPermission()

Requests microphone permission from the user.

```swift theme={null}
public static func requestRecordPermission() async -> Bool
```

<ResponseField name="return" type="Bool">
  True if permission granted
</ResponseField>

#### getAudioDevices() (macOS only)

Returns list of available audio input devices.

```swift theme={null}
public static func getAudioDevices() -> [AudioDevice]
```

<ResponseField name="return" type="[AudioDevice]">
  Array of available audio input devices
</ResponseField>

## Instance Methods

### Recording Control

#### startRecordingLive(inputDeviceID:callback:)

Starts recording audio from an input device.

```swift theme={null}
public func startRecordingLive(
    inputDeviceID: DeviceID? = nil,
    callback: (([Float]) -> Void)? = nil
) throws
```

<ParamField path="inputDeviceID" type="DeviceID?">
  Device ID to record from (nil = default device). Only used on macOS.
</ParamField>

<ParamField path="callback" type="(([Float]) -> Void)?">
  Callback invoked for each audio buffer
</ParamField>

#### startStreamingRecordingLive(inputDeviceID:)

Starts recording and returns an async stream.

```swift theme={null}
public func startStreamingRecordingLive(
    inputDeviceID: DeviceID? = nil
) -> (AsyncThrowingStream<[Float], Error>, AsyncThrowingStream<[Float], Error>.Continuation)
```

<ResponseField name="return" type="(stream: AsyncThrowingStream, continuation: Continuation)">
  Tuple containing the audio stream and its continuation for cancellation
</ResponseField>

#### pauseRecording()

Pauses recording (can be resumed).

```swift theme={null}
public func pauseRecording()
```

#### resumeRecordingLive(inputDeviceID:callback:)

Resumes recording after pause.

```swift theme={null}
public func resumeRecordingLive(
    inputDeviceID: DeviceID? = nil,
    callback: (([Float]) -> Void)? = nil
) throws
```

#### stopRecording()

Stops recording and releases resources.

```swift theme={null}
public func stopRecording()
```

### Audio Buffer Management

#### purgeAudioSamples(keepingLast:)

Removes old audio samples, keeping only recent ones.

```swift theme={null}
public func purgeAudioSamples(keepingLast keep: Int)
```

<ParamField path="keep" type="Int">
  Number of samples to keep
</ParamField>

#### setInputSuppressed(\_:)

Enables or disables input suppression (silent buffers).

```swift theme={null}
public func setInputSuppressed(_ isSuppressed: Bool)
```

<ParamField path="isSuppressed" type="Bool">
  If true, replaces input with silence
</ParamField>

#### padOrTrim(fromArray:startAt:toLength:)

Instance method for padding/trimming audio.

```swift theme={null}
open func padOrTrim(
    fromArray audioArray: [Float],
    startAt startIndex: Int,
    toLength frameLength: Int
) -> (any AudioProcessorOutputType)?
```

## Example Usage

### Load and Convert Audio

```swift theme={null}
import WhisperKit

// Load audio from file
let audioBuffer = try AudioProcessor.loadAudio(
    fromPath: "/path/to/audio.mp3",
    channelMode: .sumChannels(nil)
)

// Convert to float array
let audioArray = AudioProcessor.convertBufferToArray(buffer: audioBuffer)
print("Loaded \(audioArray.count) samples")
```

### Live Recording with Callback

```swift theme={null}
let processor = AudioProcessor()

// Request permission first
let granted = await AudioProcessor.requestRecordPermission()
guard granted else {
    print("Microphone permission denied")
    return
}

// Start recording
try processor.startRecordingLive { audioBuffer in
    print("Received buffer: \(audioBuffer.count) samples")
    // Process audio buffer
}

// Stop when done
processor.stopRecording()
```

### Streaming with AsyncStream

```swift theme={null}
let processor = AudioProcessor()

let (stream, continuation) = processor.startStreamingRecordingLive()

Task {
    do {
        for try await audioBuffer in stream {
            print("Stream received: \(audioBuffer.count) samples")
            // Process audio
        }
    } catch {
        print("Stream error: \(error)")
    }
}

// Cancel when done
continuation.finish()
```

### Channel Selection

```swift theme={null}
// Use only left channel
let leftChannel = try AudioProcessor.loadAudioAsFloatArray(
    fromPath: "/path/to/stereo.wav",
    channelMode: .specificChannel(0)
)

// Mix only specific channels
let mixed = try AudioProcessor.loadAudioAsFloatArray(
    fromPath: "/path/to/multi.wav",
    channelMode: .sumChannels([0, 2])  // Mix channels 0 and 2
)
```

### Energy-Based Voice Detection

```swift theme={null}
let processor = AudioProcessor()

try processor.startRecordingLive { buffer in
    let energy = AudioProcessor.calculateAverageEnergy(of: buffer)
    print("Energy: \(energy)")
    
    // Check if voice is present
    let hasVoice = AudioProcessor.isVoiceDetected(
        in: processor.relativeEnergy,
        nextBufferInSeconds: 0.1,
        silenceThreshold: 0.3
    )
    
    if hasVoice {
        print("Voice detected!")
    }
}
```

### Batch Loading

```swift theme={null}
let paths = [
    "/path/to/audio1.wav",
    "/path/to/audio2.mp3",
    "/path/to/audio3.m4a"
]

let results = await AudioProcessor.loadAudio(at: paths)

for (index, result) in results.enumerated() {
    switch result {
    case .success(let audioArray):
        print("File \(index): \(audioArray.count) samples")
    case .failure(let error):
        print("File \(index) failed: \(error)")
    }
}
```

### Segment Audio by Silence

```swift theme={null}
let audioArray = try AudioProcessor.loadAudioAsFloatArray(
    fromPath: "/path/to/audio.wav"
)

let nonSilentChunks = AudioProcessor.calculateNonSilentChunks(in: audioArray)

for (index, chunk) in nonSilentChunks.enumerated() {
    let startTime = Float(chunk.startIndex) / 16000.0
    let endTime = Float(chunk.endIndex) / 16000.0
    print("Segment \(index): \(startTime)s - \(endTime)s")
}
```

### Get Available Devices (macOS)

```swift theme={null}
#if os(macOS)
let devices = AudioProcessor.getAudioDevices()

for device in devices {
    print("Device: \(device.name) (ID: \(device.id))")
}

// Record from specific device
if let device = devices.first {
    try processor.startRecordingLive(inputDeviceID: device.id)
}
#endif
```