> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/argmaxinc/WhisperKit/llms.txt
> Use this file to discover all available pages before exploring further.

# Benchmarks

> Run performance benchmarks for WhisperKit models across different devices

## Overview

WhisperKit provides a comprehensive benchmarking suite to evaluate model performance across different Apple devices. Results can be uploaded to the [argmaxinc/whisperkit-evals-dataset](https://huggingface.co/datasets/argmaxinc/whisperkit-evals-dataset) on HuggingFace and viewed in the [WhisperKit Benchmarks](https://huggingface.co/spaces/argmaxinc/whisperkit-benchmarks) space.

## Prerequisites

<Warning>
  An active Apple developer account is required to run tests on physical devices.
</Warning>

Before running benchmarks:

* All external devices must be connected and paired to your Mac
* Devices must be registered with your developer account
* Devices must be in Developer Mode

<Tip>
  Press `Command + Shift + 2` in Xcode to open the devices list and track their connection status.
</Tip>

## Setup

### Download the Source

Clone the WhisperKit repository:

```bash theme={null}
git clone git@github.com:argmaxinc/WhisperKit.git
cd WhisperKit
```

### Install Dependencies

Set up your local environment with necessary dependencies:

```bash theme={null}
make setup
```

See the [Contributing Guide](/resources/contributing) for more information.

## Configuration

### Xcode Environment Variables

The model to test is provided to Xcode from Fastlane as an environment variable:

1. Open the example project:

```bash theme={null}
xed Examples/WhisperAX
```

2. Click on `WhisperAX` at the top and select `Edit Scheme`

3. Under `Environment Variables`, you'll see `MODEL_NAME` with value `$(MODEL_NAME)`

### Datasets

Configure test datasets in the global `datasets` array in:

```
Tests/WhisperKitTests/RegressionTests.swift
```

The file is prefilled with currently available datasets.

### Models

Configure test models in the `Fastfile`:

```ruby theme={null}
# Find BENCHMARK_CONFIGS and modify the models array
BENCHMARK_CONFIGS = {
  'full' => {
    models: ['tiny', 'base', 'small', 'medium', 'large-v3'],
    # ...
  },
  'debug' => {
    models: ['tiny'],
    # ...
  }
}
```

Location: `fastlane/Fastfile`

## Running Benchmarks

### List Connected Devices

Verify device connections before running tests:

```bash theme={null}
make list-devices
```

Expected output:

```ruby theme={null}
{
   :name=>"My Mac", 
   :type=>"Apple M2 Pro", 
   :platform=>"macOS", 
   :os_version=>"15.0.1", 
   :product=>"Mac14,12", 
   :id=>"XXXXXXXX-1234-5678-9012-XXXXXXXXXXXX", 
   :state=>"connected"
}
```

Verify the `:state` field shows `"connected"`.

### Debug Tests

Run quick debug tests to check for potential errors:

```bash theme={null}
make benchmark-devices DEBUG=true
```

### Full Benchmark Suite

Run the complete benchmark suite:

```bash theme={null}
make benchmark-devices
```

### Specify Target Devices

Run benchmarks on specific devices:

```bash theme={null}
make benchmark-devices DEVICES="iPhone 15 Pro Max,My Mac"
```

The `DEVICES` parameter accepts a comma-separated list of device names from `make list-devices`.

## Results

### Output Location

After tests complete, results are saved in:

* **Full results**: `fastlane/benchmark_data/` - includes `.xcresult` files with logs and attachments
* **JSON results**: `fastlane/upload_folder/benchmark_data/` - JSON files for further analysis

### Viewing Results

Results are periodically uploaded to:

* **Dataset**: [argmaxinc/whisperkit-evals-dataset](https://huggingface.co/datasets/argmaxinc/whisperkit-evals-dataset)
* **Benchmarks Space**: [WhisperKit Benchmarks](https://huggingface.co/spaces/argmaxinc/whisperkit-benchmarks)

These provide device comparisons and performance metrics across different models.

## Troubleshooting

<AccordionGroup>
  <Accordion title="Tests fail to run">
    1. Open the project in Xcode: `xed Examples/WhisperAX`
    2. Run the test `RegressionTests/testModelPerformanceWithDebugConfig` from the test navigator
    3. Check Xcode's detailed error messages
    4. If tests run successfully in Xcode, the issue is with the device or Fastlane setup
  </Accordion>

  <Accordion title="Device connection issues">
    1. Run `make list-devices` to verify device status
    2. Ensure devices are in Developer Mode
    3. Check devices are paired and trusted
    4. Try specifying a single device: `make benchmark-devices DEVICES="My Mac"`
  </Accordion>

  <Accordion title="Still having issues?">
    * Join our [Discord](https://discord.gg/G5F5GZGecC) for community support
    * Create an [issue on GitHub](https://github.com/argmaxinc/WhisperKit/issues)
  </Accordion>
</AccordionGroup>

## Command Reference

<CardGroup cols={2}>
  <Card title="make setup" icon="gear">
    Install dependencies and set up environment
  </Card>

  <Card title="make list-devices" icon="list">
    List all connected devices and their status
  </Card>

  <Card title="make benchmark-devices" icon="chart-line">
    Run full benchmark suite on all devices
  </Card>

  <Card title="make benchmark-devices DEBUG=true" icon="bug">
    Run debug benchmark configuration
  </Card>
</CardGroup>

## Next Steps

<CardGroup cols={2}>
  <Card title="View Benchmark Results" icon="chart-bar" href="https://huggingface.co/spaces/argmaxinc/whisperkit-benchmarks">
    Compare performance across devices
  </Card>

  <Card title="Supported Devices" icon="mobile" href="/resources/supported-devices">
    Check device compatibility
  </Card>

  <Card title="Contributing" icon="code-pull-request" href="/resources/contributing">
    Contribute benchmark results
  </Card>

  <Card title="Model Catalog" icon="book" href="/resources/model-catalog">
    Explore available models
  </Card>
</CardGroup>