> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/argmaxinc/WhisperKit/llms.txt
> Use this file to discover all available pages before exploring further.

# Custom Models

> Create, train, and deploy custom Whisper models with WhisperKit

WhisperKit supports loading custom fine-tuned Whisper models, allowing you to deploy specialized models optimized for your specific use case, domain, or language.

## Model Requirements

Custom models must be converted to CoreML format compatible with WhisperKit. The models consist of:

<CardGroup cols={3}>
  <Card title="Audio Encoder" icon="waveform">
    Mel spectrogram to embeddings
  </Card>

  <Card title="Text Decoder" icon="text">
    Embeddings to text tokens
  </Card>

  <Card title="Tokenizer" icon="hashtag">
    Text encoding/decoding
  </Card>
</CardGroup>

## WhisperKit Tools

The [`whisperkittools`](https://github.com/argmaxinc/whisperkittools) Python package provides utilities to:

* Convert Hugging Face Whisper models to CoreML
* Fine-tune models on custom datasets
* Optimize models for specific Apple devices
* Deploy models to Hugging Face Hub

### Installation

```bash theme={null}
pip install git+https://github.com/argmaxinc/whisperkittools.git
```

## Converting Models

### From Hugging Face Hub

Convert any Whisper model from Hugging Face:

```python theme={null}
from whisperkit.convert import convert_whisper_to_coreml

convert_whisper_to_coreml(
    model_name="openai/whisper-large-v3",
    output_dir="./models",
    compute_units="cpuAndNeuralEngine"
)
```

### From Local Checkpoint

Convert a locally fine-tuned model:

```python theme={null}
from whisperkit.convert import convert_whisper_to_coreml

convert_whisper_to_coreml(
    model_path="./my-finetuned-whisper",
    output_dir="./models",
    model_name="custom-whisper-medical",
    compute_units="cpuAndNeuralEngine"
)
```

### Conversion Options

<ParamField path="model_name" type="string">
  Hugging Face model ID (e.g., `openai/whisper-large-v3`)
</ParamField>

<ParamField path="model_path" type="string">
  Path to local model checkpoint
</ParamField>

<ParamField path="output_dir" type="string">
  Directory to save converted models
</ParamField>

<ParamField path="compute_units" type="string" default="cpuAndNeuralEngine">
  Target compute units: `cpuOnly`, `cpuAndGPU`, `cpuAndNeuralEngine`, `all`
</ParamField>

<ParamField path="quantize" type="string">
  Quantization mode: `linear`, `palettize`, or `none`
</ParamField>

## Fine-Tuning Models

### Preparing Your Dataset

Dataset should be in Hugging Face Datasets format with audio and transcription:

```python theme={null}
from datasets import Dataset, Audio

dataset = Dataset.from_dict({
    "audio": ["audio1.wav", "audio2.wav"],
    "text": ["Transcription one.", "Transcription two."]
}).cast_column("audio", Audio(sampling_rate=16000))
```

### Training Example

```python theme={null}
from transformers import WhisperForConditionalGeneration, WhisperProcessor
from transformers import Seq2SeqTrainer, Seq2SeqTrainingArguments

# Load base model
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-small")
processor = WhisperProcessor.from_pretrained("openai/whisper-small")

# Configure training
training_args = Seq2SeqTrainingArguments(
    output_dir="./whisper-finetuned",
    per_device_train_batch_size=16,
    gradient_accumulation_steps=2,
    learning_rate=1e-5,
    num_train_epochs=3,
    fp16=True,
    evaluation_strategy="steps",
    save_strategy="steps",
    save_steps=500,
    eval_steps=500,
    logging_steps=100,
)

# Train model
trainer = Seq2SeqTrainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    tokenizer=processor.feature_extractor,
)

trainer.train()
```

## Deploying to Hugging Face

After converting your model to CoreML, upload to Hugging Face Hub:

```python theme={null}
from huggingface_hub import HfApi

api = HfApi()
api.upload_folder(
    folder_path="./models/custom-whisper-medical",
    repo_id="username/custom-whisper-medical",
    repo_type="model",
)
```

## Loading Custom Models

### From Hugging Face Hub

Once uploaded, load your custom model in WhisperKit:

```swift theme={null}
import WhisperKit

let config = WhisperKitConfig(
    model: "custom-whisper-medical",
    modelRepo: "username/custom-whisper-medical"
)

let pipe = try await WhisperKit(config)
let result = try await pipe.transcribe(audioPath: "patient_recording.wav")
```

### From Local Path

Load models from local filesystem:

```swift theme={null}
import WhisperKit

let config = WhisperKitConfig(
    modelFolder: "/path/to/models/custom-whisper-medical"
)

let pipe = try await WhisperKit(config)
```

### With Compute Options

```swift theme={null}
import WhisperKit
import CoreML

let computeOptions = ModelComputeOptions(
    melCompute: .cpuAndGPU,
    audioEncoderCompute: .cpuAndNeuralEngine,
    textDecoderCompute: .cpuAndNeuralEngine,
    prefillCompute: .cpuOnly
)

let config = WhisperKitConfig(
    model: "custom-whisper-medical",
    modelRepo: "username/custom-whisper-medical",
    computeOptions: computeOptions
)

let pipe = try await WhisperKit(config)
```

## Model Repository Structure

Your Hugging Face repository should follow this structure:

```
username/custom-whisper-medical/
├── AudioEncoder.mlmodelc/
│   └── (CoreML compiled model)
├── TextDecoder.mlmodelc/
│   └── (CoreML compiled model)
├── MelSpectrogram.mlmodelc/
│   └── (CoreML compiled model)
├── generation_config.json
├── config.json
├── tokenizer.json
├── merges.txt
├── vocab.json
└── README.md
```

## Model Variants

WhisperKit supports glob patterns for model selection:

```swift theme={null}
// Select any distil large-v3 variant
let config = WhisperKitConfig(
    model: "distil*large-v3",
    modelRepo: "argmaxinc/whisperkit-coreml"
)
```

Common prefixes:

* `openai_whisper-*` - Original OpenAI models
* `distil-whisper-*` - Distilled models (faster, slightly lower accuracy)

## Optimization Techniques

### Quantization

Reduce model size and improve inference speed:

```python theme={null}
from whisperkit.convert import convert_whisper_to_coreml

convert_whisper_to_coreml(
    model_name="openai/whisper-large-v3",
    output_dir="./models",
    quantize="linear",  # Linear quantization
    # or
    quantize="palettize"  # Palettization (better compression)
)
```

### Model Pruning

Remove unnecessary weights during fine-tuning:

```python theme={null}
from transformers import WhisperForConditionalGeneration
import torch

model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-small")

# Apply structured pruning
from torch.nn.utils import prune

for module in model.modules():
    if isinstance(module, torch.nn.Linear):
        prune.l1_unstructured(module, name="weight", amount=0.3)
```

### Knowledge Distillation

Create smaller models from larger ones:

```python theme={null}
from whisperkit.distillation import distill_model

teacher_model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-large-v3")
student_model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-small")

distill_model(
    teacher=teacher_model,
    student=student_model,
    train_dataset=train_dataset,
    temperature=2.0,
    alpha=0.5
)
```

## Testing Custom Models

### CLI Testing

```bash theme={null}
swift run whisperkit-cli transcribe \
  --model-path "models/custom-whisper-medical" \
  --audio-path "test_audio.wav" \
  --verbose
```

### Programmatic Testing

```swift theme={null}
import WhisperKit

Task {
    let config = WhisperKitConfig(
        modelFolder: "models/custom-whisper-medical",
        verbose: true
    )
    
    let pipe = try await WhisperKit(config)
    
    // Test multiple files
    let testFiles = [
        "test1.wav",
        "test2.wav",
        "test3.wav"
    ]
    
    for file in testFiles {
        let result = try await pipe.transcribe(audioPath: file)
        print("\(file): \(result?.text ?? "Failed")")
    }
}
```

## Benchmarking

Compare your custom model against baselines:

```swift theme={null}
import WhisperKit

func benchmark(modelPath: String, testFiles: [String]) async throws {
    let config = WhisperKitConfig(modelFolder: modelPath)
    let pipe = try await WhisperKit(config)
    
    var totalTime: Double = 0
    var totalTokens: Int = 0
    
    for file in testFiles {
        let start = Date()
        let result = try await pipe.transcribe(audioPath: file)
        let elapsed = Date().timeIntervalSince(start)
        
        totalTime += elapsed
        totalTokens += result?.segments.reduce(0) { $0 + $1.tokens.count } ?? 0
    }
    
    print("Tokens per second: \(Double(totalTokens) / totalTime)")
    print("Real-time factor: \(totalTime / getTotalAudioDuration(testFiles))")
}
```

## Best Practices

<Card title="Model Selection" icon="bullseye">
  * Start with `openai/whisper-small` for fine-tuning (good balance)
  * Use `large-v3` for highest accuracy, `tiny` for fastest inference
  * Consider distil models for production (2-3x faster)
</Card>

<Card title="Fine-Tuning" icon="sliders">
  * Use domain-specific data (medical, legal, technical)
  * Include background noise similar to deployment environment
  * Balance dataset across accents and speakers
  * Fine-tune for 2-5 epochs to avoid overfitting
</Card>

<Card title="Optimization" icon="rocket">
  * Apply quantization for models > 200MB
  * Target `cpuAndNeuralEngine` for macOS 14+ deployment
  * Use `cpuAndGPU` for older macOS versions
  * Test on target devices before deployment
</Card>

<Card title="Validation" icon="check">
  * Measure Word Error Rate (WER) on held-out test set
  * Test edge cases (accents, noise, domain terms)
  * Compare against baseline OpenAI models
  * Profile memory usage and inference time
</Card>

## Example: Medical Transcription Model

Complete workflow for creating a medical transcription model:

```python theme={null}
# 1. Prepare dataset
from datasets import load_dataset

dataset = load_dataset("medical-transcriptions", split="train")
train_test = dataset.train_test_split(test_size=0.1)

# 2. Fine-tune model
from transformers import WhisperForConditionalGeneration, Seq2SeqTrainer

model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-small")
# ... training code ...
model.save_pretrained("./whisper-medical")

# 3. Convert to CoreML
from whisperkit.convert import convert_whisper_to_coreml

convert_whisper_to_coreml(
    model_path="./whisper-medical",
    output_dir="./models/whisper-medical-coreml",
    compute_units="cpuAndNeuralEngine",
    quantize="linear"
)

# 4. Upload to Hub
from huggingface_hub import HfApi

api = HfApi()
api.upload_folder(
    folder_path="./models/whisper-medical-coreml",
    repo_id="username/whisper-medical",
    repo_type="model"
)
```

```swift theme={null}
// 5. Deploy in app
import WhisperKit

let config = WhisperKitConfig(
    model: "whisper-medical",
    modelRepo: "username/whisper-medical"
)

let pipe = try await WhisperKit(config)
let result = try await pipe.transcribe(audioPath: "patient_note.wav")
print(result?.text ?? "")
```

## Resources

<CardGroup cols={2}>
  <Card title="WhisperKit Tools" icon="python" href="https://github.com/argmaxinc/whisperkittools">
    Python toolkit for model conversion
  </Card>

  <Card title="Model Hub" icon="database" href="https://huggingface.co/argmaxinc/whisperkit-coreml">
    Pre-converted WhisperKit models
  </Card>

  <Card title="Hugging Face Whisper" icon="face-smile" href="https://huggingface.co/models?other=whisper">
    Browse available Whisper models
  </Card>

  <Card title="Fine-tuning Guide" icon="book" href="https://huggingface.co/blog/fine-tune-whisper">
    Official Whisper fine-tuning guide
  </Card>
</CardGroup>

## Next Steps

<CardGroup cols={2}>
  <Card title="Performance Optimization" icon="gauge-high" href="/advanced/performance-optimization">
    Optimize custom model performance
  </Card>

  <Card title="Memory Management" icon="memory" href="/advanced/memory-management">
    Manage memory for large models
  </Card>
</CardGroup>