---
name: whisper
description: "สกิลสำหรับ Multimodal Whisper ใช้เมื่อต้องการแนวทาง ขั้นตอน Template หรือคำแนะนำเชิงปฏิบัติสำหรับงานวิเคราะห์ข้อมูล วิจัย AI ฐานข้อมูล วิทยาศาสตร์ข้อมูล และรายงานเชิงลึก ในหัวข้อนี้"
version: 1.0.0
author: Orchestra Research
license: MIT
tags: [Whisper, Speech Recognition, ASR, Multimodal, Multilingual, OpenAI, Speech-To-Text, Transcription, Translation, Audio Processing]
dependencies: [openAI-whisper, transformers, torch]
---

# Multimodal Whisper

OpenAI's multilingual speech recognition model.

## When to use Whisper

**Use when:**
- Speech-to-text transcription (99 languages)
- Podcast/video transcription
- Meeting notes automation
- Translation to English
- Noisy audio transcription
- Multilingual audio processing

**Metrics**:
- **72,900+ GitHub stars**
- 99 languages supported
- Trained on 680,000 hours of audio
- MIT License

**Use alternatives instead**:
- **AssemblyAI**: Managed API, speaker diarization
- **Deepgram**: Real-time streaming ASR
- **Google Speech-to-Text**: Cloud-based

## Quick start

### Installation

```bash
# Multimodal Whisper
pip install -U openAI-whisper

# Multimodal Whisper
# Multimodal Whisper
# Multimodal Whisper
# Multimodal Whisper
```

### Basic transcription

```python
import whisper

# Multimodal Whisper
model = whisper.load_model("base")

# Multimodal Whisper
result = model.transcribe("audio.mp3")

# Multimodal Whisper
print(result["text"])

# Multimodal Whisper
for segment in result["segments"]:
    print(f"[{segment['start']:.2f}s - {segment['end']:.2f}s] {segment['text']}")
```

## Model sizes

```python
# Multimodal Whisper
models = ["tiny", "base", "small", "medium", "large", "turbo"]

# Multimodal Whisper
model = whisper.load_model("turbo")  # Fastest, good quality
```

| Model | Parameters | English-only | Multilingual | Speed | VRAM |
|-------|------------|--------------|--------------|-------|------|
| tiny | 39M | ✓ | ✓ | ~32x | ~1 GB |
| base | 74M | ✓ | ✓ | ~16x | ~1 GB |
| small | 244M | ✓ | ✓ | ~6x | ~2 GB |
| medium | 769M | ✓ | ✓ | ~2x | ~5 GB |
| large | 1550M | ✗ | ✓ | 1x | ~10 GB |
| turbo | 809M | ✗ | ✓ | ~8x | ~6 GB |

**Recommendation**: Use `turbo` for best speed/quality, `base` for prototyping

## Transcription options

### Language specification

```python
# Multimodal Whisper
result = model.transcribe("audio.mp3")

# Multimodal Whisper
result = model.transcribe("audio.mp3", language="en")

# Multimodal Whisper
```

### Task selection

```python
# Multimodal Whisper
result = model.transcribe("audio.mp3", task="transcribe")

# Multimodal Whisper
result = model.transcribe("spanish.mp3", task="translate")
# Multimodal Whisper
```

### Initial prompt

```python
# Multimodal Whisper
result = model.transcribe(
    "audio.mp3",
    initial_prompt="This is a technical podcast about machine learning and AI."
)

# Multimodal Whisper
# Multimodal Whisper
# Multimodal Whisper
# Multimodal Whisper
```

### Timestamps

```python
# Multimodal Whisper
result = model.transcribe("audio.mp3", word_timestamps=True)

for segment in result["segments"]:
    for word in segment["words"]:
        print(f"{word['word']} ({word['start']:.2f}s - {word['end']:.2f}s)")
```

### Temperature fallback

```python
# Multimodal Whisper
result = model.transcribe(
    "audio.mp3",
    temperature=(0.0, 0.2, 0.4, 0.6, 0.8, 1.0)
)
```

## Command line usage

```bash
# Multimodal Whisper
whisper audio.mp3

# Multimodal Whisper
whisper audio.mp3 --model turbo

# Multimodal Whisper
whisper audio.mp3 --output_format txt     # Plain text
whisper audio.mp3 --output_format srt     # Subtitles
whisper audio.mp3 --output_format vtt     # WebVTT
whisper audio.mp3 --output_format JSON    # JSON with timestamps

# Multimodal Whisper
whisper audio.mp3 --language Spanish

# Multimodal Whisper
whisper spanish.mp3 --task translate
```

## Batch processing

```python
import os

audio_files = ["file1.mp3", "file2.mp3", "file3.mp3"]

for audio_file in audio_files:
    print(f"Transcribing {audio_file}...")
    result = model.transcribe(audio_file)

    # Save to file
    output_file = audio_file.replace(".mp3", ".txt")
    with open(output_file, "w") as f:
        f.write(result["text"])
```

## Real-time transcription

```python
# Multimodal Whisper
# Multimodal Whisper

from faster_whisper import WhisperModel

model = WhisperModel("base", device="cuda", compute_type="float16")

# Multimodal Whisper
segments, info = model.transcribe("audio.mp3", beam_size=5)

for segment in segments:
    print(f"[{segment.start:.2f}s -> {segment.end:.2f}s] {segment.text}")
```

## GPU acceleration

```python
import whisper

# Multimodal Whisper
model = whisper.load_model("turbo")

# Multimodal Whisper
model = whisper.load_model("turbo", device="cpu")

# Multimodal Whisper
model = whisper.load_model("turbo", device="cuda")

# Multimodal Whisper
```

## Integration with other tools

### Subtitle generation

```bash
# Multimodal Whisper
whisper video.mp4 --output_format srt --language English

# Multimodal Whisper
```

### With LangChain

```python
from langchain.document_loaders import WhisperTranscriptionLoader

loader = WhisperTranscriptionLoader(file_path="audio.mp3")
docs = loader.load()

# Multimodal Whisper
from langchain_chroma import Chroma
from langchain_openAI import OpenAIEmbeddings

vectorstore = Chroma.from_documents(docs, OpenAIEmbeddings())
```

### Extract audio from video

```bash
# Multimodal Whisper
ffmpeg -i video.mp4 -vn -acodec pcm_s16le audio.wav

# Multimodal Whisper
whisper audio.wav
```

## Best practices

1. **Use turbo model** - Best speed/quality for English
2. **Specify language** - Faster than auto-detect
3. **Add initial prompt** - Improves technical terms
4. **Use GPU** - 10-20× faster
5. **Batch process** - More efficient
6. **Convert to WAV** - Better compatibility
7. **Split long audio** - <30 min chunks
8. **Check language support** - Quality varies by language
9. **Use faster-whisper** - 4× faster than openAI-whisper
10. **Monitor VRAM** - Scale model size to hardware

## Performance

| Model | Real-time factor (CPU) | Real-time factor (GPU) |
|-------|------------------------|------------------------|
| tiny | ~0.32 | ~0.01 |
| base | ~0.16 | ~0.01 |
| turbo | ~0.08 | ~0.01 |
| large | ~1.0 | ~0.05 |

*Real-time factor: 0.1 = 10× faster than real-time*

## Language support

Top-supported languages:
- English (en)
- Spanish (es)
- French (fr)
- German (de)
- Italian (it)
- Portuguese (pt)
- Russian (ru)
- Japanese (ja)
- Korean (ko)
- Chinese (zh)

Full list: 99 languages total

## Limitations

1. **Hallucinations** - May repeat or invent text
2. **Long-form accuracy** - Degrades on >30 min audio
3. **Speaker identification** - No diarization
4. **Accents** - Quality varies
5. **Background noise** - Can affect accuracy
6. **Real-time latency** - Not suitable for live captioning

## Resources

- **GitHub**: https://github.com/openAI/whisper ⭐ 72,900+
- **Paper**: https://arxiv.org/abs/2212.04356
- **Model Card**: https://github.com/openAI/whisper/blob/main/model-card.md
- **Colab**: Available in repo
- **License**: MIT


