# เอกสารอ้างอิง: Integration

Complete guide to using HuggingFace Tokenizers with the Transformers library.

## AutoTokenizer

The easiest way to load tokenizers.

### Loading pretrained tokenizers

```python
from transformers import AutoTokenizer

# เอกสารอ้างอิง: Integration
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

# เอกสารอ้างอิง: Integration
print(tokenizer.is_fast)  # True

# เอกสารอ้างอิง: Integration
if tokenizer.is_fast:
    fast_tokenizer = tokenizer.backend_tokenizer
    print(type(fast_tokenizer))  # <class 'tokenizers.Tokenizer'>
```

### Fast vs slow tokenizers

| Feature                  | Fast (Rust)    | Slow (Python) |
|--------------------------|----------------|---------------|
| Speed                    | 5-10× faster   | Baseline      |
| Alignment tracking       | ✅ Full support | ❌ Limited     |
| Batch processing         | ✅ Optimized    | ⚠️ Slower      |
| Offset mapping           | ✅ Yes          | ❌ No          |
| Installation             | `tokenizers`   | Built-in      |

**Always use fast tokenizers when available.**

### Check available tokenizers

```python
from transformers import TOKENIZER_MAPPING

# เอกสารอ้างอิง: Integration
for config_class, (slow, fast) in TOKENIZER_MAPPING.items():
    if fast is not None:
        print(f"{config_class.__name__}: {fast.__name__}")
```

## PreTrainedTokenizerFast

Wrap custom tokenizers for transformers.

### Convert custom tokenizer

```python
from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
from transformers import PreTrainedTokenizerFast

# เอกสารอ้างอิง: Integration
tokenizer = Tokenizer(BPE())
trainer = BpeTrainer(
    vocab_size=30000,
    special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"]
)
tokenizer.train(files=["corpus.txt"], trainer=trainer)

# เอกสารอ้างอิง: Integration
tokenizer.save("my-tokenizer.json")

# เอกสารอ้างอิง: Integration
transformers_tokenizer = PreTrainedTokenizerFast(
    tokenizer_file="my-tokenizer.json",
    unk_token="[UNK]",
    sep_token="[SEP]",
    pad_token="[PAD]",
    cls_token="[CLS]",
    mask_token="[MASK]"
)

# เอกสารอ้างอิง: Integration
transformers_tokenizer.save_pretrained("my-tokenizer")
```

**Result**: Directory with `tokenizer.json` + `tokenizer_config.json` + `special_tokens_map.json`

### Use like any transformers tokenizer

```python
# เอกสารอ้างอิง: Integration
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("my-tokenizer")

# เอกสารอ้างอิง: Integration
outputs = tokenizer(
    "Hello world",
    padding="max_length",
    truncation=True,
    max_length=128,
    return_tensors="pt"
)

print(outputs.keys())
# เอกสารอ้างอิง: Integration
```

## Special tokens

### Default special tokens

| Model Family | CLS/BOS | SEP/EOS       | PAD     | UNK     | MASK    |
|--------------|---------|---------------|---------|---------|---------|
| BERT         | [CLS]   | [SEP]         | [PAD]   | [UNK]   | [MASK]  |
| GPT-2        | -       | <\|endoftext\|> | <\|endoftext\|> | <\|endoftext\|> | -       |
| RoBERTa      | <s>     | </s>          | <pad>   | <unk>   | <mask>  |
| T5           | -       | </s>          | <pad>   | <unk>   | -       |

### Adding special tokens

```python
# เอกสารอ้างอิง: Integration
special_tokens_dict = {
    "additional_special_tokens": ["<|image|>", "<|video|>", "<|audio|>"]
}

num_added_tokens = tokenizer.add_special_tokens(special_tokens_dict)
print(f"Added {num_added_tokens} tokens")

# เอกสารอ้างอิง: Integration
model.resize_token_embeddings(len(tokenizer))

# เอกสารอ้างอิง: Integration
text = "This is an image: <|image|>"
tokens = tokenizer.encode(text)
```

### Adding regular tokens

```python
# เอกสารอ้างอิง: Integration
new_tokens = ["COVID-19", "mRNA", "vaccine"]
num_added = tokenizer.add_tokens(new_tokens)

# เอกสารอ้างอิง: Integration
tokenizer.add_tokens(new_tokens, special_tokens=False)

# เอกสารอ้างอิง: Integration
tokenizer.add_tokens(new_tokens, special_tokens=True)
```

## Encoding and decoding

### Basic encoding

```python
# เอกสารอ้างอิง: Integration
text = "Hello, how are you?"
encoded = tokenizer(text)

print(encoded)
# เอกสารอ้างอิง: Integration
# เอกสารอ้างอิง: Integration
# เอกสารอ้างอิง: Integration
```

### Batch encoding

```python
# เอกสารอ้างอิง: Integration
texts = ["Hello world", "How are you?", "I am fine"]
encoded = tokenizer(texts, padding=True, truncation=True, max_length=10)

print(encoded['input_ids'])
# เอกสารอ้างอิง: Integration
# เอกสารอ้างอิง: Integration
# เอกสารอ้างอิง: Integration
```

### Return tensors

```python
# เอกสารอ้างอิง: Integration
outputs = tokenizer("Hello world", return_tensors="pt")
print(outputs['input_ids'].shape)  # torch.Size([1, 5])

# เอกสารอ้างอิง: Integration
outputs = tokenizer("Hello world", return_tensors="tf")

# เอกสารอ้างอิง: Integration
outputs = tokenizer("Hello world", return_tensors="np")

# เอกสารอ้างอิง: Integration
outputs = tokenizer("Hello world", return_tensors=None)
```

### Decoding

```python
# เอกสารอ้างอิง: Integration
ids = [101, 7592, 2088, 102]
text = tokenizer.decode(ids)
print(text)  # "[CLS] hello world [SEP]"

# เอกสารอ้างอิง: Integration
text = tokenizer.decode(ids, skip_special_tokens=True)
print(text)  # "hello world"

# เอกสารอ้างอิง: Integration
batch_ids = [[101, 7592, 102], [101, 2088, 102]]
texts = tokenizer.batch_decode(batch_ids, skip_special_tokens=True)
print(texts)  # ["hello", "world"]
```

## Padding and truncation

### Padding strategies

```python
# เอกสารอ้างอิง: Integration
tokenizer(texts, padding="longest")

# เอกสารอ้างอิง: Integration
tokenizer(texts, padding="max_length", max_length=128)

# เอกสารอ้างอิง: Integration
tokenizer(texts, padding=False)

# เอกสารอ้างอิง: Integration
tokenizer(texts, padding="max_length", max_length=128, pad_to_multiple_of=8)
# เอกสารอ้างอิง: Integration
```

### Truncation strategies

```python
# เอกสารอ้างอิง: Integration
tokenizer(text, truncation=True, max_length=10)

# เอกสารอ้างอิง: Integration
tokenizer(text1, text2, truncation="only_first", max_length=20)

# เอกสารอ้างอิง: Integration
tokenizer(text1, text2, truncation="only_second", max_length=20)

# เอกสารอ้างอิง: Integration
tokenizer(text1, text2, truncation="longest_first", max_length=20)

# เอกสารอ้างอิง: Integration
tokenizer(text, truncation=False)
```

### Stride for long documents

```python
# เอกสารอ้างอิง: Integration
text = "Very long document " * 1000

# เอกสารอ้างอิง: Integration
encodings = tokenizer(
    text,
    max_length=512,
    stride=128,          # Overlap between chunks
    truncation=True,
    return_overflowing_tokens=True,
    return_offsets_mapping=True
)

# เอกสารอ้างอิง: Integration
num_chunks = len(encodings['input_ids'])
print(f"Split into {num_chunks} chunks")

# เอกสารอ้างอิง: Integration
for i, chunk in enumerate(encodings['input_ids']):
    print(f"Chunk {i}: {len(chunk)} tokens")
```

**Use case**: Long document QA, sliding window inference

## Alignment and offsets

### Offset mapping

```python
# เอกสารอ้างอิง: Integration
encoded = tokenizer("Hello, world!", return_offsets_mapping=True)

for token, (start, end) in zip(
    encoded.tokens(),
    encoded['offset_mapping'][0]
):
    print(f"{token:10s} → [{start:2d}, {end:2d})")

# เอกสารอ้างอิง: Integration
# เอกสารอ้างอิง: Integration
# เอกสารอ้างอิง: Integration
# เอกสารอ้างอิง: Integration
# เอกสารอ้างอิง: Integration
# เอกสารอ้างอิง: Integration
# เอกสารอ้างอิง: Integration
```

### Word IDs

```python
# เอกสารอ้างอิง: Integration
encoded = tokenizer("Hello world", return_offsets_mapping=True)
word_ids = encoded.word_ids()

print(word_ids)
# เอกสารอ้างอิง: Integration
# เอกสารอ้างอิง: Integration
```

**Use case**: Token classification (NER, POS tagging)

### Character to token mapping

```python
text = "Machine learning is awesome"
encoded = tokenizer(text, return_offsets_mapping=True)

# เอกสารอ้างอิง: Integration
char_pos = 8  # "l" in "learning"
token_idx = encoded.char_to_token(char_pos)

print(f"Character {char_pos} is in token {token_idx}: {encoded.tokens()[token_idx]}")
# เอกสารอ้างอิง: Integration
```

**Use case**: Question answering (map answer character span to tokens)

### Sequence pairs

```python
# เอกสารอ้างอิง: Integration
encoded = tokenizer("Question here", "Answer here", return_offsets_mapping=True)

# เอกสารอ้างอิง: Integration
sequence_ids = encoded.sequence_ids()
print(sequence_ids)
# เอกสารอ้างอิง: Integration
# เอกสารอ้างอิง: Integration
```

## Model integration

### Use with transformers models

```python
from transformers import AutoModel, AutoTokenizer
import torch

# เอกสารอ้างอิง: Integration
model = AutoModel.from_pretrained("bert-base-uncased")
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

# เอกสารอ้างอิง: Integration
text = "Hello world"
inputs = tokenizer(text, return_tensors="pt")

# เอกสารอ้างอิง: Integration
with torch.no_grad():
    outputs = model(**inputs)

# เอกสารอ้างอิง: Integration
last_hidden_state = outputs.last_hidden_state
print(last_hidden_state.shape)  # [1, seq_len, hidden_size]
```

### Custom model with custom tokenizer

```python
from transformers import BertConfig, BertModel

# เอกสารอ้างอิง: Integration
from tokenizers import Tokenizer, models, trainers
tokenizer = Tokenizer(models.BPE())
trainer = trainers.BpeTrainer(vocab_size=30000)
tokenizer.train(files=["data.txt"], trainer=trainer)

# เอกสารอ้างอิง: Integration
from transformers import PreTrainedTokenizerFast
fast_tokenizer = PreTrainedTokenizerFast(
    tokenizer_object=tokenizer,
    unk_token="[UNK]",
    pad_token="[PAD]"
)

# เอกสารอ้างอิง: Integration
config = BertConfig(vocab_size=30000)
model = BertModel(config)

# เอกสารอ้างอิง: Integration
inputs = fast_tokenizer("Hello world", return_tensors="pt")
outputs = model(**inputs)
```

### Save and load together

```python
# เอกสารอ้างอิง: Integration
model.save_pretrained("my-model")
tokenizer.save_pretrained("my-model")

# เอกสารอ้างอิง: Integration
# เอกสารอ้างอิง: Integration
# เอกสารอ้างอิง: Integration
# เอกสารอ้างอิง: Integration
# เอกสารอ้างอิง: Integration
# เอกสารอ้างอิง: Integration
# เอกสารอ้างอิง: Integration

# เอกสารอ้างอิง: Integration
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("my-model")
tokenizer = AutoTokenizer.from_pretrained("my-model")
```

## Advanced features

### Multimodal tokenization

```python
from transformers import AutoTokenizer

# เอกสารอ้างอิง: Integration
tokenizer = AutoTokenizer.from_pretrained("llava-hf/llava-1.5-7b-hf")

# เอกสารอ้างอิง: Integration
tokenizer.add_special_tokens({"additional_special_tokens": ["<image>"]})

# เอกสารอ้างอิง: Integration
text = "Describe this image: <image>"
inputs = tokenizer(text, return_tensors="pt")
```

### Template formatting

```python
# เอกสารอ้างอิง: Integration
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Hello!"},
    {"role": "assistant", "content": "Hi! How can I help?"},
    {"role": "user", "content": "What's the weather?"}
]

# เอกสารอ้างอิง: Integration
if hasattr(tokenizer, "apply_chat_template"):
    text = tokenizer.apply_chat_template(messages, tokenize=False)
    inputs = tokenizer(text, return_tensors="pt")
```

### Custom template

```python
from transformers import PreTrainedTokenizerFast

tokenizer = PreTrainedTokenizerFast(tokenizer_file="tokenizer.json")

# เอกสารอ้างอิง: Integration
tokenizer.chat_template = """
{%- for message in messages %}
    {%- if message['role'] == 'system' %}
        System: {{ message['content'] }}\\n
    {%- elif message['role'] == 'user' %}
        User: {{ message['content'] }}\\n
    {%- elif message['role'] == 'assistant' %}
        Assistant: {{ message['content'] }}\\n
    {%- endif %}
{%- endfor %}
Assistant:
"""

# เอกสารอ้างอิง: Integration
text = tokenizer.apply_chat_template(messages, tokenize=False)
```

## Performance optimization

### Batch processing

```python
# เอกสารอ้างอิง: Integration
from datasets import load_dataset

dataset = load_dataset("imdb", split="train[:1000]")

# เอกสารอ้างอิง: Integration
def tokenize_function(examples):
    return tokenizer(
        examples["text"],
        padding="max_length",
        truncation=True,
        max_length=512
    )

# เอกสารอ้างอิง: Integration
tokenized_dataset = dataset.map(
    tokenize_function,
    batched=True,
    batch_size=1000,
    num_proc=4  # Parallel processing
)
```

### Caching

```python
# เอกสารอ้างอิง: Integration
tokenizer = AutoTokenizer.from_pretrained(
    "bert-base-uncased",
    use_fast=True,
    cache_dir="./cache"  # Cache tokenizer files
)

# เอกสารอ้างอิง: Integration
from functools import lru_cache

@lru_cache(maxsize=10000)
def cached_tokenize(text):
    return tuple(tokenizer.encode(text))

# เอกสารอ้างอิง: Integration
```

### Memory efficiency

```python
# เอกสารอ้างอิง: Integration
from datasets import load_dataset

dataset = load_dataset("pile", split="train", streaming=True)

def process_batch(batch):
    # Tokenize
    tokens = tokenizer(batch["text"], truncation=True, max_length=512)

    # Process tokens...

    return tokens

# เอกสารอ้างอิง: Integration
for batch in dataset.batch(batch_size=1000):
    processed = process_batch(batch)
```

## Troubleshooting

### Issue: Tokenizer not fast

**Symptom**:
```python
tokenizer.is_fast  # False
```

**Solution**: Install tokenizers library
```bash
pip install tokenizers
```

### Issue: Special tokens not working

**Symptom**: Special tokens are split into subwords

**Solution**: Add as special tokens, not regular tokens
```python
# เอกสารอ้างอิง: Integration
tokenizer.add_tokens(["<|image|>"])

# เอกสารอ้างอิง: Integration
tokenizer.add_special_tokens({"additional_special_tokens": ["<|image|>"]})
```

### Issue: Offset mapping not available

**Symptom**:
```python
tokenizer("text", return_offsets_mapping=True)
# เอกสารอ้างอิง: Integration
```

**Solution**: Use fast tokenizer
```python
from transformers import AutoTokenizer

# เอกสารอ้างอิง: Integration
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased", use_fast=True)
```

### Issue: Padding inconsistent

**Symptom**: Some sequences padded, others not

**Solution**: Specify padding strategy
```python
# เอกสารอ้างอิง: Integration
tokenizer(
    texts,
    padding="max_length",  # or "longest"
    max_length=128
)
```

## Best practices

1. **Always use fast tokenizers**:
   - 5-10× faster
   - Full alignment tracking
   - Better batch processing

2. **Save tokenizer with model**:
   - Ensures reproducibility
   - Prevents version mismatches

3. **Use batch processing for datasets**:
   - Tokenize with `.map(batched=True)`
   - Set `num_proc` for parallelism

4. **Enable caching for repeated inputs**:
   - Use `lru_cache` for inference
   - Cache tokenizer files with `cache_dir`

5. **Handle special tokens properly**:
   - Use `add_special_tokens()` for never-split tokens
   - Resize embeddings after adding tokens

6. **Test alignment for downstream tasks**:
   - Verify `offset_mapping` is correct
   - Test `char_to_token()` on samples

7. **Version control tokenizer config**:
   - Save `tokenizer_config.json`
   - Document custom templates
   - Track vocabulary changes
