Update README.md

8e7703a verified about 2 months ago

7.72 kB

license: mit
language:
  - fa
library_name: transformers
pipeline_tag: token-classification
base_model: google/canine-s
tags:
  - persian
  - farsi
  - diacritization
  - harakat
  - e'raab
  - text2text-generation
  - canine
  - character-level
metrics:
  - accuracy
  - f1
model-index:
  - name: canine-fa-diacritizer
    results:
      - task:
          type: token-classification
          name: Persian Diacritization
        metrics:
          - type: der
            value: 0.083
            name: Diacritic Error Rate (DER)
          - type: wer
            value: 0.1589
            name: Word Error Rate (WER)
          - type: accuracy
            value: 0.917
            name: Character Accuracy
          - type: f1
            value: 0.8887
            name: Macro-F1 (diacritics)
widget:
  - text: کتاب من

🟢 CANINE Persian Diacritizer (`canine-fa-diacritizer`)

Restore diacritics (حَرَکات / اِعراب) on undiacritized Persian text — one short-vowel prediction per character.

Persian is almost always written without short vowels, which makes the same letter string ambiguous to read aloud (مَرد "man" vs مُرد "died"). This model reads a raw Persian sentence and puts the missing marks back — Fatha, Damma, Kasra, Sokun, Shadda, Fathatan and the Shadda+vowel combinations — so the text becomes fully vocalized.

It's built on Google's CANINE, a tokenizer-free encoder that reads one Unicode character at a time. That's the whole trick: a normal subword tokenizer would glue several letters into one opaque token, so you couldn't attach a diacritic to each individual letter. CANINE gives a clean 1-label-per-character alignment, which is exactly what diacritization needs.

Task: Persian diacritization, framed as character-level token classification (10 classes)
Base model: google/canine-s (Apache-2.0)
Parameters: ~132M
Language: Persian / Farsi (fa)
License: MIT

🚀 How to use

This is the important part. The model outputs one label per input character; you then re-insert the corresponding diacritic mark after each character. The snippet below is fully self-contained — no extra files, no local modules.

import torch
from transformers import CanineTokenizer, CanineForTokenClassification

MODEL_ID = "PedramR/canine-fa-diacritizer"

tokenizer = CanineTokenizer.from_pretrained(MODEL_ID)
model = CanineForTokenClassification.from_pretrained(MODEL_ID).eval()

# Combining marks, keyed by the label-name pieces stored in model.config.id2label.
MARKS = {
    "FATHA": "َ", "DAMMA": "ُ", "KASRA": "ِ",
    "SOKUN": "ْ", "SHADDA": "ّ", "FATHATAN": "ً",
}

def label_to_marks(label: str) -> str:
    # e.g. "SHADDA_KASRA" -> ّ + ِ ; "NONE" -> ""
    if label == "NONE":
        return ""
    return "".join(MARKS[part] for part in label.split("_"))

@torch.no_grad()
def diacritize(text: str) -> str:
    enc = tokenizer(text, return_tensors="pt", truncation=True, max_length=2048)
    logits = model(**enc).logits
    preds = logits.argmax(-1)[0].tolist()

    # CANINE adds [CLS] at index 0 and [SEP] at the end; everything in between
    # aligns 1:1 with the input characters.
    char_preds = preds[1:1 + len(text)]

    id2label = model.config.id2label
    out = []
    for ch, pid in zip(text, char_preds):
        out.append(ch)
        out.append(label_to_marks(id2label[pid]))
    return "".join(out)


print(diacritize("کتاب من"))
# -> کِتاب مَن

Why the preds[1:1 + len(text)] slice? CANINE is character-level and adds a [CLS] token at the start and [SEP] at the end. Dropping the first prediction and taking exactly len(text) after it re-aligns predictions perfectly with your original characters. Do not strip or normalize the input first — feed the raw string so the alignment holds.

🏷️ Label scheme (10 classes)

The model predicts one of these per character. They are stored in model.config.id2label, so you never need to hard-code them.

id	label	mark inserted	id	label	mark inserted
0	`NONE`	(nothing)	5	`SHADDA`	ّ
1	`FATHA`	َ	6	`SHADDA_FATHA`	َّ
2	`DAMMA`	ُ	7	`SHADDA_DAMMA`	ُّ
3	`KASRA`	ِ	8	`SHADDA_KASRA`	ِّ
4	`SOKUN`	ْ	9	`FATHATAN`	ً

Shadda (gemination) stacks with a vowel on the same base letter, so the combinations get their own compound labels rather than being dropped.

📊 Evaluation

Evaluated on a held-out test set of 359 sentences (59,927 characters, 17,273 words). Metrics follow the Arabic-diacritization convention (Fadel et al., 2019 / CATT). The starred variants (DER* / WER*) exclude each word's final character, which in Persian is usually the syntax-dependent Ezafe Kasra and is genuinely harder than the rest of the word.

Metric	This model	Baseline (predict no diacritic)
Character accuracy	91.70 %	56.42 %
DER (Diacritic Error Rate)	8.30 %	43.58 %
WER (Word Error Rate)	15.89 %	70.29 %
DER* (ignoring word-final char)	7.63 %	39.98 %
WER* (ignoring word-final char)	11.76 %	60.22 %
Macro-F1 (diacritics)	0.8887	0.0

Per-class F1

Class	F1	Class	F1
NONE	0.932	SHADDA	0.902
FATHA	0.910	SHADDA_FATHA	0.869
DAMMA	0.893	SHADDA_DAMMA	0.925
KASRA	0.863	SHADDA_KASRA	0.806
SOKUN	0.908	FATHATAN	0.922

🧠 Training

Data: avaeziaiteam/harakat-dataset (private) — pairs of raw and diacritized Persian sentences. Base characters and per-character labels are derived by stripping the marks out of the diacritized text itself, which guarantees exact 1:1 char↔label alignment.
Splits: 7,180 deduplicated sentences → 6,462 train / 359 dev / 359 test (90 / 5 / 5, seed 42).
Objective: token classification, best checkpoint chosen by macro-F1 over diacritic classes.
Hyperparameters: 8 epochs · lr 5e-5 (cosine, 10 % warmup) · batch size 16 · weight decay 0.01 · AdamW · fp16 · length-grouped batching · truncation at 512 characters.
Hardware: single NVIDIA RTX 3090.
Frameworks: transformers, torch, datasets.

⚠️ Intended use & limitations

Intended use. Restoring diacritics on modern written Persian — as a front-end for text-to-speech, a reading aid for learners, or a preprocessing step for linguistic tooling.

Limitations.

Trained on a modest (~6.5K-sentence) corpus; expect weaker results on domains far from the training data (e.g. classical poetry, heavy technical jargon, dialectal text).
Word-final position is the hard part. The Ezafe Kasra at the end of a word depends on syntax that can reach beyond the sentence, so errors concentrate there — hence the separately reported DER*/WER*. SHADDA_KASRA is the weakest class (F1 0.806).
Training truncated inputs to 512 characters; at inference the tokenizer allows up to 2048 (CANINE's architectural max is 2046). For long documents, split into sentences.
The rare SUPERSCRIPT_ALEF mark (9 occurrences in the whole corpus) is folded into NONE and is not predicted.

📄 License & attribution

Base model: google/canine-s — Apache-2.0.
Metric definitions follow Fadel et al. (2019) and the CATT Arabic-diacritization line of work.

🟢 CANINE Persian Diacritizer (canine-fa-diacritizer)