Nguni-ByT5-large

Nguni-ByT5-large is a pretrained encoder-decoder/text-to-text language model that adapts ByT5-large for the Nguni languages - isiXhosa, isiZulu, isiNdebele, and Siswati - via multilingual continued pretraining. It outperforms base models on sequence-to-sequence generation tasks in the NGLUEni benchmark.

Paper: NGLUEni: Benchmarking and Adapting Pretrained Language Models for Nguni Languages
Francois Meyer, Haiyue Song, Abhisek Chakrabarty, Jan Buys, Raj Dabre and Hideki Tanaka
LREC-COLING 2024

Usage

1. Corrupted span prediction.

## Example from here: https://huggingface.co/docs/transformers/en/model_doc/byt5
tokenizer = AutoTokenizer.from_pretrained("francois-meyer/nguni-byt5-large")
model = AutoModelForSeq2SeqLM.from_pretrained("francois-meyer/nguni-byt5-large")
#model = T5ForConditionalGeneration.from_pretrained(model_path)

input_ids_prompt = "The dog chases a ball in the park."
input_ids = tokenizer(input_ids_prompt).input_ids

input_ids = torch.tensor([input_ids[:8] + [258] + input_ids[14:21] + [257] + input_ids[28:]]) ## Corruption

output_ids = model.generate(input_ids, max_length=100)[0].tolist()

output_ids_list = []
start_token = 0
sentinel_token = 258
while sentinel_token in output_ids:
    split_idx = output_ids.index(sentinel_token)
    output_ids_list.append(output_ids[start_token:split_idx])
    start_token = split_idx
    sentinel_token -= 1

output_ids_list.append(output_ids[start_token:])
output_string = tokenizer.batch_decode(output_ids_list)
print(output_string)

2. For any other task, you will need to fine-tune it like any other T5, mT5, byT5 model.

Citation

@inproceedings{meyer-etal-2024-nglueni,
    title = "{NGLUE}ni: Benchmarking and Adapting Pretrained Language Models for Nguni Languages",
    author = "Meyer, Francois  and
      Song, Haiyue  and
      Chakrabarty, Abhisek  and
      Buys, Jan  and
      Dabre, Raj  and
      Tanaka, Hideki",
    editor = "Calzolari, Nicoletta  and
      Kan, Min-Yen  and
      Hoste, Veronique  and
      Lenci, Alessandro  and
      Sakti, Sakriani  and
      Xue, Nianwen",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.lrec-main.1071/",
    pages = "12247--12258",
    abstract = "The Nguni languages have over 20 million home language speakers in South Africa. There has been considerable growth in the datasets for Nguni languages, but so far no analysis of the performance of NLP models for these languages has been reported across languages and tasks. In this paper we study pretrained language models for the 4 Nguni languages - isiXhosa, isiZulu, isiNdebele, and Siswati. We compile publicly available datasets for natural language understanding and generation, spanning 6 tasks and 11 datasets. This benchmark, which we call NGLUEni, is the first centralised evaluation suite for the Nguni languages, allowing us to systematically evaluate the Nguni-language capabilities of pretrained language models (PLMs). Besides evaluating existing PLMs, we develop new PLMs for the Nguni languages through multilingual adaptive finetuning. Our models, Nguni-XLMR and Nguni-ByT5, outperform their base models and large-scale adapted models, showing that performance gains are obtainable through limited language group-based adaptation. We also perform experiments on cross-lingual transfer and machine translation. Our models achieve notable cross-lingual transfer improvements in the lower resourced Nguni languages (isiNdebele and Siswati). To facilitate future use of NGLUEni as a standardised evaluation suite for the Nguni languages, we create a web portal to access the collection of datasets and publicly release our models."
}