| --- |
| library_name: transformers |
| tags: |
| - low resource |
| - trans |
| language: |
| - ban |
| - min |
| - en |
| - id |
| base_model: |
| - Yellow-AI-NLP/komodo-7b-base |
| --- |
| |
| # Model Card for Model ID |
|
|
| NusaMT-7B is a large language model fine-tuned for machine translation of low-resource Indonesian languages, with a focus on Balinese and Minangkabau. Built on LLaMA2-7B and leveraging the Komodo-7B-base model, it incorporates continued pre-training on non-English monolingual data, supervised fine-tuning, data preprocessing for cleaning parallel sentences, and synthetic data generation. |
|
|
|
|
| ## Model Details |
|
|
| ### Model Description |
|
|
|
|
| - **Developed by:** William Tan |
| - **Model type:** Decoder-only Large Language Model |
| - **Language(s) (NLP):** Balinese, Minangkabau, Indonesian, English |
| <!-- - **License:** [More Information Needed] --> |
| - **Finetuned from model:** Yellow-AI-NLP/komodo-7b-base |
|
|
| ### Model Sources |
|
|
| <!-- Provide the basic links for the model. --> |
|
|
| - **Repository:** https://github.com/williammtan/nusamt |
| - **Paper:** https://arxiv.org/abs/2410.07830 |
| - **Demo:** https://indonesiaku.com/translate |
|
|
| ## Uses |
|
|
| The model is designed for: |
| - Bidirectional translation between English/Indonesian and low-resource Indonesian languages (currently Balinese and Minangkabau) |
| - Language preservation and documentation |
| - Cross-cultural communication |
| - Educational purposes and language learning |
|
|
| ### Direct Use |
|
|
| <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. --> |
|
|
| - Integrated into translation applications |
| - Used for data augmentation in low-resource language tasks |
| - Adapted for other Indonesian regional languages |
| - Used as a foundation for developing language learning tools |
|
|
|
|
| ### Out-of-Scope Use |
|
|
| <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. --> |
|
|
| The model is not suitable for: |
|
|
| - Translation of languages outside its trained scope |
| - General text generation or chat functionality |
| - Real-time translation requiring minimal latency |
| - Critical applications where translation errors could cause harm |
|
|
| ## Bias, Risks, and Limitations |
|
|
| <!-- This section is meant to convey both technical and sociotechnical limitations. --> |
|
|
| - Limited to specific language pairs (English/Indonesian ↔ Balinese/Minangkabau) |
| - Performance varies between translation directions, with better results for translations into low-resource languages |
| - Underperforms larger models (NLLB-3.3B) in translations into high-resource languages |
| - May not capture all dialectal variations or cultural nuances |
| - Uses significantly more parameters (7 billion) compared to traditional NMT models |
| - Limited by the quality and quantity of available training data |
|
|
| ### Recommendations |
|
|
| <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. --> |
|
|
| Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. |
|
|
| ## How to Get Started with the Model |
|
|
| Use the code below to get started with the model. |
|
|
|
|
| ## Training Details |
|
|
| ### Training Data |
|
|
| NusaMT: https://huggingface.co/datasets/williamhtan/NusaMT |
|
|
| Total parallel sentences after cleaning: |
| - Balinese ↔ English: 35.6k sentences |
| - Balinese ↔ Indonesian: 44.9k sentences |
| - Minangkabau ↔ English: 16.6k sentences |
| - Minangkabau ↔ Indonesian: 22.4k sentences |
|
|
| Data sources: |
| - NLLB Mined corpus (ODC-BY license) |
| - NLLB SEED dataset (CC-BY-SA license) |
| - BASAbaliWiki (CC-BY-SA license) |
| - Bible verses from Alkitab.mobi (for non-profit scholarly use) |
| - NusaX dataset (CC-BY-SA license) |
|
|
| #### Preprocessing |
|
|
| - Length filtering (15-500 characters) |
| - Word length ratio of 2 |
| - Removal of sentences with words >20 characters |
| - Deduplication |
| - Language identification with GlotLid V3 (threshold: 0.9) |
| - LASER3 similarity scoring (threshold: 1.09) |
| - GPT-4o mini-based data cleaning |
|
|
| #### Training Hyperparameters |
|
|
| - Training regime: bfloat16 mixed precision |
| - LoRA rank: 16 |
| - Learning rate: 0.002 |
| - Batch size: 10 per device |
| - Epochs: 3 |
| - Data splits: 90% training, 5% validation, 5% testing |
| - Loss: Causal Language Modeling (CLM) |
|
|
|
|
| <!-- #### Speeds, Sizes, Times [optional] --> |
|
|
| <!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. --> |
|
|
| <!-- [More Information Needed] --> |
|
|
| ## Evaluation |
|
|
| <!-- This section describes the evaluation protocols and provides the results. --> |
|
|
| ### Testing Data, Factors & Metrics |
|
|
| #### Testing Data |
|
|
| - FLORES-200 multilingual translation benchmark |
| - Internal test set (5% of parallel data) |
|
|
|
|
| #### Metrics |
|
|
| - spBLEU (SentencePiece tokenized BLEU) |
|
|
| ### Results |
|
|
| Performance highlights: |
| - Outperforms SOTA models by up to +6.69 spBLEU in translations into Balinese |
| - Underperforms by up to -3.38 spBLEU in translations into higher-resource languages |
| - Consistently outperforms GPT-3.5, GPT-4, and GPT-4o in zero-shot translation |
|
|
| ### Table 2: spBLEU Score Comparison of the LLaMA2-7B SFT Model with Various Enhancements |
|
|
| | Models | ban → en | en → ban | ban → id | id → ban | |
| |-------------------------------|----------|----------|----------|----------| |
| | LLaMA2-7B SFT | 27.63 | 13.94 | 27.90 | 13.68 | |
| | + Monolingual Pre-training | 31.28 | 18.92 | 28.75 | 20.11 | |
| | + Mono + Backtranslation | 33.97 | 20.27 | 29.62 | 20.67 | |
| | + Mono + LLM Cleaner | 33.23 | 19.75 | 29.02 | 21.16 | |
| | + Mono + Cleaner + Backtrans. | **35.42**| **22.15**| **31.56**| **22.95**| |
|
|
| This table presents spBLEU scores for various configurations of the LLaMA2-7B model, showing the impact of monolingual pre-training, backtranslation, and LLM cleaning on translation performance across different language pairs. |
|
|
| ### Table 3: spBLEU Scores of NusaMT-7B Compared Against SoTA Models and Large GPT Models |
|
|
| | Models | ban → en | en → ban | ban → id | id → ban | min → en | en → min | min → id | id → min | |
| |-------------------------------|----------|----------|----------|----------|----------|----------|----------|----------| |
| | GPT-3.5-turbo, zero-shot | 27.17 | 11.63 | 28.17 | 13.14 | 28.75 | 11.07 | 31.06 | 11.05 | |
| | GPT-4o, zero-shot | 27.11 | 11.45 | 27.89 | 13.08 | 28.63 | 11.00 | 31.27 | 11.00 | |
| | GPT-4, zero-shot | 27.20 | 11.59 | 28.41 | 13.24 | 28.51 | 10.99 | 31.00 | 10.93 | |
| | NLLB-600M | 33.96 | 16.86 | 30.12 | 15.15 | 35.05 | 19.72 | 31.92 | 17.72 | |
| | NLLB-1.3B | 37.24 | 17.73 | 32.42 | 16.21 | 38.59 | 22.79 | 34.68 | 20.89 | |
| | NLLB-3.3B | **38.57**| 17.09 | **33.35**| 14.85 | **40.61**| **24.71**| **35.20**| 22.44 | |
| | NusaMT-7B (Ours) | 35.42 | **22.15**| 31.56 | **22.95**| 37.23 | 24.32 | 34.29 | **23.27**| |
|
|
| This table compares the performance of NusaMT-7B with state-of-the-art models and large GPT models in terms of spBLEU scores across multiple language pairs. NusaMT-7B shows significant improvements, particularly in translations into low-resource languages. |
|
|
|
|
|
|
| ## Environmental Impact |
|
|
|
|
| - **Hardware Type:** 2x NVIDIA RTX 4090 |
| - **Hours used:** 1250 |
| - **Cloud Provider:** Runpod.io |
| - **Carbon Emitted:** 210 kg CO2e |
|
|
|
|
| ## Citation |
|
|
| If you find this model useful, please cite the following works |
|
|
| ``` |
| @misc{tan2024nusamt7bmachinetranslationlowresource, |
| title={NusaMT-7B: Machine Translation for Low-Resource Indonesian Languages with Large Language Models}, |
| author={William Tan and Kevin Zhu}, |
| year={2024}, |
| eprint={2410.07830}, |
| archivePrefix={arXiv}, |
| primaryClass={cs.CL}, |
| url={https://arxiv.org/abs/2410.07830}, |
| } |
| ``` |
|
|
|
|
