--- language: - en license: cc-by-nc-sa-3.0 library_name: transformers pipeline_tag: text-classification base_model: dmis-lab/UltraBERT-150M base_model_relation: finetune datasets: - takala/financial_phrasebank tags: - finance - sentiment-analysis - financial-sentiment-analysis - ultrabert - custom_code - fp16 metrics: - accuracy - f1 model-index: - name: UltraFinBERT-150M results: - task: type: text-classification name: Financial sentiment classification dataset: type: takala/financial_phrasebank name: FinancialPhraseBank AllAgree (provided held-out test split) config: sentences_allagree split: test metrics: - type: accuracy name: Accuracy value: 0.96460177 - type: f1 name: Macro F1 value: 0.96159049 --- # UltraFinBERT-150M **UltraFinBERT-150M** is an English financial sentiment classifier obtained by full fine-tuning [dmis-lab/UltraBERT-150M](https://huggingface.co/dmis-lab/UltraBERT-150M) on the unanimous-annotation subset of **FinancialPhraseBank**. It predicts the financial polarity of a sentence from an investor-oriented perspective: **negative**, **neutral**, or **positive**. The model has 151,034,883 parameters and is distributed as **FP16 Safetensors weights**, with the tokenizer and custom UltraBERT implementation required for loading. ## Model details - Base checkpoint revision: `4cc8ca415eb1958184c491297fe5aaa94c4a8ad2`. - Architecture: UltraBERT encoder with a three-class sequence classification head. - Training method: full parameter fine-tuning; standard three-class cross-entropy. - Label mapping: `0 = negative`, `1 = neutral`, `2 = positive`. - Fine-tuning and evaluation maximum input length: **256 tokens**. - The upstream encoder supports longer context, but this classifier was trained and evaluated on short sentences; long-document performance was not evaluated. - Custom code: required (`trust_remote_code=True`). ## Training data and split The source is the provided FinancialPhraseBank AllAgree data. Input source: **provided TXT file**. Encoding: `cp1252`. Source SHA-256 fingerprint(s): `5fb370b3005b9a2759f708849a691ad2e4d2593d4c995c1afde229398a516594`. The optional TXT fallback is parsed at the last `@` separator on each non-empty line. - Original examples: **2264**. - Duplicate rows removed after NFKC, whitespace and case normalization: **5**. - Contradictory-label rows excluded: **0**. - Retained examples: **2259**. - Split method: **Stratified 70/15/15 split generated from TXT with seed 42 (fallback mode)**. - Partition sizes: **1581 train / 339 validation / 339 test**. Text is preserved in its original form after outer whitespace trimming. Normalized exact duplicates cannot cross the three splits. Semantic or article-level duplicates are not identified; no article identifiers are supplied. The test split is held out from this fine-tuning run. Exposure during upstream pretraining has not been audited. This is a custom split, not an official FinancialPhraseBank test partition. The split manifest contains hashes and labels, not the raw sentences. ## Fine-tuning procedure - AdamW: beta1 **0.9**, beta2 **0.98**, epsilon **1e-6**. - Weight decay: **0.1**, using Trainer's standard decay parameter grouping. - Linear learning-rate decay, **5% warm-up**, rounded up to a full optimizer step. - Learning-rate search: **5e-6, 8e-6, 1e-5, 3e-5, 5e-5, 8e-5**. - Selected learning rate: **8e-06**. - Maximum epochs: **10**; early stopping patience: **3** validation checks. - Per-device batch: **8**; gradient accumulation: **4**; effective batch: **32** on one GPU/process. - Gradient clipping: **1.0**; gradient checkpointing enabled; attention: **SDPA**. - Training compute precision: **bfloat16**, with FP32 trainable parameters. - Export precision: **FP16**. Evaluation compute precision: **torch.float16**. - Selection: maximum validation macro F1; validation loss breaks ties across trials. Each trial restores its best validation checkpoint. No final refit on the test or validation split is performed. The exported weights are evaluated directly. - One training seed is used. Training-seed variability has not been measured. ### Learning-rate search (validation only) | learning_rate | validation_macro_f1 | validation_accuracy | trained_epochs | |----------------:|----------------------:|----------------------:|-----------------:| | 8e-06 | 0.988832 | 0.9941 | 8 | Post-export validation macro F1: **0.9888**. Change from the restored pre-export checkpoint's validation macro F1: **+0.000000**; compute precision also differs. ## Evaluation and checkpoint comparison All three checkpoints are evaluated on exactly the same local test sentences, with their own tokenizer, right padding, the same 256-token truncation cap, and probabilities aligned by the label names in each model's configuration. Reference checkpoints are evaluated as published, without additional fine-tuning. The reference revisions are recorded in `experiment.json` and `test_comparison.json`. | model | accuracy | macro_f1 | weighted_f1 | macro_f1_ci95_low | macro_f1_ci95_high | |:----------------------------------|-----------:|-----------:|--------------:|--------------------:|---------------------:| | beethogedeon/UltraFinBERT-150M | 0.9646 | 0.9616 | 0.9642 | 0.9373 | 0.9815 | | ProsusAI/finbert | 0.9823 | 0.9775 | 0.9824 | 0.9591 | 0.9945 | | beethogedeon/Modern-FinBERT-large | 0.9823 | 0.9749 | 0.9821 | 0.9526 | 0.9916 | **Comparison caveat:** both `ProsusAI/finbert` and `beethogedeon/Modern-FinBERT-large` report fine-tuning on FinancialPhraseBank. Their original training partitions are not verified against this test, so overlap is possible. These scores are a descriptive checkpoint comparison, not a contamination-controlled ranking or evidence of superiority on unseen data. Use an independently collected external test corpus for stronger conclusions. Confidence intervals are 95% percentile intervals from **2000** class-stratified bootstrap resamples of the held-out test predictions. They describe test-sampling uncertainty, not variation across training seeds. Per-class reports, log-loss, balanced accuracy, truncation rates and confusion matrices are provided in the repository. ![Confusion matrices](confusion_matrices.png) ![Macro F1 comparison](macro_f1_comparison.png) ## Usage Install the Transformers version used in this experiment: ```bash pip install transformers==5.16.1 'torch>=2.6' safetensors ``` ```python import torch from transformers import AutoTokenizer, AutoModelForSequenceClassification model_id = "beethogedeon/UltraFinBERT-150M" device = torch.device("cuda" if torch.cuda.is_available() else "cpu") dtype = torch.float16 if device.type == "cuda" else torch.float32 tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True) tokenizer.padding_side = "right" model = AutoModelForSequenceClassification.from_pretrained( model_id, trust_remote_code=True, dtype=dtype, attn_implementation="sdpa", ).to(device).eval() texts = ["Operating profit increased compared with the previous year."] inputs = tokenizer(texts, padding=True, truncation=True, max_length=256, return_tensors="pt").to(device) with torch.inference_mode(): probabilities = model(**inputs).logits.float().softmax(dim=-1) for row in probabilities.cpu(): print(dict(zip([model.config.id2label[i] for i in range(3)], row.tolist()))) ``` For reproducible deployment, pin `revision` to a specific commit from this repository. Loading custom code executes repository Python code; review it first. CPU inference casts the stored FP16 weights to FP32. This does not change the FP16 format of the distributed checkpoint. ## Intended uses and limitations Intended for research and non-commercial financial sentence sentiment analysis consistent with the applicable license. This model estimates textual sentiment; it is not a trading signal, a forecast, or financial advice. The dataset is small and imbalanced, concentrated in English financial news and the Finnish listed-company context. Unanimously annotated examples may be easier than ambiguous real-world news. Performance on other markets, current events, sarcasm, multilingual inputs and long documents is not established. Softmax probabilities are not calibrated confidence guarantees. Evaluate on representative data before using the model in a downstream application. ## Reproducibility See `experiment.json`, `requirements.txt`, `data_audit.json`, `split_manifest.csv`, `selection.json`, `learning_rate_search.csv`, `fp16_validation.json`, `test_comparison.json`, and the classification reports. The original text corpus, optimizer state, authentication secrets and text-level prediction logs are not included in this model repository. ## License and attribution The original UltraBERT checkpoint is published under **Apache-2.0**. FinancialPhraseBank is distributed under **Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported** ([dataset card](https://huggingface.co/datasets/takala/financial_phrasebank), [license](https://creativecommons.org/licenses/by-nc-sa/3.0/)). The fine-tuned weights in this repository are released under **CC BY-NC-SA 3.0**. Upstream code retains its original licensing and attribution. Commercial use requires appropriate rights for the data and any other applicable components. No license is inferred from the licenses of the reference models. Please credit the authors of UltraBERT and FinancialPhraseBank: - Jungwoo Park, Taewhoo Lee and Jaewoo Kang. *UltraBERT: Beyond the Limit of BERT Pre-Training* (2026). https://arxiv.org/abs/2610.15028 - Pekka Malo, Ankur Sinha, Pekka Korhonen, Jyrki Wallenius and Pyry Takala. *Good debt or bad debt: Detecting semantic orientations in economic texts*. Journal of the Association for Information Science and Technology (2014). https://arxiv.org/abs/1307.5336 Reference model cards: [ProsusAI/finbert](https://huggingface.co/ProsusAI/finbert) and [beethogedeon/Modern-FinBERT-large](https://huggingface.co/beethogedeon/Modern-FinBERT-large).