Flaglab/ESNLIR-Baseline

ESNLIR — XGBoost baseline

The non-neural reference point for ESNLIR: XGBoost over bag-of-words features from both sentences. It exists to show how much of the task is solvable without contextual representations — the answer is not much, which is what makes the BERT results meaningful.

ESNLIR: Expanding Spanish NLI Benchmarks with Multi-Genre and Causal Annotation Johan R. Portela, Nicolás Pérez-Terán, Rubén Manrique — Universidad de los Andes, Bogotá Applied Informatics, Springer, 2026, pp. 345–361 — doi:10.1007/978-3-032-07175-0_23

Part of the ESNLIR collection.

Results on the ESNLIR test set (80,216 pairs, class-balanced)

accuracy	macro F1	contrasting	entailment	neutral	reasoning
0.3501	0.3481	0.345	0.436	0.278	0.341

Ten points above the 0.250 majority-class floor, and roughly half what the fine-tuned encoders reach:

model	accuracy	macro F1
Majority class	0.250	0.250
XGBoost (this model)	0.350	0.348
BERTIN RoBERTa	0.663	0.664
XLM-RoBERTa	0.676	0.676

Lexical features alone barely move the needle, so the relations in ESNLIR are not recoverable from word identity — contextual sentence representations are doing the work.

Stress tests

test	accuracy	macro F1	contrasting	entailment	neutral	reasoning
test	0.3501	0.3481	0.345	0.436	0.278	0.341
test_length_mismatch	0.3257	0.2945	0.204	0.307	0.111	0.681
test_negation	0.3381	0.3287	0.401	0.266	0.200	0.486
test_overlap	0.3409	0.3313	0.226	0.360	0.245	0.532
test_spelling	0.3490	0.3473	0.340	0.428	0.280	0.348

The perturbations push predictions hard toward reasoning (0.341 → 0.681 under length mismatch), which is what a bag-of-words model does when tokens are injected: the added words shift the feature vector rather than the meaning.

Labels

id	label
0	contrasting
1	entailment
2	neutral
3	reasoning

Label order is alphabetical, from sklearn.preprocessing.LabelEncoder.

Files

model.pkl — a pickled scikit-learn/XGBoost estimator. metrics.zip — full metrics per split, broken down by genre, domain and source corpus.

Usage

import pickle
from huggingface_hub import hf_hub_download

with open(hf_hub_download("Flaglab/ESNLIR-Baseline", "model.pkl"), "rb") as fh:
    model = pickle.load(fh)

Loading a pickle executes arbitrary code — only do this because you trust the source.

The vectorizer is not included in this repository, so the model cannot be applied to raw text as published. Rebuild the bag-of-words features with jd-rodriguezp1234/esnlir (auto_nli/model/baseline/dataset.py), which fits the vectorizer and label encoder on the training split. For inference on new text, prefer ESNLIR-XLM-RoBERTa.

Training

features	bag-of-words over both sentences
data	Flaglab/ESNLIR-dataset
max samples	1,000,000 (subsampled from the 4.4M train split)

Trained with auto_nli/model/baseline/run.py, config params/model/baseline.json.

Citation

@InProceedings{portela2025esnlirspanishmultigenredataset,
  author    = {Portela, Johan R. and P{\'e}rez-Ter{\'a}n, Nicol{\'a}s and Manrique, Rub{\'e}n},
  editor    = {Florez, Hector and Peluffo-Ordo{\~{n}}ez, Diego},
  title     = {{ESNLIR}: Expanding Spanish {NLI} Benchmarks with Multi-genre and Causal Annotation},
  booktitle = {Applied Informatics},
  year      = {2026},
  publisher = {Springer Nature Switzerland},
  address   = {Cham},
  pages     = {345--361},
  isbn      = {978-3-032-07175-0},
  doi       = {10.1007/978-3-032-07175-0_23},
}