Text Generation
Transformers
Safetensors
English
lfm2
continued-pretraining
domain-adaptation
conversational

LFM2.5-1.2B Asterion — Domain CPT

LiquidAI LFM2.5-1.2B-Base after Continued Pre-Training on the Asterion Space Operations corpus (64.3M-token stratified sample) plus reused Mars Express telemetry. De-risking sibling of the Gemma-4-12B Asterion round.

Overview

  • Stage: Continued Pre-Training (full fine-tune, CLM)
  • Lineage: LFM2.5-1.2B-Base → CPT (this model)
  • Method: Continued Pre-Training (next-token, full fine-tune) on a TOKEN-BUDGET stratified sample of the Asterion corpus (55M-token budget over the 84 doc_type × topic cells — proportional-to-params cut for a 1.2B), with 10% FineWeb-Edu replay against forgetting [replay].
  • Domain: fictional — Orbital Mining Corporation (OMC) technical docs + Mars Express telemetry.

Training

Corpus asterion-training-corpus-lfm2 — 64.3M LFM2 tokens (85.5% Asterion / 4.5% Mars telemetry / 10% replay); train 17,829 / val 500
Objective CLM (next-token), full fine-tune, bf16
LR / schedule 1e-4 cosine, warmup 0.03
Epochs / seq 2 epochs, seq 4096, eff_batch 32

Evaluation

Metric Value Note
PPL Asterion held-out 1.91 base: 7.36
PPL Mars telemetry 1.30 base: 5.96
PPL general (FineWeb-Edu) 7.31 base: 11.87 — no forgetting

Eval is a held-out, deterministic verifiable harness (synthetic tasks); baseline = the pre-SFT ParamΔ model. See noval-corp/scripts/eval_agentic.py.

Intended use & limitations

Domain-knowledge backbone for the Asterion round. Not instruction-tuned (see the -instruct-paramdelta / -agentic siblings).

Limitations:

  • Fictional domain — the model 'knows' an invented satellite fleet.
  • No chat template / instruction following (base-style CPT checkpoint).
  • Trained on ~8.5% of the source corpus (token-budget cut); the Gemma-4-12B sibling consumes the full corpus.

Innovations tested

  • Stratified token-budget sampling — round-robin over (doc_type × topic) cells up to a token budget — even coverage of the domain at a fraction of the corpus.

Citations

  • CPT replay (anti-forgetting) — Ibrahim et al., 2024. arXiv:2403.08763
  • Data-constrained scaling (≤4 epochs) — Muennighoff et al., 2023. arXiv:2305.16264

Card generated by noval-corp/scripts/gen_model_cards.py (standardized across the noval-corp model family).

Downloads last month
25
Safetensors
Model size
1B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for atenareply/lfm2.5-1.2b-asterion

Finetuned
(57)
this model
Quantizations
1 model

Papers for atenareply/lfm2.5-1.2b-asterion