ToricGT 170M ConvexTok-8192 FoT

ToricGT 170M ConvexTok-8192 FoT is the current muP-scaled ToricGT/ToricBLM checkpoint line: a width-scaled adaptation of the OpenAI Parameter Golf baseline with first-class graph-structured training, fresh ConvexTok tokenization, TokenGT-style graph tokenization, graph-output flattening for BPB scoring, embedding-space Forest-of-Thought reasoning, GFlowNet-style trajectory objectives, GraphCG disentanglement, and tropical-toric/topological/category-theoretic audit losses.

This repository now carries dated checkpoint artifacts from the structure-priority ToricBLM/FoT run and explicit example prompts for checkpoint inference. The examples are stored under examples/ so they can be used by local ToricGT inference/export scripts without copying prompt text from the model card.

Current Training Run

Item	Value
Run id	`toricblm-mup-codex8192-biomed-full-<timestamp>`
Approximate parameters	`169,698,927` generated target params
Width transfer	muP base 512, delta 768, target 1536, width multiplier 3.0
Layers	9
Heads / KV heads	12 / 6
Sequence length	1024
Effective batch tokens	196,608 initial setting for larger softmax
Tokenizer	fresh ConvexTok deterministic 8192, no BPE, no old-vocab reuse
Biomedical/control reserve	512-token target reserve from universal-modality biomedical seed syntax
Primary BPB stream	FineWeb-style ConvexTok-8192 shards exported from ToricGT curated Parquet source
Graph stream	full `AmelieSchreiber/toricgt-curated-splits` train split
Late-stage graph stream	`AmelieSchreiber/codex5.5_ToT`, train-only, 3-pass view
Late-stage mix	85% codex5.5_ToT / FoT graph stream after step 17,500
Validation	FineWeb BPB validation stream

What Is Being Trained

The run combines five layers of structure:

FineWeb BPB objective. Bits-per-byte remains the score-bearing language-model objective.
ConvexTok DAG tokenization. Tokenization is represented as a byte-boundary DAG: vertices are byte boundaries, candidate arcs are tokens, rounded/frequency-ranked candidate features become token/edge features, and the final flattened sequence remains compatible with BPB scoring. The active scale-up tokenizer is a fresh ConvexTok-8192 vocabulary with a biomedical/control reserve for future UniProt, PDB/AFDB/ESMFold, molecular-graph, atomistic-trajectory, cell-state, tropical-toric, persistence, category-theoretic, and FoT control streams.
TokenGT graph input/output. Text and graph records are graphified, with node tokens, edge tokens, endpoint structure, positional buckets, toric phase features, and optional graph-output flattening for OAI FineWeb scoring.
Embedding-space FoT and GFlowNets. The model trains branching reasoning trajectories in embedding space with Forest-of-Thought style exploration, GFlowNet rewards, trajectory memory retrieval, and analogical retrieval heads.
Mathematical audits and regularizers. Tropical attention, toric embeddings, one-dimensional-cone/vector-bundle probes, BGG category O certificates, Koszul/persistence metrics, combinatorial commutative algebra, derived signatures, and GraphCG basis-vector disentanglement are active as low-weight regularizers and W&B metrics.

Data

The model uses:

Fresh ConvexTok-8192 shards for the primary BPB objective, exported from the existing ToricGT local data root.
AmelieSchreiber/toricgt-curated-splits train split as the full curated graph training stream:
- train: 4,633,582 rows, 117 parquet shards
- validation/test are kept for evaluation and are not used for graph training.
AmelieSchreiber/codex5.5_ToT as a late-stage train-only Tree/Forest-of-Thought dataset:
- 3,890 source rows
- materialized as a deterministic three-pass late-stage view
- no validation/test split is used from this dataset.

Running a Checkpoint

This is not a standard transformers model repo. The checkpoint is loaded through the ToricGT codebase and the adapted OpenAI Parameter Golf baseline.

git clone --recursive https://github.com/amelie-iska/ToricGT.git
cd ToricGT
git checkout toricblm

# After a dated checkpoint is uploaded here:
hf download AmelieSchreiber/ToricGT_160M_FoT \
  --include "checkpoints/*_step_*.pt" \
  --local-dir ./hf_downloads/ToricGT_160M_FoT

# Example evaluation entry point from the ToricGT repo.
conda run -n tokengt env PYTHONPATH=src:external/mup:amelie-iska/parameter-golf \
  python scripts/evaluate_oai_competition_bpb.py \
  --checkpoint ./hf_downloads/ToricGT_160M_FoT/checkpoints/<dated-checkpoint>.pt \
  --tokenizer /home/iska/Documents/amelie/bio/TropicalGT/TropicalGT-I/data/toricgt/parameter_golf_convextok8192_biomed_det_full/tokenizers/fineweb_convextok_8192_biomed_det.convextok.json

The final model artifacts uploaded here will include the dated checkpoint, training config, muP base-shape file, run log, and supporting planning notes.

Example Prompts

Two nontrivial example prompts are included in this repository:

examples/enzyme_petase_design.prompt.json asks for three enzyme designs, including a PETase-like polyester hydrolase with explicit catalytic and kinetic targets and optional flow-matching structure trajectory output.
examples/de_novo_binder_panel.prompt.json asks for ten de novo protein binders against a difficult target interface, with graph records, Forest-of-Thought traces, analogical memory retrieval, persistent-homology gates, and optional multi-MODEL PDB trajectory output.

Run a prompt-conditioned FoT trace locally:

RUN_ID=toricblm-structure-priority-balanced-3day-20260709T185034Z
CKPT=checkpoints/${RUN_ID}/${RUN_ID}_epoch_002_special_structure_delta_step_002750.pt
TOK=/home/iska/Documents/amelie/bio/TropicalGT/TropicalGT-I/data/toricgt/parameter_golf_convextok8192_biomed_det_full/tokenizers/fineweb_convextok_8192_biomed_det.convextok.json

conda run -n tokengt env PYTHONPATH=src:external/mup:amelie-iska/parameter-golf \
  python scripts/export_oai_fot_trace.py \
  --checkpoint "$CKPT" \
  --tokenizer "$TOK" \
  --prompt "$(python - <<'PY'
import json
print(json.load(open('hf_model_cards/ToricGT_160M_FoT/examples/enzyme_petase_design.prompt.json'))['prompt'])
PY
)" \
  --output-json outputs/hf_examples/enzyme_petase_design/fot_trace.json

Structure denoising trajectories are exported as multi-MODEL PDB files with scripts/export_structure_flow_trajectory.py. These trajectories are flow-matching denoising paths, not molecular-dynamics simulations.

Relation to ToricBLM

This checkpoint line is intended as the scale-up bridge from ToricGT toward ToricBLM: a biomedical de novo design and scientific reasoning model built around universal equivariant graph-to-graph approximation, tropical-toric reasoning control, analogical memory retrieval, and Forest-of-Thought inference-time scaling. The model is trained for general graph-structured scientific reasoning and does not provide medical advice or patient-specific recommendations.

Limitations

This is an experimental research checkpoint line.
The current 170M run is outside the 16MB OpenAI Parameter Golf submission regime.
Mathematical auxiliary objectives are low-weight training signals and audits; they should not be interpreted as formal proof that every generated reasoning trajectory is algebraically valid.
Biomedical use requires separate safety evaluation and domain-specific validation.

AmelieSchreiber
/

ToricGT_160M_FoT