Update README.md

README.md

@@ -176,6 +176,78 @@ gen: arbazone 14, 13, 1-thiazolones4a-b and 15-cidine 2-thiazinoleic strain 60
 1024 floats carry a sentence's structure and content, but not arbitrary
 high-entropy strings.
 
+## Evaluation on general-domain text (SlimPajama, added 2026-09)
+
+Out-of-domain evaluation added after the SlimPajama models were trained: the same battery that scores them, run on this checkpoint unchanged.
+
+| | |
+|---|---|
+| teacher-forced cross-entropy (whole split, token-weighted) | **0.0141** |
+| perplexity | 1.0142 |
+| next-token accuracy | 99.58% |
+| exact reconstruction, 8,000 sentences (greedy) | **96.21%** |
+| character error rate (Levenshtein / length) | 0.69% |
+| word error rate | 1.00% |
+| cosine(enc(src), enc(reconstruction)) | 0.9988 |
+| reconstruction truncated at decoder max length | 0.00% |
+| exact reconstruction, 32 stress probes | **26/32** |
+| throughput, 8×H100 bf16 (encode / decode) | 15,098 / 1,016 sentences/s |
+
+By sentence length — reconstruction degrades past ~48 tokens, and the 4B encoder
+holds up markedly longer:
+
+| length (qwen3 tokens) | n | brivangl/qwenar-0.6b-montevideo exact / CER | brivangl/qwenar-4b-montevideo exact / CER | brivangl/qwenar-0.6b exact / CER |
+|---|---:|---:|---:|---:|
+| 0-15 | 3,064 | 100.0% / 0.00% | 100.0% / 0.00% | 99.9% / 0.00% |
+| 16-31 | 3,426 | 99.7% / 0.02% | 99.9% / 0.00% | 99.0% / 0.09% |
+| 32-47 | 1,282 | 97.0% / 0.40% | 99.3% / 0.07% | 87.6% / 1.65% |
+| 48-63 | 208 | 79.3% / 3.77% | 97.1% / 0.10% | 57.2% / 10.45% |
+| 64-79 | 14 | 7.1% / 31.85% | 28.6% / 13.36% | 7.1% / 44.94% |
+| 80-95 | 3 | 0.0% / 40.72% | 33.3% / 5.71% | 0.0% / 52.92% |
+| 96-111 | 3 | 0.0% / 42.76% | 0.0% / 24.60% | 0.0% / 50.54% |
+
+Prose vs. code (rule-based detector; code is 0.7% of the split):
+
+| content | n | brivangl/qwenar-0.6b-montevideo exact / CER | brivangl/qwenar-4b-montevideo exact / CER | brivangl/qwenar-0.6b exact / CER |
+|---|---:|---:|---:|---:|
+| prose | 7,941 | 98.7% / 0.23% | 99.6% / 0.04% | 96.3% / 0.65% |
+| code | 59 | 81.4% / 3.98% | 94.9% / 0.92% | 78.0% / 7.15% |
+
+The failure mode is the published one: the sentence frame survives while
+dense code syntax, identifiers, URLs and formula symbols scramble.
+
+```
+src: def flatten(xs): return [y for x in xs for y in (flatten(x) if isinstance(x, list) else [x])]
+gen: def flatten(xs): return [y for x in xs for y in (flatten(x) if isinstance(x, list) else [x])]
+
+src: fn parse(input: &str) -> Result<Vec<u32>, ParseIntError> { input.split(',').map(str::trim).map(str::parse).collect() }
+gen: fn parse(input) : result &<&Str<Vec<UInt32>, parseErrorIndex> { inp.split(',|map').str.parse((map).str.parse()::trim())
+     (Levenshtein 58, CER 49.15%)
+
+src: SELECT u.id, COUNT(o.id) AS orders FROM users u LEFT JOIN orders o ON o.user_id = u.id WHERE u.active GROUP BY u.id HAVING COUNT(o.id) > 3;
+gen: SELECT u.id, COUNT(o.id) AS orders FROM users u JOIN LEFT orders o ON u.o_id = u.user WHERE u.id AS ACTIVE GROUP BY u.id hOrderBy u.id; > 0)
+     (Levenshtein 46, CER 33.09%)
+
+src: func main() { ch := make(chan int, 8); go func() { ch <- 42 }(); fmt.Println(<-ch) }
+gen: func main() { chan := ch.make, int8} { go func 4(); } ch := 23() <- fmt.Println; (ca+)
+     (Levenshtein 34, CER 40.48%)
+```
+
+### Comparison with the SlimPajama-trained successors
+
+| model | encoder | d_emb | K | params | training data | steps | corpus seen | CE ↓ | tok acc ↑ | exact ↑ | CER ↓ | WER ↓ | cos ↑ | probes ↑ |
+|---|---|---:|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| brivangl/qwenar-0.6b-montevideo | pplx-embed-v1-0.6b | 1024 | 2 | 1194M | SlimPajama 9.2B sentences | 600,000 | 14% | 0.0047 | 99.85% | 98.61% | 0.26% | 0.39% | 0.9995 | 27/32 |
+| brivangl/qwenar-4b-montevideo | pplx-embed-v1-4b | 2560 | 4 | 4629M | SlimPajama 9.2B sentences | 1,000,000 | 9% | 0.0012 | 99.96% | 99.59% | 0.05% | 0.10% | 0.9998 | 28/32 |
+| **brivangl/qwenar-0.6b** | **pplx-embed-v1-0.6b** | **1024** | **2** | **1194M** | **PubMed / SlimPajama-6B / SYNTH mix** | **750,000** | **57% ¹** | **0.0141** | **99.58%** | **96.21%** | **0.69%** | **1.00%** | **0.9988** | **26/32** |
+
+All numbers on the same held-out SlimPajama split (200,000 sentences), same greedy decoding, bf16. CE / tok acc: teacher-forced over the whole split. exact / CER / WER / cos: free-running reconstruction of 8,000 sentences — CER = character Levenshtein / source length, WER = word Levenshtein / source words, cos = cosine between encoder embeddings of source and reconstruction. probes: exact reconstructions of 32 fixed stress sentences (long sentences, code, numbers, URLs).
+
+¹ this model; the successors ([`brivangl/qwenar-0.6b-montevideo`](https://huggingface.co/brivangl/qwenar-0.6b-montevideo), [`brivangl/qwenar-4b-montevideo`](https://huggingface.co/brivangl/qwenar-4b-montevideo)) were trained on the 9.2B-sentence
+SlimPajama corpus and are evaluated in-domain here. For general web text prefer
+them; this checkpoint remains the stronger choice only for biomedical text, on
+which the successors were not evaluated.
+
 ## Limitations
 
 Read this before drawing conclusions from the numbers above.