Freedream-1B-Instruct-V2-preview

Models
1

Hello guys,
This is an preview version for Freedream-1B-Instruct-V2, it is trained based on the Freedream-1B 24B tokens seen base model where the tokenizer vocabulary became 32001 that fixes the problem of ‘{’ being tokenized as unk token, special thanks to @John6666 , and the model is now available on: Digilidaahz/Freedream-1B-Instruct-V2 · Hugging Face, any techinical suggestions are appreciated.
Thanks for reading, have a good day.

2

Hi. I can confirm that the { issue we found earlier is fixed on my end too:

I gave the published V2 Preview tokenizer a separate CPU-only check, using fixed files from the V1 repository and the new V2 repository. The particular regression that prompted the earlier discussion is repaired: the V1 tokenizer encoded an isolated opening brace as [178, 0], with ID 0 being its unknown token, while the V2 tokenizer encodes { as the single token [32000] and decodes it back to {.

That is a real and useful improvement. In the V2 Preview announcement, you said the vocabulary had become 32,001 specifically to address this, and this narrow check agrees with that description. I also tried two tokenizers releases, 0.22.1 and 0.23.3, in isolated CPU environments; the results agreed for the tested inputs.

There is one adjacent detail that might be useful for a future tokenizer revision, though. Some strings containing the newly registered punctuation characters now acquire spaces inside the decoded text. This is separate from whether the missing { can be represented, and it is also separate from whether the model can produce correct code or JSON.

Here are a few short input/round-trip pairs from the released V2 tokenizer. I am showing the literal decoded text, including a possible leading space:

Original input Decoded after V2 encode/decode Why it is interesting
{ { The original missing character is now preserved.
{"a":1} { "a":1} A space appears after the opening brace, but this JSON still parses to the same value.
left{right left{ right An additional space appears inside ordinary text.
C:\Users\test\file.json C:\ Users\ test\ file.json A path-like string is no longer character-for-character identical.
https://example.org/a?b=1&c=2 https://example.org/a?b=1& c=2 A query-string-like fragment changes internally.

The initial leading space in some rows needs to be interpreted separately. The published tokenizer uses a ByteLevel pre-tokenizer with add_prefix_space=True; its ordinary leading-space behavior already existed in V1. The interesting new part here is the additional spacing at internal punctuation boundaries. The ByteLevel documentation explains the prefix-space option, while the AddedToken documentation describes the separate matching properties of registered tokens.

A low-cost default check

The cheapest next step is a small tokenizer-only regression panel: {, a simple JSON object, an embedded brace, a path with backslashes, and one URL or literal escape. For each, retain the token IDs and the exact decoded string. Check JSON parsing separately from exact-text equality. The source can be pinned and checked on CPU, without loading model weights or retraining.

When this matters

  • If only parseable JSON values matter, the whitespace in the two selected JSON examples is allowed; it need not be a blocker.
  • If exact code, paths, URLs, or escapes matter, internal spaces deserve a regression check.
  • If preserving the already-trained V2 checkpoint matters, test ID compatibility before changing tokenizer settings.
  • If building a new tokenizer and model together, byte coverage and round-trip checks fit naturally before training.

This is a raw-tokenizer test, not a V2 generation-quality benchmark. The diagnostic inputs are deliberately selected edge cases, so their counts are not real-world error rates. The details below give pinned files, reproducible commands, the structural diff, and the limits of the conclusions.

Pinned tokenizer files, environment, and a minimal reproduction

1. What I tested, and what I did not

This is a raw-tokenizer artifact comparison, not a model-generation run.

The two input files were fetched at fixed Hugging Face revisions:

Artifact Fixed revision Size SHA-256
V1 tokenizer_v2.json f0abc549c5584749b3e59193694728f1a7416983 2,292,448 bytes 6663b6059da6bc6e2edc2b12571af8c668a1b6eacb65cae81dab28fd90856434
V2 tokenizer_v2.json 6cbfa34ce45ce28d31bf841652eeb30cf4465c64 2,295,407 bytes 4ad58e6c2bf1f1ec5bfd0f27098f9c76d0553b983fa6f16fa0d92d24ea234c16

These hashes identify the JSON files used in this check. They do not establish the identity or behavior of any model checkpoint, and they are not hashes of the currently mutable main branch. A future reader should either use these pinned files or explicitly state that a newer file is being tested.

The returned CPU run completed under Python 3.13.18 on Linux. It tested tokenizers 0.22.1 and 0.23.3 independently and found identical results for the 216 ordinary test inputs. There was also a separate chat-format reference input in the input collection, intentionally excluded from that 216-case numerical summary. This distinction matters if someone attempts to compare counts from a modified panel.

The method was:

  1. Load each tokenizer from the serialized JSON.
  2. Use a ByteLevel decoder, matching the relevant low-level V1 loading path.
  3. Call encode(text, add_special_tokens=False).
  4. Preserve the raw numeric token IDs and token strings.
  5. Decode those IDs with skip_special_tokens=False so that unknown-token markers remain visible.
  6. Also inspect the default decode where useful, because skipping special tokens can hide evidence of an unknown token.
  7. Classify only what the raw sequence actually shows, rather than inferring a cause from a rendered string.

This is an encode/decode round-trip experiment. In particular, there is no inference model in the loop. Even if a tokenizer round trip is perfect, a model can still output incorrect JSON; conversely, a model may generate valid JSON in a particular trial even if selected input strings do not round-trip exactly.

1.1 A small reproduction that needs no model weights

The following standalone Python example is intended for a fresh CPU environment with Internet access to the two pinned JSON files. The larger diagnostic run used the same basic lower-level operations; this shorter presentation focuses on a handful of useful cases. The hash checks are deliberate: they stop a quiet repository update from being mistaken for the same test.

python -m pip install "tokenizers==0.23.3"

import hashlib
import json
from pathlib import Path
from urllib.request import urlopen

import tokenizers
from tokenizers import Tokenizer
from tokenizers.decoders import ByteLevel

SOURCES = {
    "v1": (
        "https://huggingface.co/Digilidaahz/Freedream-1B-Instruct-V1/"
        "resolve/f0abc549c5584749b3e59193694728f1a7416983/tokenizer_v2.json",
        "6663b6059da6bc6e2edc2b12571af8c668a1b6eacb65cae81dab28fd90856434",
    ),
    "v2": (
        "https://huggingface.co/Digilidaahz/Freedream-1B-Instruct-V2/"
        "resolve/6cbfa34ce45ce28d31bf841652eeb30cf4465c64/tokenizer_v2.json",
        "4ad58e6c2bf1f1ec5bfd0f27098f9c76d0553b983fa6f16fa0d92d24ea234c16",
    ),
}

tokenizers_by_version = {}
for label, (url, expected_sha256) in SOURCES.items():
    with urlopen(url, timeout=60) as response:
        data = response.read()
    actual_sha256 = hashlib.sha256(data).hexdigest()
    assert actual_sha256 == expected_sha256, (label, actual_sha256)
    path = Path(label + "_tokenizer_v2.json")
    path.write_bytes(data)
    tok = Tokenizer.from_file(str(path))
    tok.decoder = ByteLevel()
    tokenizers_by_version[label] = tok

print("tokenizers", tokenizers.__version__)
print("v1 brace ID:", tokenizers_by_version["v1"].token_to_id("{"))
print("v2 brace ID:", tokenizers_by_version["v2"].token_to_id("{"))

cases = [
    "{",
    '{"a":1}',
    '{"ok":true,"nested":{"x":2}}',
    "left{right",
    r"C:\Users\test\file.json",
    r"https://example.org/a?b=1&c=2",
    r'"\n" and "\u1234"',
]

for text in cases:
    print("\nINPUT:", repr(text))
    for label, tok in tokenizers_by_version.items():
        encoded = tok.encode(text, add_special_tokens=False)
        decoded = tok.decode(encoded.ids, skip_special_tokens=False)
        print(
            label,
            "ids=", encoded.ids,
            "decoded=", repr(decoded),
            "exact=", decoded == text,
            "prefix_only=", decoded == " " + text,
            "contains_id_0=", 0 in encoded.ids,
        )
        if text in cases[1:3]:
            try:
                same_json = json.loads(decoded) == json.loads(text)
            except json.JSONDecodeError:
                same_json = False
            print("   same JSON value:", same_json)

The code intentionally performs the JSON-value comparison only on the two complete JSON objects, not the lone opening brace or the other test strings. A larger regression harness can label cases explicitly, but the point of keeping this snippet compact is that a reader can see and adapt every moving part.

Only the two approximately 2.3 MB tokenizer JSON files are needed; model weights, PyTorch, and a GPU are not required.

1.2 Decoder visibility and the scope of the check

The V1 brace encoding contains ID 0. With skip_special_tokens=False, the ByteLevel decode exposes the unknown marker; with the default True it can disappear from displayed text. The raw IDs are therefore essential. The V2 brace result has no unknown ID. See the Tokenizer API for the decode option.

This experiment tested the two JSON tokenizer artifacts only. No V2 weights, inference wrapper, or GPU generation were included.

Concrete round trips, classifications, and what the numbers mean

2. Representative observations with the published files

The table near the top was deliberately short. Here is a more explicit comparison of selected cases. For Markdown table readability, UNK[0] in the V1 cells is a display abbreviation for the actual <|unk|> marker (whose ID is 0); it is not a literal output substring. Other characters, including spaces, match the recorded decoded text. The space after the opening quote in a repr representation indicates a leading U+0020 space.

Test input V1 decoded with markers visible V2 decoded with markers visible
{ ' UNK[0]' '{'
{"a":1} ' UNK[0]"a":1}' '{ "a":1}'
left{right ' leftUNK[0]right' ' left{ right'
C:\Users\test\file.json ' C:\Users\test\file.json' ' C:\ Users\ test\ file.json'
https://example.org/a?b=1&c=2 ' https://example.org/a?b=1&c=2' ' https://example.org/a?b=1& c=2'
"\n" and "\u1234" ' "\n" and "\u1234"' ' "\ n" and "\ u1234"'

The first three rows show the tradeoff in a particularly compact way. V1 could not preserve the opening brace in those selected contexts, while V2 can. At the same time, the string that follows the new brace token may be decoded with a space that was not present originally. A change can be a net improvement on one axis while leaving a separate, narrower question on another axis.

The path, URL, and escape examples show why the second axis deserves its own test. A human reader may barely notice a space, especially in a rendered forum post. A program consuming the exact string may notice it immediately. That does not mean the tokenizer is necessarily corrupting every path or every URL; these are precisely identified examples, not a general rate.

2.1 The JSON examples have a useful positive control

For the simple JSON input:

{"a":1}

the published V2 tokenizer’s round trip is:

{ "a":1}

Both strings parse to the same JSON value. The space is permitted there. Calling the latter invalid JSON would be factually wrong.

The same distinction holds for one tested nested example:

{"ok":true,"nested":{"x":2}}

whose decoded V2 text was:

{ "ok":true,"nested":{ "x":2}}

In this selected case, the decoded result is still valid JSON with an equivalent parsed value. What fails is character-for-character identity, not necessarily format validity. There could be applications where exact JSON bytes are part of a signature, comparison, or stored format, and there could be ordinary applications that only care about the parsed object. Those applications should not be assigned the same pass/fail rule.

This is also why a JSON-only test would miss some of the interesting behavior. Successful parsing of the particular object above would not reveal the changed Windows path or the new space after & in the URL-like example.

2.2 The diagnostic panel and its counts

For a broader diagnostic view, I compared 216 deliberately selected ordinary strings. They included punctuation combinations, braces, JSON and code-like fragments, paths, escapes, URLs, and a few controls. An additional chat-format reference item was kept separate from the numerical panel. Each ordinary case was placed into one mutually exclusive category:

  • exact: the decoded string equals the original.
  • prefix_only: the decoded string equals one leading space plus the original.
  • other_change: there is a further difference beyond the simple prefix-space pattern, with no unknown-token ID in that case.
  • unk: the encoded ID sequence includes unknown-token ID 0. This category takes precedence even when the decoded display also differs.

Under those definitions, the published files gave:

Published tokenizer Exact Prefix-only Other change Contains unknown token Total
V1 2 203 0 11 216
V2 Preview 22 167 25 2 216

This table is not a tokenizer error rate for ordinary writing. A hand-built panel with many punctuation and edge cases is intentionally biased toward the features under investigation. For example, the large prefix-only count is mainly evidence that the selected strings exercised the existing ByteLevel prefix-space behavior; it is not evidence that almost every natural input in the world is broken.

It is better to read this table as a compact map of the panel: some unknown-token cases were repaired, some strings now round-trip exactly, some previously prefix-only strings acquire internal changes, and a couple of deliberately included control cases still contain unknown tokens.

Across the same 216 inputs, V1 and V2 produced different encoded ID sequences for 45 cases and different decoded strings for 45 cases. That is a comparison of two concrete JSON files under a defined harness, not a claim that the model’s performance changed on exactly those 45 tasks.

The V1-to-V2 category transitions are informative if someone wants to audit the classification:

V1 category V2 category Cases
Exact Exact 2
Prefix-only Exact 17
Prefix-only Prefix-only 167
Prefix-only Other internal change 19
Contains unknown Exact 3
Contains unknown Other internal change 6
Contains unknown Contains unknown 2

The transition table is not a second independent experiment; it is simply another accounting of the same 216 inputs. It is useful because an aggregate such as “nine unknown-token cases fixed” does not tell you whether each now round-trips exactly. Some do, and some have the separate whitespace difference.

2.3 Two selected controls

The V2 panel still contains unknown ID 0 for the selected carriage-return input a\rb and NUL input a\0b (shown here in escape notation; the actual inputs contained control characters). These are specific controls, not evidence that ordinary V2 JSON requires an unknown token. The nine repaired unknown-ID cases and these two remaining controls should be reported separately.

Tokenizer structure, diagnostic variants, and compatibility with trained weights

3. What changed inside the serialized tokenizer

The V2 announcement describes a vocabulary of 32,001. The loaded V2 tokenizer reports that total, and the new brace occupies ID 32,000. Looking inside the serialized artifacts gives a useful additional distinction: the core BPE model.vocab still contains 32,000 entries in both the pinned V1 and V2 files. The top-level added_tokens list, however, grew from five entries to 22.

That is not the same as adding 17 new vocabulary IDs. The new list contains one new opening-brace ID and 16 newly registered entries for punctuation characters that already had IDs in the BPE vocabulary. Those existing characters can be routed through AddedToken matching without creating a new embedding index for each of them.

Property Published V1 Published V2 Preview
Entries in BPE model.vocab 32,000 32,000
Top-level AddedToken entries 5 22
Loaded vocabulary size, with added tokens 32,000 32,001
Direct ID for isolated { No dedicated ID 32,000
Isolated brace encode result [178, 0] [32000]

For clarity, the 17 newly recorded content/ID pairs were:

# 7     $ 8     % 9     & 10
< 32    > 34    @ 36    [ 63
\ 64    ] 65    ^ 66    _ 67
grave-accent 68    | 95    } 96    ~ 97
{ 32000

The new AddedToken records have special:false, lstrip:false, rstrip:false, and normalized:true. These are matching and decode-related properties, not a statement that the characters are control tokens for chat. The Tokenizers AddedToken API describes their behavior. Among other things, it distinguishes whether a token is treated as special from whether it consumes whitespace on either side.

Why does this distinction matter? The BPE vocabulary tells us which IDs exist, while the added-token registrations also influence how strings are matched into token spans. The ByteLevel pre-tokenizer and decoder affect how those spans become bytes and displayable characters. A plain count of vocabulary entries cannot tell us whether a punctuation boundary has changed.

This structural difference is a clue worth checking, not a proof about the author’s code. A published tokenizer JSON is an output artifact. It does not tell us exactly which training call, token registration operation, or serialization step produced every entry. I have not inspected a complete tokenizer-training script for this V2 release, so I would not identify a particular original training mistake as the root cause.

3.1 In-memory variants help separate two changes

For diagnosis only, I constructed copies of the V2 JSON in memory and loaded those copies through the same low-level Tokenizer interface. These were never published as alternative releases or used with the V2 model weights.

  • Brace-only variant: Retain the old V1 AddedToken entries plus V2’s new { entry; remove the other 16 newly registered punctuation entries.
  • Other-added-only variant: Retain the new punctuation registrations but remove the new { registration.
  • No-prefix variant: Retain V2’s registrations but set the ByteLevel pre-tokenizer’s add_prefix_space flag to False.

The same 216 deliberately selected cases produced:

Variant Exact Prefix-only Other internal change Contains UNK
Published V1 2 203 0 11
Published V2 22 167 25 2
V2 brace-only diagnostic 4 203 7 2
V2 other-added-only diagnostic 19 167 19 11
V2 no-prefix diagnostic 214 0 0 2

With the other 16 registrations removed but the brace retained, the number of internally changed cases falls from 25 to seven. That makes it reasonable to investigate the interaction between those registrations and ByteLevel boundaries. It does not establish a single exact implementation-level cause, and it does not make removal a safe patch for an existing model.

The complementary variant, which removes the brace registration while leaving the other punctuation registrations, returns to 11 unknown-token cases. This helps confirm that the explicit brace registration materially contributes to the repaired coverage in the selected panel. At the same time, it does not restore every other detail of V1 behavior, because it still contains the other new registrations.

These are controlled comparisons of tokenizer configurations, not controlled comparisons of pretrained models. No training or model forward pass was performed for the variants. Their value is diagnostic isolation.

3.2 A surprisingly good round trip is not automatically a compatible fix

The no-prefix variant deserves special caution. It makes 214 of the 216 selected strings round-trip exactly; only the two deliberately selected control-character cases remain in the unknown-ID category. On a pure text-fidelity chart, that looks attractive.

But the same comparison also showed that setting add_prefix_space=False changed the encoded ID sequences for 194 of the 216 cases relative to the released V2 tokenizer. Removing the other 16 registrations changed IDs for 40 cases. Removing only the brace registration changed IDs for nine cases.

Those ID changes mean that a downstream model can receive a different sequence of learned symbols even when the final decoded text looks more faithful. The checkpoint was not trained under our temporary diagnostic settings, and these variants were not tested against the V2 weights. In that setting, declaring the no-prefix variant a drop-in fix would cross an evidence boundary.

Future tokenizer design and inexpensive regression checks

4. If a new tokenizer is part of a future training run

The existing checkpoint and a new tokenizer trained from scratch have different constraints. The former needs compatibility checks; the latter allows design-time choices without pretending they are drop-in fixes for the released V2 weights.

4.1 Future tokenizer training has a different set of options

If a new tokenizer is being designed before a future training run, it may be worth testing more of the byte alphabet at construction time. The ByteLevel documentation says its alphabet covers 256 byte values through a visible-character mapping. The BPE Trainer documentation provides an initial_alphabet option, and the official training-from-memory example demonstrates supplying pre_tokenizers.ByteLevel.alphabet() to a trainer.

One new-build candidate, assuming the rest of a genuinely new tokenizer is designed around it, looks like:

from tokenizers import Tokenizer, decoders, models, pre_tokenizers, trainers

new_tokenizer = Tokenizer(models.BPE(unk_token="<|unk|>"))
new_tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=True)
new_tokenizer.decoder = decoders.ByteLevel()

trainer = trainers.BpeTrainer(
    vocab_size=32000,
    initial_alphabet=pre_tokenizers.ByteLevel.alphabet(),
    special_tokens=["<|unk|>"],
)

# Train with the new project's actual selected data and full token inventory:
# new_tokenizer.train_from_iterator(training_text_iterator, trainer=trainer)
# Then test the serialized tokenizer before pairing it with new model weights.

This is not a recipe for silently rewriting the published V2 tokenizer. The author’s full special-token setup, normalization policy, token IDs, chat delimiters, and embedding/output dimensions would have to be designed consistently. I have not verified which trainer options produced the original Freedream tokenizer, so I cannot say that this one option was missing or that it would have solved every present issue.

The design question can be framed simply: for the next tokenizer built together with a new model, do we want complete required-byte coverage and exact handling of the target formats to be tested before training starts? If yes, an alphabet and round-trip test is inexpensive insurance. For an already-trained checkpoint, the same question is constrained by compatibility with learned IDs.

Even then, more coverage does not automatically mean better model-generated code. It removes a representational limitation; it does not create reasoning ability, syntax awareness, or training examples. Those other aspects must be assessed independently.

4.2 A save/reload and version check can prevent confusing comparisons

It is useful to test the file the user actually downloads, not only an in-memory Tokenizer object just before it is saved. A construction script may append registrations or change pre-tokenizer settings before serialization; readers see only the serialized result.

For a future tokenizer release, a lightweight release check could:

  1. Save the tokenizer to its intended JSON file.
  2. Load it through the same public interface the author expects users to call.
  3. Run the tiny edge-case panel on the loaded file.
  4. Record its SHA-256 and the tokenizers version.
  5. Compare an actual inference prompt’s encoded IDs if a checkpoint already exists.
  6. Keep the corresponding model revision and tokenizer revision together in the reproducibility notes.

This avoids confusing a changed library wrapper, a saved-file change, a new model checkpoint, and a changed evaluation prompt. It also helps another forum reader reproduce the exact observation without chasing whatever main happens to contain at the time.

None of these checks requires evaluating all 216 diagnostic strings in every development iteration. That larger panel was useful for mapping the change, but a much smaller curated subset can be sufficient for preventing the specific regression from returning.

Relevant documentation, similar upstream reports, and the limits of this feedback

5. Related upstream discussions: similar symptoms are not identical causes

A particularly close historical report is huggingface/tokenizers issue #1174, “add_prefix_space adds a space after added tokens”. Its reproducer describes a ByteLevel tokenizer with an added token and unexpected whitespace after that token. This is useful context for the general interaction between AddedToken boundaries and prefix-space processing.

There is an important status detail: that issue was closed as completed in January 2024, with a collaborator indicating it had been fixed. It would be misleading to call #1174 a currently open, unfixed upstream bug or to imply that its exact original implementation defect is present in V2. The pinned V2 JSON still exhibited the selected round-trip differences under tokenizers 0.22.1 and 0.23.3; that is a separate present-day observation about this artifact. The historical issue is a comparison point, not a substitute for reproducing this artifact.

Another partially related report is huggingface/transformers issue #28218, which discusses additional spaces after added tokens in a Transformers-facing tokenizer setting. It provides more evidence that similar boundary symptoms have appeared in other stacks. It should not be used to assert the exact same root cause in this raw tokenizers.Tokenizer experiment.

The tokenizers issue #1044 discussion is useful for a different reason: it highlights compatibility concerns around changing prefix-space behavior for already pretrained tokenizers. That thread deals with Metaspace/SentencePiece-related behavior rather than this ByteLevel case. I cite it as a related compatibility caution, reinforced here by the direct 194/216 input-ID change measurement, not as a bug report about the Freedream model.

The official references are more appropriate for what the API actually guarantees:

These references serve different roles. Documentation states the supported API concepts. Issue reports illustrate phenomena seen in particular code and versions. The pinned Freedream JSON and CPU results establish the actual observations here. None should be substituted for another.

5.1 Scope and optional next evidence

This CPU result does not establish V2 model-generated JSON or code quality, the author’s training root cause, or any overall model ranking. The older V1 generation probes used another checkpoint and cannot be relabeled as V2 results. A matched V2 generation probe may be useful if structured-output ability becomes the question, but it is not needed to confirm the tokenizer artifact behavior.

For future readers, it is enough to retain the file revision, library version, raw IDs, skip_special_tokens setting, a few exact input/decoded pairs, and the contract being tested. A successful JSON parse, a faithful round trip, and a correct model answer remain distinct results.

Thanks again for putting the V2 Preview out and for following up on the brace issue. The concrete repair is easy to confirm, which is a nice outcome. I hope the short round-trip examples and the compatibility distinction are useful as reference points for a later revision, whenever they fit the direction you want to take the model.