I would like to suggest an experiment.
I am not a researcher or an expert in model training, so I cannot say whether this idea will work. But while experimenting with small local models, I kept wondering about one thing:
Why do we usually try to transfer knowledge directly from a very large model into a much smaller model?
For example, if the target model is 4B, trying to teach it directly from a 27B, 30B, or even larger model may create a very large gap between the teacher and the student.
It feels similar to asking a first-grade student to immediately study sixth-grade mathematics.
What if we tried something more gradual?
The experiment
Start with a small model, for example Qwen 4B.
Then progressively expose it to larger models:
Qwen 4B
↓
learn something from Qwen 8B or 9B
↓
improved 4B
↓
learn additional knowledge from Qwen 12B
↓
improved 4B
↓
learn additional knowledge from Qwen 27B
↓
improved 4B
↓
continue with a larger teacher
The important point is that the student remains the same small model.
The larger models are only temporary teachers.
After the 4B model has learned everything useful that it can from the 8B model, the 8B model is no longer needed.
Delete it.
Then bring in the next teacher.
The next teacher should ideally be used mainly to teach what the current 4B model has not already learned.
In simple terms:
4B learns from 8B
8B is removed
the improved 4B learns new things from 12B
12B is removed
the improved 4B learns new things from 27B
27B is removed
This is closer to how humans normally learn.
A second-grade student does not completely restart first-grade education.
Previously learned knowledge becomes the foundation for the next stage.
A Heretic-like approach
Another reason I think this might be interesting is the way projects such as Heretic perform repeated searches and keep trial histories.
A simplified version is:
search
→ test a modification
→ evaluate it
→ keep or reject it
→ record the result
→ continue
Heretic searches for directions that can be reduced or removed.
What if the same general idea were reversed?
search a larger teacher
→ find something useful that the small model appears to lack
→ test adding or reinforcing it
→ evaluate the result
→ keep or reject it
→ record the trial
→ continue
Instead of search and remove, this would be search and reinforce.
A JSONL checkpoint history could keep track of what has already been tested and learned.
That could become important when moving to the next teacher.
For example, if a useful direction or capability was already obtained from the 9B model, the search process could avoid spending large amounts of time rediscovering the same thing in the 27B model.
Why Qwen might be suitable
Qwen seems particularly interesting for this experiment because there are multiple model sizes within closely related model families.
That makes it possible to test progressive steps instead of jumping immediately from a tiny model to a huge one.
The first experiment does not need to be ambitious.
Something as simple as:
4B → learn from 9B
would already answer an important question:
Can a fixed-size 4B model become measurably better by selectively absorbing useful information from a somewhat larger model without simply retraining everything from scratch?
If that works, the next step could be:
the improved 4B → learn from 27B
The important measurement would not only be whether new abilities improve.
It would also be necessary to check whether abilities gained during earlier stages are preserved.
What I am suggesting
I am not claiming that a 4B model can somehow contain the full capability of a 30B model.
There is obviously a capacity limit.
What I am curious about is something more practical:
How much useful capability can a fixed-size small model accumulate if it learns progressively from increasingly capable teachers?
And:
Would this work better than asking a small student to learn directly from the largest teacher available?
Even a negative result could be interesting.
For example, perhaps a 4B model improves significantly after learning from 9B, improves slightly after 27B, and then reaches a clear limit.
That would tell us something useful about the practical knowledge capacity of a small model.
I would be very interested to see someone with more experience in model internals, distillation, pruning, representation analysis, or projects such as Heretic try an experiment along these lines.
The concept is simple:
Do not start with the largest teacher.
Let the small model grow step by step.