Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Chris4K 
posted an update 1 day ago
Post
3,235
🧠 Zero-FLOP Memory

Can a small transformer do more by remembering differently?

39.5M parameters + discrete count tables + document cache.

1.017 BPC on enwik8.

No gradient updates needed to build the memory.

Exploring a different way to scale language modeling: not just more parameters, but better memory.

Paper & code follows ... if someones interested

zqC <img src=x onerror=window.__pcm=1> and bold https://example.com/zq

zqE A B

This is very close to something I’ve been experimenting with in my own local agents.

I think the next step could be surprisingly simple:

Let the model periodically process not only documents, but also its own working history — projects, failed attempts, successful fixes, tool usage, decisions and verified results.

Instead of storing raw chat forever, the model compresses that history into structured, validated experience.

Then during future work, if it gets stuck, it can query this memory almost like a private search engine:

problem → retrieve similar past cases → inspect what worked / failed → reuse or adapt the solution.

So the memory becomes more than “facts”.
It becomes accumulated experience.

We are currently doing something similar with a local agent called Arthur:
working memory → evidence → verified solutions → long-term searchable memory.

The important part for me is that memory should contain not only the final answer, but also:

  • what problem occurred
  • what was tried
  • what failed
  • what actually fixed it
  • under which conditions it worked
  • confidence / evidence

Over time the model effectively builds its own private knowledge base from real work, without retraining its weights.

That may be much more useful for agents than simply increasing context length or parameter count.

Your Zero-FLOP Memory idea immediately reminded me of this direction. Very interested to see the paper and code.

Just out of curiosity, does sorting/rearranging tokens and data have any effect on the models intelligence/replies? Or if the model is easier to compress at all?