Hello,
i’m looking for a small llm so it can be usea as a translator between the user and bigger llm
I’m Polish so trying some things for my small project
real good polish llm’s are too big for my laptop so i was thinking about this solution.
i found this one for example
Helsinki-NLP/opus-mt-pl-en
taki model znalazłem
us-mt-pl-en
source languages: pl
target languages: en
but is it translating only from PL to ENG ?
other way around not really?
some tips for me?
Thanks!
Raf
but is it translating only from PL to ENG ?
If you want to keep the model size as small as possible, I think it’s better to use a unidirectional translation.
There seem to be quite a few models that can interpret Polish, but if you try to do it with LLM, the model size is still quite large. It’s a realistic size, but it’s still large.
Thank You,
will check some thing and play arund.
R.
Hello
Thank You for reply.
I have a laptop i5 11400
16gb ram
GeForce NVIDIA rtx 3060 6 GB bram
I have found a polish model around 7b that works good with ollama
But with transformers and some of my code it’s quite slow compared to ollama
Thanks for feedback
R.
Ollama uses a 4-bit quantized model by default (i.e. it only uses a quarter of the RAM), and it runs smoothly in an environment with a mixture of CPU and GPU.
If you set the quantization in Transformers, it will be similar, but it’s hard to get used to.
Here are a few suggestions you can try:
opus-mt-en-pl model for bidirectional translation.MarianMT or TinyLlama fine-tuned for translation, which are smaller models.Let us updated with your progress then, cheers 
Thanks for advice i will check it out!
What do You mean to get used to it?
Lose of performance of the model?
Will check definitely
Thanks!
There are various quirks to the quantization of Transformers and Transformers, so you might have some trouble getting used to them.
Ohh ok
Thanks!
Have a nice one!
R.