Error running Llama 3.1 Minitron 4B quantized model with Ollama

Donkey545
Wed, 28 Aug 2024 01:26:57 +0000

I have a similar problem with all of the Width based versions of this model. I managed to get the Q8 Depth Version to work with Ollama, but there appears to be something wrong with my setup because the model just hallucinates.

It appears that there is some work to get this problem addressed with some changes to Llama.cpp, but there may be some delay before those changes are rolled into Ollama.