How to use hugging face to fine-tune ollama's local model

nielsr
Fri, 10 May 2024 14:29:09 +0000

Hi,

File formats like GGUF are typically meant for inference on local hardware, see ggml/docs/gguf.md at master · ggerganov/ggml · GitHub.

For fine-tuning models, one typically uses one of the following libraries (in combination with GPU hardware):

Transformers, TRL, PEFT. These are libraries developed by HF making it very easy to fine-tune open-source models on your custom data.

Unsloth: GitHub - unslothai/unsloth: Finetune Llama 3.1, Mistral, Phi & Gemma LLMs 2-5x faster with 80% less memory. Claims to fine-tune models faster than the Transformers library.

Axolotl: GitHub - axolotl-ai-cloud/axolotl: Go ahead and axolotl questions. It works with a YAML-based configuration file that defines your entire fine-tuning. See e.g. axolotl/examples/llama-3 at main · axolotl-ai-cloud/axolotl · GitHub for llama-3

Torchtune: https://github.com/pytorch/torchtune: a new library developed by Meta for LLM fine-tuning.

After fine-tuning, the weights can be converted to the GGUF format, which allows local inference with ollama and llama cpp. See How to convert any HuggingFace Model to gguf file format? - GeeksforGeeks for a tutorial.

Additionally there’s Apple’s MLX library which allows to fine-tune LLMs on a Macbook. See mlx-examples/llms at main · ml-explore/mlx-examples · GitHub for fine-tuning LLMs.