Unable to load a FineTuned LLama Model to GPU for inference

bobbybelajar
Fri, 15 Dec 2023 09:11:40 +0000

My solution was to shard the merged model before uploading it Huggingface cloud. By doing this you can load the model into the memory with smaller chunks.