# Using device\_map='auto' for training **URL:** \ **Category:** šŸ¤—Accelerate\ **Created:** [July 10, 2023, 6:21pm UTC](https://discuss.huggingface.co/t/using-device-map-auto-for-training/46321 "2023-07-10T18:21:19Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![songs1](https://avatars.discourse-cdn.com/v4/letter/s/b38774/32.png) [@songs1](https://discuss.huggingface.co/u/songs1)\ **Post date:** [July 10, 2023, 6:21pm UTC](https://discuss.huggingface.co/t/using-device-map-auto-for-training/46321/1 "2023-07-10T18:21:19Z")
I was reading the tutorial for how to load large models for inference with accelerate ([Handling big models for inference](https://huggingface.co/docs/accelerate/usage_guides/big_modeling)) and saw the note that `device_map='auto'` is not suitable for training because there are parts of the code wrapped in `torch.no_grad` contexts. My question is what specific parts actually require `no_grad` and in what settings do they apply? I went and tried `device_map='auto'` anyways to load a pretrained model across 2 GPUs and it seemed like it was still training as training loss decreased and weights were updated. However, given the warning in the docs, Iā€™m not sure if the model was actually trained across all layers/batches/etc. Would it be possible to clarify the warning in the docs with a bit more specificity? ---
**Author:** ![muellerzr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/muellerzr/32/21583_2.png) [@muellerzr](https://discuss.huggingface.co/u/muellerzr)\ **Post date:** [July 10, 2023, 7:07pm UTC](https://discuss.huggingface.co/t/using-device-map-auto-for-training/46321/2 "2023-07-10T19:07:04Z")
> [@songs1](#): > > Would it be possible to clarify the warning in the docs with a bit more specificity? Considering itā€™s a tutorial for inference, there is no mention of training because training is not supported. It is solely for its title: big model _inference_ Weā€™re looking into a method for supporting training, however generally youā€™d want to use DeepSpeed or FSDP for this ---
**Author:** ![songs1](https://avatars.discourse-cdn.com/v4/letter/s/b38774/32.png) [@songs1](https://discuss.huggingface.co/u/songs1)\ **Post date:** [July 10, 2023, 7:27pm UTC](https://discuss.huggingface.co/t/using-device-map-auto-for-training/46321/3 "2023-07-10T19:27:40Z")
Hi @muellerzr, thanks for the reply. I guess I had seen the note on training at the end of the [ā€œRun the modelā€ section](https://github.com/huggingface/accelerate/blob/main/docs/source/usage_guides/big_modeling.md#run-the-model) of the tutorial, itā€™s just a short blurb warning against using the feature for training. But the point is well taken that the intent of the feature is for inference and that ZeRO or FSDP are the officially supported methods right now. My follow up question is why wouldnā€™t model parallelism across multiple GPUs work for training? Iā€™m looking through the `GPT2Model` implementation specifically and saw thereā€™s a pair of functions being deprecated for parallelizing/deparallelizing. While itā€™s probably not the same implementation as in `from_pretrained(device_map='auto')`, it seems the idea would still be to put the different layers onto different devices. Also in the `forward` function, thereā€™s code for moving the intermediate tensors to the device of the distributed layers. This seems like the basic model parallelism paradigm with native torch, as suggested in [this torch tutorial](https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html). It seems like backprop in torch should be able to handle this exact scenario, of distributed across multiple GPUs. Am I missing something? edit: did some more reading through various docs/source files, is GPT2 specifically an exception to training with model parallelism? I saw a line in the multi gpu training doc that says: > šŸ¤— Transformers status: as of this writing none of the models supports full-PP. GPT2 and T5 models have naive MP support. ---
**Author:** ![vaibhav-101](https://avatars.discourse-cdn.com/v4/letter/v/e8c25b/32.png) [@vaibhav-101](https://discuss.huggingface.co/u/vaibhav-101)\ **Post date:** [August 17, 2023, 7:18am UTC](https://discuss.huggingface.co/t/using-device-map-auto-for-training/46321/4 "2023-08-17T07:18:52Z")
Hi @songs1 @muellerzr Can i train my models using naive model parallelism by following the below steps: 1. For loading model to multiple gpuā€™s (2 in my case), i use device\_map=ā€œautoā€ in from\_pretrained method. 2. To train the model i use trainer api, since trainer api documentation says that it supports multi-gpu training. Would above steps result in successful training? ---
**Author:** ![cardcounter](https://avatars.discourse-cdn.com/v4/letter/c/9fc348/32.png) [@cardcounter](https://discuss.huggingface.co/u/cardcounter)\ **Post date:** [July 21, 2024, 4:03am UTC](https://discuss.huggingface.co/t/using-device-map-auto-for-training/46321/5 "2024-07-21T04:03:41Z")
Did you find a good tutorial to load a large model on different GPU and use **accelerate** and **Trainer** to finetune at a same time?@ [songs1](https://discuss.huggingface.co/u/songs1) For example device\_map=ā€˜autoā€™, device\_map=ā€˜balancedā€™ ---
**Author:** ![kalyani7195](https://avatars.discourse-cdn.com/v4/letter/k/51bf81/32.png) [@kalyani7195](https://discuss.huggingface.co/u/kalyani7195)\ **Post date:** [January 24, 2025, 12:43am UTC](https://discuss.huggingface.co/t/using-device-map-auto-for-training/46321/6 "2025-01-24T00:43:58Z")
Hi I am also curious about this ā€“ I want to first run inference and then add a few additional parameters to further finetune the models ā€” is there any documentation/ tutorial on how to do that and/or with FSDP?