# Device\_map="auto"

**URL:** <https://discuss.huggingface.co/t/device-map-auto/49610>\
**Category:** Beginners\
**Created:** [August 5, 2023, 7:50pm UTC](https://discuss.huggingface.co/t/device-map-auto/49610 "2023-08-05T19:50:36Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kenkentron](https://avatars.discourse-cdn.com/v4/letter/k/6bbea6/32.png) [@Kenkentron](https://discuss.huggingface.co/u/Kenkentron)\
**Post date:** [August 5, 2023, 7:50pm UTC](https://discuss.huggingface.co/t/device-map-auto/49610/1 "2023-08-05T19:50:36Z")

</div>

I was wondering if this parameter is only set for inference as the document states ([Handling big models for inference](https://huggingface.co/docs/accelerate/usage_guides/big_modeling)) or does it actually have an effect during training?

Thanks!

---

<div class="post-metadata">

**Author:** ![hansekbrand](https://avatars.discourse-cdn.com/v4/letter/h/9de053/32.png) [@hansekbrand](https://discuss.huggingface.co/u/hansekbrand)\
**Post date:** [August 6, 2023, 5:49am UTC](https://discuss.huggingface.co/t/device-map-auto/49610/2 "2023-08-06T05:49:30Z")

</div>

It loads a model onto multiple GPUs. Once loaded, the model can be run forward or backward. I have only used ”auto” for training as of yet, and it works.

If you refer to this section:

> This only supports the inference of your model, not training. Most of the computation happens behind `torch.no_grad()` context managers to avoid spending some GPU memory with intermediate activations.

I think it only applies to the offloading to CPU or disk mechanism, but not when the full model can be loaded onto several GPUs.

---

<div class="post-metadata">

**Author:** ![Kenkentron](https://avatars.discourse-cdn.com/v4/letter/k/6bbea6/32.png) [@Kenkentron](https://discuss.huggingface.co/u/Kenkentron)\
**Post date:** [August 7, 2023, 2:51pm UTC](https://discuss.huggingface.co/t/device-map-auto/49610/3 "2023-08-07T14:51:30Z")

</div>

Thanks @hansekbrand that is helpful.

With more search, I think it became clear to me that device\_map=“auto” is doing naive MP for training (ref: [Make all Transformer models compatible with model parallelism · Issue #22561 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/issues/22561)).

---

<div class="post-metadata">

**Author:** ![muellerzr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/muellerzr/32/21583_2.png) [@muellerzr](https://discuss.huggingface.co/u/muellerzr)\
**Post date:** [August 7, 2023, 2:53pm UTC](https://discuss.huggingface.co/t/device-map-auto/49610/4 "2023-08-07T14:53:33Z")

</div>

> [@Kenkentron](#):
>
> is doing naive MP for training

Correct, any form of distributed training aside MP is not supported, and as of the next version will raise a proper error if you try to do so:

> <https://github.com/huggingface/accelerate/pull/1819>
>
> Since we're happy with the warning, expands it to include any call to \`prepare\` …(and \`prepare\_model\`) and refactors it to it's own function.
> 
> Will ensure that https://github.com/huggingface/accelerate/issues/1815 will raise the right err

---

<div class="post-metadata">

**Author:** ![Kenkentron](https://avatars.discourse-cdn.com/v4/letter/k/6bbea6/32.png) [@Kenkentron](https://discuss.huggingface.co/u/Kenkentron)\
**Post date:** [August 9, 2023, 5:04pm UTC](https://discuss.huggingface.co/t/device-map-auto/49610/5 "2023-08-09T17:04:09Z")

</div>

Thanks @muellerzr, could you also take a look at a related problem I have [ZeRO uses more RAM than DDP?](https://discuss.huggingface.co/t/zero-uses-more-ram-than-ddp/49830)?

---

<div class="post-metadata">

**Author:** ![vergilus](https://avatars.discourse-cdn.com/v4/letter/v/0ea827/32.png) [@vergilus](https://discuss.huggingface.co/u/vergilus)\
**Post date:** [September 25, 2024, 9:27am UTC](https://discuss.huggingface.co/t/device-map-auto/49610/6 "2024-09-25T09:27:22Z")

</div>

training mode does not support device\_map=“auto”, it will throw an error suggesting you load model on single device.