# Yielding items from multiple datasets in parallel

**URL:** <https://discuss.huggingface.co/t/yielding-items-from-multiple-datasets-in-parallel/38467>\
**Category:** 🤗Datasets\
**Created:** [May 2, 2023, 8:48pm UTC](https://discuss.huggingface.co/t/yielding-items-from-multiple-datasets-in-parallel/38467 "2023-05-02T20:48:25Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![surya-narayanan](https://avatars.discourse-cdn.com/v4/letter/s/d2c977/32.png) [@surya-narayanan](https://discuss.huggingface.co/u/surya-narayanan)\
**Post date:** [May 2, 2023, 8:48pm UTC](https://discuss.huggingface.co/t/yielding-items-from-multiple-datasets-in-parallel/38467/1 "2023-05-02T20:48:25Z")

</div>

Hi,

I have several datasets, and want a dataloader that can sample from multiple datasets, so iterating over the dataloader yields `batch_size` number of items from each dataset.

Is that possible?

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [May 3, 2023, 2:43pm UTC](https://discuss.huggingface.co/t/yielding-items-from-multiple-datasets-in-parallel/38467/2 "2023-05-03T14:43:17Z")

</div>

Hi! You can use [interleave\_datasets](https://huggingface.co/docs/datasets/v2.12.0/en/process#interleave) for that and pass the returned dataset to the dataloader. Another option is to create one dataloader for each dataset and sample from them.

---

<div class="post-metadata">

**Author:** ![surya-narayanan](https://avatars.discourse-cdn.com/v4/letter/s/d2c977/32.png) [@surya-narayanan](https://discuss.huggingface.co/u/surya-narayanan)\
**Post date:** [May 9, 2023, 1:31am UTC](https://discuss.huggingface.co/t/yielding-items-from-multiple-datasets-in-parallel/38467/3 "2023-05-09T01:31:07Z")

</div>

Aah, I think `interleave_datasets` will yield `batch_size` items overall, from a mixture of datasets, whereas I want `batch_size` items from each dataset. Is that possible?

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [May 9, 2023, 1:41pm UTC](https://discuss.huggingface.co/t/yielding-items-from-multiple-datasets-in-parallel/38467/4 "2023-05-09T13:41:42Z")

</div>

`interleave_datasets` cycles through the given list of datasets, which means you can set the dataloader’s batch size to `batch_size` \* the number of interleaved datasets to get `batch_size` samples from each dataset in each iteration. Another option is to have a separate dataloader for each dataset.

---

<div class="post-metadata">

**Author:** ![tehranixyz](https://avatars.discourse-cdn.com/v4/letter/t/a9adbd/32.png) [@tehranixyz](https://discuss.huggingface.co/u/tehranixyz)\
**Post date:** [February 8, 2024, 12:19am UTC](https://discuss.huggingface.co/t/yielding-items-from-multiple-datasets-in-parallel/38467/5 "2024-02-08T00:19:36Z")

</div>

If we use a separate dataloader for each dataset, how the training loop will look like?  
for each epoch, how can we the batch from each one of the dataloaders and calculate the loss?  
Especially for the case where the length of dataloaders is not the same?