# Efficiently slicing dataset

**URL:** <https://discuss.huggingface.co/t/efficiently-slicing-dataset/28067>\
**Category:** 🤗Datasets\
**Created:** [December 18, 2022, 3:00pm UTC](https://discuss.huggingface.co/t/efficiently-slicing-dataset/28067 "2022-12-18T15:00:39Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![JustSaX](https://avatars.discourse-cdn.com/v4/letter/j/3be4f8/32.png) [@JustSaX](https://discuss.huggingface.co/u/JustSaX)\
**Post date:** [December 18, 2022, 3:00pm UTC](https://discuss.huggingface.co/t/efficiently-slicing-dataset/28067/1 "2022-12-18T15:00:40Z")

</div>

Hi,

What is the most efficient method of slicing a dataset? I have a dataset that only contains the input\_ids and attention mask for evaluating my model, both as list (not Tensor). I need to split the dataset into chunks as my GPU Memory is not enough to fit everything in one go.

I tried `ds.select(range(0,20))` and I tried `ds[0:20]`. Both operations take 2 seconds and even If I increase the size from 20 to 50 elements it takes 5 seconds. So basically the time increases linearly with the number of elements.

Is there a way to slice a dataset that is more time efficient?

Thanks!

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/64813_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [December 18, 2022, 11:14pm UTC](https://discuss.huggingface.co/t/efficiently-slicing-dataset/28067/2 "2022-12-18T23:14:25Z")

</div>

You can use `ds.select()`. Make sure to use a recent version of `datasets`, it has been optimized significantly and is almost instantaneous even for big datasets

---

<div class="post-metadata">

**Author:** ![JustSaX](https://avatars.discourse-cdn.com/v4/letter/j/3be4f8/32.png) [@JustSaX](https://discuss.huggingface.co/u/JustSaX)\
**Post date:** [December 22, 2022, 3:54pm UTC](https://discuss.huggingface.co/t/efficiently-slicing-dataset/28067/3 "2022-12-22T15:54:20Z")

</div>

Thanks for the help!

The problem was somewhere else in the end. The print statements in my code find out which expression takes the longest slowed down my code substantially 🤦‍♂️.

I learned my lesson 😄
