# Pretrain bert model using BOTH bert data custom data

**URL:** <https://discuss.huggingface.co/t/pretrain-bert-model-using-both-bert-data-custom-data/17309>\
**Category:** Beginners\
**Created:** [April 28, 2022, 7:28pm UTC](https://discuss.huggingface.co/t/pretrain-bert-model-using-both-bert-data-custom-data/17309 "2022-04-28T19:28:59Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Keyu](https://avatars.discourse-cdn.com/v4/letter/k/8491ac/32.png) [@Keyu](https://discuss.huggingface.co/u/Keyu)\
**Post date:** [April 28, 2022, 7:29pm UTC](https://discuss.huggingface.co/t/pretrain-bert-model-using-both-bert-data-custom-data/17309/1 "2022-04-28T19:29:00Z")

</div>

Hello, I wonder if there is a tutorial for incremental training bert model with custom data in pytorch. All I found on website is pretraining from scratch using only custom dataset. I also want to leverage bert dataset if thta’s possible.

Thanks!