I know I can intercept the first 512 lengths.But I don’t want to do this. This task can understand the classification of texts. AutoModelForSequenceClassification can be directly used for classification, but the question is whether the input can be some 512 length sentences, and use pooler layer t…

Hey @ccfeidao you might want to try one of the dedicated models like LongFormer or BigBird which have a longer context size of around 4,096 tokens. See this thread for more details :slight_smile:

My input sentence is very long(more than 512). What should I do when I want to fintune model about classify?Thanks

Beginners

ccfeidao September 3, 2021, 4:49am 4

I want to user BigBird to try my task, but have a new problem, I am very Confusion.

Topic		Replies	Views
Transformer for very big text 🤗Transformers	1	727	May 6, 2022
Model for big context window Beginners	0	220	June 30, 2024
Token Classification Models on (Very) Long Text Models	8	11671	March 9, 2023
Text classification training on long text Intermediate	3	5501	June 18, 2024
Passing Inputs Longer Than 512 Tokens After Pretraining a T5 Model: Is It Safe? 🤗Transformers	3	228	November 20, 2025

My input sentence is very long(more than 512). What should I do when I want to fintune model about classify?Thanks

Related topics