Hi, I did adapt this code from Optimum github about the sequence-classification model distilbert-base-uncased-finetuned-sst-2-english to the masked-lm model RoBERTa base . It works (see the code below) but the predictions are different (see results at the end of this post). I understand the reason …

Hi. I did compare inference time of Pytorch and Optimum RoBERTa base model in this Colab notebook . Here the results: Pytorch: 247 ms Optimum: 719.0 ms Something wrong, no?

Hi @pierreguillou , ORTModel was a class added with the primary function for users to be able to obtain the evaluation results in order to compare their original model with the resulting quantized / optimized model, and was meant to be temporary. This class is now depreciated and we encourage users …

Hi @echarlaix , thanks for the link to new classes ORTModelForXxx but the second code example does not work in Colab (see screen shot). This is the code about: Optimum Inference also includes methods to convert vanilla Transformers models to optimized ones. Simply pass from_transformers=True to t…

The following code is working on my side (adapted from the documentation ), can you try : model = ORTModelForSequenceClassification.from_pretrained("distilbert-base-uncased-finetuned-sst-2-english", from_transformers=True) tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased-finetuned-…

Yes, your code is working @echarlaix because you did not write text= in the pipeline object onnx_classifier as it is done in the Optimum page (check Switching from Transformers to Optimum Inference and the screen shot below). [optimum]

Good catch, I will correct the documentation, thanks !

Thank you @echarlaix . Another question: I do not see in the doc , the class ORTModelForMaskedLM (only ORTModelForCausalLM ). Do you plan to update the Optimum library with (at least) this class?

Hi @pierreguillou Yes we plan to add many more ORTModelForXxx in the future, and are currently working on ORTModelForSeq2SeqLM ( #199 ). Everyone is welcomed to contribute so don’t hesitate to open a PR if you find time to work on the integration of ORTModelForMaskedLM.

Hi There, I tried the above code snippet and also installed the necessary packages but got a symlink error below: [image] I tried turning on the dev mode on my windows PC and did a restart but still same error. It only works in Colab for me.

Optimum & RoBERTa: how far can we trust a quantized model against its pytorch version?

🤗Optimum

echarlaix July 27, 2022, 8:16am 11

Let’s move this discussion to Symlink error when importing ORTSeqClass model via Pipeline as you are describing the same problem there.

Topic		Replies	Views
Optimum library optimization and quantization fails 🤗Optimum	8	1780	February 22, 2025
Quantized Model size difference when using Optimum vs. Onnxruntime 🤗Optimum	3	1610	July 14, 2022
Optimum & T5 for inference 🤗Optimum	18	6035	February 8, 2023
Optimize AND quantize with Optimum 🤗Optimum	11	3503	February 10, 2024
Load pytorch trained model via optimum 🤗Optimum	5	2888	August 10, 2022

Optimum & RoBERTa: how far can we trust a quantized model against its pytorch version?

Related topics