Hi, I just watched the video of the Workshop: Going Production: Deploying, Scaling & Monitoring Hugging Face Transformer models (11/02/2021) from Hugging Face. With the informations about how to deploy ( timeline start: 28:14 ), I created a notebook instance (type: ml.m5.xlarge) on AWS SageMaker whe…

for large DL models such as transformers, inference on CPU is slower than on GPU. And T5 is much bigger than the distillbert used in the demo. 700ms is actually not that bad for a CPU transformer :slight_smile: try replacing m5.xlarge by g4dn.xlarge to reduce latency.

Hello @OlivierCR . You are right about GPU vs CPU inference time but I’m doing tests with the same configuration for the 2 models (distilbert-base-uncased and T5 base). About models size, we are not talking here of large DL models. distilbert-base-uncased: 66 millions parameters ( fonte ) / Inferen…

Hey @pierreguillou , Thanks for opening the thread and I am happy to hear the workshop material was enough to get you started! So currently the models aren’t optimized automatically. So when you would like to run optimized models you would need to optimize them currently by yourself and then provid…

Hello @philschmid . Thanks for your answer. [image] philschmid: That’s because both models have different architecture and trained on different tasks and methods for inference. For example, T5 uses the .generate method with a beam search to create your translation, which means it is not running…

Upload the ONNX T5 base model to S3 in AWS You can use for example boto3 or the cli to upload files to S3 and you can find documentation on how the create the model.tar.gz here: Deploy models to Amazon SageMaker Use the ONNX T5 base model in AWS SageMaker DLC in order to make inferences Cur…

Hi @philschmid . I’m back to you about using AWS SageMaker for inference with a Text2Text-Generation model like T5. My objective is to use an ONNX T5 model for inference but in order to understand the logic behind the SageMaker Hugging Face Inference Toolkit , I started with a T5 model from the HF h…

@philschmid : very stange. I think I found how to pass parameters but when I pass the same parameters than the ones I used in a Colab notebook, I got 2 different predictions… Code from my Colab notebook model_name = "xxx" API_TOKEN = 'xxxx' # API token max_target_length = 32 num_beams = 1 text2te…

@pierreguillou when using generative models it is not guaranteed that the output is always exactly the same. Especially when converting the model to a ONNX Model. What different output are you seeing? Are you using the same tranformers version?

[image] philschmid: @pierreguillou when using generative models it is not guaranteed that the output is always exactly the same Hi @philschmid . I’m with difficulty to understand your sentence. When I have a model (generative or not), the same text in input, and of course the same values for a…

[image] pierreguillou: Last point : in AWS Sagemaker Inference, I did not use the ONNX model but the Pytorch T5 model. As I saw a different output, it means that the DLC for inference does some compression on T5 (ONNX or similar?) that could explain the different output? No SageMaker is not d…

How to deploy a T5 model to AWS SageMaker for fast inference?

Amazon SageMaker

GenV February 28, 2022, 2:38pm 13

Hi @pierreguillou, I have the same problem. I created a post here. Did you solve the problem?

Topic		Replies	Views
Deploying open llm - google/flan-t5-large model on AWS inferentia2 Amazon SageMaker	0	460	September 14, 2023
Deploying T5-style models via Sagemaker Endpoint: 'T5LayerFF' object has no attribute 'config' Amazon SageMaker	5	1508	November 7, 2022
Help for inference.py code Amazon SageMaker	10	4135	March 8, 2022
Inference Hyperparameters Amazon SageMaker	29	5045	October 8, 2021
Deploying Open AI's whisper on Sagemaker Amazon SageMaker	54	16538	April 12, 2024

How to deploy a T5 model to AWS SageMaker for fast inference?

Related topics